Debugging a Quarkus gRPC blocking issue with database transactions
I was working on a Quarkus application with uses gPRC and Hibernate JPA. The code flow is quite straightforward. OrderGrpcService
is the annotated gPRC service implementation. Its createOrder
method calls createOrder
method of OrderService
, which is annotated with @Transactional
. The gPRC method is annotated with @io.smallrye.common.annotation.Blocking
. When running in the dev mode, the gRPC request failed with io.quarkus.runtime.BlockingOperationNotAllowedException
error with message Cannot start a JTA transaction from the IO thread
. The error message is very clear. When debugging the createOrder
method of OrderService
, it shows the current thread is an I/O thread from Vert.x.
After doing some searches, it's clearly that using @Blocking
is enough to work with transactions in gPRC. I also created a simple application with minimal dependencies. That application worked, so it's not an issue of Quarkus itself. So I dug into Quarkus source code.
The support of @Blocking
is implemented as a gPRC server interceptor, io.quarkus.grpc.runtime.supports.blocking.BlockingServerInterceptor
. This interceptor offloads the execution of the gRPC method on a worker thread if the method is annotated with @Blocking
.
To debug this interceptor, I started Quarkus dev server in suspended mode. This allows debugging of Quarkus deployment extensions.
quarkus dev --suspend
I added a breakpoint of the gatherGrpcInterceptors method in GrpcServerProcessor
. It turned out that BlockingServerInterceptor
was not included in the list of interceptors. That's why the blocking behavior was not enabled.
When searching for usage of BlockingServerInterceptor
, it showed that it would be added by GrpcServerRecorder when there were methods annotated with @Blocking
in the gPRC service. During the debugging, it turned out that the blockingMethodsPerService
map was empty, so Quarkus didn't find any blocking methods in gRPC service classes. The blockingMethodsPerService
map was initialized in the initializeGrpcServer
method of GrpcServerRecorder
.
The initializeGrpcServer
method of GrpcServerRecorder
was called in the initializeServer
method of GrpcServerProcessor
. Blocking methods were discovered by checking the List<BindableServiceBuildItem>
object which contained all BindableServiceBuildItem
s. But the list was empty.
BindableService
s were discovered in the discoverBindableServices method of GrpcServerProcessor
by calling getAllKnownImplementors(GrpcDotNames.BINDABLE_SERVICE)
method of org.jboss.jandex.IndexView
. The root cause was that BindableService
s were not found in the Jandex index.
Then I realized that the gRPC service was actually defined in another order-service-api
module, not the current order-service
. Although Jandex Maven plugin was configured in the order-service-api
module, after I checked the target\classes
directory, I couldn't find the META-INF\jandex.idx
file. This was the root cause of this issue. Although the gPRC service OrderGrpcService
was defined in the current order-service
module, its parent class OrderServiceImplBase
was generated in another module. When no Jandex index for the order-service-api
module, the OrderServiceImplBase
couldn't be found, thus its subclass OrderGrpcService
was not discoverable.
After running mvn install
for the order-service-api
module, the issue was resolved.