Debugging a Quarkus gRPC blocking issue with database transactions

Debugging a Quarkus gRPC blocking issue with database transactions

I was working on a Quarkus application with uses gPRC and Hibernate JPA. The code flow is quite straightforward. OrderGrpcService is the annotated gPRC service implementation. Its createOrder method calls createOrder method of OrderService, which is annotated with @Transactional. The gPRC method is annotated with @io.smallrye.common.annotation.Blocking. When running in the dev mode, the gRPC request failed with io.quarkus.runtime.BlockingOperationNotAllowedException error with message Cannot start a JTA transaction from the IO thread. The error message is very clear. When debugging the createOrder method of OrderService, it shows the current thread is an I/O thread from Vert.x.

After doing some searches, it's clearly that using @Blocking is enough to work with transactions in gPRC. I also created a simple application with minimal dependencies. That application worked, so it's not an issue of Quarkus itself. So I dug into Quarkus source code.

The support of @Blocking is implemented as a gPRC server interceptor, io.quarkus.grpc.runtime.supports.blocking.BlockingServerInterceptor. This interceptor offloads the execution of the gRPC method on a worker thread if the method is annotated with @Blocking.

To debug this interceptor, I started Quarkus dev server in suspended mode. This allows debugging of Quarkus deployment extensions.

quarkus dev --suspend

I added a breakpoint of the gatherGrpcInterceptors method in GrpcServerProcessor. It turned out that BlockingServerInterceptor was not included in the list of interceptors. That's why the blocking behavior was not enabled.

When searching for usage of BlockingServerInterceptor, it showed that it would be added by GrpcServerRecorder when there were methods annotated with @Blocking in the gPRC service. During the debugging, it turned out that the blockingMethodsPerService map was empty, so Quarkus didn't find any blocking methods in gRPC service classes. The blockingMethodsPerService map was initialized in the initializeGrpcServer method of GrpcServerRecorder.

The initializeGrpcServer method of GrpcServerRecorder was called in the initializeServer method of GrpcServerProcessor. Blocking methods were discovered by checking the List<BindableServiceBuildItem> object which contained all BindableServiceBuildItems. But the list was empty.

BindableServices were discovered in the discoverBindableServices method of GrpcServerProcessor by calling getAllKnownImplementors(GrpcDotNames.BINDABLE_SERVICE) method of org.jboss.jandex.IndexView. The root cause was that BindableServices were not found in the Jandex index.

Then I realized that the gRPC service was actually defined in another order-service-api module, not the current order-service. Although Jandex Maven plugin was configured in the order-service-api module, after I checked the target\classes directory, I couldn't find the META-INF\jandex.idx file. This was the root cause of this issue. Although the gPRC service OrderGrpcService was defined in the current order-service module, its parent class OrderServiceImplBase was generated in another module. When no Jandex index for the order-service-api module, the OrderServiceImplBase couldn't be found, thus its subclass OrderGrpcService was not discoverable.

After running mvn install for the order-service-api module, the issue was resolved.

© 2022 VividCode