Google-cloud-spanner:Error in connection java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception #3889
Description
I am trying to run my application which connects to google cloud spanner in a yarn container and the application gets blocked. Upon further investigation I found that the following error was being thrown
java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Environment details
- OS: linux
- Java version: 1.8
- google-cloud-java version(s): google-spanner-0.52-beta
The code I am trying to execute:
Spanner spannerClient=SpannerOptions.newBuilder().build().getService();
InstanceAdminClient instanceAdminClient = spannerClient.getInstanceAdminClient();
Page<Database> databasePage = instanceAdminClient.getInstance(instanceID).listDatabases();
for (Database database : databasePage.getValues()) {
String databaseName = database.getId().getDatabase();
log.info("Database-->"+databaseName);
}This code gets blocked while running in yarn container. Upon inspecting I got the following thread dump:
sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(int, long) @bci=139, line=1037 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(int, long) @bci=25, line=1328 (Interpreted frame)
- java.util.concurrent.CountDownLatch.await(long, java.util.concurrent.TimeUnit) @bci=10, line=277 (Interpreted frame)
- com.google.cloud.spanner.SpannerImpl.backoffSleep(io.grpc.Context, long) @bci=67, line=200 (Interpreted frame)
- com.google.cloud.spanner.SpannerImpl.backoffSleep(io.grpc.Context, com.shaded.scanner.google.api.client.util.BackOff) @bci=5, line=170 (Interpreted frame)
- com.google.cloud.spanner.SpannerImpl.runWithRetries(java.util.concurrent.Callable) @bci=105, line=238 (Interpreted frame)
- com.google.cloud.spanner.SpannerImpl$InstanceAdminClientImpl.getInstance(java.lang.String) @bci=25, line=660 (Interpreted frame)
- com.test.spannerscanner.utils.GoogleSpannerManager.execute() @bci=289, line=93 (Interpreted frame)
Upon further debugging I found the runWithRetries method in SpannerImpl is throwing SpannerException due to which the code is getting blocked. I examined the SpannerException and in the detailed message I found
java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
The code works fine in my local environment but in yarn container it gets blocked due to the SpannerException. Any guidance in how to resolve the issue is greatly appreciated.