Description
Discovered in the Otel Draft PR: #2500
Issue
For Otel, this is causing the number of attempts to always be recorded as 1 instead of the number of retries actually attempted. It does not impact retries being invoked, but will record incorrect metrics.
Steps for creating a UnaryCallable (1 is the inner most callable and 5 is the last callable invoked):
HttpJson Echo Unary:
- HttpJsonDirectCallable
- HttpJsonUnaryRequestParamCallable (optional -- created if params exist)
- TracedUnaryCallable
- HttpJsonExceptionCallable
- RetryingCallable
gRPC Echo Unary:
- GrpcDirectCallable
- GrpcUnaryRequestParamCallable (optional -- created if params exist)
- GrpcExceptionCallable
- RetryingCallable
- TracedUnaryCallable
TracedUnaryCallable
Inside the TracedUnaryCallable, the callback is invoked once in innerCallable has been invoked. The callback implementation is the TraceFinisher.
From the implementation, it'll record an operation success or failure, which is intended to invoked after all the retries are exhausted (as the retries are attempts).
Possible Solution
We should move creating the TracedUnaryCallable to the last step for HttpJson. This would require refactoring this file: https://github.com/googleapis/sdk-platform-java/blob/7902a41c87240d607179d07c28cce462ea135c5f/gax-java/gax-httpjson/src/main/java/com/google/api/gax/httpjson/HttpJsonCallableFactory.java
Concerns
Would this impact existing behavior? Do we need to do this for the other Callable types (StreamingCallables?)