Open
Description
At the moment, relying on open telemetry tools to monitor token usage is not possible
due to the fact that not all open telemetry standard fields are set by the telemetry module.
Solution:
Include standard telemetry fields on spans, as recommended by the open telemetry standard
https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-events/
For example, the following fields are not set, despite recommended by the otel standard:
gen_ai.request.max_tokens
gen_ai.request.top_p
gen_ai.usage.output_tokens
gen_ai.usage.input_tokens
gen_ai.response.finish_reasons
** happy to send a pr **