-
|
Description: I deployed Jaeger all-in-one to a Kubernetes cluster with Cassandra as storage backend. Some applications are using OTLP exporters to send traces to Jaeger, but I'm encountering the following error in the logs: Environment:
Configuration Attempted: I've tried increasing the following collector parameters, but the issue persists: - name: COLLECTOR_QUEUE_SIZE
value: "300000"
- name: COLLECTOR_NUM_WORKERS
value: "10000"
- name: COLLECTOR_QUEUE_SIZE_MEMORY
value: "500000"
- name: COLLECTOR_BATCH_SIZE
value: "1000"Observations: When checking Jaeger metrics, the Question: Given that the queue metrics show no accumulation, what other approaches or configurations could help resolve this "sending queue is full" error? Are there other queues or bottlenecks I should investigate? Any insights or suggestions would be greatly appreciated. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments
-
|
Hi @figureair! I'm Dosu and I’m helping the jaeger team. The "sending queue is full" error comes from the OpenTelemetry Collector's internal queue, not Jaeger's legacy collector queue. The To resolve or diagnose this, focus on tuning the OpenTelemetry Collector pipeline inside Jaeger all-in-one. Key areas to check:
Also, check for bottlenecks in Cassandra by monitoring Jaeger's write latency metrics ( Common pitfalls: YAML syntax errors, incorrect exporter names (use If you need config examples or want to share your pipeline YAML, I can help review it for queue/batch tuning. To reply, just mention @dosu. How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other |
Beta Was this translation helpful? Give feedback.
-
|
With AI assistance, I identified the key bottleneck in the Here's the modification I made: exporterhelper.WithQueue(exporterhelper.QueueBatchConfig{
Enabled: true,
NumConsumers: 10,
QueueSize: 10000,
}), |
Beta Was this translation helpful? Give feedback.
With AI assistance, I identified the key bottleneck in the
newSpanProcessormethod, which has a default configurationexporterhelper.WithQueue(exporterhelper.NewDefaultQueueConfig()). This queue defaults to 1000, so I tried customizing the queue length to 10000. After rebuilding, the "sending queue is full" errors from other applications disappeared. Of course, this may not be the best practice - theoretically, reducing the application's sampling rate would be the correct approach.Here's the modification I made: