Multiline python parser breaks on caret (^) lines in stacktraces, causing split log records #11516
Description
Bug Report
Describe the bug
The built-in python multiline parser in the multiline filter prematurely terminates Python stacktraces when encountering caret-only (^) marker lines.
As a result, a single Python traceback is ingested as multiple separate log records instead of a single merged record.
This occurs even when:
- the
tailinput uses thecriparser - the
multilinefilter is configured withparser: python - logs are properly CRI-parsed and the
logfield contains only the application message
Caret-only lines (e.g. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^) cause the multiline state machine to exit unexpectedly.
To Reproduce
- Example log message if applicable:
2026-03-02T13:23:38.768156672+00:00 stderr F 2026-03-02 13:23:38,767 ERROR yaook.op.daemon.infra.yaook.cloud/v1.mysqlusers.yaook.nova-api-api-5g8hd failed to reconcile state <yaook.op.infra.resources.MySQLUser component='user'>
2026-03-02T13:23:38.768156672+00:00 stderr F Traceback (most recent call last):
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/site-packages/yaook/statemachine/statemachine.py", line 81, in _ensure_state
2026-03-02T13:23:38.768156672+00:00 stderr F await state.reconcile(
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/site-packages/yaook/statemachine/resources/external.py", line 194, in reconcile
2026-03-02T13:23:38.768156672+00:00 stderr F await self.update(ctx, dependencies)
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/site-packages/yaook/op/infra/resources.py", line 482, in update
2026-03-02T13:23:38.768156672+00:00 stderr F await loop.run_in_executor(
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
2026-03-02T13:23:38.768156672+00:00 stderr F result = self.fn(*self.args, **self.kwargs)
2026-03-02T13:23:38.768156672+00:00 stderr F ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/site-packages/yaook/op/infra/resources.py", line 428, in _update_user
2026-03-02T13:23:38.768156672+00:00 stderr F with connect_mysql(connection_template) as conn:
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/contextlib.py", line 137, in __enter__
2026-03-02T13:23:38.768156672+00:00 stderr F return next(self.gen)
2026-03-02T13:23:38.768156672+00:00 stderr F ^^^^^^^^^^^^^^
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/site-packages/yaook/op/infra/resources.py", line 87, in connect_mysql
2026-03-02T13:23:38.768156672+00:00 stderr F conn = pymysql.connect(
2026-03-02T13:23:38.768156672+00:00 stderr F ^^^^^^^^^^^^^^^^
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 365, in __init__
2026-03-02T13:23:38.768156672+00:00 stderr F self.connect()
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 680, in connect
2026-03-02T13:23:38.768156672+00:00 stderr F self._get_server_information()
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 1099, in _get_server_information
2026-03-02T13:23:38.768156672+00:00 stderr F packet = self._read_packet()
2026-03-02T13:23:38.768156672+00:00 stderr F ^^^^^^^^^^^^^^^^^^^
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 751, in _read_packet
2026-03-02T13:23:38.768156672+00:00 stderr F packet_header = self._read_bytes(4)
2026-03-02T13:23:38.768156672+00:00 stderr F ^^^^^^^^^^^^^^^^^^^
2026-03-02T13:23:38.768156672+00:00 stderr F File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 805, in _read_bytes
2026-03-02T13:23:38.768156672+00:00 stderr F raise err.OperationalError(
2026-03-02T13:23:38.768156672+00:00 stderr F pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')- Steps to reproduce the problem:
- Configure Fluent Bit with
tailinput andcrimultiline parser to match thecripattern:tail: bufferChunkSize: 256K bufferMaxSize: 5M db: /fluent-bit/tail/tail-yaook-operators-containers.db dbSync: Normal excludePath: /var/log/containers/*fluent-bit*.log, memBufLimit: 50MB multilineParser: cri,python path: /var/log/containers/*_yaook_operator-*.log pathKey: log_path refreshIntervalSeconds: 10 rotateWaitSeconds: 300 storageType: filesystem tag: kube.tail.yaook.operators.*
- Add a
multilinefilter to (try to) concatenate the traceback:filters: - multiline: buffer: true emitterMemBufLimit: 10 emitterType: memory flushMs: 2000 keyContent: log parser: python
Expected behavior
The entire Python traceback (including caret marker line) should be merged into a single log record.
Caret-only lines ( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^) are a valid and common part of Python error output and should not termiante the multiline state machine.
Screenshots
(As an example, does not exactly reflect the above log lines)
Your Environment
- Version used: Fluent Bit v4.2.3
- Configuration: tail input and multiline filter (see above).
- Environment name and version (e.g. Kubernetes? What version?):
- Kubernetes
Client Version: v1.32.7 Kustomize Version: v5.5.0 Server Version: v1.31.10
- Fluent Bit 4.2.3 with Fluent Operator 3.4.2
- Kubernetes
- Server type and version: containerd (CRI logs)
- Operating System and version: Ubuntu 22.04
- Filters and plugins:
- tail (cri)
- multiline (python)
Additional context
This issue affects production Kubernetes environments where Python-based operators and controllers emit stacktraces.
Fragmented stacktraces:
- Increase document count in OpenSearch
- Break readability in log viewers
- Complicate alerting and correlation
- Require workarounds (custom multiline parser or dropping caret lines)
Caret marker lines are common in modern Python error formatting and should be supported by the built-in python multiline parser.
Reviewing the history of related issues (#9290, #7478, #5333, #6542), I see that several were automatically closed, but this problem still appears to persist for years now! I am curious how others are handling Python tracebacks in production environments.
It seems unlikely that we are the only operators encountering multiline Python stacktraces containing caret marker lines.