Multiline from python app in k8s #6542
Description
Bug Report
Describe the bug
Hi.
I need to configure multiline parsing for python app in k8s env.
To Reproduce
- Example log message if applicable (taken from
kubectl logoutput):
2022-12-13 13:42:33.663 | INFO | uvicorn.protocols.http.httptools_impl:send:441 - 1.1.1.1:443 - "GET /api HTTP/1.1" 500
Exception in ASGI application
Traceback (most recent call last):
File "/opt/.venv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 376, in run_asgi
...
2022-12-13 13:42:33.688 | DEBUG | elasticapm.transport.base:_process_queue:165 - flushing due to time since last flush 9.536s > max_flush_time 9.536s
- Steps to reproduce the problem: apply the config :)
Expected behavior
One document in opensearch for multiline log instead of separated documents.
Screenshots
Your Environment
- Version used: helm chart 0.20.3
- Configuration:
config:
customParsers: |
[MULTILINE_PARSER]
name multiline-regex-test
type regex
flush_timeout 1000
# rules | state name | regex pattern | next state
# ------|---------------|-----------------------------------
rule "start_state" "/{\"log\":\"Traceback.+\"}/" "cont"
rule "cont" "/{\"log\":\"[^\d].+\"}/" "cont"
key_content log
[MULTILINE_PARSER]
name multiline-regex-test1
type regex
flush_timeout 1000
# rules | state name | regex pattern | next state
# ------|---------------|-----------------------------------
rule "start_state" "/Traceback/" "cont"
rule "cont" "/^[^\d].+/" "cont"
key_content log
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*app*.log
multiline.parser docker, cri, multiline-regex-test1
Log_Level debug
Tag app.*
Mem_Buf_Limit 5MB
Buffer_Chunk_Size 5MB
Buffer_Max_Size 5MB
Mem_Buf_Limit 50MB
Skip_Long_Lines Off
filters: |
[FILTER]
Name kubernetes
Match app.*
Kube_Tag_Prefix app.var.log.containers.
Merge_Log On
Keep_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser Off
K8S-Logging.Exclude Off
outputs: |
[OUTPUT]
Name opensearch
Match app.*
Host XYZ
Port 443
Logstash_Format On
Logstash_Prefix app
Logstash_DateFormat %Y.%m.%d
Suppress_Type_Name On
tls On
Retry_Limit False
Replace_Dots On
HTTP_User user
HTTP_Passwd password
- Environment name and version (e.g. Kubernetes? What version?): kubernetes, v1.20
Additional context
I tried to set multiline.parser multiline-regex-test and it provides the ugly output (each line contains "log" word) but multiline works:
I tried different variants but it still doesn't work