warnings spamming logs: "wrong number of labels for upstream peer, empty labels will be used instead" in nginx-plus ingress controller #5607
Description
Describe the bug
our access logs are spammed with these warnings in nginx-plus ingress controller :
"wrong number of labels for upstream peer, empty labels will be used instead"
NOTE: THIS BEHAVIOUR IS NOT AVAILABLE IN NGINX INGRESS CONTROLLER( FREE/OSS) ONLY IN NGINX -PLUS INGRESS CONTROLLER
=================================================================================================
origin of this error: https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/collector/nginx_plus.go#L838
level.Warn(c.logger).Log("msg", "wrong number of labels for upstream, empty labels will be used instead", "upstream", name, "expected", len(c.variableLabelNames.UpstreamServerVariableLabelNames), "got", len(varLabelValues))
=================================================================================================
To Reproduce
Steps to reproduce the behavior:
- Deploy 3.4.3/3.5.0/3.5.1 of nginx-plus ingress controller.
- View access.logs: K logs -f
- See error:
=================================================================================================
evel=warn ts=2024-05-23T08:06:54.604Z caller=nginx_plus.go:838 msg="wrong number of labels for upstream, empty labels will be used instead" upstream=xxxxxxxxxxxxx expected=4 got=0
level=warn ts=2024-05-23T08:06:54.604Z caller=nginx_plus.go:849 msg="wrong number of labels for upstream peer, empty labels will be used instead" upstream=xxxxxxx-dev peer=10.x.x.x :443 expected=1 got=0
level=warn ts=2024-05-23T08:06:54.604Z caller=nginx_plus.go:838 msg="wrong number of labels for upstream, empty labels will be used instead" upstream=xxxxxxxxx-dev expected=4 got=0
level=warn ts=2024-05-23T08:06:54.604Z caller=nginx_plus.go:849 msg="wrong number of labels for upstream peer, empty labels will be used instead" upstream=xxxxxxxxxx-dev peer=10.x.x.x expected=1 got=0
level=warn ts=2024-05-23T08:06:55.138Z caller=nginx_plus.go:838 msg="wrong number of labels for upstream, empty labels will be used instead" upstream=xxxxxxxxxxxx-dev expected=4 got=0
level=warn ts=2024-05-23T08:06:55.138Z caller=nginx_plus.go:849 msg="wrong number of labels for upstream peer, empty labels will be used instead" upstream=xxxxxxxxxxx-dev peer=10.x.x.x expected=1 got=0
level=warn ts=2024-05-23T08:06:55.138Z caller=nginx_plus.go:838 msg="wrong number of labels for upstream, empty labels will be used instead" upstream=xxxxxxxxxx-dev expected=4 got=0
level=warn ts=2024-05-23T08:06:55.138Z caller=nginx_plus.go:849 msg="wrong number of labels for upstream peer, empty labels will be used instead" upstream=xxxxxxxxxx-dev peer=10.x.x.x expected=1 got=0
===================================================================================
Expected behavior
our access.logs should have actual access.logs which is working but these warnings are spamming our logs
NOTE: THIS BEHAVIOUR IS NOT AVAILABLE IN NGINX INGRESS CONTROLLER( FREE/OSS)
Your environment
- Version of the Ingress Controller - release version : nginx-plus ingress controller - 3.4.3/3.5.0/3.5.1
- Version of Kubernetes:
Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.13
WARNING: version difference between client (1.28) and server (1.26) exceeds the supported minor version skew of +/-1 - Kubernetes platform (e.g. Mini-kube or GCP)
- Using NGINX Plus
Additional context
Add any other context about the problem here. Any log files you want to share.