8000 gauge.set_function() doesn't work in multiprocess mode · Issue #504 · prometheus/client_python · GitHub
[go: up one dir, main page]
Skip to content

gauge.set_function() doesn't work in multiprocess mode #504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
MatthewMaclean opened this issue Jan 29, 2020 · 6 comments
Open

gauge.set_function() doesn't work in multiprocess mode #504

MatthewMaclean opened this issue Jan 29, 2020 · 6 comments

Comments

@MatthewMaclean
Copy link

Multiprocess mode's collect() reads the registry files and aggregates metrics that have been written to prometheus_multiproc_dir.

This doesn't work with gauge.set_function() which does not record its value. The provided function is just called during collection.

That means with the following code:

registry = CollectorRegistry()
Gauge("test", "test", registry=self._registry).set_function(lambda: 100)
multiprocess.MultiProcessCollector(registry)

The output will be:

# HELP test test
# TYPE test gauge
test 100.0
# HELP test Multiprocess metric
# TYPE test gauge
test{pid="10705"} 0.0

Current side effects:

  • If any mode other than all or liveall is used, the pid tag won't be included. This results in duplicate metrics being reported to Prometheus. Prometheus currently only uses the first metric it reads, which is non-deterministic due to iteration over the registry's dictionary.
  • If registry=None to avoid double reporting, only the default value of 0.0 is reported.
  • Current way to work around it is to use the mode all and to ignore gauges in Prometheus and with the tag pid.

Proposal:
I'm not sure how you could incorporate set_function into the multiprocess registry and I'm not convinced how useful of a feature it would be. Is it reasonable to add a new multiprocess_mode: exclude which would prevent the incorrect 0.0 value being reported? Or would it be better to just add documentation to recommend using two independent registries?
@brian-brazil
Copy link
Contributor

This classifies as a custom collector, which it's not possible to make work with multiprocess mode.

@MatthewMaclean
Copy link
Author

Makes sense, I'll look/ask about ways to work around that in the mailing list.

MatthewMaclean added a commit to MatthewMaclean/client_python that referenced this issue Jan 30, 2020
@MatthewMaclean
Update README.md
4572f68 
Clarifying in the Multiprocess Mode instructions that gauge.set_function is also a custom collector.
prometheus#504
MatthewMaclean added a commit to MatthewMaclean/client_python that referenced this issue Jan 30, 2020
@MatthewMaclean
gauge.set_function is a custom collector
4fc4590 
Clarifying in the Multiprocess Mode instructions that gauge.set_function is also a custom collector.
prometheus#504
@MatthewMaclean MatthewMaclean mentioned this issue Jan 30, 2020
Closed
@hdost
Copy link
hdost commented Dec 6, 2022

Given that multiprocess is getting some traction is it worth re-opening this?

@hdost hdost reopened this Dec 6, 2022
@csmarchbanks
Copy link
Member

I'm still not sure how this could be done in multiprocess mode. You would need to run the function for each process which is not a pattern we have today and would need to integrate with whatever is starting the process I believe. If there is a simpler way I am all ears, but otherwise I think it should probably be kept as closed.

@hdost
Copy link
hdost commented Dec 18, 2022

Perhaps just documentation or a failure logged. Right now you just have to wonder

@csmarchbanks
Copy link
Member

👍 Documentation on the function is a good idea to avoid confusion.

aapeliv added a commit to aapeliv/client_python that referenced this issue Mar 19, 2025
@aapeliv
Add note on gauge.set_function not working with mp, see prometheus#504
a983b66
@aapeliv aapeliv mentioned this issue Mar 19, 2025
Merged
aapeliv added a commit to aapeliv/client_python that referenced this issue Mar 19, 2025
@aapeliv
Add note on gauge.set_function not working with mp, see prometheus#504
8777cb3 
Signed-off-by: Aapeli <git@aapelivuorinen.com>
csmarchbanks pushed a commit that referenced this issue Mar 20, 2025
@aapeliv
Add note on gauge.set_function not working with mp, see #504 (#1098)
23ab826 
Signed-off-by: Aapeli <git@aapelivuorinen.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hdost @MatthewMaclean @csmarchbanks @brian-brazil
0