-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(server): add more prometheus metrics #11089
base: main
Are you sure you want to change the base?
feat(server): add more prometheus metrics #11089
Conversation
Where could I get the user created metric? |
Probably immich/server/src/cores/user.core.ts Line 28 in d23aa5e
|
Ok, I thought there was a better option. Ty |
} | ||
this.metricRepository.host.addToGauge(`immich.host.photos`, serverStats.photos) | ||
this.metricRepository.host.addToGauge(`immich.host.videos`, serverStats.videos) | ||
this.metricRepository.host.addToGauge(`immich.host.usage`, serverStats.usage) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Btw @mertalev, am I being stupid or you cannot send metrics without creating "sums"
Currently this keeps adding up the one photo I have. I can't use a Counter because it can decrease and the Gauge is adding stuff.
I am very noob in Otel, I am used to send prometheus data with bash or python directly :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not as easy to do it for persistent data like that. Options:
- Increment whenever an asset gets created or removed at the server level (might be difficult to keep in sync with actual)
- Query Postgres for the data (accurate, but counting in Postgres is very slow since it's a full table scan, so doing it all the time is a very bad idea).
- Have a trigger in Postgres that increments values in a separate counting table when assets are added or removed (fast retrieval for metrics, but makes insertions and deletes slower, not sure by how much).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exactly, I'll think about it! It is missing that on my grafana
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But to just set it in this handler and not automatically, I'm sure you can set the gauges to explicitly be those values. The metric repo doesn't expose everything from the underlying otel library.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I think the up/down counter is different from a gauge. This should be what you're looking for.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried that but then it got stuck on 1... I need to try again. Thank you
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also need to find an endpoint that gets called more often (that one is just when the admin opens that panel) or create a cron for that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can add it to the nightly jobs here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can add it to the nightly jobs here.
That might not be needed with the Observable Gauge since it triggers the function on scrape
I attempted with it but the value is still stuck at 0. Do you have any other ideas?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It shouldn't make a new observable gauge with each call. The gauge should be created when the service is created and reference variables in the service. These variables should be updated when the user statistics handler is called.
Definitely don't let this run every time the metrics are scraped.
Implementation of #11069
Added the following metrics:
in progress...