[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(server): add more prometheus metrics #11089

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

joaopedrocg27
Copy link
Contributor
@joaopedrocg27 joaopedrocg27 commented Jul 14, 2024

Implementation of #11069

Added the following metrics:

  • immich_users_deleted_total
  • immich_host_version
  • immich_host_version_latest

in progress...

@joaopedrocg27
Copy link
Contributor Author

Where could I get the user created metric?

@bo0tzz
Copy link
Member
bo0tzz commented Jul 14, 2024

Probably

async createUser(dto: Partial<UserEntity> & { email: string }): Promise<UserEntity> {

@joaopedrocg27
Copy link
Contributor Author

Probably

async createUser(dto: Partial<UserEntity> & { email: string }): Promise<UserEntity> {

Ok, I thought there was a better option. Ty

}
this.metricRepository.host.addToGauge(`immich.host.photos`, serverStats.photos)
this.metricRepository.host.addToGauge(`immich.host.videos`, serverStats.videos)
this.metricRepository.host.addToGauge(`immich.host.usage`, serverStats.usage)
Copy link
Contributor Author
@joaopedrocg27 joaopedrocg27 Jul 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw @mertalev, am I being stupid or you cannot send metrics without creating "sums"

Currently this keeps adding up the one photo I have. I can't use a Counter because it can decrease and the Gauge is adding stuff.

I am very noob in Otel, I am used to send prometheus data with bash or python directly :)

Copy link
Contributor
@mertalev mertalev Jul 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not as easy to do it for persistent data like that. Options:

  • Increment whenever an asset gets created or removed at the server level (might be difficult to keep in sync with actual)
  • Query Postgres for the data (accurate, but counting in Postgres is very slow since it's a full table scan, so doing it all the time is a very bad idea).
  • Have a trigger in Postgres that increments values in a separate counting table when assets are added or removed (fast retrieval for metrics, but makes insertions and deletes slower, not sure by how much).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly, I'll think about it! It is missing that on my grafana

Copy link
Contributor
@mertalev mertalev Jul 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But to just set it in this handler and not automatically, I'm sure you can set the gauges to explicitly be those values. The metric repo doesn't expose everything from the underlying otel library.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I think the up/down counter is different from a gauge. This should be what you're looking for.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried that but then it got stuck on 1... I need to try again. Thank you

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also need to find an endpoint that gets called more often (that one is just when the admin opens that panel) or create a cron for that

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add it to the nightly jobs here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add it to the nightly jobs here.

That might not be needed with the Observable Gauge since it triggers the function on scrape

I attempted with it but the value is still stuck at 0. Do you have any other ideas?

Copy link
Contributor
@mertalev mertalev Jul 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shouldn't make a new observable gauge with each call. The gauge should be created when the service is created and reference variables in the service. These variables should be updated when the user statistics handler is called.

Definitely don't let this run every time the metrics are scraped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants