10000 Added UWSGI sharedarea support by Lispython · Pull Request #129 · prometheus/client_python · GitHub
[go: up one dir, main page]

Skip to content

Added UWSGI sharedarea support #129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Conversation

Lispython
Copy link

Added new value class UWSGISharedareaInit, that store values on uwsgi sharedarea.

@brian-brazil
Copy link
Contributor

Can you explain how this works and why it's better than the approach we already use?

@Lispython
Copy link
Author

You mean prometheus_multiproc_dir ?

It's use http://uwsgi-docs.readthedocs.io/en/latest/SharedArea.html to store metrics.

It's fast and safe.You don't need to wipe prometheus_multiproc_dir between restarts.

@brian-brazil
Copy link
Contributor

How does it work in the backend?

From a quick peek the code you have can be no faster than what we already have, as it's doing the same locking (which is the slow bit).

@Lispython
Copy link
Author

backend? What do you mean?

I create sample project https://github.com/Lispython/prometheus_client_example to describe problems with prometheus_multiproc_dir.

Can you confirm that https://github.com/Lispython/prometheus_client_example/blob/master/flask_app.py and https://github.com/Lispython/prometheus_client_example/blob/master/uwsgi.ini#L4 is a correct usage of lib for multiprocessed applications?

@Lispython
Copy link
Author

@brian-brazil

@brian-brazil
Copy link
Contributor

backend? What do you mean?

Is is using IPC, SHM, mmapped files, anonymous mmaped files, network sockets, unix sockets or what?

I'm not inclined to add an additional method of doing things that only works for some users and doesn't bring a major benefit with it.

@Lispython
Copy link
Author

It can use mmapped files or anonymous mmaped files.

@brian-brazil whats about second question?

About prometheus_multiproc_dir

Try to make slow load (100 RPS during 1 minute) to application from example. Our application config have only 4 workers. Whet you call generate_latest function, it's generate different results for every request, not 6000 calls.

@brian-brazil
Copy link
Contributor

It can use mmapped files or anonymous mmaped files.

How is it doing interprocess locking?

Try to make slow load (100 RPS during 1 minute) to application from example.

How slow is the /metrics to render? I'd presume it's pretty fast.

@Lispython
Copy link
Author

@brian-brazil you don't understand.

Step 1. Run application and make load via make tank.

Step 2. Check uwsgi last log string.
dev_prometheustest_flask_app_1 | [pid: 10|app: 0|req: 1500/6000] 172.23.0.3 () {28 vars in 289 bytes} [Tue Jan 24 14:54:14 2017] GET / => generated 31 bytes in 0 msecs (HTTP/1.0 200) 2 headers in 80 bytes (1 switches on core 0)

Uwsgi processed 6000 requests.

Step 3. Get metrics request.

*   Trying 10.9.8.8...
* Connected to application (10.9.8.8) port 8051 (#0)
> GET /metrics HTTP/1.1
> Host: application:8051
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Content-Length: 157
<
# Process № 13
# HELP app:requests_total Total count of requests
# TYPE app:requests_total counter
app:requests_total{method="GET",url_rule="total"} 1502.0
* Connection #0 to host application left intact

Process pid=13
1502 not equal 6000 requests

Get another metrics request.

mbp-alex:uwsgi Alexandr$ curl -v http://application:8051/metrics
*   Trying 10.9.8.8...
* Connected to application (10.9.8.8) port 8051 (#0)
> GET /metrics HTTP/1.1
> Host: application:8051
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Content-Length: 157
<
# Process № 11
# HELP app:requests_total Total count of requests
# TYPE app:requests_total counter
app:requests_total{method="GET",url_rule="total"} 1496.0
* Connection #0 to host application left intact

Process pid=11
1496 requests not equal 6000

And every /metrics request return different requests counter.

@brian-brazil
Copy link
Contributor
brian-brazil commented Jan 24, 2017

Given that's roughly a quarter of the requests, I suspect you haven't hooked in the mutliproc setup properly.

@Lispython
Copy link
Author

If this is not right setup https://github.com/Lispython/prometheus_client_example/blob/master/uwsgi.ini#L4

Can you show right setup?

@brian-brazil
Copy link
Contributor

Have you hooked in the multiproc collector?

@Lispython
Copy link
Author

Some results.

*   Trying 10.9.8.8...
* Connected to application (10.9.8.8) port 8051 (#0)
> GET /metrics HTTP/1.1
> Host: application:8051
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Content-Length: 294
<
# Process in 11
# HELP app:requests_total Multiprocess metric
# TYPE app:requests_total counter
app:requests_total{method="GET",url_rule="total"} 1497.0
# HELP app:requests_total Total count of requests
# TYPE app:requests_total counter
app:requests_total{method="GET",url_rule="total"} 1495.0
* Connection #0 to host application left intact```

@brian-brazil
Copy link
Contributor

Are you pre-forking? We don't support that currently.

@Lispython
Copy link
Author
[uwsgi]
chdir=/usr/src/app/
env = APP_ROLE=dev_uwsgi
env = prometheus_multiproc_dir=/usr/src/app/storage/prometheus/
wsgi-file = /usr/src/app/flask_app.py
master=True
vacuum=True
max-requests=5000
harakiri=120
post-buffering=65536
workers=4
stats=/tmp/uwsgi-app.stats
buffer-size=65536
http = 0.0.0.0:8051
thunder-lock=True

@rhariady
Copy link

@Lispython i tried using your branch, but unfortunately the result is still different each time i call the metrics endpoint. Am i missing something in the configuration?

This is how i use the prometheus client:

import time
import os

from prometheus_client import generate_latest
from prometheus_client import CollectorRegistry, Counter, Histogram
from flask import request, Response

REGISTRY = CollectorRegistry()

FLASK_REQUEST_LATENCY = Histogram('flask_request_latency_seconds', 'Flask Request Latency',
				  ['method', 'endpoint'], registry=REGISTRY)
FLASK_REQUEST_COUNT = Counter('flask_request_count', 'Flask Request Count',
				['method', 'endpoint', 'http_status'], registry=REGISTRY)

class Prometheus(object):

    def __init__(self, app=None):
        self.app = app
        if app is not None:
            self.init_app(app)

    def init_app(self, app):

        app.before_request(self.before_request)
        app.after_request(self.after_request)
        
        @app.route("/metrics")
        def metrics():
            text = "# Process in {0}\n".format(os.getpid())
            return Response(text + generate_latest(REGISTRY), mimetype="text/plain")

    @staticmethod
    def before_request():
        request.start_time = time.time()

    @staticmethod  
    def after_request(response):
        request_latency = time.time() - request.start_time
        if request.url_rule and 'metrics' not in request.url_rule.rule:

            FLASK_REQUEST_LATENCY.labels(request.method, request.path).observe(request_latency)
            FLASK_REQUEST_COUNT.labels(request.method, request.path, response.status_code).inc()

        return response

And this is my uwsgi .ini file

[uwsgi]
master = true
die-on-term = true
module = <module_name>:<callable>
memory-report = true
enable-threads = true
max-requests=5000
harakiri=120
post-buffering=65536
workers=2
threads=1
#enable-threads=True
#listen=4000
# socket=0.0.0.0:8997
stats=/tmp/uwsgi-app.stats
#logger=syslog:uwsgi_app_stage,local0
buffer-size=65536
thunder-lock=True

sharedarea=1000
sharedarea=1000 

and this is the result

# Process in 8512
# HELP flask_request_latency_seconds Flask Request Latency
# TYPE flask_request_latency_seconds histogram
flask_request_latency_seconds_bucket{endpoint="/users",le="0.005",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.01",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.025",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.05",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.075",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.1",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.25",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.5",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.75",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="1.0",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="2.5",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="5.0",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="7.5",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="10.0",method="GET"} 48.0
flask_request_latency_seconds_bucket{endpoint="/users",le="+Inf",method="GET"} 48.0
flask_request_latency_seconds_count{endpoint="/users",method="GET"} 48.0
flask_request_latency_seconds_sum{endpoint="/users",method="GET"} 0.03195333480834961
# HELP flask_request_count Flask Request Count
# TYPE flask_request_count counter
flask_request_count{endpoint="/users",http_status="401",method="GET"} 48.0

and second time

# Process in 8513
# HELP flask_request_latency_seconds Flask Request Latency
# TYPE flask_request_latency_seconds histogram
flask_request_latency_seconds_bucket{endpoint="/users",le="0.005",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.01",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.025",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.05",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.075",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.1",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.25",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.5",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="0.75",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="1.0",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="2.5",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="5.0",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="7.5",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="10.0",method="GET"} 52.0
flask_request_latency_seconds_bucket{endpoint="/users",le="+Inf",method="GET"} 52.0
flask_request_latency_seconds_count{endpoint="/users",method="GET"} 52.0
flask_request_latency_seconds_sum{endpoint="/users",method="GET"} 0.03392148017883301
# HELP flask_request_count Flask Request Count
# TYPE flask_request_count counter
flask_request_count{endpoint="/users",http_status="401",method="GET"} 52.0

@Lispython
Copy link
Author

Hi, @orhan89 , you right. After this discussion I make another tests and got similar results. generate_latest get values from local memory, not from _ValueClass( _UWSGISharedareaDict).

Without a complete refactoring it is impossible to do what we need.

I create my own library that can support extensible storages (local memory, uwsgi sharedarea, mmap, etc) and uwsgi exporter included:

https://github.com/Lispython/pyprometheus

Demo project: https://github.com/Lispython/pyprometheus_demo

Sorry for poor docs and examples, its still in progress.

@Lispython Lispython closed this Feb 26, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
0