10000 Return an explicit 500 if there's an exception generating metrics by FauxFaux · Pull Request #85 · prometheus/client_python · GitHub
[go: up one dir, main page]
Skip to content

Return an explicit 500 if there's an exception generating metrics #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Return an explicit 500 if there's an exception generating metrics #85

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion prometheus_client/exposition.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

from __future__ import unicode_literals

import logging
import os
import socket
import time
Expand Down Expand Up @@ -70,10 +71,18 @@ def generate_latest(registry=core.REGISTRY):

class MetricsHandler(BaseHTTPRequestHandler):
def do_GET(self):
try:
content = generate_latest(core.REGISTRY)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest removing the try and letting the exception propagate up.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That causes an empty body to be returned, not a 500. I felt that a 500 was more useful than just closing the tcp connection (although I suspect Prometheus would handle the case in the same way).

     def do_GET(self):
-        try:
-            content = generate_latest(core.REGISTRY)
-        except:
-            logging.exception('error generating response')
-            self.send_error(500, 'error generating metrics')
-            self.end_headers()
-            return
-
+        content = generate_latest(core.REGISTRY)
         self.send_response(200)
% curl -v http://localhost:23456/
*   Trying ::1...
* connect to ::1 port 23456 failed: Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 23456 (#0)
> GET / HTTP/1.1
> Host: localhost:23456
> User-Agent: curl/7.47.0
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prometheus would handle it the same. If we're going to explicitly return a 500 we should also include the full exception in the response.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many security policies discourage returning the stack-trace (or any details about the actual problem) in the response from 500 errors by default. Maybe you could argue that people typically authenticate these metrics endpoints, so it's not a problem? I don't, so would rather the stack-trace wasn't returned. If you decide to put it in, it will just be stripped by our load balancers anyway, so I don't mind either way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would generally expect metrics endpoints not to be exposed to the outside world, and even then for the information you can gleam from the errors to be minimal.

except:
logging.exception('error generating response')
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't use the "convenience" functions directly on the logging module. They log to the root logger making it impossible to filter (or even redirect) these messages. Instead use a module level logger like so:

import logging

log = logging.getLogger(__name__)

# ... code ...

log.exception("whatever")

self.send_error(500, 'error generating metrics')
self.end_headers()
return

self.send_response(200)
self.send_header('Content-Type', CONTENT_TYPE_LATEST)
self.end_headers()
self.wfile.write(generate_latest(core.REGISTRY))
self.wfile.write(content)

def log_message(self, format, *args):
return
Expand Down
0