process_collector.py assumes /proc/self/fd is readable for process_open_fds #95

RalphCorderoy · 2016-08-18T11:08:27Z

Apache with prefork MPM, not threaded, runs process_collector.py as the chosen non-root user, but /proc/self/fd/ is still owned by root:root so its 0500 permissions cause an OSError exception, e.errno == EACCES, on the os.listdir() attempt. A patch to give up on that metric when that occurs: process_open_fds.txt

hedgepigdaniel · 2017-08-10T06:33:11Z

So annoying, also an issue with Ubuntu 12.04 built in uwsgi init.d service.

I assume this is a problem whenever uwsgi does its own priviledge deescalation

https://unix.stackexchange.com/questions/75045/files-in-proc-pid-e-g-ssh-agent-chrome-are-not-owned-by-user-but-by-root

When used in https://github.com/korfuri/django-prometheus the stacktrace looks like this:

Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 638, in __init__
    self.handle()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
    method()
  File "/home/engine/venv/local/lib/python2.7/site-packages/prometheus_client/exposition.py", line 86, in do_GET
    output = generate_latest(registry)
  File "/home/engine/venv/local/lib/python2.7/site-packages/prometheus_client/exposition.py", line 63, in generate_latest
    for metric in registry.collect():
  File "/home/engine/venv/local/lib/python2.7/site-packages/prometheus_client/core.py", line 97, in collect
    for metric in collector.collect():
  File "/home/engine/venv/local/lib/python2.7/site-packages/prometheus_client/process_collector.py", line 84, in collect
    len(os.listdir(os.path.join(pid, 'fd'))))
OSError: [Errno 13] Permission denied: '/proc/self/fd'

hedgepigdaniel · 2017-08-10T06:58:41Z

Thanks @RalphCorderoy for the patch, it fixed the problem for me (I assume at the expense of file handle information) - I committed it here if anyone wants to try it: https://github.com/prometheus/client_python/compare/master...hedgepigdaniel:ignore-fd-permissions?expand=1

hedgepigdaniel referenced this issue in hedgepigdaniel/client_python Sep 14, 2017

Ignore EACCESS when reading open file handles

529c8f5

blopker pushed a commit to blopker/client_python that referenced this issue Oct 20, 2017

Fix prometheus#95: catch OSError when collecting file descriptors.

4fa3e53

blopker mentioned this issue Oct 20, 2017

Fix #95: catch OSError when collecting file descriptors. #205

Merged

brian-brazil closed this as completed in 6625f89 Oct 20, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

process_collector.py assumes /proc/self/fd is readable for process_open_fds #95

process_collector.py assumes /proc/self/fd is readable for process_open_fds #95

Uh oh!

Uh oh!

process_collector.py assumes /proc/self/fd is readable for process_open_fds #95

process_collector.py assumes /proc/self/fd is readable for process_open_fds #95

Comments

Uh oh!

Uh oh!