segfault due to font objects when multi-threading

Although we are very much no thread safe, that this is a "segfault" failure so I think it should be addressed. Users should be able to take their toes off with threads, not their whole leg.

I discovered this in an application my group develops at BNL which is more-or-less

import faulthandler

import io
import time
import threading
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvas

faulthandler.enable()

fig = Figure()
cav = FigureCanvas(fig)
ax = fig.subplots()
ln, = ax.plot(range(5), label='bob')
ax.legend()

def bad_idea(fig, n):
    time.sleep(1)
    print(f'go {n}')
    for j in range(100):
        #        time.sleep(.01)
        fig.gca().legend()
        fig.tight_layout()
    print(f'done {n}')


fig.savefig(io.BytesIO())
threads = [threading.Thread(target=bad_idea, name=f'bad_thread_{j}', args=(fig, j)).start()
           for j in range(10)]


for j in range(5):
    fig.canvas.draw()

but maybe a bit less over-the-top (we only have 1 background thread and it only fires occasionally), but the goal here is to make sure we always lose the race. Tracing this back I eventually got to this being related to fonts and reduced the reproducing case to:

import faulthandler
import time
import threading
import matplotlib.font_manager as fm
from matplotlib.ft2font import LOAD_NO_HINTING

faulthandler.enable()


def bad_idea(fm, n):
    time.sleep(1)
    print(f"go {n}")
    for j in range(100):
        font = fm.get_font(fm.findfont("Dej
DC01
aVu Sans"))
        # font.clear()
        # font.set_size(12, 72.0)
        font.set_text(str(n), 0.0, flags=LOAD_NO_HINTING)
    print(f"done {n}")


threads = [
    threading.Thread(target=bad_idea, name=f"bad_thread_{j}", args=(fm, j)).start()
    for j in range(10)
]

If you turn down the number of threads you can also get various errors out of freetype

$ python /tmp/bad_font.py
go 0
go 2
go 1
Exception in thread bad_thread_2:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/tmp/bad_font.py", line 17, in bad_idea
    font.set_text(str(n), 0.0, flags=LOAD_NO_HINTING)
RuntimeError: In set_text: Could not get glyph (error code 0x12)
Exception in thread bad_thread_0:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/tmp/bad_font.py", line 17, in bad_idea
    font.set_text(str(n), 0.0, flags=LOAD_NO_HINTING)
RuntimeError: In set_text: Could not load glyph (error code 0x14)
done 1

vs

$ python /tmp/bad_font.py
go 1
go 2
go 0
Exception in thread bad_thread_2:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Fatal Python error: Segmentation fault

Thread 0x00007fefdac67640 (most recent call first):
  File "/usr/lib/python3.9/threading.py", line 1214 in invoke_excepthook
  File "/usr/lib/python3.9/threading.py", line 956 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

Current thread 0x00007fefd8466640 (most recent call first):
  File "/tmp/bad_font.py", line 17 in bad_idea
  File "/usr/lib/python3.9/threading.py", line 892 in run
  File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

Thread 0x00007fefd5c65640 (most recent call first):
  File "/usr/lib/python3.9/posixpath.py", line 167 in islink
  File "/usr/lib/python3.9/posixpath.py", line 425 in _joinrealpath
  File "/usr/lib/python3.9/posixpath.py", line 391 in realpath
  File "/home/tcaswell/source/p/matplotlib/mpl-main/lib/matplotlib/font_manager.py", line 1228 in findfont
  File "/tmp/bad_font.py", line 14 in bad_idea
  File "/usr/lib/python3.9/threading.py", line 892 in run
  File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

Thread 0x00007fefe6309740 (most recent call first):
  File "/usr/lib/python3.9/threading.py", line 1428 in _shutdown
Segmentation fault (core dumped)

I bisected this back to https://github.com/matplotlib/matplotlib/pull/15104/files and 97477d7 specifically.

I think we have a couple of options here:

revert that PR temporairily (maybe we want to do this for 3.4?)
add some locking at the c-level in the ft2font
add some locking at the python level
make the font cache a thread-local

I have some code to implement locking, but it gets a bit gnarly (as I have both a lock and a semaphore to try and make sure we always use fonts safely). Sorting out how to make sure we always lock around font access is complicated because we cache the font objects in at least 2 places

matplotlib/lib/matplotlib/font_manager.py

Lines 1397 to 1413 in a510e32

    
           _get_font = lru_cache(64)(ft2font.FT2Font) 
        
           # FT2Font objects cannot be used across fork()s because they reference the same 
        
           # FT_Library object.  While invalidating *all* existing FT2Fonts after a fork 
        
           # would be too complicated to be worth it, the main way FT2Fonts get reused is 
        
           # via the cache of _get_font, which we can empty upon forking (in Py3.7+). 
        
           if hasattr(os, "register_at_fork"): 
        
               os.register_at_fork(after_in_child=_get_font.cache_clear) 
        
           def get_font(filename, hinting_factor=None): 
        
               # Resolving the path avoids embedding the font twice in pdf/ps output if a 
        
               # single font is selected using two different relative paths. 
        
               filename = _cached_realpath(filename) 
        
               if hinting_factor is None: 
        
                   hinting_factor = rcParams['text.hinting_factor'] 
        
               return _get_font(filename, hinting_factor, 
        
                                _kerning_factor=rcParams['text.kerning_factor'])

and

matplotlib/lib/matplotlib/_mathtext.py

Lines 222 to 253 in a510e32

    
           class TruetypeFonts(Fonts): 
        
               """ 
        
               A generic base class for all font setups that use Truetype fonts 
        
               (through FT2Font). 
        
               """ 
        
               def __init__(self, default_font_prop, mathtext_backend): 
        
                   super().__init__(default_font_prop, mathtext_backend) 
        
                   self.glyphd = {} 
        
                   self._fonts = {} 
        
                   filename = findfont(default_font_prop) 
        
                   default_font = get_font(filename) 
        
                   self._fonts['default'] = default_font 
        
                   self._fonts['regular'] = default_font 
        
               @_api.deprecated("3.4") 
        
               def destroy(self): 
        
                   self.glyphd = None 
        
                   super().destroy() 
        
               def _get_font(self, font): 
        
                   if font in self.fontmap: 
        
                       basename = self.fontmap[font] 
        
                   else: 
        
                       basename = font 
        
                   cached_font = self._fonts.get(basename) 
        
                   if cached_font is None and os.path.exists(basename): 
        
                       cached_font = get_font(basename) 
        
                       self._fonts[basename] = cached_font 
        
                       self._fonts[cached_font.postscript_name] = cached_font 
        
                       self._fonts[cached_font.postscript_name.lower()] = cached_font 
        
                   return cached_font

and because every backend has slightly different helpers to interact with the font objects.

We also have some existing locking around the font cache

matplotlib/lib/matplotlib/backends/backend_agg.py

Lines 72 to 83 in a510e32

    
           # we want to cache the fonts at the class level so that when 
        
           # multiple figures are created we can reuse them.  This helps with 
        
           # a bug on windows where the creation of too many figures leads to 
        
           # too many open file handles.  However, storing them at the class 
        
           # level is not thread safe.  The solution here is to let the 
        
           # FigureCanvas acquire a lock on the fontd at the start of the 
        
           # draw, and release it when it is done.  This allows multiple 
        
           # renderers to share the cached fonts, but only one figure can 
        
           # draw at time and so the font cache is used by only one 
        
           # renderer at a time. 
        
           lock = threading.RLock()

but despite my comments on gitter, I no longer think this code is relevant (and this lock should probably move to font_manager.py anyway).

Matplotlib version

Operating system: linux
Matplotlib version (import matplotlib; print(matplotlib.__version__)): 3.3.x +
Matplotlib backend (print(matplotlib.get_backend())): any
Python version: 3.7-3.9
Jupyter version (if applicable): N/A
Other libraries:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions