Description
I'm getting a crash on macOS Catalina when running some multiprocessing code after doing some plotting with matplotlib.
Looks like this:
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
Fatal Python error: Aborted
and using print(psutil.Process().open_files()) I figured out that likely the issue is that this MPL TTF font file is opened multiple times and (I guess) for some reason closing the file occurs only later, from my multiprocessing code, and that fails on macOS Catalina.
open files: [popenfile(path='/Users/deil/anaconda3/envs/gammapy-dev/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf', fd=5), popenfile(path='/Users/deil/anaconda3/envs/gammapy-dev/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf', fd=6), popenfile(path='/Users/deil/anaconda3/envs/gammapy-dev/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansDisplay.ttf', fd=7), popenfile(path='/Users/deil/anaconda3/envs/gammapy-dev/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansDisplay.ttf', fd=8)]
I tried to extract a minimal test case that doesn't involve our software, but couldn't so far, sorry.
But you should be able to reproduce the crash by copy & pasting these commands:
git clone https://github.com/gammapy/gammapy.git
cd gammapy
conda env create -f environment-dev.yml
pip install -e .
pytest -s -v gammapy/cube
Log: https://gist.github.com/cdeil/a75211856a3bcab751ead707df9708c9#file-gistfile1-txt-L88
I know MPL isn't thread-safe. But it should be possible to do some plotting, and then later after plotting is done to run some multiprocessing code, no?
Is it normal that the file handle for DejaVuSansDisplay.ttf remains open?
Shouldn't MPL load the file content and close the file handle directly?
Maybe that is a bug?
I don't think it matters for this issue, but in case it does -- we are calling plt.close() after all plotting from our tests, because previously we had a problem with too many open figures (see [here](
https://github.com/gammapy/gammapy/blob/d4db11d559b210c0e0d03bba75ddcff9c03e0511/gammapy/utils/testing.py#L209-L230
(gammapy-dev) hfm-1804a:gammapy deil$ python --version
Python 3.7.0
(gammapy-dev) hfm-1804a:gammapy deil$ python -c 'import matplotlib; print(matplotlib.__version__)'
3.1.1
(gammapy-dev) hfm-1804a:gammapy deil$ python -c 'import matplotlib; print(matplotlib.get_backend())'
MacOSX