[Bug]: ImportError when using Matplotlib v3.8.0 in Python package tests #26827
Comments
|
breaks my CI too |
|
Your installs are crossed. You are picking up mpl_toolkits from a different distribution. On the other hand this happens pretty commonly - I forget the explanation and maybe indicates an issue with how we install the wheels? |
|
This looks like an installation issue? It is certainly true that However, the line that is erroring in 52df591#diff-a67656224bfb26dc6de0a0f89e398cd6999377d1482aca3e867ee550ff05be57 |
|
I will also point out that |
|
@jklymak - "You are ..." sound like we did something actively. I assume we did not. The nightly CI build in jenkins fail. This seems to be just dependency hell as I have seen quite a few times in the last few months. |
|
Avoiding 3.8.0 fixes the jenkins problem and does not harm the github CI. The github CI also worked with the 3.8.0 version so there might be indeed be a sandboxing problem and some crossover between packages. I donot even know what mpl_toolkits is and why it would have anything to do with the way I am using matplotlib. My code is quite short and it surprises me that it should have such hidden dependencies '''
Created on 2020-07-05
@author: wf
'''
import matplotlib.pyplot as pl
8000
t
from collections import Counter
import numpy as np
import os
class Plot(object):
'''
create Plot based on counters
see https://stackoverflow.com/questions/19198920/using-counter-in-python-to-build-histogram
'''
def __init__(self, valueList,title,xlabel=None,ylabel=None,gformat='.png',fontsize=12,plotdir=None,debug=False):
'''
Constructor
'''
self.counter=Counter(valueList)
self.valueList=valueList
self.title=title
self.xlabel=xlabel
self.ylabel=ylabel
self.fontsize=fontsize
self.gformat=gformat
self.debug=debug
path=os.path.dirname(__file__)
if plotdir is not None:
self.plotdir=plotdir
else:
self.plotdir=path+"/../plots/"
os.makedirs(self.plotdir,exist_ok=True)
def titleMe(self):
''' set my title and labels '''
plt.title(self.title, fontsize=self.fontsize)
if self.xlabel is not None:
plt.xlabel(self.xlabel)
if self.ylabel is not None:
plt.ylabel(self.ylabel)
def showMe(self,mode='show',close=True):
''' show me in the given mode '''
if mode=="show":
plt.show()
else:
plt.savefig(self.plotdir+self.title+self.gformat)
if close:
plt.close()
def barchart(self,mode='show'):
''' barchart based histogram for the given counter '''
labels, values = zip(*self.counter.items())
indexes = np.arange(len(labels))
width = 1
self.titleMe()
plt.bar(indexes, values, width)
plt.xticks(indexes + width * 0.5, labels)
plt.yticks(np.arange(1,max(values)+1,step=1))
self.showMe(mode)
def showDebug(self):
print(" value list: ",self.valueList)
print("counter items: ",self.counter.items())
print("counter values: ",self.counter.values())
print("counter keys: ",self.counter.keys())
def hist(self,mode="show"):
''' create histogram for the given counter '''
if self.debug:
self.showDebug()
self.titleMe()
# see https://stackoverflow.com/a/2162045/1497139
plt.hist(self.valueList,bins=len(self.counter.keys()))
self.showMe(mode)
pass
|
|
I am getting the same problem on Linux Mint 21.2 after I upgraded to 3.8.0. The import sequence jump from The latter directory is part of the matplotlib 3.5.1 that is bundled with Mint. I uninstalled the Mint package and the import then started working correctly. ~/.local/lib/python3.10/site-packages comes before /usr/lib/python3/dist-packages in my sys.path, so is something in 3.8.0 changing it at run time? I have another PC with Mint 21.2 which is still on matplotlib 3.7.2 and it doe not have this problem. |
|
As to why you have multiple and why python is choosing different locations for Even with Matplotlib 3.7, I'd have expected this to at least be warning, as that was deprecated in 3.6. |
|
mpl_toolkits is a namespace package that we ship with Matplotlib (that is where the 3d and axis artist code live). For 3.8 we did change to using the "new-style" version of doing namespace packages (see #25244 #25381) because upstream started to actively warn about using the old way. It looks like all of these issues are in cases where What is the version of setuptools on the cases where it is broken? I strongly suspect that is going to be the determining factor of when this works or not. |
|
I'm not changing sys.path in my own code. Just executing the following line in the interpreter will trigger it: |
|
Ah, yeah, the namespace package definition did also change, though I think that the changes to packaging are likely orthogonal here... setuptools has been stripping out the changes silently for quite a while, the fact that it is a namespace package is probably important though, since namespace packages can have multiple locations on a filesystem. I'd have thought they would resolve in |
|
When I made the change, I found that setuptools had already been stripping out I think that much is true, and that even from wheels/recent setuptools installs from previous versions, the OS version would still be selected for mpl_toolkits.
I had been under the impression that it was okay to have a mix of "present" and "absent" That said, setuptools is kind of forcing our hand here. Even if we did not make the change to be explicit about how we package it, setuptools was stripping it from the wheels, and they are deprecating the old style entirely. And the upgrade path is just not good... I suspect that people who install via |
|
I resolved the ImportError issue by following these steps:
sudo apt remove python3-matplotlib
pip uninstall matplotlib
pip install matplotlibAfter completing these steps, I can now import Matplotlib in Python without encountering any errors: >>> import matplotlib.pyplot as pltIt's worth noting that while I resolved the issue, I don't remember the initial installation method, but I have been updating Matplotlib via pip since then. |
That would be my guess as well, and just that making doscstring private is now making old 3D incompatible with new matplotlib. |
|
I will also point out the frustrating state of being with this sentence in the
(emphasis mine) But we have to migrate the existing package, because setuptools is taking it away (and silently changing it so that our distribution has already been using new style namespace packages, so we are kind of stuck. |
|
I know this has been discussed before, but I forget the answer: is there some compelling reason we have mpl_toolkits install in a separate namespace from the main lib? They are released together, so I don't see the advantage. Conversely, if we are going to do this, it seems we need to be able to install mpl_toolkits as a versioned dependency of matplotlib if we are going to import it. |
|
mpl_toolkits is a namespace package, meaning that other things can install into the same namespace. In practice, I think this is done almost exclusively by I think I found like two other things, but they weren't maintained: Is the only one I actually remember finding (most searches only showed copies of the mpl source vendored in various locations.) |
|
Discussed briefly on the call: this import should at least be a conditional import and Warning if the import fails. |
|
@jaraco we do seem to be having some regression that may or may not be related to Matplotlib moving to new-style namespace packages. Is there a better fix for this? |
|
https://stackoverflow.com/questions/77153636/importerror-cannot-import-name-docstring-from-matplotlib where |
This solved my problem, thank you. |
I'd be happy to investigate. Are there steps I can take to repro? |
|
@jaraco I think it would be good to have step-by-step instructions for the setup that could ideally be run in a container or something akin to your CI |
|
I can reproduce this in an Ubuntu container using: The regular If I downgrade to 3.7.3: and the toolkit is in the user directory as expected. |
|
I reinstalled the matplotlib package that comes with Linux Mint 21.2 and I tried stepping through the So I tried using pdb: I got to the following point: I then ran python3 in a new terminal window and and confirmed that the How does Could this be the cause of the problem? |
|
I found the I unpacked it and found it includes a file: That file contains the following code: Presumably the python interpreter always runs this code at startup and adds the distribution's EDIT: I then ran a new |
|
If I replicate my reproducer in Fedora (36 if you want to have 3.5.* installed in the system location), then I cannot reproduce: This may have something to do with Debian's patching to use |
This file doesn't exist in 3.8.0 installed on Ubuntu or Fedora. However, it's not particularly version-specific, so if you copy it from |
|
I had a similar issue to many people whose |
|
I'm running into this in my dev environment, will update with whatever fix I find out. Show TracebackUninstalling and reinstalling matplotlib with pip from source did not work, and I did not have a system matplotlib package installed. My imports of matplotlib and mpl_toolkits.mplot3d are pointing towards the same source folder. Frustratingly, this only happens when debugging the file with VSCode and not setting a breakpoint. If I run the script not in debug mode, it works fine. If I set a breakpoint on the first line ( |
|
I just upgraded to 3.8.2 and I'm experiencing the related #27342 issue (Ubuntu 22.04, Python 3.10). When I went to run |
|
@nhf216 Are you getting errors or warnings? Can you update |
|
@tacaswell I'm just getting warnings. I updated the version of matplotlib with sagemath up to 3.8.2 also, and now I'm getting this warning there too. It turns out I have an Anaconda installation from years ago with some old versions of matplotlib hanging around in there. I'm assuming that's the source of the warnings. Dealing with that is painful. |
|
Did you try the |
|
@barronh I tried that (after you commented), and it did not fix anything. But, I did confirm that it's a system package (Version 3.5.1) causing the issue, not any of the Anaconda packages. I also confirmed that the |
|
@reticulatus #26827 (comment)
by /usr/lib/python3/dist-packages/matplotlib*nspkg.pth invoked from site.py. To remove nspkg.pth, we need to install python3-matplotlib 3.8 debian package, but it is not available yet, so I created one. https://github.com/cielavenir/salsa-matplotlib/releases/tag/ciel%2F3.8.3_1 (edit 20240312: simplified build process a bit)
It does import. |
|
This issue is blocking building docs for the debian packaging of matplotlib (3.8.0). The problem is as described by #26827 (comment) or #26827 (comment). The issue is that mpl_toolkits from matplotlib 3.6.3 is installed as a system package (python3-matplotlib) in I've tried many variations to solve it: adding an empty But every attempt to import the new |
|
Do you need python3-sphinx-gallery to build matplotlib? I dont think so |
|
The docs, man, the docs. |
Sorry for the long delay in responding. I'd fully intended to follow up. I don't remember why I did not. The recent activity revived my awareness of the issue. Following the repro instructions, I'm able to replicate the failure with this Dockerfile: from ubuntu:22.04
ENV DEBIAN_FRONTEND=noninteractive
run apt update -yy
run apt install -y python3-matplotlib python3-pip
run pip install --user -U matplotlib
cmd python3 -i -c 'import matplotlib; import mpl_toolkits.mplot3d'I can see now that the debian-installed matplotlib is using the pkg_resources-style namespace package for mpl_toolkits but the 3.8 pip-installed version is using the native namespace packages. As you've discovered, these two forms of namespace packages are incompatible within the same environment. It doesn't matter whether
The reason adding the Since matplotlib is migrating, it means that dual-installations will be incompatible across the 3.7/3.8 boundary, so users will need to somehow keep these versions isolated. Probably the two easiest ways to achieve this isolation is to use a virtualenv for the newer installation or to uninstall the older version from dist-packages. Upgrading the package-installed version also looks to be a viable alternative. Deleting the -nspth.pkg and mpl_toolkits/init.py from dist-packages should also work, though I'd feel uneasy about that, given that those files are managed by the package manager. I wish I had a better suggestion. If we could go back in time, one way to ease the transition is to use
There's no version of setuptools that fixes this issue. It's an inherent limitation in the design. |
|
As far as the doc build goes, the blocking problem is specifically with the autosummary generated for doc/api/toolkits/mplot3d/axes3d.rst. That is, the warning about the version mismatch with I can get a "successful" doc built by removing the I'll apply that workaround to get the docs building for Debian's python-matplotlib-doc. |
|
For me the solution was to |
|
yeah, removing python3-matplotlib works, but in my case freecad had dependency, so installing python3-matplotlib 3.8 was the only solution for me. |
Works!!! Thanks a lot!!! |
|
I think I tracked down the cause of this on my local machine. When debugging in VSCode on WSL, there are multiple paths that point towards the same file. One is the local linux filepath ( The solution (if I'm right about this) was to go through my breakpoints and make sure you aren't splitting them across files on the separate paths. |
|
Per #26827 (comment) we went across a hard compatibly edge with mpl 3.7 -> 3.8 and having overlapping installs of anything < 3.8 and anything >= 3.8 is going to have this issue. I've opened a PR ( #29673 ) add this to the API changes for 3.8 to close this issue. I don't think there is anything we can do at this point that would not be as or more disruptive than that status quo. |
Bug summary
I am encountering an ImportError when attempting to use Matplotlib v3.8.0 in my Python package's tests, specifically when running them with pytest. This issue does not occur with older versions of Matplotlib, such as v3.7.3.
Code for reproduction
Actual outcome
When running the tests, I receive the following ImportError:
Expected outcome
I expected my tests to run successfully with Matplotlib v3.8.0, just as they did with v3.7.3.
Additional information
Has this worked in earlier versions?
It is working in previous versions (eg, v3.7.3).
Do you know why this bug is happening?
When running the following command to get the Matplotlib Backend:
I got the following error:
Operating system
Pop!_OS 22.04 LTS
Matplotlib Version
3.8.0
Matplotlib Backend
ImportError: cannot import name 'docstring' from 'matplotlib' (/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/init.py)
Python version
3.10.12
Jupyter version
6.5.1 (Not using Jupyter notebook)
Installation
pip
The text was updated successfully, but these errors were encountered: