[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explicit OpenCV flavor requirement conflicts with existing installs of OpenCV #473

Open
faustomorales opened this issue Nov 1, 2019 · 20 comments
Labels
TODO Planned to be added to the library or changed in the future

Comments

@faustomorales
Copy link
faustomorales commented Nov 1, 2019

The current practice of specifying a requirement for opencv-python-headless causes conflicts for other installations of OpenCV. For example, if a user requires modules from opencv-contrib-python that are not available in opencv-python-headless but they install imgaug, pip will install both versions and thusly have conflicting versions of OpenCV with different features. To do this, I have to do:

pip install imgaug
pip uninstall -y opencv-python-headless
pip install opencv-contrib-python-headless

It seems this problem was contemplated in #324 so I suppose I'm just here to confirm those fears are well-founded.

From what I can tell, imgaug supports any of the four opencv-*-python-* flavors. In light of that, I humbly propose the following as options (with a preference for the first):

  1. Removing opencv-*-* from the install_requires in setup.py. This is the simplest option and is compatible with the current installation instructions in the README which ask the user to install OpenCV on their own first. It is at this point that the user could choose the OpenCV flavor that makes sense for their application.
  2. Implementing more logic to check for existing OpenCV versions. I think this will be kind of a pain but is what scikit-image seems to do and it may create problems for people who use package management systems like pipenv or poetry where package installation sequence is ill-defined. It also will have sequencing issues for users who build their own customized versions of opencv-python in order to use non-free algorithms in OpenCV.

I will gladly submit a PR for this if the relevant folks agree this is okay. Thanks!

@aleju aleju added the TODO Planned to be added to the library or changed in the future label Nov 1, 2019
@rokusottervanger
Copy link
rokusottervanger commented Jan 24, 2020

Here's another interesting edge case to keep in mind:
I'm working from an NVidia Jetson Xavier with Jetpack 4.2.3. This comes preinstalled with opencv 3.3.1:

user@xavier-001:~$ python3 -c "import cv2; print(cv2.__version__)"
3.3.1

However, trying to pip3 install imgaug gives this error:

ERROR: Could not find a version that satisfies the requirement opencv-python-headless (from imgaug) (from versions: none)
ERROR: No matching distribution found for opencv-python-headless (from imgaug)

The workaround I used to install imgaug is as follows:
pip3 install the requirements of imgaug manually (from requirements.txt), except opencv-python-headless.
pip3 install --no-deps imgaug
This will probably become more and more of a nuisance when dealing with packages/modules depending on imgaug, as their installation will err or at least warn.

@aleju
Copy link
Owner
aleju commented Feb 6, 2020

There is now a new release online in which setup.py accepts opencv-python, opencv-python-headless, opencv-contrib-python and opencv-contrib-python-headless -- if they are already installed. Otherwise it will fall back to opencv-python-headless as its dependency. This improves compatibility with libraries that use other opencv flavors than headless.

This change does not fix the issue that imgaug might end up being installed first, chooses its default opencv flavor (headless) and later on installed libraries are then incompatible with that flavor. At least in some cases that can be handled by installing the other libraries first and then in a separate command imgaug.

I'm rather reluctant currently to remove opencv from the dependencies list. For professional users that would probably be the most convenient choice. I suspect however that the majority of all users is rather inexperienced and would encounter problems if the library did not fully install itself, even if there was a seemingly clear error message. There is also the issue that removing the opencv dependency would probably break many existing builds that rely on opencv being auto-installed via imgaug.

@faustomorales
Copy link
Author
faustomorales commented Feb 9, 2020

I support an approach that helps inexperienced users. Experienced users can typically find workarounds like those suggested in this thread. I do wonder under what circumstances a user would rely on imgaug without also relying on OpenCV directly in their codebase, but that may just be a failure of my imagination. Bottom line, I'm okay closing this issue. Thanks for carefully considering the relevant tradeoffs.

@ntabris
Copy link
ntabris commented Feb 11, 2020

As far as I can tell, the modification to setup.py works great when installing imgaug from source but doesn't help when installing the the pypi wheel. I'm guessing this is because the wheel METADATA for imgaug 0.4.0 has Requires-Dist: opencv-python.

Specifically, if I run pip install opencv-python-headless imgaug, I end up with both opencv-python and opencv-python-headless:

> pip install opencv-python-headless imgaug

Collecting opencv-python-headless
  Using cached opencv_python_headless-4.2.0.32-cp36-cp36m-macosx_10_9_x86_64.whl (40.1 MB)
Collecting imgaug
  Using cached imgaug-0.4.0-py2.py3-none-any.whl (948 kB)
...
Collecting opencv-python
  Using cached opencv_python-4.2.0.32-cp36-cp36m-macosx_10_9_x86_64.whl (47.9 MB)
...
Installing collected packages: numpy, opencv-python-headless, Shapely, Pillow, imageio, scipy, opencv-python, pyparsing, six, cycler, kiwisolver, python-dateutil, matplotlib, PyWavelets, decorator, networkx, scikit-image, imgaug
Successfully installed Pillow-7.0.0 PyWavelets-1.1.1 Shapely-1.7.0 cycler-0.10.0 decorator-4.4.1 imageio-2.6.1 imgaug-0.4.0 kiwisolver-1.1.0 matplotlib-3.1.3 networkx-2.4 numpy-1.18.1 opencv-python-4.2.0.32 opencv-python-headless-4.2.0.32 pyparsing-2.4.6 python-dateutil-2.8.1 scikit-image-0.16.2 scipy-1.4.1 six-1.14.0

I couldn't find a way list alternative dependencies in a wheel, and it appears the idea was considered and rejected in PEP 426.

@fvisconti
Copy link

Doesn't a virtual env resolve the issue?

@ntabris
Copy link
ntabris commented Jul 2, 2020

Doesn't a virtual env resolve the issue?

No, this doesn't give us a way to pip install imgaug==0.4 without it installing opencv-python (which we don't want). We've pinned our package to require imgaug==0.3.0 so that we can use opencv-python-headless.

@agchang-cgl
Copy link

I have also encountered this issue where we are trying to break the dependency on opencv-python (i.e. the non-headless version), but it makes its way in due to albumentations==0.4.6 -> imgaug==0.4.0.

I'm no Python packaging expert, but It seems like the wheel was made in an environment where opencv-python was installed so it made its way into the wheel dependencies. Can we get a new wheel with opencv-python-headless, please?

@adampl
Copy link
adampl commented Sep 10, 2020

Same issue here - @aleju what can be done about this?

@bertsky
Copy link
bertsky commented Sep 22, 2020

Does anyone know if pyproject.toml (replacing setup.py) could help us, or is this really a wheel-building issue?

@bertsky
Copy link
bertsky commented Oct 6, 2020

I'm no Python packaging expert, but It seems like the wheel was made in an environment where opencv-python was installed so it made its way into the wheel dependencies. Can we get a new wheel with opencv-python-headless, please?

I second that.

As a workaround (besides holding at 0.3.0) one can simply avoid the binary/wheel method for imgaug, as in:

pip install --no-binary imgaug imgaug # ... other deps

...or in a requirements.txt file...

# ... other deps
imgaug --no-binary imgaug
# ... other deps

And there is yet another option: I have created a pseudo-package re-branding opencv-python-headless as opencv-python (and trying to re-export its install-time version number, too). Thus, you could install this package prior to installing some dependency chain to avoid dragging in opencv-python proper (and thereby replacing headless).

I wonder, though, why the OpenCV Python bindings did not decide to solve the contrib and X11 options via the extras mechanism. So packages which depend on these features could depend on opencv-python[contrib] or opencv[contrib,x11], or in turn make them conditional in extras of their own (e.g. extras_require={'plot':['opencv-python[x11]']})...

@JohnPaton
Copy link

Just spent a whole morning getting bit by this. There should at least be a section in the README about how to install with different opencv flavors, and even better an officially supported way of making it happen.

@patzm
Copy link
patzm commented Nov 24, 2020

@aleju, I can also confirm that the uploaded wheel triggers the opencv-python dependency (as opposed to opencv-python-headless). I think this could be resolved by either of the following two things:

  • re-upload the wheel, making sure that the correct library (opencv-python-headless) is referenced in it
  • do a new release and then upload again

Would that be feasible?

@haolsun
Copy link
haolsun commented Jan 8, 2021

pip install --upgrade pip
Have a try.

@patzm
Copy link
patzm commented Jan 11, 2021

Can you quickly elaborate what changed?

@bertsky
Copy link
bertsky commented Jan 14, 2021
  • re-upload the wheel, making sure that the correct library (opencv-python-headless) is referenced in it

IINM, that could be as simple as:

pip download imgaug
wheel unpack imgaug*.whl
sed -i s/opencv-python/opencv-python-headless/ imgaug-0.4.0/imgaug-0.4.0.dist-info/METADATA
wheel pack --build-number 2 imgaug-0.4.0
twine upload imgaug-0.4.0-2-py2.py3-none-any.whl

Anyone with the credentials for imgaug on PyPI could do that.

  • do a new release and then upload again

Yes, a true post-release according to PEP 440 (like 0.4.0.post1) would probably be better.

@faustomorales
Copy link
Author

On Windows, this creates another interesting error if you are trying to install imgaug while having an open Python process that is using OpenCV.

ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: <REDACTED>\\.venv\\Lib\\site-packages\\cv2\\cv2.cp38-win_amd64.pyd'
Consider using the `--user` option or check the permissions.

I worked around this using --no-dependencies.

@adampl
Copy link
adampl commented Apr 20, 2021

@patzm Could you, being German, try to write/call @aleju ? :) Here are his contact details: http://www.ajung.name/impressum.html

@patzm
Copy link
patzm commented Apr 20, 2021

It seems like @aleju stopped working on GitHub around June last year. If he doesn't intend to (exclusively) continue maintaining this repo, maybe someone can offer a helping hand 🤔. I opened #764 for that matter. Let's discuss over there. If Alex doesn't respond in say a month, I can for sure give him a call 😉.

@patzm
Copy link
patzm commented Jun 9, 2021

Yo apparently the recent release v1.0.0 of albumentations avoids this issue. Check out the release notes: https://github.com/albumentations-team/albumentations/releases/tag/1.0.0

@sisrfeng
Copy link

Yo apparently the recent release v1.0.0 of albumentations avoids this issue. Check out the release notes: https://github.com/albumentations-team/albumentations/releases/tag/1.0.0

In some projects (e.g. paddleOCR), there is imgaug in the "requirements.txt" file. Is there any convenient method to replace it with albumentations?
@patzm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
TODO Planned to be added to the library or changed in the future
Projects
None yet
Development

No branches or pull requests