10000 [CI fix] add import_array in cython files where numpy is cimport-ed by emmanuelle · Pull Request #4592 · scikit-image/scikit-image · GitHub
[go: up one dir, main page]

Skip to content

[CI fix] add import_array in cython files where numpy is cimport-ed #4592

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 18, 2020

Conversation

emmanuelle
Copy link
Member

Closes #4577 (hopefully).

Lots of places where import_array should be added, but let's check first whether the CI is green.

@grlee77
Copy link
Contributor
grlee77 commented Apr 17, 2020

Thanks for looking into this @emanuelle

I can confirm that the same issue occurs in PyWavelets and adding import_array() a few places fixed it there.

I also ran into some failures similar to the remaining ones seen here. Specifically, the errors stating

TypeError: Expected dict, got float correspond to: cython/cython#3511 and have already been fixed on Cython master.

grlee77
grlee77 previously approved these changes Apr 17, 2020
@grlee77 grlee77 dismissed their stale review April 17, 2020 22:22

accidentally marked approved while still debugging

@emmanuelle
Copy link
Member Author

Thanks a lot for the tip @grlee77 I would have spent a lot of time today trying to understand this TypeError. So we wait until Cython ships a new version of their alpha I guess?

@emmanuelle
Copy link
Member Author

@grlee77 since you already took a look, the failures in th 8000 e denoise tests are indeed related to this fused types bug which was corrected in Cython master but the colormixer failing tests seem to be a different kind of problem. It's a TypeError: add() takes exactly 4 positional arguments (5 given) when calling a function defined in a cython file. Does this ring a bell? I will try to understand what's going on but maybe it's something obvious...

@grlee77
Copy link
Contributor
grlee77 commented Apr 18, 2020

It's a TypeError: add() takes exactly 4 positional arguments (5 given) when calling a function defined in a cython file. Does this ring a bell? I will try to understand what's going on but maybe it's something obvious...

I was not familiar with this one, but did take a quick look. skimage.io._plugins._colormixer.add does work fine when I call it directly. The issue seems to be when assigning it as a class attribute as done in the test suite. In this case, it becomes a bound method and thus an extra self argument is being passed in. I don't know the details yet of why/how this used to work and what changed to cause it to no longer work now. It is probably worth reporting to the Cython team.

The following change (based on this Stack Overflow question) to the test class, does get it to work for me (the change is just the addition of staticmethod() around the cython function):

class TestColorMixerAdd(ColorMixerTest):
    op = staticmethod(cm.add)
    py_op = np.add
    positive = 50
    positive_clip = 56
    negative = -50
    negative_clip = -220


class TestColorMixerMul(ColorMixerTest):
    op = staticmethod(cm.multiply)
    py_op = np.multiply
    positive = 1.2
    positive_clip = 2
    negative = 0.5
    negative_clip = -0.5

@grlee77
Copy link
Contributor
grlee77 commented Apr 18, 2020

I can open a Cython issue related to the change in class attribute/bound method behavior.

update: see cython/cython#3529

@emmanuelle
Copy link
Member Author

thank you so much @grlee77 !

@grlee77
Copy link
Contributor
grlee77 commented Apr 18, 2020

An additional benefit of adding these import_array() calls is that they may also improve PyPy compatibility. I had to add a patch with them to the conda-forge/pywavelets-feedstock to get that package to run the tests successfully with PyPy on ppc64le architecture (without the patch it would segfault). I assume the same may be true for scikit-image.

def _nl_means_denoising_2d(cnp.ndarray[np_floats, ndim=3] image, Py_ssize_t s=7,
Py_ssize_t d=13, double h=0.1, double var=0.):
def _nl_means_denoising_2d(cnp.ndarray[np_floats, ndim=3] image, Py_ssize_t s,
Py_ssize_t d, double h, double var):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's ok to remove default values here since the defaults are given in the Python function and passed to the Cython one.

@emmanuelle emmanuelle changed the title [WIP] add import_array in cython files where numpy is cimport-ed [CI fix] add import_array in cython files where numpy is cimport-ed Apr 18, 2020
@emmanuelle
Copy link
Member Author

Lo and behold, the CI is now green on the --pre jobs (at least on Azure and Travis) using Cython alpha :-). Let's wait for all jobs to complete and if it's green we should merge this ASAP ! @jni

Many thanks to @grlee77 and the Cython team (@scoder @da-woods @mattip) for their previous help!

@da-woods
Copy link

Just about "removing default values" - this looks to be working around a bug that we think we've fixed. It might be worth waiting for Cython alpha 2? (unless the default values really aren't needed, or it's really important to get the CI to build it as soon as possible)

@emmanuelle
Copy link
Member Author

@da-woods indeed we should remove the default values as soon as alpha 2 is out but I prefer to have the CI green, it's quite scary for contributors to see all these failed jobs, and also we become less attentive to other potential CI failures (thinking yeah the CI fails these days...). The workaround does not change at all the behaviour of the function.

< 8000 div data-view-component="true" class="comment-reactions js-reactions-container js-reaction-buttons-container social-reactions reactions-container d-none">

Copy link
Contributor
@JDWarner JDWarner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve. Removing the defaults in _nl_means helper functions is fine because as @emmanuelle says, these are hidden helper functions called from pure Python wrappers which preserve the user-facing defaults. Good to know we could use that motif in the future, but they were never necessary in these Cython helper functions.

Copy link
Member
@rfezzani rfezzani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 Thank you @emmanuelle for all your investigations.

@rfezzani rfezzani merged commit 2fc3c5d into scikit-image:master Apr 18, 2020
@jni
Copy link
Member
jni commented Apr 19, 2020

🎉 !!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI fails on Azure for Default-Python-x64-Pre build
6 participants
0