8000 RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. · Issue #11788 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. #11788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ron819 opened this issue Aug 20, 2018 · 34 comments
Closed

RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. #11788

ron819 opened this issue Aug 20, 2018 · 34 comments
Labels
06 - Regression 50 - Duplicate

Comments

@ron819
Copy link
ron819 commented Aug 20, 2018

I'm running Python 3.5.2 and numpy 1.15.0

I'm seeing:

/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)

According to this stackoverflow question I shouldn't see this warring as it was silence since numpy 1.8

Why do I see it? and does it means anything?
@mattip
Copy link
Member
mattip commented Aug 20, 2018

It is a harmless. You can either ignore it or go back to 1.14.5. Will be fixed in the upcoming 1.15.1
Duplicate of #11628. Leave this one open until 1.15.1 is released

@ron819
Copy link
Author
ron819 commented Aug 20, 2018

@mattip Maybe it's best to edit the stackoverflow question with details about the issue with 1.15 if it would appear there this topic would have not been created. The question there is the first search result of google regarding the error.

@ldconejo ldconejo mentioned this issue Sep 6, 2018
Closed
@ChrisBarker-NOAA
Copy link
Contributor

I'm running 1.15.1, and getting the error still :-(

I'm running 1.1.5.1 fron conda-forge on OS-X, with python2.7, if that's relevant.

I'm also making heavy use of Cython -- maybe it's .pyd files are out of date??

@ChrisBarker-NOAA
Copy link
Contributor

NOTE: I'm running Cython code compiled against an older numpy -- maybe that's the issue??

@rgommers
Copy link
Member

You shouldn't get those. Are you clearing the warning filters somewhere? It's this one in numpy's main __init__.py:

warnings.filterwarnings("ignore", message="numpy.dtype size changed")

@charris
Copy link
Member
charris commented Sep 19, 2018

What pytest version?

@randyzwitch randyzwitch mentioned this issue Sep 19, 2018
Closed
@ChrisBarker-NOAA
Copy link
Contributor 
$ pytest --version
This is pytest version 3.8.0

I have noticed a lot more warnings in other contexts as well -- so maybe pytest changed its defaults.

But should numpy be putting out this warning at all?

@rgommers -- I'm not doing anything (myself) about warnings -- but maybe pytest is?

But why is numpy set to ignore a warning that it itself raises???

@rgommers
Copy link
Member

But why is numpy set to ignore a warning that it itself raises???

Long story, the warning comes from Cython and the Cython devs don't want to turn it off. Hence we filter it in __init__.py.

Your warning filters get reset somewhere, that must be the cause. Maybe a new pytest release.

@rgommers
Copy link
Member

Anyway, this is definitely fixed in 1.15.1, so closing. If you find it's a pytest change rather than something in your own code and do think it's something numpy should fix, please open a new issue.

@mattip
Copy link
Member
mattip commented Sep 19, 2018

For the "why" - see cython/cython#2498. From the positive feedback to a possible solution in this comment, there is hope it might get solved.

@charris
Copy link
Member
charris commented Sep 20, 2018

Matti, that isn't the problem with this warning, what is happening is that the C struct has been changed across numpy versions. That is back compatible if the new members are added to the end of the structure, because extension module compiled again earlier numpy versions will never access them. However, cython will complain if numpy reports a structure size different than it was for the numpy that the code was generated against. See #5343 for that added member.

@ChrisBarker-NOAA
Copy link
Contributor

For clarity for future readers, and to make sure I understand the issue, here's a summary:

let's see if I have this right

Problem:

  • The warnings are not created by numpy, but by cython (see above for the link to the issue)

    • cython may fix this soon

  • numpy has addressed this by suppressing the warning it its configuration, so most folks won't see it.

  • pytest has (fairly recently) updated its handling of warnings -- so newer pytest versions will override numpy's suppression of the warning, so you will see it again. (https://docs.pytest.org/en/latest/warnings.html)

  • As for the technical details: Cython is checking that the size of certain structs match between how the extension was complied and what is used at runtime by numpy. If the struct has been expanded since the extension was built (or since the cython numpy.pxd was written??) then it will give a warning. But, in fact, it is not dangerous, as long as the struct is larger, rather than smaller that it used to be.

cython is also (or is this the cause) still using a oder, deprecated numpy API (which leads to a bunch of warnings at compile time) -- updating that is under discussion.

Solutions:

  • ignore the warnings, and hope that Cython updates fix them in the future.

  • Adjust your pytest configuration to suppress these warnings.

NOTE: that PEP 565 recommends that test runners do not suppress warnings:

https://www.python.org/dev/peps/pep-0565/#recommended-filter-settings-for-test-runners

-- makes some sense, there is a lot that you don't want to see at runtime that you probably should see when you run your tests -- at least by default.

Final note: if anyone with good pytest and warnings foo can post a way to suppress this particular warning, that would be much appreciated :-)

@ChrisBarker-NOAA
Copy link
Contributor

@charris: thanks for the explanation -- I wonder if Cython could check to see if the struct is smaller than it was, rather than just different -- then you'd only get the warning when it really mattered!

I guess that's a suggestion for the cython folks

-CHB

@charris
Copy link
Member
charris commented Sep 20, 2018

@ChrisBarker-NOAA I think there is something like that in the code. I don't recall exactly, but you can find the relevant code in the Cython generated C if you are willing to wade through that.

@rgommers
Copy link
Member

pytest has (fairly recently) updated its handling of warnings -- so newer pytest versions will override numpy's suppression of the warning, so you will see it again. (https://docs.pytest.org/en/latest/warnings.html)

just checking: are you sure? that's a long doc page, and it's not clear to me that pytest really changed. did you find a difference in behavior with an older version?

The relevant discussion on the Cython tracker is cython/cython#2221.

@charris
Copy link
Member
charris commented Sep 20, 2018

Two things are involved.

  • Pytest.ini currently overrides the command line. Hence errors need to be suppressed in both
    pytest.ini and in the __init__.py file.
  • Pytest 3.8 started issuing warnings raised during the collection phase.

Pytest will change so that the command line overrides pytests.ini, probably in the 3.9 series.

@rgommers
Copy link
Member

Thanks @charris. That does not sound like a good change. NumPy only has control over its own __init__.py, not over the users command line nor over pytest.ini of other packages. The conclusion seems to be that we need to either

  • convince pytest to change its ways
  • convince cython devs to hide the warning better or give us an option to remove it at compile time

or am I missing something?

@ChrisBarker-NOAA
Copy link
Contributor

I think pytest is doing the right thing -- there is a warning raised, it should be up to the test framework, and the user to determine what warnings they want see.

The idea of numpy (or any package) suppressing warnings that are raised by other code seems pretty dangerous to me. I can see why it seems like the right thing to do in this case, but not in general. (now that I think about it, scipy has cython generated code -- so this is going to come up there)

Frankly, it's a cython issue -- it's raising a warning that it shouldn't. Until someone gets around to fixing it in cython, I think we'll have to live with it. And since I don't have the time to try to make a PR for cython, I know I'll have to live with it.

At least if googling will quickly let people know that don't have to sorry about it -- then we're OK.

-CHB

@mattip
Copy link
Member
mattip commented Sep 20, 2018 •
edited by charris
Loading

@charris is correct that this not cython/cython#2498, rather cython/cython#2221.

@rgommers
Copy link
Member

Frankly, it's a cython issue -- it's raising a warning that it shouldn't.

Well, it's debatable. The Cython devs are saying that the check and warning are useful to them sometimes. It's just never useful for numpy. They really should give us a better option to disable it though.

I think pytest is doing the right thing -- there is a warning raised, it should be up to the test framework, and the user to determine what warnings they want see.

I don't think that's right. Packages should certainly be able to decide which of the warnings that come from their own code or from code below them in the stack should be suppressed.

Say we have a user that writes some code that uses, e.g., seaborn. We then have a stack (just an example, could be incorrect in the details) seaborn -> pandas -> numpy. The warning comes from cython-generated code within numpy. Expecting the user to see, understand or decide on warnings that come from numpy or even lower in the stack makes very little sense. If numpy or pandas devs decide that a warning from numpy is never relevant, there is zero reason to let a seaborn user see it. In summary: pytest is wrong here.

@ChrisBarker-NOAA ChrisBarker-NOAA mentioned this issue Sep 20, 2018
Closed
@ChrisBarker-NOAA
Copy link
Contributor

I think there is no good answer with Python's warning system. For example:

" If numpy or pandas devs decide that a warning from numpy is never relevant, there is zero reason to let a seaborn user see it. "

absolutely -- but the warning doesn't necessarily come from numpy -- in fact, does it ever? numpy used some cython code (yes?) -- but then it gets compiled all together -- so there would never be a numpy version mis-match, would there?

In my case, I'm getting the warning when I run the tests on my "gridded" package, which is pure-python, but depends on (also mine) cell_tree2d package, which is written in cython, and depends on numpy.

So should the numpy project be able to dictate what warnings are seen from my code? In this case, it's all pretty numpy-focused, but doesn't seem like a good policy to build into Python or pytest.

The core problem here is that the numpy devs and the cython devs disagree about how useful this warning is. But if it's useful for Cython developers (and i can sure see how it could be), it seems to me that it should be something that can be turned on and off at cython-compile time.

-CHB

@rgommers
Copy link
Member

but the warning doesn't necessarily come from numpy -- in fact, does it ever?

okay:) let me rephrase: the warning is caused by numpy (because it decided to increase the size of dtype).

The core problem here is that the numpy devs and the cython devs disagree about how useful this warning is.

cython/cython#2221 seems to be converging on a solution

@mattip
Copy link
Member
mattip commented Sep 29, 2018

Fixed in cython/cython#2627 and merged for the upcoming cython release.

@charris
Copy link
Member
charris commented Sep 29, 2018

or am I missing something?

Late response here, but pytest.ini makes warnings errors and currently overrides the init file, which is why the filters need to be in both in order to operate with, or without, the pytest.ini file. I think the proposed change will give the init file priority, or at least enable the test runners to disable/enable warnings on the command line, which I think is a good thing.

@LiMengyang990726
Copy link

You shouldn't get those. Are you clearing the warning filters somewhere? It's this one in numpy's main __init__.py:

warnings.filterwarnings("ignore", message="numpy.dtype size changed")

Hi! I have added this part into my code shown as follows

import warnings

warnings.filterwarnings("ignore", message="numpy.dtype size changed")
warnings.filterwarnings("ignore", message="numpy.ufunc size changed")

While the warning is still there, how can I eliminate it?

@mattip
Copy link
Member
mattip commented Oct 19, 2018

Assuming "your code" is something you are compiling with cython; Cython 0.29, released only recently, should not generate this warning. Can you try upgrading?

@CharlesNord
Copy link

Upgrade to numy 1.15.2 could solve the problem.
conda install numpy=1.15.2

@mattip mattip mentioned this issue Oct 29, 2018
Merged
@fmaussion fmaussion mentioned this issue Nov 22, 2018
Closed
@ebo
Copy link
ebo commented Jan 10, 2019 •
edited
Loading

upgrading to numpy-1.15.4 did not fix it.

BTW, I am currently using cypython-0.29.2

@bfarzin bfarzin mentioned this issue Jan 10, 2019
Closed
@mattip
Copy link
Member
mattip commented Jan 10, 2019

@ebo This issue is closed. Please open a new one with details about how you are obtaining the packages and the exact error message you are getting. Chances are it is a problem not with NumPy itself rather with something interacting with NumPy, and we need to know what that something is.

@mattip
Copy link
Member
mattip commented Jan 10, 2019

Note that the message changed in Cython 0.29, it should now print size changed, may indicate binary incompatibility. Expected %zd from C header, got %zd from PyObject. If your warning is not formatted like this you are using C files generated from an older version of Cython.

@adarob adarob mentioned this issue Jan 15, 2019
Closed
@nicain nicain mentioned this issue Jan 21, 2019
Merged
@CAM-Gerlach CAM-Gerlach mentioned this issue Jan 23, 2019
Closed
@ericaivarsson
Copy link
ericaivarsson commented Feb 4, 2019 •
edited
Loading

Hi, I got the warning after importing the following packages:
import json
import matplotlib.pyplot as plt
import pandas as pd

Maybe this helps in figuring out where the issue is caused. My warning is:
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
return f(*args, **kwds)

@xylar xylar mentioned this issue Feb 19, 2019
Closed
@FilippoBovo FilippoBovo mentioned this issue Mar 25, 2019
Open
@psteinb psteinb mentioned this issue Apr 18, 2019
Closed
@danielballan danielballan mentioned this issue May 16, 2019
Merged
@leofang leofang mentioned this issue May 11, 2020
Merged
@Tejash-Shah Tejash-Shah mentioned this issue Oct 3, 2020
Merged
1 task
@taldcroft taldcroft mentioned this issue Jan 16, 2021
Closed
@advhscmoe
Copy link

I'm running 1.15.1, and getting the error still :-(

I'm running 1.1.5.1 fron conda-forge on OS-X, with python2.7, if that's relevant.

I'm also making heavy use of Cython -- maybe it's .pyd files are out of date??

I am using 1.19 and still getting it.

@mattip
Copy link
Member
mattip commented Mar 6, 2021

@advhscmoe please open a new issue and paste the entire traceback. This issue is closed.

@thuiop thuiop mentioned this issue Jul 28, 2021
Closed
@waisyousofi
Copy link

I went to requirements.txt and added this changes
numpy==1.22.0
and worked for me.

@numpy numpy locked as resolved and limited conversation to collaborators Feb 7, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
06 - Regression 50 - Duplicate
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
0