Pandas wrongly read Scipy sparse matrix #29814
Comments
|
Thanks for the report! I can confirm this bug on latest master. Under the hood, the first column of the dataframe is created as: So you see that also here the result is wrong. But, the actual indices are correct (indicating [2, 3] for elements 3 and 4 not being sparse), but it's the value of the third element that is wrong (0 instead of 1). If you change the "fill_value", this becomes clearer: This might help to find the cause. Always welcome to look into it! |
|
Maybe, I found the cause. This problem occurred due to unexpected sparse matrix. pandas/pandas/core/arrays/sparse/array.py Lines 418 to 422 in 79e1fc3 The above code is sorting sparse index in ascending order. |
|
One of my projects also got hit by this problem. I can confirm that @m7142yosuke's solution works for me. I implemented that solution as a monkey patch, and our tests pass again. dwhswenson/contact_map#69 @m7142yosuke, do you want to make a PR of this? If you don't want to, I can start a PR. But you found the bug and found the solution; you should get the credit. It looks like anyone creating sparse dataframes from scipy.sparse matrices may be getting incorrect results from pandas, so this is an important bug to fix. |
|
@dwhswenson |
|
So on latest master, the original bug report now is correct: This might be due to #32825, a PR that refactored But, the SparseArray.from_spmatrix is still buggy: |
|
Does #28992 related? |
Uh oh!
There was an error while loading. Please reload this page.
Code Sample, a copy-pastable example if possible
Problem description
sparse_data.todense()and the other matrixes should be same, but not.(i.e. the [2, 0] value in bottom 2 matrixes should be 1.)
Expected Output
When converting sparse matrix to Pandas DataFrame values of the sparse array should stay the same.
Output of
pd.show_versions()python : 3.7.5.final.0
pandas : 0.25.3
numpy : 1.17.4
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 41.6.0
Cython : None
pytest : 5.3.0
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.2
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
The text was updated successfully, but these errors were encountered: