8000 Pandas wrongly read Scipy sparse matrix · Issue #29814 · pandas-dev/pandas · GitHub
[go: up one dir, main page]
Skip to content

Pandas wrongly read Scipy sparse matrix #29814

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
m7142yosuke opened this issue Nov 23, 2019 · 6 comments · Fixed by #31991
Closed

Pandas wrongly read Scipy sparse matrix #29814

m7142yosuke opened this issue Nov 23, 2019 · 6 comments · Fixed by #31991
Labels
Bug Sparse Sparse Data Type
Milestone
1.1

Comments

@m7142yosuke
Copy link
Contributor
m7142yosuke commented Nov 23, 2019 •
edited by jorisvandenbossche
Loading

Code Sample, a copy-pastable example if possible

>>> from scipy.sparse import coo_matrix
>>> sparse_data = coo_matrix((
...     [0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
...     ([0, 1, 1, 2, 2, 3, 3], [0, 1, 2, 0, 2, 0, 1])
... ))
>>> sparse_data.todense()
matrix([[0., 0., 0.],
        [0., 1., 1.],
        [1., 0., 1.],
        [1., 1., 0.]])
>>> pd.DataFrame.sparse.from_spmatrix(sparse_data)
     0    1    2
0  0.0  0.0  0.0
1  0.0  1.0  1.0
2  0.0  0.0  1.0
3  1.0  1.0  0.0
>>> pd.DataFrame.sparse.from_spmatrix(sparse_data.tocsr())
     0    1    2
0  0.0  0.0  0.0
1  0.0  1.0  1.0
2  0.0  0.0  1.0
3  1.0  1.0  0.0

Problem description

sparse_data.todense() and the other matrixes should be same, but not.
(i.e. the [2, 0] value in bottom 2 matrixes should be 1.)

Expected Output

When converting sparse matrix to Pandas DataFrame values of the sparse array should stay the same.

Output of pd.show_versions()

python : 3.7.5.final.0
pandas : 0.25.3
numpy : 1.17.4
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 41.6.0
Cython : None
pytest : 5.3.0
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.2
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
@jorisvandenbossche jorisvandenbossche added Bug Sparse Sparse Data Type labels Nov 23, 2019
@jorisvandenbossche jorisvandenbossche added this to the Contributions Welcome milestone Nov 23, 2019
@jorisvandenbossche
Copy link
Member

Thanks for the report! I can confirm this bug on latest master.

Under the hood, the first column of the dataframe is created as:

In [25]: a = pd.SparseArray.from_spmatrix(sparse_data.tocsc()[:, 0])   

In [26]: a 
Out[26]: 
[0.0, 0.0, 0.0, 1.0]
Fill: 0.0
IntIndex
Indices: array([2, 3], dtype=int32)

So you see that also here the result is wrong. But, the actual indices are correct (indicating [2, 3] for elements 3 and 4 not being sparse), but it's the value of the third element that is wrong (0 instead of 1). If you change the "fill_value", this becomes clearer:

In [27]: a.fill_value = np.nan

In [28]: a    
Out[28]: 
[nan, nan, 0.0, 1.0]
Fill: nan
IntIndex
Indices: array([2, 3], dtype=int32)

This might help to find the cause. Always welcome to look into it!

@m7142yosuke
Copy link
Contributor Author
m7142yosuke commented Nov 23, 2019 •
edited
Loading

Maybe, I found the cause. This problem occurred due to unexpected sparse matrix.
Related code is below

pandas/pandas/core/arrays/sparse/array.py

Lines 418 to 422 in 79e1fc3

arr = data.data
idx, _ = data.nonzero()
loc = np.argsort(idx)
arr = arr.take(loc)
idx.sort()

The above code is sorting sparse index in ascending order.
But the code has a bag which is assuming data.data has no explicit zero.
So, I think it is appropriate to use eliminate_zeros method if Sparse matrix has explicit zeros.

@dwhswenson dwhswenson mentioned this issue Feb 8, 2020
Merged
@dwhswenson
Copy link

One of my projects also got hit by this problem. I can confirm that @m7142yosuke's solution works for me. I implemented that solution as a monkey patch, and our tests pass again. dwhswenson/contact_map#69

@m7142yosuke, do you want to make a PR of this? If you don't want to, I can start a PR. But you found the bug and found the solution; you should get the credit.

It looks like anyone creating sparse dataframes from scipy.sparse matrices may be getting incorrect results from pandas, so this is an important bug to fix.

@m7142yosuke
Copy link
Contributor Author

@dwhswenson
Thank you for your comment.
I thought my solution was wrong, because no reply.
I want to make a PR

@m7142yosuke m7142yosuke mentioned this issue Feb 15, 2020
Merged
5 tasks
@dbarbier dbarbier mentioned this issue Feb 22, 2020
Merged
@jorisvandenbossche
Copy link
Member
jorisvandenbossche commented Mar 25, 2020 •
edited
Loading

So on latest master, the original bug report now is correct:

In [3]: pd.DataFrame.sparse.from_spmatrix(sparse_data) 
Out[3]: 
     0    1    2
0  0.0  0.0  0.0
1  0.0  1.0  1.0
2  1.0  0.0  1.0
3  1.0  1.0  0.0

This might be due to #32825, a PR that refactored from_spmatrix for performance.

But, the SparseArray.from_spmatrix is still buggy:

In [7]: pd.arrays.SparseArray.from_spmatrix(sparse_data.tocsc()[:, 0]) 
Out[7]: 
[0.0, 0.0, 0.0, 1.0]
Fill: 0.0
IntIndex
Indices: array([2, 3], dtype=int32)

@jorisvandenbossche jorisvandenbossche modified the milestones: Contributions Welcome, 1.1 Mar 25, 2020
@dwhswenson dwhswenson mentioned this issue May 31, 2020
Open
@HYChou0515
Copy link

Does #28992 related?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Sparse Sparse Data Type
Projects
None yet
Milestone
1.1
Development

Successfully merging a pull request may close this issue.

BUG: Fix wrong reading sparse matrix
4 participants
@jorisvandenbossche @dwhswenson @HYChou0515 @m7142yosuke
0