8000 Regression for unpickling due to #11166 · Issue #11408 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

Regression for unpickling due to #11166 #11408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jnothman opened this issue Jul 3, 2018 · 2 comments · Fixed by #11471
Closed

Regression for unpickling due to #11166 #11408

jnothman opened this issue Jul 3, 2018 · 2 comments · Fixed by #11471
Labels
Blocker
Milestone
0.20

Comments

@jnothman
Copy link
Member
jnothman commented Jul 3, 2018

As reported in #11166 (comment):

This has broken CircleCI on master, which is trying, if I understand correctly, to load pickles which reference sklearn.externals.joblib.numpy_pickle, resulting in an ImportError.

Firstly, I presume to fix Circle, we just need to clear a cache somewhere (but I've not worked out where).

Secondly, are we unnecessarily breaking pickles from previous versions? Should we see if we can find a way to make this function importable when loading a joblib pickle?

See subsequent discussion there.

This issue is just to make sure we have some resolution before 0.20 release
@jnothman jnothman added this to the 0.20 milestone Jul 3, 2018
@jnothman
Copy link
Member Author
jnothman commented Jul 5, 2018

I'm getting annoyed by all the circle failures. Should we just switch the .joblib and ._joblib?

@jnothman
Copy link
Member Author

I've tried using six.moves magic, and tried using slightly less magic, but I'm not making much progress. I agree it's quite annoying that we might need to leave sklearn.externals.joblib there and have uses of sklearn.externals._joblib in examples, etc.

One option might be to stop using sklearn.externals.joblib internally, by moving all components we use to sklearn.utils`.

from sklearn.externals.joblib import load


def test_old_pickle(tmpdir):
    # Check that a pickle that references sklearn.external.joblib can load
    f = tmpdir.join('foo.pkl')
    f.write(b'\x80\x03csklearn.externals.joblib.numpy_pickle\nNumpyArrayWrappe'
            b'r\nq\x00)\x81q\x01}q\x02(X\n\x00\x00\x00allow_mmapq\x03\x88X\x05'
            b'\x00\x00\x00shapeq\x04K\x01\x85q\x05X\x05\x00\x00\x00dtypeq\x06c'
            b'numpy\ndtype\nq\x07X\x02\x00\x00\x00i8q\x08K\x00K\x01\x87q\tRq\n'
            b'(K\x03X\x01\x00\x00\x00<q\x0bNNNJ\xff\xff\xff\xffJ\xff\xff\xff'
            b'\xffK\x00tq\x0cbX\x08\x00\x00\x00subclassq\rcnumpy\nndarray\nq'
            b'\x0eX\x05\x00\x00\x00orderq\x0fX\x01\x00\x00\x00Cq\x10ub\x01\x00'
            b'\x00\x00\x00\x00\x00\x00.', mode='wb')

    load(str(f))

This was referenced Jul 10, 2018
Closed
Merged
@lesteve lesteve mentioned this issue Jul 15, 2018
Merged
ogrisel pushed a commit that referenced this issue Jul 17, 2018
@jnothman @ogrisel
Restructure access to vendored/site Joblib (#11471)
14e7c32 
In order to fix #11408, this swaps `joblib` and `_joblib`. It however, allows users to access joblib's `Memory` or `Parallel` functionality without accessing `sklearn.externals._joblib` by importing `Memory`, `Parallel`, etc. into `sklearn.utils`.
@rth rth mentioned this issue Jan 5, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Blocker
Projects
None yet
Milestone
0.20
Development

Successfully merging a pull request may close this issue.

[MRG+1] Restructure access to vendored/site Joblib jnothman/scikit-learn
1 participant
@jnothman
0