8000 MeCab 실행시 PicklingError: Could not pickle the task to send it to the workers. 오류 · Issue #1 · rickiepark/introduction_to_ml_with_python · GitHub
[go: up one dir, main page]
Skip to content

MeCab 실행시 PicklingError: Could not pickle the task to send it to the workers. 오류 #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

8000 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
YongBeomKim opened this issue Apr 1, 2019 · 4 comments
Closed

MeCab 실행시 PicklingError: Could not pickle the task to send it to the workers. 오류 #1

YongBeomKim opened this issue Apr 1, 2019 · 4 comments

Comments

@YongBeomKim
Copy link
YongBeomKim commented Apr 1, 2019 •
edited
Loading

마지막장 konlpy 응용 부분에서 scikit-learn 을 0.20 최신 버전으로 실습하면
MeCab을 활용한 GridSearch 부분에서 n_job을 -1로 변경하고 실행하면 오류가 나네요 ㅠㅠ
대신 scikit-learn 을 0.19로 내리면 잘 작동 됩니다. 관련해서 아시는 분들 도움 부탁드려요

mecab_pipe = make_pipeline(TfidfVectorizer(tokenizer = mecab_tokenizer), 
                           LogisticRegression(solver = 'liblinear'),
                           memory = cache_dir)
mecab_grid = GridSearchCV(mecab_pipe, mecab_param_grid, n_jobs=-1, cv=3)
mecab_grid.fit(text_train, y_train)  # 그리드 서치를 수행
/usr/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result
PicklingError: Could not pickle the task to send it to the workers.
@YongBeomKim YongBeomKim changed the title MeCab 실행시 PicklingError: Could not pickle the task to send it to the workers. 오류가 발생 MeCab 실행시 PicklingError: Could not pickle the task to send it to the workers. 오류 Apr 1, 2019
@rickiepark
Copy link
Owner

안녕하세요. 재현해 보고 다시 코멘트 드리겠습니다. 감사합니다.

@rickiepark
Copy link
Owner

@YongBeomKim 안녕하세요. 새 책을 마감하느라 까맣게 잊고 있었네요. ㅠ.ㅠ
간단하게 Mecab 클래스를 감싸서 pickling을 할 수 있네요.
테스트가 완료되면 푸시하고 다시 알려 드리겠습니다.
즐거운 주말 되세요. :)

@YongBeomKim
Copy link
Author

넵 ^^;;
아직까지 0.19로 잘 되어서 이걸로 작업하고 있습니다...

@rickiepark
Copy link
Owner

n_jobs=-1로 주니 제법(?) 빨리 끝나네요! :)
사이킷런 0.20 버전부터 사용하는 joblib 0.12 버전이 멀티프로세싱 라이브러리를 loky로 바꾸어서 생기는 문제 같습니다.(scikit-learn/scikit-learn#11741)
간단한 해결책은 피클링이 가능하도록 상태 관리 메서드를 추가해 감싸는 것입니다.(https://stackoverflow.com/questions/9310053/how-to-make-my-swig-extension-module-work-with-pickle)

class PicklableMecab(Mecab):

    def __init__(self, *args):
        self.args = args
        Mecab.__init__(self, *args)
    
    def __setstate__(self, state):
        self.__init__(*state['args'])

    def __getstate__(self):
        return {'args': self.args}

mecab = PicklableMecab()

실행 결과는 주피터 노트북을 참고해 주세요.(https://github.com/rickiepark/introduction_to_ml_with_python/blob/master/07-konlpy.ipynb)
konlpy에도 PR 넣어야 겠네요.
감사합니다!

@rickiepark rickiepark mentioned this issue May 24, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rickiepark @YongBeomKim
0