8000 Problem with packing/unpacking tuples which is big obstacle in complex structures · Issue #202 · msgpack/msgpack-python · GitHub
[go: up one dir, main page]
Skip to content

Problem with packing/unpacking tuples which is big obstacle in complex structures #202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ChameleonRed opened this issue Jul 20, 2016 · 7 comments
Closed

Problem with packing/unpacking tuples which is big obstacle in complex structures #202

ChameleonRed opened this issue Jul 20, 2016 · 7 comments

Comments

@ChameleonRed
Copy link
ChameleonRed commented Jul 20, 2016 •
edited
Loading

Here is some problem with pack/unpack tuples. As I know msgpack not distinguish between list and tuple and there is not hook to force list or tuple be ExtType. It generates frustrating problems.

See simple example class with hash - nothing special:

import msgpack

class Period(object):
    def __init__(self, key):
        self.key = key

    def __hash__(self):
        return hash(self.key)

    def __eq__(self, other):
        self.key == self.key

def encode(o):
    if type(o) is Period:
        return msgpack.ExtType(0, msgpack.dumps(o.__dict__))

def decode_ext(code, data):
    if code == 0:
        o = Period.__new__(Period)
        o.__dict__ = msgpack.loads(data)
        return o

o = {Period((2016, 7)): 112, Period((2016, 8)): 231}

print o
s = msgpack.dumps(o, default=encode)
print s
o2 = msgpack.loads(s, ext_hook=decode_ext)
print o2

It generates problem during unpacking which cannot be solved easily I think:

C:\root\Python27-64\python.exe "C:/Users/Cezary Wagner/PycharmProjects/msgpack_learn/src/02_tuple_wrong_pack.py"
Traceback (most recent call last):
{<__main__.Period object at 0x0000000002941668>: 231, <__main__.Period object at 0x0000000002941AC8>: 112}
  File "C:/Users/Cezary Wagner/PycharmProjects/msgpack_learn/src/02_tuple_wrong_pack.py", line 28, in <module>
��
    o2 = msgpack.loads(s, ext_hook=decode_ext)
 ��key��������
  File "msgpack/_unpacker.pyx", line 139, in msgpack._unpacker.unpackb (msgpack/_unpacker.cpp:139)
 ��key�����p
  File "C:/Users/Cezary Wagner/PycharmProjects/msgpack_learn/src/02_tuple_wrong_pack.py", line 8, in __hash__
    return hash(self.key)
TypeError: unhashable type: 'list'

Process finished with exit code 1

Do you have any idea how to reconstruct tuple to tuples and list to lists using msgpack if it possible at all?
@methane methane mentioned this issue Jul 21, 2016
Merged
@methane
Copy link
Member
methane commented Jul 21, 2016

@ChameleonRed Could you try this?
https://ci.appveyor.com/api/buildjobs/g3824gsxbnbhw7k7/artifacts/dist/msgpack_python-0.4.7-cp35-cp35m-win_amd64.whl

@ChameleonRed
Copy link
Author
ChameleonRed commented Jul 22, 2016 •
edited
Loading

I will try soon. Current I create __msgpack_post_decode__ to fix all object with use custom functions.

Same problem is with str and unicode. That is very ugly.

@methane methane mentioned this issue Jul 29, 2016
Closed
@methane methane reopened this Jul 29, 2016
@methane
Copy link
Member
methane commented Jul 29, 2016

I'm sorry, #203 solves #201, not this.

@methane
Copy link
Member
methane commented Jul 29, 2016

Basically speaking, msgpack is json-like, simple and cross-language format.
It means, if you want to serialize complex type, you shouldn't use msgpack.
Pickle is for you.

In this case, there are options for you: use_bin_type in packer and use_list, encoding in unpacker.
You should read options carefully, before filing issue.

There are many library users, and only one library maintainer. Filing issue to ask question
to library owner cause burn-out easily.
Users should try to solve their issue at user community first, to reduce developer's load.

@methane methane closed this as completed Jul 29, 2016
@ChameleonRed
Copy link
Author
ChameleonRed commented Jul 29, 2016 •
edited
Loading

I am not asking question at all but suggest you change in library since it can not be solve simple by use use_bin_type, use_list or encoding as it was explained.

It need to force treat tuple and other types as input for "default" encoder - you need add exceptions not pack tuple or list as list, tuple or list as tuple - that will not work - why hash(list()) is exception, tuple().append() is not possible.

It limits creation of generic decoder, encoder - it is not limited by msgpack at all.

Can you propose some solution?

  • exception for types (exclude i.e. tuple from standard types and allow pass to default)
  • hook before standard packing tuple or list or any other type
  • external encoder enveloping tuples or list or other types before pass objects to msgpack
  • something what is you proposal

@ChameleonRed
Copy link
Author
ChameleonRed commented Jul 29, 2016 •
edited
Loading

You can be sure that I do not waste your time with questions. I experienced programmer so I can fork and fix you library without problem if there is not law restrictions. You code is nice and clear to be fixed (pure python).

@methane
Copy link
Member
methane commented Jul 29, 2016

I am not asking question at all but suggest you change in library.

Have you searched existing issues, or checked master branch at least?
There is strict_types option in master branch, already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@methane @ChameleonRed
0