8000 gh-99631: Add custom `loads` and `dumps` support for the `shelve` mod… · python/cpython@dda70fa · GitHub
[go: up one dir, main page]
Skip to content

Commit dda70fa

Browse files
furkanondereendebakptencukoupicnixz
authored
gh-99631: Add custom loads and dumps support for the shelve module (#118065)
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com> Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
1 parent c564847 commit dda70fa

File tree

4 files changed

+336
-36
lines changed

4 files changed

+336
-36
lines changed

‎Doc/library/shelve.rst

Lines changed: 61 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,8 @@ This includes most class instances, recursive data types, and objects containing
1717
lots of shared sub-objects. The keys are ordinary strings.
1818

1919

20-
.. function:: open(filename, flag='c', protocol=None, writeback=False)
20+
.. function:: open(filename, flag='c', protocol=None, writeback=False, *, \
21+
serializer=None, deserializer=None)
2122

2223
Open a persistent dictionary. The filename specified is the base filename for
2324
the underlying database. As a side-effect, an extension may be added to the
@@ -41,13 +42,32 @@ lots of shared sub-objects. The keys are ordinary strings.
4142
determine which accessed entries are mutable, nor which ones were actually
4243
mutated).
4344

45+
By default, :mod:`shelve` uses :func:`pickle.dumps` and :func:`pickle.loads`
46+
for serializing and deserializing. This can be changed by supplying
47+
*serializer* and *deserializer*, respectively.
48+
49+
The *serializer 8000 * argument must be a callable which takes an object ``obj``
50+
and the *protocol* as inputs and returns the representation ``obj`` as a
51+
:term:`bytes-like object`; the *protocol* value may be ignored by the
52+
serializer.
53+
54+
The *deserializer* argument must be callable which takes a serialized object
55+
given as a :class:`bytes` object and returns the corresponding object.
56+
57+
A :exc:`ShelveError` is raised if *serializer* is given but *deserializer*
58+
is not, or vice-versa.
59+
4460
.. versionchanged:: 3.10
4561
:const:`pickle.DEFAULT_PROTOCOL` is now used as the default pickle
4662
protocol.
4763

4864
.. versionchanged:: 3.11
4965
Accepts :term:`path-like object` for filename.
5066

67+
.. versionchanged:: next
68+
Accepts custom *serializer* and *deserializer* functions in place of
69+
:func:`pickle.dumps` and :func:`pickle.loads`.
70+
5171
.. note::
5272

5373
Do not rely on the shelf being closed automatically; always call
@@ -129,7 +149,8 @@ Restrictions
129149
explicitly.
130150

131151

132-
.. class:: Shelf(dict, protocol=None, writeback=False, keyencoding='utf-8')
152+
.. class:: Shelf(dict, protocol=None, writeback=False, \
153+
keyencoding='utf-8', *, serializer=None, deserializer=None)
133154
134155
A subclass of :class:`collections.abc.MutableMapping` which stores pickled
135156
values in the *dict* object.
@@ -147,6 +168,9 @@ Restrictions
147168
The *keyencoding* parameter is the encoding used to encode keys before they
148169
are used with the underlying dict.
149170

171+
The *serializer* and *deserializer* parameters have the same interpretation
172+
as in :func:`~shelve.open`.
173+
150174
A :class:`Shelf` object can also be used as a context manager, in which
151175
case it will be automatically closed when the :keyword:`with` block ends.
152176

@@ -161,8 +185,13 @@ Restrictions
161185
:const:`pickle.DEFAULT_PROTOCOL` is now used as the default pickle
162186
protocol.
163187

188+
.. versionchanged:: next
189+
Added the *serializer* and *deserializer* parameters.
164190

165-
.. class:: BsdDbShelf(dict, protocol=None, writeback=False, keyencoding='utf-8')
191+
192+
.. class:: BsdDbShelf(dict, protocol=None, writeback=False, \
193+
keyencoding='utf-8', *, \
194+
serializer=None, deserializer=None)
166195
167196
A subclass of :class:`Shelf` which exposes :meth:`!first`, :meth:`!next`,
168197
:meth:`!previous`, :meth:`!last` and :meth:`!set_location` methods.
@@ -172,18 +201,27 @@ Restrictions
172201
modules. The *dict* object passed to the constructor must support those
173202
methods. This is generally accomplished by calling one of
174203
:func:`!bsddb.hashopen`, :func:`!bsddb.btopen` or :func:`!bsddb.rnopen`. The
175-
optional *protocol*, *writeback*, and *keyencoding* parameters have the same
176-
interpretation as for the :class:`Shelf` class.
204+
optional *protocol*, *writeback*, *keyencoding*, *serializer* and *deserializer*
205+
parameters have the same interpretation as in :func:`~shelve.open`.
206+
207+
.. versionchanged:: next
208+
Added the *serializer* and *deserializer* parameters.
177209

178210

179-
.. class:: DbfilenameShelf(filename, flag='c', protocol=None, writeback=False)
211+
.. class:: DbfilenameShelf(filename, flag='c', protocol=None, \
212+
writeback=False, *, serializer=None, \
213+
deserializer=None)
180214
181215
A subclass of :class:`Shelf` which accepts a *filename* instead of a dict-like
182216
object. The underlying file will be opened using :func:`dbm.open`. By
183217
default, the file will be created and opened for both read and write. The
184-
optional *flag* parameter has the same interpretation as for the :func:`.open`
185-
function. The optional *protocol* and *writeback* parameters have the same
186-
interpretation as for the :class:`Shelf` class.
218+
optional *flag* parameter has the same interpretation as for the
219+
:func:`.open` function. The optional *protocol*, *writeback*, *serializer*
220+
and *deserializer* parameters have the same interpretation as in
221+
:func:`~shelve.open`.
222+
223+
.. versionchanged:: next
224+
Added the *serializer* and *deserializer* parameters.
187225

188226

189227
.. _shelve-example:
@@ -225,6 +263,20 @@ object)::
225263
d.close() # close it
226264

227265

266+
Exceptions
267+
----------
268+
269+
.. exception:: ShelveError
270+
271+
Exception raised when one of the arguments *deserializer* and *serializer*
272+
is missing in the :func:`~shelve.open`, :class:`Shelf`, :class:`BsdDbShelf`
273+
and :class:`DbfilenameShelf`.
274+
275+
The *deserializer* and *serializer* arguments must be given together.
276+
277+
.. versionadded:: next
278+
279+
228280
.. seealso::
229281

230282
Module :mod:`dbm`

‎Lib/shelve.py

Lines changed: 38 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -56,12 +56,17 @@
5656
the persistent dictionary on disk, if feasible).
5757
"""
5858

59-
from pickle import DEFAULT_PROTOCOL, Pickler, Unpickler
59+
from pickle import DEFAULT_PROTOCOL, dumps, loads
6060
from io import BytesIO
6161

6262
import collections.abc
6363

64-
__all__ = ["Shelf", "BsdDbShelf", "DbfilenameShelf", "open"]
64+
__all__ = ["ShelveError", "Shelf", "BsdDbShelf", "DbfilenameShelf", "open"]
65+
66+
67+
class ShelveError(Exception):
68+
pass
69+
6570

6671
class _ClosedDict(collections.abc.MutableMapping):
6772
'Marker for a closed dict. Access attempts raise a ValueError.'
@@ -82,7 +87,7 @@ class Shelf(collections.abc.MutableMapping):
8287
"""
8388

8489
def __init__(self, dict, protocol=None, writeback=False,
85-
keyencoding="utf-8"):
90+
keyencoding="utf-8", *, serializer=None, deserializer=None):
8691
self.dict = dict
8792
if protocol is None:
8893
protocol = DEFAULT_PROTOCOL
@@ -91,6 +96,16 @@ def __init__(self, dict, protocol=None, writeback=False,
9196
self.cache = {}
9297
self.keyencoding = keyencoding
9398

99+
if serializer is None and deserializer is None:
100+
self.serializer = dumps
101+
self.deserializer = loads
102+
elif (serializer is None) ^ (deserializer is None):
103+
raise ShelveError("serializer and deserializer must be "
104+
"defined together")
105+
else:
106+
self.serializer = serializer
107+
self.deserializer = deserializer
108+
94109
def __iter__(self):
95110
for k in self.dict.keys():
96111
yield k.decode(self.keyencoding)
@@ -110,19 +125,17 @@ def __getitem__(self, key):
110125
try:
111126
value = self.cache[key]
112127
except KeyError:
113-
f = BytesIO(self.dict[key.encode(self.keyencoding)])
114-
value = Unpickler(f).load()
128+
f = self.dict[key.encode(self.keyencoding)]
129+
value = self.deserializer(f)
115130
if self.writeback:
116131
self.cache[key] = value
117132
return value
118133

119134
def __setitem__(self, key, value):
120135
if self.writeback:
121136
self.cache[key] = value
122-
f = BytesIO()
123-
p = Pickler(f, self._protocol)
124-
p.dump(value)
125-
self.dict[key.encode(self.keyencoding)] = f.getvalue()
137+
serialized_value = self.serializer(value, self._protocol)
138+
self.dict[key.encode(self.keyencoding)] = serialized_value
126139

127140
def __delitem__(self, key):
128141
del self.dict[key.encode(self.keyencoding)]
@@ -191,33 +204,29 @@ class BsdDbShelf(Shelf):
191204
"""
192205

193206
def __init__(self, dict, protocol=None, writeback=False,
194-
keyencoding="utf-8"):
195-
Shelf.__init__(self, dict, protocol, writeback, keyencoding)
207+
keyencoding="utf-8", *, serializer=None, deserializer=None):
208+
Shelf.__init__(self, dict, protocol, writeback, keyencoding,
209+
serializer=serializer, deserializer=deserializer)
196210

197211
def set_location(self, key):
198212
(key, value) = self.dict.set_location(key)
199-
f = BytesIO(value)
200-
return (key.decode(self.keyencoding), Unpickler(f).load())
213+
return (key.decode(self.keyencoding), self.deserializer(value))
201214

202215
def next(self):
203216
(key, value) = next(self.dict)
204-
f = BytesIO(value)
205-
return (key.decode(self.keyencoding), Unpickler(f).load())
217+
return (key.decode(self.keyencoding), self.deserializer(value))
206218

207219
def previous(self):
208220
(key, value) = self.dict.previous()
209-
f = BytesIO(value)
210-
return (key.decode(self.keyencoding), Unpickler(f).load())
221+
return (key.decode(self.keyencoding), self.deserializer(value))
211222

212223
def first(self):
213224
(key, value) = self.dict.first()
214-
f = BytesIO(value)
215-
return (key.decode(self.keyencoding), Unpickler(f).load())
225+
return (key.decode(self.keyencoding), self.deserializer(value))
216226

217227
def last(self):
218228
(key, value) = self.dict.last()
219-
f = BytesIO(value)
220-
return (key.decode(self.keyencoding), Unpickler(f).load())
229+
return (key.decode(self.keyencoding), self.deserializer(value))
221230

222231

223232
class DbfilenameShelf(Shelf):
@@ -227,9 +236,11 @@ class DbfilenameShelf(Shelf):
227236
See the module's __doc__ string for an overview of the interface.
228237
"""
229238

230-
def __init__(self, filename, flag='c', protocol=None, writeback=False):
239+
def __init__(self, filename, flag='c', protocol=None, writeback=False, *,
240+
serializer=None, deserializer=None):
231241
import dbm
232-
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
242+
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback,
243+
serializer=serializer, deserializer=deserializer)
233244

234245
def clear(self):
235246
"""Remove all items from the shelf."""
@@ -238,8 +249,8 @@ def clear(self):
238249
self.cache.clear()
239250
self.dict.clear()
240251

241-
242-
def open(filename, flag='c', protocol=None, writeback=False):
252+
def open(filename, flag='c', protocol=None, writeback=False, *,
253+
serializer=None, deserializer=None):
243254
"""Open a persistent dictionary for reading and writing.
244255
245256
The filename parameter is the base filename for the underlying
@@ -252,4 +263,5 @@ def open(filename, flag='c', protocol=None, writeback=False):
252263
See the module's __doc__ string for an overview of the interface.
253264
"""
254265

255-
return DbfilenameShelf(filename, flag, protocol, writeback)
266+
return DbfilenameShelf(filename, flag, protocol, writeback,
267+
serializer=serializer, deserializer=deserializer)

0 commit comments

Comments
 (0)
0