8000 custom json (de)serialisation (#823) · ag-python/pydantic@973de05 · GitHub
[go: up one dir, main page]
Skip to content

Commit 973de05

Browse files
samuelcolvinsamuelcolvin
authored
custom json (de)serialisation (pydantic#823)
* custom json (d)encoders, fix pydantic#714 * add docs
1 parent 2539835 commit 973de05

File tree

11 files changed

+125
-29
lines changed

11 files changed

+125
-29
lines changed

‎changes/714-samuelcolvin.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Allow custom JSON decoding and encoding via ``json_loads`` and ``json_dumps`` ``Config`` properties.

‎docs/examples/json_orjson.py

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
from datetime import datetime
2+
import orjson
3+
from pydantic import BaseModel
4+
5+
def orjson_dumps(v, *, default):
6+
# orjson.dumps returns bytes, to match standard json.dumps we need to decode
7+
return orjson.dumps(v, default=default).decode()
8+
9+
class User(BaseModel):
10+
id: int
11+
name = 'John Doe'
12+
signup_ts: datetime = None
13+
14+
class Config:
15+
json_loads = orjson.loads
16+
json_dumps = orjson_dumps
17+
18+
19+
user = User.parse_raw('{"id": 123, "signup_ts": 1234567890, "name": "John Doe"}')
20+
print(user.json())
21+
#> {"id":123,"signup_ts":"2009-02-13T23:31:30+00:00","name":"John Doe"}

‎docs/examples/json_ujson.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
from datetime import datetime
2+
import ujson
3+
from pydantic import BaseModel
4+
5+
class User(BaseModel):
6+
id: int
7+
name = 'John Doe'
8+
signup_ts: datetime = None
9+
10+
class Config:
11+
json_loads = ujson.loads
12+
13+
user = User.parse_raw('{"id": 123, "signup_ts": 1234567890, "name": "John Doe"}')
14+
print(user)
15+
#> User id=123 signup_ts=datetime.datetime(2009, 2, 13, 23, 31, 30, tzinfo=datetime.timezone.utc) name='John Doe'

‎docs/index.rst

Lines changed: 38 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -91,19 +91,20 @@ To test if *pydantic* is compiled run::
9191
import pydantic
9292
print('compiled:', pydantic.compiled)
9393

94-
If you want *pydantic* to parse json faster you can add `ujson <https://pypi.python.org/pypi/ujson>`_
95-
as an optional dependency. Similarly *pydantic's* email validation relies on
96-
`email-validator <https://github.com/JoshData/python-email-validator>`_ ::
94+
If you require email validation you can add `email-validator <https://github.com/JoshData/python-email-validator>`_
95+
as an optional dependency. Similarly, use of ``Literal`` relies on
96+
`typing-extensions <https://pypi.org/project/typing-extensions/>`_::
9797

98-
pip install pydantic[ujson]
99-
# or
10098
pip install pydantic[email]
99+
# or
100+
pip install pydantic[typing_extensions]
101101
# or just
102-
pip install pydantic[ujson,email]
102+
pip install pydantic[email,typing_extensions]
103103

104104
Of course you can also install these requirements manually with ``pip install ...``.
105105

106-
Pydantic is also available on `conda <https://www.anaconda.com>`_ under the `conda-forge <https://conda-forge.org>`_ channel::
106+
Pydantic is also available on `conda <https://www.anaconda.com>`_ under the `conda-forge <https://conda-forge.org>`_
107+
channel::
107108

108109
conda install pydantic -c conda-forge
109110

@@ -945,13 +946,14 @@ Options:
945946
Pass in a dictionary with keys matching the error messages you want to override (default: ``{}``)
946947
:arbitrary_types_allowed: whether to allow arbitrary user types for fields (they are validated simply by checking if the
947948
value is instance of that type). If ``False`` - ``RuntimeError`` will be raised on model declaration (default: ``False``)
948-
:json_encoders: customise the way types are encoded to json, see :ref:`JSON Serialisation <json_dump>` for more
949-
details.
950949
:orm_mode: allows usage of :ref:`ORM mode <orm_mode>`
951950
:alias_generator: callable that takes field name and returns alias for it
952951
:keep_untouched: tuple of types (e. g. descriptors) that won't change during model creation and won't be
953-
included in the model schemas.
952+
included in the model schemas
954953
:schema_extra: takes a ``dict`` to extend/update the generated JSON Schema
954+
:json_loads: custom function for decoding JSON, see :ref:`custom JSON (de)serialisation <json_encode_decode>`
955+
:json_dumps: custom function for encoding JSON, see :ref:`custom JSON (de)serialisation <json_encode_decode>`
956+
:json_encoders: customise the way types are encoded to JSON, see :ref:`JSON Serialisation <json_dump>`
955957

956958
.. warning::
957959

@@ -1186,6 +1188,9 @@ Example:
11861188
By default timedelta's are encoded as a simple float of total seconds. The ``timedelta_isoformat`` is provided
11871189
as an optional alternative which implements ISO 8601 time diff encoding.
11881190

1191+
See :ref:`below <json_encode_decode>` for details on how to use other libraries for more performant JSON encoding
1192+
and decoding
1193+
11891194
``pickle.dumps(model)``
11901195
~~~~~~~~~~~~~~~~~~~~~~~
11911196

@@ -1213,6 +1218,29 @@ Of course same can be done on any depth level:
12131218

12141219
Same goes for ``json`` and ``copy`` methods.
12151220

1221+
.. _json_encode_decode:
1222+
1223+
Custom JSON (de)serialisation
1224+
.............................
1225+
1226+
To improve the performance of encoding and decoding JSON, alternative JSON implementations can be used via the
1227+
``json_loads`` and ``json_dumps`` properties of ``Config``, e.g. `ujson <https://pypi.python.org/pypi/ujson>`_.
1228+
1229+
.. literalinclude:: examples/json_ujson.py
1230+
1231+
(This script is complete, it should run "as is")
1232+
1233+
``ujson`` generally cannot be used to dump JSON since it doesn't support encoding of objects like datetimes and does
1234+
not accept a ``default`` fallback function argument. To do this you may use another library like
1235+
`orjson <https://github.com/ijl/orjson>`_.
1236+
1237+
.. literalinclude:: examples/json_orjson.py
1238+
1239+
(This script is complete, it should run "as is")
1240+
1241+
Note that ``orjson`` takes care of ``datetime`` encoding natively, making it faster than ``json.dumps`` but
1242+
meaning you cannot always customise encoding using ``Config.json_encoders``.
1243+
12161244
Abstract Base Classes
12171245
.....................
12181246

‎pydantic/env_settings.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
import json
21
import os
32
from typing import Any, Dict, Optional, cast
43

@@ -47,7 +46,7 @@ def _build_environ(self) -> Dict[str, Optional[str]]:
4746
if env_val:
4847
if field.is_complex():
4948
try:
50-
env_val = json.loads(env_val)
49+
env_val = self.__config__.json_loads(env_val) # type: ignore
5150
except ValueError as e:
5251
raise SettingsError(f'error parsing JSON for "{env_name}"') from e
5352
d[field.alias] = env_val

‎pydantic/main.py

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -63,11 +63,13 @@ class BaseConfig:
6363
validate_assignment = False
6464
error_msg_templates: Dict[str, str] = {}
6565
arbitrary_types_allowed = False
66-
json_encoders: Dict[AnyType, AnyCallable] = {}
6766
orm_mode: bool = False
6867
alias_generator: Optional[Callable[[str], str]] = None
6968
keep_untouched: Tuple[type, ...] = ()
7069
schema_extra: Dict[str, Any] = {}
70+
json_loads: Callable[[str], Any] = json.loads
71+
json_dumps: Callable[..., str] = json.dumps
72+
json_encoders: Dict[AnyType, AnyCallable] = {}
7173

7274
@classmethod
7375
def get_field_schema(cls, name: str) -> Dict[str, str]:
@@ -307,7 +309,7 @@ def json(
307309
data = self.dict(include=include, exclude=exclude, by_alias=by_alias, skip_defaults=skip_defaults)
308310
if self._custom_root_type:
309311
data = data['__root__']
310-
return json.dumps(data, default=encoder, **dumps_kwargs)
312+
return self.__config__.json_dumps(data, default=encoder, **dumps_kwargs)
311313

312314
@classmethod
313315
def parse_obj(cls: Type['Model'], obj: Any) -> 'Model':
@@ -334,7 +336,12 @@ def parse_raw(
334336
) -> 'Model':
335337
try:
336338
obj = load_str_bytes(
337-
b, proto=proto, content_type=content_type, encoding=encoding, allow_pickle=allow_pickle
339+
b,
340+
proto=proto,
341+
content_type=content_type,
342+
encoding=encoding,
343+
allow_pickle=allow_pickle,
344+
json_loads=cls.__config__.json_loads,
338345
)
339346
except (ValueError, TypeError, UnicodeDecodeError) as e:
340347
raise ValidationError([ErrorWrapper(e, loc='__obj__')], cls)
@@ -437,7 +444,7 @@ def schema(cls, by_alias: bool = True) -> 'DictStrAny':
437444
def schema_json(cls, *, by_alias: bool = True, **dumps_kwargs: Any) -> str:
438445
from .json import pydantic_encoder
439446

440-
return json.dumps(cls.schema(by_alias=by_alias), default=pydantic_encoder, **dumps_kwargs)
447+
return cls.__config__.json_dumps(cls.schema(by_alias=by_alias), default=pydantic_encoder, **dumps_kwargs)
441448

442449
@classmethod
443450
def __get_validators__(cls) -> 'CallableGenerator':

‎pydantic/parse.py

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,25 @@
1+
import json
12
import pickle
23
from enum import Enum
34
from pathlib import Path
4-
from typing import Any, Union
5+
from typing import Any, Callable, Union
56

67
from .types import StrBytes
78

8-
try:
9-
import ujson as json
10-
except ImportError:
11-
import json # type: ignore
12-
139

1410
class Protocol(str, Enum):
1511
json = 'json'
1612
pickle = 'pickle'
1713

1814

1915
def load_str_bytes(
20-
b: StrBytes, *, content_type: str = None, encoding: str = 'utf8', proto: Protocol = None, allow_pickle: bool = False
16+
b: StrBytes,
17+
*,
18+
content_type: str = None,
19+
encoding: str = 'utf8',
20+
proto: Protocol = None,
21+
allow_pickle: bool = False,
22+
json_loads: Callable[[str], Any] = json.loads,
2123
) -> Any:
2224
if proto is None and content_type:
2325
if content_type.endswith(('json', 'javascript')):
@@ -32,7 +34,7 @@ def load_str_bytes(
3234
if proto == Protocol.json:
3335
if isinstance(b, bytes):
3436
b = b.decode(encoding)
35-
return json.loads(b)
37+
return json_loads(b)
3638
elif proto == Protocol.pickle:
3739
if not allow_pickle:
3840
raise RuntimeError('Trying to decode with pickle with allow_pickle=False')

‎pydantic/validators.py

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
import json
21
import re
32
import sys
43
from collections import OrderedDict
@@ -424,9 +423,9 @@ def constr_strip_whitespace(v: 'StrBytes', field: 'Field', config: 'BaseConfig')
424423
return v
425424

426425

427-
def validate_json(v: Any) -> Any:
426+
def validate_json(v: Any, config: 'BaseConfig') -> Any:
428427
try:
429-
return json.loads(v)
428+
return config.json_loads(v) # type: ignore
430429
except ValueError:
431430
raise errors.JsonError()
432431
except TypeError:

‎requirements.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22
-r docs/requirements.txt
33
-r tests/requirements.txt
44

5-
ujson==1.35
65
email-validator==1.0.4
76
dataclasses==0.6; python_version < '3.7'
87
typing-extensions==3.7.4

‎setup.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,6 @@ def extra(self):
9999
'dataclasses>=0.6;python_version<"3.7"'
100100
],
101101
extras_require={
102-
'ujson': ['ujson>=1.35'],
103102
'email': ['email-validator>=1.0.3'],
104103
'typing_extensions': ['typing-extensions>=3.7.2']
105104
},

0 commit comments

Comments
 (0)
0