pydantic · sydney-runkle · Jun 6, 2024 · May 20, 2024 · May 20, 2024 · May 20, 2024
diff --git a/docs/api/experimental.md b/docs/api/experimental.md
@@ -0,0 +1,4 @@
+::: pydantic.experimental.pipeline
+    options:
+        members:
+            - _Pipeline
diff --git a/docs/concepts/experimental.md b/docs/concepts/experimental.md
@@ -0,0 +1,116 @@
+# Experimental Features
+
+In this section you will find documentation for new, experimental features in Pydantic. These features are subject to change or removal, and we are looking for feedback and suggestions before making them a permanent part of Pydantic.
+
+See our [Version Policy](../version-policy.md#experimental-features) for more information on experimental features.
+
+## Feedback
+
+We welcome feedback on experimental features! Please open an issue on the [Pydantic GitHub repository](https://github.com/pydantic/pydantic/issues/new/choose) to share your thoughts, requests, or suggestions.
+
+We also encourage you to read through existing feedback and add your thoughts to existing issues.
+
+## Warnings on Import
+
+When you import an experimental feature from the `experimental` module, you'll see a warning message that the feature is experimental. You can disable this warning with the following:
+
+```python
+import warnings
+
+from pydantic import PydanticExperimentalWarning
+
+warnings.filterwarnings('ignore', category=PydanticExperimentalWarning)
+```
+
+## Pipeline API
+
+Pydantic v2.8.0 introduced an experimental "pipeline" API that allows composing of parsing (validation), constraints and transformations in a more type-safe manner than existing APIs. This API is subject to change or removal, we are looking for feedback and suggestions before making it a permanent part of Pydantic.
+
+??? api "API Documentation"
+    [`pydantic.experimental.pipeline`][pydantic.experimental.pipeline]<br>
+
+Generally, the pipeline API is used to define a sequence of steps to apply to incoming data during validation. The pipeline API is designed to be more type-safe and composable than the existing Pydantic API.
+
+Each step in the pipeline can be:
+* A validation step that runs pydantic validation on the provided type
+* A transformation step that modifies the data
+* A constraint step that checks the data against a condition
+* A predicate step that checks the data against a condition and raises an error if it returns `False`
+
+<!-- TODO: (@sydney-runkle) add more documentation once we solidify the API during the experimental phase -->
+
+```python
+from __future__ import annotations
+
+from datetime import datetime
+
+from typing_extensions import Annotated
+
+from pydantic import BaseModel
+from pydantic.experimental.pipeline import validate_as, validate_as_deferred
+
+
+class User(BaseModel):
+    name: Annotated[str, validate_as(str).str_lower()]  # (1)!
+    age: Annotated[int, validate_as(int).gt(0)]  # (2)!
+    username: Annotated[str, validate_as(str).str_pattern(r'[a-z]+')]  # (3)!
+    password: Annotated[
+        str,
+        validate_as(str)
+        .transform(str.lower)
+        .predicate(lambda x: x != 'password'),  # (4)!
+    ]
+    favorite_number: Annotated[  # (5)!
+        int,
+        (validate_as(int) | validate_as(str).str_strip().validate_as(int)).gt(
+            0
+        ),
+    ]
+    friends: Annotated[list[User], validate_as(...).len(0, 100)]  # (6)!
+    family: Annotated[  # (7)!
+        list[User],
+        validate_as_deferred(lambda: list[User]).transform(lambda x: x[1:]),
+    ]
+    bio: Annotated[
+        datetime,
+        validate_as(int)
+        .transform(lambda x: x / 1_000_000)
+        .validate_as(...),  # (8)!
+    ]
+```
+
+1. Lowercase a string.
+2. Constrain an integer to be greater than zero.
+3. Constrain a string to match a regex pattern.
+4. You can also use the lower level transform, constrain and predicate methods.
+5. Use the `|` or `&` operators to combine steps (like a logical OR or AND).
+6. Calling `validate_as(...)` with `Ellipsis`, `...` as the first positional argument implies `validate_as(<field type>)`. Use `validate_as(Any)` to accept any type.
+7. For recursive types you can use `validate_as_deferred` to reference the type itself before it's defined.
+8. You can call `validate_as()` before or after other steps to do pre or post processing.
+
+### Mapping from `BeforeValidator`, `AfterValidator` and `WrapValidator`
+
+The `validate_as` method is a more type-safe way to define `BeforeValidator`, `AfterValidator` and `WrapValidator`:
+
+```python
+from typing_extensions import Annotated
+
+from pydantic.experimental.pipeline import transform, validate_as
+
+# BeforeValidator
+Annotated[int, validate_as(str).str_strip().validate_as(...)]  # (1)!
+# AfterValidator
+Annotated[int, transform(lambda x: x * 2)]  # (2)!
+# WrapValidator
+Annotated[
+    int,
+    validate_as(str)
+    .str_strip()
+    .validate_as(...)
+    .transform(lambda x: x * 2),  # (3)!
+]
+```
+
+1. Strip whitespace from a string before parsing it as an integer.
+2. Multiply an integer by 2 after parsing it.
+3. Strip whitespace from a string, validate it as an integer, then multiply it by 2.
diff --git a/docs/version-policy.md b/docs/version-policy.md
@@ -29,6 +29,54 @@ In all cases we will aim to minimize churn and do so only when justified by the
 
 We expect to make new major releases roughly once a year going forward, although as mentioned above, any associated breaking changes should be trivial to fix compared to the V1-to-V2 transition.
 
+## Experimental Features
+
+At Pydantic, we like to move quickly and innovate! To that end, we may introduce experimental features in minor releases.
+
+!!! abstract "Usage Documentation"
+    To learn more about our current experimental features, see the [experimental features documentation](./concepts/experimental.md).
+
+Please keep in mind, experimental features are active works in progress. If these features are successful, they'll eventually beocme part of Pydantic. If unsuccessful, said features will be removed with little notice. While in its experimental phase, a feature's API and behaviors may not be stable, and it's very possible that changes made to the feature will not be backward-compatible.
+
+### Naming Conventions
+
+We use one of the following naming conventions to indicate that a feature is experimental:
+
+1. The feature is located in the `experimental` module. In this case, you can access the feature like this:
+
+    ```python test="skip" lint="skip"
+    from pydantic.experimental import feature_name
+    ```
+
+2. The feature is located in the main module, but prefixed with `experimental_`. This case occurs when we add a new field, argument, or method to an existing data structure already within the main `pydantic` module.
+
+New features with these naming conventions are subject to change or removal, and we are looking for feedback and suggestions before making them a permanent part of Pydantic. See the [feedback section](./concepts/experimental.md#feedback) for more information.
+
+### Importing Experimental Features
+
+When you import an experimental feature from the `experimental` module, you'll see a warning message that the feature is experimental. You can disable this warning with the following:
+
+```python
+import warnings
+
+from pydantic import PydanticExperimentalWarning
+
+warnings.filterwarnings('ignore', category=PydanticExperimentalWarning)
+```
+
+### Lifecycle of Experimental Features
+
+1. A new feature is added, either in the `experimental` module or with the `experimental_` prefix.
+2. The behavior is often modified during patch/minor releases, with potential API/behavior changes.
+3. If the feature is successful, we promote it to Pydantic with the following steps:
+    a. If it was in the `experimental` module, the feature is cloned to Pydantic's main module. The original experimental feature still remains in the `experimental` module, but it will show a warning when used. If the feature was already in the main Pydantic module, we create a copy of the feature without the `experimental_` prefix, so the feature exists with both the official and experimental names. A deprecation warning is attached to the experimental version.
+    b. At some point, the code of the experimental feature is removed, but there will still be a stub of the feature that provides an error message with appropriate instructions.
+    c. As a last step, the experimental version of the feature is entirely removed from the codebase.
+
+If the feature is unsuccessful or unpopular, it's removed with little notice. A stub will remain in the location of the deprecated feature with an error message.
+
+Thanks to [streamlit](https://docs.streamlit.io/develop/quick-reference/prerelease) for the inspiration for the lifecycle and naming conventions of our new experimental feature patterns.
+
 ## Support for Python versions
 
 Pydantic will drop support for a Python version when the following conditions are met:

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -104,6 +104,7 @@ nav:
   - Settings Management: concepts/pydantic_settings.md
   - Performance: concepts/performance.md
   - Pydantic Plugins: concepts/plugins.md
+  - Experimental: concepts/experimental.md
 - API Documentation:
   - Pydantic:
     - BaseModel: api/base_model.md
@@ -124,6 +125,7 @@ nav:
     - Version Information: api/version.md
     - Pydantic Plugins: api/plugin.md
     - Annotated Handlers: api/annotated_handlers.md
+    - Experimental: api/experimental.md
   - Pydantic Core:
     - pydantic_core: api/pydantic_core.md
     - pydantic_core.core_schema: api/pydantic_core_schema.md

diff --git a/pydantic/__init__.py b/pydantic/__init__.py
@@ -45,7 +45,12 @@
     from .type_adapter import TypeAdapter
     from .types import *
     from .validate_call_decorator import validate_call
-    from .warnings import PydanticDeprecatedSince20, PydanticDeprecatedSince26, PydanticDeprecationWarning
+    from .warnings import (
+        PydanticDeprecatedSince20,
+        PydanticDeprecatedSince26,
+        PydanticDeprecationWarning,
+        PydanticExperimentalWarning,
+    )
 
     # this encourages pycharm to import `ValidationError` from here, not pydantic_core
     ValidationError = pydantic_core.ValidationError
@@ -203,6 +208,7 @@
     'PydanticDeprecatedSince20',
     'PydanticDeprecatedSince26',
     'PydanticDeprecationWarning',
+    'PydanticExperimentalWarning',
     # annotated handlers
     'GetCoreSchemaHandler',
     'GetJsonSchemaHandler',
@@ -354,6 +360,7 @@
     'PydanticDeprecatedSince20': (__spec__.parent, '.warnings'),
     'PydanticDeprecatedSince26': (__spec__.parent, '.warnings'),
     'PydanticDeprecationWarning': (__spec__.parent, '.warnings'),
+    'PydanticExperimentalWarning': (__spec__.parent, '.warnings'),
     # annotated handlers
     'GetCoreSchemaHandler': (__spec__.parent, '.annotated_handlers'),
     'GetJsonSchemaHandler': (__spec__.parent, '.annotated_handlers'),

diff --git a/pydantic/_internal/_generate_schema.py b/pydantic/_internal/_generate_schema.py
@@ -1805,7 +1805,7 @@ def _apply_annotations(
         pydantic_js_annotation_functions: list[GetJsonSchemaFunction] = []
 
         def inner_handler(obj: Any) -> CoreSchema:
-            from_property = self._generate_schema_from_property(obj, obj)
+            from_property = self._generate_schema_from_property(obj, source_type)
             if from_property is None:
                 schema = self._generate_schema_inner(obj)
             else:

diff --git a/pydantic/_internal/_internal_dataclass.py b/pydantic/_internal/_internal_dataclass.py
@@ -1,7 +1,4 @@
 import sys
-from typing import Any, Dict
-
-dataclass_kwargs: Dict[str, Any]
 
 # `slots` is available on Python >= 3.10
 if sys.version_info >= (3, 10):

diff --git a/pydantic/experimental/__init__.py b/pydantic/experimental/__init__.py
@@ -0,0 +1,10 @@
+"""The "experimental" module of pydantic contains potential new features that are subject to change."""
+
+import warnings
+
+from pydantic.warnings import PydanticExperimentalWarning
+
+warnings.warn(
+    'This module is experimental, its contents are subject to change and deprecation.',
+    category=PydanticExperimentalWarning,
+)