8000 PEP 751: A file format to list Python dependencies for installation reproducibility by brettcannon · Pull Request #3870 · python/peps · GitHub
[go: up one dir, main page]

Skip to content

PEP 751: A file format to list Python dependencies for installation reproducibility #3870

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 59 commits into from
Jul 24, 2024
Merged
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
5e9e274
Write up to the specification
brettcannon Mar 20, 2024
f4fc728
Tweak rationale
brettcannon Mar 21, 2024
68a6dcd
Write down the file format
brettcannon Mar 21, 2024
761922e
Add in the `[[tool]]` table array
brettcannon Mar 21, 2024
61a72f5
Add expectations for lockers
brettcannon Mar 21, 2024
86e44fa
Add expectations for installers for per-file locking
brettcannon Mar 21, 2024
a960f76
Add some clarifications
brettcannon Mar 21, 2024
08f59d3
Example workflow for package locking
brettcannon Mar 22, 2024
82393b2
Bit of a clarification
brettcannon Mar 22, 2024
8c6801d
Add `package.tool`
brettcannon Mar 23, 2024
6b188ba
Add `package.dependents`
brettcannon Mar 23, 2024
6cced99
Fix a typo
brettcannon Mar 23, 2024
0080fc2
Add `package.description`
brettcannon Mar 23, 2024
1214a78
Add `package.direct`, `package.directory`, and `package.multiple-entr…
brettcannon Mar 25, 2024
92de818
Rename `package.simple-repo-package-url` and add `package.files.simpl…
brettcannon Mar 26, 2024
619bd6e
Updates based on some review comments
brettcannon Mar 28, 2024
244957c
Use a fake PEP number to make CI happy
brettcannon Mar 28, 2024
6a92b8f
Add an explicit note that the directory to write a `pylock.toml` file…
brettcannon Mar 28, 2024
2eb7166
Make the hash algorithm its own setting
brettcannon Mar 29, 2024
3623ccf
Fill out the ACKS section
brettcannon Mar 29, 2024
22fc531
Outline rejected ideas
brettcannon Mar 29, 2024
3182912
Add a rejected idea
brettcannon Mar 30, 2024
0e1df0c
Record the rejected idea of new core metadata version for metadata co…
brettcannon May 24, 2024
48811ee
Answer the rejected idea of having the installer do resolution
brettcannon May 25, 2024
b1c2fdc
Fill in rejected ideas around file names
brettcannon May 25, 2024
135a13e
Add rejected ideas around the file format
brettcannon May 27, 2024
450a50f
Finish filling in the rejected ideas
brettcannon May 27, 2024
aa7840f
Minor tweaks
brettcannon May 31, 2024
f6b02ef
Finish proofreading
brettcannon Jun 7, 2024
b35057a
Clarify that installers should never guess which `[[file-lock]]` to i…
brettcannon Jul 11, 2024
c16848b
reST fix
brettcannon Jul 11, 2024
15b2c05
Fix reST links
brettcannon Jul 11, 2024
040fa85
Add a TOML link target
brettcannon Jul 11, 2024
902ab69
Clarify that the file format is designed to facilitate diff reading
brettcannon Jul 23, 2024
c2677c1
Merge branch 'main' of github.com:python/peps into lock-file
brettcannon Jul 24, 2024
e1ec016
Refer to PoCs
brettcannon Jul 24, 2024
08e6fdb
Tweak some wording
brettcannon Jul 24, 2024
a3a1485
Pick a PEP number
brettcannon Jul 24, 2024
462b7e0
Update CODEOWNERS
brettcannon Jul 24, 2024
53a0ba6
Merge branch 'main' into lock-file
brettcannon Jul 24, 2024
75d6615
Clean up CODEOWNERS
brettcannon Jul 24, 2024
beaee5c
Merge branch 'lock-file' of https://github.com/brettcannon/peps into …
brettcannon Jul 24, 2024
dec8120
Fix lint failures
brettcannon Jul 24, 2024
a4d1840
Fix another lint failure
brettcannon Jul 24, 2024
b0d4877
Nest sections more
brettcannon Jul 24, 2024
1b39e49
Accept/address some comments
brettcannon Jul 24, 2024
9031e5c
Apply suggestions from code review
brettcannon Jul 24, 2024
fd139c3
Apply suggestions from code review
brettcannon Jul 24, 2024
89ff321
Apply suggestions from code review
brettcannon Jul 24, 2024
a914c12
Apply suggestions from code review
brettcannon Jul 24, 2024
8c21909
Adam likes periods
brettcannon Jul 24, 2024
98b4e82
More details around hashing
brettcannon Jul 24, 2024
e27b73d
Clarify `package.direct` defaults to `false`
brettcannon Jul 24, 2024
c0d46a1
Specify ``[[package.files]]` should be sorted by name
brettcannon Jul 24, 2024
0c45e87
Clarify that `[[package.build-requires]]` is locked
brettcannon Jul 24, 2024
ce18c5c
Clean up a sentence
brettcannon Jul 24, 2024
f5b574e
Fix a section title
brettcannon Jul 24, 2024
17ce7cd
Tweak abstract
brettcannon Jul 24, 2024
6df00c5
Merge branch 'main' into lock-file
brettcannon Jul 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Tweak rationale
  • Loading branch information
brettcannon committed Mar 21, 2024
commit f4fc7284c521f2a981d0e65ec5396d78914e090b
68 changes: 56 additions & 12 deletions peps/pep-9999.rst
Original file line number Diff line number Diff line change
Expand Up @@ -70,22 +70,66 @@ section either already implement this approach of evaluating dependencies in
isolation or have suggested they could in
`Poetry's case <https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists/46593/83>`__.

The lock file format is designed to support two locking scenarios. The first is
locking to specific files for an environment. This is to support workflows where
the environment(s) are known ahead of time and specifying what exactly to
install is important. This is similar to what ``pip freeze`` and pip-tools_

Locking Scenarios
-----------------

The lock file format is designed to support two locking scenarios.


Per-file Locking
~~~~~~~~~~~~~~~~

*Per-file locking* operates under the premise that one wants to install exact
same files in matching environments. As such, the lock file specifies the
requirements for an environment and then matches that environment requirements
with the files to install. There can be multiple environments specified in a
single file, each with their own set of files to install. By specifying the
exact files to install avoids needs performing a resolution to decide what to
install.

The motivation for this approach to locking is for those who have controlled
environments that they work with. For instance, if you have specific, controlled
development and production environments then you can use per-file locking to
make sure the **same** files are installed in either environment for everyone.

This is similar to what ``pip freeze`` and pip-tools_
support, but with more strictness of the exact files as well as incorporating
into the file format the ability to specify the locked files for multiple
environments.

The second locking scenario is to lock to package versions. This is the approach
taken by PDM_ (and indirectly by Poetry_). This is to support workflows where
the environment(s) are not known ahead of time. This can happen in open source
projects where contributors can have a myriad of different environments but
there's a desire to lock the packages used for e.g. building the project's
documentation. Since trying to lock the files for all possible environments
could be tedious at best and very expensive at worst, locking to the package
acts as a pragmatic compromise.

Package Locking
~~~~~~~~~~~~~~~

*Package locking* lists the packages and their versions that *may* apply to any
environment being installed for. The of packages and their version are evaluated
individually and independently from any other packages and versions listed in
the file. This allows installation to be linear -- read each package and version
and make an isolated decision as to whether it should be installed -- and thus
not require the installer to perform a resolution to decide what to install.

The motivation of this approach comes from
`PDM lock files <https://frostming.com/en/2024/pdm-lockfile/>`__. By listing the
potential packages and versions that may be installed, what packages may be
installed is controlled for in a way that's easy to reason about. This also
allows for not specifying the exact environments that would be supported by the
lock file so there's more flexibility and potentially easier
-- and thus faster -- lock generation. This approach supports scenarios like
open source projects that want to lock what people should use to build the
documentation as open source projects do not necessarily know upfront what
environments their contributors are working from.

One issue with this approach has with current packaging standards is that to be
wholly accurate with what dependencies could be installed, every distribution
file would need to be downloaded and analyzed. This is is not always practical
if a `project has a lot of distribution files <https://pypi.org/project/charset-normalizer/#files>`__,
especially when varying metadata between distribution files is considered rare.
As such, this PEP includes a proposal of a new `core metadata`_ field to specify
if metadata varies in any way between distribution files for a package version.

As already mentioned, this approach is supported by PDM_. Poetry_ has
`shown some interest <https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists/46593/83>`__.


Specification
Expand Down
0