10000 PEP 751: A file format to list Python dependencies for installation reproducibility by brettcannon · Pull Request #3870 · python/peps · GitHub
[go: up one dir, main page]

Skip to content

PEP 751: A file format to list Python dependencies for installation reproducibility #3870

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 59 commits into from
Jul 24, 2024
Merged
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
5e9e274
Write up to the specification
brettcannon Mar 20, 2024
f4fc728
Tweak rationale
brettcannon Mar 21, 2024
68a6dcd
Write down the file format
brettcannon Mar 21, 2024
761922e
Add in the `[[tool]]` table array
brettcannon Mar 21, 2024
61a72f5
Add expectations for lockers
brettcannon Mar 21, 2024
86e44fa
Add expectations for installers for per-file locking
brettcannon Mar 21, 2024
a960f76
Add some clarifications
brettcannon Mar 21, 2024
08f59d3
Example workflow for package locking
brettcannon Mar 22, 2024
82393b2
Bit of a clarification
brettcannon Mar 22, 2024
8c6801d
Add `package.tool`
brettcannon Mar 23, 2024
6b188ba
Add `package.dependents`
brettcannon Mar 23, 2024
6cced99
Fix a typo
brettcannon Mar 23, 2024
0080fc2
Add `package.description`
brettcannon Mar 23, 2024
1214a78
Add `package.direct`, `package.directory`, and `package.multiple-entr…
brettcannon Mar 25, 2024
92de818
Rename `package.simple-repo-package-url` and add `package.files.simpl…
brettcannon Mar 26, 2024
619bd6e
Updates based on some review comments
brettcannon Mar 28, 2024
244957c
Use a fake PEP number to make CI happy
brettcannon Mar 28, 2024
6a92b8f
Add an explicit note that the directory to write a `pylock.toml` file…
brettcannon Mar 28, 2024
2eb7166
Make the hash algorithm its own setting
brettcannon Mar 29, 2024
3623ccf
Fill out the ACKS section
brettcannon Mar 29, 2024
22fc531
Outline rejected ideas
brettcannon Mar 29, 2024
3182912
Add a rejected idea
brettcannon Mar 30, 2024
0e1df0c
Record the rejected idea of new core metadata version for metadata co…
brettcannon May 24, 2024
48811ee
Answer the rejected idea of having the installer do resolution
brettcannon May 25, 2024
b1c2fdc
Fill in rejected ideas around file names
brettcannon May 25, 2024
135a13e
Add rejected ideas around the file format
brettcannon May 27, 2024
450a50f
Finish filling in the rejected ideas
brettcannon May 27, 2024
aa7840f
Minor tweaks
brettcannon May 31, 2024
f6b02ef
Finish proofreading
brettcannon Jun 7, 2024
b35057a
Clarify that installers should never guess which `[[file-lock]]` to i…
brettcannon Jul 11, 2024
c16848b
reST fix
brettcannon Jul 11, 2024
15b2c05
Fix reST links
brettcannon Jul 11, 2024
040fa85
Add a TOML link target
brettcannon Jul 11, 2024
902ab69
Clarify that the file format is designed to facilitate diff reading
brettcannon Jul 23, 2024
c2677c1
Merge branch 'main' of github.com:python/peps into lock-file
brettcannon Jul 24, 2024
e1ec016
Refer to PoCs
brettcannon Jul 24, 2024
08e6fdb
Tweak some wording
brettcannon Jul 24, 2024
a3a1485
Pick a PEP number
brettcannon Jul 24, 2024
462b7e0
Update CODEOWNERS
brettcannon Jul 24, 2024
53a0ba6
Merge branch 'main' into lock-file
brettcannon Jul 24, 2024
75d6615
Clean up CODEOWNERS
brettcannon Jul 24, 2024
beaee5c
Merge branch 'lock-file' of https://github.com/brettcannon/peps into …
brettcannon Jul 24, 2024
dec8120
Fix lint failures
brettcannon Jul 24, 2024
a4d1840
Fix another lint failure
brettcannon Jul 24, 2024
b0d4877
Nest sections more
brettcannon Jul 24, 2024
1b39e49
Accept/address some comments
brettcannon Jul 24, 2024
9031e5c
Apply suggestions from code review
brettcannon Jul 24, 2024
fd139c3
Apply suggestions from code review
brettcannon Jul 24, 2024
89ff321
Apply suggestions from code review
brettcannon Jul 24, 2024
a914c12
Apply suggestions from code review
brettcannon Jul 24, 2024
8c21909
Adam likes periods
brettcannon Jul 24, 2024
98b4e82
More details around hashing
brettcannon Jul 24, 2024
e27b73d
Clarify `package.direct` defaults to `false`
brettcannon Jul 24, 2024
c0d46a1
Specify ``[[package.files]]` should be sorted by name
brettcannon Jul 24, 2024
0c45e87
Clarify that `[[package.build-requires]]` is locked
brettcannon Jul 24, 2024
ce18c5c
Clean up a sentence
brettcannon Jul 24, 2024
f5b574e
Fix a section title
brettcannon Jul 24, 2024
17ce7cd
Tweak abstract
brettcannon Jul 24, 2024
6df00c5
Merge branch 'main' into lock-file
brettcannon Jul 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updates based on some review comments
- Drop core metadata 2.4
- Suggest SHA-256 or better
- Suggest lockers set the simple Repo package URL
- Rearrange the order of some keys
  • Loading branch information
brettcannon committed Mar 28, 2024
commit 619bd6e0cc1f2f92f5494d9e891c041f3fa357e4
72 changes: 28 additions & 44 deletions peps/pep-9999.rst
Original file line number Diff line number Diff line change
Expand Up @@ -124,36 +124,13 @@ open source projects that want to lock what people should use to build the
documentation as open source projects do not necessarily know upfront what
environments their contributors are working from.

One issue with this approach has with current packaging standards is that to be
wholly accurate with what dependencies could be installed, every distribution
file would need to be downloaded and analyzed. This is is not always practical
if a `project has a lot of distribution files <https://pypi.org/project/charset-normalizer/#files>`__,
especially when varying metadata between distribution files is considered rare.
As such, this PEP includes a proposal of a new `core metadata`_ field to specify
if metadata varies in any way between distribution files for a package version.

As already mentioned, this approach is supported by PDM_. Poetry_ has
`shown some interest <https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists/46593/83>`__.


Specification
=============

Core Metadata 2.4
-----------------

A new core metadata version of 2.4 is to be introduced. When metadata uses this
version the contained metadata MUST be consistent across files for the same
package and version unless it is marked as ``Dynamic``. This effectively means
**all** wheel files will have the same metadata and source distributions _may_
also have the same metadata **if** all the fields static (i.e. `Dynamic` is not
set).

This should allow for the reading of the metadata from any distribution file to
know whether examining the other distribution files have the same or differing
metadata.


File Name
---------

Expand Down Expand Up @@ -312,6 +289,8 @@ other is disallowed.
- String
- Storess the `project detail`_ URL from the `Simple Repository API`_
- Useful for generating Packaging URLs (aka *PURLs*)
- When possible, lockers SHOULD include this key to assist with generating
`software bill of materials`_ (aka SBOMs)


``package.marker``
Expand Down Expand Up @@ -389,18 +368,16 @@ other is disallowed.
- Necessary for installers to decide what to install when using package locking


``package.files.hash``
``package.files.lock``
''''''''''''''''''''''

- String
- The hash of the file contents
- The format is ``f"{hashname}={hashvalue}"`` which is the same as the used by
the `Simple Repository API`_ and its HTML form
- Only a single hash value is used to allow the table to be written inline
- Using a single string to store both the hash algorithm and value instead of
separate keys for the two values is to make the inline table shorter
- Used by installers to verify the file contents match what the locker worked
with
- Required when ``[[file-lock]]`` is used
- Array of strings
- An array of ``file-lock.name`` values which signify that the file is to be
installed when the corresponding ``[[file-lock]]`` table applies to the
environment
- There MUST only be a single file with any one ``file-lock.name`` entry per
package, regardless of version


``package.files.simple-repo-package-url``
Expand All @@ -413,6 +390,8 @@ other is disallowed.
this file **only**
- This is to support :pep:`708` when some files override what's provided by
another `Simple Repository API`_ index
- Lockers SHOULD include this key when appropriate to assist with generating
`software bill of materials`_ (aka SBOMs)


``package.files.origin``
Expand All @@ -425,16 +404,21 @@ other is disallowed.
for the file if not already downloaded/available


``package.files.lock``
``package.files.hash``
''''''''''''''''''''''

- Required when ``[[file-lock]]`` is used
- Array of strings
- An array of ``file-lock.name`` values which signify that the file is to be
installed when the corresponding ``[[file-lock]]`` table applies to the
environment
- There MUST only be a single file with any one ``file-lock.name`` entry per
package, regardless of version
- String
- The hash of the file contents
- The format is ``f"{hashname}={hashvalue}"`` which is the same as the used by
the `Simple Repository API`_ and its HTML form
- Only a single hash value is used to allow the table to be written inline
for readability and compactness purposes
- Using a single string to store both the hash algorithm and value instead of
separate keys for the two values is to make the inline table shorter
- Used by installers to verify the file contents match what the locker worked
with
- Lockers SHOULD use a hash algorithm that is as least as strong as
`SHA-256 <https://en.wikipedia.org/wiki/SHA-2>`__


``[package.vcs]``
Expand Down Expand Up @@ -527,9 +511,8 @@ other is disallowed.
Expectations for Lockers
------------------------

- When creating a lock file for ``[package-lock]`` and a package and version are
not using core metadata 2.4 as proposed by this PEP, the locker SHOULD read
the metadata of all files listed in ``[[package.files]]`` to make sure all
- When creating a lock file for ``[package-lock]``, the locker SHOULD read
the metadata of **all** files listed in ``[[package.files]]`` to make sure all
potential metadata cases are covered
- If a locker chooses not to check every file for its metadata, the tool MUST
either provide the user with the option to have all files checked (whether
Expand Down Expand Up @@ -670,5 +653,6 @@ _Poetry: https://python-poetry.org/
_project detail: https://packaging.python.org/en/latest/specifications/simple-repository-api/#project-detail
_pyproject.toml specification: https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-specification
_Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/
_ software bill of materials: https://www.cisa.gov/sbom
_version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/
_wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/
0