Permalink
Comparing changes
Choose two branches to see what’s changed or to start a new pull request.
If you need to, you can also or
learn more about diff comparisons.
Open a pull request
Create a new pull request by comparing changes across two branches. If you need to, you can also .
Learn more about diff comparisons here.
...
- 10 commits
- 17 files changed
- 2 contributors
Commits on Mar 21, 2025
-
CI: Publish each master commit with a unique version on TestPyPI
This follows commit 7d52df7 "CI: Ignore version duplicates when publishing to TestPyPI" with a better design: - Only publish builds from `master` branch on TestPyPI. - Version every non-tag commit with a `.devN` suffix, e.g. `1.36.2.dev1`. This prevents duplicates on TestPyPI. - `twine check` built packages. See discussion at #721 (comment) for more details.
Commits on Mar 23, 2025
-
tests: Use correct encoding for path
Before this change, build_temp_workspace() would always encode a path using UTF-8 and the strict error handler [1]. Most of the time, this is fine, but systems do not necessarily use UTF-8 and the strict error handler for paths [2]. [1]: <https://docs.python.org/3.12/library/stdtypes.html#str.encode> [2]: <https://docs.python.org/3.12/glossary.html#term-filesystem-encoding-and-error-handler>
-
Before this commit, test_run_default_format_output_in_tty() changed the value of sys.stdout, but it would never change it back to the original value. This commit makes sure that it gets changed back. At the moment, this commit doesn’t make a user-visible difference. A future commit will add a new test named test_ignored_from_file_with_multiple_encodings(). That new test requires that stdout gets restored, or else it will fail.
-
tests: Move code for deleting env vars to __init__
The motivation behind this change is to make it easier to create a future commit. That future commit will make yamllint change its behavior if an environment variable named YAMLLINT_FILE_ENCODING is found. That new environment variable will potentially cause interference with many different tests. Before this change, environment variables would only be deleted when the tests.test_cli module was used. At the moment, it’s OK to do that because that’s the only test module that will fail if certain environment variables are set. Once yamllint is updated to look for the YAMLLINT_FILE_ENCODING variable, pretty much every test will be likely to fail if YAMLLINT_FILE_ENCODING is set to a certain values. This change makes the code for deleting environment variables get run for all tests (not just tests.test_cli). As an alternative, we could have kept most of the code for deleting environment variables in tests/test_cli.py, and only included code for deleting YAMLLINT_FILE_ENCODING in tests/__init__.py. I decided to put all of the environment variable deletion code in tests/__init__.py in order to make things more consistent and easier to understand. I had also considered adding a function for deleting environment variables to tests/common.py and then adding this to every test module that needs to have environment variables deleted: from tests.common import remove_env_vars_that_might_interfere setUpModule = remove_env_vars_that_might_interfere() I decided to not do that because pretty much every single test module will fail if YAMLLINT_FILE_ENCODING is set to certain values, and there’s a lot of test modules.
-
decoder: Autodetect encoding of most YAML files
Before this change, yamllint would open YAML files using open()’s default encoding. As long as UTF-8 mode isn’t enabled, open() defaults to using the system’s locale encoding [1][2]. This can cause problems in multiple different scenarios. The first scenario involves linting UTF-8 YAML files on Linux systems. Most of the time, the locale encoding on Linux systems is set to UTF-8 [3][4], but it can be set to something else [5]. In the unlikely event that someone was using Linux with a locale encoding other than UTF-8, there was a chance that yamllint would crash with a UnicodeDecodeError. The second scenario involves linting UTF-8 YAML files on Windows systems. The locale encoding on Windows systems is the system’s ANSI code page [6]. The ANSI code page on Windows systems is NOT set to UTF-8 by default [7]. In the very likely event that someone was using Windows with a locale encoding other than UTF-8, there was a chance that yamllint would crash with a UnicodeDecodeError. Additionally, using open()’s default encoding is a violation of the YAML spec. Chapter 5.2 says: “On input, a YAML processor must support the UTF-8 and UTF-16 character encodings. For JSON compatibility, the UTF-32 encodings must also be supported. If a character stream begins with a byte order mark, the character encoding will be taken to be as indicated by the byte order mark. Otherwise, the stream must begin with an ASCII character. This allows the encoding to be deduced by the pattern of null (x00) characters.” [8] In most cases, this change fixes all of those problems by implementing the YAML spec’s character encoding detection algorithm. Now, as long as YAML files begin with either a byte order mark or an ASCII character, yamllint will (in most cases) automatically detect them as being UTF-8, UTF-16 or UTF-32. Other character encodings are not supported at the moment. Even with this change, there is still one specific situation where yamllint still uses the wrong character encoding. Specifically, this change does not affect the character encoding used for stdin. This means that at the moment, these two commands may use different character encodings when decoding file.yaml: $ yamllint file.yaml $ cat file.yaml | yamllint - A future commit will update yamllint so that it uses the same character encoding detection algorithm for stdin. It’s possible that this change will break things for existing yamllint users. This change allows users to use the YAMLLINT_FILE_ENCODING to override the autodetection algorithm just in case they’ve been using yamllint on weird nonstandard YAML files. Credit for the idea of having tests with pre-encoded strings and having an environment variable for overriding the character encoding autodetection algorithm goes to @adrienverge [9]. Fixes #218. Fixes #238. Fixes #347. [1]: <https://docs.python.org/3.12/library/functions.html#open> [2]: <https://docs.python.org/3.12/library/os.html#utf8-mode> [3]: <https://www.gnu.org/software/libc/manual/html_node/Extended-Char-Intro.html> [4]: <https://wiki.musl-libc.org/functional-differences-from-glibc.html#Character-sets-and-locale> [5]: <https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/SUPPORTED;h=c8b63cc2fe2b4547f2fb1bff6193da68d70bd563;hb=36f2487f13e3540be9ee0fb51876b1da72176d3f> [6]: <https://docs.python.org/3.12/glossary.html#term-locale-encoding> [7]: <https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page> [8]: <https://yaml.org/spec/1.2.2/#52-character-encodings> [9]: <#630 (comment)>
-
decoder: Autodetect decoding of stdin
Before this change, yamllint would use a character encoding autodetection algorithm in order to determine the character encoding of all YAML files that it processed, unless the YAML file was sent to yamllint via stdin. This change makes it so that yamllint always uses the character encoding detection algorithm, even if the YAML file is sent to yamllint via stdin. Before this change, one of yamllint’s tests would replace sys.stdin with a StringIO object. This change makes it so that that test replaces sys.stdin with a file object instead of a StringIO object. Before this change, it was OK to use a StringIO object because yamllint never tried to access sys.stdin.buffer. It’s no longer OK to use a StringIO because yamllint now tries to access sys.stdin.buffer. File objects do have a buffer attribute, so we can use a file object instead.
-
decoder: Autodetect encoding for ignore-from-file
Before this change, yamllint would decode files on the ignore-from-file list using open()’s default encoding [1][2]. This can cause decoding to fail in some situations (see the previous commit message for details). This change makes yamllint automatically detect the encoding for files on the ignore-from-file list. It uses the same algorithm that it uses for detecting the encoding of YAML files, so the same limitations apply: files must use UTF-8, UTF-16 or UTF-32 and they must begin with either a byte order mark or an ASCII character. [1]: <https://docs.python.org/3.12/library/fileinput.html#fileinput.input> [2]: <https://docs.python.org/3.12/library/fileinput.html#fileinput.FileInput>
-
tests: Stop using open()’s default encoding
In general, using open()’s default encoding is a mistake [1]. This change makes sure that every time open() is called, the encoding parameter is specified. Specifically, it makes it so that all tests succeed when run like this: python -X warn_default_encoding -W error::EncodingWarning -m unittest discover [1]: <https://peps.python.org/pep-0597/#using-the-default-encoding-is-a-common-mistake>
-
CI: Fail when open()’s default encoding is used
The previous few commits have removed all calls to open() that use its default encoding. That being said, it’s still possible that code added in the future will contain that same mistake. This commit makes it so that the CI test job will fail if that mistake is made again. Unfortunately, it doesn’t look like coverage.py allows you to specify -X options [1] or warning filters [2] when running your tests [3]. To work around this problem, I’m running all of the Python code, including coverage.py itself, with -X warn_default_encoding and -W error::EncodingWarning. As a result, the CI test job will also fail if coverage.py uses open()’s default encoding. Hopefully, coverage.py won’t do that. If it does, then we can always temporarily revert this commit. [1]: <https://docs.python.org/3.12/using/cmdline.html#cmdoption-X> [2]: <https://docs.python.org/3.12/using/cmdline.html#cmdoption-W> [3]: <https://coverage.readthedocs.io/en/7.4.0/cmd.html#execution-coverage-run>
-
Loading
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff v1.36.2...v1.37.0