Repository navigation
Read files as UTF-8 instead of using the locale encoding - #106
Merged
Merged
Conversation
`read()` and `_get_from_file_version()` both call `Path.read_text()` with no
`encoding`, so the file is decoded with the locale encoding. On any non-UTF-8
locale (e.g. `LC_ALL=C`, or a Chinese/Japanese Windows install, where
`locale.getpreferredencoding()` is `cp936`/`cp932`) a non-ASCII byte makes the
build fail:
UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 in position 5387
That byte was an em dash in a comment in a `pyproject.toml`. TOML is UTF-8 by
spec, and a `from-file` source with a `pattern` is meant to be a source file
whose non-version content can be non-ASCII too, so pass `encoding="utf-8"`
explicitly at both call sites.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
read()and_get_from_file_version()both callPath.read_text()without anencoding, so the file is decoded with the locale encoding (locale.getpreferredencoding(False)).On any non-UTF-8 locale —
LC_ALL=C, or a Chinese/Japanese Windows install where the preferred encoding iscp936/cp932— a single non-ASCII byte anywhere inpyproject.tomlbreaks the build. I hit this building a real project:Position 5387 was an em dash (
—, U+2014) inside a comment. The version string itself was pure ASCII; the failure came from unrelated prose elsewhere in the document. Worse, some byte sequences decode without raising and silently mojibake the document instead.This is a spec violation rather than a platform quirk: TOML is UTF-8 by definition ("A TOML file must be a valid UTF-8 encoded Unicode document"), so the locale should never get a vote. The same applies to a
from-filesource: whenpatternis set it is meant to point at a source file such as__init__.py, whose surrounding content can obviously be non-ASCII.Fix
Pass
encoding="utf-8"explicitly at both call sites.Tests
Two regression tests in
tests/test_main.py, plus fixtures with non-ASCII content (tests/fixtures/with-non-ascii/,tests/fixtures/with-from-file-non-ascii/).They run a child interpreter rather than monkeypatching
locale: the encodingopen()defaults to is resolved at interpreter start-up, so patchinglocale.getpreferredencodingin-process has no effect. The child getsLC_ALL=CwithPYTHONUTF8=0andPYTHONCOERCECLOCALE=0, which disables both routes back to UTF-8 (PEP 540 UTF-8 mode and PEP 538 C locale coercion), andPYTHONIOENCODING=utf-8so it can hand the decoded text back over stdout intact. That makes the tests meaningful on the Ubuntu CI runners, not just on a Windows CJK machine.Verification
mainand pass with the fix. Without it, on this machine:58 passed.ruff check src testsandruff format --check src testsclean.