Skip to content

Read files as UTF-8 instead of using the locale encoding - #106

Merged
ninoseki merged 2 commits into
ninoseki:mainfrom
LHMQ878:read-files-as-utf8
Aug 18, 2026
Merged

ninoseki merged 2 commits into
ninoseki:mainfrom
LHMQ878:read-files-as-utf8

Conversation

@LHMQ878

@LHMQ878 LHMQ878 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Problem

read() and _get_from_file_version() both call Path.read_text() without an encoding, so the file is decoded with the locale encoding (locale.getpreferredencoding(False)).

On any non-UTF-8 locale — LC_ALL=C, or a Chinese/Japanese Windows install where the preferred encoding is cp936/cp932 — a single non-ASCII byte anywhere in pyproject.toml breaks the build. I hit this building a real project:

UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 in position 5387
  File ".../uv_dynamic_versioning/main.py", line 17, in read
    return pyproject.read_text()

Position 5387 was an em dash (—, U+2014) inside a comment. The version string itself was pure ASCII; the failure came from unrelated prose elsewhere in the document. Worse, some byte sequences decode without raising and silently mojibake the document instead.

This is a spec violation rather than a platform quirk: TOML is UTF-8 by definition ("A TOML file must be a valid UTF-8 encoded Unicode document"), so the locale should never get a vote. The same applies to a from-file source: when pattern is set it is meant to point at a source file such as __init__.py, whose surrounding content can obviously be non-ASCII.

Fix

Pass encoding="utf-8" explicitly at both call sites.

Tests

Two regression tests in tests/test_main.py, plus fixtures with non-ASCII content (tests/fixtures/with-non-ascii/, tests/fixtures/with-from-file-non-ascii/).

They run a child interpreter rather than monkeypatching locale: the encoding open() defaults to is resolved at interpreter start-up, so patching locale.getpreferredencoding in-process has no effect. The child gets LC_ALL=C with PYTHONUTF8=0 and PYTHONCOERCECLOCALE=0, which disables both routes back to UTF-8 (PEP 540 UTF-8 mode and PEP 538 C locale coercion), and PYTHONIOENCODING=utf-8 so it can hand the decoded text back over stdout intact. That makes the tests meaningful on the Ubuntu CI runners, not just on a Windows CJK machine.

Verification

  • Both new tests fail on main and pass with the fix. Without it, on this machine:
    UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 in position 71
    UnicodeDecodeError: 'gbk' codec can't decode byte 0xb1 in position 23
    
  • Full suite: 58 passed.
  • ruff check src tests and ruff format --check src tests clean.

LHMQ878 and others added 2 commits August 18, 2026 07:39
`read()` and `_get_from_file_version()` both call `Path.read_text()` with no
`encoding`, so the file is decoded with the locale encoding. On any non-UTF-8
locale (e.g. `LC_ALL=C`, or a Chinese/Japanese Windows install, where
`locale.getpreferredencoding()` is `cp936`/`cp932`) a non-ASCII byte makes the
build fail:

    UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 in position 5387

That byte was an em dash in a comment in a `pyproject.toml`. TOML is UTF-8 by
spec, and a `from-file` source with a `pattern` is meant to be a source file
whose non-version content can be non-ASCII too, so pass `encoding="utf-8"`
explicitly at both call sites.
@ninoseki
ninoseki merged commit 33b50e8 into ninoseki:main Aug 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants