Repository navigation
fix(encoding): decode git's file names as Python does, and write LF (Windows); km 1.0.2 - #6
Merged
Merged
Conversation
pasrom
force-pushed
the
fix/git-output-utf8
branch
2 times, most recently
from
September 24, 2026 07:17
7b8930e to
40f6b8a
Compare
pasrom
force-pushed
the
fix/git-output-utf8
branch
2 times, most recently
from
September 24, 2026 08:30
4d57ed8 to
ae6cc03
Compare
km decoded git's output with the system's preferred encoding. On Windows that is a code page (cp1252), while git prints file names as UTF-8, so a tracked file with a non-ASCII name came back mangled and named no file: - km validate over the whole repo stopped with FileNotFoundError before checking anything; - km gen-index, demote and serve, which skip names that are not files, silently left the doc out: gen-index said "up to date" while the doc was missing from its list. Every git call now goes through one helper, paths.git(), that decodes the output with the file-system encoding, the way Python turns file names into text: UTF-8 on Windows and macOS, the locale's encoding on Linux, so each listed name opens its file on all three. Bytes that do not decode are kept (surrogateescape) instead of crashing: git writes its messages in the locale, which crashed km outside a repo. validate lists files through common.tracked_md() instead of three copies of the git call, and git lists them once per run; that also stops a tracked doc deleted but not yet staged from crashing validate. promote reads its child validator as UTF-8, and km's console output escapes a character the console encoding lacks instead of crashing. The same platform wrote CRLF: every file km writes now goes through paths.write_text(), UTF-8 with LF, so a generated _index.md no longer shows as changed on Windows. Tests run km with EncodingWarning as an error, so any call that relies on the default encoding fails them. tests/encoding_smoke.sh builds a brain with an u-umlaut and an o with double acute in its file names and checks that gen-index lists them and writes LF, that validate reports them, and that the work tree stays as committed. A new Windows CI job runs it, and the release waits for that job; team_smoke runs it under every macOS locale that behaves like Windows. On Linux a non-UTF-8 locale changes the file-name encoding too, which Windows never does: the smoke job generates a Latin-1 locale, and a test checks that a non-ASCII name is still validated there. Decoding as UTF-8 regardless of platform would have failed it. A further test checks a git error in a Latin-1 locale.
pasrom
force-pushed
the
fix/git-output-utf8
branch
from
September 24, 2026 08:37
ae6cc03 to
92a75a9
Compare
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (reported from a Windows machine)
km validateover the whole repo crashed on Windows as soon as a tracked.mdhad a non-ASCII character in its name. km decoded git's output with the system code page (cp1252) while git writes file names as UTF-8, sokünstler.mdcame back as a name that does not exist:FileNotFoundErrorbefore anything was checked.gen-index,demoteandservedid not crash but silently skipped the file:gen-indexreported "up to date" while the doc was missing from its list. Linux and macOS were fine, so CI never saw it.Fix
paths.git(), decodes git's output with the file-system encoding, the way Python turns file names into text: UTF-8 on Windows and macOS, the locale's encoding on Linux. Each listed name opens its file on all three. Undecodable bytes are kept (surrogateescape) instead of crashing; git writes its messages in the locale.validatelists files throughcommon.tracked_md(), with git called once per run; a tracked doc deleted but not yet staged no longer crashes it.promotereads its child validator as UTF-8; console output escapes characters the console lacks.paths.write_text(): UTF-8 with LF. On Windows km wrote CRLF, so generated_index.mdfiles showed as changed.Guards
EncodingWarningas an error.tests/encoding_smoke.sh: a brain with an ü and an ő in its file names; gen-index lists them and writes LF, validate reports them, the work tree stays as committed. A new Windows CI job runs it (the release waits for it); team_smoke runs it under the macOS locales that behave like Windows.Test plan
Version bump to 1.0.2: merging is the release.