Skip to content

fix(encoding): decode git's file names as Python does, and write LF (Windows); km 1.0.2 - #6

Merged
pasrom merged 2 commits into
mainfrom
fix/git-output-utf8
Sep 24, 2026
Merged

pasrom merged 2 commits into
mainfrom
fix/git-output-utf8

Conversation

@pasrom

@pasrom pasrom commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

Problem (reported from a Windows machine)

km validate over the whole repo crashed on Windows as soon as a tracked .md had a non-ASCII character in its name. km decoded git's output with the system code page (cp1252) while git writes file names as UTF-8, so künstler.md came back as a name that does not exist: FileNotFoundError before anything was checked. gen-index, demote and serve did not crash but silently skipped the file: gen-index reported "up to date" while the doc was missing from its list. Linux and macOS were fine, so CI never saw it.

Fix

  • One helper, paths.git(), decodes git's output with the file-system encoding, the way Python turns file names into text: UTF-8 on Windows and macOS, the locale's encoding on Linux. Each listed name opens its file on all three. Undecodable bytes are kept (surrogateescape) instead of crashing; git writes its messages in the locale.
  • validate lists files through common.tracked_md(), with git called once per run; a tracked doc deleted but not yet staged no longer crashes it.
  • promote reads its child validator as UTF-8; console output escapes characters the console lacks.
  • Every file km writes goes through paths.write_text(): UTF-8 with LF. On Windows km wrote CRLF, so generated _index.md files showed as changed.

Guards

  • Tests run km with EncodingWarning as an error.
  • tests/encoding_smoke.sh: a brain with an ü and an ő in its file names; gen-index lists them and writes LF, validate reports them, the work tree stays as committed. A new Windows CI job runs it (the release waits for it); team_smoke runs it under the macOS locales that behave like Windows.
  • On Linux a non-UTF-8 locale changes the file-name encoding too. The smoke job generates a Latin-1 locale and a test checks a non-ASCII name is still validated there.
  • A git error in a Latin-1 locale is reported, not a crash.

Test plan

  • Reproduced first: crash with a Latin-1 locale, clean with UTF-8
  • Suites 30/30, 30/30, 92/92 + encoding 4/4 locally; CI green on Linux and Windows
  • The Windows job bites: on the old code it fails through the EncodingWarning guard, and without the guard because gen-index leaves the non-ASCII file out (throwaway PRs throwaway: windows job without the fix #7, throwaway: windows job on the old code, no guard #8, closed)
  • Mutations: decoding as cp1252 fails encoding_smoke; removing console escaping fails it; removing the existence filter fails the deleted-doc tests
  • Linux Latin-1 (Docker): decoding as UTF-8 regardless of platform, the state before review, fails the new test; the fs-encoding decoding and the old code pass
  • /simplify and /code-review (high) on the final diff, findings folded in; write_jsonl building the bundle in memory kept (a brain's served bundle is small)

Version bump to 1.0.2: merging is the release.

@pasrom
pasrom force-pushed the fix/git-output-utf8 branch 2 times, most recently from 7b8930e to 40f6b8a Compare September 24, 2026 07:17
@pasrom
pasrom force-pushed the fix/git-output-utf8 branch 2 times, most recently from 4d57ed8 to ae6cc03 Compare September 24, 2026 08:30
km decoded git's output with the system's preferred encoding. On
Windows that is a code page (cp1252), while git prints file names as
UTF-8, so a tracked file with a non-ASCII name came back mangled and
named no file:

- km validate over the whole repo stopped with FileNotFoundError before
  checking anything;
- km gen-index, demote and serve, which skip names that are not files,
  silently left the doc out: gen-index said "up to date" while the doc
  was missing from its list.

Every git call now goes through one helper, paths.git(), that decodes
the output with the file-system encoding, the way Python turns file
names into text: UTF-8 on Windows and macOS, the locale's encoding on
Linux, so each listed name opens its file on all three. Bytes that do
not decode are kept (surrogateescape) instead of crashing: git writes
its messages in the locale, which crashed km outside a repo. validate
lists files through common.tracked_md() instead of three copies of the
git call, and git lists them once per run; that also stops a tracked
doc deleted but not yet staged from crashing validate. promote reads
its child validator as UTF-8, and km's console output escapes a
character the console encoding lacks instead of crashing.

The same platform wrote CRLF: every file km writes now goes through
paths.write_text(), UTF-8 with LF, so a generated _index.md no longer
shows as changed on Windows.

Tests run km with EncodingWarning as an error, so any call that relies
on the default encoding fails them. tests/encoding_smoke.sh builds a
brain with an u-umlaut and an o with double acute in its file names and
checks that gen-index lists them and writes LF, that validate reports
them, and that the work tree stays as committed. A new Windows CI job
runs it, and the release waits for that job; team_smoke runs it under
every macOS locale that behaves like Windows. On Linux a non-UTF-8
locale changes the file-name encoding too, which Windows never does:
the smoke job generates a Latin-1 locale, and a test checks that a
non-ASCII name is still validated there. Decoding as UTF-8 regardless
of platform would have failed it. A further test checks a git error in
a Latin-1 locale.
@pasrom
pasrom force-pushed the fix/git-output-utf8 branch from ae6cc03 to 92a75a9 Compare September 24, 2026 08:37
@pasrom pasrom changed the title fix: read git's output as UTF-8 and write LF (Windows), km 1.0.2 fix(encoding): decode git's file names as Python does, and write LF (Windows); km 1.0.2 Sep 24, 2026
@pasrom
pasrom merged commit bb236d6 into main Sep 24, 2026
4 checks passed
@pasrom
pasrom deleted the fix/git-output-utf8 branch September 24, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant