Windows compat pass: utf-8 everywhere, hardened guards, windows-latest CI

.gitattributes

@@ -0,0 +1 @@
+* text=auto eol=lf

.github/workflows/test.yml

@@ -7,8 +7,16 @@ on:
 
 jobs:
   pytest:
-    name: "Tier 1: deterministic tests"
-    runs-on: ubuntu-latest
+    name: "Tier 1: deterministic tests (${{ matrix.os }})"
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        # windows-latest gives the Python layer (incl. the MCP server, which
+        # the protocol tests spawn as a real subprocess) empirical Windows
+        # coverage. Do NOT set PYTHONUTF8 here: Windows' cp1252 default is
+        # exactly what the unicode round-trip tests must run under.
+        os: [ubuntu-latest, windows-latest]
     steps:
       - uses: actions/checkout@v6
       - uses: actions/setup-python@v6
@@ -17,6 +25,7 @@ jobs:
       - run: pip install pytest pyyaml
       - run: pytest tests/ -q
       - name: Vendored-fidelity audit (every deviation traces to a PORTING rule)
+        if: matrix.os == 'ubuntu-latest'
         run: python3 scripts/upstream_audit.py
 
   plugin-validate:

.mcp.json

@@ -5,7 +5,8 @@
       "args": ["${CLAUDE_PLUGIN_ROOT}/scripts/mcp_server.py"],
       "env": {
         "CLAUDITY_PLUGIN_ROOT": "${CLAUDE_PLUGIN_ROOT}",
-        "PYTHONUNBUFFERED": "1"
+        "PYTHONUNBUFFERED": "1",
+        "PYTHONUTF8": "1"
       }
     }
   }

CLAUDE.md

@@ -19,6 +19,9 @@ the contract before editing anything:
 - **Zero dependencies**: everything runs on python3 stdlib. Nothing is pip-
   installed, ever — that includes the MCP server (hand-rolled stdio JSON-RPC,
   see the "Transport" note in PORTING.md).
+- **All file I/O in scripts/ passes `encoding="utf-8"`** — Windows' locale
+  default corrupts unicode markdown; the windows-latest CI job runs the
+  unicode round-trip tests under that default to enforce it.
 - **Versioning**: bump `.claude-plugin/plugin.json` and
   `.claude-plugin/marketplace.json` together (test-enforced). Releases via
   `claude plugin tag`.

PORTING.md

@@ -61,6 +61,14 @@ policy (each is also flagged in the server docstring):
   prefix). Upstream's tool and its own decision-guidance CLI disagree on id
   format; a session that used both double-recorded under `02` and
   `decision-02-<slug>`. The tool now writes the same key the CLI does.
+- **`encoding="utf-8"` on all file I/O** in the vendored protocol scripts
+  (upstream omits it in `mailbox.py`, `packet_status.py`, `initialize.py`):
+  Windows' locale default is cp1252, which corrupts unicode markdown.
+  Locked by a unicode round-trip test that runs on the windows-latest CI job.
+- **`Path.is_relative_to` traversal guards** in the server (upstream uses a
+  string-prefix comparison, which a sibling directory like
+  `.clarity-protocol-evil` passes on any OS, and which case-insensitive
+  filesystems weaken further).
 
 ### Retired rules
 

README.md

@@ -48,7 +48,13 @@ It is not affiliated with or endorsed by Microsoft or Anthropic. See
 - Claude Code with plugin support (a recent version)
 - `python3` 3.10+ on PATH (the bundled scripts are stdlib-only)
 - `git` (the protocol directory is designed to be committed)
-- Tested on macOS and Linux; Windows is untested
+- Tested on macOS and Linux. On Windows: the Python layer (including the
+  bundled MCP server) is CI-verified on `windows-latest`, but Claude Code
+  itself running the plugin natively on Windows has not been validated.
+  Two known requirements: Claude Code's Bash tool needs Git for Windows,
+  and the plugin invokes `python3` — the python.org installer registers
+  `python`/`py` but not `python3` (the Microsoft Store build does; or
+  create an alias/shim so `python3` resolves)
 
 ## Install
 

TESTING.md

@@ -73,6 +73,15 @@ suite fails if a sequential run's total exceeds the budget.
 | 09-message | message-clarification writes a non-template general-audience `summary.md` and records it |
 | 10-record | the headless MCP canary: an ambient risk report lands as exactly one `mailboxes/failure-brainstorm/` item via the `record_failure` tool |
 
+**Windows.** Tier 1 runs on both `ubuntu-latest` and `windows-latest` in
+CI — the Windows job is the empirical compatibility evidence for the Python
+layer (the MCP protocol tests spawn the real server, and the unicode
+round-trip tests run under Windows' cp1252 locale default — never set
+PYTHONUTF8 in CI, it would mask the bugs they guard against). The e2e harness
+(`tests/e2e/`) is POSIX-only by design (bash, rsync, shasum); its tests skip
+on win32. Claude Code natively running the plugin on Windows remains
+unvalidated — see the README prerequisites.
+
 **Plugin isolation.** Sessions never see the real repo: each suite run
 snapshots the plugin payload (working tree minus `.git`/`.venv`/caches) to
 `/tmp/claudity-e2e-plugin.*` and every `claude` invocation gets the snapshot

scripts/mailbox.py

@@ -18,7 +18,9 @@
 Copyright (c) Microsoft Corporation). Modifications per PORTING.md:
 the clarity_agent.app_paths imports (_PROTOCOL_DIR_DOT,
 _PROTOCOL_DIR_VISIBLE, find_protocol_dir) are inlined below so the file
-runs standalone. Note: this module shadows the stdlib ``mailbox`` module
+runs standalone; all file I/O passes encoding="utf-8" (Windows'
+locale default corrupts unicode markdown). Note: this module shadows
+the stdlib ``mailbox`` module
 for code importing with scripts/ on sys.path; nothing in this repo uses
 the stdlib module.
 """
@@ -220,7 +222,7 @@ def load_config(self) -> dict[str, Any]:
         """
         if not self.exists:
             raise MailboxNotFoundError(f"Mailbox '{self.name}' does not exist")
-        return json.loads(self.config_path.read_text())
+        return json.loads(self.config_path.read_text(encoding="utf-8"))
 
     def save_config(self, config: dict[str, Any]) -> None:
         """Write config to _config.json in both mailbox and archive.
@@ -230,9 +232,9 @@ def save_config(self, config: dict[str, Any]) -> None:
         if not self.exists:
             raise MailboxNotFoundError(f"Mailbox '{self.name}' does not exist")
         content: str = json.dumps(config, indent=2) + "\n"
-        self.config_path.write_text(content)
+        self.config_path.write_text(content, encoding="utf-8")
         if self.archive_dir.exists():
-            self.archive_config_path.write_text(content)
+            self.archive_config_path.write_text(content, encoding="utf-8")
 
     # --- Factory ---
 
@@ -255,8 +257,8 @@ def create(
         mailbox.archive_dir.mkdir(parents=True, exist_ok=True)
 
         content: str = json.dumps(config, indent=2) + "\n"
-        mailbox.config_path.write_text(content)
-        mailbox.archive_config_path.write_text(content)
+        mailbox.config_path.write_text(content, encoding="utf-8")
+        mailbox.archive_config_path.write_text(content, encoding="utf-8")
 
         return mailbox
 
@@ -324,7 +326,7 @@ def write(self, name: str, content: str) -> Path:
         while True:
             path = self.mailbox_dir / f"{now}-{counter:02d}-{name}.md"
             try:
-                with path.open("x") as f:
+                with path.open("x", encoding="utf-8") as f:
                     f.write(content)
                 return path
             except FileExistsError:
@@ -446,7 +448,7 @@ def create_lock(self, thinker_name: str, expires_at: datetime) -> Path:
             "expires_at": expires_at.isoformat(),
         }
         lock_path: Path = self.mailbox_dir / f"_{thinker_name}{self._LOCK_SUFFIX}"
-        lock_path.write_text(json.dumps(lock_data, indent=2) + "\n")
+        lock_path.write_text(json.dumps(lock_data, indent=2) + "\n", encoding="utf-8")
         return lock_path
 
     def remove_lock(self, thinker_name: str) -> None:
@@ -477,7 +479,7 @@ def active_locks(self) -> list[LockInfo]:
         for path in self.mailbox_dir.iterdir():
             if path.name.startswith("_") and path.name.endswith(self._LOCK_SUFFIX):
                 try:
-                    data: dict[str, Any] = json.loads(path.read_text())
+                    data: dict[str, Any] = json.loads(path.read_text(encoding="utf-8"))
                     created = datetime.fromisoformat(data["created_at"])
                     expires = datetime.fromisoformat(data["expires_at"])
                     if expires > now:
@@ -502,7 +504,7 @@ def clean_expired_locks(self) -> list[str]:
         for path in self.mailbox_dir.iterdir():
             if path.name.startswith("_") and path.name.endswith(self._LOCK_SUFFIX):
                 try:
-                    data = json.loads(path.read_text())
+                    data = json.loads(path.read_text(encoding="utf-8"))
                     expires = datetime.fromisoformat(data["expires_at"])
                     if expires <= now:
                         path.unlink()
@@ -539,7 +541,7 @@ def list_nonempty_mailboxes(protocol_dir: Path) -> list[MailboxInfo]:
         if not items:
             continue
 
-        config: dict[str, Any] = json.loads(config_path.read_text())
+        config: dict[str, Any] = json.loads(config_path.read_text(encoding="utf-8"))
         result.append({
             "name": entry.name,
             "config": config,

scripts/mcp_server.py

@@ -50,6 +50,10 @@
   (``decision-NN-<slug>``) instead of upstream's bare number prefix, so
   the tool and the decision-guidance CLI step write the same
   ``decisionState`` key instead of double-recording.
+- The path-traversal guards use ``Path.is_relative_to`` instead of
+  upstream's string-prefix comparison, which a sibling directory like
+  ``.clarity-protocol-evil`` passes on any OS and case-insensitive
+  filesystems weaken further.
 """
 
 from __future__ import annotations
@@ -338,7 +342,7 @@ def read_protocol_document(
     proto_dir = _resolve_protocol_dir(project_dir)
     file_path = (proto_dir / document_path).resolve()
 
-    if not str(file_path).startswith(str(proto_dir.resolve())):
+    if not file_path.is_relative_to(proto_dir.resolve()):
         return "Error: path traversal not allowed."
     if not file_path.exists():
         return f"Error: document not found: {document_path}"
@@ -359,7 +363,7 @@ def write_protocol_document(
     proto_dir = _resolve_protocol_dir(project_dir)
     file_path = (proto_dir / document_path).resolve()
 
-    if not str(file_path).startswith(str(proto_dir.resolve())):
+    if not file_path.is_relative_to(proto_dir.resolve()):
         return "Error: path traversal not allowed."
 
     file_path.parent.mkdir(parents=True, exist_ok=True)
@@ -576,7 +580,7 @@ def protocol_document_resource(path: str) -> str:
     """
     proto_dir = _resolve_protocol_dir()
     file_path = (proto_dir / path).resolve()
-    if not str(file_path).startswith(str(proto_dir.resolve())):
+    if not file_path.is_relative_to(proto_dir.resolve()):
         return "Error: path traversal not allowed."
     if not file_path.exists():
         return f"Document not found: {path}"

scripts/protocol_init.py

@@ -11,7 +11,8 @@
 the AGENTS.md snippet refresh is removed (Claudity installs its snippet
 into CLAUDE.md via the embed command); ensure_suggestion_box imports
 from the sibling mailbox module; the default config identifies Claudity
-instead of a clarity-agent install.
+instead of a clarity-agent install; all file I/O passes encoding="utf-8"
+(Windows' locale default corrupts unicode markdown).
 
 Template contents are verbatim from upstream — their placeholder text
 must keep matching TEMPLATE_MARKERS in protocol_status.py, which is how
@@ -56,7 +57,7 @@ def _claudity_version() -> str:
     """Read the plugin version (single-sourced from plugin.json)."""
     manifest = Path(__file__).resolve().parent.parent / ".claude-plugin" / "plugin.json"
     try:
-        return str(json.loads(manifest.read_text())["version"])
+        return str(json.loads(manifest.read_text(encoding="utf-8"))["version"])
     except (OSError, ValueError, KeyError):
         return "unknown"
 

scripts/protocol_status.py

@@ -20,7 +20,8 @@
 Copyright (c) Microsoft Corporation). Modifications per PORTING.md:
 the clarity_agent package imports (app_paths.protocol_dir,
 mailbox.list_nonempty_mailboxes, failure_state.check_failure_state) are
-inlined below so the file runs standalone.
+inlined below so the file runs standalone, and all file I/O passes
+encoding="utf-8" (Windows' locale default corrupts unicode markdown).
 """
 
 from __future__ import annotations
@@ -259,7 +260,7 @@ def _list_nonempty_mailboxes(protocol_dir: Path) -> list[dict[str, Any]]:
             continue
         result.append({
             "name": entry.name,
-            "config": json.loads(config_path.read_text()),
+            "config": json.loads(config_path.read_text(encoding="utf-8")),
             "item_count": len(items),
         })
     return result
@@ -288,7 +289,7 @@ class _FailureState:
 
 
 def _has_management_plan(file_path: Path) -> bool:
-    text = file_path.read_text()
+    text = file_path.read_text(encoding="utf-8")
     match = _MGMT_HEADER_RE.search(text)
     if not match:
         return False
@@ -343,7 +344,7 @@ def is_template(file_path: Path) -> bool:
     try:
         content: str = file_path.read_text(encoding="utf-8").strip()
     except UnicodeDecodeError:
-        content = file_path.read_text().strip()
+        content = file_path.read_text(encoding="utf-8").strip()
     if not content:
         return True
     for marker in TEMPLATE_MARKERS:
@@ -362,15 +363,15 @@ def load_config(protocol_dir: Path) -> ClarityConfig:
     config_path: Path = protocol_dir / "config.json"
     if not config_path.exists():
         return {}
-    with open(config_path) as f:
+    with open(config_path, encoding="utf-8") as f:
         result: ClarityConfig = json.load(f)
         return result
 
 
 def save_config(protocol_dir: Path, config: ClarityConfig) -> None:
     """Write config.json to the protocol directory, preserving formatting."""
     config_path: Path = protocol_dir / "config.json"
-    with open(config_path, "w") as f:
+    with open(config_path, "w", encoding="utf-8") as f:
         json.dump(config, f, indent=2)
         f.write("\n")
 
@@ -589,7 +590,7 @@ def check_resolved_in_open_questions(protocol_dir: Path) -> list[str]:
     warnings: list[str] = []
     current_question: str | None = None
 
-    for line in oq_path.read_text().splitlines():
+    for line in oq_path.read_text(encoding="utf-8").splitlines():
         heading_match = re.match(r"^##\s+(Q\d+:.+)", line)
         if heading_match:
             current_question = heading_match.group(1).strip()

scripts/upstream_audit.py

@@ -27,7 +27,7 @@
 from pathlib import Path
 
 REPO = Path(__file__).resolve().parent.parent
-CFG = json.loads((REPO / "upstream.json").read_text())
+CFG = json.loads((REPO / "upstream.json").read_text(encoding="utf-8"))
 PIN = CFG["pin"]
 CACHE = Path(f"/tmp/claudity-upstream-pin-{PIN[:12]}")
 RAW = f"https://raw.githubusercontent.com/{CFG['repo']}/{PIN}"
@@ -80,7 +80,7 @@
 def fetch(upstream_path: str, offline: bool) -> str | None:
     dst = CACHE / upstream_path
     if dst.exists():
-        return dst.read_text()
+        return dst.read_text(encoding="utf-8")
     if offline:
         return None
     dst.parent.mkdir(parents=True, exist_ok=True)
@@ -90,7 +90,7 @@ def fetch(upstream_path: str, offline: bool) -> str | None:
     except Exception as e:  # noqa: BLE001
         print(f"FETCH ERROR {upstream_path}: {e}", file=sys.stderr)
         return None
-    dst.write_text(text)
+    dst.write_text(text, encoding="utf-8")
     return text
 
 
@@ -148,32 +148,32 @@ def main() -> None:
 
         if entry.get("fidelity") == "inlined":
             short = PIN[:7]
-            ok = local_path.exists() and f"clarity-agent@{short}" in local_path.read_text()
+            ok = local_path.exists() and f"clarity-agent@{short}" in local_path.read_text(encoding="utf-8")
             print(f"  {'✓' if ok else '✗'}  {local}  (inlines {up}; {'pin header present' if ok else 'MISSING pin header'})")
             failures += 0 if ok else 1
             continue
 
         if entry.get("fidelity") == "adapted":
             n = len([l for l in difflib.unified_diff(
-                upstream_text.splitlines(), local_path.read_text().splitlines(), lineterm="")
+                upstream_text.splitlines(), local_path.read_text(encoding="utf-8").splitlines(), lineterm="")
                 if l.startswith(("+", "-")) and not l.startswith(("+++", "---"))])
             print(f"  i  {local}  (adapted — {n} changed lines; contract is its header note + UPSTREAM.md row)")
             continue
 
         if local in QUOTED:
-            ok = upstream_text.strip() in local_path.read_text()
+            ok = upstream_text.strip() in local_path.read_text(encoding="utf-8")
             print(f"  {'✓' if ok else '✗'}  {local}  (license quoted {'intact' if ok else 'MISSING'})")
             failures += 0 if ok else 1
             continue
 
         if local.endswith(INFORMATIONAL_SUFFIXES):
             n = len([l for l in difflib.unified_diff(
-                upstream_text.splitlines(), local_path.read_text().splitlines(), lineterm="")
+                upstream_text.splitlines(), local_path.read_text(encoding="utf-8").splitlines(), lineterm="")
                 if l.startswith(("+", "-")) and not l.startswith(("+++", "---"))])
             print(f"  i  {local}  ({n} changed lines; documented in its docstring)")
             continue
 
-        local_text = local_path.read_text()
+        local_text = local_path.read_text(encoding="utf-8")
         if local in VERBATIM:
             ok = local_text == upstream_text
             print(f"  {'✓' if ok else '✗'}  {local}  ({'byte-identical' if ok else 'DIVERGED — must be verbatim'})")

tests/test_e2e_isolation.py

@@ -12,10 +12,16 @@
 from __future__ import annotations
 
 import subprocess
+import sys
 from pathlib import Path
 
 import pytest
 
+pytestmark = pytest.mark.skipif(
+    sys.platform == "win32",
+    reason="the e2e harness is POSIX-only (bash, rsync, shasum); see TESTING.md",
+)
+
 REPO = Path(__file__).resolve().parent.parent
 LIB = REPO / "tests" / "e2e" / "lib.sh"