Repository navigation
Conversation
The corpus report quoted bare percentages. Print a fixed footer after the aggregate line, before the first assertion so it also appears when a gate fails, and say in docs what the false-positive rate covers.
The sentence splitter, the repetition scanner and the oral-density count each kept a private definition of a Han character, so text with Bopomofo, Extension A or 𠮷 was handled differently by each pass. All three now call is_cjk_ideograph, which also gains Extension J and the Compatibility Supplement. The sentence splitter keeps treating CJK opening marks such as the corner bracket as a sentence start after a dot, but no longer counts the ideographic comma or full stop as CJK text before an abbreviation.
The translation guide told agents to keep lists under 40% of a document and dashes under two per paragraph. A count to hit rewards editing for its own sake, and a lone em dash is not evidence anyway. Reword both as reasons not to edit. Add tests that scan the setup texts, the MCP prompts and the tool schema descriptions for rewrite-goal phrasing, skipping the caller-supplied ai_threshold and max_errors lines. Say in docs/mcp.md that axis scores rank documents and per-sentence findings justify edits.
Seven native corpus cases use the native_antiovercorrection_ prefix: suoyou in technical prose, a list item, an opinion command, mid-sentence shuoshihua, a parenthetical self-correction, a tutorial rhetorical question and a counter-mainstream claim with evidence. Two translationese guards go in translationese_good.txt, because the corpus harness enables only the ai_* detectors and cannot fail on a translationese detector.
Lint and convert only the string values of a JSON file, never its keys, so structure and identifiers survive a pass over config and data files. The lint, hook and MCP paths skip whole-text Simplified conversion for JSON, and a fix holding a quote, a backslash or a control character is dropped because it would break the string. Bound the placeholder scan to its head limit.
Record every row of the humanizer-tw residual table with a probe sentence and verdict in a TSV, and reparse it in a new integration test so the audit cannot rot. Add a typo rule for 併 used as 並, one character wide with merge-sense exceptions and low editorial confidence so it is never auto-fixed. Gate 社區 on tech-community clues, since the residential sense is valid zh-TW. Leave 構建 ungated: it has no distinct zh-TW reading. Keep the 的-chain threshold at four per clause. Three is ordinary prose, and the corpus harness never runs translationese, so lowering it had no false-positive evidence. A test pins both sides.
The harness switched on only the four ai flags, so a native case could not fail on a translationese or rhythm detector. Cases now take optional detect_translationese and detect_rhythm booleans, and an optional detector tag that the native report counts as a traps line. Existing trap cases are tagged. Eight native cases run with both detectors on: three translationese guards and five with three de in one clause.
Add cutoff, helper opener, bio filler, fake-candor, 廢話不多說 and fake-depth frame openers, skip positional tells inside an open quotation, and flag a paragraph-final 由此可見. Each is gated so native zh-TW prose in the corpus stays quiet.
A 是 between the first and last 的 splits the span into a subject and a predicate (X的N是Y的), so it is no longer one stacked modifier. Treat it as a predicate marker, with 是非 and 國是 as words that do not open one. Add the two sentences from item 52 to the native corpus.
Three or more consecutive bare sentences of two to six CJK characters read as staged drama. The check sits on the opt-in rhythm axis as an advisory Translationese finding with no suggestion, so it stays out of the calibrated scores and every fix tier, and --rhythm cannot move the AI score. A new RhythmStaccato phase family marks it advisory, and the lint and fixer tests expect one more finding. Colons, commas, quotes, digits and question marks disqualify a sentence, which keeps numbered steps, list items and dialogue silent; a native corpus case with steps and short labelled lines pins that.
Add four rhetorical reveal tells, tighten the positional ones, and spare ordinary speech for the chatbot openers. Drop a disabled watchlist rule that never earned a second signal.
An ignored test scores convert against Microsoft's published VS Code zh-Hant strings at a pinned upstream commit, with OpenCC s2twp as the baseline. It reports exact line match and changed-span precision, recall and F1, and writes disagreements to a TSV without failing on them. A fetch script verifies per-file sha256 pins before use, and a pinned run fails on a download error instead of writing a partial pairs file. The span metric carries inserted text, so converting a to c does not score as correct when the reference says b. Identical strings skip the alignment, and strings with a carriage return are skipped. The corpus report now prints a blake3 digest of each corpus's sorted case ids and inputs, so a quoted figure can be tied to its inputs.
A rule can now carry a register field, formal or casual, and fires only when the document resolves to that register. The register is the same one the register flag and the MCP parameter select, so an explicit override applies to gated rules too. The gate runs before overlap resolution, the match lives in RegisterGate::admits, and the register is resolved once per scan and shared with the structural passes. Use it for the officialese forms a 公文 mandates: 茲將, 茲因, 特此通知, 特此說明, 特此公告 and 鑑於 are Info ai_filler rules restricted to casual documents. A native 公文 case stays silent, and the check-ruleset lint validates the new field.
Gate the staccato check on verse, drop 講白了,, and tune the self Q&A and stock conclusion gates. Add native bio and news cases to the corpus and fix what they hit.
The corpus harness counted issues or whole documents, so a fixer that rewrote one correct term and one that mangled a document scored alike. Add a keep rate: of the editable spans the scanner reports that the gold output leaves alone, the share the fixer also leaves alone. Gold edits come from a character diff of input against expected_fixed, shared with the external corpus test through tests/common. Report it per corpus beside the other metrics, weighted by repeat and counted once per span. Gate the once-per-span rate strictly at 100.0 on both dirty corpora (11 and 29 spans): one miss is a 9 or 3 point drop, so a tolerance would hide a real overcorrection, and it bounds the weighted rate printed beside it. The native corpus draws no span, so its rate is n/a and ungated, and a dirty corpus that measures no span fails instead of passing on nothing.
Authored native cases probed two limits. Short academic closers of a results paragraph fired the 由此可見 detector, since the 20 character cap is only a length proxy, and 茲將 fired in news sentences that report a 公告. The closer now stays silent when the paragraph above it names reported-study vocabulary: at least one subject, instrument or statistic cue and three distinct cues in all, because one word such as 樣本 or 顯著 also appears in a product blurb. 茲將 gains the exceptions 公告,茲將 and 公告內容,茲將, a position test that spares a quoted notice while 茲將 in the author's own voice still fires. Residual: a bare verdict that borrows three study words is spared, and so is a 公告 comma frame in an AI sentence.
The spelling automaton kept the longest match before the register gate ran, so a gated match the gate then dropped also hid any shorter ungated rule inside it. Build a per-register automaton that leaves out the rules the register rejects, and their absorbers, lazily on first use. The scan runs the full automaton and stops at the first gated hit the document's register rejects, then discards its issues and makes one pass over that subset. If neither subset matcher builds, it makes one pass over the full automaton and skips rejected hits, the old behavior. Documents that meet no rejected gated pattern pay one field check per gated hit. The post-scan gate pass is removed. The full automaton and the subsets share one PatternSet shape with a single hit iterator, and the gate lives on the compiled rule. The count-only bench helper now applies gates too, as a side effect of sharing the scan. It is a benchmark entry point with no production caller, so this is intended. Add behavioral tests on a synthetic ruleset: a gated rule the register drops no longer hides a shorter ungated rule inside it. They cover the charwise and bytewise subsets, a crossing rule, the opposite gate, and the unbuildable-subset fallback.
Blank excluded text once, lazily and by character, and hand the blanked view to the cue searches so quoted and code spans no longer trigger study and hedging cues.
A review that read the source found the hedging density pass measured paragraph length on the blanked view, where each excluded byte becomes a space, so a long code span inflated the length and the threshold and spared a promotion that should happen. Read the length from the raw text and the hedges from the view, and take the emitter like the neighbouring passes so the view is built only when a paragraph is long enough to need it. Pin it with a test that fails when the length is read from the view.
The shared character diff kept insertions in a set keyed by position and character, so inserting two identical characters at one place scored the same as inserting one, and an output that differed from the reference could score as a perfect match in the external benchmark. Number the insertions made before a position, so each is its own edit, and pin it with a test that fails when the number is dropped.
Plain prose in doc comments, one reflowed sentence, and a simpler study closer gate with its evidence tests.
The ruleset lint rejects a new rule whose source is not a fixture path or a corpus case id. The comma form of the biographical filler rule now cites the same fixture as its sibling, so the lint passes against the base branch as CI runs it.
The sibling's fixture never contains the comma form, so it only satisfied the provenance lint. The disabled comma-form rule now cites the native corpus case where the phrase appears, which is the evidence for it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the linter's AI-style coverage and makes it quieter on native zh-TW prose, and fixes how JSON files are handled. New detectors flag more AI-style openers, rhetorical reveal tells and bursts of short sentences, each gated against the native corpus; spelling rules now carry a register gate so officialese is spared in a formal document; and cue searches read a blanked view so quoted and code spans no longer trigger them. JSON files are linted and converted by string value only, never by key, and the lint, hook and MCP paths skip whole-text Simplified conversion for JSON, with fixes that would break a string (quote, backslash, control character) dropped. The branch also adds a pinned external corpus benchmark and an OpenCC residual audit.
Left out: the CLI
convertcommand still converts JSON string values while lint, hook and MCP skip JSON conversion entirely; unifying those into one policy is a separate decision.Summary by cubic
Expands AI-style and translationese detection while protecting native zh-TW prose, and makes JSON handling preserve keys and structure. JSON linting scans string values only; conversion behavior depends on the command.
convertconverts string values while preserving keys, escapes, and placeholders.Known residual: CLI
convertconverts JSON string values, while lint, hooks, and MCP skip JSON conversion.Written for commit 783d391. Summary will update on new commits.