How we verify RTK's output stays transparent, on every release

[aeppling]

Hey RTK community!

RTK's contract is simple: same behaviour, same exit code, only the noise removed. Filtering the output of hundreds of tools, each with countless flag combinations, means edge cases will show up, so we don't rely on chance to find them.

For several weeks now, every merge to develop has gone through a dedicated Transparency Lab before it reaches a release.

How it works

1. Differential fuzzing. Over 1,000 cases covering both built-in commands and TOML filters, with many flag combinations. Each case runs both raw and through RTK, with real tools on real projects.

2. Deterministic oracles: the hard gate. These check that the exit code matches, that output goes to the right stream, that no signal is lost, that the output is never larger than raw, and that nothing crashes or gets blocked. Results are reproducible, and a finding must show up twice before it counts.

3. Decision model as judge: advisory. Some questions can't be settled by a rule. For those, a model compares raw and filtered output and flags possible over-filtering. It guides improvements and never overrides a deterministic failure.

4. Reports to maintainers. The results go straight to the maintainers, so regressions are caught before a release.

The oracles are tested too: each one must catch planted defects and stay quiet on legitimate compression.

Your feedback is part of it

Many of the issues you reported have been fixed and are now permanent test cases, so they can't quietly come back. Thank you to everyone who took the time to report them.

What you should do

Update to the latest release to get all of these fixes.

Limits, honestly

We cover every command and flag RTK declares. Native flags passed through to the underlying tool are effectively unbounded, so we report what we covered rather than claim completeness.

Seen something odd?

Open an issue with the raw output and the RTK output. It becomes a test case. Questions about the approach are welcome in this thread.