Skip to content

test: every assertion can fail, the forecast test runs on the real platform, and two Windows arm64 tests stop flaking - #90

Merged
gutyoh merged 3 commits into
mainfrom
fix/test-trust-cleanup
Nov 18, 2026
Merged

gutyoh merged 3 commits into
mainfrom
fix/test-trust-cleanup

Conversation

@gutyoh

@gutyoh gutyoh commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Problem

Three kinds of test that couldn't be trusted:

  1. 58 bare negations across 16 suites. A line like ! grep -q x file in the middle of a bats test never fails it. bash exempts a negated command from set -e, which is bats-core's own gotcha and #462, and ShellCheck's SC2314. Only a negation on a test's last line was ever checked.
  2. A platform-pinned test. statusline-forecast.bats set PLATFORM=linux, so on macOS get_file_age ran GNU stat -c, read the mod file's age as unknown, and "the mod's forecast wins over the average" failed on every Mac without GNU stat first on PATH. It looked time-dependent; it was platform-dependent.
  3. Two Windows arm64 flakes in full-search.bats.

Change

  • Negations: each of the 58 lines now ends in || false. Lines already written ! cmd || { …; false; } were left alone. All tests still pass, so none of them was hiding a bug.
    • A portability test refuses a bare negation in any suite.
  • Platform: the forecast test now uses detect_test_platform, as helpers.bash asks every statusline test to.
    • A portability test refuses a pinned PLATFORM= unless the line carries # pinned: <why>. The two deliberate pins, the Keychain branch and bar.sh under bash 3.2, now say why.
  • Windows arm64:
    • The slow-store bound is 25 s. Two gets take 18 s and a third would take 27 s or more, so the test still proves the third get is never made.
    • The "saved verbatim" test asserts the run's status first. Now a flake prints the script's own lines instead of only "No such file". It is diagnosed, not retried blind.

Tests

  • Full suite on macOS: 1519 of 1519, under bash 5.3 and under stock bash 3.2. It's the first fully green Mac run of the suite: the forecast test passes there now.

Each control shows that its fix catches what it says:

Control Result
A bare ! true added to a suite The new guard names its line
@test bare { ! true; true; } against { ! true || false; true; } ok against not ok: the old form passes whatever it checks
PLATFORM=linux restored in the forecast test on macOS The test fails again
  • just lint and just format-check are clean.

🤖 Generated with Claude Code

gutyoh and others added 3 commits November 18, 2026 22:11
…atform, and two Windows arm64 tests stop flaking

- 58 bare `! cmd` lines across 16 suites end in `|| false`. bash exempts
  a negated command from errexit, so each one passed whatever it checked
  unless it was its test's last line. All still pass. A portability test
  now refuses a bare negation in any suite.
- statusline-forecast pinned PLATFORM=linux, so on macOS get_file_age ran
  GNU `stat -c`, read the mod file's age as unknown, and "the mod's
  forecast wins" failed on every Mac without GNU stat first on PATH. It
  uses detect_test_platform, as helpers.bash asks; a test may pin the
  platform only with `# pinned: <why>` on the line, which a portability
  test checks.
- "a store slow on every get" timed two 9 s fnox gets against 19 s; on
  Windows arm64 process start-up ate the last second. 25 s still proves
  the third get is never made (it would take 27 s).
- "saved verbatim" failed once on Windows arm64 with no output to read:
  it now asserts the run's status first, so a flake prints its cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d-line negations, and scans the helpers

`! cmd || true` fails nothing either, so a negation must end in || false,
|| { or || { ...; false; }; one after && or ; outside a condition counts
too, and tests/helpers.bash is scanned with the suites. Controls: each of
`! true || true`, `x=1 && ! true` and `true; ! false` added to a suite is
caught.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e and readonly pins

Controls: `local PLATFORM=linux`, `export PLATFORM="macos"` and
`declare -g PLATFORM=windows`, each added to a suite, are caught.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gutyoh
gutyoh merged commit e464157 into main Nov 18, 2026
15 checks passed
@gutyoh
gutyoh deleted the fix/test-trust-cleanup branch November 18, 2026 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant