Skip to content

Break the failed-run test's Bot read from the run on, not before it - #1729

Merged
davidmckayv merged 1 commit into
mainfrom
fix/failed-run-restore-flake
Nov 16, 2026
Merged

davidmckayv merged 1 commit into
mainfrom
fix/failed-run-restore-flake

Conversation

@davidmckayv

@davidmckayv davidmckayv commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

channel-failed-run-restore failed once in shard 1 on #1724's first attempt and passed when rerun. The failing case was "a message that is not sent because the run is refused and the Bot cannot be read still says the back end couldn't be reached": Unable to find an element by: [data-testid="transcript-stopped"] after 1,091 ms, with the turn still showing Thinking.

Cause

The test's timing, not a race in the product. The sentence always comes, about a second after the send.

Traced on a 4 vCPU x86 box, with the Bot read failing (502):

  1. Before a run, the screen reads the Bot's facts, and a fact never read before is waited for with the screen's own retry (factsFor in lib/copilot/bot-offer.ts, the app's retry: 1). The fixture's Bot read failed from the moment the page opened, so that read failed and was retried a second later.
  2. Only then was the run asked for. It was refused, the read after the refusal failed, and the sentence was drawn, all within a few milliseconds.

The send-to-sentence time was 1,001 to 1,012 ms at every load from idle down to a quarter of a CPU. The test's findByTestId waits 1,000 ms. Which of two timers of about a second fired first was down to the event loop. The two cases where the Bot read breaks made three reads where the others made two: the extra one is the retry.

Fix

What these tests are about is the read after the run went wrong. The fixture's Bot read now fails only once a run has been asked for. That is how it goes for a person already talking to the Bot, whose facts the screen already holds. The unrelated one-second retry is gone from the path, and the sentence comes within milliseconds of the send. The tests still wait on the sentence itself, and the product is unchanged.

Evidence

All runs are on a 4 vCPU x86 Ubuntu 24.04 box (the shape of ubuntu-latest). Each is a fresh bun test, with the CPU held down by stress-ng or systemd-run -p CPUQuota=.

What decides it: send to sentence, in the four "still says the back end" cases

Case (run / Bot read) Before, idle Before, quarter CPU After, idle After, quarter CPU
refused / breaks (the one that failed in CI) 1,001 to 1,006 ms up to 1,003 ms 1 ms 1 ms
breaks / breaks 1,001 to 1,003 ms up to 1,003 ms 1 ms up to 2 ms
refused / found 1 ms up to 3 ms 1 ms up to 2 ms
unreachable / found 2 ms up to 87 ms 2 ms up to 89 ms

Before the fix, both broken-read cases sat on the 1,000 ms wait with no margin. After it, every case is at least 10 times inside it. The test's findByTestId still waits its default 1,000 ms; nothing was given more time.

Pass counts

Condition Before (main) After (this PR)
Whole file: idle, stress-ng, one CPU, half a CPU (20 runs each) 80 of 80 80 of 80
The four cases: a quarter of a CPU (30), stress-ng (30) 60 of 60 60 of 60
Shard 1's 171 files in CI's order up to this one, in one process: idle (10), stress-ng (10) 20 of 20 20 of 20

The one-in-many failure CI saw did not come back in 160 runs here, before or after, so the counts alone don't separate the two. The timing does. Before, the case CI caught passed only because of how the event loop ordered two timers about a second long. After, nothing is waiting on that second.

On this head, CI ran the file and it passed, and biome check passes. The product is unchanged; only the test fixture moved.

A message that is not sent because the run is refused and the Bot cannot be read failed once in CI. The fixture's Bot read failed from the moment the page opened, so the read the screen makes before a run failed too, and the screen waits that out with its own retry a second later (factsFor in lib/copilot/bot-offer.ts). The sentence then came about 1 s after the send, against a findBy that waits 1 s, and which came first was down to the event loop. What these tests are about is the read after the run goes wrong, so the Bot read now fails only once a run has been asked for, as it does for a person already talking to the Bot. The sentence then comes within milliseconds.
@davidmckayv
davidmckayv force-pushed the fix/failed-run-restore-flake branch from 06b8a2f to f83fced Compare November 16, 2026 20:52
@davidmckayv
davidmckayv merged commit ea7f43a into main Nov 16, 2026
16 checks passed
@davidmckayv
davidmckayv deleted the fix/failed-run-restore-flake branch November 16, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant