Repository navigation
Break the failed-run test's Bot read from the run on, not before it - #1729
Merged
Merged
Conversation
davidmckayv
requested review from
MikeRyanDev,
dustywusty,
guidovizoso,
mxmzb,
samjulien and
tylerslaton
as code owners
November 16, 2026 19:56
A message that is not sent because the run is refused and the Bot cannot be read failed once in CI. The fixture's Bot read failed from the moment the page opened, so the read the screen makes before a run failed too, and the screen waits that out with its own retry a second later (factsFor in lib/copilot/bot-offer.ts). The sentence then came about 1 s after the send, against a findBy that waits 1 s, and which came first was down to the event loop. What these tests are about is the read after the run goes wrong, so the Bot read now fails only once a run has been asked for, as it does for a person already talking to the Bot. The sentence then comes within milliseconds.
davidmckayv
force-pushed
the
fix/failed-run-restore-flake
branch
from
November 16, 2026 20:52
06b8a2f to
f83fced
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
channel-failed-run-restorefailed once in shard 1 on #1724's first attempt and passed when rerun. The failing case was "a message that is not sent because the run is refused and the Bot cannot be read still says the back end couldn't be reached":Unable to find an element by: [data-testid="transcript-stopped"]after 1,091 ms, with the turn still showing Thinking.Cause
The test's timing, not a race in the product. The sentence always comes, about a second after the send.
Traced on a 4 vCPU x86 box, with the Bot read failing (502):
factsForinlib/copilot/bot-offer.ts, the app'sretry: 1). The fixture's Bot read failed from the moment the page opened, so that read failed and was retried a second later.The send-to-sentence time was 1,001 to 1,012 ms at every load from idle down to a quarter of a CPU. The test's
findByTestIdwaits 1,000 ms. Which of two timers of about a second fired first was down to the event loop. The two cases where the Bot read breaks made three reads where the others made two: the extra one is the retry.Fix
What these tests are about is the read after the run went wrong. The fixture's Bot read now fails only once a run has been asked for. That is how it goes for a person already talking to the Bot, whose facts the screen already holds. The unrelated one-second retry is gone from the path, and the sentence comes within milliseconds of the send. The tests still wait on the sentence itself, and the product is unchanged.
Evidence
All runs are on a 4 vCPU x86 Ubuntu 24.04 box (the shape of
ubuntu-latest). Each is a freshbun test, with the CPU held down bystress-ngorsystemd-run -p CPUQuota=.What decides it: send to sentence, in the four "still says the back end" cases
Before the fix, both broken-read cases sat on the 1,000 ms wait with no margin. After it, every case is at least 10 times inside it. The test's
findByTestIdstill waits its default 1,000 ms; nothing was given more time.Pass counts
stress-ng, one CPU, half a CPU (20 runs each)stress-ng(30)stress-ng(10)The one-in-many failure CI saw did not come back in 160 runs here, before or after, so the counts alone don't separate the two. The timing does. Before, the case CI caught passed only because of how the event loop ordered two timers about a second long. After, nothing is waiting on that second.
On this head, CI ran the file and it passed, and
biome checkpasses. The product is unchanged; only the test fixture moved.