Skip to content

Start five more browser tests' setup before the tests, not inside a timed hook - #1723

Merged
davidmckayv merged 2 commits into
mainfrom
fix/browser-test-setup-flake
Nov 16, 2026
Merged

davidmckayv merged 2 commits into
mainfrom
fix/browser-test-setup-flake

Conversation

@davidmckayv

@davidmckayv davidmckayv commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

The same fix as #1720, applied to the five other real-browser tests whose setup ran inside a beforeAll: a2ui-frame, update-banner-layout, composer-dictation-notice-layout, checkbox-contrast and conversation-intro-halo.

Cause

A beforeAll gets bun's default of 5 s. Each of these files bundles a page or builds a stylesheet in that hook, and four of them also start Chromium there. On a cold runner, when the file is the first one bun test opens, that can take longer than 5 s. The file then fails with a beforeEach/afterEach hook timed out before it measures anything. transcript-rail-clearance failed that way on main.

Fix

The setup now runs at the top of each file, before any test. It waits for the bundle (or the stylesheet) to be built and for Chromium to report it has started. Playwright's own launch timeout bounds that wait. No hook clock and no larger number replaces the old limit. checkbox-contrast only bundled in its hook, so only the bundle moves. Its browsers still start inside each test, within that test's own two minutes.

Evidence

Each run is a fresh bun test of one file alone, so it is the first file opened. The runs were on a 4 vCPU x86 Ubuntu 24.04 box (the shape of ubuntu-latest), with the page cache dropped before each run and the CPU held down with systemd-run -p CPUQuota=. Before is main at 7b4968a; after is this PR.

File One CPU, before One CPU, after A quarter of a CPU, before A quarter of a CPU, after
a2ui-frame 0 of 10 pass, 10 hook timeouts 10 of 10 0 of 5 pass, 5 hook timeouts 5 of 5
composer-dictation-notice-layout 5 of 10 pass, 5 hook timeouts 10 of 10 0 of 5 pass, 5 hook timeouts 5 of 5
update-banner-layout 10 of 10 10 of 10 5 of 5 5 of 5
checkbox-contrast 10 of 10 10 of 10 5 of 5 5 of 5
conversation-intro-halo 10 of 10 10 of 10 5 of 5 5 of 5

a2ui-frame and composer-dictation-notice-layout time out in setup the same way transcript-rail-clearance did. The other three have lighter setups and did not time out here. Their setup ran under the same 5 s limit, though, so they get the same pattern rather than waiting to cross it on a slower runner. Every failure before the fix was a hook timeout. After it, nothing failed.

In CI on this head, all five files ran and passed in the "app (real browser)" job. biome format, biome lint and the app's tsc pass. Without OPENBOT_APP_BROWSER every file still skips.

…ot in a timed hook

A beforeAll gets 5 s. In a2ui-frame, update-banner-layout, composer-dictation-notice-layout, checkbox-contrast and conversation-intro-halo the setup bundles a page or builds a stylesheet and starts Chromium, which can take longer than that on a cold runner when the file is the first one bun test opens. The setup now runs at the top of each file and waits for the bundle and for Chromium's launch, which Playwright bounds itself, as transcript-rail-clearance does.
@davidmckayv
davidmckayv force-pushed the fix/browser-test-setup-flake branch from 26e1990 to c8d85b5 Compare November 16, 2026 18:57
@davidmckayv
davidmckayv merged commit f612af0 into main Nov 16, 2026
16 checks passed
@davidmckayv
davidmckayv deleted the fix/browser-test-setup-flake branch November 16, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant