Skip to content

chore: run e2e tests in Playwright container - #1888

Merged
latin-panda merged 11 commits into
masterfrom
use-playwright-container
Sep 17, 2026
Merged

latin-panda merged 11 commits into
masterfrom
use-playwright-container

Conversation

@latin-panda

@latin-panda latin-panda commented Sep 7, 2026 •

Copy link
Copy Markdown
Collaborator

What has been done to verify that this works as intended?

Run CI multiple times with different commits

Why is this the best possible solution? Were any other approaches considered?

  • Reproducible locally with Docker. Before, we had differences between local macOS and Linux vs GHA's runner.
    • Downside: we must bump the Playwright image tag when upgrading @playwright/test. It fails loudly if missed, so we'll find out fast and fix it. I consider it worth it because of the improvement made in this branch.
  • Before, 6 jobs each downloaded browsers from a CDN, and a job was sometimes canceled when the download was slow. Now jobs use the Playwright container image, so there's no per-browser install step and less exposure to network flakiness.
  • One shared cache action, cache keys computed once.
  • Removed the node-version matrix, bc it had a single entry and only fed the cache action. Node now comes from package.json (setup-node) and the Playwright image. This is what renames the check names (see below).
  • It renames the e2e jobs (drops the node version from the name). The three old names (Test web-forms e2e / Functional e2e (24.16.0, <browser>) are required status checks in branch protection, so after this merges, rebased and newly created branches will show them stuck at "Expected" and can't merge until our GH Admin updates the required check names. The same would happen anyway on any future node-version bump, so it's worth fixing the naming now.
    • To update branch protection, I think it's in Settings -> Branches -> required status checks. I'll also give the team a heads-up.

How does this change impact users? Describe intentional behavior changes from code updates. What are the regression risks?

N/A

Does this change require updates to user documentation? If so, please file an issue here and include the link below.

No

@changeset-bot

changeset-bot Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 76751e5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@latin-panda latin-panda changed the title experiment: use playwright container chore: run e2e tests in Playwright container Sep 9, 2026
@latin-panda
latin-panda marked this pull request as ready for review September 9, 2026 16:40
@latin-panda

Copy link
Copy Markdown
Collaborator Author

@garethbowen Last week, I had an issue with checks timing out because the browser download took too long, even though the tests weren't broken. I found in Playwright docs that they have containers. So far, it looks good to me. What do you think?

@garethbowen garethbowen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find!

To be confident of reliability changes like this we usually run the build multiple times (maybe 100). Have you tried that?

It looks like there are multiple threads here, not just the container change.

Does this also resolve getodk/central#2060 ? @alxndrsn

Comment thread .github/actions/node-cache/action.yml Outdated
Comment thread .github/workflows/wf-ci-e2e.yml
Comment thread .github/workflows/wf-ci-e2e.yml
Comment thread .github/workflows/wf-ci-e2e.yml
Comment thread .github/workflows/wf-ci.yml Outdated
@alxndrsn

Copy link
Copy Markdown
Contributor

Does this also resolve getodk/central#2060 ? @alxndrsn

It might very well side-step it, but it doesn't seem to right now:

https://github.com/getodk/central-frontend/actions/runs/34334779372/job/102523218655?pr=1888

2026-09-09T15:13:22.7806563Z [e2e-tester] Playwright: Version 1.60.0
2026-09-09T15:13:22.7807069Z [e2e-tester] Installing playwright deps...
2026-09-09T15:13:23.5450341Z Installing dependencies...
2026-09-09T15:13:23.5541478Z Switching to root user to install dependencies...
2026-09-09T15:13:23.6514198Z Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [144 B]
2026-09-09T15:13:23.6958403Z Hit:2 http://azure.archive.ubuntu.com/ubuntu noble InRelease
2026-09-09T15:13:23.6978958Z Get:3 http://azure.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
2026-09-09T15:13:23.7040767Z Get:4 http://azure.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
2026-09-09T15:13:23.7079522Z Get:5 http://azure.archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]

@garethbowen garethbowen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there!

Comment thread .github/actions/node-cache/action.yml Outdated
Comment thread .github/workflows/wf-ci-e2e.yml
Comment thread packages/web-forms/tests/fixtures/3-notes-with-markdown.xml

@garethbowen garethbowen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@latin-panda
latin-panda merged commit c2d2e6a into master Sep 17, 2026
263 of 264 checks passed
@latin-panda
latin-panda deleted the use-playwright-container branch September 17, 2026 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants