Repository navigation
[test] Improve running speed of E2E testing #30
Copy link
Copy link
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Activity
/fix
Reacted by github-actions
Here’s a concrete fix plan based on a quick repo scan (scripts + e2e Docker + emulator config).
Probable root causes
- Cold boot every run because snapshots are explicitly disabled (
-no-snapshot,-no-snapshot-save) intest/integration/docker-agent-e2e.mjs, so the emulator always does a full boot. - The Docker image bakes a Play Store system image (
system-images;android-34;google_apis_playstore;x86_64) which is slower to boot thangoogle_apis. - In CI, image layers may be rebuilt without cache, re-downloading Android SDK bits and recreating the AVD (
docker/e2e/Dockerfile).
Likely files/modules involved
scripts/test-e2e-docker.shdocker/e2e/Dockerfiledocker/e2e/entrypoint.shtest/integration/docker-agent-e2e.mjssrc/device/emulator-manager.ts
Reproduction/verification steps
- Run
npm run test:e2e:dockerand time from “emulator start” to “boot completed” in the logs. - Re-run in the same environment and compare boot time (should still be cold boot today).
- After applying changes, re-run twice to confirm the second run uses a warm snapshot and is faster.
Minimal fix options (recommended first)
- Recommended: Enable quickboot snapshot for the E2E emulator.
- Remove
-no-snapshotand-no-snapshot-savefromcfg.emulator.extraArgsintest/integration/docker-agent-e2e.mjs. - Add
-snapshotquickboot(or similar name) toextraArgs. - Bake a snapshot in the Docker image by starting the emulator once during build, waiting for boot, then saving a snapshot. This snapshot will live in the AVD directory inside the image and load quickly on subsequent runs.
- Switch system image to non-Play Store: change
google_apis_playstoretogoogle_apisindocker/e2e/Dockerfileand keep the rest. This is usually a noticeably faster boot with less background setup. - CI layer caching: ensure build uses Docker buildx cache (inline cache, or cache-to/cache-from in GitHub Actions) so the SDK/AVD layers are reused. This addresses download + AVD creation time but not cold-boot time.
Suggested minimal implementation sketch (recommended approach)
- In
test/integration/docker-agent-e2e.mjs, change the emulator args to allow snapshots and request a named snapshot:- Remove
-no-snapshotand-no-snapshot-save. - Add
-snapshotquickboot.
- Remove
- In
docker/e2e/Dockerfile, afteravdmanager create avd ..., run a one-time boot and save:emulator -avd OpenPocket_AVD -no-window -no-audio -gpu swiftshader_indirect -no-boot-anim -accel off &adb wait-for-device+adb shell getprop sys.boot_completedloopadb emu avd snapshot save quickbootadb emu kill
- Keep
-no-snapshot-savefor runtime if you want determinism, but keep-snapshot quickbootso it loads the baked snapshot.
Test cases to add/update
- Add a timing/log assertion to ensure boot time decreases on the second run (e.g., compare elapsed boot wait for two sequential runs in the same container).
- Add a simple check that
adb emu avd snapshot listcontainsquickbootbefore starting the agent to catch missing snapshot early.
Risk and rollback notes
- Snapshot-based boot can introduce state drift if the snapshot gets stale. Mitigation: bake the snapshot during image build and keep runtime
-no-snapshot-savefor deterministic runs. - Switching away from Play Store images may break tests that rely on Play Services/Play Store. Roll back by restoring
google_apis_playstore. - If snapshot boot causes flakiness, roll back to cold boot by re-adding
-no-snapshotand-no-snapshot-save.
Lightweight checks performed
- Inspected
scripts/test-e2e-docker.sh,docker/e2e/Dockerfile,docker/e2e/entrypoint.sh,test/integration/docker-agent-e2e.mjs, andsrc/device/emulator-manager.tsfor emulator flags and boot flow.
Questions to confirm before changes
- Do E2E tests require Play Store / Play Services, or is
google_apissufficient? - Are you OK with baking a snapshot during Docker build (slower build, faster test runs), or do you prefer snapshot creation at runtime?
- Is there already a GitHub Actions cache for Docker layers we should align with?
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Currently e2e testing on
npm run test:e2e:docker. This is because of emulator boot time taking really long, especially on Github Workflow hosts. We need to optimize by caching more state