Skip to content

[test] Improve running speed of E2E testing #30

Description

@YingjingLu

Currently e2e testing on npm run test:e2e:docker. This is because of emulator boot time taking really long, especially on Github Workflow hosts. We need to optimize by caching more state

Activity

tsubasakong commented on Feb 23, 2026

@tsubasakong
Collaborator

/fix

github-actions commented on Feb 23, 2026

@github-actions

Here’s a concrete fix plan based on a quick repo scan (scripts + e2e Docker + emulator config).

Probable root causes

  • Cold boot every run because snapshots are explicitly disabled (-no-snapshot, -no-snapshot-save) in test/integration/docker-agent-e2e.mjs, so the emulator always does a full boot.
  • The Docker image bakes a Play Store system image (system-images;android-34;google_apis_playstore;x86_64) which is slower to boot than google_apis.
  • In CI, image layers may be rebuilt without cache, re-downloading Android SDK bits and recreating the AVD (docker/e2e/Dockerfile).

Likely files/modules involved

  • scripts/test-e2e-docker.sh
  • docker/e2e/Dockerfile
  • docker/e2e/entrypoint.sh
  • test/integration/docker-agent-e2e.mjs
  • src/device/emulator-manager.ts

Reproduction/verification steps

  1. Run npm run test:e2e:docker and time from “emulator start” to “boot completed” in the logs.
  2. Re-run in the same environment and compare boot time (should still be cold boot today).
  3. After applying changes, re-run twice to confirm the second run uses a warm snapshot and is faster.

Minimal fix options (recommended first)

  1. Recommended: Enable quickboot snapshot for the E2E emulator.
  • Remove -no-snapshot and -no-snapshot-save from cfg.emulator.extraArgs in test/integration/docker-agent-e2e.mjs.
  • Add -snapshot quickboot (or similar name) to extraArgs.
  • Bake a snapshot in the Docker image by starting the emulator once during build, waiting for boot, then saving a snapshot. This snapshot will live in the AVD directory inside the image and load quickly on subsequent runs.
  1. Switch system image to non-Play Store: change google_apis_playstore to google_apis in docker/e2e/Dockerfile and keep the rest. This is usually a noticeably faster boot with less background setup.
  2. CI layer caching: ensure build uses Docker buildx cache (inline cache, or cache-to/cache-from in GitHub Actions) so the SDK/AVD layers are reused. This addresses download + AVD creation time but not cold-boot time.

Suggested minimal implementation sketch (recommended approach)

  • In test/integration/docker-agent-e2e.mjs, change the emulator args to allow snapshots and request a named snapshot:
    • Remove -no-snapshot and -no-snapshot-save.
    • Add -snapshot quickboot.
  • In docker/e2e/Dockerfile, after avdmanager create avd ..., run a one-time boot and save:
    • emulator -avd OpenPocket_AVD -no-window -no-audio -gpu swiftshader_indirect -no-boot-anim -accel off &
    • adb wait-for-device + adb shell getprop sys.boot_completed loop
    • adb emu avd snapshot save quickboot
    • adb emu kill
  • Keep -no-snapshot-save for runtime if you want determinism, but keep -snapshot quickboot so it loads the baked snapshot.

Test cases to add/update

  • Add a timing/log assertion to ensure boot time decreases on the second run (e.g., compare elapsed boot wait for two sequential runs in the same container).
  • Add a simple check that adb emu avd snapshot list contains quickboot before starting the agent to catch missing snapshot early.

Risk and rollback notes

  • Snapshot-based boot can introduce state drift if the snapshot gets stale. Mitigation: bake the snapshot during image build and keep runtime -no-snapshot-save for deterministic runs.
  • Switching away from Play Store images may break tests that rely on Play Services/Play Store. Roll back by restoring google_apis_playstore.
  • If snapshot boot causes flakiness, roll back to cold boot by re-adding -no-snapshot and -no-snapshot-save.

Lightweight checks performed

  • Inspected scripts/test-e2e-docker.sh, docker/e2e/Dockerfile, docker/e2e/entrypoint.sh, test/integration/docker-agent-e2e.mjs, and src/device/emulator-manager.ts for emulator flags and boot flow.

Questions to confirm before changes

  • Do E2E tests require Play Store / Play Services, or is google_apis sufficient?
  • Are you OK with baking a snapshot during Docker build (slower build, faster test runs), or do you prefer snapshot creation at runtime?
  • Is there already a GitHub Actions cache for Docker layers we should align with?

Generated by issue-fix for issue #30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions