Skip to content

test(e2e): run the deployment scenario page as live tests - #14702

Draft
kaofelix wants to merge 6 commits into
v1from
test/varar-scenario-oaths
Draft

kaofelix wants to merge 6 commits into
v1from
test/varar-scenario-oaths

Conversation

@kaofelix

Copy link
Copy Markdown
Collaborator

Adds examples/scenarios.md, a table of deployment scenarios in which each row names an example and what the machine needs, and a live test, scenarios::page, that runs every row the machine can run. Each row it runs goes through plan, apply, a reply check, an unchanged plan, and destroy; each row it can't run is reported as skipped with the need the machine lacks.

Problem

The scenario matrix for GTC DC lives in a spreadsheet, and the examples it names are checked by hand on each machine.
The repository has live HCL cases, but each case is its own document and tofu test file, so a docs table of scenarios and the tests that run them are kept in sync by hand, and nothing says which rows a given machine can run.

Change

  • examples/scenarios.md: one table, with the columns Scenario, Example (a link), Needs, Model, Ask, and Expect. Needs uses a small vocabulary: linux, wsl2, arm64, amd64, docker, gpu NAME, kubernetes, and $NAME for a credential. Model: stub serves the stub model in place of every endpoint, so the row needs no GPU or credential.
  • crates/nemoclaw-e2e/tests/e2e/scenarios.rs:
    • Parses the page strictly. A table with other columns, an empty cell, an unknown need, an Example without exactly one link, a multi-word Expect, or a repeated ID fails the parse and names the row, so a row is never dropped silently.
    • Checks this machine against each row's needs.
    • Binds the example to the machine: a fresh UID, free gateway and service ports, a free subnet, the engine socket, and zero-digest images resolved from local REPO:TAG builds.
    • Runs the row through hcl_live::case.
    • Afterward, whatever the outcome, destroys the deployment and removes every Docker object labelled with its UID.
    • Optionally uses a model already serving on the machine instead of the services the example declares.
  • crates/nemoclaw-e2e/tests/e2e/hcl/scenarios/lifecycle.tftest.hcl: one lifecycle shared by every row, reusing the cli, verifier, and model modules. The row arrives as TF_VAR_scenario. The model module gains optional port and enabled variables; existing cases are unchanged.

Rejected alternatives:

  • Varar oaths, as spiked in #12916.
    • A Varar step can only pass or fail, so a row this machine can't run can't be reported as skipped.
    • A row here is a command to run, not values in prose for the code to match.
    • The spike's findings still apply, because Varar is still at 0.8.1: a sentence that never matches is silent, and oaths can't link to files.
  • One generated tofu test file per row. The lifecycle is the same for every row, so generating files would add a sync step without checking anything new.

Deliberately left out:

  • The runner deploys only managed Docker gateways, so the Kubernetes row is declared but would fail to bind.
  • No CI job runs the page yet. The stub row needs only Docker and could join cargo ci live-docker once the runner takes that job's image names.
  • A row whose example reads a credential must list it in Needs. The runner doesn't yet derive credential needs from the example.

Beyond CI

  • On macOS with Docker Desktop, using a bundle and OpenClaw image built from this branch, T-144 passed plan, apply, the stub's reply, unchanged plan, and destroy, and left no object labelled with its UID. The other six rows were reported as skipped, each with the need the machine lacks.
  • Breaking the row failed it as expected:
    • Changing Expect to another word failed the reply check, which named the agent.
    • A misspelled need failed the page parse, which named the row.
    • Dropping the stub model failed apply on the missing NVIDIA_API_KEY.
  • Unverified: the DGX Spark, Station, WSL2, and Kubernetes rows. No Spark row has run: sparky's GPU memory is held by another deployment's model service, and using that service needs its key.

@copy-pr-bot

copy-pr-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the v1 NemoClaw v1 branch label Nov 20, 2026
@kaofelix

Copy link
Copy Markdown
Collaborator Author
Authoring session — opening
Field Status Value Evidence or limit
Harness recorded Claude Code 2.1.329 Session record version field
Skills recorded nemoclaw-contributor-create-pr (injected) Skill invocations to cutoff; the hc manual was read only in part and isn't counted
Instruction files recorded AGENTS.md (injected); WRITING.md (complete read) Startup injection and file reads to cutoff
Tool calls recorded Bash: 117; Write: 5; Edit: 3; ToolSearch: 2; Read: 1; Skill: 1; AskUserQuestion: 1; Google Drive read_file_content: 1 Wrapper calls, including failed ones, to cutoff; the Bash wrapper hides nested commands, including those run on a remote host through hc
Tokens recorded input: 23307944; cached input: 22645191; output: 86009 Final usage per assistant response in the session record to cutoff; no subagents ran
Session window recorded 2026-11-19T22:52Z to 2026-11-20T15:08Z; 16 h 16 min First event to cutoff, including idle time overnight
Compactions recorded 0 Complete record to cutoff
Models recorded claude-opus-5-5 Assistant message model fields; no changes
Reasoning levels recorded medium Effort field on every request; no changes

@github-actions

Copy link
Copy Markdown
Contributor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v1 NemoClaw v1 branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants