Skip to content

chore(#18): recommendation document for test orchestration layer - #116

Open
sugat009 wants to merge 1 commit into
mainfrom
18-test-orchestration-layer-recommendation
Open

chore(#18): recommendation document for test orchestration layer#116
sugat009 wants to merge 1 commit into
mainfrom
18-test-orchestration-layer-recommendation

Conversation

@sugat009

Copy link
Copy Markdown
Member

Captures the #18 discovery findings as a committed recommendation document, alongside the other layer-recommendation docs in designs/layer_recommendations. It reorganizes the existing discovery comment (the all-Mocha/no-Jest finding and its component table, the npm-scripts catalog, the thin-wrapper orchestration approach, and the CTRF, dependency-cruiser, and nyc-diff future enhancements) into the established recommendation-doc format. This is a future layer under the QA Supervisor, consumed after the Test Environment Layer, and is not yet implemented.

Closes #18

@Hareet Hareet left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read this against what our end-to-end QA demo actually exercised — the core recommendation (thin wrapper, no custom runner) is right, and we can now back it with evidence: the demo's tier-2 step was exactly this shape (shell out to the target config repo's own pinned mocha, capture, classify) and it closed a live red→green loop. npm run resolving the repo's local binaries also gives version parity for free — the same property we had to build a seam for on the cht-conf side (CHT_CONF_BIN).

Suggested additions that would make the implementation spec easier, all demo-earned:

  1. Spec the exec contract. exec('npm test') hides where the failure modes live:
    per-invocation timeout + SIGTERM, stdin: 'ignore' (tools that prompt will hang an
    autonomous loop), stdout+stderr captured, and a never-rejecting result
    ({exitCode, timedOut, startError, output}) so the supervisor aggregates without
    try/catch. #144's runChtConf is this contract already — generalize it.
  2. Child-env allowlist. npm scripts run arbitrary target-project code; LLM keys must not
    reach them. #144's minimalEnv is the precedent — one line here prevents a regression.
  3. Expected-outcome runs. A QA loop needs direction-aware results: the generated
    regression test must FAIL pre-fix (red) and PASS post-fix (green); a test that passes on
    red is vacuous. Give invocations expected: 'pass' | 'fail' and report conformance.
  4. Environment column in the npm-scripts table: none / self-provisioning
    (integration-all-local, wdio-local build Docker → they inherit the human-gated
    no-Docker posture) / instance-consuming (input: the EnvironmentHandle #144 ships —
    worth naming as the concrete handoff type).
  5. Browser suites need an explicit call. Karma/wdio launch Chrome; in a hardened
    container (cap_drop ALL) that means baked Chromium + --no-sandbox/
    --disable-dev-shm-usage — we debugged exactly this for the Enketo harness. Either spec
    the bake or scope v1 to non-browser suites explicitly.
  6. Structured output is cheaper than the CTRF section implies. Mocha's built-in json
    reporter via passthrough (npm run unit-api -- --reporter json; note npm silently drops
    flags placed before --) gets machine-readable results in v1 with zero new deps. CTRF
    then becomes the later cross-runner unification (wdio/Allure, karma) rather than the
    first structured format. If parsing spec text anywhere, strip ANSI first.
  7. Flaky policy: bounded auto-retry (once, timeout-classified failures only, both outputs
    kept) rather than "prompt to re-run" — a human prompt mid-loop breaks autonomy.
  8. Config-repo family: the config/default + cht-conf rows in the component table are the
    family the cht-conf-extension's tier-2 harness runner already serves in miniature; when
    this layer is built, that runner should become a consumer (or documented special case).

None of this blocks merging the recommendation doc as-is — happy to see it land and carry
these as amendments or into the #18 implementation spec.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

[Phase 2] Discovery: Test Orchestration Layer (npm scripts & Coverage)

2 participants