chore(#18): recommendation document for test orchestration layer - #116
chore(#18): recommendation document for test orchestration layer#116sugat009 wants to merge 1 commit into
Conversation
Hareet
left a comment
There was a problem hiding this comment.
Read this against what our end-to-end QA demo actually exercised — the core recommendation (thin wrapper, no custom runner) is right, and we can now back it with evidence: the demo's tier-2 step was exactly this shape (shell out to the target config repo's own pinned mocha, capture, classify) and it closed a live red→green loop. npm run resolving the repo's local binaries also gives version parity for free — the same property we had to build a seam for on the cht-conf side (CHT_CONF_BIN).
Suggested additions that would make the implementation spec easier, all demo-earned:
- Spec the exec contract.
exec('npm test')hides where the failure modes live:
per-invocation timeout + SIGTERM,stdin: 'ignore'(tools that prompt will hang an
autonomous loop), stdout+stderr captured, and a never-rejecting result
({exitCode, timedOut, startError, output}) so the supervisor aggregates without
try/catch. #144'srunChtConfis this contract already — generalize it. - Child-env allowlist. npm scripts run arbitrary target-project code; LLM keys must not
reach them. #144'sminimalEnvis the precedent — one line here prevents a regression. - Expected-outcome runs. A QA loop needs direction-aware results: the generated
regression test must FAIL pre-fix (red) and PASS post-fix (green); a test that passes on
red is vacuous. Give invocationsexpected: 'pass' | 'fail'and report conformance. - Environment column in the npm-scripts table: none / self-provisioning
(integration-all-local,wdio-localbuild Docker → they inherit the human-gated
no-Docker posture) / instance-consuming (input: theEnvironmentHandle#144 ships —
worth naming as the concrete handoff type). - Browser suites need an explicit call. Karma/wdio launch Chrome; in a hardened
container (cap_drop ALL) that means baked Chromium +--no-sandbox/
--disable-dev-shm-usage— we debugged exactly this for the Enketo harness. Either spec
the bake or scope v1 to non-browser suites explicitly. - Structured output is cheaper than the CTRF section implies. Mocha's built-in
json
reporter via passthrough (npm run unit-api -- --reporter json; note npm silently drops
flags placed before--) gets machine-readable results in v1 with zero new deps. CTRF
then becomes the later cross-runner unification (wdio/Allure, karma) rather than the
first structured format. If parsingspectext anywhere, strip ANSI first. - Flaky policy: bounded auto-retry (once, timeout-classified failures only, both outputs
kept) rather than "prompt to re-run" — a human prompt mid-loop breaks autonomy. - Config-repo family: the config/default + cht-conf rows in the component table are the
family the cht-conf-extension's tier-2 harness runner already serves in miniature; when
this layer is built, that runner should become a consumer (or documented special case).
None of this blocks merging the recommendation doc as-is — happy to see it land and carry
these as amendments or into the #18 implementation spec.
Captures the #18 discovery findings as a committed recommendation document, alongside the other layer-recommendation docs in designs/layer_recommendations. It reorganizes the existing discovery comment (the all-Mocha/no-Jest finding and its component table, the npm-scripts catalog, the thin-wrapper orchestration approach, and the CTRF, dependency-cruiser, and nyc-diff future enhancements) into the established recommendation-doc format. This is a future layer under the QA Supervisor, consumed after the Test Environment Layer, and is not yet implemented.
Closes #18