Skip to content

Add private operations KB replay and Journey-3 history - #36

Merged
gouhongshen merged 6 commits into
mainfrom
feat/j3-ops-regression
Nov 18, 2026
Merged

gouhongshen merged 6 commits into
mainfrom
feat/j3-ops-regression

Conversation

@gouhongshen

@gouhongshen gouhongshen commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Journey-3 adds 60 slow production-question replays to the existing 57 knowledge cases. Two standard Ubuntu runners execute concurrently, each with up to 10 isolated case processes. cases (documents) runs J3-01–J3-57. cases (operations) runs J3-58–J3-117 for both operations agents. QA runs remain manual; PR CI only tests the framework offline.

  • Select each source agent's 30 slowest distinct requests from the last seven days, including successful, failed and canceled requests. Freeze the original question clock and keep prior conversation as quoted context.
  • Copy four structured KBs, agent configurations and supported Q&A dependencies into QA without channel bindings. All 272 copied tables passed final QA row-count verification (35,470,556 rows); effective enabled source sets match production (77/64/64/64).
  • Keep immutable private questions, independently calculated goldens and evidence in QA. Fifty-three selected questions have usable knowledge-answer references; seven require data outside the copied KBs and remain explicitly unscored. Recover and checksum-verify the original 91-row CSV for the identity-mapping question rather than substituting the current registration population.
  • Save each case's question, agent, latency, quality, timestamp, tool/model rounds, failure reason, trace/session IDs, versions, report and evidence reference to journey3_benchmark.results. Export history once after both case jobs finish and generate numeric charts, CSV and JSON. The history page overlays all cases by default, with Documents, Operations and individual case views. Each case version keeps a separate line and missing grades leave gaps.
  • Publish only allowlisted IDs, states and numeric metrics for every Journey-3 case. Questions, business names, answers, citations, SQL/tool data, raw traces and child-process logs stay in QA.
  • Verify live enabled source registrations and required agent bindings before scoring. Block benchmark-control-table or registered reference/evidence-file access from producing a quality score.
  • Preserve green framework completion when the agent fails, times out or cannot deliver; persistence/orchestration failures remain explicit harness errors.

Validation:

  • 511 Python tests and 17 browser tests passed locally. The two-runner change verifies the exact 57/60 matrix split and metrics-only report aggregation in an offline smoke. Browser coverage verifies the new history scopes, the shared time axis, isolated versions, and legend highlighting.
  • All 60 current private case versions and 53 usable reference shapes/checksums were verified against their registered QA versions.
  • Live checks verify source governance, private agent bindings, both sealed snapshots, private-file access detection, QA history persistence and sanitized report/chart generation.
  • Final post-restore QA smoke: the actual agent task failed after 86 seconds with insufficient_credit. Its trace/session and 29 events were captured; report, QA archive/history and metrics-only publication completed with harness exit 0. No successful new model-quality evaluation is claimed.

Readiness limitations: the public-cloud replica still lacks five MCP tools and a dependent custom skill because QA does not expose the required MCP exports. It is blocked rather than scored as an equivalent replica. Model credit must be available for real answer-quality measurements. No shared QA feature flags, MOI/SDK code, production settings or schedules were changed.

@gouhongshen gouhongshen changed the title feat(journey-3): add private operations replay and historical metrics Add private operations KB replay and Journey-3 history Nov 17, 2026
@gouhongshen
gouhongshen marked this pull request as ready for review November 18, 2026 05:58
@gouhongshen
gouhongshen merged commit ca757a7 into main Nov 18, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant