feat(hdx-eval): add dashboard-build eval scenario

[brandon-pereira]

Summary

Adds a dashboard-build eval scenario that tests an agent's ability to create multi-tile observability dashboards via MCP tools. Unlike the existing investigation scenarios (error-root-cause, latency-spike, etc.) which evaluate text answers, this evaluates created artifacts — the dashboards themselves are inspected post-run via the API.

The scenario is intentionally hard. The agent must handle vague user prompts, distractor services, impossible metric requests, messy severity data, and cross-dashboard drill-downs — all within a 15-turn budget. Current baseline: 75% combined score (92% programmatic, 78% judge), leaving room for improvement as the MCP tools and prompts evolve.

What the scenario tests

The agent receives a realistic user request for two dashboards ("Service Health Overview" + "Service Detail") and must:

• Discover sources and schemas via list_sources / describe_source
• Create dashboards with 18-22 tiles across 8 display types (line, stacked_bar, table, number, heatmap, pie, search, markdown, raw SQL)
• Wire cross-dashboard onClick drill-downs with ServiceName filters
• Handle data traps: 4 of 7 services are internal infrastructure (not user-facing), debug-proxy has a fake 15% error rate, inventory-service has a misleading latency red herring, SeverityText has mixed casing, and CPU/memory metrics don't exist
• Verify tiles return data via query_tile
• Report what was built and flag data quality caveats

What's in this PR

• Scenario generator (dashboard-build/generate.ts) — seeds 2M traces + 4M logs across 7 services (3 user-facing + 4 distractors) with deliberate data traps
• Post-run inspection (dashboardInspection.ts) — fetches created dashboards via the v2 API, queries every tile, and formats structured evidence for the LLM judge including heuristic distractor-awareness signals
• Ground truth + rubric (ground-truth.json) — 24 programmatic regex checks and 7 weighted judge criteria
• HyperDX API client extensions (api.ts) — dashboard CRUD, tile querying, and MCP JSON-RPC for post-run inspection
• System prompt — minimal dashboard-building instructions with DATA REVIEW guidance and conciseness constraint
• Unit tests — volume targets, determinism, service distribution, error spikes, messy severity, latency red herring

Uses the scenario hooks framework merged in PR #2547 (buildSystemPrompt, allowedToolPatterns, judgeSystemPreamble, postRunInspection).

Eval results (75% baseline)

Metric	Value
Combined score	75%
Programmatic score	92%
Judge mean	78%
Tool calls (mean)	23
Tool-error penalty	8pp
Duration (mean)	282s
Completion rate	3/3 final_answer

Judge Criterion	Score
tool_efficiency	5.0/5
cross_dashboard_drill	4.7/5
computed_expressions	4.0/5
structure_and_design	4.0/5
verification	4.0/5
tile_correctness	3.7/5
data_awareness	2.7/5

data_awareness (2.7/5) is the main gap — the agent partially flags data traps but doesn't consistently scope dashboards to user-facing services or handle all the planted red herrings. This is by design: the scenario is meant to be hard enough that MCP prompt improvements show measurable gains.

[changeset-bot]

🦋 Changeset detected

Latest commit: 3208f04

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@hyperdx/hdx-eval	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	Jul 8, 2026 6:40pm
hyperdx-storybook	Ready	Preview, Comment	Jul 8, 2026 6:40pm

[greptile-apps]

Greptile Summary

This PR adds a dashboard-build eval scenario that tests an agent's ability to create multi-tile observability dashboards via MCP tools. It introduces a scenario generator, a post-run artifact inspection pipeline, and a complete grading rubric — architected cleanly via the existing scenario hooks framework (buildSystemPrompt, postRunInspection, etc.).

• Scenario generator seeds 2M traces and 4M logs across 7 services (3 user-facing + 4 distractors) with deliberate data traps: messy severity casing, a latency red herring, a misleading distractor error rate, and mixed deployment environments — backed by a thorough deterministic unit-test suite.
• Post-run inspection (dashboardInspection.ts) fetches created dashboards via the v2 API, queries every tile, and formats structured evidence for the LLM judge including heuristic distractor-awareness signals.
• logs.ts gains a deriveSeverityNumber helper and an explicit severityNumber field in LogInput, fixing silent mis-mapping of non-canonical severity texts (e.g., warning → 13, fatal → 21) that previously fell through to INFO.

Confidence Score: 5/5

Safe to merge — all changes are confined to the eval package with no impact on production code paths.

The new scenario generator, inspection pipeline, and grading integration are well-tested and cleanly isolated. All findings are in heuristic evidence-collection logic (a false-positive risk for crossDashboardOnClickValid, a case-sensitive fallback regex, and a silent skip in grade.ts), none of which affect the correctness of scores or the integrity of production data.

No files require special attention — all issues are in heuristic evidence-formatting code within the eval package.

Important Files Changed

Filename	Overview
packages/hdx-eval/src/scenarios/dashboard-build/generate.ts	New scenario generator seeding 2M traces + 4M logs across 7 services with intentional data traps. Deterministic via seeded RNG. crossDashboardOnClickValid check in the post-run inspection hook doesn't filter out self-referential onClick links.
packages/hdx-eval/src/grading/dashboardInspection.ts	New post-run inspection module fetching created dashboards via v2 API, querying every tile, and building structured evidence for the LLM judge. ID_REGEX fallback is case-sensitive for hex (lowercase only). Distractor-awareness heuristics are well-commented.
packages/hdx-eval/src/grading/grade.ts	Adds post-run inspection integration and inspection summary logging. Silently skips inspection when postRunInspection is defined but inspectionConfig is absent; a warning would improve observability.
packages/hdx-eval/src/hyperdx/api.ts	Adds dashboard CRUD, tile querying via MCP JSON-RPC, and SSE/JSON response parsing. clickstack_query_tile tool name is consistent with the rest of the codebase.
packages/hdx-eval/src/tests/dashboard-build.test.ts	Comprehensive unit tests covering volume targets, determinism, service distribution, error spikes, messy severity mapping, and latency red herring at 1% volume factor.
packages/hdx-eval/src/generators/logs.ts	Adds optional severityNumber to LogInput and introduces deriveSeverityNumber to correctly resolve non-canonical severity texts (warning, fatal, information) to their OTel numbers.
packages/hdx-eval/src/harness/systemPrompt.ts	Adds SAMPLING_CAVEAT_BLOCK constant and routes scenarios with buildSystemPrompt hook to their own prompt builder, cleanly supporting non-investigation scenario types.
packages/hdx-eval/src/tests/systemPrompt.test.ts	Adds tests for SAMPLING_CAVEAT_BLOCK injection and dashboard-specific system prompt. The relative p.length < investigationPrompt.length comparison correctly verifies conciseness.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant CLI as cli.ts (grade)
    participant Grade as grade.ts
    participant Inspect as dashboardInspection.ts
    participant API as HyperdxApiClient
    participant Judge as LLM Judge

    CLI->>Grade: "gradeBatch(batchDir, {inspectionConfig})"
    Grade->>Grade: runProgrammaticChecks(finalAnswer, rubric)
    Grade->>Grade: computeToolErrorStats(record)

    alt "scenario.postRunInspection && inspectionConfig"
        Grade->>Inspect: inspectDashboards(toolCalls, apiUrl, ...)
        Inspect->>Inspect: extractDashboardIds(toolCalls)
        loop for each dashboardId
            Inspect->>API: getDashboardV2(id, accessKey)
            loop for each tile
                Inspect->>API: queryTileWithEvidence(tileId)
                API-->>Inspect: TileQueryEvidence
            end
        end
        Inspect->>Inspect: analyzeDistractorAwareness(result)
        Inspect-->>Grade: PostRunInspectionResult
    else no inspectionConfig
        Grade->>Grade: silent skip
    end

    Grade->>Judge: "judgeTrajectory({inspectionEvidence, rubric})"
    Judge-->>Grade: JudgeResult
    Grade-->>CLI: "GradeRecord {combinedScore}"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant CLI as cli.ts (grade)
    participant Grade as grade.ts
    participant Inspect as dashboardInspection.ts
    participant API as HyperdxApiClient
    participant Judge as LLM Judge

    CLI->>Grade: "gradeBatch(batchDir, {inspectionConfig})"
    Grade->>Grade: runProgrammaticChecks(finalAnswer, rubric)
    Grade->>Grade: computeToolErrorStats(record)

    alt "scenario.postRunInspection && inspectionConfig"
        Grade->>Inspect: inspectDashboards(toolCalls, apiUrl, ...)
        Inspect->>Inspect: extractDashboardIds(toolCalls)
        loop for each dashboardId
            Inspect->>API: getDashboardV2(id, accessKey)
            loop for each tile
                Inspect->>API: queryTileWithEvidence(tileId)
                API-->>Inspect: TileQueryEvidence
            end
        end
        Inspect->>Inspect: analyzeDistractorAwareness(result)
        Inspect-->>Grade: PostRunInspectionResult
    else no inspectionConfig
        Grade->>Grade: silent skip
    end

    Grade->>Judge: "judgeTrajectory({inspectionEvidence, rubric})"
    Judge-->>Grade: JudgeResult
    Grade-->>CLI: "GradeRecord {combinedScore}"

Loading

_{Reviews (5): Last reviewed commit: "Merge branch 'main' into brandon/brandon..." | Re-trigger Greptile}

[github-actions]

E2E Test Results

✅ All tests passed • 227 passed • 3 skipped • 1494s

Status	Count
✅ Passed	227
❌ Failed	0
⚠️ Flaky	1
⏭️ Skipped	3

Tests ran across 4 shards in parallel.

View full report →

[github-actions]

🔴 Tier 4 — Critical

Touches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD.

Why this tier:

• Large diff: 1757 production lines changed (threshold: 1000)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

• Production files changed: 11
• Production lines changed: 1757 (+ 317 in test files, excluded from tier calculation)
• Branch: brandon/brandon-dashboard-evals
• Author: brandon-pereira

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[github-actions]

Deep Review

Scope: packages/hdx-eval — new dashboard-build eval scenario (generator, HyperDX API-client extensions, post-run dashboard inspection + judge-evidence heuristics, LogRow.severityText widening, system-prompt sampling caveat). Base e2f750e3.
Mode: report-only (read-only, no edits).

✅ No critical issues found. The happy path is sound — the new MCP/v2 response parsing was verified against the current server response shape, existing severity consumers are unaffected by the type widening, and inspection failures are already caught in grade.ts. The findings below are correctness/maintainability risks in new eval-scoring logic (internal tooling, no production surface), so all land at P2 or below.

🟡 P2 — recommended

• packages/hdx-eval/src/grading/dashboardInspection.ts:519 — extractIntendedTileConfigs keys the intended-config map by tile name with last-write-wins (:137), and the scenario deliberately creates two dashboards that share tile names; formatDashboardEvidence then prints only tile.intendedConfig ?? tile.config, so dashboard 1's tile is shown to the judge with dashboard 2's config and its own actual config is never emitted — contradicting the code comment that claims the judge sees both.
  • Fix: Key the intended-config map by (dashboardId, tileName) or tileId, or always print both intended and actual config in the evidence.
  • _{adversarial-reviewer, correctness-reviewer}
• packages/hdx-eval/src/grading/dashboardInspection.ts:437 — latencyBrokenDownByEndpoint ANDs spanname and quantile/duration substrings over a blob concatenating every tile's config, so a top-endpoints table (SpanName) plus any separate latency tile satisfies both conditions and the signal reports true even when no latency tile is actually segmented by endpoint — inflating the highest-weighted data_awareness judge criterion for agents that never saw through the inventory latency red herring.
  • Fix: Evaluate the signal per-tile, requiring SpanName in a groupBy and quantile/Duration in the same tile's config.
  • _{correctness-reviewer, testing-reviewer}
• packages/hdx-eval/src/grading/dashboardInspection.ts:404 — filtersOutDistractors inspects only the first occurrence of each distractor name (lower.indexOf(svc)) within a 60-char window, so exclusion done in a later tile is missed and the judge is told the agent failed to filter distractors — the primary data_awareness failure mode the scenario grades.
  • Fix: Scan all occurrences of each distractor name (loop over indexOf or a global regex) instead of only the first.
  • _{adversarial-reviewer}
• packages/hdx-eval/src/generators/logs.ts:22 — deriveSeverityNumber re-encodes the case/alias taxonomy already expressed by normalizeSeverityText/SEVERITY_NUMBER_BY_TEXT in templates.ts, and the two disagree (deriveSeverityNumber('fatal') → 21 vs normalizeSeverityText('fatal') → ERROR/17), so editing one without the other silently changes emitted severity numbers.
  • Fix: Share a single severity-taxonomy source (map + normalizer) between logs.ts and templates.ts instead of the inline duplicate.
  • _{maintainability-reviewer, api-contract-reviewer}
• packages/hdx-eval/src/grading/dashboardInspection.ts:354 — DISTRACTOR_SERVICES/USER_FACING_SERVICES duplicate the service names already defined in generate.ts (SERVICE_WEIGHTS) and in ground-truth.json; renaming a service in the generator makes the awareness heuristics silently return false with no error, feeding wrong hints to the judge.
  • Fix: Derive both lists from ground-truth.json (already imported) or a shared constants module.
  • _{maintainability-reviewer}
• packages/hdx-eval/src/grading/dashboardInspection.ts:188 — inspectDashboards calls login() before any evidence collection even though getDashboardV2/queryTileWithEvidence use Bearer auth (only deleteDashboard needs the cookie), and cleanup at :307 is not in a finally; a login failure throws, grade.ts:265 swallows it to a console.warn, and the judge then scores on the agent's text alone (contradicting its own artifact-first preamble) while the created dashboards leak uncleaned on the shared eval account.
  • Fix: Collect Bearer-auth evidence before login, run cleanup in a finally, and surface a login failure into result.errors rather than rejecting the whole inspection.
  • _{adversarial-reviewer, reliability-reviewer}
• packages/hdx-eval/src/grading/dashboardInspection.ts:392 — the pure, scoring-backbone functions that drive eval results (analyzeDistractorAwareness, extractDashboardIds, extractIntendedTileConfigs, and api.ts extractMcpContent) have no unit tests and are module-private, so the regex/window/first-occurrence bugs above ship unguarded and future edits mis-score silently.
  • Fix: Export these functions and add table-driven tests over hand-built fixtures covering the exclusion-window, allow-list, name-collision, and MCP JSON-RPC/SSE branches.
  • _{testing-reviewer, correctness-reviewer}

🔵 P3 nitpicks (6)

• packages/hdx-eval/src/hyperdx/api.ts:330 — MCP parsing assumes the payload is wrapped in a result key and that SSE frames start with the literal data: (space required); a bare-array/{data:[]} payload or a spaceless/multi-line SSE frame would report hasData:false for a data-bearing tile. Contingent on the envelope differing from the currently-verified server shape.
  • Fix: Apply the array/data/object-scan fallback to inner itself when inner.result is absent, and strip the data: prefix tolerantly while reassembling multi-line SSE data fields.
• packages/hdx-eval/src/grading/dashboardInspection.ts:425 — the scopesToUserFacing regex \bin\s*\( matches the in ( inside a NOT IN ('web-gateway',…) clause, so a where-clause excluding the user-facing services can flip the allow-list signal true.
  • Fix: Add a negative lookbehind / boundary check for a preceding not\s+ before in.
• packages/hdx-eval/src/hyperdx/api.ts:100 — the const err = new Error(...); (err as ApiError).status = ...; throw err pattern is repeated three times (request and twice in getDashboardV2).
  • Fix: Extract an apiError(message, status): ApiError helper.
• packages/hdx-eval/src/grading/dashboardInspection.ts:209 — HyperdxDashboard omits the containers field the v2 API returns, forcing a double as cast whose hand-written element shape (collapsed, title, tabs) bypasses the type checker and would be undefined at runtime if the API renamed a field.
  • Fix: Add containers?: […] to HyperdxDashboard and read it directly.
• packages/hdx-eval/src/hyperdx/api.ts:36 — TileQueryEvidence is module-private but its exact shape is hand-duplicated as TileEvidence.queryResult in dashboardInspection.ts, so the two can drift.
  • Fix: Export TileQueryEvidence and reuse it as the queryResult type.
• packages/hdx-eval/src/grading/dashboardInspection.ts:72 — the ID_REGEX comment claims it matches "UUIDs, and other alphanumeric IDs" but [a-f0-9-]{24,36} only matches lowercase hex + hyphens.
  • Fix: Correct the comment or broaden the pattern to match its stated intent.

---

Reviewers (8): correctness, adversarial, kieran-typescript, api-contract, reliability, testing, maintainability, project-standards.

Testing gaps:

• No unit tests exist for any pure function in dashboardInspection.ts or extractMcpContent in api.ts; the only new test file covers the data generator exclusively.
• deriveSeverityNumber's fallback branches (WARN*/FATAL/ERR*/DEB*/TRACE) are never exercised — every caller passes an explicit severityNumber, so the messy-alias derivation it was written for is uncovered.
• No test asserts the three severity-number sources (SEVERITY_NUMBER, SEVERITY_NUMBER_BY_TEXT, deriveSeverityNumber) agree, leaving the divergence risk uncaught.

[pulpdrew]

LGTM, with some non-blocking questions and ideas.

First idea: in the future it might be good to run a similar eval on a source with a very different schema to see how the agent copes with that (since many of our dashboard examples use the standard schema).

[pulpdrew]

LGTM