[AbhitejJohn]

Summary

• Keep column headers visible: The seven Skill Value headers stay in view when you scroll.
• Move agentic-workflows to the bottom: The plugin appears last; all other plugins keep their order.
• Search the full hierarchy: Case-insensitive substring search matches plugin or skill names. A plugin match shows all its skills and models; a skill match shows its parent plugin and all its models. Search filters cached rows once per animation frame without rebuilding the table, and clearing it restores the prior expanded state.
• Fix a Vally eval lint blocker for main: Move three binlog dormancy flags from unsupported top-level expect_activation fields to Vally 0.18's supported tags.expect_activation: "false" form. This keeps the same prompts, graders, and expected non-activation. The main push workflow failed on this binlog schema error; the PR's full-corpus lint now passes. A similar error in the proposed telemetry eval belongs to #1036, not to main or this PR.

Why: The dashboard needs easier navigation, and the binlog eval metadata must use the supported Vally schema so it does not block corpus validation after merge. Impact: Model and judge filters still work with search; binlog dormancy coverage remains in place.

Related issue

N/A (dashboard UX and eval lint fix).

Validation

• node --test eng\dashboard\dashboard-filtering.test.js eng\dashboard\dashboard-freshness.test.js eng\dashboard\dashboard-verdict.test.js eng\dashboard\skill-value.test.js — passed, 19/19 tests.
• node --test eng\evaluation-tools\lint-corpus.test.mjs — passed, 5/5 tests.
• python eng/eval-quality/check_eval_quality.py — passed with no errors (with process-local TEMP and TMP on Q:\ because C:\ was low on space).
• python "$env:USERPROFILE\.copilot\session-state\9040bcb0-2970-4582-9711-7476739e638f\files\dashboard-playwright.py" — passed in Chromium with a local-only sample-data fixture: sticky header after scrolling, plugin order, hierarchy matches, no-match state, collapse restore, model-filter intersection, table reuse, and no page errors. The fixture is not part of this PR.
• git diff --check origin/main...HEAD — passed.
• CI evaluation-tools run: Validate full eval corpus passed, 111/111 specs. Existing scoring-defaults-applied warnings are non-blocking.
• Bound PR evaluation — passed all six Vally jobs; all PR checks passed or were skipped.

Checklist

• I searched existing issues and pull requests to avoid duplicates.
• I kept this pull request focused and avoided unrelated refactors.
• I added or updated tests, evals, or documentation when changing skill or agent behavior.
• I updated CODEOWNERS when adding or moving owned content.
• I updated all marketplace manifests when plugin metadata changed.
• I updated eng/known-domains.txt for any new external domains referenced by skill content.

The last three items are not applicable: no ownership, plugin metadata, or skill-content domains changed.

Evaluation changes only

For every eval-related change, including a new eval:

• Scenarios are necessary, distinct, and use natural prompts.
• Graders cover the full result, accept the golden result, and reject a realistic mutation.
• No-op, dormancy, and statistical power are covered where needed.
• I ran the applicable production evaluation path and recorded the result above.
• If this fixes a failed eval, I classified the failure before editing skill content.
• If this broadly changes routing or behavior, I checked separate model-family evidence.

No prompts, graders, or trial counts changed: the three dormancy scenarios retain their activation expectations. The first, second, and last unchecked items are unchanged or not applicable; this PR does not broadly change routing or behavior. Vally 0.18 rejected a top-level field; this was not a skill regression.

[Copilot]

🟡 Changes recommended

Changing model filters during search loses the expansion state that clearing search should restore.

1 open issue

• Model filter redraw loses pre-search tree expansion state · New

What changed in this PR

Improves navigation in the dashboard’s Skill Value table.

Changes:

• Keeps column headers visible while scrolling.
• Adds case-insensitive plugin and skill search with cached row filtering.
• Places agentic-workflows last and adds search regression tests.

File	Description
eng/​dashboard/​skill-value.test.js	Tests header styling, search behavior, ordering, and filter interaction.
eng/​dashboard/​skill-value.js	Implements search, hierarchy visibility, and plugin ordering.
eng/​dashboard/​dashboard.html	Styles search controls and sticky headers.

_{Review effort: Balanced}

[github-actions]

👋 @AbhitejJohn — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[Copilot]

🟡 Changes recommended

Widening filters during search can leave model rows visible beneath collapsed parents after clearing search.

1 open issue

• Expanded search filters fail to preserve collapsed model visibility · New

1 resolved since last review

• Model filter redraw loses pre-search tree expansion state

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The focused UI changes have regression coverage and no unresolved blocking findings.

No open issues 🎉

1 resolved since last review

• Expanded search filters fail to preserve collapsed model visibility

_{Review effort: Balanced}

[github-actions]

✅ Evaluation passed for 4006ac8. cc @AbhitejJohn @JanKrivanek — please review.

[Copilot]

🟢 Approval recommended

No blocking issues remain, and regression coverage addresses both previously reported expansion-state problems.

No open issues 🎉

_{Review effort: Balanced}

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage

[github-actions]

📊 Skill and Agent Evaluation Results

6 model/target results across 3 targets and 2 models — ✅ 0 improved, ➖ 6 results without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 664894e3c119e3dd011028be54bc4403cd4347bf; 2 judge models.

Measurement health: 6 expected / 6 observed / 6 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
agent.build-perf	claude-sonnet-5	➖ Baseline signal, tie-limited	n=5; 1W/2T/2L; d=3; p=0.500; net -20.0%	—	—	Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
agent.build-perf	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=5; 3W/1T/1L; d=4; p=0.312; net +40.0%	—	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
agent.msbuild	claude-sonnet-5	➖ Improvement signal, unproven	n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%	—	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
agent.msbuild	gpt-5.6-luna	➖ Baseline signal, tie-limited	n=6; 1W/2T/3L; d=4; p=0.312; net -33.3%	—	—	Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
troubleshoot-binlog-mcp	claude-sonnet-5	➖ Improvement signal, tie-limited	n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%; 3 dormancy excluded	🟡 0.40	Activation: isolated 7/7; plugin 6/7; Activation-only stop: isolated 2 failed runs; Activation-only stop: plugin 3 failed runs	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
troubleshoot-binlog-mcp	gpt-5.6-luna	➖ No preference	n=7; 0W/7T/0L; d=0; p=1.000; net +0.0%; 3 dormancy excluded	🟡 0.48	Activation: isolated 7/7; plugin 6/7; Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 2 failed runs	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Baseline signal, tie-limited — agent.build-perf (claude-sonnet-5)

Why: Net win -20.0% (1W/2T/2L over 5 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -8.0% across 5 paired run(s) — no improvement

Next action: Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 1W/2T/2L; d=3; p=0.500; net -20.0%

Repeated-run reliability (not used by the gate): 5 paired runs (1W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Follow a host launch failure despite package discovery	Eligible	+0.0%	+0.0%	0/1/0
▼ Investigate the reader connection before optimizing a build	Eligible	-100.0%	-40.0%	0/0/1
= Keep an unresponsive launch inconclusive	Eligible	+0.0%	+0.0%	0/1/0
▼ Resume measured analysis after host tool discovery	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Follow a host launch failure despite package discovery: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — agent.build-perf (gpt-5.6-luna)

Why: Net win +40.0% (3W/1T/1L over 5 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +28.0% across 5 paired run(s) — not credible — 1 of 5 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the agent is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 3W/1T/1L; d=4; p=0.312; net +40.0%

Repeated-run reliability (not used by the gate): 5 paired runs (3W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Continue performance analysis when reader repair is declined	Eligible	+0.0%	+0.0%	0/1/0
▼ Investigate the reader connection before optimizing a build	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Continue performance analysis when reader repair is declined: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — agent.msbuild (claude-sonnet-5)

Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +40.0% across 6 paired run(s) — not credible (sign test p=0.188 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%

Repeated-run reliability (not used by the gate): 6 paired runs (4W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose broken incremental build behavior	Eligible	+0.0%	+0.0%	0/1/0
▼ Route a slow build to performance analysis	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Diagnose broken incremental build behavior: Position-swap inconsistent (forward: baseline, reverse: skill). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Baseline signal, tie-limited — agent.msbuild (gpt-5.6-luna)

Why: Net win -33.3% (1W/2T/3L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference -23.3% across 6 paired run(s) — no improvement

Next action: Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/2T/3L; d=4; p=0.312; net -33.3%

Repeated-run reliability (not used by the gate): 6 paired runs (1W/2T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Advise on project file organization	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose broken incremental build behavior	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose the binlog connection before changing the project	Eligible	+0.0%	+0.0%	0/1/0
▼ Route a slow build to performance analysis	Eligible	-100.0%	-40.0%	0/0/1
▼ Triage a build failure and route to appropriate analysis	Eligible	-100.0%	-100.0%	0/0/1

Illustrative judge evidence:

• Advise on project file organization: While Response B provides superior explanation of how Directory.Build.props works (a key rubric criterion), Response A delivers better technical quality. Response A correctly avoids centralizing OutputType, which is architecturally sound for .NET projects where different proje...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — troubleshoot-binlog-mcp (claude-sonnet-5)

Why: Net win +57.1% (4W/3T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +12.0% across 10 paired run(s), 3 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%; 3 dormancy excluded

Warnings: Activation: isolated 7/7; plugin 6/7; Activation-only stop: isolated 2 failed runs; Activation-only stop: plugin 3 failed runs

Overfit: Moderate (score 0.40)

Repeated-run reliability (not used by the gate): 10 paired runs (4W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Do not apply binlog-specific diagnosis to another server	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Explain a package resolution failure behind a closed connection	Eligible	+0.0%	+0.0%	0/1/0
= Follow a launch mapping error after successful search	Eligible	+0.0%	+0.0%	0/1/0
= Leave binlog capture to build logging	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Leave recorded build errors to build diagnosis	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▲ Preserve a connected cached server despite empty search	Eligible	+100.0%	+40.0%	1/0/0
= Recognize partial search errors despite exit zero	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Explain a package resolution failure behind a closed connection: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — troubleshoot-binlog-mcp (gpt-5.6-luna)

Why: Net win +0.0% (0W/7T/0L over 7 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 10 paired run(s), 3 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 0W/7T/0L; d=0; p=1.000; net +0.0%; 3 dormancy excluded

Warnings: Activation: isolated 7/7; plugin 6/7; Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 2 failed runs

Overfit: Moderate (score 0.48)

Repeated-run reliability (not used by the gate): 10 paired runs (0W/10T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose a closed connection by executing the registered launcher	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose the SDK selected by the workspace instead of the installed SDK	Eligible	+0.0%	+0.0%	0/1/0
= Do not apply binlog-specific diagnosis to another server	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Explain a package resolution failure behind a closed connection	Eligible	+0.0%	+0.0%	0/1/0
= Follow a launch mapping error after successful search	Eligible	+0.0%	+0.0%	0/1/0
= Keep a timed-out availability check inconclusive	Eligible	+0.0%	+0.0%	0/1/0
= Leave binlog capture to build logging	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Leave recorded build errors to build diagnosis	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Preserve a connected cached server despite empty search	Eligible	+0.0%	+0.0%	0/1/0
= Recognize partial search errors despite exit zero	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Diagnose a closed connection by executing the registered launcher: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1337 in dotnet/skills, download eval artifacts with gh run download 39679145374 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/664894e3c119e3dd011028be54bc4403cd4347bf/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.