[AbhitejJohn]

Summary

Generalize required-arm evaluation recovery to recognize the Copilot SDK 1.0.26 Session disconnected while waiting for a response error as a bounded transient executor failure.

Why: The SDK upgrade on trusted main changed the failure contract, so the existing timeout-only classifier planned zero retries for affected official evaluations.

Impact: Baseline and skilled failures with the exact disconnect message can recover without rerunning or replacing successful slots. Retry group and attempt limits, stable shard matching, audit provenance, plugin-arm exclusion, and strict invalidation of persistent or unrecognized failures remain unchanged. The broader No response was returned. Send your message again to retry. message remains ineligible because package evidence does not tie it to the same bounded transport failure class. This fix is required before rerunning #1351.

Related issue

N/A. Required before rerunning #1351.

Validation

• node --test eng\vally-adapter\retry-executor-timeouts.test.mjs — passed, 14 tests.
• node --test eng\vally-adapter\*.test.mjs eng\dashboard\*.test.js — passed, 235 tests.
• node --test eng\evaluation-tools\*.test.mjs — passed, 15 tests.
• python eng\evaluation\test_token_failover.py — passed, 51 tests.
• python eng\evaluation\test_workflow_targets.py — passed, 7 tests.
• python eng\evaluation\test_pr_triage_retry.py — passed, 9 tests.
• node eng\evaluation-tools\lint-corpus.mjs tests — passed with existing corpus warnings only.
• actionlint 1.7.7 -shellcheck= -pyflakes= .github/workflows/evaluation-run.yml — passed.
• git diff --check — passed.

Checklist

• I searched existing issues and pull requests to avoid duplicates.
• I kept this pull request focused and avoided unrelated refactors.
• I added or updated tests, evals, or documentation when changing skill or agent behavior.
• I updated CODEOWNERS when adding or moving owned content.
• I updated all marketplace manifests when plugin metadata changed.
• I updated eng/known-domains.txt for any new external domains referenced by skill content.

Evaluation changes only

For every eval-related change, including a new eval:

• Scenarios are necessary, distinct, and use natural prompts.
• Graders cover the full result, accept the golden result, and reject a realistic mutation.
• No-op, dormancy, and statistical power are covered where needed.
• I ran the applicable production evaluation path and recorded the result above.
• If this fixes a failed eval, I classified the failure before editing skill content.
• If this broadly changes routing or behavior, I checked separate model-family evidence.

[Copilot]

🟢 Approval recommended

No active review issues are recorded for the change extending bounded recovery to exact Copilot SDK session-disconnect errors.

No open issues 🎉

What changed in this PR

Extends the evaluation pipeline’s bounded recovery to handle the exact Copilot SDK session-disconnect error, addressing a blocker for rerunning #1351 while preserving successful results and existing retry limits.

Changes:

• Generalizes timeout classification to recognize specific transient executor failures while excluding the broader “No response was returned” error.
• Updates tests to cover disconnect recovery, successful-slot preservation, and retry provenance.
• Aligns recovery documentation, workflow comments, and warning text with the expanded classification.

File	Description
eng/​vally-adapter/​retry-executor-timeouts.test.mjs	Adds disconnect classification and recovery coverage.
eng/​vally-adapter/​retry-executor-timeouts.mjs	Recognizes exact timeout and session-disconnect failures throughout recovery.
eng/​vally-adapter/​README.md	Documents eligible errors and bounded recovery behavior.
eng/​vally-adapter/​InvestigatingResults.md	Updates investigation guidance for disconnects and excluded errors.
.github/​workflows/​evaluation-run.yml	Updates recovery comments and warning wording without changing workflow control flow.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues block the change extending bounded recovery to Copilot SDK session disconnects.

No open issues 🎉

_{Review effort: Balanced}

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 9be2ecff951d77814ebdc6ce16e1d73d6339ffbb to retry this exact commit.

14 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[Copilot]

🟢 Approval recommended

No active review issues remain for the change extending bounded recovery to recognized Copilot SDK disconnects.

No open issues 🎉

_{Review effort: Balanced}

[github-actions]

📊 Skill, Agent, and Workflow Evaluation Results

Workflow results measure offline prompt decisions and proposed outputs, not live Actions jobs or publication.

28 model/target results across 14 targets and 2 models — ✅ 0 improved, ➖ 14 results without a clear winner, ⚠️ 6 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only), 🔻 8 objective regressions.

Measurement identity: evaluated commit cc27c9daf60fcf163190156fa97a9083b1ff5a3a; 2 judge models.

Measurement health: 28 expected / 28 observed / 28 written; 0 missing, 0 unexpected, 6 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: enabled; 8 results show a proven objective regression.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
build-failure-analysis	claude-sonnet-5	🔻 Objective regression	n=10; 5W/3T/2L; d=7; p=0.227; net +30.0%	—	—	Inspect objective completion losses and fix them before merge.
build-failure-analysis	gpt-5.6-luna	🔻 Objective regression	n=10; 3W/2T/5L; d=8; p=0.363; net -20.0%	—	—	Inspect objective completion losses and fix them before merge.
dotnet-maui-doctor	claude-sonnet-5	➖ Improvement signal, unproven	n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%	🔴 0.51	Activation: isolated 8/8; plugin 7/8	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
dotnet-maui-doctor	gpt-5.6-luna	➖ No preference	n=8; 0W/8T/0L; d=0; p=1.000; net +0.0%	🟡 0.31	—	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
maui-app-lifecycle	claude-sonnet-5	⚠️ Underpowered	n=4; 3W/0T/1L; d=4; p=0.312; net +50.0%	🔴 0.65	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-app-lifecycle	gpt-5.6-luna	⚠️ Underpowered	n=4; 0W/4T/0L; d=0; p=1.000; net +0.0%	🟡 0.45	Activation-only stop: plugin 1 failed run	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-collectionview	claude-sonnet-5	➖ No preference	n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%	🟡 0.42	Activation-only stop: plugin 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
maui-collectionview	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=5; 1W/4T/0L; d=1; p=0.500; net +20.0%	🟡 0.20	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
maui-data-binding	claude-sonnet-5	⚠️ Underpowered	n=4; 2W/2T/0L; d=2; p=0.250; net +50.0%	🟡 0.41	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-data-binding	gpt-5.6-luna	⚠️ Underpowered	n=4; 0W/4T/0L; d=0; p=1.000; net +0.0%	🟡 0.39	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-dependency-injection	claude-sonnet-5	➖ Improvement signal, tie-limited	n=5; 2W/3T/0L; d=2; p=0.250; net +40.0%	🔴 0.52	Activation-only stop: isolated 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
maui-dependency-injection	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=5; 1W/4T/0L; d=1; p=0.500; net +20.0%	🟡 0.44	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
maui-safe-area	claude-sonnet-5	⚠️ Underpowered	n=4; 2W/2T/0L; d=2; p=0.250; net +50.0%	🟡 0.36	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-safe-area	gpt-5.6-luna	⚠️ Underpowered	n=4; 0W/4T/0L; d=0; p=1.000; net +0.0%	🔴 0.51	Activation-only stop: isolated 1 failed run	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-shell-navigation	claude-sonnet-5	➖ Baseline signal, tie-limited	n=5; 0W/4T/1L; d=1; p=0.500; net -20.0%	🔴 0.58	—	Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
maui-shell-navigation	gpt-5.6-luna	➖ No preference	n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%	🟡 0.28	—	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
maui-theming	claude-sonnet-5	➖ Improvement signal, tie-limited	n=5; 2W/3T/0L; d=2; p=0.250; net +40.0%	🔴 0.70	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
maui-theming	gpt-5.6-luna	➖ No preference	n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%	🔴 0.53	—	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
msbuild-quality-review	claude-sonnet-5	🔻 Objective regression	n=10; 7W/3T/0L; d=7; p=0.008; net +70.0%	—	—	Inspect objective completion losses and fix them before merge.
msbuild-quality-review	gpt-5.6-luna	🔻 Objective regression	n=10; 4W/5T/1L; d=5; p=0.188; net +30.0%	—	—	Inspect objective completion losses and fix them before merge.
process-api-net11	claude-sonnet-5	➖ Improvement signal, tie-limited	n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 2 dormancy excluded	🟡 0.44	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
process-api-net11	gpt-5.6-luna	➖ No preference	n=6; 0W/6T/0L; d=0; p=1.000; net +0.0%; 2 dormancy excluded	🔴 0.57	Activation-only stop: isolated 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
system-text-json-net11	claude-sonnet-5	➖ Improvement signal, tie-limited	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 2 dormancy excluded	🟡 0.50	Activation: isolated 6/6; plugin 4/6	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
system-text-json-net11	gpt-5.6-luna	➖ No preference	n=6; 0W/6T/0L; d=0; p=1.000; net +0.0%; 2 dormancy excluded	🟡 0.35	Activation: isolated 6/6; plugin 5/6	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
test-failure-analysis	claude-sonnet-5	🔻 Objective regression	n=12; 12W/0T/0L; d=12; p=0.000; net +100.0%	—	—	Inspect objective completion losses and fix them before merge.
test-failure-analysis	gpt-5.6-luna	🔻 Objective regression	n=12; 8W/3T/1L; d=9; p=0.020; net +58.3%	—	—	Inspect objective completion losses and fix them before merge.
unskip-closed-tests	claude-sonnet-5	🔻 Objective regression	n=11; 3W/7T/1L; d=4; p=0.312; net +18.2%	—	—	Inspect objective completion losses and fix them before merge.
unskip-closed-tests	gpt-5.6-luna	🔻 Objective regression	n=11; 4W/2T/5L; d=9; p=0.500; net -9.1%	—	—	Inspect objective completion losses and fix them before merge.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

🔻 Objective regression — build-failure-analysis (claude-sonnet-5)

Why: Net win +30.0% (5W/3T/2L over 10 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +6.0% across 10 paired run(s) — not credible (sign test p=0.227 > 0.05) — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 5W/3T/2L; d=7; p=0.227; net +30.0%

Repeated-run reliability (not used by the gate): 10 paired runs (5W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Bound a restore diagnosis to the searched feed	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a silent target-process failure	Eligible	+0.0%	+0.0%	0/1/0
= Recognize a failed pipeline with a clean build	Eligible	+0.0%	+0.0%	0/1/0
▼ Report the absence of all binary logs without a diagnosis	Eligible	-100.0%	-100.0%	0/0/1
▼ Suppress obsolete analysis after a head update	Eligible	-100.0%	-100.0%	0/0/1

Illustrative judge evidence:

• Bound a restore diagnosis to the searched feed: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — build-failure-analysis (gpt-5.6-luna)

Why: Net win -20.0% (3W/2T/5L over 10 preference-eligible stimulus vote(s), sign test p=0.363), mean preference -8.0% across 10 paired run(s) — no improvement — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 3W/2T/5L; d=8; p=0.363; net -20.0%

Repeated-run reliability (not used by the gate): 10 paired runs (3W/2T/5L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Preserve a silent target-process failure	Eligible	-100.0%	-40.0%	0/0/1
▼ Recognize a failed pipeline with a clean build	Eligible	-100.0%	-40.0%	0/0/1
▼ Report the absence of all binary logs without a diagnosis	Eligible	-100.0%	-100.0%	0/0/1
= Resolve a disposal warning promoted to an error	Eligible	+0.0%	+0.0%	0/1/0
= Retain useful findings while naming a missing module	Eligible	+0.0%	+0.0%	0/1/0
▼ Suppress obsolete analysis after a head update	Eligible	-100.0%	-40.0%	0/0/1
▼ Withhold conclusions when the failed leg has no log	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Preserve a silent target-process failure: Both responses successfully complete the task by writing evidence-backed comments to result.json, correctly identifying the exit 137 failure as genuine despite no compiler errors, and appropriately noting that the underlying cause is unproven. The key difference lies in how st...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — msbuild-quality-review (claude-sonnet-5)

Why: Net win +70.0% (7W/3T/0L over 10 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +34.0% across 10 paired run(s) — credibly better — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 7W/3T/0L; d=7; p=0.008; net +70.0%

Repeated-run reliability (not used by the gate): 10 paired runs (7W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid findings when the changed file cannot be read	Eligible	+0.0%	+0.0%	0/1/0
= Catch a property evaluated before its discriminator exists	Eligible	+0.0%	+0.0%	0/1/0
= Preserve explicit FSharp compilation order	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid findings when the changed file cannot be read: Both results are evidence-grounded, correctly produce a supported noop decision, and document the missing diff/content gap without speculation. Response A is somewhat more detailed, while Response B is more concise; neither has a material quality advantage.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — msbuild-quality-review (gpt-5.6-luna)

Why: Net win +30.0% (4W/5T/1L over 10 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +12.0% across 10 paired run(s) — not credible (sign test p=0.188 > 0.05) — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 4W/5T/1L; d=5; p=0.188; net +30.0%

Repeated-run reliability (not used by the gate): 10 paired runs (4W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Accept a valid required import in the packed layout	Eligible	+0.0%	+0.0%	0/1/0
= Avoid findings when the changed file cannot be read	Eligible	+0.0%	+0.0%	0/1/0
= Catch a property evaluated before its discriminator exists	Eligible	+0.0%	+0.0%	0/1/0
▼ Filter intentionally broken fixture changes	Eligible	-100.0%	-100.0%	0/0/1
= Preserve explicit FSharp compilation order	Eligible	+0.0%	+0.0%	0/1/0
= Preserve the caller extension chain	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Accept a valid required import in the packed layout: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — test-failure-analysis (claude-sonnet-5)

Why: Net win +100.0% (12W/0T/0L over 12 preference-eligible stimulus vote(s), sign test p=0.000), mean preference +65.0% across 12 paired run(s) — credibly better — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=12; 12W/0T/0L; d=12; p=0.000; net +100.0%

Repeated-run reliability (not used by the gate): 12 paired runs (12W/0T/0L).

🔻 Objective regression — test-failure-analysis (gpt-5.6-luna)

Why: Net win +58.3% (8W/3T/1L over 12 preference-eligible stimulus vote(s), sign test p=0.020), mean preference +33.3% across 12 paired run(s) — credibly better — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=12; 8W/3T/1L; d=9; p=0.020; net +58.3%

Repeated-run reliability (not used by the gate): 12 paired runs (8W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Distinguish recovered retries from current failures	Eligible	+0.0%	+0.0%	0/1/0
▼ Group matching terminal failures across modules	Eligible	-100.0%	-100.0%	0/0/1
= Prioritize explicit watchdog evidence over downstream failure text	Eligible	+0.0%	+0.0%	0/1/0
= Reject findings for an obsolete tested merge	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Distinguish recovered retries from current failures: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — unskip-closed-tests (claude-sonnet-5)

Why: Net win +18.2% (3W/7T/1L over 11 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +7.3% across 11 paired run(s) — not credible — 7 of 11 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the agent is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=11; 3W/7T/1L; d=4; p=0.312; net +18.2%

Repeated-run reliability (not used by the gate): 11 paired runs (3W/7T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Defer a completed issue unrelated to the ignored test	Eligible	+0.0%	+0.0%	0/1/0
= Defer an incomplete partial class with nested tests	Eligible	+0.0%	+0.0%	0/1/0
= Do not treat a zero-result verification exit as authorization	Eligible	+0.0%	+0.0%	0/1/0
= Preserve distinct candidates across modules with identical attributes	Eligible	+0.0%	+0.0%	0/1/0
= Reject a candidate bound to the wrong syntax owner	Eligible	+0.0%	+0.0%	0/1/0
▼ Reject a legacy manifest without source-bound identities	Eligible	-100.0%	-40.0%	0/0/1
= Reject a manifest for an obsolete source commit	Eligible	+0.0%	+0.0%	0/1/0
= Select a class site with complete direct-test enumeration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Defer a completed issue unrelated to the ignored test: Position-swap inconsistent (forward: tie, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — unskip-closed-tests (gpt-5.6-luna)

Why: Net win -9.1% (4W/2T/5L over 11 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -3.6% across 11 paired run(s) — no improvement — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=11; 4W/2T/5L; d=9; p=0.500; net -9.1%

Repeated-run reliability (not used by the gate): 11 paired runs (4W/2T/5L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Defer an incomplete partial class with nested tests	Eligible	-100.0%	-40.0%	0/0/1
▼ Reject a candidate bound to the wrong syntax owner	Eligible	-100.0%	-40.0%	0/0/1
▼ Reject a legacy manifest without source-bound identities	Eligible	-100.0%	-40.0%	0/0/1
▼ Reject a manifest for an obsolete source commit	Eligible	-100.0%	-40.0%	0/0/1
= Select a class site with complete direct-test enumeration	Eligible	+0.0%	+0.0%	0/1/0
= Select a method whose tracking pull request is merged	Eligible	+0.0%	+0.0%	0/1/0
▼ Select a source-bound method with a completed issue	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Defer an incomplete partial class with nested tests: Both responses arrive at the correct decision (noop) with sound reasoning and proper adherence to constraints. Response A provides marginally better quality by: (1) including meaningful content in the 'limitations' field to document the evidence gap, and (2) offering more gran...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-app-lifecycle (claude-sonnet-5)

Why: Net win +50.0% (3W/0T/1L over 4 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +35.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 3W/0T/1L; d=4; p=0.312; net +50.0%

Overfit: High (score 0.65)

Repeated-run reliability (not used by the gate): 4 paired runs (3W/0T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Platform-specific lifecycle mapping	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Platform-specific lifecycle mapping: Both provide essentially functional native lifecycle hook code, but A is more complete and precise: it distinguishes pause versus fully backgrounded behavior, provides an accurate Window.Stopped mapping, and has clearer platform conditional setup. B is concise and mostly corre...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-app-lifecycle (gpt-5.6-luna)

Why: Net win +0.0% (0W/4T/0L over 4 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 0W/4T/0L; d=0; p=1.000; net +0.0%

Warnings: Activation-only stop: plugin 1 failed run

Overfit: Moderate (score 0.45)

Repeated-run reliability (not used by the gate): 4 paired runs (0W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid legacy Xamarin.Forms lifecycle methods	Eligible	+0.0%	+0.0%	0/1/0
= Platform-specific lifecycle mapping	Eligible	+0.0%	+0.0%	0/1/0
= Save and restore state on background	Eligible	+0.0%	+0.0%	0/1/0
= Window lifecycle event subscription	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid legacy Xamarin.Forms lifecycle methods: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-data-binding (claude-sonnet-5)

Why: Net win +50.0% (2W/2T/0L over 4 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +20.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 2W/2T/0L; d=2; p=0.250; net +50.0%

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 4 paired runs (2W/2T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Create and use an IValueConverter	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose missing BindingContext and non-compiled bindings	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Create and use an IValueConverter: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-data-binding (gpt-5.6-luna)

Why: Net win +0.0% (0W/4T/0L over 4 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 0W/4T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.39)

Repeated-run reliability (not used by the gate): 4 paired runs (0W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Create and use an IValueConverter	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose missing BindingContext and non-compiled bindings	Eligible	+0.0%	+0.0%	0/1/0
= Implement MVVM ViewModel with ObservableObject	Eligible	+0.0%	+0.0%	0/1/0
= Set up compiled bindings with x:DataType on a page	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Create and use an IValueConverter: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-safe-area (claude-sonnet-5)

Why: Net win +50.0% (2W/2T/0L over 4 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +35.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 2W/2T/0L; d=2; p=0.250; net +50.0%

Overfit: Moderate (score 0.36)

Repeated-run reliability (not used by the gate): 4 paired runs (2W/2T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Edge-to-edge layout with SafeAreaEdges	Eligible	+0.0%	+0.0%	0/1/0
= Keyboard avoidance with safe area for chat UI	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Edge-to-edge layout with SafeAreaEdges: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-safe-area (gpt-5.6-luna)

Why: Net win +0.0% (0W/4T/0L over 4 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 0W/4T/0L; d=0; p=1.000; net +0.0%

Warnings: Activation-only stop: isolated 1 failed run

Overfit: High (score 0.51)

Repeated-run reliability (not used by the gate): 4 paired runs (0W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid deprecated UseSafeArea in new .NET 10 project	Eligible	+0.0%	+0.0%	0/1/0
= Edge-to-edge layout with SafeAreaEdges	Eligible	+0.0%	+0.0%	0/1/0
= Handle notch and status bar safe areas on iOS	Eligible	+0.0%	+0.0%	0/1/0
= Keyboard avoidance with safe area for chat UI	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid deprecated UseSafeArea in new .NET 10 project: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — dotnet-maui-doctor (claude-sonnet-5)

Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +62.5% across 8 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%

Warnings: Activation: isolated 8/8; plugin 7/8

Overfit: High (score 0.51)

Repeated-run reliability (not used by the gate): 8 paired runs (6W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Determine required Android SDK packages for specific .NET version	Eligible	+0.0%	+0.0%	0/1/0
▲ Guardrail against workload update and repair	Eligible	+100.0%	+100.0%	1/0/0
▼ Plan complete MAUI setup on Windows	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Determine required Android SDK packages for specific .NET version: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — dotnet-maui-doctor (gpt-5.6-luna)

Why: Net win +0.0% (0W/8T/0L over 8 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 8 paired run(s) — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 0W/8T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.31)

Repeated-run reliability (not used by the gate): 8 paired runs (0W/8T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Determine required Android SDK packages for specific .NET version	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose non-Microsoft JDK causing build failure	Eligible	+0.0%	+0.0%	0/1/0
= Fix stale MAUI workloads after SDK update	Eligible	+0.0%	+0.0%	0/1/0
= Guardrail against workload update and repair	Eligible	+0.0%	+0.0%	0/1/0
= Plan Linux MAUI environment for Android	Eligible	+0.0%	+0.0%	0/1/0
= Plan complete MAUI setup on Windows	Eligible	+0.0%	+0.0%	0/1/0
= Plan macOS MAUI setup with Xcode	Eligible	+0.0%	+0.0%	0/1/0
= Prevent incorrect JAVA_HOME configuration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Determine required Android SDK packages for specific .NET version: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — maui-collectionview (claude-sonnet-5)

Why: Net win +0.0% (0W/5T/0L over 5 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 5 paired run(s) — no improvement

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%

Warnings: Activation-only stop: plugin 1 failed run

Overfit: Moderate (score 0.42)

Repeated-run reliability (not used by the gate): 5 paired runs (0W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid ListView and ViewCell mistakes	Eligible	+0.0%	+0.0%	0/1/0
= Basic CollectionView with data binding and DataTemplate	Eligible	+0.0%	+0.0%	0/1/0
= Grid layout with CollectionView	Eligible	+0.0%	+0.0%	0/1/0
= ItemSizingStrategy placement for uniform items	Eligible	+0.0%	+0.0%	0/1/0
= Selection and pull-to-refresh with CollectionView	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid ListView and ViewCell mistakes: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — maui-collectionview (gpt-5.6-luna)

Why: Net win +20.0% (1W/4T/0L over 5 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +8.0% across 5 paired run(s) — not credible — 4 of 5 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 1W/4T/0L; d=1; p=0.500; net +20.0%

Overfit: Moderate (score 0.20)

Repeated-run reliability (not used by the gate): 5 paired runs (1W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid ListView and ViewCell mistakes	Eligible	+0.0%	+0.0%	0/1/0
= Basic CollectionView with data binding and DataTemplate	Eligible	+0.0%	+0.0%	0/1/0
= Grid layout with CollectionView	Eligible	+0.0%	+0.0%	0/1/0
= Selection and pull-to-refresh with CollectionView	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid ListView and ViewCell mistakes: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — maui-dependency-injection (claude-sonnet-5)

Why: Net win +40.0% (2W/3T/0L over 5 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +16.0% across 5 paired run(s) — not credible — 3 of 5 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 2W/3T/0L; d=2; p=0.250; net +40.0%

Warnings: Activation-only stop: isolated 1 failed run

Overfit: High (score 0.52)

Repeated-run reliability (not used by the gate): 5 paired runs (2W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid AddScoped pitfall in MAUI	Eligible	+0.0%	+0.0%	0/1/0
= Register services with correct lifetimes in MauiProgram.cs	Eligible	+0.0%	+0.0%	0/1/0
= Shell navigation auto-resolves DI-registered pages	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid AddScoped pitfall in MAUI: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — maui-dependency-injection (gpt-5.6-luna)

Why: Net win +20.0% (1W/4T/0L over 5 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +8.0% across 5 paired run(s) — not credible — 4 of 5 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 1W/4T/0L; d=1; p=0.500; net +20.0%

Overfit: Moderate (score 0.44)

Repeated-run reliability (not used by the gate): 5 paired runs (1W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose a page whose injected dependencies are missing	Eligible	+0.0%	+0.0%	0/1/0
= Platform-specific service registration with fallback	Eligible	+0.0%	+0.0%	0/1/0
= Register services with correct lifetimes in MauiProgram.cs	Eligible	+0.0%	+0.0%	0/1/0
= Shell navigation auto-resolves DI-registered pages	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Diagnose a page whose injected dependencies are missing: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Baseline signal, tie-limited — maui-shell-navigation (claude-sonnet-5)

Why: Net win -20.0% (0W/4T/1L over 5 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -8.0% across 5 paired run(s) — no improvement

Next action: Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 0W/4T/1L; d=1; p=0.500; net -20.0%

Overfit: High (score 0.58)

Repeated-run reliability (not used by the gate): 5 paired runs (0W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose common Shell navigation mistakes	Eligible	+0.0%	+0.0%	0/1/0
= Handle back navigation and unsaved changes guard	Eligible	+0.0%	+0.0%	0/1/0
= Navigate with parameters using GoToAsync	Eligible	+0.0%	+0.0%	0/1/0
▼ Set up Shell navigation with tabs and flyout	Eligible	-100.0%	-40.0%	0/0/1
= Stable routes for deep linking into tabs	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Diagnose common Shell navigation mistakes: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — maui-shell-navigation (gpt-5.6-luna)

Why: Net win +0.0% (0W/5T/0L over 5 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 5 paired run(s) — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.28)

Repeated-run reliability (not used by the gate): 5 paired runs (0W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose common Shell navigation mistakes	Eligible	+0.0%	+0.0%	0/1/0
= Handle back navigation and unsaved changes guard	Eligible	+0.0%	+0.0%	0/1/0
= Navigate with parameters using GoToAsync	Eligible	+0.0%	+0.0%	0/1/0
= Set up Shell navigation with tabs and flyout	Eligible	+0.0%	+0.0%	0/1/0
= Stable routes for deep linking into tabs	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Diagnose common Shell navigation mistakes: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — maui-theming (claude-sonnet-5)

Why: Net win +40.0% (2W/3T/0L over 5 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +16.0% across 5 paired run(s) — not credible — 3 of 5 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 2W/3T/0L; d=2; p=0.250; net +40.0%

Overfit: High (score 0.70)

Repeated-run reliability (not used by the gate): 5 paired runs (2W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add light/dark mode support using AppThemeBinding	Eligible	+0.0%	+0.0%	0/1/0
= Create custom themes with ResourceDictionary switching	Eligible	+0.0%	+0.0%	0/1/0
= Detect and respond to system theme changes	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add light/dark mode support using AppThemeBinding: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — maui-theming (gpt-5.6-luna)

Why: Net win +0.0% (0W/5T/0L over 5 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 5 paired run(s) — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%

Overfit: High (score 0.53)

Repeated-run reliability (not used by the gate): 5 paired runs (0W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add light/dark mode support using AppThemeBinding	Eligible	+0.0%	+0.0%	0/1/0
= Avoid common theming mistakes	Eligible	+0.0%	+0.0%	0/1/0
= Create custom themes with ResourceDictionary switching	Eligible	+0.0%	+0.0%	0/1/0
= Detect and respond to system theme changes	Eligible	+0.0%	+0.0%	0/1/0
= Swap theme dictionaries without destroying app styles	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add light/dark mode support using AppThemeBinding: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — process-api-net11 (claude-sonnet-5)

Why: Net win +33.3% (2W/4T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +25.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 6 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 2 dormancy excluded

Overfit: Moderate (score 0.44)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/6T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Deadlock-free full-output read via instance ReadAllText in .NET 11	Eligible	+0.0%	+0.0%	0/1/0
= Fire-and-forget process start in .NET 11	Eligible	+0.0%	+0.0%	0/1/0
= Non-activation: Running a process on .NET 8	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Non-activation: capture process output on .NET 10	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Stream tagged process output lines in .NET 11	Eligible	+0.0%	+0.0%	0/1/0
= Whitelist inherited handles for child process in .NET 11	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Deadlock-free full-output read via instance ReadAllText in .NET 11: Both provide a valid minimal .NET 11 program that configures redirection before Process.Start, awaits the first-party tuple-returning API, and includes a correct net11.0 project file. A is somewhat more explicit about UseShellExecute = false; B is more concise, and neither...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — process-api-net11 (gpt-5.6-luna)

Why: Net win +0.0% (0W/6T/0L over 6 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 0W/6T/0L; d=0; p=1.000; net +0.0%; 2 dormancy excluded

Warnings: Activation-only stop: isolated 1 failed run

Overfit: High (score 0.57)

Repeated-run reliability (not used by the gate): 8 paired runs (0W/8T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Auto-teardown child processes on parent exit in .NET 11	Eligible	+0.0%	+0.0%	0/1/0
= Deadlock-free full-output read via instance ReadAllText in .NET 11	Eligible	+0.0%	+0.0%	0/1/0
= Fire-and-forget process start in .NET 11	Eligible	+0.0%	+0.0%	0/1/0
= Non-activation: Running a process on .NET 8	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Non-activation: capture process output on .NET 10	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Run and capture process output in .NET 11	Eligible	+0.0%	+0.0%	0/1/0
= Stream tagged process output lines in .NET 11	Eligible	+0.0%	+0.0%	0/1/0
= Whitelist inherited handles for child process in .NET 11	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Auto-teardown child processes on parent exit in .NET 11: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — system-text-json-net11 (claude-sonnet-5)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +32.5% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 2 dormancy excluded

Warnings: Activation: isolated 6/6; plugin 4/6

Overfit: Moderate (score 0.50)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Apply PascalCase to dictionary keys on .NET 11	Eligible	+100.0%	+100.0%	1/0/0
▲ Deserialize PascalCase JSON into lowercase members on .NET 11	Eligible	+100.0%	+100.0%	1/0/0
= Non-activation: Newtonsoft.Json PascalCase contract resolver	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Non-activation: camelCase JSON serialization on .NET 8	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Serialize JSON in .NET 11 with PascalCase property names	Eligible	+0.0%	+0.0%	0/1/0
= Serialize with strongly typed metadata without casting	Eligible	+0.0%	+0.0%	0/1/0
▼ Type-safe JsonTypeInfo access without exceptions in .NET 11	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Serialize JSON in .NET 11 with PascalCase property names: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — system-text-json-net11 (gpt-5.6-luna)

Why: Net win +0.0% (0W/6T/0L over 6 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 0W/6T/0L; d=0; p=1.000; net +0.0%; 2 dormancy excluded

Warnings: Activation: isolated 6/6; plugin 5/6

Overfit: Moderate (score 0.35)

Repeated-run reliability (not used by the gate): 8 paired runs (0W/8T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Apply PascalCase to dictionary keys on .NET 11	Eligible	+0.0%	+0.0%	0/1/0
= Deserialize PascalCase JSON into lowercase members on .NET 11	Eligible	+0.0%	+0.0%	0/1/0
= Non-activation: Newtonsoft.Json PascalCase contract resolver	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Non-activation: camelCase JSON serialization on .NET 8	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Probe source-generated metadata without exception control flow	Eligible	+0.0%	+0.0%	0/1/0
= Serialize JSON in .NET 11 with PascalCase property names	Eligible	+0.0%	+0.0%	0/1/0
= Serialize with strongly typed metadata without casting	Eligible	+0.0%	+0.0%	0/1/0
= Type-safe JsonTypeInfo access without exceptions in .NET 11	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Apply PascalCase to dictionary keys on .NET 11: Position-swap inconsistent (forward: A, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1369 in dotnet/skills, download eval artifacts with gh run download 41754714768 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/cc27c9daf60fcf163190156fa97a9083b1ff5a3a/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[AbhitejJohn]

Exact-head evaluation passed for $expected: https://github.com/dotnet/skills/actions/runs/41754714768

@Evangelink @YuliiaKovalova @y87feng, please review and approve when ready.

[Copilot]

🟢 Approval recommended

No active review issues are reported for the per-attempt Copilot authentication isolation change.

No open issues 🎉

1 resolved since last review

• Preserve existing GH_TOKEN when applying the token fallback

_{Review effort: Balanced}

[github-actions]

📊 Skill, Agent, and Workflow Evaluation Results

Workflow results measure offline prompt decisions and proposed outputs, not live Actions jobs or publication.

20 model/target results across 10 targets and 2 models — ✅ 2 improved, ➖ 8 results without a clear winner, ⚠️ 6 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only), 🔻 4 objective regressions.

Measurement identity: evaluated commit 9f4b279a8d92342e41e21ca3fb56f949cd594c40; 2 judge models.

Measurement health: 20 expected / 20 observed / 20 written; 0 missing, 0 unexpected, 6 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: enabled; 4 results show a proven objective regression.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
build-failure-analysis	claude-sonnet-5	🔻 Objective regression	n=10; 5W/1T/4L; d=9; p=0.500; net +10.0%	—	—	Inspect objective completion losses and fix them before merge.
build-failure-analysis	gpt-5.6-luna	🔻 Objective regression	n=10; 4W/3T/3L; d=7; p=0.500; net +10.0%	—	—	Inspect objective completion losses and fix them before merge.
create-datadriven-aspnetcore	claude-sonnet-5	➖ Improvement signal, tie-limited	n=5; 2W/3T/0L; d=2; p=0.250; net +40.0%	🟡 0.45	Activation: isolated 5/5; plugin 4/5	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
create-datadriven-aspnetcore	gpt-5.6-luna	➖ No preference	n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%	🟡 0.28	—	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
csharp-scripts	claude-sonnet-5	⚠️ Underpowered	n=1; 0W/1T/0L; d=0; p=1.000; net +0.0%	🟡 0.50	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
csharp-scripts	gpt-5.6-luna	⚠️ Underpowered	n=1; 0W/1T/0L; d=0; p=1.000; net +0.0%	🟡 0.40	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
dotnet-pinvoke	claude-sonnet-5	⚠️ Underpowered	n=2; 1W/1T/0L; d=1; p=0.500; net +50.0%	🟡 0.31	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
dotnet-pinvoke	gpt-5.6-luna	⚠️ Underpowered	n=2; 0W/2T/0L; d=0; p=1.000; net +0.0%	🟡 0.43	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
msbuild-quality-review	claude-sonnet-5	🔻 Objective regression	n=10; 6W/2T/2L; d=8; p=0.145; net +40.0%	—	—	Inspect objective completion losses and fix them before merge.
msbuild-quality-review	gpt-5.6-luna	➖ Improvement signal, unproven	n=10; 5W/4T/1L; d=6; p=0.109; net +40.0%	—	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
nuget-trusted-publishing	claude-sonnet-5	⚠️ Underpowered	n=3; 0W/2T/1L; d=1; p=0.500; net -33.3%	🟡 0.33	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
nuget-trusted-publishing	gpt-5.6-luna	⚠️ Underpowered	n=3; 0W/3T/0L; d=0; p=1.000; net +0.0%	🟡 0.40	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
optimizing-ef-core-queries	claude-sonnet-5	➖ Improvement signal, tie-limited	n=11; 3W/8T/0L; d=3; p=0.125; net +27.3%; 1 dormancy excluded	🟡 0.40	Activation: isolated 9/11; plugin 9/11	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
optimizing-ef-core-queries	gpt-5.6-luna	➖ Mixed evidence	n=11; 1W/9T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	—	—	Compare winning and losing scenarios to isolate where the target helps versus hurts.
test-failure-analysis	claude-sonnet-5	✅ Improved	n=12; 11W/1T/0L; d=11; p=0.000; net +91.7%	—	—	None.
test-failure-analysis	gpt-5.6-luna	✅ Improved	n=12; 8W/4T/0L; d=8; p=0.004; net +66.7%	—	—	None.
unskip-closed-tests	claude-sonnet-5	➖ Improvement signal, unproven	n=11; 5W/3T/3L; d=8; p=0.363; net +18.2%	—	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
unskip-closed-tests	gpt-5.6-luna	🔻 Objective regression	n=11; 3W/4T/4L; d=7; p=0.500; net -9.1%	—	—	Inspect objective completion losses and fix them before merge.
vectorization	claude-sonnet-5	➖ Improvement signal, tie-limited	n=9; 3W/5T/1L; d=4; p=0.312; net +22.2%; 1 dormancy excluded	🟡 0.42	Activation: isolated 8/9; plugin 4/9; Activation-only stop: isolated 3 failed runs; Activation-only stop: plugin 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
vectorization	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=9; 2W/7T/0L; d=2; p=0.250; net +22.2%; 1 dormancy excluded	🔴 0.60	Activation: isolated 7/9; plugin 5/9	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

🔻 Objective regression — build-failure-analysis (claude-sonnet-5)

Why: Net win +10.0% (5W/1T/4L over 10 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -2.0% across 10 paired run(s) — not credible (sign test p=0.500 > 0.05) — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 5W/1T/4L; d=9; p=0.500; net +10.0%

Repeated-run reliability (not used by the gate): 10 paired runs (5W/1T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Bound a restore diagnosis to the searched feed	Eligible	-100.0%	-40.0%	0/0/1
▼ Report the absence of all binary logs without a diagnosis	Eligible	-100.0%	-100.0%	0/0/1
= Resolve a disposal warning promoted to an error	Eligible	+0.0%	+0.0%	0/1/0
▼ Retain useful findings while naming a missing module	Eligible	-100.0%	-40.0%	0/0/1
▼ Suppress obsolete analysis after a head update	Eligible	-100.0%	-100.0%	0/0/1

Illustrative judge evidence:

• Bound a restore diagnosis to the searched feed: The outputs are substantively equivalent and correct. A is marginally stronger because its evidence list also cites the changed dependency file and its limitations state the offline-only scope more directly, improving traceability and epistemic precision.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — build-failure-analysis (gpt-5.6-luna)

Why: Net win +10.0% (4W/3T/3L over 10 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -8.0% across 10 paired run(s) — not credible (sign test p=0.500 > 0.05) — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 4W/3T/3L; d=7; p=0.500; net +10.0%

Repeated-run reliability (not used by the gate): 10 paired runs (4W/3T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Bound a restore diagnosis to the searched feed	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a silent target-process failure	Eligible	+0.0%	+0.0%	0/1/0
▼ Report the absence of all binary logs without a diagnosis	Eligible	-100.0%	-100.0%	0/0/1
▼ Suppress obsolete analysis after a head update	Eligible	-100.0%	-100.0%	0/0/1
▼ Suppress obsolete analysis after the merge revision moves	Eligible	-100.0%	-100.0%	0/0/1
= Withhold conclusions when the failed leg has no log	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Bound a restore diagnosis to the searched feed: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — msbuild-quality-review (claude-sonnet-5)

Why: Net win +40.0% (6W/2T/2L over 10 preference-eligible stimulus vote(s), sign test p=0.145), mean preference +28.0% across 10 paired run(s) — not credible (sign test p=0.145 > 0.05) — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 6W/2T/2L; d=8; p=0.145; net +40.0%

Repeated-run reliability (not used by the gate): 10 paired runs (6W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid duplicating default source items	Eligible	+0.0%	+0.0%	0/1/0
= Avoid findings when the changed file cannot be read	Eligible	+0.0%	+0.0%	0/1/0
▼ Catch a property evaluated before its discriminator exists	Eligible	-100.0%	-40.0%	0/0/1
▼ Identify a generated-file collision across projects and frameworks	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Avoid duplicating default source items: Position-swap inconsistent (forward: baseline, reverse: skill). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — unskip-closed-tests (gpt-5.6-luna)

Why: Net win -9.1% (3W/4T/4L over 11 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -3.6% across 11 paired run(s) — no improvement — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=11; 3W/4T/4L; d=7; p=0.500; net -9.1%

Repeated-run reliability (not used by the gate): 11 paired runs (3W/4T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Defer a completed issue unrelated to the ignored test	Eligible	+0.0%	+0.0%	0/1/0
= Defer an incomplete partial class with nested tests	Eligible	+0.0%	+0.0%	0/1/0
= Defer open and not-planned tracking items	Eligible	+0.0%	+0.0%	0/1/0
▼ Preserve distinct candidates across modules with identical attributes	Eligible	-100.0%	-40.0%	0/0/1
= Reject a legacy manifest without source-bound identities	Eligible	+0.0%	+0.0%	0/1/0
▼ Reject a manifest for an obsolete source commit	Eligible	-100.0%	-40.0%	0/0/1
▼ Select a class site with complete direct-test enumeration	Eligible	-100.0%	-40.0%	0/0/1
▼ Select a method whose tracking pull request is merged	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Defer a completed issue unrelated to the ignored test: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — csharp-scripts (claude-sonnet-5)

Why: Net win +0.0% (0W/1T/0L over 1 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 1 paired run(s) — underpowered (1 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=1; 0W/1T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.50)

Repeated-run reliability (not used by the gate): 1 paired run (0W/1T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Test a C# language feature with a script	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Test a C# language feature with a script: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — csharp-scripts (gpt-5.6-luna)

Why: Net win +0.0% (0W/1T/0L over 1 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 1 paired run(s) — underpowered (1 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=1; 0W/1T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.40)

Repeated-run reliability (not used by the gate): 1 paired run (0W/1T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Test a C# language feature with a script	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Test a C# language feature with a script: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — dotnet-pinvoke (claude-sonnet-5)

Why: Net win +50.0% (1W/1T/0L over 2 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +20.0% across 2 paired run(s) — underpowered (2 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=2; 1W/1T/0L; d=1; p=0.500; net +50.0%

Overfit: Moderate (score 0.31)

Repeated-run reliability (not used by the gate): 2 paired runs (1W/1T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Generate LibraryImport declaration from C header (.NET Framework)	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Generate LibraryImport declaration from C header (.NET Framework): Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — dotnet-pinvoke (gpt-5.6-luna)

Why: Net win +0.0% (0W/2T/0L over 2 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 2 paired run(s) — underpowered (2 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=2; 0W/2T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.43)

Repeated-run reliability (not used by the gate): 2 paired runs (0W/2T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Generate LibraryImport declaration from C header (.NET 8+)	Eligible	+0.0%	+0.0%	0/1/0
= Generate LibraryImport declaration from C header (.NET Framework)	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Generate LibraryImport declaration from C header (.NET 8+): Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — nuget-trusted-publishing (claude-sonnet-5)

Why: Net win -33.3% (0W/2T/1L over 3 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -33.3% across 3 paired run(s) — underpowered (3 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=3; 0W/2T/1L; d=1; p=0.500; net -33.3%

Overfit: Moderate (score 0.33)

Repeated-run reliability (not used by the gate): 3 paired runs (0W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Migrate existing workflow from API key to trusted publishing	Eligible	+0.0%	+0.0%	0/1/0
= Set up NuGet publishing without mentioning trusted publishing	Eligible	+0.0%	+0.0%	0/1/0
▼ Set up trusted publishing for a new NuGet library	Eligible	-100.0%	-100.0%	0/0/1

Illustrative judge evidence:

• Migrate existing workflow from API key to trusted publishing: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — nuget-trusted-publishing (gpt-5.6-luna)

Why: Net win +0.0% (0W/3T/0L over 3 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 3 paired run(s) — underpowered (3 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=3; 0W/3T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.40)

Repeated-run reliability (not used by the gate): 3 paired runs (0W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Migrate existing workflow from API key to trusted publishing	Eligible	+0.0%	+0.0%	0/1/0
= Set up NuGet publishing without mentioning trusted publishing	Eligible	+0.0%	+0.0%	0/1/0
= Set up trusted publishing for a new NuGet library	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Migrate existing workflow from API key to trusted publishing: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — create-datadriven-aspnetcore (claude-sonnet-5)

Why: Net win +40.0% (2W/3T/0L over 5 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +28.0% across 5 paired run(s) — not credible — 3 of 5 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 2W/3T/0L; d=2; p=0.250; net +40.0%

Warnings: Activation: isolated 5/5; plugin 4/5

Overfit: Moderate (score 0.45)

Repeated-run reliability (not used by the gate): 5 paired runs (2W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Scaffold Blazor CRUD components	Eligible	+100.0%	+100.0%	1/0/0
= Scaffold MVC CRUD with an existing DbContext	Eligible	+0.0%	+0.0%	0/1/0
= Scaffold Minimal API child and required parent CRUD	Eligible	+0.0%	+0.0%	0/1/0
= Scaffold Minimal API endpoints with OpenAPI	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Scaffold MVC CRUD with an existing DbContext: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — create-datadriven-aspnetcore (gpt-5.6-luna)

Why: Net win +0.0% (0W/5T/0L over 5 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 5 paired run(s) — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.28)

Repeated-run reliability (not used by the gate): 5 paired runs (0W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Scaffold Blazor CRUD components	Eligible	+0.0%	+0.0%	0/1/0
= Scaffold MVC CRUD with an existing DbContext	Eligible	+0.0%	+0.0%	0/1/0
= Scaffold Minimal API child and required parent CRUD	Eligible	+0.0%	+0.0%	0/1/0
= Scaffold Minimal API endpoints with OpenAPI	Eligible	+0.0%	+0.0%	0/1/0
= Scaffold Razor Pages CRUD with EF Core and SQLite	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Scaffold Blazor CRUD components: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — msbuild-quality-review (gpt-5.6-luna)

Why: Net win +40.0% (5W/4T/1L over 10 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +16.0% across 10 paired run(s) — not credible (sign test p=0.109 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=10; 5W/4T/1L; d=6; p=0.109; net +40.0%

Repeated-run reliability (not used by the gate): 10 paired runs (5W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Accept a valid required import in the packed layout	Eligible	+0.0%	+0.0%	0/1/0
= Avoid findings when the changed file cannot be read	Eligible	+0.0%	+0.0%	0/1/0
▼ Filter intentionally broken fixture changes	Eligible	-100.0%	-100.0%	0/0/1
= Identify a generated-file collision across projects and frameworks	Eligible	+0.0%	+0.0%	0/1/0
= Preserve explicit FSharp compilation order	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Accept a valid required import in the packed layout: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — optimizing-ef-core-queries (claude-sonnet-5)

Why: Net win +27.3% (3W/8T/0L over 11 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +15.0% across 12 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 8 of 11 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 3W/8T/0L; d=3; p=0.125; net +27.3%; 1 dormancy excluded

Warnings: Activation: isolated 9/11; plugin 9/11

Overfit: Moderate (score 0.40)

Repeated-run reliability (not used by the gate): 12 paired runs (3W/9T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Improve slow category discount maintenance	Eligible	+0.0%	+0.0%	0/1/0
▲ Keep a hot status queue server-side and identify its missing index	Eligible	+100.0%	+40.0%	1/0/0
= Review a slow catalog search repository	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Review customer directory search	Eligible	+0.0%	+0.0%	0/1/0
= Review order history pagination behavior	Eligible	+0.0%	+0.0%	0/1/0
= Review order operations query boundaries	Eligible	+0.0%	+0.0%	0/1/0
= Review route planning data access	Eligible	+0.0%	+0.0%	0/1/0
= Review saved order search implementation	Eligible	+0.0%	+0.0%	0/1/0
= Review statement export implementation	Eligible	+0.0%	+0.0%	0/1/0
= Split independent collection loads without changing results	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Improve slow category discount maintenance: The actual patches are functionally identical aside from whitespace/naming of the setters lambda, and both demonstrate the same large targeted performance improvement and equivalence. Neither has a material final-result advantage.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Mixed evidence — optimizing-ef-core-queries (gpt-5.6-luna)

Why: Net win +0.0% (1W/9T/1L over 11 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 12 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 1W/9T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Repeated-run reliability (not used by the gate): 12 paired runs (1W/10T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Audit a back-office operations service riddled with query problems	Eligible	-100.0%	-40.0%	0/0/1
= Improve slow category discount maintenance	Eligible	+0.0%	+0.0%	0/1/0
= Review a slow catalog search repository	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Review customer directory search	Eligible	+0.0%	+0.0%	0/1/0
= Review order history pagination behavior	Eligible	+0.0%	+0.0%	0/1/0
= Review order operations query boundaries	Eligible	+0.0%	+0.0%	0/1/0
= Review pre-merge data access changes	Eligible	+0.0%	+0.0%	0/1/0
= Review route planning data access	Eligible	+0.0%	+0.0%	0/1/0
= Review saved order search implementation	Eligible	+0.0%	+0.0%	0/1/0
= Review statement export implementation	Eligible	+0.0%	+0.0%	0/1/0
= Split independent collection loads without changing results	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Audit a back-office operations service riddled with query problems: Both agents successfully completed the task with all 11 methods improved and equivalent to baseline. However, Run A was more thorough in applying .AsNoTracking() to read-only product queries (ListActiveProducts and SearchProducts), resulting in slightly better performance im...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — unskip-closed-tests (claude-sonnet-5)

Why: Net win +18.2% (5W/3T/3L over 11 preference-eligible stimulus vote(s), sign test p=0.363), mean preference +7.3% across 11 paired run(s) — not credible (sign test p=0.363 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 5W/3T/3L; d=8; p=0.363; net +18.2%

Repeated-run reliability (not used by the gate): 11 paired runs (5W/3T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Defer a completed issue unrelated to the ignored test	Eligible	-100.0%	-40.0%	0/0/1
▼ Defer an incomplete partial class with nested tests	Eligible	-100.0%	-40.0%	0/0/1
= Defer open and not-planned tracking items	Eligible	+0.0%	+0.0%	0/1/0
▼ Reject a legacy manifest without source-bound identities	Eligible	-100.0%	-40.0%	0/0/1
= Reject a manifest for an obsolete source commit	Eligible	+0.0%	+0.0%	0/1/0
= Select a method whose tracking pull request is merged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Defer a completed issue unrelated to the ignored test: Both reach the safe, correct deferral decision. A is modestly stronger because its comment action and populated finding clearly communicate the evidence, mismatch classification, limitations, and required human follow-up, whereas B's noop has no findings despite having useful ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — vectorization (claude-sonnet-5)

Why: Net win +22.2% (3W/5T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +14.0% across 10 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 9 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 3W/5T/1L; d=4; p=0.312; net +22.2%; 1 dormancy excluded

Warnings: Activation: isolated 8/9; plugin 4/9; Activation-only stop: isolated 3 failed runs; Activation-only stop: plugin 1 failed run

Overfit: Moderate (score 0.42)

Repeated-run reliability (not used by the gate): 10 paired runs (3W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Detect architecture-specific behavior change	Eligible	+0.0%	+0.0%	0/1/0
= Detect empty-span reference access	Eligible	+0.0%	+0.0%	0/1/0
▲ Detect unsigned tail offset underflow	Eligible	+100.0%	+40.0%	1/0/0
= Extend an existing vectorized path	Eligible	+0.0%	+0.0%	0/1/0
= Ignore unrelated parser performance request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Preserve product empty-input contract	Eligible	+0.0%	+0.0%	0/1/0
= Reject fragile backwards reference	Eligible	+0.0%	+0.0%	0/1/0
▲ Repair vectorized reduction safely	Eligible	+100.0%	+40.0%	1/0/0
▼ Vectorize conditional increment safely	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Detect architecture-specific behavior change: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — vectorization (gpt-5.6-luna)

Why: Net win +22.2% (2W/7T/0L over 9 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +8.0% across 10 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 7 of 9 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 2W/7T/0L; d=2; p=0.250; net +22.2%; 1 dormancy excluded

Warnings: Activation: isolated 7/9; plugin 5/9

Overfit: High (score 0.60)

Repeated-run reliability (not used by the gate): 10 paired runs (2W/8T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Detect architecture-specific behavior change	Eligible	+0.0%	+0.0%	0/1/0
▲ Detect empty-span reference access	Eligible	+100.0%	+40.0%	1/0/0
= Detect unsigned tail offset underflow	Eligible	+0.0%	+0.0%	0/1/0
= Detect unsupported vector element type	Eligible	+0.0%	+0.0%	0/1/0
= Extend an existing vectorized path	Eligible	+0.0%	+0.0%	0/1/0
= Ignore unrelated parser performance request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Preserve product empty-input contract	Eligible	+0.0%	+0.0%	0/1/0
= Reject fragile backwards reference	Eligible	+0.0%	+0.0%	0/1/0
= Repair vectorized reduction safely	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Detect architecture-specific behavior change: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 2 results are in Full Results.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1369 in dotnet/skills, download eval artifacts with gh run download 41905893626 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/9f4b279a8d92342e41e21ca3fb56f949cd594c40/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

✅ Evaluation passed for 9f4b279. cc @AbhitejJohn @JanKrivanek — please review.

[Copilot]

🟡 Changes recommended

Unconditionally replacing COPILOT_HOME hides persisted CLI authentication and can break comparisons and retries for callers without token environment variables.

1 open issue

• Preserve inherited COPILOT_HOME when no token is set · New

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues remain for the authentication-isolation and token-priority changes.

No open issues 🎉

1 resolved since last review

• Preserve inherited COPILOT_HOME when no token is set

_{Review effort: Balanced}

[github-actions]

📊 Skill and Agent Evaluation Results

8 model/target results across 4 targets and 2 models — ✅ 0 improved, ➖ 8 results without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 36fe491315c3b07509cf26e828c8c9a24bee77f4; 2 judge models.

Measurement health: 8 expected / 8 observed / 8 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
agent.build-perf	claude-sonnet-5	➖ Improvement signal, tie-limited	n=5; 3W/1T/1L; d=4; p=0.312; net +40.0%	—	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
agent.build-perf	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=5; 2W/2T/1L; d=3; p=0.500; net +20.0%	—	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
build-perf-diagnostics	claude-sonnet-5	➖ Improvement signal, tie-limited	n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 2 dormancy excluded	🟡 0.42	Activation: isolated 6/6; plugin 3/6	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
build-perf-diagnostics	gpt-5.6-luna	➖ No preference	n=6; 0W/6T/0L; d=0; p=1.000; net +0.0%; 2 dormancy excluded	🔴 0.53	Activation: isolated 6/6; plugin 2/6	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
csharp-refactoring	claude-sonnet-5	➖ Improvement signal, tie-limited	n=17; 3W/13T/1L; d=4; p=0.312; net +11.8%; 1 dormancy excluded	🟡 0.39	Activation: isolated 14/17; plugin 9/17; Activation-only stop: plugin 2 failed runs	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
csharp-refactoring	gpt-5.6-luna	➖ Baseline signal, tie-limited	n=17; 0W/16T/1L; d=1; p=0.500; net -5.9%; 1 dormancy excluded	—	Activation-only stop: plugin 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
writing-mstest-tests	claude-sonnet-5	➖ Improvement signal, unproven	n=15; 4W/9T/2L; d=6; p=0.344; net +13.3%	🔴 0.52	Activation: isolated 15/15; plugin 14/15; Activation-only stop: isolated 2 failed runs; Activation-only stop: plugin 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
writing-mstest-tests	gpt-5.6-luna	➖ Baseline signal, tie-limited	n=15; 0W/13T/2L; d=2; p=0.250; net -13.3%	—	Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 2 failed runs	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Improvement signal, tie-limited — agent.build-perf (claude-sonnet-5)

Why: Net win +40.0% (3W/1T/1L over 5 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +16.0% across 5 paired run(s) — not credible — 1 of 5 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the agent is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 3W/1T/1L; d=4; p=0.312; net +40.0%

Repeated-run reliability (not used by the gate): 5 paired runs (3W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Investigate the reader connection before optimizing a build	Eligible	+0.0%	+0.0%	0/1/0
▼ Resume measured analysis after host tool discovery	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Investigate the reader connection before optimizing a build: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — agent.build-perf (gpt-5.6-luna)

Why: Net win +20.0% (2W/2T/1L over 5 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +8.0% across 5 paired run(s) — not credible — 2 of 5 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the agent is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 2W/2T/1L; d=3; p=0.500; net +20.0%

Repeated-run reliability (not used by the gate): 5 paired runs (2W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Continue performance analysis when reader repair is declined	Eligible	-100.0%	-100.0%	0/0/1
= Investigate the reader connection before optimizing a build	Eligible	+0.0%	+0.0%	0/1/0
= Keep an unresponsive launch inconclusive	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Continue performance analysis when reader repair is declined: Response A provides a working, standards-compliant approach using StructuredLogViewer, which is the officially supported MSBuild tool for analyzing binlog files. Response B's core command is fundamentally flawed—you cannot pass a .binlog file to dotnet msbuild as a project i...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — build-perf-diagnostics (claude-sonnet-5)

Why: Net win +16.7% (1W/5T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +0.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 6 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 2 dormancy excluded

Warnings: Activation: isolated 6/6; plugin 3/6

Overfit: Moderate (score 0.42)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline a known repeated-restore fix	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a Copy task dominating build time	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a pathological ResolveAssemblyReference time	Eligible	+0.0%	+0.0%	0/1/0
▲ Diagnose a single custom target dominating one project's build	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose evaluation overhead before any target runs	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose per-project overhead across many small projects	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose slow build for a small project	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Diagnose a Copy task dominating build time: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — build-perf-diagnostics (gpt-5.6-luna)

Why: Net win +0.0% (0W/6T/0L over 6 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 0W/6T/0L; d=0; p=1.000; net +0.0%; 2 dormancy excluded

Warnings: Activation: isolated 6/6; plugin 2/6

Overfit: High (score 0.53)

Repeated-run reliability (not used by the gate): 8 paired runs (0W/8T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a known repeated-restore fix	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a Copy task dominating build time	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a pathological ResolveAssemblyReference time	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a single custom target dominating one project's build	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose evaluation overhead before any target runs	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose per-project overhead across many small projects	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose slow build for a small project	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Diagnose a Copy task dominating build time: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — csharp-refactoring (claude-sonnet-5)

Why: Net win +11.8% (3W/13T/1L over 17 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +11.1% across 18 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 13 of 17 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=17; 3W/13T/1L; d=4; p=0.312; net +11.8%; 1 dormancy excluded

Warnings: Activation: isolated 14/17; plugin 9/17; Activation-only stop: plugin 2 failed runs

Overfit: Moderate (score 0.39)

Repeated-run reliability (not used by the gate): 18 paired runs (3W/14T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Consolidate duplicate public helpers without breaking shipped callers	Eligible	+0.0%	+0.0%	0/1/0
= Decline a framework and package upgrade dressed up as a refactor	Eligible	+0.0%	+0.0%	0/1/0
▲ Decline a new-feature request dressed up as a refactor	Eligible	+100.0%	+100.0%	1/0/0
▼ Extract shared discount logic without collapsing distinct inputs	Eligible	-100.0%	-40.0%	0/0/1
= Extract subtotal calculation without changing arithmetic	Eligible	+0.0%	+0.0%	0/1/0
= Inline a pass-through wrapper and update callers	Eligible	+0.0%	+0.0%	0/1/0
= Keep a shipped nullable contract out of a refactor	Eligible	+0.0%	+0.0%	0/1/0
= Merge two near-identical types into one parameterized type	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a serialized name while renaming a C# property	Eligible	+0.0%	+0.0%	0/1/0
= Rename a helper used in every target-framework branch	Eligible	+0.0%	+0.0%	0/1/0
= Rename a member declared by a generated partial source	Eligible	+0.0%	+0.0%	0/1/0
= Rename a method without breaking a configured reflection name	Eligible	+0.0%	+0.0%	0/1/0
= Rename a shipped public method without breaking existing callers	Eligible	+0.0%	+0.0%	0/1/0
= Rename an internal member used by a friend assembly	Eligible	+0.0%	+0.0%	0/1/0
= Rename one binding without touching same-text symbols	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant for an ordinary feature request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Consolidate duplicate public helpers without breaking shipped callers: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Baseline signal, tie-limited — csharp-refactoring (gpt-5.6-luna)

Why: Net win -5.9% (0W/16T/1L over 17 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -5.6% across 18 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=17; 0W/16T/1L; d=1; p=0.500; net -5.9%; 1 dormancy excluded

Warnings: Activation-only stop: plugin 1 failed run

Repeated-run reliability (not used by the gate): 18 paired runs (0W/17T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Consolidate duplicate public helpers without breaking shipped callers	Eligible	+0.0%	+0.0%	0/1/0
= Decline a framework and package upgrade dressed up as a refactor	Eligible	+0.0%	+0.0%	0/1/0
= Decline a new-feature request dressed up as a refactor	Eligible	+0.0%	+0.0%	0/1/0
= Extract shared discount logic without collapsing distinct inputs	Eligible	+0.0%	+0.0%	0/1/0
= Extract subtotal calculation without changing arithmetic	Eligible	+0.0%	+0.0%	0/1/0
▼ Inline a pass-through wrapper and update callers	Eligible	-100.0%	-100.0%	0/0/1
= Keep a shipped nullable contract out of a refactor	Eligible	+0.0%	+0.0%	0/1/0
= Keep behavior-changing bug fixes out of a behavior-preserving refactor	Eligible	+0.0%	+0.0%	0/1/0
= Merge two near-identical types into one parameterized type	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a serialized name while renaming a C# property	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a shipped wrapper while inlining ordinary callers	Eligible	+0.0%	+0.0%	0/1/0
= Rename a helper used in every target-framework branch	Eligible	+0.0%	+0.0%	0/1/0
= Rename a member declared by a generated partial source	Eligible	+0.0%	+0.0%	0/1/0
= Rename a method without breaking a configured reflection name	Eligible	+0.0%	+0.0%	0/1/0
= Rename a shipped public method without breaking existing callers	Eligible	+0.0%	+0.0%	0/1/0
= Rename an internal member used by a friend assembly	Eligible	+0.0%	+0.0%	0/1/0
= Rename one binding without touching same-text symbols	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant for an ordinary feature request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Consolidate duplicate public helpers without breaking shipped callers: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — writing-mstest-tests (claude-sonnet-5)

Why: Net win +13.3% (4W/9T/2L over 15 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +21.3% across 15 paired run(s) — not credible (sign test p=0.344 > 0.05)

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=15; 4W/9T/2L; d=6; p=0.344; net +13.3%

Warnings: Activation: isolated 15/15; plugin 14/15; Activation-only stop: isolated 2 failed runs; Activation-only stop: plugin 1 failed run

Overfit: High (score 0.52)

Repeated-run reliability (not used by the gate): 15 paired runs (4W/9T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Apply checked-in MSTest naming and category conventions	Eligible	+100.0%	+100.0%	1/0/0
= Apply environment retry and cleanup policies	Eligible	+0.0%	+0.0%	0/1/0
= Configure concurrent execution with a serial exception	Eligible	+0.0%	+0.0%	0/1/0
= Correct expected and actual failure labels	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose failed numeric boundaries	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose failed string conditions	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose identity, null, absence, and emptiness	Eligible	+0.0%	+0.0%	0/1/0
= Improve collection failure diagnostics	Eligible	+0.0%	+0.0%	0/1/0
▼ Make a data source compile-time safe	Eligible	-100.0%	-40.0%	0/0/1
= Modernize legacy MSTest code safely	Eligible	+0.0%	+0.0%	0/1/0
= Respect a pinned framework version	Eligible	+0.0%	+0.0%	0/1/0
= Verify an exception at the throwing call	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Apply environment retry and cleanup policies: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Baseline signal, tie-limited — writing-mstest-tests (gpt-5.6-luna)

Why: Net win -13.3% (0W/13T/2L over 15 preference-eligible stimulus vote(s), sign test p=0.250), mean preference -9.3% across 15 paired run(s) — no improvement

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=15; 0W/13T/2L; d=2; p=0.250; net -13.3%

Warnings: Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 2 failed runs

Repeated-run reliability (not used by the gate): 15 paired runs (0W/13T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Apply checked-in MSTest naming and category conventions	Eligible	-100.0%	-40.0%	0/0/1
= Apply environment retry and cleanup policies	Eligible	+0.0%	+0.0%	0/1/0
= Configure concurrent execution with a serial exception	Eligible	+0.0%	+0.0%	0/1/0
= Correct expected and actual failure labels	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose failed numeric boundaries	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose failed string conditions	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose identity, null, absence, and emptiness	Eligible	+0.0%	+0.0%	0/1/0
= Improve collection failure diagnostics	Eligible	+0.0%	+0.0%	0/1/0
▼ Initialize test state in lifecycle order	Eligible	-100.0%	-100.0%	0/0/1
= Make a data source compile-time safe	Eligible	+0.0%	+0.0%	0/1/0
= Modernize legacy MSTest code safely	Eligible	+0.0%	+0.0%	0/1/0
= Report runtime type mismatches as assertion failures	Eligible	+0.0%	+0.0%	0/1/0
= Respect a pinned framework version	Eligible	+0.0%	+0.0%	0/1/0
= Stop in-flight work when a timed test expires	Eligible	+0.0%	+0.0%	0/1/0
= Verify an exception at the throwing call	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Apply checked-in MSTest naming and category conventions: Both responses successfully apply all four rubric criteria with technically correct solutions. The key differentiator is criterion 3 (fixture-generation exception handling): Response A demonstrated explicit understanding of why the test class should remain unsealed, showing co...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1369 in dotnet/skills, download eval artifacts with gh run download 42109615479 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/36fe491315c3b07509cf26e828c8c9a24bee77f4/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 36fe491315c3b07509cf26e828c8c9a24bee77f4 to retry this exact commit.

55 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 36fe491315c3b07509cf26e828c8c9a24bee77f4 to retry this exact commit.

67 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

⏳ Commit 36fe491 is already being evaluated — follow it here: https://github.com/dotnet/skills/actions/runs/42146813295. Not starting a duplicate run.

[github-actions]

📊 Skill, Agent, and Workflow Evaluation Results

Workflow results measure offline prompt decisions and proposed outputs, not live Actions jobs or publication.

36 model/target results across 18 targets and 2 models — ✅ 1 improved, ➖ 24 results without a clear winner, ⚠️ 6 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only), 🔻 5 objective regressions.

Measurement identity: evaluated commit 36fe491315c3b07509cf26e828c8c9a24bee77f4; 2 judge models.

Measurement health: 36 expected / 36 observed / 36 written; 0 missing, 0 unexpected, 6 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: enabled; 5 results show a proven objective regression.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
build-failure-analysis	claude-sonnet-5	➖ Improvement signal, unproven	n=10; 7W/1T/2L; d=9; p=0.090; net +50.0%	—	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
build-failure-analysis	gpt-5.6-luna	🔻 Objective regression	n=10; 3W/2T/5L; d=8; p=0.363; net -20.0%	—	—	Inspect objective completion losses and fix them before merge.
dotnet-maui-doctor	claude-sonnet-5	➖ Improvement signal, unproven	n=8; 4W/3T/1L; d=5; p=0.188; net +37.5%	🟡 0.44	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
dotnet-maui-doctor	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=8; 1W/7T/0L; d=1; p=0.500; net +12.5%	🟡 0.49	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
maui-app-lifecycle	claude-sonnet-5	⚠️ Underpowered	n=4; 2W/2T/0L; d=2; p=0.250; net +50.0%	🔴 0.63	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-app-lifecycle	gpt-5.6-luna	⚠️ Underpowered	n=4; 0W/4T/0L; d=0; p=1.000; net +0.0%	🟡 0.46	Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 1 failed run	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-collectionview	claude-sonnet-5	➖ Mixed evidence	n=5; 1W/3T/1L; d=2; p=0.750; net +0.0%	🟡 0.45	Activation-only stop: plugin 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
maui-collectionview	gpt-5.6-luna	➖ No preference	n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%	🟡 0.39	—	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
maui-data-binding	claude-sonnet-5	⚠️ Underpowered	n=4; 1W/3T/0L; d=1; p=0.500; net +25.0%	🔴 0.57	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-data-binding	gpt-5.6-luna	⚠️ Underpowered	n=4; 1W/3T/0L; d=1; p=0.500; net +25.0%	🔴 0.54	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-dependency-injection	claude-sonnet-5	➖ Improvement signal, tie-limited	n=5; 3W/2T/0L; d=3; p=0.125; net +60.0%	🔴 0.50	Activation: isolated 5/5; plugin 4/5; Activation-only stop: isolated 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
maui-dependency-injection	gpt-5.6-luna	➖ No preference	n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%	🟡 0.47	Activation: isolated 5/5; plugin 4/5	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
maui-safe-area	claude-sonnet-5	⚠️ Underpowered	n=4; 1W/3T/0L; d=1; p=0.500; net +25.0%	🟡 0.31	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-safe-area	gpt-5.6-luna	⚠️ Underpowered	n=4; 0W/3T/1L; d=1; p=0.500; net -25.0%	🟡 0.34	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
maui-shell-navigation	claude-sonnet-5	➖ Improvement signal, tie-limited	n=5; 4W/1T/0L; d=4; p=0.063; net +80.0%	🔴 0.58	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
maui-shell-navigation	gpt-5.6-luna	➖ No preference	n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%	🟡 0.43	—	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
maui-theming	claude-sonnet-5	➖ Improvement signal, tie-limited	n=5; 1W/4T/0L; d=1; p=0.500; net +20.0%	🔴 0.60	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
maui-theming	gpt-5.6-luna	➖ No preference	n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%	🔴 0.57	—	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
msbuild-quality-review	claude-sonnet-5	➖ Improvement signal, unproven	n=10; 7W/1T/2L; d=9; p=0.090; net +50.0%	—	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
msbuild-quality-review	gpt-5.6-luna	🔻 Objective regression	n=10; 5W/3T/2L; d=7; p=0.227; net +30.0%	—	—	Inspect objective completion losses and fix them before merge.
template-authoring	claude-sonnet-5	➖ Improvement signal, tie-limited	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%	🟡 0.38	Activation: isolated 7/7; plugin 5/7	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
template-authoring	gpt-5.6-luna	➖ Mixed evidence	n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%	🟡 0.42	Activation-only stop: plugin 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
template-comparison	claude-sonnet-5	➖ Improvement signal, tie-limited	n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%	🟡 0.32	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
template-comparison	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=7; 1W/6T/0L; d=1; p=0.500; net +14.3%	🟡 0.32	Activation: isolated 7/7; plugin 6/7	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
template-discovery	claude-sonnet-5	➖ Improvement signal, tie-limited	n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%	🟡 0.40	Activation: isolated 7/7; plugin 6/7	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
template-discovery	gpt-5.6-luna	➖ No preference	n=7; 0W/7T/0L; d=0; p=1.000; net +0.0%	🟡 0.48	Activation: isolated 7/7; plugin 6/7	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
template-instantiation	claude-sonnet-5	➖ No preference	n=7; 0W/7T/0L; d=0; p=1.000; net +0.0%	🟡 0.34	—	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
template-instantiation	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%	🔴 0.53	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
template-smart-defaults	claude-sonnet-5	➖ Improvement signal, tie-limited	n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%	🔴 0.50	Activation: isolated 7/7; plugin 4/7	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
template-smart-defaults	gpt-5.6-luna	➖ No preference	n=7; 0W/7T/0L; d=0; p=1.000; net +0.0%	🔴 0.57	Activation: isolated 7/7; plugin 5/7	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
template-validation	claude-sonnet-5	➖ Improvement signal, unproven	n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%	🟡 0.49	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
template-validation	gpt-5.6-luna	➖ No preference	n=7; 0W/7T/0L; d=0; p=1.000; net +0.0%	🔴 0.61	—	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.
test-failure-analysis	claude-sonnet-5	🔻 Objective regression	n=12; 10W/2T/0L; d=10; p=0.001; net +83.3%	—	—	Inspect objective completion losses and fix them before merge.
test-failure-analysis	gpt-5.6-luna	✅ Improved	n=12; 10W/1T/1L; d=11; p=0.006; net +75.0%	—	—	None.
unskip-closed-tests	claude-sonnet-5	🔻 Objective regression	n=11; 5W/5T/1L; d=6; p=0.109; net +36.4%	—	—	Inspect objective completion losses and fix them before merge.
unskip-closed-tests	gpt-5.6-luna	🔻 Objective regression	n=11; 2W/5T/4L; d=6; p=0.344; net -18.2%	—	—	Inspect objective completion losses and fix them before merge.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

🔻 Objective regression — build-failure-analysis (gpt-5.6-luna)

Why: Net win -20.0% (3W/2T/5L over 10 preference-eligible stimulus vote(s), sign test p=0.363), mean preference -20.0% across 10 paired run(s) — no improvement — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 3W/2T/5L; d=8; p=0.363; net -20.0%

Repeated-run reliability (not used by the gate): 10 paired runs (3W/2T/5L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Group a removed API cascade in the second build leg	Eligible	-100.0%	-40.0%	0/0/1
▼ Recognize a failed pipeline with a clean build	Eligible	-100.0%	-40.0%	0/0/1
▼ Report the absence of all binary logs without a diagnosis	Eligible	-100.0%	-100.0%	0/0/1
▼ Retain useful findings while naming a missing module	Eligible	-100.0%	-40.0%	0/0/1
▼ Suppress obsolete analysis after a head update	Eligible	-100.0%	-100.0%	0/0/1
= Suppress obsolete analysis after the merge revision moves	Eligible	+0.0%	+0.0%	0/1/0
= Withhold conclusions when the failed leg has no log	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Group a removed API cascade in the second build leg: Response A is moderately better because it uses the semantically correct 'review' action for PR review comments (versus Response B's generic 'comment'), and provides an explicit, copy-paste-ready code suggestion for the fix. While Response B has slightly better evidence granul...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — msbuild-quality-review (gpt-5.6-luna)

Why: Net win +30.0% (5W/3T/2L over 10 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +12.0% across 10 paired run(s) — not credible (sign test p=0.227 > 0.05) — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 5W/3T/2L; d=7; p=0.227; net +30.0%

Repeated-run reliability (not used by the gate): 10 paired runs (5W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Accept a valid required import in the packed layout	Eligible	+0.0%	+0.0%	0/1/0
▼ Avoid findings when the changed file cannot be read	Eligible	-100.0%	-40.0%	0/0/1
= Catch a property evaluated before its discriminator exists	Eligible	+0.0%	+0.0%	0/1/0
▼ Filter intentionally broken fixture changes	Eligible	-100.0%	-100.0%	0/0/1
= Limit findings to the changed in-scope policy	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Accept a valid required import in the packed layout: Position-swap inconsistent (forward: tie, reverse: skill). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — test-failure-analysis (claude-sonnet-5)

Why: Net win +83.3% (10W/2T/0L over 12 preference-eligible stimulus vote(s), sign test p=0.001), mean preference +58.3% across 12 paired run(s) — credibly better — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=12; 10W/2T/0L; d=10; p=0.001; net +83.3%

Repeated-run reliability (not used by the gate): 12 paired runs (10W/2T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid a visible response for an empty complete bundle	Eligible	+0.0%	+0.0%	0/1/0
= Keep a newer final result ahead of late preliminary evidence	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid a visible response for an empty complete bundle: Both produce the correct noop result with empty findings, appropriate limitations, and a sound explanation that no evidence to report is not evidence of test success. B's denied shell command did not impair its final result.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — unskip-closed-tests (claude-sonnet-5)

Why: Net win +36.4% (5W/5T/1L over 11 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +14.5% across 11 paired run(s) — not credible (sign test p=0.109 > 0.05) — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=11; 5W/5T/1L; d=6; p=0.109; net +36.4%

Repeated-run reliability (not used by the gate): 11 paired runs (5W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Defer an incomplete partial class with nested tests	Eligible	+0.0%	+0.0%	0/1/0
= Defer open and not-planned tracking items	Eligible	+0.0%	+0.0%	0/1/0
= Reject a candidate bound to the wrong syntax owner	Eligible	+0.0%	+0.0%	0/1/0
▼ Reject a legacy manifest without source-bound identities	Eligible	-100.0%	-40.0%	0/0/1
= Reject a manifest for an obsolete source commit	Eligible	+0.0%	+0.0%	0/1/0
= Select a source-bound method with a completed issue	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Defer an incomplete partial class with nested tests: Both produce the safe, schema-consistent noop outcome and clearly explain why no selection can be made without rewriting or supplementing the manifest. B is slightly more explicit about governing deferral rules, while A is slightly more concise; neither has a meaningful correc...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — unskip-closed-tests (gpt-5.6-luna)

Why: Net win -18.2% (2W/5T/4L over 11 preference-eligible stimulus vote(s), sign test p=0.344), mean preference -7.3% across 11 paired run(s) — no improvement — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=11; 2W/5T/4L; d=6; p=0.344; net -18.2%

Repeated-run reliability (not used by the gate): 11 paired runs (2W/5T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Defer a completed issue unrelated to the ignored test	Eligible	-100.0%	-40.0%	0/0/1
▼ Defer an incomplete partial class with nested tests	Eligible	-100.0%	-40.0%	0/0/1
= Defer open and not-planned tracking items	Eligible	+0.0%	+0.0%	0/1/0
▼ Do not treat a zero-result verification exit as authorization	Eligible	-100.0%	-40.0%	0/0/1
▼ Preserve distinct candidates across modules with identical attributes	Eligible	-100.0%	-40.0%	0/0/1
= Reject a legacy manifest without source-bound identities	Eligible	+0.0%	+0.0%	0/1/0
= Reject a manifest for an obsolete source commit	Eligible	+0.0%	+0.0%	0/1/0
= Select a class site with complete direct-test enumeration	Eligible	+0.0%	+0.0%	0/1/0
= Select a method whose tracking pull request is merged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Defer a completed issue unrelated to the ignored test: Both responses correctly identify that the candidate is not safe to attempt due to the mismatch between the socket-timeout ignore and the README-only tracking item. However, Response A is superior in execution: it provides comprehensive findings with classification and confide...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-app-lifecycle (claude-sonnet-5)

Why: Net win +50.0% (2W/2T/0L over 4 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +35.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 2W/2T/0L; d=2; p=0.250; net +50.0%

Overfit: High (score 0.63)

Repeated-run reliability (not used by the gate): 4 paired runs (2W/2T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Platform-specific lifecycle mapping	Eligible	+0.0%	+0.0%	0/1/0
= Window lifecycle event subscription	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Platform-specific lifecycle mapping: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-app-lifecycle (gpt-5.6-luna)

Why: Net win +0.0% (0W/4T/0L over 4 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 0W/4T/0L; d=0; p=1.000; net +0.0%

Warnings: Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 1 failed run

Overfit: Moderate (score 0.46)

Repeated-run reliability (not used by the gate): 4 paired runs (0W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid legacy Xamarin.Forms lifecycle methods	Eligible	+0.0%	+0.0%	0/1/0
= Platform-specific lifecycle mapping	Eligible	+0.0%	+0.0%	0/1/0
= Save and restore state on background	Eligible	+0.0%	+0.0%	0/1/0
= Window lifecycle event subscription	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid legacy Xamarin.Forms lifecycle methods: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-data-binding (claude-sonnet-5)

Why: Net win +25.0% (1W/3T/0L over 4 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +25.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 1W/3T/0L; d=1; p=0.500; net +25.0%

Overfit: High (score 0.57)

Repeated-run reliability (not used by the gate): 4 paired runs (1W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Create and use an IValueConverter	Eligible	+0.0%	+0.0%	0/1/0
= Implement MVVM ViewModel with ObservableObject	Eligible	+0.0%	+0.0%	0/1/0
= Set up compiled bindings with x:DataType on a page	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Create and use an IValueConverter: Position-swap inconsistent (forward: A, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-data-binding (gpt-5.6-luna)

Why: Net win +25.0% (1W/3T/0L over 4 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +25.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 1W/3T/0L; d=1; p=0.500; net +25.0%

Overfit: High (score 0.54)

Repeated-run reliability (not used by the gate): 4 paired runs (1W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Create and use an IValueConverter	Eligible	+0.0%	+0.0%	0/1/0
= Implement MVVM ViewModel with ObservableObject	Eligible	+0.0%	+0.0%	0/1/0
= Set up compiled bindings with x:DataType on a page	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Create and use an IValueConverter: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-safe-area (claude-sonnet-5)

Why: Net win +25.0% (1W/3T/0L over 4 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +25.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 1W/3T/0L; d=1; p=0.500; net +25.0%

Overfit: Moderate (score 0.31)

Repeated-run reliability (not used by the gate): 4 paired runs (1W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid deprecated UseSafeArea in new .NET 10 project	Eligible	+0.0%	+0.0%	0/1/0
= Edge-to-edge layout with SafeAreaEdges	Eligible	+0.0%	+0.0%	0/1/0
= Handle notch and status bar safe areas on iOS	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid deprecated UseSafeArea in new .NET 10 project: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — maui-safe-area (gpt-5.6-luna)

Why: Net win -25.0% (0W/3T/1L over 4 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -10.0% across 4 paired run(s) — underpowered (4 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=4; 0W/3T/1L; d=1; p=0.500; net -25.0%

Overfit: Moderate (score 0.34)

Repeated-run reliability (not used by the gate): 4 paired runs (0W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid deprecated UseSafeArea in new .NET 10 project	Eligible	+0.0%	+0.0%	0/1/0
= Edge-to-edge layout with SafeAreaEdges	Eligible	+0.0%	+0.0%	0/1/0
= Handle notch and status bar safe areas on iOS	Eligible	+0.0%	+0.0%	0/1/0
▼ Keyboard avoidance with safe area for chat UI	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Avoid deprecated UseSafeArea in new .NET 10 project: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — build-failure-analysis (claude-sonnet-5)

Why: Net win +50.0% (7W/1T/2L over 10 preference-eligible stimulus vote(s), sign test p=0.090), mean preference +26.0% across 10 paired run(s) — not credible (sign test p=0.090 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=10; 7W/1T/2L; d=9; p=0.090; net +50.0%

Repeated-run reliability (not used by the gate): 10 paired runs (7W/1T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Report the absence of all binary logs without a diagnosis	Eligible	-100.0%	-100.0%	0/0/1
= Retain useful findings while naming a missing module	Eligible	+0.0%	+0.0%	0/1/0
▼ Suppress obsolete analysis after the merge revision moves	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Report the absence of all binary logs without a diagnosis: A fulfills the central requested outcome: a useful, evidence-grounded diagnostic comment that sends a human to the originating build while explicitly acknowledging the missing diagnostics. B is cautious and evidence-grounded, but its noop choice omits the required useful comme...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — dotnet-maui-doctor (claude-sonnet-5)

Why: Net win +37.5% (4W/3T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +45.0% across 8 paired run(s) — not credible (sign test p=0.188 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 4W/3T/1L; d=5; p=0.188; net +37.5%

Overfit: Moderate (score 0.44)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Determine required Android SDK packages for specific .NET version	Eligible	-100.0%	-40.0%	0/0/1
= Fix stale MAUI workloads after SDK update	Eligible	+0.0%	+0.0%	0/1/0
= Plan Linux MAUI environment for Android	Eligible	+0.0%	+0.0%	0/1/0
= Plan complete MAUI setup on Windows	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Determine required Android SDK packages for specific .NET version: Both responses are strong and produce the same concrete current package list. A is more concise and robust as an answer to determining requirements from the installed SDK/workload itself; B offers useful manifest-inspection detail but makes the reproducible path unnecessarily ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — dotnet-maui-doctor (gpt-5.6-luna)

Why: Net win +12.5% (1W/7T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +12.5% across 8 paired run(s) — not credible — 7 of 8 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 1W/7T/0L; d=1; p=0.500; net +12.5%

Overfit: Moderate (score 0.49)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/7T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Determine required Android SDK packages for specific .NET version	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose non-Microsoft JDK causing build failure	Eligible	+0.0%	+0.0%	0/1/0
= Fix stale MAUI workloads after SDK update	Eligible	+0.0%	+0.0%	0/1/0
= Guardrail against workload update and repair	Eligible	+0.0%	+0.0%	0/1/0
= Plan Linux MAUI environment for Android	Eligible	+0.0%	+0.0%	0/1/0
= Plan macOS MAUI setup with Xcode	Eligible	+0.0%	+0.0%	0/1/0
= Prevent incorrect JAVA_HOME configuration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Determine required Android SDK packages for specific .NET version: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Mixed evidence — maui-collectionview (claude-sonnet-5)

Why: Net win +0.0% (1W/3T/1L over 5 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 5 paired run(s) — no improvement

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 1W/3T/1L; d=2; p=0.750; net +0.0%

Warnings: Activation-only stop: plugin 1 failed run

Overfit: Moderate (score 0.45)

Repeated-run reliability (not used by the gate): 5 paired runs (1W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid ListView and ViewCell mistakes	Eligible	+0.0%	+0.0%	0/1/0
= Basic CollectionView with data binding and DataTemplate	Eligible	+0.0%	+0.0%	0/1/0
= Grid layout with CollectionView	Eligible	+0.0%	+0.0%	0/1/0
▼ ItemSizingStrategy placement for uniform items	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Avoid ListView and ViewCell mistakes: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — maui-collectionview (gpt-5.6-luna)

Why: Net win +0.0% (0W/5T/0L over 5 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 5 paired run(s) — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.39)

Repeated-run reliability (not used by the gate): 5 paired runs (0W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid ListView and ViewCell mistakes	Eligible	+0.0%	+0.0%	0/1/0
= Basic CollectionView with data binding and DataTemplate	Eligible	+0.0%	+0.0%	0/1/0
= Grid layout with CollectionView	Eligible	+0.0%	+0.0%	0/1/0
= ItemSizingStrategy placement for uniform items	Eligible	+0.0%	+0.0%	0/1/0
= Selection and pull-to-refresh with CollectionView	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid ListView and ViewCell mistakes: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — maui-dependency-injection (claude-sonnet-5)

Why: Net win +60.0% (3W/2T/0L over 5 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +36.0% across 5 paired run(s) — not credible — 2 of 5 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 3W/2T/0L; d=3; p=0.125; net +60.0%

Warnings: Activation: isolated 5/5; plugin 4/5; Activation-only stop: isolated 1 failed run

Overfit: High (score 0.50)

Repeated-run reliability (not used by the gate): 5 paired runs (3W/2T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid AddScoped pitfall in MAUI	Eligible	+0.0%	+0.0%	0/1/0
▲ Diagnose a page whose injected dependencies are missing	Eligible	+100.0%	+40.0%	1/0/0
= Shell navigation auto-resolves DI-registered pages	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid AddScoped pitfall in MAUI: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — maui-dependency-injection (gpt-5.6-luna)

Why: Net win +0.0% (0W/5T/0L over 5 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 5 paired run(s) — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%

Warnings: Activation: isolated 5/5; plugin 4/5

Overfit: Moderate (score 0.47)

Repeated-run reliability (not used by the gate): 5 paired runs (0W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid AddScoped pitfall in MAUI	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a page whose injected dependencies are missing	Eligible	+0.0%	+0.0%	0/1/0
= Platform-specific service registration with fallback	Eligible	+0.0%	+0.0%	0/1/0
= Register services with correct lifetimes in MauiProgram.cs	Eligible	+0.0%	+0.0%	0/1/0
= Shell navigation auto-resolves DI-registered pages	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid AddScoped pitfall in MAUI: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — maui-shell-navigation (claude-sonnet-5)

Why: Net win +80.0% (4W/1T/0L over 5 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +32.0% across 5 paired run(s) — not credible — 1 of 5 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 4W/1T/0L; d=4; p=0.063; net +80.0%

Overfit: High (score 0.58)

Repeated-run reliability (not used by the gate): 5 paired runs (4W/1T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Set up Shell navigation with tabs and flyout	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Set up Shell navigation with tabs and flyout: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — maui-shell-navigation (gpt-5.6-luna)

Why: Net win +0.0% (0W/5T/0L over 5 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 5 paired run(s) — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.43)

Repeated-run reliability (not used by the gate): 5 paired runs (0W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose common Shell navigation mistakes	Eligible	+0.0%	+0.0%	0/1/0
= Handle back navigation and unsaved changes guard	Eligible	+0.0%	+0.0%	0/1/0
= Navigate with parameters using GoToAsync	Eligible	+0.0%	+0.0%	0/1/0
= Set up Shell navigation with tabs and flyout	Eligible	+0.0%	+0.0%	0/1/0
= Stable routes for deep linking into tabs	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Diagnose common Shell navigation mistakes: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — maui-theming (claude-sonnet-5)

Why: Net win +20.0% (1W/4T/0L over 5 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +8.0% across 5 paired run(s) — not credible — 4 of 5 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 1W/4T/0L; d=1; p=0.500; net +20.0%

Overfit: High (score 0.60)

Repeated-run reliability (not used by the gate): 5 paired runs (1W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add light/dark mode support using AppThemeBinding	Eligible	+0.0%	+0.0%	0/1/0
= Avoid common theming mistakes	Eligible	+0.0%	+0.0%	0/1/0
= Create custom themes with ResourceDictionary switching	Eligible	+0.0%	+0.0%	0/1/0
= Detect and respond to system theme changes	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add light/dark mode support using AppThemeBinding: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — maui-theming (gpt-5.6-luna)

Why: Net win +0.0% (0W/5T/0L over 5 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 5 paired run(s) — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=5; 0W/5T/0L; d=0; p=1.000; net +0.0%

Overfit: High (score 0.57)

Repeated-run reliability (not used by the gate): 5 paired runs (0W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add light/dark mode support using AppThemeBinding	Eligible	+0.0%	+0.0%	0/1/0
= Avoid common theming mistakes	Eligible	+0.0%	+0.0%	0/1/0
= Create custom themes with ResourceDictionary switching	Eligible	+0.0%	+0.0%	0/1/0
= Detect and respond to system theme changes	Eligible	+0.0%	+0.0%	0/1/0
= Swap theme dictionaries without destroying app styles	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add light/dark mode support using AppThemeBinding: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — msbuild-quality-review (claude-sonnet-5)

Why: Net win +50.0% (7W/1T/2L over 10 preference-eligible stimulus vote(s), sign test p=0.090), mean preference +20.0% across 10 paired run(s) — not credible (sign test p=0.090 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=10; 7W/1T/2L; d=9; p=0.090; net +50.0%

Repeated-run reliability (not used by the gate): 10 paired runs (7W/1T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Avoid findings when the changed file cannot be read	Eligible	-100.0%	-40.0%	0/0/1
▼ Identify a generated-file collision across projects and frameworks	Eligible	-100.0%	-100.0%	0/0/1
= Limit findings to the changed in-scope policy	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid findings when the changed file cannot be read: Both results are correct, schema-shaped noop proposals based solely on the supplied unavailable-read evidence. A is marginally stronger because its reason and limitations more fully document why no alternative evidence can support a review.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — template-authoring (claude-sonnet-5)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +20.0% across 7 paired run(s) — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%

Warnings: Activation: isolated 7/7; plugin 5/7

Overfit: Moderate (score 0.38)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a safe restore post-action	Eligible	+0.0%	+0.0%	0/1/0
▲ Author a reusable item template	Eligible	+100.0%	+40.0%	1/0/0
= Create template from existing project	Eligible	+0.0%	+0.0%	0/1/0
▼ Preserve central package management conventions	Eligible	-100.0%	-40.0%	0/0/1
= Restrict and classify an SDK-specific template	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add a safe restore post-action: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Mixed evidence — template-authoring (gpt-5.6-luna)

Why: Net win +0.0% (1W/5T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 7 paired run(s) — no improvement

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%

Warnings: Activation-only stop: plugin 1 failed run

Overfit: Moderate (score 0.42)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a choice parameter with conditional project content	Eligible	+0.0%	+0.0%	0/1/0
▼ Add a safe restore post-action	Eligible	-100.0%	-40.0%	0/0/1
= Author a reusable item template	Eligible	+0.0%	+0.0%	0/1/0
= Create template from existing project	Eligible	+0.0%	+0.0%	0/1/0
= Preserve central package management conventions	Eligible	+0.0%	+0.0%	0/1/0
= Restrict and classify an SDK-specific template	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add a choice parameter with conditional project content: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — template-comparison (claude-sonnet-5)

Why: Net win +42.9% (3W/4T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +25.7% across 7 paired run(s) — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Choose MVC or Razor Pages for an admin portal	Eligible	+0.0%	+0.0%	0/1/0
= Choose a Blazor or Razor Pages template	Eligible	+0.0%	+0.0%	0/1/0
= Choose between blazor and blazorwasm	Eligible	+0.0%	+0.0%	0/1/0
= Compare REST and gRPC service templates	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Choose MVC or Razor Pages for an admin portal: Position-swap inconsistent (forward: A, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — template-comparison (gpt-5.6-luna)

Why: Net win +14.3% (1W/6T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.7% across 7 paired run(s) — not credible — 6 of 7 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/6T/0L; d=1; p=0.500; net +14.3%

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/6T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Choose MVC or Razor Pages for an admin portal	Eligible	+0.0%	+0.0%	0/1/0
= Choose a Blazor or Razor Pages template	Eligible	+0.0%	+0.0%	0/1/0
= Choose between blazor and blazorwasm	Eligible	+0.0%	+0.0%	0/1/0
= Compare REST and gRPC service templates	Eligible	+0.0%	+0.0%	0/1/0
= Decide which template fits a background processing scenario	Eligible	+0.0%	+0.0%	0/1/0
= Select a test project template for an enterprise suite	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Choose MVC or Razor Pages for an admin portal: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — template-discovery (claude-sonnet-5)

Why: Net win +42.9% (3W/4T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +25.7% across 7 paired run(s) — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Moderate (score 0.40)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Find template for web API project	Eligible	+0.0%	+0.0%	0/1/0
= Inspect a single worker template before creation	Eligible	+0.0%	+0.0%	0/1/0
= Inspect web API template parameters	Eligible	+0.0%	+0.0%	0/1/0
▲ Preview project creation with dry run	Eligible	+100.0%	+40.0%	1/0/0
= Resolve ambiguous project intent to multiple candidates	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Find template for web API project: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — template-discovery (gpt-5.6-luna)

Why: Net win +0.0% (0W/7T/0L over 7 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 7 paired run(s) — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 0W/7T/0L; d=0; p=1.000; net +0.0%

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Moderate (score 0.48)

Repeated-run reliability (not used by the gate): 7 paired runs (0W/7T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Find installed test project template choices	Eligible	+0.0%	+0.0%	0/1/0
= Find template for web API project	Eligible	+0.0%	+0.0%	0/1/0
= Inspect a single worker template before creation	Eligible	+0.0%	+0.0%	0/1/0
= Inspect web API template parameters	Eligible	+0.0%	+0.0%	0/1/0
= Preview project creation with dry run	Eligible	+0.0%	+0.0%	0/1/0
= Resolve ambiguous project intent to multiple candidates	Eligible	+0.0%	+0.0%	0/1/0
= Search NuGet for specialized template	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Find installed test project template choices: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — template-instantiation (claude-sonnet-5)

Why: Net win +0.0% (0W/7T/0L over 7 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 7 paired run(s) — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 0W/7T/0L; d=0; p=1.000; net +0.0%

Overfit: Moderate (score 0.34)

Repeated-run reliability (not used by the gate): 7 paired runs (0W/7T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Adapt a generated test project to central package management	Eligible	+0.0%	+0.0%	0/1/0
= Compose an application and test solution	Eligible	+0.0%	+0.0%	0/1/0
= Create a console application	Eligible	+0.0%	+0.0%	0/1/0
= Create a controller-based web API	Eligible	+0.0%	+0.0%	0/1/0
= Create a worker service	Eligible	+0.0%	+0.0%	0/1/0
= Create and build a class library	Eligible	+0.0%	+0.0%	0/1/0
= Match a neighboring project's target framework	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Adapt a generated test project to central package management: Both runs produce the same correct CPM-converted xUnit project, preserve the pre-existing central version, and verify a clean restoring build. Run B avoids the template's initial automatic restore failure using --no-restore, but that is not a final-result quality advantage bec...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — template-instantiation (gpt-5.6-luna)

Why: Net win +42.9% (3W/4T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +17.1% across 7 paired run(s) — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%

Overfit: High (score 0.53)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Compose an application and test solution	Eligible	+0.0%	+0.0%	0/1/0
= Create a controller-based web API	Eligible	+0.0%	+0.0%	0/1/0
= Create a worker service	Eligible	+0.0%	+0.0%	0/1/0
= Match a neighboring project's target framework	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Compose an application and test solution: Both Response A and Response B successfully completed all requirements of the task with identical final results. Both created the solution file, class library, and test project at the correct paths; added the project reference from tests to library; added both projects to the ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

Details for 4 results were omitted to keep this comment under GitHub's limit. Open the workflow summary or Full Results for the complete breakdown.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1369 in dotnet/skills, download eval artifacts with gh run download 42175093041 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/36fe491315c3b07509cf26e828c8c9a24bee77f4/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

✅ Evaluation passed for 36fe491. cc @AbhitejJohn @JanKrivanek — please review.