[YuliiaKovalova]

Summary

Improve the consolidated dotnet/msbuild skill from observed losing answers, missed activation, and unnecessary work, while preserving the original evaluation's nine preference-eligible tasks, seven dormancy cases, three repeats, time budget, concurrency, and scoring thresholds.

No baseline win is claimed, and this head is not promotion-ready. The latest published content is c173383d5d4d28b217e56988c9d34c5fdef74e18. Its one new default-profile comparison is complete: Sonnet has valid but unproven 6W / 1T / 2L, p=0.14453125; Luna has 0W / 6T / 3L, p=0.125 and fails the framework-migration dormancy contract. Complete raw artifacts, not green workflow jobs, determine these conclusions.

• Keep required imports required, including through aggregation files; verify hook order and optional/required presence combinations.
• Distinguish property evaluation from item evaluation. Correct only the premature property assignment, preserve project-owned frameworks and working package items, and retain F# source order and actual SDK defaults.
• Route successful-but-stale build repairs explicitly. Prefer built-in membership-state tracking, distinguish path identities from content hashes, and test destructive cases in a disposable copy rather than reconstructing original source files.
• Answer settled XML/package/timing questions directly, without mandatory reference reads or checkout discovery. Resolve packed forwarders inside the package and preserve a valid no-change result.
• Compare old/new setting values under matched commands, input/output states, restore policy, instrumentation, and repeated measurements. Default to the original workload; do not substitute a no-op merely to obtain easier timings.
• Keep executable grading isolated from agent edits and host SDK/NuGet configuration. This content iteration changes only SKILL.md and three existing references; production stimuli, graders, fixtures, dependencies, and gate settings are unchanged from 6d24709.

The specialist plugin and unrelated work in #1311 are excluded. Current main's upstream reader/session-history repair through Vally 0.18 (#1309) is retained.

Evidence-led content iteration

The earlier losing-trial explanations included:

"its identity-only fingerprint is mischaracterized as detecting edits and its trajectory had to recreate a deleted original spec."

"proposes an architectural change (moving TargetFramework to props) as its primary solution"

The current entry puts those decisions before reference browsing: preserve framework ownership; move only the early property assignment; keep actual input files in Inputs; use WriteLinesToFile / WriteOnlyWhenDifferent for membership state; and preserve exact source bytes during verification.

Native SDK 10.0.403 evaluation-only probes confirmed that an early property condition misses a framework assigned in the project body while the conditional package item in the same import is already correct. Moving only the property restores FEATURE, with package version and F# source order unchanged. A globally supplied framework already satisfies the early condition. The standard F# SDK already sets EnableDefaultCompileItems=false; the skill does not adopt a judge's incorrect claim that adding that switch is a required repair. These probes did not restore or compile the project.

A separate native custom-target probe started with no inputs: the built-in task created an empty membership file and output, generation ran once, and the second unchanged Build preserved both timestamps and skipped generation.

Non-voting development diagnostics

All snapshots and results are retained separately. Before/after below means previous skill versus candidate skill, not no skill versus skill. These local trials do not add independent preference votes.

1. A matched six-task check exposed both gains and regressions. Both models answered the F# phase review correctly with only the skill call, avoiding reference reads. Both Sonnet incremental arms skipped activation and rewrote protected spec bytes, so their token difference is excluded from content-attributed savings. Across the five activated Sonnet pairs, tokens increased 34%, mainly from avoidable package inspection. Luna's six activated pairs passed and used 55% fewer tokens than the previous skill, not than no skill. Neither aggregate is final-candidate baseline-efficiency evidence.
2. A refined all-16 functional check passed 16/16 on Luna and 15/16 on Sonnet. Both stayed dormant on all seven exclusion cases. Sonnet now activated for incremental repair and produced correct build wiring, but its verification deleted original specs and recreated JSON without preserving the original trailing newline. The immutable-input grader correctly rejected this. Sonnet also skipped two other intended activations; favorable prompt grades did not erase those limitations.
3. Moving concrete byte-copy/finally/hash guidance from the shared footer to the repair decision point changed the actual behavior. A five-task follow-up passed 5/5 per family, with 4/4 intended activations and 1/1 assertion dormancy each. Both families tested deletions and empty-input behavior in disposable project copies and passed the real incremental and fixed-package consumer graders. Packed-import, F# and timing advice each used only the skill call, without reference or shell reads.
4. Manual inspection still found a passing Sonnet plan choosing a no-op/arbitrary edit rather than retaining the original workload. The final one-line A/B clarification was checked on the same existing advisory stimulus once per family: both activated and passed. Sonnet still proposed plain Build as its example, so consistent adherence to the original-workload default is not established. This is retained as a limitation, not hidden by the passing rubric or followed by unchanged-content reruns.

The final entry is 2,912 BPE tokens / 14,791 working-tree bytes, larger than the previous entry. The plugin validator's size warning is report-only. The goal is fewer unnecessary round trips and more correct decisions, not an asserted entry-size reduction or whole-task cost win.

Full paired evidence

Each distinct stimulus contributes one vote; repeated trials and model families are not pooled into extra votes.

Evaluated content	Sonnet wins / ties / losses; p	Luna wins / ties / losses; p	Result
4001fb3	4 / 5 / 0; 0.0625	3 / 5 / 1; 0.3125	Improvement unproven
2a43604	6 / 1 / 2; 0.14453125	4 / 4 / 1; 0.1875	Unproven; one Luna assertion-dormancy activation
6042e75	5 / 2 / 2; 0.2265625	1 / 6 / 2; 0.5	Unproven; all seven dormancy contracts passed on both
c173383	6 / 1 / 2; 0.14453125	0 / 6 / 3; 0.125	Sonnet unproven; Luna directionally worse and fails migration dormancy

The c173383 run is a single new-content comparison at the unchanged default profile: claude-sonnet-5 and gpt-5.6-luna, with their standard judges, baseline/isolated/plugin arms, and production repeats and budgets. It is bound to that exact PR commit, not the workflow's main-branch head_sha. It is not an unchanged-content rerun to seek significance.

Accounting is complete for both families: 288/288 trials, 48/48 in each family/arm, no errors, no unmatched comparisons, and no missing or duplicate repeat identities. Both canonical records are VALID_NO_CHANGE; Luna's stateReason is activation_contract_failed, not an environment or measurement failure.

Sonnet passed all seven isolated dormancy contracts. Intended-task activation was 25/27 isolated and 20/27 plugin, not the 9/9 and 8/9 case-level presence shown in the automated warning. The isolated arm passed all nine new-contract executions. The plugin arm passed eight of nine: one incremental repair never activated the skill, rewrote a protected spec during verification, and also stopped shared build servers. The native grader correctly rejected it. Source-preserving repair is therefore not established across the normal plugin menu, despite the successful activated development follow-up.

Luna activated on 27/27 intended trials in both isolated and plugin arms and passed all 27/27 new-contract executions across its three arms. However, the isolated skill also activated on the framework-migration request in two of three repeats, leaving 6/7 dormancy contracts satisfied. The full plugin stayed dormant on all 21 off-target trials. The migration answers remained migration plans, but that does not excuse violation of the explicit routing contract.

The two Sonnet preference losses remain advisory depth/restraint issues: the unsupported-speedup answer was less concisely scoped than baseline, and the dependency-wait answer omitted some compiler/analyzer/generator detail and controlled equivalent-build/output validation. Passing prompt grades are not presented as eliminating those shortcomings.

Luna's three preference losses concern capture overhead, clearer hook-repair reasoning and observed HookOrder evidence, and a more concrete A/B plan with explicit no-cache-purge/server-shutdown constraints. These are reported judge preferences, not failed native hook wiring. A separate incremental repeat passed native replay but failed the explanation rubric: the handoff listed correct outcomes without explaining why current-list timestamps forget removed/older inputs or how the membership file fixes that blind spot.

Actual target-task cost

Within each run, the table covers the same nine preference tasks times three repeats per arm. Dormancy and judge usage are excluded; cached input is not counted twice.

Latest c173383:

Executor	No-skill tokens	Isolated-skill tokens	Plugin tokens	Overhead, isolated / plugin
Sonnet	3,172,164	3,189,819	3,347,488	+0.6% / +5.5%
Luna	1,795,332	1,564,112	1,555,924	-12.9% / -13.3%

Sonnet made 160 / 141 / 147 executor requests in baseline / isolated / plugin target arms. This is less overhead than in the previous run, but still not a token-efficiency win over its own current baseline. Missed intended activations also confound attributing lower overhead to better activated workflows. Different runs' absolute token totals are not treated as a matched experiment.

Luna made 132 / 111 / 108 requests. Its target-task token savings are observed with full intended activation, but do not establish a quality win: the preference direction is negative and the isolated dormancy gate fails.

Previous full run, 6042e75:

Executor	No-skill tokens	Isolated-skill tokens	Plugin tokens	Overhead, isolated / plugin
Sonnet	2,309,183	3,647,410	4,272,444	+58% / +85%
Luna	1,431,589	1,704,840	1,707,482	+19% / +19%

Sonnet remains above its current no-skill token baseline. Luna's measured target-task savings do not override its quality/routing results or establish an across-model promotion. Development reference-read savings are narrower evidence and are not substituted for the full comparison.

Executable grading and review hardening

Earlier commits in this PR:

• Verify complete spec membership and immutable source bytes before and after each build; reject a target that rewrites the grader's input edit.
• Preserve the complete supplied feed and exact package bytes. Consume the fixed package rather than repacking sources; allow unrelated review output without weakening import checks.
• Stage authoritative checker, fixture hashes, and SDK policy only during disposable grading. Agent-written checker/baseline replacements cannot rebaseline changed inputs.
• Use fresh hook markers, allow valid aggregation alternatives, and accept explicit missing-hook failures without requiring one literal filename.
• Resolve real paths in CLI entry guards, cover Windows junction execution, and run contract regressions in CI.
• Isolate the fixture SDK policy from the preview authoring SDK and restore the disposable consumer with an explicit local-feed NuGet.Config.

At c173383, all 54 new-contract grades are accounted for: 18/18 hooks, 18/18 fixed-package consumers, and 17/18 incremental repairs passed. The 53 passing executions resolved SDK 10.0.401. The remaining Sonnet plugin repair was rejected for source mutation before SDK execution, not for SDK resolution or inherited NuGet mapping.

Raw artifacts exposed two host defects separately from content quality. Earlier runs had all 54 contract executions fail before replay because the selected SDK patch/preview was unavailable. The 10.0.100 minimum / latestMajor policy was verified in an owned host view exposing only 10.0.204, where all 47 then-existing tests and four production goldens passed. The later hosted run resolved SDK 10.0.401 in all 54 executions: all 36 hook/incremental replays passed, while inherited package-source mapping rejected all 18 package consumers.

Grader-only commit 6d24709 fixed that mapping dependency. Its hostile-mapping regression failed before the fix and passed afterward; all 18 unchanged recorded package submissions replayed successfully under the reproduced mapping without new model calls or votes. The two Luna "activation-only stop" warnings were package code-grader failures, not missing answers. No runtime continuation workaround was added.

Validation and provenance

• Current plugin/reference/frontmatter and whitespace checks pass. The description is 978 characters; the MSBuild size warning is disclosed above.
• Frozen candidate hashes match all 13 skill files; committed Markdown matches the snapshot after platform newline normalization. Fixture-byte comparisons remain exact and are not normalized.
• Native phase/default/source-order and initially-empty-state probes passed on SDK 10.0.403.
• The isolated PR worktree is used throughout; the original checkout's unrelated HEAD and staged tree are preserved.
• The unchanged grader code was last verified by hosted Linux workflow tests at 6d24709: all three jobs and 48/48 Node cases, no skips. Its four production goldens and hostile-mapping replay also passed. These are prior deterministic results, not newly claimed model outcomes at c173383.
• The current hosted comparison completed, but its auxiliary session-replay publisher reported a separate publishing failure. Downloaded model artifacts and verdicts are complete; publishing secrets were not changed.
• Earlier installed-reference diagnostics recorded 12/12 exact-path native reads per family with source hashes and native histories. They precede this revision of three reference documents; paths and reader dependencies are unchanged, but those old hashes are not presented as current-content smoke evidence.

Dependencies remain Vally 0.18 / Copilot SDK 1.0.14. The bundled runtime reports 0.0.0; that value is retained rather than interpreted as a released CLI version. Existing environment-alias deprecation and default-scoring warnings remain.

Related issue and checklist

Follow-up to #1241; no issue is closed.

• Preserved unrelated work, production stimuli, repeats, budgets, and thresholds.
• Kept plugin metadata, marketplace manifests, dependencies, and external domains unchanged.
• Classified fixture/host, activation, content, preservation, and cost problems separately.
• Retained unfavorable local results and real immutable-input failures.
• Kept golden acceptance, mutation rejection, no-change behavior, and dormancy coverage.
• Demonstrated statistically supported baseline improvement at the final head.
• Passed every explicit dormancy contract at the final head.
• Full normal-menu/coexistence promotion qualification.

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
⚠️	dotnet	setup-local-sdk	4/8	50%

Uncovered: dotnet/setup-local-sdk

• [Pitfall] dotnet app.dll wrong runtime (line 386)
• [CodePattern] [guid] (line 104)
• [CodePattern] [ordered] (line 301)
• [CodePattern] [pscustomobject] (line 301)

[Copilot]

🟡 Changes recommended

The graders can accept build-time changes to protected inputs and deletion or corruption of the supplied package.

2 open issues

• Snapshot spec content before build to detect in-build rewrites · New
• Verify supplied package artifact preservation in no-change scenario · New

What changed in this PR

This PR strengthens the consolidated MSBuild skill’s import and performance guidance, adds executable correctness contracts, and introduces a smoke test for installed-reference access through the logged SDK evaluation path.

Changes:

• Clarifies required-import behavior and controlled, workload-specific performance comparisons.
• Adds hook, incremental-generation, and package-consumer fixtures, graders, and regression tests.
• Adds exact-path reference-read checks with retained execution evidence and troubleshooting guidance.

File	Description
tests/​dotnet/​msbuild/​references/​incremental-membership.patch	Adds a golden input-membership repair.
tests/​dotnet/​msbuild/​references/​hook-composition.patch	Adds a golden optional/required import repair.
tests/​dotnet/​msbuild/​graders/​check-contracts.test.mjs	Tests golden repairs and broken mutations.
tests/​dotnet/​msbuild/​graders/​check-contracts.mjs	Implements executable correctness and preservation checks.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​src/​Producer/​Producer.csproj	Defines the package producer.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​src/​Producer/​Packaging.props	Configures external nuspec packaging.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​src/​Producer/​buildTransitive/​common/​Example.props	Supplies the package contract property.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​src/​Producer/​build/​net8.0/​Example.props	Imports the required packed asset.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​eng/​packaging/​Example.nuspec	Maps source assets to package paths.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​consumer/​Consumer.proj	Exercises restored package imports.
tests/​dotnet/​msbuild/​fixtures/​incremental/​Specs/​beta.json	Supplies a removable generator input.
tests/​dotnet/​msbuild/​fixtures/​incremental/​Specs/​alpha.json	Supplies an editable generator input.
tests/​dotnet/​msbuild/​fixtures/​incremental/​Generator.targets	Generates catalog content and records executions.
tests/​dotnet/​msbuild/​fixtures/​incremental/​Build.proj	Reproduces stale input-membership behavior.
tests/​dotnet/​msbuild/​fixtures/​hooks/​required.targets	Records required-hook import order.
tests/​dotnet/​msbuild/​fixtures/​hooks/​optional.targets.example	Supplies the optional-hook definition.
tests/​dotnet/​msbuild/​fixtures/​hooks/​Build.proj	Reproduces mixed-list import guarding.
tests/​dotnet/​msbuild/​eval.yaml	Adds executable contracts and a controlled timing scenario.
plugins/​dotnet/​skills/​msbuild/​SKILL.md	Strengthens import, timing, and reference-reading guidance.
plugins/​dotnet/​skills/​msbuild/​references/​extension-points.md	Clarifies optional versus required imports.
plugins/​dotnet/​skills/​msbuild/​references/​build-perf-baseline.md	Clarifies fair comparisons and scoped conclusions.
plugins/​dotnet/​README.md	Documents contracts and reference-access qualification.
eng/​vally-adapter/​InvestigatingResults.md	Documents the logged reference-access smoke.
eng/​evaluation-tools/​smoke-skill-references.test.mjs	Tests reference-read acceptance and rejection.
eng/​evaluation-tools/​smoke-skill-references.mjs	Implements exact-path reads and evidence retention.

_{Review effort: Balanced}

[github-actions]

👋 @YuliiaKovalova — this PR has 2 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[YuliiaKovalova]

Addressed all six additional findings in 7a20ca5:

1. Incremental grading captures expected contents and source hashes before each build, then rejects build-time spec or generator rewrites.
2. The complete supplied feed is protected by exact hashes, and the consumer uses the supplied package without repacking. Deletion, corruption, and feed removal are covered.
3. The checker and authoritative hashes/SDK policy are staged only during disposable grading. Agent-visible snapshot commands are removed; integration tests verify that agent-written checker/baseline replacements cannot rebaseline tampered fixtures.
4. Both entry guards compare real paths; actual Windows junction subprocess tests prevent silent no-op exits.
5. CI now installs the repository-selected SDK and runs the contract regressions alongside the evaluation-tool tests, with fixture and SDK-policy path triggers.
6. Fresh hook markers reject hard-coded HookOrder; benign extra review files are accepted; missing-hook failures do not require literal filename wording.

Local validation: 44 tests passed, all four production-oracle goldens passed (including the three executable graders), 111/111 eval specs validated, and the evaluation quality gate reported no errors. Remote CI is separate from these local results.

[Copilot]

🟡 Changes recommended

The incremental grader accepts added spec files, weakening preservation checks and allowing its empty-input scenario to remain nonempty.

1 open issue

• Detect extra Specs files before incremental replay · New

2 resolved since last review

• Verify supplied package artifact preservation in no-change scenario
• Snapshot spec content before build to detect in-build rewrites

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The executable grading boundary and logged SDK integration warrant final human review across platform and runtime boundaries.

No open issues 🎉

1 resolved since last review

• Detect extra Specs files before incremental replay

_{Review effort: Balanced}

[github-actions]

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 0 improved, ➖ 2 results without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 4001fb3258109fccb5fa7aa425803c584e1e7585; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
msbuild	claude-sonnet-5	➖ Improvement signal, tie-limited	n=9; 4W/5T/0L; d=4; p=0.063; net +44.4%; 7 dormancy excluded	🟡 0.41	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
msbuild	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=9; 3W/5T/1L; d=4; p=0.312; net +22.2%; 7 dormancy excluded	🟡 0.47	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Improvement signal, tie-limited — msbuild (claude-sonnet-5)

Why: Net win +44.4% (4W/5T/0L over 9 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +3.8% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 9 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 4W/5T/0L; d=4; p=0.063; net +44.4%; 7 dormancy excluded

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 48 paired runs (15W/24T/9L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Assess a supported workload-specific build improvement	Eligible	+0.0%	+0.0%	1/1/1
= Keep SDK installation separate from build analysis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep an npm failure outside MSBuild	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep failing assertions out of build troubleshooting	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep framework migration separate from build cleanup	Excluded (activation contract)	-33.3%	-33.3%	1/0/2
▼ Keep legacy project conversion with the migration workflow	Excluded (activation contract)	-66.7%	-46.7%	0/1/2
= Keep runtime CPU profiling out of build diagnosis	Excluded (activation contract)	+0.0%	+0.0%	1/1/1
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	0/3/0
= Reject an unsupported build speedup claim	Eligible	+0.0%	-20.0%	1/1/1
= Repair stale generation after input membership changes	Eligible	+0.0%	+0.0%	0/3/0
= Separate dependency waits from expensive compilation	Eligible	+0.0%	+0.0%	1/1/1

Illustrative judge evidence:

• Assess a supported workload-specific build improvement: Both outputs are high-quality, appropriately cautious, and satisfy all core requirements. A is slightly stronger overall because it keeps its caveats grounded in the given evidence and frames the no-checkout limitation constructively, whereas B adds an unsupported survivorship...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — msbuild (gpt-5.6-luna)

Why: Net win +22.2% (3W/5T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +1.3% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 9 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 3W/5T/1L; d=4; p=0.312; net +22.2%; 7 dormancy excluded

Overfit: Moderate (score 0.47)

Repeated-run reliability (not used by the gate): 48 paired runs (4W/40T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Assess a supported workload-specific build improvement	Eligible	+0.0%	+0.0%	0/3/0
= Diagnose a failed build from its only surviving artifact	Eligible	+0.0%	+0.0%	0/3/0
= Keep SDK installation separate from build analysis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep a C sharp rename out of build authoring	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep an npm failure outside MSBuild	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep failing assertions out of build troubleshooting	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep framework migration separate from build cleanup	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep legacy project conversion with the migration workflow	Excluded (activation contract)	+0.0%	+0.0%	1/1/1
= Keep runtime CPU profiling out of build diagnosis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Leave a valid packed import contract unchanged	Eligible	+0.0%	+0.0%	0/3/0
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	0/3/0
= Reject an unsupported build speedup claim	Eligible	+0.0%	+0.0%	0/3/0
▼ Separate dependency waits from expensive compilation	Eligible	-33.3%	-13.3%	0/2/1

Illustrative judge evidence:

• Assess a supported workload-specific build improvement: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1348 in dotnet/skills, download eval artifacts with gh run download 39805673893 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/4001fb3258109fccb5fa7aa425803c584e1e7585/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

✅ Evaluation passed for 4001fb3. cc @AbhitejJohn @JanKrivanek @webreidi @dotnet/msbuild @YuliiaKovalova — please review.

[github-actions]

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 0 improved, ➖ 1 result without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 1 activation contract failure, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 2a4360407051ad94e24bbeabd18c2d915401251e; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
msbuild	claude-sonnet-5	➖ Improvement signal, unproven	n=9; 6W/1T/2L; d=8; p=0.145; net +44.4%; 7 dormancy excluded	—	Activation: isolated 9/9; plugin 8/9	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
msbuild	gpt-5.6-luna	⛔ Activation contract failed	n=9; 4W/4T/1L; d=5; p=0.188; net +33.3%; 7 dormancy excluded	—	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — msbuild (gpt-5.6-luna)

Why: Net win +33.3% (4W/4T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +6.7% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=9; 4W/4T/1L; d=5; p=0.188; net +33.3%; 7 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Repeated-run reliability (not used by the gate): 48 paired runs (9W/38T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Capture a failing invocation without repairing it	Eligible	+0.0%	+0.0%	0/3/0
= Keep SDK installation separate from build analysis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep failing assertions out of build troubleshooting	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep framework migration separate from build cleanup	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep legacy project conversion with the migration workflow	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Leave a valid packed import contract unchanged	Eligible	+0.0%	+0.0%	0/3/0
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	0/3/0
▼ Reject an unsupported build speedup claim	Eligible	-33.3%	-13.3%	0/2/1
= Separate dependency waits from expensive compilation	Eligible	+0.0%	+0.0%	0/3/0

Illustrative judge evidence:

• Capture a failing invocation without repairing it: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — msbuild (claude-sonnet-5)

Why: Net win +44.4% (6W/1T/2L over 9 preference-eligible stimulus vote(s), sign test p=0.145), mean preference +14.6% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.145 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 6W/1T/2L; d=8; p=0.145; net +44.4%; 7 dormancy excluded

Warnings: Activation: isolated 9/9; plugin 8/9

Repeated-run reliability (not used by the gate): 48 paired runs (21W/19T/8L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Keep SDK installation separate from build analysis	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
▼ Keep a C sharp rename out of build authoring	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
▼ Keep an npm failure outside MSBuild	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
▼ Keep legacy project conversion with the migration workflow	Excluded (activation contract)	-33.3%	-13.3%	1/0/2
= Keep runtime CPU profiling out of build diagnosis	Excluded (activation contract)	+0.0%	+0.0%	1/1/1
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	0/3/0
▼ Reject an unsupported build speedup claim	Eligible	-33.3%	-13.3%	0/2/1
▲ Repair stale generation after input membership changes	Eligible	+33.3%	+13.3%	1/2/0
▼ Separate dependency waits from expensive compilation	Eligible	-33.3%	-13.3%	0/2/1

Illustrative judge evidence:

• Preserve a new hook when an optional predecessor is absent: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1348 in dotnet/skills, download eval artifacts with gh run download 39963851868 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/2a4360407051ad94e24bbeabd18c2d915401251e/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 0 improved, ➖ 2 results without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 6042e75ae40e51da6e54bef62af61e8cdf49f82d; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
msbuild	claude-sonnet-5	➖ Improvement signal, unproven	n=9; 5W/2T/2L; d=7; p=0.227; net +33.3%; 7 dormancy excluded	—	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
msbuild	gpt-5.6-luna	➖ Baseline signal, tie-limited	n=9; 1W/6T/2L; d=3; p=0.500; net -11.1%; 7 dormancy excluded	—	Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Improvement signal, unproven — msbuild (claude-sonnet-5)

Why: Net win +33.3% (5W/2T/2L over 9 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +3.3% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.227 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 5W/2T/2L; d=7; p=0.227; net +33.3%; 7 dormancy excluded

Repeated-run reliability (not used by the gate): 48 paired runs (13W/26T/9L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Keep SDK installation separate from build analysis	Excluded (activation contract)	-66.7%	-26.7%	0/1/2
▼ Keep a C sharp rename out of build authoring	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep failing assertions out of build troubleshooting	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep framework migration separate from build cleanup	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep legacy project conversion with the migration workflow	Excluded (activation contract)	-66.7%	-46.7%	0/1/2
▼ Keep runtime CPU profiling out of build diagnosis	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Leave a valid packed import contract unchanged	Eligible	+0.0%	+0.0%	0/3/0
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	0/3/0
▼ Reject an unsupported build speedup claim	Eligible	-66.7%	-26.7%	0/1/2
▼ Repair stale generation after input membership changes	Eligible	-33.3%	-13.3%	0/2/1

Illustrative judge evidence:

• Leave a valid packed import contract unchanged: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Baseline signal, tie-limited — msbuild (gpt-5.6-luna)

Why: Net win -11.1% (1W/6T/2L over 9 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -1.7% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 1W/6T/2L; d=3; p=0.500; net -11.1%; 7 dormancy excluded

Warnings: Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 1 failed run

Repeated-run reliability (not used by the gate): 48 paired runs (4W/38T/6L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Assess a supported workload-specific build improvement	Eligible	+0.0%	+0.0%	0/3/0
= Capture a failing invocation without repairing it	Eligible	+0.0%	+0.0%	0/3/0
= Diagnose a failed build from its only surviving artifact	Eligible	+0.0%	+0.0%	0/3/0
= Keep SDK installation separate from build analysis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep a C sharp rename out of build authoring	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep an npm failure outside MSBuild	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep failing assertions out of build troubleshooting	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
▼ Keep framework migration separate from build cleanup	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep legacy project conversion with the migration workflow	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▲ Leave a valid packed import contract unchanged	Eligible	+33.3%	+13.3%	1/2/0
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	1/1/1
= Reject an unsupported build speedup claim	Eligible	+0.0%	+0.0%	1/1/1
▼ Repair stale generation after input membership changes	Eligible	-33.3%	-13.3%	0/2/1
▼ Review framework conditions without breaking valid F sharp items	Eligible	-33.3%	-13.3%	0/2/1
= Separate dependency waits from expensive compilation	Eligible	+0.0%	+0.0%	0/3/0

Illustrative judge evidence:

• Assess a supported workload-specific build improvement: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1348 in dotnet/skills, download eval artifacts with gh run download 39973929027 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/6042e75ae40e51da6e54bef62af61e8cdf49f82d/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[YuliiaKovalova]

Further iteration is published through 6d24709: direct advisory answers, matched old/new workload guidance, assertion-versus-build routing, and critical-path attribution.

Baseline improvement is still unproven. The latest full skill-content run at 6042e75 reports Sonnet 5W/2T/2L, p=0.2265625, and Luna 1W/6T/2L, p=0.5. Both pass all seven dormancy contracts. Target-task token totals remain above the no-skill baseline; both isolated and plugin costs are now disclosed in the PR body.

All 54 hosted command graders selected SDK10.0.401; all 36 hook/incremental replays passed. The 18 package failures were instead inherited host NuGet source mapping. The final grader-only repair uses an explicit local-feed restore config. All 18 actual unchanged submissions now pass local Windows/SDK10.0.402 replay through production grading staging under reproduced hostile mapping, without new model calls or preference votes. The hosted Linux regression also passes.

The two Luna activation-only warnings are not continuation stops here: both answers are substantive correct no-change explanations with passing prompt grades. The failed package code grader caused the misleading classification; no runtime workaround was added. Final-head SKILL.md and eval.yaml are identical to the measured 6042e75 content. The deterministic replays do not replace the official preference verdict.

[github-actions]

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 0 improved, ➖ 1 result without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 1 activation contract failure, 📉 0 preference losses (report only).

Measurement identity: evaluated commit c173383d5d4d28b217e56988c9d34c5fdef74e18; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
msbuild	claude-sonnet-5	➖ Improvement signal, unproven	n=9; 6W/1T/2L; d=8; p=0.145; net +44.4%; 7 dormancy excluded	—	Activation: isolated 9/9; plugin 8/9	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
msbuild	gpt-5.6-luna	⛔ Activation contract failed	n=9; 0W/6T/3L; d=3; p=0.125; net -33.3%; 7 dormancy excluded	—	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — msbuild (gpt-5.6-luna)

Why: Net win -33.3% (0W/6T/3L over 9 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +0.0% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=9; 0W/6T/3L; d=3; p=0.125; net -33.3%; 7 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Repeated-run reliability (not used by the gate): 48 paired runs (4W/40T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Assess a supported workload-specific build improvement	Eligible	+0.0%	+0.0%	0/3/0
▼ Capture a failing invocation without repairing it	Eligible	-33.3%	-13.3%	0/2/1
= Diagnose a failed build from its only surviving artifact	Eligible	+0.0%	+0.0%	0/3/0
= Keep SDK installation separate from build analysis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep a C sharp rename out of build authoring	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep an npm failure outside MSBuild	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
▲ Keep framework migration separate from build cleanup	Excluded (activation contract)	+66.7%	+26.7%	2/1/0
= Keep runtime CPU profiling out of build diagnosis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Leave a valid packed import contract unchanged	Eligible	+0.0%	+0.0%	0/3/0
▼ Preserve a new hook when an optional predecessor is absent	Eligible	-33.3%	-13.3%	0/2/1
▼ Reject an unsupported build speedup claim	Eligible	-33.3%	-13.3%	0/2/1
= Repair stale generation after input membership changes	Eligible	+0.0%	+0.0%	0/3/0
= Review framework conditions without breaking valid F sharp items	Eligible	+0.0%	+0.0%	0/3/0
= Separate dependency waits from expensive compilation	Eligible	+0.0%	+0.0%	0/3/0

Illustrative judge evidence:

• Assess a supported workload-specific build improvement: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — msbuild (claude-sonnet-5)

Why: Net win +44.4% (6W/1T/2L over 9 preference-eligible stimulus vote(s), sign test p=0.145), mean preference +11.3% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.145 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 6W/1T/2L; d=8; p=0.145; net +44.4%; 7 dormancy excluded

Warnings: Activation: isolated 9/9; plugin 8/9

Repeated-run reliability (not used by the gate): 48 paired runs (20W/17T/11L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Assess a supported workload-specific build improvement	Eligible	+0.0%	+0.0%	1/1/1
= Keep SDK installation separate from build analysis	Excluded (activation contract)	+0.0%	+0.0%	1/1/1
▼ Keep a C sharp rename out of build authoring	Excluded (activation contract)	-66.7%	-26.7%	0/1/2
▼ Keep an npm failure outside MSBuild	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep legacy project conversion with the migration workflow	Excluded (activation contract)	+0.0%	+0.0%	1/1/1
▼ Keep runtime CPU profiling out of build diagnosis	Excluded (activation contract)	-66.7%	-26.7%	0/1/2
▲ Preserve a new hook when an optional predecessor is absent	Eligible	+66.7%	+26.7%	2/1/0
▼ Reject an unsupported build speedup claim	Eligible	-33.3%	-13.3%	0/2/1
▼ Separate dependency waits from expensive compilation	Eligible	-66.7%	-26.7%	0/1/2

Illustrative judge evidence:

• Assess a supported workload-specific build improvement: Both final answers are correct, well-scoped, and satisfy every rubric item. A is modestly stronger overall because it provides a clearer reporting formulation and a substantially more complete yet relevant discussion of measurement limits; B is concise and accurate but less in...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1348 in dotnet/skills, download eval artifacts with gh run download 40328576800 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/c173383d5d4d28b217e56988c9d34c5fdef74e18/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[YuliiaKovalova]

Published the next content refinement in c173383: property/item evaluation phases and framework ownership, built-in membership tracking, byte-preserving verification in disposable copies, optional reference reads, and matched original-workload comparisons. Production stimuli, graders, repeats, budgets, and thresholds are unchanged.

The single full comparison is complete, with all 288 trials accounted for:

Family	Distinct-task W / T / L	Result
Sonnet	6 / 1 / 2; p=0.14453125	Positive but unproven; 7/7 dormancy contracts passed
Luna	0 / 6 / 3; p=0.125	Directionally worse; migration dormancy failed in 2/3 isolated repeats

This head is not yet a demonstrated baseline improvement or promotion-ready. Actual intended activation was Sonnet 25/27 isolated and 20/27 plugin, versus Luna 27/27 in both arms. One unactivated Sonnet plugin repair modified protected source bytes and was correctly rejected; the other 53/54 new-contract grades passed. Luna's approximately 13% lower target-task token use does not cancel its preference/routing results. Complete costs, losing-case explanations, native results, and prior unfavorable diagnostics are retained in the PR description.