Prepare affected-test selection rollout

azure-pipelines.yml

@@ -290,6 +290,10 @@ stages:
         condition: and(succeeded(), eq(variables._BuildConfig, 'Debug'), eq(variables['MSBuildCacheBuildSucceeded'], 'true'))
 
       - template: /eng/pipelines/steps/test-windows-debug-coverage.yml
+        parameters:
+          # Enable only after the prerequisites in docs/affected-test-selection.md are satisfied.
+          enableAffectedTests: false
+          affectedTestsMode: collect
 
     - task: PublishBuildArtifacts@1
       displayName: 'Publish cache seed build binlogs'
@@ -326,6 +330,8 @@ stages:
     - checkout: self
       fetchDepth: 0
       clean: false
+    - pwsh: ./eng/validate-affected-tests.ps1
+      displayName: Validate affected-test rollout
     - bash: |
         set -euo pipefail
 
@@ -710,6 +716,10 @@ stages:
         - ${{ if eq(parameters.SkipTests, False) }}:
 
           - template: /eng/pipelines/steps/test-windows-debug-coverage.yml
+            parameters:
+              # Enable only after the prerequisites in docs/affected-test-selection.md are satisfied.
+              enableAffectedTests: false
+              affectedTestsMode: run
 
           # Integration tests are redundant in Release—they spawn child processes that already cover
           # both Debug and Release configurations. Use --test-modules to run only unit tests.

docs/README.md

@@ -39,6 +39,7 @@ For technical reasoning and implementation details, you can refer to the list of
 
 ## Guides
 
+- [Affected-test selection rollout](affected-test-selection.md): disabled adoption layout and activation prerequisites.
 - [Testing WinUI apps](winui-testing.md): packaged (MSIX) vs unpackaged WinUI, and how the test host is started for each.
 
 ## Design notes

docs/affected-test-selection.md

@@ -0,0 +1,67 @@
+# Affected-test selection rollout
+
+This repository is prepared to adopt the experimental affected-test workflow from
+[dotnet/sdk#55574](https://github.com/dotnet/sdk/pull/55574). The workflow is Microsoft.Testing.Platform-only and
+builds on the composable filter-provider support from
+[testfx#10235](https://github.com/microsoft/testfx/pull/10235).
+
+The rollout is intentionally disabled. SDK `11.0.100-rc.1.26406.108` contains the affected-test commands but fails
+`dotnet tool restore` on clean agents because it predates
+[dotnet/sdk#55595](https://github.com/dotnet/sdk/pull/55595). The repository remains on the stable SDK until a fixed
+daily is published. The affected-test extension package and its public local-filesystem storage contract are also not
+available yet. Ordinary test commands therefore remain unchanged.
+
+## Prepared layout
+
+- `global.json` defines the repository-specific `test.affectedTests` change and instrumentation scopes.
+- The trusted main-branch Windows Debug test is the future `--collect-test-map` entry point.
+- The Windows Debug PR test is the future `--affected-tests` entry point.
+- Both pipeline call sites pass `enableAffectedTests: false`. The inactive template branches restore the map through
+  Azure Pipelines `Cache@2` and set `DOTNET_CLI_ENABLE_AFFECTED_TESTS=1` only for the affected-test commands.
+- `eng/validate-affected-tests.ps1` protects the disabled state and verifies that the public SDK gate and command names
+  do not drift.
+
+`DOTNET_CLI_TEST_AFFECTED_TESTS_MODE` is an SDK-to-extension authorization marker. Repository scripts and pipeline
+definitions must not set it.
+
+## Storage design
+
+The map should use the extension's local-filesystem provider rooted at
+`$(Pipeline.Workspace)\affected-test-map`. Azure Pipelines `Cache@2` transfers that directory between runs without
+credentials:
+
+- trusted main builds can restore the previous map and publish a new immutable cache entry;
+- PR and fork-PR builds can read the target branch's cache scope but cannot write to it;
+- the cache prefix includes its manual compatibility version, OS, architecture, and configuration;
+- the unique build ID suffix lets every successful main collection publish a new map;
+- prefix restore selects the newest compatible map.
+
+Azure Pipelines caches expire after seven days without activity. A cache miss is therefore an expected state, not a
+test failure: the PR lane runs the unchanged full test command. The same fallback runs when the extension rejects a
+missing, stale, or incompatible map, and scheduled or manual builds always keep full validation.
+
+Selected-test runs do not publish their partial coverage as the repository coverage report. Collection and full
+fallback runs still publish complete coverage.
+
+Pipeline artifacts should contain only non-secret diagnostics or a mapping snapshot suitable for troubleshooting.
+They are not the cross-run source of truth because artifact lookup and retention are tied to individual builds.
+
+The `storage` property is deliberately absent from `test.affectedTests` until the extension package publishes the exact
+local-filesystem provider schema. Adding an invented provider or path setting now would create configuration that
+cannot be validated.
+
+## Activation checklist
+
+1. Update `global.json` to an SDK newer than `11.0.100-rc.1.26406.108` that contains dotnet/sdk#55595, then validate
+   `dotnet tool restore` on a clean agent.
+2. Add the publicly available affected-test extension package through `Directory.Packages.props` and the test project
+   infrastructure, following the repository's normal dependency-flow and package-source policy.
+3. Add `test.affectedTests.storage` using the package's published local-filesystem schema and point it at the Pipeline
+   Cache directory.
+4. Update `affectedTestsCacheVersion` whenever the persisted map format or its compatibility dimensions change.
+5. Enable `collect` in the main-branch cache-seed call site and publish non-secret diagnostics as an Azure DevOps
+   artifact.
+6. After a compatible map exists, enable `run` in the PR call site. Keep the full test command available as an explicit
+   rollback by setting `enableAffectedTests` back to `false`.
+7. Validate a documentation-only change, a product change with a narrow affected set, a force-all change, a missing or
+   incompatible map, a fork PR without secrets, and a collection failure before making selection required.

eng/pipelines/steps/test-windows-debug-coverage.yml

@@ -3,22 +3,118 @@
 # Assumes the consuming job declares:
 #   - the matrix variable `_BuildConfig` (Debug or Release)
 #   - the job-level environment variables from eng/pipelines/variables/test-env-vars.yml
+parameters:
+- name: enableAffectedTests
+  type: boolean
+  default: false
+- name: affectedTestsMode
+  type: string
+  default: disabled
+  values:
+  - disabled
+  - collect
+  - run
+- name: affectedTestsCacheVersion
+  type: string
+  default: v1
+
 steps:
 # Because the build step is using -ci, restore is done in a local .packages directory.
 # NUGET_PACKAGES must point to that directory so test project evaluation imports the restored package props/targets.
 #
 # -p:TestingPlatformCaptureOutput=false streams each test executable's output to the AzDO console instead of
 # buffering it until failure, preserving live progress during long-running test sessions.
-- script: |
-    echo ##vso[task.setvariable variable=TestStepRan]true
-    dotnet test -c $(_BuildConfig) --no-build -bl:$(BUILD.SOURCESDIRECTORY)\artifacts\TestResults\$(_BuildConfig)\TestStep.binlog -p:UsingDotNetTest=true -p:TestingPlatformCaptureOutput=false
-  name: Test
-  displayName: Test
-  condition: and(succeeded(), eq(variables._BuildConfig, 'Debug'))
-  env:
-    # Secret variables are not automatically exposed to scripts. Fork PR builds do not receive this token,
-    # so report-azdo history queries no-op there; trusted branch builds exercise them end-to-end.
-    SYSTEM_ACCESSTOKEN: $(System.AccessToken)
+- ${{ if eq(parameters.enableAffectedTests, false) }}:
+  - script: |
+      echo ##vso[task.setvariable variable=PublishCoverageReport]true
+      dotnet test -c $(_BuildConfig) --no-build -bl:$(BUILD.SOURCESDIRECTORY)\artifacts\TestResults\$(_BuildConfig)\TestStep.binlog -p:UsingDotNetTest=true -p:TestingPlatformCaptureOutput=false
+    name: Test
+    displayName: Test
+    condition: and(succeeded(), eq(variables._BuildConfig, 'Debug'))
+    env:
+      # Secret variables are not automatically exposed to scripts. Fork PR builds do not receive this token,
+      # so report-azdo history queries no-op there; trusted branch builds exercise them end-to-end.
+      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
+
+# These branches are intentionally disabled at their call sites. They can be enabled only after the affected-test
+# extension package and its local-filesystem storage schema have flowed into this repository. The SDK owns
+# DOTNET_CLI_TEST_AFFECTED_TESTS_MODE; pipelines must not set it.
+- ${{ if and(eq(parameters.enableAffectedTests, true), eq(parameters.affectedTestsMode, 'collect')) }}:
+  - task: Cache@2
+    displayName: Restore and publish affected-test map
+    inputs:
+      key: '"affected-tests" | "${{ parameters.affectedTestsCacheVersion }}" | "$(Agent.OS)" | "$(Agent.OSArchitecture)" | "$(_BuildConfig)" | "$(Build.BuildId)"'
+      restoreKeys: |
+        "affected-tests" | "${{ parameters.affectedTestsCacheVersion }}" | "$(Agent.OS)" | "$(Agent.OSArchitecture)" | "$(_BuildConfig)"
+      path: '$(Pipeline.Workspace)\affected-test-map'
+      cacheHitVar: AffectedTestsMapCacheRestored
+    condition: and(succeeded(), eq(variables._BuildConfig, 'Debug'))
+
+  - script: |
+      echo ##vso[task.setvariable variable=PublishCoverageReport]true
+      dotnet test -c $(_BuildConfig) --no-build -bl:$(BUILD.SOURCESDIRECTORY)\artifacts\TestResults\$(_BuildConfig)\TestStep.binlog -p:UsingDotNetTest=true -p:TestingPlatformCaptureOutput=false --collect-test-map
+    name: Test
+    displayName: Test and collect affected-test map
+    condition: and(succeeded(), eq(variables._BuildConfig, 'Debug'))
+    env:
+      DOTNET_CLI_ENABLE_AFFECTED_TESTS: 1
+      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
+
+- ${{ if and(eq(parameters.enableAffectedTests, true), eq(parameters.affectedTestsMode, 'run')) }}:
+  - task: Cache@2
+    displayName: Restore affected-test map
+    inputs:
+      key: '"affected-tests" | "${{ parameters.affectedTestsCacheVersion }}" | "$(Agent.OS)" | "$(Agent.OSArchitecture)" | "$(_BuildConfig)" | "$(Build.BuildId)"'
+      restoreKeys: |
+        "affected-tests" | "${{ parameters.affectedTestsCacheVersion }}" | "$(Agent.OS)" | "$(Agent.OSArchitecture)" | "$(_BuildConfig)"
+      path: '$(Pipeline.Workspace)\affected-test-map'
+      cacheHitVar: AffectedTestsMapCacheRestored
+    condition: and(succeeded(), eq(variables._BuildConfig, 'Debug'), eq(variables['Build.Reason'], 'PullRequest'))
+
+  - pwsh: |
+      dotnet test -c $(_BuildConfig) --no-build -bl:$(BUILD.SOURCESDIRECTORY)\artifacts\TestResults\$(_BuildConfig)\TestStep.binlog -p:UsingDotNetTest=true -p:TestingPlatformCaptureOutput=false --affected-tests
+      $exitCode = $LASTEXITCODE
+      if ($exitCode -eq 0) {
+        Write-Host "##vso[task.setvariable variable=AffectedTestsSucceeded]true"
+        exit 0
+      }
+
+      if ($exitCode -in 2, 8) {
+        Write-Host "##vso[task.logissue type=error]Affected-test execution failed with exit code $exitCode."
+        exit $exitCode
+      }
+
+      Write-Host "##vso[task.logissue type=warning]Affected-test selection failed; running the full test suite."
+      exit 0
+    name: TestAffected
+    displayName: Test affected changes
+    condition: and(succeeded(), eq(variables._BuildConfig, 'Debug'), eq(variables['Build.Reason'], 'PullRequest'), ne(variables['AffectedTestsMapCacheRestored'], 'false'))
+    env:
+      DOTNET_CLI_ENABLE_AFFECTED_TESTS: 1
+      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
+
+  # Scheduled and manual runs keep their full-validation contract. PRs also fall back when the cache is absent or the
+  # extension rejects a missing, stale, or incompatible map.
+  - pwsh: |
+      Write-Host "##vso[task.setvariable variable=PublishCoverageReport]true"
+
+      $testResultsDirectory = "$(Build.SourcesDirectory)\artifacts\TestResults\$(_BuildConfig)"
+      if (Test-Path $testResultsDirectory) {
+        Get-ChildItem $testResultsDirectory -Filter *.coverage -File -ErrorAction SilentlyContinue |
+          Remove-Item -Force
+      }
+
+      dotnet test -c $(_BuildConfig) --no-build -bl:$(BUILD.SOURCESDIRECTORY)\artifacts\TestResults\$(_BuildConfig)\TestStep.binlog -p:UsingDotNetTest=true -p:TestingPlatformCaptureOutput=false
+      exit $LASTEXITCODE
+    name: Test
+    displayName: Test (affected-test fallback)
+    condition: and(succeeded(), eq(variables._BuildConfig, 'Debug'), or(ne(variables['Build.Reason'], 'PullRequest'), eq(variables['AffectedTestsMapCacheRestored'], 'false'), ne(variables['AffectedTestsSucceeded'], 'true')))
+    env:
+      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
+
+- ${{ if and(eq(parameters.enableAffectedTests, true), eq(parameters.affectedTestsMode, 'disabled')) }}:
+  - pwsh: throw "affectedTestsMode must be 'collect' or 'run' when enableAffectedTests is true."
+    displayName: Reject invalid affected-test configuration
 
 # Publish a portable, compact report separately from the large TestResults artifact so external consumers can
 # retrieve it without downloading every test result and binary .coverage file.
@@ -52,7 +148,7 @@ steps:
       }
 
       Write-Host "##vso[task.setvariable variable=CoverageReportReady]true"
-  condition: and(succeededOrFailed(), eq(variables._BuildConfig, 'Debug'), eq(variables['TestStepRan'], 'true'))
+  condition: and(succeededOrFailed(), eq(variables._BuildConfig, 'Debug'), eq(variables['PublishCoverageReport'], 'true'))
 
 - task: PublishBuildArtifacts@1
   displayName: 'Publish Cobertura coverage report'