Skip to content

Auto-generated execution tests, found by mutation testing Mesa - #4720

Merged
alan-baker merged 8 commits into
gpuweb:mainfrom
ambergorzynski:mutation-tests
Nov 12, 2026
Merged

alan-baker merged 8 commits into
gpuweb:mainfrom
ambergorzynski:mutation-tests

Conversation

@ambergorzynski

Copy link
Copy Markdown
Contributor

Issue: #4299

@afd and I have been working on generating CTS tests through mutation testing, which we discussed some time ago in #4299. This draft proposes a first batch of five tests, so we can agree on their form before generating more.

Recap

The idea: we deliberately mutate (i.e. introduce a small bug into) part of a WebGPU implementation or a downstream driver, and run the CTS against the mutated version. If no test fails, the CTS has a gap: it doesn't detect that change. We then use a WGSL fuzzer to find a shader that does fail on the mutated code, and propose adding it as a test, so that the CTS will catch future bugs in that part of the code.

Each test is a small fuzzer-generated program that the unmodified compiler runs correctly and the mutated one does not. For these five tests:

  1. We mutated Mesa's NIR shader compiler (through llvmpipe/lavapipe, run with Dawn) and ran the CTS against each mutant.
  2. For mutants that no CTS test detected, we used the WGSL fuzzer WGSLsmith to find programs whose output changes when the mutant is present.
  3. We reduced each program automatically, to what we hope is a reasonable size.

In the earlier discussion, @dneto0 made some helpful suggestions that we have incorporated here: the tests live in a separate bucket, organised by the implementation that was mutated, and each one has a machine-readable label saying what was mutated and where.

What the tests look like

The tests are in src/webgpu/mutation/mesa/, in two files: no_input.spec.ts (shaders without an input buffer) and uniform_input.spec.ts (shaders that read a uniform buffer). Each test runs its shader once and compares the whole output buffer with the expected bytes. For example:

kMutations['shader_10c2ef9c'] = [
  { package: 'dawn', version: '9de0fd67086127228e8aa28e69616da37220cfc3' },
  {
    package: 'mesa',
    version: 'ba8eaab4f07e33c0b74fa92c60852cba2518bf2e',
    mutated: {
      file: 'src/compiler/nir/nir_constant_expressions.c',
      line: 50895,
      generatedFrom: 'src/compiler/nir/nir_constant_expressions.py',
    },
  },
];
g.test('shader_10c2ef9c')
  .desc(
    `A fuzzer-generated compute shader. Found by mutation testing a constant folding compiler pass.`
  )
  .fn(t => {
    runGeneratedTest(
      t,
      {
        // s_output: c = 4
        output: new Uint8Array([4, 0, 0, 0]),
      },
      `
struct StorageBuffer {
    c: i32,
}
...
`
    );
  });

As suggested:

  • They are in a separate bucket, organised by the implementation that was mutated (webgpu:mutation,mesa,*).
  • Each test has a machine-readable label, kMutations: git commits and the mutated source location. (generatedFrom marks a file that Mesa generates during the build.) The Mutation type is shared, in src/webgpu/mutation/mutation_info.ts.
Test Mutated Mesa source
shader_10c2ef9c nir_constant_expressions.c, line 50895
shader_101299f4 nir_constant_expressions.c, line 48527
shader_7ea91b04 nir_range_analysis.c, line 1178
shader_429a8325 nir_constant_expressions.c, line 51437
shader_7be24930 nir_opt_if.c, line 839

Questions

  • Where should the bucket live? We used a subdirectory, webgpu:mutation,mesa,*, as suggested by @dneto0. The alternative is a separate top-level tree, webgpu-mutation:*; the earlier discussion left this open.
  • Is the kMutations format useful? Each entry is written just above its test and collected in a table each file exports. Should it contain more or less detail, e.g. the mutation itself rather than just its line?
  • How much should each program be simplified? Further simplification is possible, but gets harder to scale hundreds of tests.
  • Bytes or typed values? Buffer contents are byte arrays, as the fuzzer reports them, with a comment giving the values. We could instead write them as values of their WGSL types, member by member (e.g. new Float32Array([1, 1, 0, 0, 10, 10]) for a mat3x2<f32>), which is easier to read. Would you prefer the typed form?
  • Padding: The output buffer starts zero-filled, and padding bytes must still be 0 afterwards. Should we pre-fill with a sentinel instead, as padding.spec.ts does? And if we move to typed values, should padding still be checked, or only the struct members?
  • Test names: a hash of the shader text (shader_<8 hex digits>), with a description naming the area of the mutation. Is there a naming scheme you'd prefer?

+@afd

@alan-baker alan-baker left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this seems good to me. The tests are quite simple with minimal runner code. Left a few comments.

For future expansion are there plans for how to integrate language features and enable extensions?

Comment thread src/webgpu/mutation/mutation_info.ts Resolved
@alan-baker

Copy link
Copy Markdown
Contributor

To answer the questions in the description:

Where should the bucket live? We used a subdirectory, webgpu:mutation,mesa,, as suggested by @dneto0. The alternative is a separate top-level tree, webgpu-mutation:; the earlier discussion left this open.

I think where they are is ok. The other place I would consider is webgpu,shader,execution,mutation,*.

Is the kMutations format useful? Each entry is written just above its test and collected in a table each file exports. Should it contain more or less detail, e.g. the mutation itself rather than just its line?
How much should each program be simplified? Further simplification is possible, but gets harder to scale hundreds of tests.

I think I commented about this, but I think we should consider more documentation and maybe some alterations to the structure. Good to have the information though.

Bytes or typed values? Buffer contents are byte arrays, as the fuzzer reports them, with a comment giving the values. We could instead write them as values of their WGSL types, member by member (e.g. new Float32Array([1, 1, 0, 0, 10, 10]) for a mat3x2), which is easier to read. Would you prefer the typed form?
Padding: The output buffer starts zero-filled, and padding bytes must still be 0 afterwards. Should we pre-fill with a sentinel instead, as padding.spec.ts does? And if we move to typed values, should padding still be checked, or only the struct members?

I'm ok with either. Typed is easier to read, but the comments list those values. So no strong opinions. It's ok to check the padding is untouched since WGSL promises that.

Test names: a hash of the shader text (shader_<8 hex digits>), with a description naming the area of the mutation. Is there a naming scheme you'd prefer?

These threw me at first, but I could think of a better naming scheme. The nature of mutation testing doesn't really lend itself to being able to provide a good descriptive name for the test. Maybe you could base on the change you made to Mesa, but not that would be any more insightful if someone sees one of these fail.

@ambergorzynski

Copy link
Copy Markdown
Contributor Author

Thanks for the review! I’ve addressed the inline comments, and answered the rest here:

  • Language features / extensions: Our natural next step would be to mutate Dawn (specifically Tint) directly instead of Mesa. In terms of additional WGSL language features / extensions we can play around with WGSLsmith's generation - it would be very helpful to get some guidance from you on which areas would be the most useful or interesting to explore.
  • Location: Moved to webgpu,shader,execution,mutation,mesa,*. Agreed, it fits better alongside the other shader execution tests.
  • kMutations: I’ve expanded the docs and restructured the type (see the inline thread). Let me know if you’d like other structural changes.
  • Simplification, bytes vs typed values, padding: I’ve kept these as they are for now.
  • Naming: Agreed that the hashes aren’t particularly meaningful, but I also struggled to come up with something better. I worry that if we include the Mesa details in the name it could become outdated as Mesa changes. If a test fails, people will see it’s under shader,execution,mutation,mesa which perhaps gives enough context at first? They can then look at the test for the full mutation info if needed, including the location of the change and the commit.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Results for build job (at b9445ec):

(No pr-comment-body.txt artifact found from this job.)

@alan-baker alan-baker left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes. The only substantial comment is the suggestion to restructure the test data a bit.

In terms of additional WGSL language features / extensions we can play around with WGSLsmith's generation - it would be very helpful to get some guidance from you on which areas would be the most useful or interesting to explore.

Most of the language features would be good to target, but I'd suggest focusing on buffer_view first. I think there'd be a lot of good ground to cover with that feature.

Location: Moved to webgpu,shader,execution,mutation,mesa,*. Agreed, it fits better alongside the other shader execution tests.

+1, I like the new location.

kMutations: I’ve expanded the docs and restructured the type (see the inline thread). Let me know if you’d like other structural changes.

The structure looks good to me now.

Naming: Agreed that the hashes aren’t particularly meaningful, but I also struggled to come up with something better. I worry that if we include the Mesa details in the name it could become outdated as Mesa changes. If a test fails, people will see it’s under shader,execution,mutation,mesa which perhaps gives enough context at first? They can then look at the test for the full mutation info if needed, including the location of the change and the commit.

I think this is partially addressed by the restructuring I proposed, but I don't think we need to worry too much about the actual case names.

Move the shader, buffer data and mutation info for each test into a
single kTestData table, and run them as cases of one test per file.

@dneto0 dneto0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks very much for the contribution!

It all looks sensible to me.

Re: placement in the tree. I think it's ok where it is. If the suite is reasonably quick then nobody should complain. Putting them in the middle of other execution tests means everyone will run them by default, which means better coverage for everyone.

LGTM. I'll leave final approval to @alan-baker

@alan-baker alan-baker left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes. Feel free to update the branch and mark this as ready and I can merge it.

If you have more tests waiting please contribute them in reasonable sized batches to make reviewing simpler (e.g. 10 - 20 tests at a time).

@ambergorzynski
ambergorzynski marked this pull request as ready for review November 11, 2026 11:32
@ambergorzynski

Copy link
Copy Markdown
Contributor Author

Great thanks! Have updated and marked as ready. I'll add more tests in small batches shortly.

@alan-baker alan-baker changed the title Draft for discussion: auto-generated execution tests, found by mutation testing Mesa Auto-generated execution tests, found by mutation testing Mesa Nov 12, 2026
@alan-baker
alan-baker enabled auto-merge (squash) November 12, 2026 14:46
@alan-baker
alan-baker merged commit 73e5dc9 into gpuweb:main Nov 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants