Repository navigation
Auto-generated execution tests, found by mutation testing Mesa - #4720
Conversation
alan-baker
left a comment
There was a problem hiding this comment.
Overall this seems good to me. The tests are quite simple with minimal runner code. Left a few comments.
For future expansion are there plans for how to integrate language features and enable extensions?
|
To answer the questions in the description:
I think where they are is ok. The other place I would consider is
I think I commented about this, but I think we should consider more documentation and maybe some alterations to the structure. Good to have the information though.
I'm ok with either. Typed is easier to read, but the comments list those values. So no strong opinions. It's ok to check the padding is untouched since WGSL promises that.
These threw me at first, but I could think of a better naming scheme. The nature of mutation testing doesn't really lend itself to being able to provide a good descriptive name for the test. Maybe you could base on the change you made to Mesa, but not that would be any more insightful if someone sees one of these fail. |
|
Thanks for the review! I’ve addressed the inline comments, and answered the rest here:
|
alan-baker
left a comment
There was a problem hiding this comment.
Thanks for the changes. The only substantial comment is the suggestion to restructure the test data a bit.
In terms of additional WGSL language features / extensions we can play around with WGSLsmith's generation - it would be very helpful to get some guidance from you on which areas would be the most useful or interesting to explore.
Most of the language features would be good to target, but I'd suggest focusing on buffer_view first. I think there'd be a lot of good ground to cover with that feature.
Location: Moved to webgpu,shader,execution,mutation,mesa,*. Agreed, it fits better alongside the other shader execution tests.
+1, I like the new location.
kMutations: I’ve expanded the docs and restructured the type (see the inline thread). Let me know if you’d like other structural changes.
The structure looks good to me now.
Naming: Agreed that the hashes aren’t particularly meaningful, but I also struggled to come up with something better. I worry that if we include the Mesa details in the name it could become outdated as Mesa changes. If a test fails, people will see it’s under shader,execution,mutation,mesa which perhaps gives enough context at first? They can then look at the test for the full mutation info if needed, including the location of the change and the commit.
I think this is partially addressed by the restructuring I proposed, but I don't think we need to worry too much about the actual case names.
Move the shader, buffer data and mutation info for each test into a single kTestData table, and run them as cases of one test per file.
dneto0
left a comment
There was a problem hiding this comment.
Thanks very much for the contribution!
It all looks sensible to me.
Re: placement in the tree. I think it's ok where it is. If the suite is reasonably quick then nobody should complain. Putting them in the middle of other execution tests means everyone will run them by default, which means better coverage for everyone.
LGTM. I'll leave final approval to @alan-baker
alan-baker
left a comment
There was a problem hiding this comment.
Thanks for the changes. Feel free to update the branch and mark this as ready and I can merge it.
If you have more tests waiting please contribute them in reasonable sized batches to make reviewing simpler (e.g. 10 - 20 tests at a time).
|
Great thanks! Have updated and marked as ready. I'll add more tests in small batches shortly. |
Issue: #4299
@afd and I have been working on generating CTS tests through mutation testing, which we discussed some time ago in #4299. This draft proposes a first batch of five tests, so we can agree on their form before generating more.
Recap
The idea: we deliberately mutate (i.e. introduce a small bug into) part of a WebGPU implementation or a downstream driver, and run the CTS against the mutated version. If no test fails, the CTS has a gap: it doesn't detect that change. We then use a WGSL fuzzer to find a shader that does fail on the mutated code, and propose adding it as a test, so that the CTS will catch future bugs in that part of the code.
Each test is a small fuzzer-generated program that the unmodified compiler runs correctly and the mutated one does not. For these five tests:
In the earlier discussion, @dneto0 made some helpful suggestions that we have incorporated here: the tests live in a separate bucket, organised by the implementation that was mutated, and each one has a machine-readable label saying what was mutated and where.
What the tests look like
The tests are in
src/webgpu/mutation/mesa/, in two files:no_input.spec.ts(shaders without an input buffer) anduniform_input.spec.ts(shaders that read a uniform buffer). Each test runs its shader once and compares the whole output buffer with the expected bytes. For example:As suggested:
webgpu:mutation,mesa,*).kMutations: git commits and the mutated source location. (generatedFrommarks a file that Mesa generates during the build.) TheMutationtype is shared, insrc/webgpu/mutation/mutation_info.ts.shader_10c2ef9cnir_constant_expressions.c, line 50895shader_101299f4nir_constant_expressions.c, line 48527shader_7ea91b04nir_range_analysis.c, line 1178shader_429a8325nir_constant_expressions.c, line 51437shader_7be24930nir_opt_if.c, line 839Questions
webgpu:mutation,mesa,*, as suggested by @dneto0. The alternative is a separate top-level tree,webgpu-mutation:*; the earlier discussion left this open.kMutationsformat useful? Each entry is written just above its test and collected in a table each file exports. Should it contain more or less detail, e.g. the mutation itself rather than just its line?new Float32Array([1, 1, 0, 0, 10, 10])for amat3x2<f32>), which is easier to read. Would you prefer the typed form?padding.spec.tsdoes? And if we move to typed values, should padding still be checked, or only the struct members?shader_<8 hex digits>), with a description naming the area of the mutation. Is there a naming scheme you'd prefer?+@afd