Add a way to run the reference tests from a PR label - #21918
Conversation
Adding the `browsertest` label to a PR dispatches a run to the private mozilla/pdf.js.pdfs repository, which runs the Firefox reference tests on Linux and Windows, sharded across several machines, and posts the results (with a link to the reftest analyzer hosted on mozilla/pdf.js.refs) as a comment. Every push to master dispatches a reference update in the same way. The test runner gets a `--shard=k/N` option to split the manifest across machines and a `--summaryFile` option to write the final counts as JSON.
calixteman
added
test
To Review
|
I'm merging this PR because I want to test it to make sure that everything is working. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #21918 +/- ##
==========================================
+ Coverage 90.29% 90.31% +0.01%
==========================================
Files 269 269
Lines 67541 67535 -6
==========================================
+ Hits 60988 60996 +8
+ Misses 6553 6539 -14
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| numFBFFailures, | ||
| numEqFailures, | ||
| numEqNoSnapshot, | ||
| runtime, |
There was a problem hiding this comment.
Looking at the output in #21920 (comment), it may be helpful to also print the runtime there?
For the "old" bots the runtime was usually fairly stable, at least on Linux, so having that available might help catch inadvertent performance regressions?
| GH_TOKEN: ${{ github.token }} | ||
| PR: ${{ github.event.pull_request.number }} | ||
| run: | | ||
| gh api "repos/${GITHUB_REPOSITORY}/issues/${PR}/labels/browsertest" \ |
There was a problem hiding this comment.
Here, and elsewhere in this PR, would it also work to use /pull/ instead of /issues/ since we're always dealing with PRs for this workflow and that might make the intent here clearer?
|
Thank you for implementing this! |
timvandermeij
removed
the
To Review
4 participants
Adding the
browsertestlabel to a PR dispatches a run to the private mozilla/pdf.js.pdfs repository, which runs the Firefox reference tests on Linux and Windows, sharded across several machines, and posts the results (with a link to the reftest analyzer hosted on mozilla/pdf.js.refs) as a comment. Every push to master dispatches a reference update in the same way.The test runner gets a
--shard=k/Noption to split the manifest across machines and a--summaryFileoption to write the final counts as JSON.