fix(ci): pin reviewed native eval diagnostics

[mrizzi]

Jira

TC-6726

Change

Update only NATIVE_EVAL_SOURCE_SHA in the trusted main workflow to the reviewed immutable PR #299 commit c7ca8495f1a83c51d191f55e391b17c5b29e97da.

The difference from the previous pin is limited to native runner diagnostics and their regression coverage. No native eval suite files are added to main.

Why

Run 37471650779 failed with an incomplete result and no assertion outcomes. Its private execution/scoring logs were unavailable, so the exact cause remains unresolved.

The new safe artifact adds allowlisted phase/category names and integer phase exit codes. Raw stderr, transcripts and credentials remain private. Error categories are advisory hints from the first 64 KiB of phase stderr; they are not proof of a root cause or grading results. Judge failures can surface as summary / invalid-summary.

Cases, assertions, grading, credential isolation and CI permissions remain unchanged.

Validation

• Reviewed diagnostic source: 481 tests passed, 1 skipped; focused diagnostics: 14 passed.
• Native dependency/runtime preflight passed without inference.
• Source and this pin branch: Skillsaw passed with existing warnings; Claude plugin validation passed; git diff --check passed.
• Read-only review found no remaining blockers.

Next step

After human review and merge, rerun the PR #299 Eval PR trigger (run 37483925429), which creates a new consumer using the updated main pin. Do not rerun the old Eval PR Run consumer: it retains its original main revision. Inspect the new safe diagnostic artifact, then fix the evidenced native failure if it persists.

The obsolete old-pin consumer from the diagnostic source push was cancelled to avoid redundant model execution. This PR improves failure visibility; it does not claim the original native eval failure is fixed.

Summary by Sourcery

Pin the native evaluation workflow to the reviewed diagnostics revision.

Enhancements:

• Pin the native evaluation workflow to the reviewed immutable diagnostics commit for improved failure visibility.

CI:

• Update the trusted evaluation workflow’s native evaluation source revision.

[sourcery-ai]

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Updates the trusted native eval workflow to consume the reviewed immutable PR #299 commit containing safer native runner diagnostics, improving failure visibility without changing evaluation behavior or claiming to fix the underlying native failure.

File-Level Changes

Change	Details	Files
Pin the trusted workflow to the reviewed immutable native-evaluation source commit.	Replace the previous NATIVE_EVAL_SOURCE_SHA with commit c7ca849. Keep the change limited to selecting the diagnostic-enhanced source revision; native eval cases, assertions, grading, credentials, and permissions are unchanged.	.github/workflows/eval-pr-run.yml

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Needs a human reviewer. This changes which native evaluation code the CI workflow executes, so a wrong or compromised pinned commit could run unintended code with the workflow's available permissions and expose CI data. Reverting restores the previous pin for future runs, but any credentials or data exposed by an already-run job would not be recoverable by the revert.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook