test(nightly): constrain AGPL source-stability allowance by mldangelo-oai · Pull Request #1817 · promptfoo/modelaudit

mldangelo-oai · 2026-08-21T01:42:29Z

Summary

Reproduce the Windows/Python 3.11 failure from scheduled Nightly run 32330980736, where the organized-assets scan correctly failed closed after pickle call-graph source stability changed.
Permit only the exact canonical agpl_model.pkl source-stability diagnostic with reason source_search_context_changed; require its malicious verdict, S204 finding, inconclusive status, has_errors, success=False, and exit code 2.
Reject copied or embedded same-name assets, unrelated operational failures, hidden issue/check/marker-only failures, and nested serialized diagnostics. Explicit operational markers are checked before otherwise benign pickle_source or parse-detail handling.
Exercise cache-enabled stable→changed and changed→stable scans. Both outcomes are intentionally non-cacheable, so the contract asserts zero entries and hits while proving each transition rescans and preserves its expected finding, verdict, status, and exit code.
Preserve existing coverage-only and optional-dependency behavior, including the Windows profile without h5py.

Validation

Failing-first: reproduced the original Nightly source-stability failure, a canonical diagnostic accepted after has_errors was cleared, and operational markers hidden behind pickle_source/parse_error details.
Focused transition, fail-closed, path, nested-diagnostic, and optional-dependency controls: 84 passed, 11 skipped before all-CI dependency sync; native exact-head review repeated the owning files with 86 passed, 9 skipped.
Required Python 3.10 precommit gate: Ruff format (424 files unchanged), Ruff check, and mypy (479 source files) passed; full offline unit suite: 23,484 passed, 806 skipped in 17:47.
Mandatory native prepublish review: two independent passes plus a fresh verifier, with 0 candidates and 0 blockers, bound to exact head 8c0d986ed6d8381950b159c0be76af88ef129f06.

chatgpt-codex-connector · 2026-08-21T01:42:45Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-08-28T12:04:30.438279Z	`ff34854`	Manual request
🔒 Security Review	✅ Completed 2026-08-28T12:04:25.364618Z	`ff34854`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

mldangelo-oai · 2026-08-21T01:43:12Z

@codex review

chatgpt-codex-connector · 2026-08-21T01:45:40Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6f36d5f138

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f36d5f138

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai · 2026-08-25T13:31:32Z

@codex review

mldangelo-oai · 2026-08-25T13:31:38Z

@codex security review

chatgpt-codex-connector · 2026-08-25T13:34:01Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 8c0d986ed6

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

mldangelo-oai · 2026-08-27T06:05:23Z

@codex review

mldangelo-oai · 2026-08-27T06:05:32Z

@codex security review

chatgpt-codex-connector · 2026-08-27T06:07:33Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 9ab449f208

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9ab449f208

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai · 2026-08-27T17:12:58Z

@codex review

mldangelo-oai · 2026-08-27T17:13:00Z

@codex security review

chatgpt-codex-connector · 2026-08-27T17:15:05Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: e3d7d510bc

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e3d7d510bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai · 2026-08-28T01:41:26Z

@codex review

mldangelo-oai · 2026-08-28T01:41:27Z

@codex security review

chatgpt-codex-connector · 2026-08-28T01:43:46Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6cda7b057c

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector · 2026-08-28T01:44:13Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6cda7b057c

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector · 2026-08-28T01:44:13Z

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 6cda7b057c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai · 2026-08-28T07:53:31Z

@codex review

mldangelo-oai · 2026-08-28T07:53:32Z

@codex security review

chatgpt-codex-connector · 2026-08-28T07:55:15Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 979dfa6fba

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector · 2026-08-28T07:55:37Z

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 979dfa6fba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-28T07:55:49Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 979dfa6fba

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

mldangelo-oai · 2026-08-28T12:02:23Z

@codex review

mldangelo-oai · 2026-08-28T12:02:29Z

@codex security review

chatgpt-codex-connector · 2026-08-28T12:04:23Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: ff348541ac

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector · 2026-08-28T12:04:29Z

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: ff348541ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai added 18 commits August 20, 2026 13:14

fix(tests): preserve fail-closed nightly asset scans

1a37114

fix(tests): reject mixed nightly scan failures

e9c7756

fix(tests): cover nested and marker-only scan errors

9d021b9

fix(tests): validate every nightly failure diagnostic

2f48728

test(nightly): reject concurrent scan-size truncation

88db74f

test(nightly): retain security findings and coverage semantics

b659446

test(nightly): preserve nested scanner timeout failures

ab96b16

test(nightly): reject nested scanner-error diagnostics

65735d8

test(nightly): honor optional scanner dependency profiles

b41185d

test(nightly): catch alternate scanner exception metadata

bf1291e

test(nightly): reject core scanner timeout diagnostics

36090f9

test(nightly): scope source instability to known agpl fixture

de0a52e

test(nightly): reject independent archive budget failures

ac9577d

test(nightly): reject masked archive read failures

3247c8c

test(nightly): validate independent archive failure controls

a69bad8

test(nightly): preserve failed checks and security findings

4d12a16

test(nightly): inspect consolidated failed scan checks

73159a6

test(nightly): reject hidden operational outcome reasons

6f36d5f

mldangelo-oai added codex codex-automation labels Aug 21, 2026

chatgpt-codex-connector Bot reviewed Aug 21, 2026

View reviewed changes

Comment thread tests/test_nightly_asset_contracts.py Outdated

github-code-quality Bot found potential problems Aug 21, 2026

View reviewed changes

Comment thread tests/test_nightly_asset_contracts.py Fixed

test(nightly): constrain AGPL source-stability allowance

8c0d986

mldangelo-oai changed the title ~~fix(tests): preserve fail-closed nightly asset scans~~ test(nightly): constrain AGPL source-stability allowance Aug 25, 2026

mldangelo-oai mentioned this pull request Aug 26, 2026

fix(ci): route hash timing test to performance lane #1824

Merged

test(nightly): harden asset scan contracts

9ab449f

chatgpt-codex-connector Bot reviewed Aug 27, 2026

View reviewed changes

Comment thread tests/test_security_asset_integration.py Outdated

Comment thread tests/test_security_asset_integration.py Outdated

Comment thread tests/test_security_asset_integration.py Outdated

test(nightly): bind archive provenance evidence

e3d7d51

chatgpt-codex-connector Bot reviewed Aug 27, 2026

View reviewed changes

test(nightly): fix archive contract portability

6cda7b0

test(nightly): honor stored ZIP member names

979dfa6

test(nightly): simplify source-stability contract

ff34854

mldangelo-oai merged commit 918cc85 into main Aug 28, 2026
42 of 44 checks passed

mldangelo-oai deleted the fix/nightly-asset-source-stability branch August 28, 2026 13:28

Uh oh!

Conversation

mldangelo-oai commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

chatgpt-codex-connector Bot commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

mldangelo-oai commented Aug 21, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

mldangelo-oai commented Aug 25, 2026

Uh oh!

mldangelo-oai commented Aug 25, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 25, 2026

Uh oh!

mldangelo-oai commented Aug 27, 2026

Uh oh!

mldangelo-oai commented Aug 27, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mldangelo-oai commented Aug 27, 2026

Uh oh!

mldangelo-oai commented Aug 27, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mldangelo-oai commented Aug 28, 2026

Uh oh!

mldangelo-oai commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

Uh oh!

mldangelo-oai commented Aug 28, 2026

Uh oh!

mldangelo-oai commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

Uh oh!

mldangelo-oai commented Aug 28, 2026

Uh oh!

mldangelo-oai commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

mldangelo-oai commented Aug 21, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Aug 21, 2026 •

edited

Loading