Repository navigation
Fix/fw06 restore corpus boundary - #6
Merged
Merged
Conversation
…inst
The stage 2 image refuses to build:
FATAL: the suite and its recording disagree:
the recording answers corpus 2c2e7339..., this corpus is 67dfd21e...
The rename before the public release swept the fixed multipart boundary in
corpus.py along with everything else that carried the old name. It should not
have moved: the boundary is an opaque request input, not a name. It lands in
the Content-Type header and again in the body, so it feeds fingerprint(), and
data/golden-statea.json was captured before the rename. Restoring it returns
fingerprint() to
2c2e73398e2c5d2b604ee8bf06ebc7501f0b33a2138936f8023db70cc80e925d, the value the
recording carries.
It went unnoticed because the boundary appears nowhere in the recording itself
-- the upload responses never echo it -- so the rename had nothing to rewrite
there and changed only the input to the hash.
The drift is that token and nothing else. The 87 case ids already agreed, the
four profiles and their flag lists are untouched, and the fixture aggregate the
recording was captured against still matches the committed docroot. The
recording itself is not edited here. Nothing about what this suite measures
changes.
Restored rather than re-captured, for the reason 85cc9ef gives for fw05: a
fresh capture would be a different artifact from the one the published results
were graded against, and the reward would be a nicer spelling for a token that
names nothing.
The audit's provenance gate moves with it. _ORACLE_TOKENS is there to catch a
submission that recognises the boundary it actually receives, so leaving it
spelled SWERefactorBenchBoundary would have left it searching for a string no
request contains. It needs restoring in its own right rather than as tidiness:
the old spelling was also caught by _HARNESS_TOKENS' "SWERefactor" prefix, and
the restored one is not, so that gate is now the only thing covering it.
Verified by building the image: the self-check passes, 87 records against the
restored fingerprint, and the suite still collects 478 checks.
Reported in #5.
… build The fingerprint check that caught the fw06 drift lives in each task's stage Dockerfile, so it only runs when an image is built. That is how a rename reached a release with the corpus and the recording disagreeing, and it is why the drift was reported from outside rather than found here. The same comparison runs in CI now: four suites, one second, no docker. It re-computes each stored fingerprint from the harness that would send the requests, and for fw06 it also compares the record ids against the case ids. The second test is the one that keeps this honest. It scans every behavioural data directory for a stamped fingerprint and fails if the set of suites that stamp one has drifted from the set checked here, so a new suite cannot ship a recording that nothing re-computes. Verified in both directions: green as committed, and red on fw06 with the renamed boundary put back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.