Skip to content

build01: stop calling an omission permitted that §1.4 never permitted - #18

Merged
hongdy22 merged 1 commit into
mainfrom
fix/optional-check-semantics
Sep 14, 2026
Merged

hongdy22 merged 1 commit into
mainfrom
fix/optional-check-semantics

Conversation

@hongdy22

@hongdy22 hongdy22 commented Sep 14, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #10.

Rewritten. The first version recorded an absent macro as a pass, and also changed fw02. Both are reverted — the scoring was never wrong.

skip is how every check in this tree records that the thing being asked about was not produced. It is charged, it does not pass, and srb_skip_ok only decides the wording. That is the published rule, in scores_raw/LEGEND.md:

skip 进分母算 0:别的模型能跑出产物、这个模型跳过了,那是它自己的问题。
(a skip counts in the denominator as 0: other models produced the artifact and this one skipped it, which is its own problem)

It is applied that way with no exception. Across the published tables, 313 skipped checks on 48 module rows fall on 42 cells, and all 42 scored 0.

These 25 macros are required, on all three counts that decide it:

  • the instruction never said they could be omitted — §1.4 asks for "the same macro set" and instruction.md contains no optional, permitted, may omit or skip;
  • they are the right thing to ask: State A's configure defines all 25, which is where the recorded values in data/macros.json come from;
  • models answer them — this row is 281 of those 313 skips, and beside them it reads 189 pass and 40 fail, with 9 of 20 cells passing at least one.

So no published score changes. What was wrong was the suite calling the omission (permitted) and carrying srb_skip_ok, which told a submission an omission was free when it was not. Both are gone; absence is now recorded as a failure saying the contract does not permit it, and a wrong value fails as before.

Regression: infra/tests/test_build01_macro_contract.py, 15 checks, fails on main (2 of 15). The seven infra suites stay at 556 passed.

test_ssp_can_be_turned_off had the same wording and is fixed in #17, which already edits that function.

@hongdy22
hongdy22 force-pushed the fix/optional-check-semantics branch from 0943df9 to bbbb4f6 Compare September 14, 2026 05:46
@hongdy22 hongdy22 changed the title build01, fw02: a permitted absence passes its check instead of skipping it build01: stop calling an omission permitted that §1.4 never permitted Sep 14, 2026
`srb_skip_ok` decides how a skip is recorded, not whether it is charged, and
`pytest_module`'s docstring draws the line it was narrowed to: where the
reference has no answer the check is not a question and is deleted from its
suite; where the reference does have one, a submission that cannot be asked has
failed to produce the thing being asked about, and the submissions that did
produce it should not be the only ones paying for the check.

`test_optional_macro_value_when_present` sat on the wrong side of that line.
State A's configure defines all 25 of these macros -- they are where the
recorded values in macros.json come from -- and across the preserved build01
cells the row reads 189 pass and 40 fail beside its skips.  It is a question
submissions answer.  §1.4 asks for the same macro set and names no exception,
and nothing else in the instruction mentions optional, permitted or skip.

So the marker goes and the reason stops saying "(permitted)".  The machinery is
unchanged: absence still raises `pytest.skip`, and without the marker the
harness records it as a failure saying the contract does not permit it.  The
`OPTIONAL` name stays, and a comment says what the group actually is -- 25
macros graded by value in one check rather than by presence in a second, which
is the only way they differ from the 64 above.

This reverses the earlier direction on this branch, which recorded an absent
macro as a pass.  That was a judgement about autoconf bookkeeping substituted
for the rule the tree already had, and the corpus settles it the other way.
@hongdy22
hongdy22 force-pushed the fix/optional-check-semantics branch from bbbb4f6 to 63d97fd Compare September 14, 2026 06:19
@hongdy22
hongdy22 merged commit d6c9d08 into main Sep 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

build01 and fw02: Clarify optional-check semantics under the Stage II all-pass gate

1 participant