ci: wire test-floor identity ratcheting into report-results job by orieg · Pull Request #221 · orieg/php-judy

orieg · 2026-10-08T03:59:44Z

What

Adds test-floor identity ratcheting to the Consolidated Test & Benchmark Report job (.github/workflows/ci.yml). Every test ID that passed on the base ref must pass on head — a test that was passing on base and becomes missing, skipped, or failed on head is reported as Test Dropped From Suite.

Why

The existing static AST count (language pack) can't catch runtime-visible test erosion:

Tests behind disabled conditions (e.g., #[cfg(...)], @pytest.mark.skipif)
Tests generated dynamically at runtime (no test ID exists in AST)
Tests moved out of collection paths not covered by language packs

Configuring test_report closes these gaps by ratcheting the set of executed test identities across base and head from JUnit XML.

Changes

New baseline: `reports/junit.xml`

258 tests, all passing on PHP 8.5.8 Linux x64 NTS (local run)
Generated by merging all platform-specific junit artifacts into a single file
Serves as the identity floor — every test ID in it must pass on head

report-results job additions:

Full checkout — reads reports/junit.xml from git (base report); replaces sparse checkout
Merge step — combines all junit-* artifacts into reports/junit-head.xml (head report)
test-floor step — runs discipline check --enable test-floor --test-report reports/junit-head.xml; only on non-fork PRs (same guard as the existing EnricoMi action step)

Validation

Local v0.18.0 discipline check --base origin/main: PASS (27 gates, 0 errors, 0 warnings; 1 override for intentional narrowing)
YAML parses correctly
All 258 local tests pass with JUnit output

CI time impact

<30s total: merge script ~2s + discipline XML parsing.

Gate notes

The ci-integrity/verification-step-narrowed finding is lifted by the allow-gate-weakening: directive — test-floor ratcheting intentionally only runs on PRs (on push-to-main the merge base equals head so no meaningful ratchet is possible). This mirrors the existing EnricoMi publish-unit-test-result-action guard which also skips pushes and fork PRs.

Adds test-floor ratcheting to the Consolidated Test & Benchmark Report job so every test ID that passed on the base ref must pass on head. This catches runtime-visible test erosion (dropped/skipped tests) that static AST counts cannot see. Changes: - Commit baseline JUnit XML from local test run as reports/junit.xml (258 tests, all passing on PHP 8.5.8 Linux x64 NTS) - Add full checkout step in report-results to read the baseline from git - Add merge step combining all platform-specific junit artifacts into a single reports/junit-head.xml for the head report - Add discipline test-floor step with test_report pointing to the merged file; runs only on non-fork PRs (same guard as the existing EnricoMi publish-unit-test-result-action step) CI time impact: <30s total (merge script ~2s + discipline XML parsing). allow-gate-weakening: ci-integrity test-floor identity ratcheting intentionally narrowed to pull_request events only — on push-to-main the merge base equals head so no meaningful ratchet is possible; the step mirrors the existing EnricoMi publish-unit-test-result-action guard which also skips pushes and fork PRs

github-actions · 2026-10-08T04:00:00Z

discipline: passed

Base main (origin/main) (merge base b5dbc42e26) · 27 gate(s) passed · 2168 item(s) examined · 0 warning(s) · 1 override(s)

Findings lifted by an override:

ci-integrity on ci-integrity: ci-integrity test-floor identity ratcheting intentionally narrowed to pull_request events only — on push-to-main the merge base equals head so no meaningful ratchet is possible; the step mirrors the existing EnricoMi publish-unit-test-result-action guard which also skips pushes and fork PRs

_{discipline explain <gate> prints what a gate checks and how a finding is lifted. This comment is updated on every run; the check's status is the verdict.}

… git objects The sparse-checkout input makes actions/checkout fetch with --filter=blob:none (partial clone). discipline embeds libgit2, which cannot fetch promisor blobs on demand, so every git-mode gate (assertion-reduction, test-floor, ...) dies in report-results with: gate `assertion-reduction` could not run: object not found - cannot read header for (<sha>); class=Odb (9); code=NotFound (-3) Drop the sparse checkout and use a plain fetch-depth: 0 checkout, matching the discipline sentinel job which already does exactly this and passes. The baseline (reports/junit.xml) and baselines/ are both part of the full tree, so nothing else in the job loses access. Verified locally against origin/main: test-floor git mode passes with an empty base (this PR adds the baseline; the ratchet engages on the next PR once reports/junit.xml exists on main).

github-actions · 2026-10-08T05:17:19Z

Test Results

	PHP	Platform	Arch	TS	Tests	Pass	Fail	Skip	Duration
✅	8.4	Linux	x64	-	258	258	0	0	9.7s
✅	Total				258	258	0	0

Benchmark Results

500,000 elements, 7 iterations (median), Judy 2.7.1

Run Status

Platform	PHP 8.4
Linux	✅

Memory (Linux, PHP 8.4)

Peak RSS of a child process building one structure, n = 500,000, minus an empty-process floor. Judy's memory advantage is type- and scale-dependent — see BENCHMARK.md for the curve across sizes on a dedicated host.

Type	Judy	PHP array	Judy vs PHP array
Bitset	< 440.0 KB	10.0 MB	> 23x smaller
Int to Int	4.3 MB	10.1 MB	2.3x smaller
Int to Mixed	42.3 MB	40.0 MB	1.06x larger
Int to Packed	18.3 MB	40.0 MB	2.2x smaller
String to Int	10.1 MB	40.1 MB	4.0x smaller
String to Mixed	48.2 MB	70.1 MB	1.5x smaller
String to Int (Hash)	31.4 MB	40.0 MB	1.3x smaller
String to Mixed (Hash)	69.4 MB	70.0 MB	about the same
String to Int (Adaptive)	31.4 MB	40.0 MB	1.3x smaller
String to Mixed (Adaptive)	69.6 MB	70.0 MB	about the same

< marks a structure smaller than this instrument can resolve at n = 500,000: run-to-run movement in the empty-process floor puts the resolution at 440.0 KB. The bound is real, the exact figure is not available at this size — BENCHMARK.md measures the same types up to n = 8M, where they are well clear of the floor.

API Speedups (Linux, PHP 8.4)

Speedup = PHP time / Judy time. Bold = Judy faster.

Operation	Judy	PHP	Speedup
putAll()	19.5 ms	28.5 ms	1.5x
fromArray()	19.9 ms	28.5 ms	1.4x
toArray()	23.4 ms	36.8 ms	1.6x
getAll()	1.1 ms	2.0 ms	1.8x
keys()	10.5 ms	24.1 ms	2.3x
values()	11.0 ms	24.0 ms	2.2x
sumValues()	6.9 ms	19.1 ms	2.8x
populationCount()	0.00 ms	20.7 ms	O(1) vs O(n)
deleteRange()	38.3 ms	55.1 ms	1.4x
equals()	11.4 ms	44.9 ms	3.9x
union() Bitset	31.1 ms	4.9 ms	0.2x
intersect() Bitset	13.7 ms	5.8 ms	0.4x
diff() Bitset	13.5 ms	3.2 ms	0.2x
xor() Bitset	26.8 ms	11.5 ms	0.4x
mergeWith() Int→Int	51.2 ms	67.7 ms	1.3x
mergeWith() Str→Int	233.9 ms	255.2 ms	1.1x
forEach() Int→Int	27.4 ms	23.2 ms	0.8x
filter() Int→Int	43.0 ms	39.4 ms	0.9x
map() Int→Int	48.5 ms	45.7 ms	0.9x
forEach() Str→Int	65.9 ms	59.2 ms	0.9x
filter() Str→Int	122.9 ms	114.8 ms	0.9x
map() Str→Int	162.6 ms	158.7 ms	~1.0x

O(1) vs O(n) marks a row whose ratio is decided by the complexity classes rather than measured: populationCount() reads libJudy's cached population, the PHP arm counts every element. It scales with n and says nothing about this change. judy-bench.php declines the same comparison for size($lo, $hi) for the same reason.

Core Types (Linux, PHP 8.4)

Speedup = PHP time / Judy time. Bold = Judy faster.

Type	Op	Judy	PHP array	Speedup
Bitset	Write	17.5 ms	7.1 ms	0.4x
Bitset	Read	10.6 ms	3.9 ms	0.4x
Bitset	Iter	18.7 ms	3.1 ms	0.2x
Bitset	Free	0.01 ms	0.31 ms	53.4x
Int to Int	Write	24.0 ms	7.4 ms	0.3x
Int to Int	Read	12.0 ms	3.9 ms	0.3x
Int to Int	Iter	19.0 ms	3.1 ms	0.2x
Int to Int	Free	0.20 ms	0.32 ms	1.6x
Int to Mixed	Write	34.3 ms	11.6 ms	0.3x
Int to Mixed	Read	14.6 ms	5.8 ms	0.4x
Int to Mixed	Iter	20.9 ms	4.0 ms	0.2x
Int to Mixed	Free	13.8 ms	4.4 ms	0.3x
String to Int	Write	91.1 ms	22.2 ms	0.2x
String to Int	Read	38.1 ms	7.8 ms	0.2x
String to Int	Iter	57.9 ms	4.8 ms	0.1x
String to Int	Free	18.4 ms	1.3 ms	0.1x
String to Int (Adaptive)	Write	163.8 ms	22.4 ms	0.1x
String to Int (Adaptive)	Read	25.2 ms	7.9 ms	0.3x
String to Int (Adaptive)	Iter	68.3 ms	4.8 ms	0.1x
String to Int (Adaptive)	Free	40.8 ms	1.2 ms	0.0x

All types detail

Core Types (Linux, PHP 8.4) — All

Speedup = PHP time / Judy time. Bold = Judy faster.

Type	Op	Judy	PHP array	Speedup
Bitset	Write	17.5 ms	7.1 ms	0.4x
Bitset	Read	10.6 ms	3.9 ms	0.4x
Bitset	Iter	18.7 ms	3.1 ms	0.2x
Bitset	Free	0.01 ms	0.31 ms	53.4x
Int to Int	Write	24.0 ms	7.4 ms	0.3x
Int to Int	Read	12.0 ms	3.9 ms	0.3x
Int to Int	Iter	19.0 ms	3.1 ms	0.2x
Int to Int	Free	0.20 ms	0.32 ms	1.6x
Int to Mixed	Write	34.3 ms	11.6 ms	0.3x
Int to Mixed	Read	14.6 ms	5.8 ms	0.4x
Int to Mixed	Iter	20.9 ms	4.0 ms	0.2x
Int to Mixed	Free	13.8 ms	4.4 ms	0.3x
Int to Packed	Write	46.4 ms	10.8 ms	0.2x
Int to Packed	Read	31.9 ms	6.1 ms	0.2x
Int to Packed	Iter	38.7 ms	4.1 ms	0.1x
Int to Packed	Free	8.8 ms	4.2 ms	0.5x
String to Int	Write	91.1 ms	22.2 ms	0.2x
String to Int	Read	38.1 ms	7.8 ms	0.2x
String to Int	Iter	57.9 ms	4.8 ms	0.1x
String to Int	Free	18.4 ms	1.3 ms	0.1x
String to Mixed	Write	114.3 ms	25.6 ms	0.2x
String to Mixed	Read	41.9 ms	11.4 ms	0.3x
String to Mixed	Iter	68.1 ms	6.5 ms	0.1x
String to Mixed	Free	63.3 ms	7.5 ms	0.1x
String to Int (Hash)	Write	167.1 ms	22.8 ms	0.1x
String to Int (Hash)	Read	25.4 ms	7.9 ms	0.3x
String to Int (Hash)	Iter	69.0 ms	4.8 ms	0.1x
String to Int (Hash)	Free	41.7 ms	1.3 ms	0.0x
String to Mixed (Hash)	Write	230.6 ms	25.8 ms	0.1x
String to Mixed (Hash)	Read	28.7 ms	11.4 ms	0.4x
String to Mixed (Hash)	Iter	90.4 ms	6.6 ms	0.1x
String to Mixed (Hash)	Free	120.6 ms	6.8 ms	0.1x
String to Int (Adaptive)	Write	163.8 ms	22.4 ms	0.1x
String to Int (Adaptive)	Read	25.2 ms	7.9 ms	0.3x
String to Int (Adaptive)	Iter	68.3 ms	4.8 ms	0.1x
String to Int (Adaptive)	Free	40.8 ms	1.2 ms	0.0x
String to Mixed (Adaptive)	Write	235.8 ms	27.0 ms	0.1x
String to Mixed (Adaptive)	Read	27.6 ms	11.1 ms	0.4x
String to Mixed (Adaptive)	Iter	91.4 ms	6.6 ms	0.1x
String to Mixed (Adaptive)	Free	119.1 ms	6.5 ms	0.1x

Release Comparison (2.6.0 → 2.7.1)

Interleaved A/B comparison against v2.6.0 (pie install) on the same CI runner: 5 benchmark groups × 5 rounds × 2 arms, 300,000 elements, median of 3 per child. Each row is the median of the paired current/baseline ratios with a 95% percentile-bootstrap CI; it is flagged only when the whole drift-adjusted CI clears ±10%. Advisory — benchmarks never gate a merge.

Run-wide median delta +0.20% (PHP-array control +0.22%); runner common-mode +0.22% divided out of every row.

Summary: 0 regressions, 97 unchanged, 3 not evaluated (unstable)

Benchmark	Baseline	Current	Delta	95% CI	Status
adv.filter.int_to_int	26.9 ms	26.3 ms	-2.5%	[-5.1%, -0.3%]	~same
adv.filter.string_to_int	72.6 ms	72.0 ms	-1.1%	[-2.6%, +1.2%]	~same
adv.forEach.int_to_int	16.6 ms	16.5 ms	-1.0%	[-2.8%, +1.7%]	~same
adv.forEach.string_to_int	38.2 ms	39.4 ms	+3.0%	[+0.2%, +4.6%]	~same
adv.map.int_to_int	30.0 ms	28.8 ms	-4.0%	[-4.5%, -1.7%]	~same
adv.map.string_to_int	93.3 ms	95.1 ms	+1.5%	[+0.6%, +1.9%]	~same
adv.optiter.foreach.string_to_int_adaptive.default	67.9 ms	64.0 ms	+4.8%	[-14.4%, +13.0%]	~same
adv.optiter.foreach.string_to_int_adaptive.optimized	42.4 ms	43.1 ms	+2.1%	[+0.5%, +4.3%]	~same
adv.optiter.foreach.string_to_int_hash.optimized	33.4 ms	32.8 ms	-1.0%	[-2.4%, +0.3%]	~same
adv.optiter.increment.string_to_int_hash.default	40.0 ms	39.6 ms	+0.2%	[-11.5%, +5.1%]	~same
adv.optiter.overwrite.string_to_int_adaptive.default	56.3 ms	58.3 ms	+9.0%	[-10.8%, +10.6%]	~same
adv.optiter.overwrite.string_to_int_adaptive.optimized	55.6 ms	51.4 ms	-5.7%	[-10.5%, +12.4%]	~same
adv.optiter.overwrite.string_to_int_hash.default	39.3 ms	40.7 ms	+4.2%	[-12.6%, +8.1%]	~same
adv.optiter.ratio.foreach.string_to_int_adaptive	62.4 ms	67.0 ms	-1.8%	[-10.4%, +19.0%]	~same
adv.optiter.ratio.foreach.string_to_int_hash	58.8 ms	59.5 ms	+2.0%	[-8.7%, +11.3%]	~same
adv.optiter.ratio.increment.string_to_int_hash	129.0 ms	114.8 ms	-6.0%	[-17.6%, +13.7%]	~same
adv.optiter.ratio.overwrite.string_to_int_adaptive	93.6 ms	86.0 ms	-1.2%	[-14.8%, +1.9%]	~same
adv.optiter.ratio.overwrite.string_to_int_hash	108.6 ms	111.6 ms	+1.7%	[-10.9%, +21.2%]	~same
adv.optiter.ratio.toArray.string_to_int_adaptive	61.6 ms	63.5 ms	+9.2%	[-1.4%, +12.7%]	~same
adv.optiter.ratio.toArray.string_to_int_hash	53.3 ms	56.9 ms	+5.9%	[-0.9%, +10.3%]	~same
adv.optiter.ratio.values.string_to_int_adaptive	58.0 ms	57.4 ms	+0.4%	[-8.9%, +12.0%]	~same
adv.optiter.ratio.values.string_to_int_hash	49.9 ms	49.2 ms	+2.2%	[-8.7%, +10.7%]	~same
adv.optiter.toArray.string_to_int_adaptive.default	72.7 ms	68.9 ms	-5.6%	[-15.7%, +1.0%]	~same
adv.optiter.toArray.string_to_int_adaptive.optimized	45.6 ms	43.9 ms	-1.4%	[-5.0%, +6.6%]	~same
adv.optiter.toArray.string_to_int_hash.default	61.9 ms	57.6 ms	-6.2%	[-11.4%, +3.5%]	~same
adv.optiter.toArray.string_to_int_hash.optimized	32.5 ms	32.8 ms	-0.4%	[-2.0%, +2.7%]	~same
adv.optiter.values.string_to_int_adaptive.default	59.0 ms	58.3 ms	-1.9%	[-10.2%, +10.1%]	~same
adv.optiter.values.string_to_int_adaptive.optimized	34.0 ms	33.4 ms	-1.3%	[-3.7%, +0.8%]	~same
adv.optiter.values.string_to_int_hash.default	50.4 ms	49.4 ms	-2.8%	[-14.7%, +6.8%]	~same
adv.optiter.values.string_to_int_hash.optimized	24.5 ms	24.0 ms	-1.3%	[-5.4%, +0.7%]	~same
api.deleteRange.int_to_int	24.9 ms	23.0 ms	-7.5%	[-10.7%, -6.4%]	~same
api.equals.int_to_int	7.4 ms	7.0 ms	-5.3%	[-5.8%, -3.7%]	~same
api.fromArray.int_to_int	14.1 ms	13.7 ms	-3.3%	[-4.3%, -1.9%]	~same
api.getAll.int_to_int	0.53 ms	0.59 ms	+10.6%	[-13.4%, +12.2%]	~same
api.increment.int_to_int	14.4 ms	13.8 ms	-3.9%	[-4.3%, -1.3%]	~same
api.keys.int_to_int	6.9 ms	6.5 ms	-6.0%	[-16.6%, -2.1%]	~same
api.mergeWith.int_to_int	32.1 ms	31.7 ms	-1.1%	[-3.7%, -0.9%]	~same
api.mergeWith.string_to_int	139.5 ms	140.6 ms	+1.2%	[-0.1%, +1.8%]	~same
api.populationCount.int_to_int	0.00 ms	0.00 ms	-0.2%	[-50.1%, -0.2%]	~same
api.putAll.int_to_int	14.1 ms	13.5 ms	-5.0%	[-6.6%, -0.7%]	~same
api.range.keys.int_to_int	0.58 ms	0.55 ms	-4.6%	[-20.5%, -3.2%]	~same
api.range.size.string_to_int	2.3 ms	2.3 ms	-2.0%	[-5.2%, +4.8%]	~same
api.range.size1000.int_to_int	0.71 ms	0.67 ms	-6.3%	[-6.8%, -5.9%]	~same
api.setop.diff.bitset	7.7 ms	8.0 ms	+3.2%	[+2.6%, +3.3%]	~same
api.setop.diff.int_to_int	10.4 ms	10.2 ms	-3.2%	[-6.3%, +0.2%]	~same
api.setop.diff.string_to_int	65.7 ms	66.1 ms	+0.8%	[-0.3%, +1.2%]	~same
api.setop.intersect.bitset	7.8 ms	8.2 ms	+4.4%	[+3.4%, +7.1%]	~same
api.setop.intersect.int_to_int	10.5 ms	10.2 ms	-0.9%	[-4.9%, +0.1%]	~same
api.setop.intersect.string_to_int	73.2 ms	74.4 ms	+1.1%	[-0.6%, +2.5%]	~same
api.setop.union.bitset	18.4 ms	18.9 ms	+2.9%	[+1.3%, +5.6%]	~same
api.setop.union.int_to_int	27.2 ms	25.8 ms	-5.7%	[-6.2%, -5.1%]	~same
api.setop.union.string_to_int	161.6 ms	165.6 ms	+2.2%	[+0.7%, +2.6%]	~same
api.setop.xor.bitset	15.5 ms	16.3 ms	+4.6%	[+3.2%, +7.0%]	~same
api.setop.xor.int_to_int	22.1 ms	20.8 ms	-6.0%	[-7.8%, -4.4%]	~same
api.sumValues.int_to_int	4.9 ms	4.7 ms	-8.5%	[-9.2%, +0.9%]	~same
api.toArray.int_to_int	15.9 ms	15.8 ms	-0.1%	[-7.8%, +0.4%]	~same
api.values.int_to_int	7.5 ms	7.5 ms	-2.1%	[-16.2%, +2.1%]	~same
core.bitset.free	0.00 ms	0.00 ms	+2.3%	[-2.2%, +11.9%]	~same
core.bitset.iter	10.8 ms	11.7 ms	+7.5%	[+3.4%, +10.1%]	~same
core.bitset.read	6.6 ms	6.5 ms	-4.3%	[-5.5%, +5.3%]	~same
core.bitset.write	10.2 ms	10.5 ms	+2.2%	[+1.8%, +5.2%]	~same
core.int_to_int.free	0.31 ms	0.31 ms	-0.2%	[-7.2%, +39.5%]	~same
core.int_to_int.iter	11.6 ms	11.4 ms	-1.7%	[-5.9%, +5.8%]	~same
core.int_to_int.read	6.7 ms	7.6 ms	+9.3%	[+7.5%, +16.7%]	~same
core.int_to_int.write	14.6 ms	14.8 ms	+0.6%	[-0.6%, +7.5%]	~same
core.int_to_mixed.free	7.7 ms	7.7 ms	-0.6%	[-4.0%, +1.0%]	~same
core.int_to_mixed.iter	12.4 ms	12.4 ms	-0.6%	[-0.9%, +1.8%]	~same
core.int_to_mixed.read	8.3 ms	8.6 ms	+2.5%	[+2.3%, +5.1%]	~same
core.int_to_mixed.write	20.3 ms	20.1 ms	-2.2%	[-2.9%, +0.2%]	~same
core.int_to_packed.free	5.2 ms	4.9 ms	-6.6%	[-20.1%, -4.4%]	~same
core.int_to_packed.iter	22.9 ms	23.5 ms	+2.1%	[+1.3%, +4.5%]	~same
core.int_to_packed.read	18.3 ms	19.1 ms	+3.6%	[+2.9%, +8.6%]	~same
core.int_to_packed.write	28.2 ms	27.5 ms	-3.2%	[-4.9%, -1.7%]	~same
core.string_to_int.free	10.1 ms	10.3 ms	+2.0%	[+0.4%, +3.1%]	~same
core.string_to_int.iter	32.8 ms	33.3 ms	+1.3%	[+0.7%, +1.9%]	~same
core.string_to_int.read	21.1 ms	21.1 ms	+0.1%	[-0.7%, +0.6%]	~same
core.string_to_int.write	51.8 ms	52.1 ms	+0.5%	[-0.7%, +2.3%]	~same
core.string_to_int_adaptive.free	21.5 ms	20.8 ms	+2.1%	[-7.9%, +14.4%]	~same
core.string_to_int_adaptive.iter	39.5 ms	39.6 ms	0.0%	[-1.7%, +1.8%]	~same
core.string_to_int_adaptive.read	14.6 ms	14.6 ms	0.0%	[-0.9%, +3.7%]	~same
core.string_to_int_adaptive.write	87.5 ms	88.5 ms	+1.1%	[-3.4%, +14.5%]	~same
core.string_to_int_hash.free	20.6 ms	20.8 ms	+1.8%	[-18.5%, +4.9%]	~same
core.string_to_int_hash.iter	40.1 ms	40.3 ms	0.0%	[-1.0%, +0.9%]	~same
core.string_to_int_hash.read	14.8 ms	14.9 ms	+0.5%	[+0.1%, +5.2%]	~same
core.string_to_int_hash.write	86.7 ms	91.8 ms	+6.3%	[-1.0%, +7.6%]	~same
core.string_to_mixed.free	33.5 ms	33.6 ms	+0.6%	[-1.6%, +1.6%]	~same
core.string_to_mixed.iter	35.1 ms	36.6 ms	+3.0%	[+0.3%, +8.4%]	~same
core.string_to_mixed.read	23.2 ms	23.4 ms	+0.5%	[-1.4%, +0.7%]	~same
core.string_to_mixed.write	64.2 ms	64.1 ms	-0.1%	[-1.6%, +1.2%]	~same
core.string_to_mixed_adaptive.free	63.1 ms	63.3 ms	-0.3%	[-5.0%, +13.1%]	~same
core.string_to_mixed_adaptive.iter	49.1 ms	48.5 ms	-0.1%	[-2.3%, +5.8%]	~same
core.string_to_mixed_adaptive.read	16.3 ms	16.3 ms	-0.1%	[-5.1%, +4.5%]	~same
core.string_to_mixed_adaptive.write	128.8 ms	129.5 ms	-1.8%	[-8.9%, +11.9%]	~same
core.string_to_mixed_hash.free	61.7 ms	62.7 ms	+1.2%	[-1.8%, +9.9%]	~same
core.string_to_mixed_hash.iter	48.5 ms	48.9 ms	+1.0%	[-5.6%, +7.0%]	~same
core.string_to_mixed_hash.read	16.5 ms	16.6 ms	+1.3%	[-0.2%, +3.3%]	~same
core.string_to_mixed_hash.write	124.7 ms	122.9 ms	+0.4%	[-6.9%, +4.9%]	~same

3 benchmarks not evaluated (within-arm scatter exceeded the effect being claimed)

Benchmark	Baseline	Current	Delta	95% CI	Status
adv.optiter.foreach.string_to_int_hash.default	57.1 ms	55.1 ms	-3.0%	[-12.4%, +9.6%]	— unstable
adv.optiter.increment.string_to_int_hash.optimized	48.7 ms	44.1 ms	-5.7%	[-26.9%, +14.4%]	— unstable
adv.optiter.overwrite.string_to_int_hash.optimized	43.1 ms	41.7 ms	+7.1%	[-12.1%, +10.1%]	— unstable

Memory Comparison

Memory: 0 regressions, 10 unchanged

Type	Baseline	Current	Delta	Status
Bitset	160 B	160 B	0.0%	~same
Int to Int	160 B	160 B	0.0%	~same
Int to Mixed	4.6 MB	4.6 MB	0.0%	~same
Int to Packed	8.0 MB	8.0 MB	0.0%	~same
String to Int	64.2 KB	64.2 KB	0.0%	~same
String to Int (Adaptive)	64.2 KB	64.2 KB	0.0%	~same
String to Int (Hash)	64.2 KB	64.2 KB	0.0%	~same
String to Mixed	4.6 MB	4.6 MB	0.0%	~same
String to Mixed (Adaptive)	4.6 MB	4.6 MB	0.0%	~same
String to Mixed (Hash)	4.6 MB	4.6 MB	0.0%	~same

Full benchmark output

Linux — PHP 8.4

══════════════════════════════════════════════════════════════════════════════════════════
  php-judy Benchmark Suite — 500,000 elements, 7 iterations (median)
  PHP 8.4.26 | Judy ext 2.7.1 | Linux x86_64
  Suite: all | Groups: core.int,core.str,api.batch,api.setops,adv.iter,adv.sso
  memory_limit: -1
  Memory: peak RSS of a child process, empty-process floor subtracted, median of 3
══════════════════════════════════════════════════════════════════════════════════════════

── Core: Integer-Keyed Types ────────────────────────────────────────────────

  Type                         Write(ms)      Read(ms)      Iter(ms)      Free(ms)        Memory
  ────────────────────────  ────────────  ────────────  ────────────  ────────────  ────────────
  BITSET                           17.53         10.57         18.66          0.01    < 440.0 KB
    PHP array                       7.10          3.86          3.12          0.31       10.0 MB
  INT_TO_INT                       24.01         12.01         19.01          0.20        4.3 MB
    PHP array                       7.35          3.86          3.12          0.32       10.1 MB
  INT_TO_MIXED                     34.33         14.63         20.91         13.79       42.3 MB
    PHP array                      11.65          5.82          3.97          4.40       40.0 MB
  INT_TO_PACKED                    46.37         31.86         38.75          8.77       18.3 MB
    PHP array                      10.81          6.06          4.09          4.22       40.0 MB

── Core: String-Keyed Types ────────────────────────────────────────────────

  Type                         Write(ms)      Read(ms)      Iter(ms)      Free(ms)        Memory
  ────────────────────────  ────────────  ────────────  ────────────  ────────────  ────────────
  STRING_TO_INT                    91.11         38.10         57.88         18.42       10.1 MB
    PHP array                      22.19          7.80          4.81          1.27       40.1 MB
  STRING_TO_MIXED                 114.31         41.89         68.08         63.32       48.2 MB
    PHP array                      25.60         11.35          6.47          7.47       70.1 MB
  STR_TO_INT_HASH                 167.06         25.41         69.01         41.67       31.4 MB
    PHP array                      22.84          7.87          4.81          1.29       40.0 MB
  STR_TO_MIX_HASH                 230.58         28.70         90.38        120.55       69.4 MB
    PHP array                      25.81         11.41          6.57          6.77       70.0 MB
  STR_INT_ADAPTIVE                163.80         25.19         68.27         40.82       31.4 MB
    PHP array                      22.42          7.92          4.82          1.22       40.0 MB
  STR_MIX_ADAPTIVE                235.81         27.64         91.41        119.12       69.6 MB
    PHP array                      27.05         11.09          6.65          6.54       70.0 MB

── API: Batch Operations (INT_TO_INT) ────────────────────────────────────────

  Operation                           Judy(ms)       PHP(ms)     Speedup
  ──────────────────────────────  ────────────  ────────────  ──────────
  putAll()                               19.48         28.49        1.5x
  fromArray()                            19.90         28.49        1.4x
  toArray()                              23.43         36.85        1.6x
  getAll(50100 keys)                      1.14          2.02        1.8x
  increment(500000 ops/1000 keys)         22.64         10.14        0.4x

── API: Native Methods (keys, values, sum, popCount, deleteRange, equals) ────

  Operation                           Judy(ms)       PHP(ms)     Speedup
  ──────────────────────────────  ────────────  ────────────  ──────────
  keys() INT_TO_INT                      10.52         24.07        2.3x
  values() INT_TO_INT                    11.00         24.01        2.2x
  keys($lo,$hi) INT_TO_INT                0.91         24.29       26.6x
    vs slice()->keys()                    2.69             -        2.9x
  size($lo,$hi) INT x1000                 0.77             -        O(1)
  size($lo,$hi) STRING_TO_INT             3.79          5.11        1.3x
  sumValues() INT_TO_INT                  6.93         19.10        2.8x
  populationCount() INT_TO_INT            0.00         20.67   229677.1x
  deleteRange() INT_TO_INT mid50%         38.33         55.12        1.4x
  equals() INT_TO_INT                    11.41         44.94        3.9x

── API: Set Operations (BITSET, 50% overlap) ────────────────────────────────

  Operation                           Judy(ms)       PHP(ms)     Speedup
  ──────────────────────────────  ────────────  ────────────  ──────────
  union() BITSET                         31.12          4.88        0.2x
  intersect() BITSET                     13.75          5.82        0.4x
  diff() BITSET                          13.51          3.24        0.2x
  xor() BITSET                           26.84         11.54        0.4x
  union() INT_TO_INT                     40.72         84.89        2.1x
  intersect() INT_TO_INT                 16.69         41.92        2.5x
  diff() INT_TO_INT                      16.95         42.41        2.5x
  xor() INT_TO_INT                       34.16         84.31        2.5x
  union() STRING_TO_INT                 279.74        316.62        1.1x
  intersect() STRING_TO_INT             118.99        139.14        1.2x
  diff() STRING_TO_INT                  112.91        135.35        1.2x
  mergeWith() INT_TO_INT                 51.18         67.66        1.3x
  mergeWith() STRING_TO_INT             233.88        255.22        1.1x

── Advanced: C-Level forEach/filter/map ──────────────────────────────────────

  Operation                        C-level(ms)       PHP(ms)     Speedup
  ──────────────────────────────  ────────────  ────────────  ──────────
  forEach() INT_TO_INT                   27.42         23.17        0.8x
  forEach() STRING_TO_INT                65.86         59.21        0.9x
  filter() INT_TO_INT (50% pass)         42.98         39.36        0.9x
  filter() STRING_TO_INT (50% pass)        122.94        114.83        0.9x
  map() INT_TO_INT (*2)                  48.55         45.70        0.9x
  map() STRING_TO_INT (*2)              162.63        158.65        1.0x

── Advanced: optimizeIteration trade (opt-in key-index payload mirror) ──────

  Operation                        default(ms)    optim.(ms)   optim./def.
  ──────────────────────────────  ────────────  ────────────  ────────────
  foreach() HASH kl16                    52.73         32.48         61.6%
  values() HASH kl16                     47.20         23.73         50.3%
  toArray() HASH kl16                    57.96         32.64         56.3%
  overwrite() HASH kl16                  35.02         39.76        113.5%
  increment() HASH kl16                  47.25         40.22         85.1%
    HASH kl16: honoured?          yes
  foreach() ADAPTIVE kl40                61.81         41.75         67.5%
  values() ADAPTIVE kl40                 53.28         32.49         61.0%
  toArray() ADAPTIVE kl40                66.30         44.16         66.6%
  overwrite() ADAPTIVE kl40              55.61         49.68         89.3%
    ADAPTIVE kl40: honoured?      yes
  skipped — increment() ADAPTIVE kl40: Judy::increment() is only supported for INT_TO_INT, STRING_TO_INT and STRING_TO_INT_HASH types

  Ratios are optimized as a percentage of default, measured inside this
  build; below 100 is the traversal win, above 100 the write price. Lower
  is better on every row, so a lost win reads as a rise.

── Advanced: Adaptive SSO — short keys (<8 bytes) via JudyL ─────────────────

  Operation                   STR_TO_INT    STR_INT_HASH   STR_INT_ADPTV    Best ratio
  ──────────────────────  ──────────────  ──────────────  ──────────────  ────────────
  insert (short keys)          72.39 ms      123.99 ms      114.40 ms       1.7x
  read (random access)         22.79 ms       21.04 ms       18.02 ms       1.3x
  foreach (iterate all)        48.85 ms       88.29 ms       88.01 ms       1.8x

── Advanced: Adaptive SSO — long keys (>=8 bytes) fallback to JudyHS ────────

  Operation                   STR_TO_INT    STR_INT_HASH   STR_INT_ADPTV    Best ratio
  ──────────────────────  ──────────────  ──────────────  ──────────────  ────────────
  insert (long keys)          118.15 ms      205.23 ms      205.51 ms       1.7x
  read (random access)         39.79 ms       34.74 ms       32.42 ms       1.2x
  foreach (iterate all)        84.29 ms       94.86 ms       93.96 ms       1.1x

══════════════════════════════════════════════════════════════════════════════════════════
  Benchmark complete — 2026-10-08 05:30:02
  All timings: median of 7 iterations via hrtime(true), 1 warmup run
  Memory: peak RSS of a child process building one structure of 500,000 elements,
          minus an empty-process floor. Measured with getrusage(), not
          memory_get_usage(), for the reason above. Judy's memory advantage is
          scale- and type-dependent — see BENCHMARK.md for the curve.
══════════════════════════════════════════════════════════════════════════════════════════

  JSON results written to: bench.json

The report-results job still carried a second "Checkout full repo for test-floor baseline" step from the first wiring attempt. actions/checkout runs `git clean -ffdx` by default, so the second checkout deleted the just-downloaded artifacts/ directory (and any untracked file including the merged reports/junit-head.xml) before the merge step ran. Observed in CI: "Merged 0 suites -> reports/junit-head.xml" while the download step had just pulled junit-linux-8.4 (258 tests). The empty head report made the ratchet a silent no-op and gutted the EnricoMi check and the consolidated report too. The single full checkout (fetch-depth: 0, no sparse-checkout) already provides reports/junit.xml, so the second checkout is redundant. Verified the merge script against the real runner layout (artifacts/junit-*/): "Merged 1 suites" -> 258 testcases, and test-floor reports 258 examined with "not present in base ref; identity ratchet will be enforced on subsequent changes".

The ratchet wired in #221 never actually enforced anything. Two bugs: 1. Path mismatch. test_report pointed at reports/junit-head.xml, a filename the merge step generates but that is never committed. In git mode discipline reads the BASE report from the base ref at the same path as test_report, so the lookup always missed and the step passed trivially with "not present in base ref; identity ratchet will be enforced on subsequent changes" -- a promise that could never come true. 2. Serialization mismatch (found on the first live run of the fix). Having pointed test_report at reports/junit.xml, the ratchet then flagged four tests as "dropped" for no reason. Cause: the merge step built the head with xml.dom.minidom.toprettyxml(), which normalizes attribute quoting and HTML-entity escaping (name="...offsetSet's..." instead of name='...offsetSet's...'). The committed baseline is raw run-tests.php output, so every test name containing an apostrophe disagreed with the head and test-floor read it as a drop. Discipline compares the name strings as written, not XML-decoded. Fix both: point test_report at the committed reports/junit.xml baseline (git mode then finds base from the base ref and head from disk at the same path), and merge the head by concatenating the raw <testsuite> blocks from each artifact verbatim, preserving the escaping the baseline uses. Verified with the local v0.18.0 CLI (the same engine CI runs): - fixed wiring, head == baseline: "verified 258 passed base test identity/identities against head report", PASS - minidom-re-serialized head: 4 false "Test Dropped From Suite" errors (the exact names CI flagged) -- reproduced, then fixed by raw concat - one testcase genuinely dropped: caught as Test Dropped From Suite - full local discipline check on the diff: PASS (27 gates)

…#224) The ratchet wired in #221 never actually enforced anything. Two bugs: 1. Path mismatch. test_report pointed at reports/junit-head.xml, a filename the merge step generates but that is never committed. In git mode discipline reads the BASE report from the base ref at the same path as test_report, so the lookup always missed and the step passed trivially with "not present in base ref; identity ratchet will be enforced on subsequent changes" -- a promise that could never come true. 2. Serialization mismatch (found on the first live run of the fix). Having pointed test_report at reports/junit.xml, the ratchet then flagged four tests as "dropped" for no reason. Cause: the merge step built the head with xml.dom.minidom.toprettyxml(), which normalizes attribute quoting and HTML-entity escaping (name="...offsetSet's..." instead of name='...offsetSet's...'). The committed baseline is raw run-tests.php output, so every test name containing an apostrophe disagreed with the head and test-floor read it as a drop. Discipline compares the name strings as written, not XML-decoded. Fix both: point test_report at the committed reports/junit.xml baseline (git mode then finds base from the base ref and head from disk at the same path), and merge the head by concatenating the raw <testsuite> blocks from each artifact verbatim, preserving the escaping the baseline uses. Verified with the local v0.18.0 CLI (the same engine CI runs): - fixed wiring, head == baseline: "verified 258 passed base test identity/identities against head report", PASS - minidom-re-serialized head: 4 false "Test Dropped From Suite" errors (the exact names CI flagged) -- reproduced, then fixed by raw concat - one testcase genuinely dropped: caught as Test Dropped From Suite - full local discipline check on the diff: PASS (27 gates)

orieg added 2 commits October 7, 2026 21:51

fix(ci): include reports/ in sparse-checkout for test-floor baseline

d31d64e

orieg merged commit 7d89638 into main Oct 8, 2026
19 checks passed

orieg deleted the chore/test-floor-ratcheting branch October 8, 2026 05:32

orieg mentioned this pull request Oct 8, 2026

ci: fix test-floor ratchet never engaging (base path mismatch) #224

Merged

orieg mentioned this pull request Oct 8, 2026

🚨 [Nightly CI Failure] Valgrind Memory Leak Detected on main #222

Closed

Navigation Menu

Repository navigation

ci: wire test-floor identity ratcheting into report-results job - #221

ci: wire test-floor identity ratcheting into report-results job#221
orieg merged 4 commits into
mainfrom
chore/test-floor-ratcheting

orieg commented Oct 8, 2026

Uh oh!

github-actions Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 8, 2026 •

edited

Loading

Core Types (Linux, PHP 8.4) — All

Linux — PHP 8.4

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

orieg commented Oct 8, 2026

What

Why

Changes

New baseline: reports/junit.xml

report-results job additions:

Validation

CI time impact

Gate notes

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

discipline: passed

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Benchmark Results

Run Status

Memory (Linux, PHP 8.4)

API Speedups (Linux, PHP 8.4)

Core Types (Linux, PHP 8.4)

Core Types (Linux, PHP 8.4) — All

Release Comparison (2.6.0 → 2.7.1)

Memory Comparison

Linux — PHP 8.4

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

New baseline: `reports/junit.xml`

github-actions Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading