feat(plan): saturated pools cut to measured parallelism (#14 phase 2) - #18
Merged
Merged
Conversation
…ts the aggregate fallback (#14 phase 2) Phase 1 (#17) gave the REPORT an aggregate CPU shape, but the ceiling the allocator actually uses (cpuCeilingFor) still required per-request readings - for sub-50ms workloads it stayed 0 and only the report told the truth. It now falls back to the aggregate share under the same Trusted gate. And the saturated case gets the honest fill: under saturation every worker reads 'busy' while queuing for a core, so the share's denominator absorbs the oversubscription factor and share-based fill circles back to the current size, whatever it is. The kernel's tick deltas cannot be inflated by workers that only wait - the cores the pool actually drives ARE its parallelism. A starved pool with a trusted baseline and a trusted aggregate is now cut to ceil(measured cores) x headroom instead of merely held: 16 workers saturating 2 cores converge to 4, where the worker-count sweep measured the throughput optimum (~+30% over 10-20 workers on CPU-bound endpoints). Gated like every other cut: on the pool's Trusted baseline AND on --cpu - a cap below the configured ceiling IS a cut, and cuts are opt-in. Without the flag the starved pool is held exactly as in phase 1, and the report row (SaturationMeasured) still shows what --cpu would do.
The first cut of phase 2 put the cores-based fill in the starved-gate block and a share-based fallback in cpuCeilingFor. Live, on a 2-core box with an oversized static pool, those two disagreed every other round: starved rounds said ceiling 4 (honest cores), queue-drained rounds said ceiling 80 (the share's busy denominator counts queued workers, so 25 workers on 2 cores read share 5% and fill 40) - and the pool flapped 4 -> 25 -> 4 with a SIGUSR2 reload per flip, collapsing throughput from 391 to ~110 rps. aggregateFill is now the ONE place that chooses: calm host -> the share is honest, fill from per-worker cost; saturated host -> fill IS the measured cores (tick deltas cannot be inflated by workers that only wait). Both the allocator (cpuCeilingFor) and the report (cpuOf) call it, so every kind of round produces the same number and the plan cannot flap. The starved-gate block goes back to phase-1 hold-only - the cut now arrives through the ceiling, in one place, still gated on --cpu and the Trusted baseline. A pool only claims the cores reading when it drives >= half the box: an io-shaped pool on a host made busy by a NEIGHBOR falls through to the share, whose fill is large and non-binding, exactly as io should be. Regression tests: the queue-drained flap round holds at 4; the io-neighbor pool is not CPU-capped; saturated cpuCeilingFor returns 4 where the poisoned share said 80.
Contributor
Author
|
Live proof, third and final run (2 CPUs / 1 GiB, 16→24-worker static pool, sub-3ms CPU flood via wrk -t2 -c32, embedded in cbox-init, 60s measurement windows, quiet host):
One The three runs tell the whole story:
Worker-sweep cross-check: the sweep measured 396-400 rps at 2-4 workers and ~305 at 20+ on this endpoint - the tuner now lands exactly on the measured optimum, from live evidence only. |
… instrument the aggregate Run 2 of the live proof never cut: a co-located build left the container ~1.15 effective cores of its 2-CPU quota, a hair under the 0.5x-cores threshold, and nothing in the logs could say so. Three changes: - Threshold 0.5x -> 0.35x of the box's cores. A genuinely CPU-bound pool's measured cores DEFLATE under host contention; an io pool's stay far below either number. Run 3 with this calibration: cut at the trust point, 24 -> 4 workers, +25% throughput, fifteen consecutive stable windows, ONE resize event in 46 minutes. - The 'Pool bound by CPU' log line now carries agg_cores, agg_busy and saturation_measured, so the regime rule's choice is visible on a live system instead of undiagnosable (PoolCPU.AggCores/AggBusy). - testing/mutations.py: the two ceiling-guard patterns follow the refactored code shape (they were NO MATCH, failing CI), and the saturated-cores branch gets its own mutation - removing it kills the flap-round regression test, which is exactly the guard-test relation the sweep exists to enforce.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes #14. Phase 1 (#17) covered the histogram's blind spots with an aggregate CPU shape - but only in the report;
cpuCeilingFor, the ceiling the allocator uses, still required per-request readings and stayed 0 for sub-50 ms workloads.The design (after a live lesson):
aggregateFillis the ONE place that reads the tick-delta aggregate, with a regime rule:ceil(1.9 cores) × headroom 2.0 = 4.Both the allocator and the report call it, so every kind of round produces the same number. The first cut of this PR had the cores rule only in the starved-gate block and the share fallback in
cpuCeilingFor- live, those two flapped against each other every other round (queue drained for one scrape → ceiling 80 → regrow toward memory's 30 → starved round → cut to 4 → SIGUSR2 reload per flip), collapsing throughput from 391 to ~110 rps. The flap round is now a regression test.Safety:
--cpuAND theTrustedbaseline (a cap below the configured ceiling IS a cut; cuts are opt-in). Without the flag: phase-1 hold, report shows what--cpuwould do (SaturationMeasured).MinAggCPURounds): hold, never cut on an unproven signal.Live evidence (2 CPUs, 16→24-worker static pool, sub-3ms CPU flood, embedded in cbox-init):
A fresh 45-minute run with the regime rule is queued; expect the cut to hold at 4 with ~390 rps to the end - numbers land as a comment.
Tests: oversized-starved pool converges to 4 (CPUBound); flap round holds at 4; io-neighbor not capped; untrusted aggregate holds; no-flag holds; saturated
cpuCeilingForreturns 4 where the poisoned share said 80. Full gate green.