feat(plan): aggregate CPU shape from tick deltas - #17
Merged
Merged
Conversation
…ind spots covered The per-request CPU histogram has two structural blind spots, both hit by exactly the workloads the CPU ceiling exists for: requests under the 50ms floor never inform it, and a saturated pool has no idle worker at scrape time so nothing is sampled at all (#14). Worker tick deltas have neither problem - fast CPU-heavy requests sum to unmistakable pool-level CPU. PoolState now keeps EWMAs (alpha 0.2) of per-interval pool cores (tick deltas / wall, already computed for BoxCost) and the busy-worker snapshot; their ratio is the aggregate share per busy worker. A single ActiveNow snapshot lies for fast requests, but the EWMA of many converges on true utilization (Little's law). Idle intervals (< 0.05 cores) teach nothing, and the signal is trusted only after MinAggCPURounds (default 5) active intervals with real concurrency. plan falls back to the aggregate when the histogram has too few readings, and PoolCPU.AggregateBased says so. Verified end-to-end embedded in cbox-init on the php-baseimages benchmark: a 3ms-request flood that always left cpu_readings at 0 now classifies, engages the ceiling ('Pool bound by CPU rather than memory ... held there'), and the pool holds its size through 100s of saturation. Known phase-2 limitation, documented on #14: under saturation the ActiveNow denominator absorbs the oversubscription factor (8 busy workers sharing 2 cores read as ~21% each, not ~95%), so the derived fill converges toward the current size - conservative, never dangerous. The sharp fill under saturation is ceil(aggregate cores) x headroom; that lands separately.
sylvesterdamgaard
added a commit
that referenced
this pull request
Sep 10, 2026
…#18) * feat(plan): saturated pools cut to measured parallelism; allocator gets the aggregate fallback (#14 phase 2) Phase 1 (#17) gave the REPORT an aggregate CPU shape, but the ceiling the allocator actually uses (cpuCeilingFor) still required per-request readings - for sub-50ms workloads it stayed 0 and only the report told the truth. It now falls back to the aggregate share under the same Trusted gate. And the saturated case gets the honest fill: under saturation every worker reads 'busy' while queuing for a core, so the share's denominator absorbs the oversubscription factor and share-based fill circles back to the current size, whatever it is. The kernel's tick deltas cannot be inflated by workers that only wait - the cores the pool actually drives ARE its parallelism. A starved pool with a trusted baseline and a trusted aggregate is now cut to ceil(measured cores) x headroom instead of merely held: 16 workers saturating 2 cores converge to 4, where the worker-count sweep measured the throughput optimum (~+30% over 10-20 workers on CPU-bound endpoints). Gated like every other cut: on the pool's Trusted baseline AND on --cpu - a cap below the configured ceiling IS a cut, and cuts are opt-in. Without the flag the starved pool is held exactly as in phase 1, and the report row (SaturationMeasured) still shows what --cpu would do. * fix(plan): one regime rule for the aggregate fill - the live run flapped The first cut of phase 2 put the cores-based fill in the starved-gate block and a share-based fallback in cpuCeilingFor. Live, on a 2-core box with an oversized static pool, those two disagreed every other round: starved rounds said ceiling 4 (honest cores), queue-drained rounds said ceiling 80 (the share's busy denominator counts queued workers, so 25 workers on 2 cores read share 5% and fill 40) - and the pool flapped 4 -> 25 -> 4 with a SIGUSR2 reload per flip, collapsing throughput from 391 to ~110 rps. aggregateFill is now the ONE place that chooses: calm host -> the share is honest, fill from per-worker cost; saturated host -> fill IS the measured cores (tick deltas cannot be inflated by workers that only wait). Both the allocator (cpuCeilingFor) and the report (cpuOf) call it, so every kind of round produces the same number and the plan cannot flap. The starved-gate block goes back to phase-1 hold-only - the cut now arrives through the ceiling, in one place, still gated on --cpu and the Trusted baseline. A pool only claims the cores reading when it drives >= half the box: an io-shaped pool on a host made busy by a NEIGHBOR falls through to the share, whose fill is large and non-binding, exactly as io should be. Regression tests: the queue-drained flap round holds at 4; the io-neighbor pool is not CPU-capped; saturated cpuCeilingFor returns 4 where the poisoned share said 80. * fix(plan): calibrate the saturation threshold from run 2's live miss; instrument the aggregate Run 2 of the live proof never cut: a co-located build left the container ~1.15 effective cores of its 2-CPU quota, a hair under the 0.5x-cores threshold, and nothing in the logs could say so. Three changes: - Threshold 0.5x -> 0.35x of the box's cores. A genuinely CPU-bound pool's measured cores DEFLATE under host contention; an io pool's stay far below either number. Run 3 with this calibration: cut at the trust point, 24 -> 4 workers, +25% throughput, fifteen consecutive stable windows, ONE resize event in 46 minutes. - The 'Pool bound by CPU' log line now carries agg_cores, agg_busy and saturation_measured, so the regime rule's choice is visible on a live system instead of undiagnosable (PoolCPU.AggCores/AggBusy). - testing/mutations.py: the two ceiling-guard patterns follow the refactored code shape (they were NO MATCH, failing CI), and the saturated-cores branch gets its own mutation - removing it kills the flap-round regression test, which is exactly the guard-test relation the sweep exists to enforce.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the actionable core of #14's remaining symptom.
The two blind spots (both hit by exactly the workloads
--cpuexists for): requests under the 50 ms floor never inform the per-request histogram, and a saturated pool has no idle worker at scrape time so nothing is sampled at all. Worker tick deltas (already computed for BoxCost) have neither problem.Mechanism:
PoolStatekeeps EWMAs (α 0.2) of per-interval pool cores and the busy-worker snapshot; their ratio is the aggregate share per busy worker. SingleActiveNowsnapshots lie for fast requests — the EWMA of many converges on utilization (Little's law). Idle intervals teach nothing; trusted afterMinAggCPURounds(default 5) active intervals.planfalls back to it when the histogram is blind, andPoolCPU.AggregateBasedsays so.Verified end-to-end (embedded in cbox-init, php-baseimages harness): a 3 ms-request flood that always left
cpu_readingsat 0 now classifies and engages the ceiling — "Pool bound by CPU rather than memory … held there" — and the pool holds its size through 100 s of saturation.Documented phase-2 limitation (comment coming on #14): under saturation the ActiveNow denominator absorbs the oversubscription factor (8 busy workers sharing 2 cores read as ~21% each), so the derived fill converges toward current size — conservative, never dangerous. The sharp fill under saturation is
ceil(aggregate cores) × headroom; separate change.Tests: fast-CPU-heavy classification, guards (min rounds, idle intervals, io-shaped pool), full suite + vet + gofmt + sbom green.