|
| 1 | +# Spec: Bulkhead middleware (0.5.0, Epic 3 slice 2) |
| 2 | + |
| 3 | +**Date:** 2026-06-05 |
| 4 | +**Topic slug:** `bulkhead` |
| 5 | +**Status:** drafted, awaiting user review |
| 6 | +**Target release:** 0.5.0 (0.4.0 release notes already drafted without this slice). |
| 7 | +**Epic 3 stories rolled in:** 3-5 (Bulkhead). |
| 8 | + |
| 9 | +## Purpose |
| 10 | + |
| 11 | +Ship `httpware`'s second resilience primitive: a `Bulkhead` middleware that caps in-flight requests through an `AsyncClient`. This is distinct from `httpx2.Limits`, which caps the *connection pool* — Bulkhead caps the *number of concurrent requests* the caller has waiting on the network at once. Use cases: |
| 12 | + |
| 13 | +- Protect a downstream service from being overwhelmed by a busy client. |
| 14 | +- Bound the client's own resource usage (memory, response handlers in flight). |
| 15 | +- Fail fast under load instead of building up unbounded queues of waiting requests. |
| 16 | + |
| 17 | +This is slice 2 of Epic 3; the only remaining Epic 3 work after this is `3-6` extension-slot docs (rides with the last code PR or in a docs-only follow-up). |
| 18 | + |
| 19 | +## Non-goals |
| 20 | + |
| 21 | +Items deliberately deferred so this slice ships clean: |
| 22 | + |
| 23 | +- **No per-host / per-route partitioning.** One Bulkhead = one semaphore. If a future caller needs per-downstream caps, they install separate Bulkheads on separate clients (or wait for a future spec that adds keying). Adding partitioning later is purely additive. |
| 24 | +- **No separate `BulkheadLimit` type.** Unlike `Retry`+`RetryBudget` (where the budget needs to be separable to share across multiple `Retry` middlewares), `Bulkhead` is *already* the sharable unit — reuse the instance across `AsyncClient(middleware=[shared])` calls. |
| 25 | +- **No queue metrics surface.** No `bulkhead.in_flight` / `bulkhead.queued` properties. Observability lands in Epic 5 (`5-1` Layer 1 hooks); revisit then. |
| 26 | +- **No fallback / shed-load callbacks.** Caller catches `BulkheadFullError` and decides what to do. |
| 27 | +- **No connection-pool integration.** `Bulkhead` is a middleware; `httpx2.Limits` is a transport-level concern. They compose but stay independent. |
| 28 | + |
| 29 | +## Architecture & module layout |
| 30 | + |
| 31 | +```text |
| 32 | +src/httpware/middleware/resilience/ |
| 33 | +├── __init__.py # existing; add Bulkhead re-export |
| 34 | +├── budget.py # existing |
| 35 | +├── retry.py # existing |
| 36 | +├── _backoff.py # existing |
| 37 | +└── bulkhead.py # NEW |
| 38 | +``` |
| 39 | + |
| 40 | +`Bulkhead` middleware in `bulkhead.py`. `BulkheadFullError(ClientError)` lands in `src/httpware/errors.py` alongside the existing exception tree. Both re-exported from `httpware/__init__.py`. No new optional extra — pure stdlib (`asyncio.Semaphore`, `asyncio.timeout`). |
| 41 | + |
| 42 | +File-per-middleware mirrors slice-1's layout — each unit is independently testable. |
| 43 | + |
| 44 | +## Public API |
| 45 | + |
| 46 | +### `Bulkhead` middleware |
| 47 | + |
| 48 | +```python |
| 49 | +class Bulkhead: |
| 50 | + """Concurrency limiter middleware. See module docstring for behavior.""" |
| 51 | + |
| 52 | + def __init__( |
| 53 | + self, |
| 54 | + *, |
| 55 | + max_concurrent: int, # required; no default |
| 56 | + acquire_timeout: float | None = 1.0, # seconds; None = wait forever; 0 = fail fast |
| 57 | + ) -> None: ... |
| 58 | + |
| 59 | + async def __call__( |
| 60 | + self, |
| 61 | + request: httpx2.Request, |
| 62 | + next: Next, |
| 63 | + ) -> httpx2.Response: ... |
| 64 | +``` |
| 65 | + |
| 66 | +Constructor validation: |
| 67 | + |
| 68 | +- `max_concurrent < 1` → `ValueError("max_concurrent must be >= 1")` |
| 69 | +- `acquire_timeout is not None and acquire_timeout < 0` → `ValueError("acquire_timeout must be >= 0")` |
| 70 | +- `acquire_timeout == 0` is accepted (fail-fast: no wait). |
| 71 | + |
| 72 | +Internal state: one `asyncio.Semaphore(max_concurrent)` instantiated in `__init__`. Created with the running event loop's binding rules — semaphores in asyncio are not loop-bound until first use in 3.10+, so the middleware is safe to construct outside an event loop. |
| 73 | + |
| 74 | +`max_concurrent` is required because there is no universally-correct default — the right value depends on downstream capacity, request latency, and the caller's SLA. Forcing the choice avoids hiding a production-shaping default. |
| 75 | + |
| 76 | +### `BulkheadFullError` |
| 77 | + |
| 78 | +```python |
| 79 | +class BulkheadFullError(ClientError): |
| 80 | + """Raised when acquire_timeout elapses before a Bulkhead slot becomes available.""" |
| 81 | + |
| 82 | + max_concurrent: int |
| 83 | + acquire_timeout: float | None |
| 84 | + |
| 85 | + def __init__(self, *, max_concurrent: int, acquire_timeout: float | None) -> None: |
| 86 | + self.max_concurrent = max_concurrent |
| 87 | + self.acquire_timeout = acquire_timeout |
| 88 | + super().__init__( |
| 89 | + f"bulkhead full (max_concurrent={max_concurrent}, " |
| 90 | + f"acquire_timeout={acquire_timeout})" |
| 91 | + ) |
| 92 | + |
| 93 | + def __reduce__(self) -> tuple[Any, ...]: ... # picklable via module-level reconstructor |
| 94 | +``` |
| 95 | + |
| 96 | +Picklable via the same `_reconstruct_*` pattern used for `StatusError` and `RetryBudgetExhaustedError` (see `src/httpware/errors.py:61-62, 149-156`). Tested explicitly. |
| 97 | + |
| 98 | +`BulkheadFullError` inherits from `ClientError` so callers catching `ClientError` already handle it. We do NOT inherit from `httpware.TimeoutError` — semantically, a bulkhead-full event is a backpressure signal, not a network timeout. Conflating them would mislead callers writing observability/alerting around timeout vs overload. |
| 99 | + |
| 100 | +## Behavior |
| 101 | + |
| 102 | +### `Bulkhead.__call__` algorithm |
| 103 | + |
| 104 | +```python |
| 105 | +async def __call__(self, request, next): |
| 106 | + try: |
| 107 | + if self._acquire_timeout is None: |
| 108 | + await self._sem.acquire() |
| 109 | + else: |
| 110 | + async with asyncio.timeout(self._acquire_timeout): |
| 111 | + await self._sem.acquire() |
| 112 | + except asyncio.TimeoutError as exc: |
| 113 | + raise BulkheadFullError( |
| 114 | + max_concurrent=self._max_concurrent, |
| 115 | + acquire_timeout=self._acquire_timeout, |
| 116 | + ) from exc |
| 117 | + |
| 118 | + try: |
| 119 | + return await next(request) |
| 120 | + finally: |
| 121 | + self._sem.release() |
| 122 | +``` |
| 123 | + |
| 124 | +Why two-stage (`acquire` then `try/finally`) rather than `async with self._sem`: |
| 125 | + |
| 126 | +- If we used `async with self._sem`, a CancelledError fired between the `__aenter__` return and the start of `next()` would still pass through `__aexit__`, which is fine — `asyncio.Semaphore.__aexit__` releases. But it is too easy to misread the structure as "the timeout wraps both acquire and execute," which is wrong. |
| 127 | +- The explicit acquire makes the contract obvious: only the acquisition is bounded by `acquire_timeout`; `next()` runs as long as it needs to. |
| 128 | +- The `try/finally` makes release deterministic across every exit path (success, exception from `next()`, cancellation during `next()`). |
| 129 | + |
| 130 | +### Cancellation semantics |
| 131 | + |
| 132 | +- **Cancellation before `acquire()` returns**: `CancelledError` propagates; no slot held; no release. |
| 133 | +- **Cancellation during `next()`**: `try/finally` releases the slot; `CancelledError` propagates. |
| 134 | +- **Cancellation between `acquire()` return and start of `next()`** (vanishingly small window): `try/finally` releases the slot; `CancelledError` propagates. |
| 135 | + |
| 136 | +### Middleware chain position |
| 137 | + |
| 138 | +Documented recommendation (not enforced): |
| 139 | + |
| 140 | +```python |
| 141 | +AsyncClient(middleware=[ |
| 142 | + Bulkhead(max_concurrent=10), # outermost: cap total concurrency |
| 143 | + Retry(...), # retries happen *inside* the Bulkhead slot |
| 144 | +]) |
| 145 | +``` |
| 146 | + |
| 147 | +Rationale: with `Bulkhead` outside `Retry`, a single request occupies one slot across all its retry attempts. Concurrency stays bounded by `max_concurrent` even under retry storms. The opposite order — `Retry` outside `Bulkhead` — lets each retry attempt re-acquire a slot, which inflates effective concurrency under the exact load conditions retry is meant to absorb. |
| 148 | + |
| 149 | +If a caller wants per-attempt rejection (each retry can be rejected independently), they put `Retry` outside; that is also valid and documented as a trade-off in the docstring. |
| 150 | + |
| 151 | +### Sharing across clients |
| 152 | + |
| 153 | +A `Bulkhead` instance is the sharable unit. Reuse: |
| 154 | + |
| 155 | +```python |
| 156 | +shared = Bulkhead(max_concurrent=20) |
| 157 | +client_a = AsyncClient(base_url="https://upstream.example.com/v1", middleware=[shared]) |
| 158 | +client_b = AsyncClient(base_url="https://upstream.example.com/v2", middleware=[shared]) |
| 159 | +``` |
| 160 | + |
| 161 | +Both clients then contend for the same 20 slots. No special API surface — the semaphore is just a Python attribute on the shared object. |
| 162 | + |
| 163 | +## Errors raised |
| 164 | + |
| 165 | +- **`BulkheadFullError(ClientError)`** — raised when `acquire_timeout` elapses without acquiring a slot. Fields: `max_concurrent`, `acquire_timeout`. Picklable. |
| 166 | +- **`asyncio.CancelledError`** — propagates unchanged. Never caught by `Bulkhead`. |
| 167 | +- **All other exceptions from `next()`** — propagate unchanged. `Bulkhead` only intercepts the acquisition-timeout path. |
| 168 | + |
| 169 | +## Testing |
| 170 | + |
| 171 | +Per `planning/engineering.md §6`: |
| 172 | + |
| 173 | +- **`tests/test_bulkhead.py`** — unit tests via `httpx2.MockTransport`: |
| 174 | + - succeeds when slots available (no wait, no `BulkheadFullError`) |
| 175 | + - serializes correctly when at capacity (second call waits for first to release) |
| 176 | + - `BulkheadFullError` raised after `acquire_timeout` elapses |
| 177 | + - `BulkheadFullError` carries the configured `max_concurrent` / `acquire_timeout` |
| 178 | + - `BulkheadFullError` is picklable (round-trip) |
| 179 | + - `acquire_timeout=0` fails fast (immediate raise when full) |
| 180 | + - `acquire_timeout=None` waits forever (use `asyncio.wait_for` to time out from outside the test) |
| 181 | + - slot released after success (next request acquires immediately) |
| 182 | + - slot released after exception from `next()` (semaphore not leaked) |
| 183 | + - slot released on `asyncio.CancelledError` propagation |
| 184 | + - `max_concurrent < 1` rejected at construction with `ValueError` |
| 185 | + - negative `acquire_timeout` rejected at construction with `ValueError` |
| 186 | + - shared `Bulkhead` instance across two `AsyncClient`s enforces the joint cap |
| 187 | + |
| 188 | +- **`tests/test_bulkhead_props.py`** — Hypothesis property tests: |
| 189 | + - **Invariant 1**: For any interleaving of N coroutines through a single `Bulkhead(max_concurrent=K)`, the observed in-flight count never exceeds K. |
| 190 | + - **Invariant 2**: With `acquire_timeout=0`, the total successful acquisitions across the run equals K + (number of releases observed before each acquire attempt). |
| 191 | + - **Invariant 3**: `BulkheadFullError` is raised iff acquire takes longer than `acquire_timeout`. |
| 192 | + |
| 193 | +- **No time injection.** Unlike `RetryBudget` (where deterministic time matters for TTL math), `Bulkhead`'s only time-sensitive operation is `asyncio.timeout` around the acquire. Tests use real `asyncio.sleep` with sub-100ms values; the suite stays under ~50ms even with the property tests. |
| 194 | + |
| 195 | +- **Coverage target**: 100% line coverage. |
| 196 | + |
| 197 | +## Open questions deferred to implementation |
| 198 | + |
| 199 | +- **Construct-outside-loop, use-inside-loop test.** `asyncio.Semaphore` in Python 3.10+ binds lazily on first use (no construct-inside-loop requirement). The project floor is 3.11, so this is already fine — but a test that constructs `Bulkhead` at module scope and uses it inside an `async def` test pins the behavior and survives any future stdlib regression. |
| 200 | +- **Property test seeding.** Hypothesis-driven concurrency tests can flap if the event loop scheduler is non-deterministic. Use `@settings(deadline=None)` and a sufficient `max_examples` to make flakes detectable, not deterministic. Same pattern as `test_budget_props.py`. |
| 201 | + |
| 202 | +## References |
| 203 | + |
| 204 | +- `planning/engineering.md` §6 (testing patterns, property tests for concurrency-sensitive code), §8 (roadmap — `3-5` Bulkhead) |
| 205 | +- `planning/specs/2026-06-05-retry-and-retry-budget-design.md` (slice-1 patterns mirrored here: module-per-middleware, `__reduce__` for picklability, public re-export) |
| 206 | +- Hystrix / Resilience4j bulkhead patterns (semaphore-based; conceptual reference, no API parity intended) |
0 commit comments