|
| 1 | +# The fill-model hazard in externally-sourced strategies: what to establish before a foreign number means anything here |
| 2 | + |
| 3 | +**Date:** 2026-08-27 · **Issue:** #529 · **Origin:** |
| 4 | +[`docs/superpowers/specs/2026-08-23-strategy-api-expressiveness-prd.md`](../superpowers/specs/2026-08-23-strategy-api-expressiveness-prd.md) |
| 5 | +§4.2 ("a warning worth encoding, not a feature"). |
| 6 | + |
| 7 | +**This document reports no new measurement.** Every number in it is cited to the experiment |
| 8 | +record that produced it. It exists because the hazard it describes is not in any engine — the |
| 9 | +engines are correct — but in how a person reads someone else's backtest. Where a strategy |
| 10 | +candidate comes from (the [scout skill](../../.claude/skills/keel-asset-scout/SKILL.md), a |
| 11 | +paper, another framework's example gallery), the questions below are what make its published |
| 12 | +numbers interpretable in keel, and they are cheaper to answer than a port is to write. |
| 13 | + |
| 14 | +--- |
| 15 | + |
| 16 | +## 1. The hazard |
| 17 | + |
| 18 | +Strategies published for other frameworks routinely **name their own entry price and assume the |
| 19 | +fill**. Jesse's idiom is literally: |
| 20 | + |
| 21 | +```python |
| 22 | +def go_long(self): |
| 23 | + entry = self.price |
| 24 | + stop = entry - self.atr * self.hp['stop_loss_atr_rate'] |
| 25 | + qty = utils.risk_to_qty(self.balance, 2, entry, stop) |
| 26 | + self.buy = qty, entry # <- a resting order at a chosen price |
| 27 | + self.stop_loss = qty, stop |
| 28 | +``` |
| 29 | + |
| 30 | +**keel does not work that way, and has measured what the difference costs.** Since #258 (the |
| 31 | +fix for #257), `backtest()` fills an entry at the **next bar's open plus slippage** |
| 32 | +(`keel/strategy/backtest.py`: *"`candles[i+1].open` plus slippage, the first price obtainable |
| 33 | +once the signal exists"*), because that is what `execution/executor.py` actually places live — |
| 34 | +`order_type="market"`, `limit_price=None`, an unconditional market IOC. It does not rest an |
| 35 | +order and wait for a level; #260 records that the executor discards a rule's conditional entry |
| 36 | +price entirely. |
| 37 | + |
| 38 | +Correcting `pullback_continuation` to that fill model — the worked example, from |
| 39 | +[`2026-08-13-restated-under-a-production-faithful-engine.md`](2026-08-13-restated-under-a-production-faithful-engine.md) |
| 40 | +§3 — **dropped its median gross profit factor from 0.9292 to 0.7736 (§3.1 prints the same pair |
| 41 | +as 0.9219 → 0.7736) and doubled its median trade count from 60 to 124** (§3.1: 58 → 124). The |
| 42 | +mechanism is legible: `entry = signal_candle.high + buffer_ticks` was a *confirmation |
| 43 | +condition*, and a market fill removes it, so the rule takes the trades it was designed to |
| 44 | +decline. The doubling *is* the count of those trades. |
| 45 | + |
| 46 | +So: **any externally-sourced strategy ported naively inherits an optimism keel has already |
| 47 | +priced.** Its published numbers are not wrong for its own framework; they are unreachable in |
| 48 | +this one. This compounds with the cost picture — round-trip friction here is **~2.5% of |
| 49 | +notional** at taker (`2 × fee_pct + 2 × slippage_pct`: 2.50% at the 1.2% taker rate, 1.30% at |
| 50 | +maker, per |
| 51 | +[`2026-08-12-fee-curve-and-rsi-meanrev.md`](2026-08-12-fee-curve-and-rsi-meanrev.md)). A |
| 52 | +strategy whose edge depends on getting a chosen price, evaluated in a framework that assumes it |
| 53 | +does, is optimistic on **both axes at once** — and neither optimism is visible in the headline |
| 54 | +return. |
| 55 | + |
| 56 | +### 1.1 A caveat the worked example must not lose |
| 57 | + |
| 58 | +The 0.92 → 0.77 drop is **not** evidence that resting entries add edge. Both sides of that |
| 59 | +comparison lose money gross (0.9219 with the filter, 0.7736 without): the entry condition |
| 60 | +separated **bad from worse**, not good from toxic, and the 08-13 record explicitly forbids |
| 61 | +citing it as evidence that offset entries add alpha. What it *is* evidence for is #260 — |
| 62 | +production silently overrides a rule's stated entry logic, so **the landmine is the next rule |
| 63 | +that expresses a condition through its entry price**. The number makes the hazard's size legible; |
| 64 | +it does not make the remedy "support resting entries" (that is #333, prerequisite-gated on its |
| 65 | +own terms). |
| 66 | + |
| 67 | +## 2. Why a document and not code |
| 68 | + |
| 69 | +There is nothing to enforce. The fill model is already correct — deliberately matched to what |
| 70 | +the executor places, a property the Jesse comparison explicitly names as a keel side of the |
| 71 | +trade. The hazard is in how a **human** reads someone else's results: a reasonable person sees a |
| 72 | +good backtest elsewhere and does not know which questions to ask. The failure mode has already |
| 73 | +occurred once at the engine level (#257: two engine defects produced "plausible, internally |
| 74 | +consistent output" for the life of the project while 2,712 tests passed); this note is the |
| 75 | +reading-side analogue of the invariants the 08-13 record demanded — make the assumption visible |
| 76 | +in the ordinary output of an ordinary evaluation, to a reader who is not hunting for it. |
| 77 | + |
| 78 | +## 3. The four questions |
| 79 | + |
| 80 | +Each below: what to establish, why it matters *here* (with keel's own measurements), and what a |
| 81 | +wrong answer costs. They are asked **before** porting effort, not after a disappointing |
| 82 | +backtest. |
| 83 | + |
| 84 | +### Q1 — What fill model produced the published numbers? |
| 85 | + |
| 86 | +**Establish:** whether entries were resting limit/stop orders at a chosen price or market |
| 87 | +orders, and whether fills are assumed at the chosen level. If the strategy's code names an entry |
| 88 | +price it expects to get (Jesse's `self.buy = qty, entry`), that is a resting entry. |
| 89 | + |
| 90 | +**Why it matters here:** keel's engine fills market-style at the next bar's open (#258) and its |
| 91 | +live executor never rests an order (#260). The worked example above is the measured cost of |
| 92 | +ignoring the difference: gross PF 0.9292 → 0.7736 and median trade count 60 → 124 for |
| 93 | +`pullback_continuation` — a strategy whose edge lives in entry timing loses that edge entirely, |
| 94 | +and *takes extra trades in the process*, because the confirmation it was demanding no longer |
| 95 | +gates anything. |
| 96 | + |
| 97 | +**What a wrong answer costs:** the backtest measures a different strategy than the one |
| 98 | +published, and the shortfall is invisible — keel's engine is internally consistent, so the |
| 99 | +output looks perfectly plausible while bearing no relation to the source's claim. And per §1.1, |
| 100 | +the correction is not "add resting entries" (#333 is deferred until a price-conditional rule |
| 101 | +earns it); it is declining to trust the number. |
| 102 | + |
| 103 | +### Q2 — What cost regime were they developed under? |
| 104 | + |
| 105 | +**Establish:** fee per leg (maker or taker), slippage assumption, and whether the source states |
| 106 | +them at all. "Not stated" is a real answer and a finding. |
| 107 | + |
| 108 | +**Why it matters here:** keel is **cost-bound, not signal-bound** (PRD §0). Round-trip friction |
| 109 | +is `2 × fee_pct + 2 × slippage_pct` — 2.50% of notional at taker, 1.30% at maker |
| 110 | +([`2026-08-12-fee-curve-and-rsi-meanrev.md`](2026-08-12-fee-curve-and-rsi-meanrev.md)) — the |
| 111 | +same order of magnitude as the per-trade edge of everything ever measured here. Seven cells |
| 112 | +(five distinct assets) showed gross PF > 1.0 in the restated intersection and **all seven died |
| 113 | +at the maker rate**, |
| 114 | +before the taker rate actually paid was reached |
| 115 | +([`2026-08-13-restated-under-a-production-faithful-engine.md`](2026-08-13-restated-under-a-production-faithful-engine.md) |
| 116 | +§3). |
| 117 | + |
| 118 | +**What a wrong answer costs:** a result developed on equities at 5bp, or crypto perps at 2bp, |
| 119 | +does not transfer to a 1.2% taker venue, and nothing in the abstract will say so. Combined with |
| 120 | +Q1 this is the compounding: a chosen-price entry *and* a cheap-cost assumption are both |
| 121 | +optimistic at once, multiplicatively. |
| 122 | + |
| 123 | +### Q3 — What sample size and evaluation window? |
| 124 | + |
| 125 | +**Establish:** the trade count the headline rests on, the period it was measured over, and |
| 126 | +whether that period spans more than one regime. |
| 127 | + |
| 128 | +**Why it matters here:** `rsi_meanrev` showed the best gross PF of the three shipped rules — |
| 129 | +median **1.1631** on a median of **38 trades** (the pre-correction anchor) — and collapsed to |
| 130 | +**0.8396** across 82 cells at `n≥100` on the corrected engine (1.1251 at median n=42 on the same |
| 131 | +engine; the level shift appears within either engine: 1.1631 → 0.8938 old, 1.1251 → 0.8396 |
| 132 | +faithful) |
| 133 | +([`2026-08-13-restated-under-a-production-faithful-engine.md`](2026-08-13-restated-under-a-production-faithful-engine.md) |
| 134 | +§4). It was not a decline as trades accumulated; it was a level shift at the measurability |
| 135 | +floor. The window trap bit here too: ZEC-`turtle` printed 1.555 gross across 2021–26 while |
| 136 | +running three consecutive losing years and compressing 92.7% of its lifetime PnL into 2025–26 |
| 137 | +([`2026-08-12-shipped-defaults-intersection.md`](2026-08-12-shipped-defaults-intersection.md) |
| 138 | +§3.2/§6 — the faithful engine later restated the same cell at 1.442, which strengthens rather |
| 139 | +than weakens the point). |
| 140 | + |
| 141 | +**What a wrong answer costs:** a headline profit factor on a small sample is a lottery ticket, |
| 142 | +not an edge. Porting effort is spent on variance, and the resulting backtest — even run honestly |
| 143 | +at keel's own costs and fill model — cannot distinguish the two, because n is exactly what it |
| 144 | +lacks. |
| 145 | + |
| 146 | +### Q4 — Does it need a capability keel does not have? |
| 147 | + |
| 148 | +**Establish:** whether the strategy uses pyramiding, partial exits, conditional entries at a |
| 149 | +price, per-bar position management (`update_position`-style hooks), per-strategy mutable state, |
| 150 | +or fill-event callbacks. |
| 151 | + |
| 152 | +**Why it matters here:** the Jesse comparison (PRD §2/§3) mapped each of these to its keel |
| 153 | +issue — conditional entry at a chosen price is **#333** (deferred until a price-conditional rule |
| 154 | +earns it); partial exits and per-bar live stop management are **#502** (blocked on the |
| 155 | +bracket/OCO order kind — `scale_out` exists with no caller); pluggable rules are **#447**; and |
| 156 | +the executor's discarding of conditional entry prices is **#260**. `detect()` is deliberately a |
| 157 | +pure function of candles with no account, balance or venue access — statefulness is a trade |
| 158 | +keel makes on purpose, not a gap to paper over. |
| 159 | + |
| 160 | +**What a wrong answer costs:** the port silently approximates the strategy — no pyramiding, no |
| 161 | +partial exits, entries at the open — and the approximation is *not* the published strategy. The |
| 162 | +backtest then measures a strawman of your own construction, and neither a good nor a bad result |
| 163 | +from it is evidence about the original claim. |
| 164 | + |
| 165 | +## 4. The checklist — answer before porting effort |
| 166 | + |
| 167 | +Copy this into the evaluation. **"Unknown" is a real answer: for Q1–Q3 it means treat the |
| 168 | +published number as unusable; for Q4 it means the strategy is not evaluable here yet.** |
| 169 | + |
| 170 | +```markdown |
| 171 | +External strategy evaluation — fill-model hazard checklist (#529) |
| 172 | + |
| 173 | +Strategy / source: |
| 174 | +Date evaluated: |
| 175 | + |
| 176 | +1. Fill model. Were the published numbers produced with resting limit/stop |
| 177 | + entries at a chosen price (rather than market-style fills)? |
| 178 | + [ ] no [ ] yes — keel's next-open fill takes the trades the strategy |
| 179 | + meant to decline; budget the pullback_continuation example |
| 180 | + (gross PF 0.92 -> 0.77, trade count doubled) |
| 181 | + [ ] unknown — treat as yes |
| 182 | + |
| 183 | +2. Cost regime. Does the source state fees (per leg, maker/taker) and |
| 184 | + slippage? |
| 185 | + [ ] yes — stated: ___________ (compare: keel round-trip is 2.50% taker) |
| 186 | + [ ] no / not stated — the number is unusable as evidence |
| 187 | + |
| 188 | +3. Sample size and window. Does the source state its trade count and |
| 189 | + evaluation window, and does the window span more than one regime? |
| 190 | + [ ] yes — n = ___, window = ___, regimes = ___ |
| 191 | + [ ] no — a small-sample headline is a lottery ticket |
| 192 | + (rsi_meanrev: 1.1631 at median n=38 vs 0.8396 at n>=100) |
| 193 | + |
| 194 | +4. Capability. Does it require pyramiding, partial exits, conditional |
| 195 | + entries, or per-bar position hooks? |
| 196 | + [ ] no — evaluable as-is |
| 197 | + [ ] yes — which: ___________ (#333 / #502 / #447 / #260 territory; |
| 198 | + not evaluable here until those land) |
| 199 | + |
| 200 | +Verdict: [ ] proceed to port [ ] do not port — reason: ___________ |
| 201 | +``` |
| 202 | + |
| 203 | +A "proceed" here does not mean the strategy is good; it means its published numbers are |
| 204 | +*comparable*. Everything after that is keel's own gate: backtest at the real fee and fill model, |
| 205 | +`n≥100`, and the promotion floors — which is where every candidate so far has died. |
| 206 | + |
| 207 | +## 5. Where this is enforced for machine-generated proposals |
| 208 | + |
| 209 | +The [keel-asset-scout skill](../../.claude/skills/keel-asset-scout/SKILL.md) (the proposing side |
| 210 | +of the §5 proposer/decider asymmetry) already requires its param-proposal citations to carry |
| 211 | +`cost_regime`, `sample_size` and `evaluation_window` fields, and frames performance claims |
| 212 | +against the next-open fill model. This document is the human-facing statement of the same |
| 213 | +discipline; the two are kept in step. |
| 214 | + |
| 215 | +## 6. Provenance of every number |
| 216 | + |
| 217 | +| number | source | |
| 218 | +|---|---| |
| 219 | +| gross PF 0.9292 → 0.7736; median n 60 → 124; n≥100 cells 4 → 14 | [`2026-08-13-restated-under-a-production-faithful-engine.md`](2026-08-13-restated-under-a-production-faithful-engine.md) §3 table | |
| 220 | +| gross 0.9219 → 0.7736; median n 58 → 124; "the doubling *is* the count of those trades" | same, §3.1 | |
| 221 | +| next-bar-open market fill; `order_type="market"`, `limit_price=None`; "free optionality on the entry price … and unbounded patience" | same, §1 (#258); `keel/strategy/backtest.py` module docstring; `keel/execution/executor.py` ("Entry routing is unconditional market (#258, #260)") | |
| 222 | +| round-trip friction 2.50% taker / 1.30% maker (`2 × fee_pct + 2 × slippage_pct`) | [`2026-08-12-fee-curve-and-rsi-meanrev.md`](2026-08-12-fee-curve-and-rsi-meanrev.md) | |
| 223 | +| `rsi_meanrev` 1.1631 at median n=38 (old engine); 0.8396 across 82 cells at n≥100, 1.1251 at median n=42 (faithful engine); old-engine n≥100 pair 1.1631 → 0.8938 across 76 cells | [`2026-08-13-restated-under-a-production-faithful-engine.md`](2026-08-13-restated-under-a-production-faithful-engine.md) §4 | |
| 224 | +| ZEC-`turtle` 1.555 gross, three consecutive losing years, 92.7% of PnL in 2025–26 | [`2026-08-12-shipped-defaults-intersection.md`](2026-08-12-shipped-defaults-intersection.md) §3.2/§6 (faithful-engine restatement: 1.442, 08-13 §2) | |
| 225 | +| "keel is cost-bound, not signal-bound"; Jesse `self.buy = qty, entry`; capability-to-issue map (#333/#447/#502) | [`docs/superpowers/specs/2026-08-23-strategy-api-expressiveness-prd.md`](../superpowers/specs/2026-08-23-strategy-api-expressiveness-prd.md) §0/§2/§3 | |
| 226 | +| Jesse `go_long` snippet | issue #529, quoting the idiom documented in the PRD §2 comparison | |
| 227 | + |
| 228 | +**Related:** #333 (resting-order routing, deferred), #502 (bracket/OCO — partial exits and live |
| 229 | +stop management), #447 (pluggable rules), #260 (executor discards conditional entries), #259 / |
| 230 | +#523 (slippage model and its cap), #257/#258 (the fill-model fix itself). |
0 commit comments