Skip to content

Commit 7d0ccf0

Browse files
committed
feat: url persister
1 parent 720ecb7 commit 7d0ccf0

25 files changed

Lines changed: 5144 additions & 14 deletions

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@
66
# LevelDB
77
**/data/leveldb/
88

9+
# URL persister
10+
**/data/urls.yaml
11+
**/data/urls.yaml.bak
12+
913
# Coverage artifacts
1014
coverage.out
1115
coverage.internal.filtered.out

‎AGENTS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ wait0 is an ultra-fast cache-first HTTP reverse proxy written in Go that serves
3131
│ ├── proxy/ # Request handling/origin fetch/response headers
3232
│ ├── revalidation/ # Revalidate and warmup orchestration
3333
│ ├── discovery/ # Sitemap discovery and URL normalization
34+
│ ├── urlpersist/ # Durable list of cached URLs (YAML file + restore)
3435
│ ├── stats/ # Metrics collector, periodic stats loop, proc probes
3536
│ ├── cachevariant/ # Cache-Variant Expr compilation and evaluation
3637
│ └── cache/ # Cache internals (RAM + LevelDB + codec)
@@ -60,6 +61,7 @@ wait0 is an ultra-fast cache-first HTTP reverse proxy written in Go that serves
6061
| internal/wait0/proxy/controller.go | Main request handling path and cache hit/miss/bypass flow |
6162
| internal/wait0/cachevariant/engine.go | Cache-Variant Expr parsing, compilation, header projection, and evaluation |
6263
| internal/wait0/revalidation/controller.go | Async revalidation and warmup orchestration |
64+
| internal/wait0/urlpersist/controller.go | Tracks successfully cached URLs, flushes them to YAML, and reseeds them on start |
6365
| internal/wait0/cache_ram.go / cache_disk.go | Root cache facades wrapping `internal/wait0/cache` |
6466
| internal/wait0/config.go | YAML configuration loading and rule parsing |
6567
| debug/debug-compose.yml | Spins up reproducible local debug environment |

‎README.md‎

Lines changed: 54 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,9 +53,12 @@ rules:
5353
```bash
5454
docker run --rm -p 8082:8082 \
5555
-v "$(pwd)/wait0.yaml:/wait0.yaml:ro" \
56+
-v wait0-data:/data \
5657
devforth/wait0:latest
5758
```
5859

60+
The image declares `/data` as a volume and wait0 writes everything durable there, including the LevelDB cache and the [URL persister](#url-persister) file. Mount a named volume as shown; with `--rm` and no named volume that data goes to an anonymous volume and is lost on every stop.
61+
5962
3. Alternative: build a tiny wrapper image with your config baked in.
6063

6164
`Dockerfile`:
@@ -133,7 +136,7 @@ Notes:
133136

134137
| Guide | Description |
135138
|-------|-------------|
136-
| [Usage and configuration](#usage-and-configuration) | [How wait0 works](#how-wait0-works), [cache bypass](#cache-bypass), [cache variants](#cache-variants), [query parameters](#query-parameter-caching), [sitemap warmup](#sitemap-warmup), and the [config reference](#config-file-reference) |
139+
| [Usage and configuration](#usage-and-configuration) | [How wait0 works](#how-wait0-works), [cache bypass](#cache-bypass), [cache variants](#cache-variants), [query parameters](#query-parameter-caching), [sitemap warmup](#sitemap-warmup), [URL persister](#url-persister), and the [config reference](#config-file-reference) |
137140
| [For Developers](docs/for-developers.md) | Build/test commands, config reference, runtime options |
138141
| [API Endpoints](docs/api-endpoints.md) | Proxy behavior, invalidation API, schemas, status codes |
139142
| [Cache variant complexity](docs/cache-variant-complexity.md) | Root/subkey storage model and operation complexity |
@@ -246,6 +249,38 @@ Each warmup rule starts its first loop immediately and processes one complete UR
246249

247250
Dashboard statistics retain only the latest completed warmup loop per rule. Fastest, slowest, largest, and smallest URL rankings are capped at 10 entries while they are collected, so wait0 does not accumulate warmup history or discarded ranking candidates.
248251

252+
### URL persister
253+
254+
`urlPersister` remembers every URL that was successfully stored in the cache and writes that list to a YAML file. On the next start the file is read back and each remembered URL is registered as an inactive cache entry, so the existing warmup machinery refetches it. This is what carries a working set across the start-up disk cache wipe described in [Redeploy Note](#redeploy-note), which otherwise leaves every restart cold.
255+
256+
```yaml
257+
urlPersister:
258+
enabled: true
259+
file: './data/urls.yaml'
260+
flushEvery: '30s'
261+
restoreOnStart: true
262+
maxUrls: 50000
263+
forgetAfterFailures: 3
264+
```
265+
266+
Mount the directory that holds `file` as a volume, otherwise the list disappears with the container and the feature does nothing. The default path `./data/urls.yaml` resolves inside `/data`, which the official image already declares as a volume. The file must not be placed inside `./data/leveldb`; that directory is wiped on start, and such a path is rejected when the config is loaded.
267+
268+
The URL persister works with a sitemap or without one, and can replace one: its list is built from URLs traffic actually requested and wait0 actually cached, so it also covers pages no sitemap advertises. Sitemap-discovered URLs join the same list once they have been cached.
269+
270+
Notes:
271+
272+
- Tracking is immediate. `flushEvery` only controls how often the in-memory list is written to the file, and a write is skipped entirely when nothing changed.
273+
- Restoring a URL only registers it; nothing is fetched by the restore itself. A restored URL is refetched only when a matching rule has a `warmUp` block. Paths with no matching rule, or a rule with `bypass: true`, are skipped.
274+
- Cache variants are remembered by the request-header values their `Cache-Variant` expressions reference, which is what makes a variant replayable after restart. There is one record per distinct tuple of evaluated variant values, not one per request: a thousand `User-Agent` strings that all evaluate to `mobile` collapse into a single record.
275+
- Query-aware entries created by `varyByQueryParams` are remembered with their query and stay separate records.
276+
- `maxUrls` caps the file. When it is exceeded, the least recently used URLs are dropped.
277+
- When a remembered URL stops being cacheable its cache entry is deleted and the record counts one failure; the record is forgotten once it reaches `forgetAfterFailures`, or on the first failure when the value is `0`. A successful store resets the counter. Warmup cannot retry a URL after its entry is deleted, so a URL that only warmup touches costs one failed request per start; a URL that clients still request counts a failure per request. Either way a dead URL is never hammered, and a brief origin outage does not erase the file.
278+
- The file contains URL identities only: paths, query strings, variant values, and the referenced request headers. No response bodies and no response headers are written.
279+
- A missing or corrupt file never blocks startup. wait0 falls back to the `.bak` copy written beside it, then to an empty list.
280+
- While enabled, the stats API adds a `url_persister` object reporting `records`, `restored`, and `last_flush_unix`. It is omitted when the feature is off.
281+
282+
> **Warning:** when `urlPersister` is enabled, do not reference per-user request headers such as `Cookie` or `Authorization` in `Cache-Variant` expressions. Their values are written into this file in plain text so the variant can be replayed later, and an expression that returns such a value verbatim also echoes it in the `X-Wait0-Cache-Variant-Key` response header. Use expressions that collapse requests into a small shared set of values such as `mobile`/`desktop` or a country code.
283+
249284
### Config file reference
250285

251286
This example contains every current configuration option. Durations use Go syntax such as `10s`, `1m`, or `2h`; sizes accept bytes or `k`, `m`, and `g` suffixes.
@@ -301,6 +336,23 @@ urlsDiscover:
301336
sitemaps:
302337
- 'https://example.com/sitemap.xml'
303338
339+
urlPersister:
340+
# Remembers successfully cached URLs across restarts; default false.
341+
enabled: true
342+
# File the list is written to; must be outside ./data/leveldb, which is
343+
# wiped on start. Default './data/urls.yaml'.
344+
file: './data/urls.yaml'
345+
# How often the list is written to the file; default 30s. Tracking itself is
346+
# immediate, and an unchanged list is not rewritten.
347+
flushEvery: '30s'
348+
# Reads the file on start and registers each URL for warmup; default true.
349+
restoreOnStart: true
350+
# Maximum remembered URLs; least recently used are dropped. Default 50000.
351+
maxUrls: 50000
352+
# Failing starts a URL survives before it is forgotten; 0 forgets it on the
353+
# first failure. Default 3.
354+
forgetAfterFailures: 3
355+
304356
rules:
305357
# Matches path prefixes; combine alternatives with |.
306358
- match: PathPrefix(/api) | PathPrefix(/admin)
@@ -385,3 +437,4 @@ If you do not restart between deploys, proactively refresh cache using invalidat
385437
- On a cache-path miss or revalidation, an origin non-`2xx` is not cached and any existing key is evicted.
386438
- Invalidation is asynchronous: accept request -> resolve keys by `paths` and `tags` -> delete keys -> recrawl in background. Path invalidation clears all cached query-aware variants for that path.
387439
- Warmup monitors discovered cache variants and may expand `warmupRequestHeaderPresets` into additional Cartesian request-header combinations.
440+
- `urlPersister` keeps successfully cached URL identities in a YAML file outside LevelDB and, on start, re-registers them as inactive entries so warmup repopulates the wiped disk cache.

‎debug/wait0.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,17 @@ urlsDiscover:
3535
sitemaps:
3636
- https://devforth.io/sitemap.xml
3737

38+
# Remembers successfully cached URLs so the start-up LevelDB wipe does not
39+
# leave the cache cold. Uncomment to try it locally; the file is written
40+
# relative to the working directory and must stay outside ./data/leveldb.
41+
# urlPersister:
42+
# enabled: true
43+
# file: './data/urls.yaml'
44+
# flushEvery: '30s'
45+
# restoreOnStart: true
46+
# maxUrls: 50000
47+
# forgetAfterFailures: 3
48+
3849
rules:
3950
- match: PathPrefix(/api) | PathPrefix(/admin) | PathPrefix(/lab)
4051
priority: 1

‎docs/api-endpoints.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,11 @@ Status: `200 OK`
128128
"crawled_urls": 60,
129129
"crawl_percentage": 75
130130
},
131+
"url_persister": {
132+
"records": 1240,
133+
"restored": 1180,
134+
"last_flush_unix": 1755252664
135+
},
131136
"rules": [
132137
{
133138
"id": 0,
@@ -175,6 +180,10 @@ The table below explains each field in the stats payload, including what it mean
175180
| `sitemap.discovered_urls` | integer | Number of unique cached keys whose discovery source is sitemap. | Count of unique keys where `discovered_by == "sitemap"` (case-insensitive). | Recomputed per snapshot. |
176181
| `sitemap.crawled_urls` | integer | Number of sitemap-discovered keys that are currently active (not inactive seed entries). | Count of sitemap keys where `inactive == false`. | Recomputed per snapshot. |
177182
| `sitemap.crawl_percentage` | float | Share of sitemap-discovered keys currently crawled/active. | `crawled_urls * 100 / discovered_urls`; `0` if `discovered_urls == 0`. | Recomputed per snapshot. |
183+
| `url_persister` | object | Durable URL list state. Omitted entirely when `urlPersister.enabled` is false. | Read from the in-memory registry. | Absent key means the feature is off. |
184+
| `url_persister.records` | integer | URLs currently remembered, including variants as separate records. | Size of the registry, which is capped by `maxUrls`. | Recomputed per snapshot. |
185+
| `url_persister.restored` | integer | Records seeded from the file at the last start. | Excludes records skipped because no rule matched or the rule bypasses. | `0` when `restoreOnStart` is false or the file was missing. |
186+
| `url_persister.last_flush_unix` | integer (unix seconds) | When the file was last written. | Updated only on a successful write; a flush with nothing dirty does not write. | `0` before the first write. |
178187
| `rules[]` | array | One entry for every configured rule, including rules with zero URLs or responses. | Configuration order after priority sorting. Cached keys are assigned to the first matching rule. | Recomputed per snapshot. |
179188
| `rules[].urls` | integer | Unique logical root URLs assigned to this rule, including inactive discovery seeds. | Union of matching RAM and disk keys after variant subkeys are folded into their base root. | `0` when empty. |
180189
| `rules[].responses` | integer | Active concrete responses assigned to this rule. | Matching non-manifest keys where `inactive == false`; each concrete variant is one response. | `0` when the rule has no responses. |

‎docs/for-developers.md‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,19 @@ For dashboard:
155155
| `initalDelay` | duration | Legacy typo still supported |
156156
| `rediscoverEvery` | duration | Periodic rediscovery interval (`> 0`) |
157157

158+
## `urlPersister`
159+
160+
| Field | Type | Default | Notes |
161+
|-------|------|---------|------|
162+
| `enabled` | bool | `false` | Remembers successfully cached URLs in a YAML file; everything below applies only when `true` |
163+
| `file` | path | `./data/urls.yaml` | Destination file; a `.bak` copy of the previous generation is kept beside it |
164+
| `flushEvery` | duration | `30s` | File write interval (`> 0`); tracking is immediate and an unchanged list is not rewritten |
165+
| `restoreOnStart` | bool | `true` | Reads the file on start and registers each URL as an inactive cache entry |
166+
| `maxUrls` | int | `50000` | Cap on remembered URLs (`> 0`); the least recently used are dropped |
167+
| `forgetAfterFailures` | int | `3` | Failures a URL survives before it is forgotten (`>= 0`); `0` forgets it on the first failure, and a successful store resets the counter. Warmup contributes at most one failure per start because a deleted entry is never retried; client requests contribute one each |
168+
169+
Validation rules: `file` must not resolve inside `./data/leveldb`, `flushEvery` must be `> 0`, `maxUrls` must be `> 0`, `forgetAfterFailures` must be `>= 0`.
170+
158171
## `logging`
159172

160173
| Field | Type | Notes |
@@ -179,7 +192,13 @@ For dashboard:
179192
- Origin `2xx` responses with `Cache-Control: no-cache` or `no-store` are not stored; either directive received during revalidation deletes the existing entry.
180193
- Warmup loops never overlap: wait0 completes the current rule snapshot, waits `pauseBetweenRuns`, then loads a fresh snapshot.
181194
- Variant warmup replays the original URL with the referenced request headers saved on each discovered concrete response. `warmupRequestHeaderPresets` adds its full Cartesian product and can substantially increase cache entries and origin traffic.
182-
- `Cache-Variant` expressions may reference every request header, including cookies and authorization. Referenced values are persisted for later refresh, so protecting sensitive values is the operator's responsibility.
195+
- `Cache-Variant` expressions may reference every request header, including cookies and authorization. Referenced values are persisted for later refresh, so protecting sensitive values is the operator's responsibility. With `urlPersister` enabled those values also leave the process and land in the persisted YAML file in plain text.
196+
- `urlPersister.file` must live outside `./data/leveldb`: that directory is deleted on every start when `WAIT0_INVALIDATE_DISK_CACHE_ON_START` is `true`, and surviving exactly that wipe is the point of the feature. A path inside it is rejected by config validation, so wait0 refuses to start.
197+
- Mount the directory containing `urlPersister.file` as a volume. The image declares `/data`, and the default `./data/urls.yaml` resolves inside it.
198+
- A missing, truncated, or otherwise corrupt persisted file never blocks startup: wait0 logs the problem, retries the `.bak` copy, and then continues with an empty list.
199+
- Restoring writes inactive placeholder entries only; it fetches nothing. Restored URLs are refetched by the normal warmup loop, so they need a matching rule with a `warmUp` block, and paths matching a `bypass: true` rule are skipped.
200+
- One record is stored per cache key, so a variant family contributes one record per distinct tuple of evaluated variant values rather than one per request. Records carrying a superseded `Cache-Variant` fingerprint are pruned when a new generation is stored.
201+
- `GET /wait0` includes a `url_persister` object (`records`, `restored`, `last_flush_unix`) while the feature is enabled and omits it otherwise.
183202
- `logging.debug_headers` may select any of `X-Wait0`, `X-Wait0-Reason`, `X-Wait0-Revalidated-At`, `X-Wait0-Revalidated-By`, `X-Wait0-Discovered-By`, `X-Wait0-Revalidate-At`, `X-Wait0-Revalidate-Entropy`, and `X-Wait0-Cache-Variant-Key`.
184203
- Omit `logging.debug_headers` to enable all diagnostics; set `debug_headers: []` to disable all of them.
185204
- `X-Wait0` response header identifies behavior (`hit`, `miss`, `bypass`, `ignore-by-cookie`, `ignore-by-request-header`, `ignore-by-status`, `bad-gateway`).

‎internal/wait0/cache/disk.go‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,11 @@ type diskOp struct {
2828
putKey string
2929
putEnt *Entry
3030
delKey string
31+
// accessUnix overrides the last-access stamp for this put. Zero keeps the
32+
// default of "now", which is what every live write wants; restoring a
33+
// remembered URL passes the time it was actually last used so warmup order
34+
// survives a restart.
35+
accessUnix int64
3136
}
3237

3338
type Disk struct {
@@ -178,6 +183,13 @@ func (d *Disk) PutAsync(key string, ent Entry) {
178183
d.ops <- diskOp{putKey: key, putEnt: &clone}
179184
}
180185

186+
// PutAsyncWithAccess stores ent with an explicit last-access time instead of
187+
// the write time.
188+
func (d *Disk) PutAsyncWithAccess(key string, ent Entry, accessUnix int64) {
189+
clone := ent
190+
d.ops <- diskOp{putKey: key, putEnt: &clone, accessUnix: accessUnix}
191+
}
192+
181193
func (d *Disk) Delete(key string) {
182194
d.ops <- diskOp{delKey: key}
183195
}
@@ -222,13 +234,16 @@ func (d *Disk) writerLoop() {
222234
continue
223235
}
224236
if op.putKey != "" {
225-
d.applyPutOrTouch(op.putKey, op.putEnt)
237+
d.applyPutOrTouch(op.putKey, op.putEnt, op.accessUnix)
226238
}
227239
}
228240
}
229241

230-
func (d *Disk) applyPutOrTouch(key string, ent *Entry) {
242+
func (d *Disk) applyPutOrTouch(key string, ent *Entry, accessUnix int64) {
231243
now := time.Now().Unix()
244+
if accessUnix > 0 {
245+
now = accessUnix
246+
}
232247

233248
d.mu.Lock()
234249
meta := d.index[key]

‎internal/wait0/cache_disk.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,10 @@ func (d *diskCache) PutAsync(key string, ent CacheEntry) {
6262
d.inner.PutAsync(key, fromWait0Entry(ent))
6363
}
6464

65+
func (d *diskCache) PutAsyncWithAccess(key string, ent CacheEntry, accessUnix int64) {
66+
d.inner.PutAsyncWithAccess(key, fromWait0Entry(ent), accessUnix)
67+
}
68+
6569
func (d *diskCache) Delete(key string) {
6670
d.inner.Delete(key)
6771
}

0 commit comments

Comments
 (0)