You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7d0ccf0
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: README.md
+54-1Lines changed: 54 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -53,9 +53,12 @@ rules:
53
53
```bash
54
54
docker run --rm -p 8082:8082 \
55
55
-v "$(pwd)/wait0.yaml:/wait0.yaml:ro" \
56
+
-v wait0-data:/data \
56
57
devforth/wait0:latest
57
58
```
58
59
60
+
The image declares `/data` as a volume and wait0 writes everything durable there, including the LevelDB cache and the [URL persister](#url-persister) file. Mount a named volume as shown; with `--rm` and no named volume that data goes to an anonymous volume and is lost on every stop.
61
+
59
62
3. Alternative: build a tiny wrapper image with your config baked in.
60
63
61
64
`Dockerfile`:
@@ -133,7 +136,7 @@ Notes:
133
136
134
137
| Guide | Description |
135
138
|-------|-------------|
136
-
| [Usage and configuration](#usage-and-configuration) | [How wait0 works](#how-wait0-works), [cache bypass](#cache-bypass), [cache variants](#cache-variants), [query parameters](#query-parameter-caching), [sitemap warmup](#sitemap-warmup), and the [config reference](#config-file-reference) |
139
+
| [Usage and configuration](#usage-and-configuration) | [How wait0 works](#how-wait0-works), [cache bypass](#cache-bypass), [cache variants](#cache-variants), [query parameters](#query-parameter-caching), [sitemap warmup](#sitemap-warmup), [URL persister](#url-persister), and the [config reference](#config-file-reference) |
| [Cache variant complexity](docs/cache-variant-complexity.md) | Root/subkey storage model and operation complexity |
@@ -246,6 +249,38 @@ Each warmup rule starts its first loop immediately and processes one complete UR
246
249
247
250
Dashboard statistics retain only the latest completed warmup loop per rule. Fastest, slowest, largest, and smallest URL rankings are capped at 10 entries while they are collected, so wait0 does not accumulate warmup history or discarded ranking candidates.
248
251
252
+
### URL persister
253
+
254
+
`urlPersister`remembers every URL that was successfully stored in the cache and writes that list to a YAML file. On the next start the file is read back and each remembered URL is registered as an inactive cache entry, so the existing warmup machinery refetches it. This is what carries a working set across the start-up disk cache wipe described in [Redeploy Note](#redeploy-note), which otherwise leaves every restart cold.
255
+
256
+
```yaml
257
+
urlPersister:
258
+
enabled: true
259
+
file: './data/urls.yaml'
260
+
flushEvery: '30s'
261
+
restoreOnStart: true
262
+
maxUrls: 50000
263
+
forgetAfterFailures: 3
264
+
```
265
+
266
+
Mount the directory that holds `file` as a volume, otherwise the list disappears with the container and the feature does nothing. The default path `./data/urls.yaml` resolves inside `/data`, which the official image already declares as a volume. The file must not be placed inside `./data/leveldb`; that directory is wiped on start, and such a path is rejected when the config is loaded.
267
+
268
+
The URL persister works with a sitemap or without one, and can replace one: its list is built from URLs traffic actually requested and wait0 actually cached, so it also covers pages no sitemap advertises. Sitemap-discovered URLs join the same list once they have been cached.
269
+
270
+
Notes:
271
+
272
+
- Tracking is immediate. `flushEvery` only controls how often the in-memory list is written to the file, and a write is skipped entirely when nothing changed.
273
+
- Restoring a URL only registers it; nothing is fetched by the restore itself. A restored URL is refetched only when a matching rule has a `warmUp` block. Paths with no matching rule, or a rule with `bypass: true`, are skipped.
274
+
- Cache variants are remembered by the request-header values their `Cache-Variant` expressions reference, which is what makes a variant replayable after restart. There is one record per distinct tuple of evaluated variant values, not one per request: a thousand `User-Agent` strings that all evaluate to `mobile` collapse into a single record.
275
+
- Query-aware entries created by `varyByQueryParams` are remembered with their query and stay separate records.
276
+
- `maxUrls`caps the file. When it is exceeded, the least recently used URLs are dropped.
277
+
- When a remembered URL stops being cacheable its cache entry is deleted and the record counts one failure; the record is forgotten once it reaches `forgetAfterFailures`, or on the first failure when the value is `0`. A successful store resets the counter. Warmup cannot retry a URL after its entry is deleted, so a URL that only warmup touches costs one failed request per start; a URL that clients still request counts a failure per request. Either way a dead URL is never hammered, and a brief origin outage does not erase the file.
278
+
- The file contains URL identities only: paths, query strings, variant values, and the referenced request headers. No response bodies and no response headers are written.
279
+
- A missing or corrupt file never blocks startup. wait0 falls back to the `.bak` copy written beside it, then to an empty list.
280
+
- While enabled, the stats API adds a `url_persister` object reporting `records`, `restored`, and `last_flush_unix`. It is omitted when the feature is off.
281
+
282
+
> **Warning:** when `urlPersister` is enabled, do not reference per-user request headers such as `Cookie` or `Authorization` in `Cache-Variant` expressions. Their values are written into this file in plain text so the variant can be replayed later, and an expression that returns such a value verbatim also echoes it in the `X-Wait0-Cache-Variant-Key` response header. Use expressions that collapse requests into a small shared set of values such as `mobile`/`desktop` or a country code.
283
+
249
284
### Config file reference
250
285
251
286
This example contains every current configuration option. Durations use Go syntax such as `10s`, `1m`, or `2h`; sizes accept bytes or `k`, `m`, and `g` suffixes.
@@ -301,6 +336,23 @@ urlsDiscover:
301
336
sitemaps:
302
337
- 'https://example.com/sitemap.xml'
303
338
339
+
urlPersister:
340
+
# Remembers successfully cached URLs across restarts; default false.
341
+
enabled: true
342
+
# File the list is written to; must be outside ./data/leveldb, which is
343
+
# wiped on start. Default './data/urls.yaml'.
344
+
file: './data/urls.yaml'
345
+
# How often the list is written to the file; default 30s. Tracking itself is
346
+
# immediate, and an unchanged list is not rewritten.
347
+
flushEvery: '30s'
348
+
# Reads the file on start and registers each URL for warmup; default true.
349
+
restoreOnStart: true
350
+
# Maximum remembered URLs; least recently used are dropped. Default 50000.
351
+
maxUrls: 50000
352
+
# Failing starts a URL survives before it is forgotten; 0 forgets it on the
353
+
# first failure. Default 3.
354
+
forgetAfterFailures: 3
355
+
304
356
rules:
305
357
# Matches path prefixes; combine alternatives with |.
306
358
- match: PathPrefix(/api) | PathPrefix(/admin)
@@ -385,3 +437,4 @@ If you do not restart between deploys, proactively refresh cache using invalidat
385
437
- On a cache-path miss or revalidation, an origin non-`2xx` is not cached and any existing key is evicted.
386
438
- Invalidation is asynchronous: accept request -> resolve keys by `paths` and `tags` -> delete keys -> recrawl in background. Path invalidation clears all cached query-aware variants for that path.
387
439
- Warmup monitors discovered cache variants and may expand `warmupRequestHeaderPresets` into additional Cartesian request-header combinations.
440
+
- `urlPersister`keeps successfully cached URL identities in a YAML file outside LevelDB and, on start, re-registers them as inactive entries so warmup repopulates the wiped disk cache.
Copy file name to clipboardExpand all lines: docs/api-endpoints.md
+9Lines changed: 9 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -128,6 +128,11 @@ Status: `200 OK`
128
128
"crawled_urls": 60,
129
129
"crawl_percentage": 75
130
130
},
131
+
"url_persister": {
132
+
"records": 1240,
133
+
"restored": 1180,
134
+
"last_flush_unix": 1755252664
135
+
},
131
136
"rules": [
132
137
{
133
138
"id": 0,
@@ -175,6 +180,10 @@ The table below explains each field in the stats payload, including what it mean
175
180
|`sitemap.discovered_urls`| integer | Number of unique cached keys whose discovery source is sitemap. | Count of unique keys where `discovered_by == "sitemap"` (case-insensitive). | Recomputed per snapshot. |
176
181
|`sitemap.crawled_urls`| integer | Number of sitemap-discovered keys that are currently active (not inactive seed entries). | Count of sitemap keys where `inactive == false`. | Recomputed per snapshot. |
177
182
|`sitemap.crawl_percentage`| float | Share of sitemap-discovered keys currently crawled/active. |`crawled_urls * 100 / discovered_urls`; `0` if `discovered_urls == 0`. | Recomputed per snapshot. |
183
+
|`url_persister`| object | Durable URL list state. Omitted entirely when `urlPersister.enabled` is false. | Read from the in-memory registry. | Absent key means the feature is off. |
184
+
|`url_persister.records`| integer | URLs currently remembered, including variants as separate records. | Size of the registry, which is capped by `maxUrls`. | Recomputed per snapshot. |
185
+
|`url_persister.restored`| integer | Records seeded from the file at the last start. | Excludes records skipped because no rule matched or the rule bypasses. |`0` when `restoreOnStart` is false or the file was missing. |
186
+
|`url_persister.last_flush_unix`| integer (unix seconds) | When the file was last written. | Updated only on a successful write; a flush with nothing dirty does not write. |`0` before the first write. |
178
187
|`rules[]`| array | One entry for every configured rule, including rules with zero URLs or responses. | Configuration order after priority sorting. Cached keys are assigned to the first matching rule. | Recomputed per snapshot. |
179
188
|`rules[].urls`| integer | Unique logical root URLs assigned to this rule, including inactive discovery seeds. | Union of matching RAM and disk keys after variant subkeys are folded into their base root. |`0` when empty. |
180
189
|`rules[].responses`| integer | Active concrete responses assigned to this rule. | Matching non-manifest keys where `inactive == false`; each concrete variant is one response. |`0` when the rule has no responses. |
|`enabled`| bool |`false`| Remembers successfully cached URLs in a YAML file; everything below applies only when `true`|
163
+
|`file`| path |`./data/urls.yaml`| Destination file; a `.bak` copy of the previous generation is kept beside it |
164
+
|`flushEvery`| duration |`30s`| File write interval (`> 0`); tracking is immediate and an unchanged list is not rewritten |
165
+
|`restoreOnStart`| bool |`true`| Reads the file on start and registers each URL as an inactive cache entry |
166
+
|`maxUrls`| int |`50000`| Cap on remembered URLs (`> 0`); the least recently used are dropped |
167
+
|`forgetAfterFailures`| int |`3`| Failures a URL survives before it is forgotten (`>= 0`); `0` forgets it on the first failure, and a successful store resets the counter. Warmup contributes at most one failure per start because a deleted entry is never retried; client requests contribute one each |
168
+
169
+
Validation rules: `file` must not resolve inside `./data/leveldb`, `flushEvery` must be `> 0`, `maxUrls` must be `> 0`, `forgetAfterFailures` must be `>= 0`.
170
+
158
171
## `logging`
159
172
160
173
| Field | Type | Notes |
@@ -179,7 +192,13 @@ For dashboard:
179
192
- Origin `2xx` responses with `Cache-Control: no-cache` or `no-store` are not stored; either directive received during revalidation deletes the existing entry.
180
193
- Warmup loops never overlap: wait0 completes the current rule snapshot, waits `pauseBetweenRuns`, then loads a fresh snapshot.
181
194
- Variant warmup replays the original URL with the referenced request headers saved on each discovered concrete response. `warmupRequestHeaderPresets` adds its full Cartesian product and can substantially increase cache entries and origin traffic.
182
-
-`Cache-Variant` expressions may reference every request header, including cookies and authorization. Referenced values are persisted for later refresh, so protecting sensitive values is the operator's responsibility.
195
+
-`Cache-Variant` expressions may reference every request header, including cookies and authorization. Referenced values are persisted for later refresh, so protecting sensitive values is the operator's responsibility. With `urlPersister` enabled those values also leave the process and land in the persisted YAML file in plain text.
196
+
-`urlPersister.file` must live outside `./data/leveldb`: that directory is deleted on every start when `WAIT0_INVALIDATE_DISK_CACHE_ON_START` is `true`, and surviving exactly that wipe is the point of the feature. A path inside it is rejected by config validation, so wait0 refuses to start.
197
+
- Mount the directory containing `urlPersister.file` as a volume. The image declares `/data`, and the default `./data/urls.yaml` resolves inside it.
198
+
- A missing, truncated, or otherwise corrupt persisted file never blocks startup: wait0 logs the problem, retries the `.bak` copy, and then continues with an empty list.
199
+
- Restoring writes inactive placeholder entries only; it fetches nothing. Restored URLs are refetched by the normal warmup loop, so they need a matching rule with a `warmUp` block, and paths matching a `bypass: true` rule are skipped.
200
+
- One record is stored per cache key, so a variant family contributes one record per distinct tuple of evaluated variant values rather than one per request. Records carrying a superseded `Cache-Variant` fingerprint are pruned when a new generation is stored.
201
+
-`GET /wait0` includes a `url_persister` object (`records`, `restored`, `last_flush_unix`) while the feature is enabled and omits it otherwise.
183
202
-`logging.debug_headers` may select any of `X-Wait0`, `X-Wait0-Reason`, `X-Wait0-Revalidated-At`, `X-Wait0-Revalidated-By`, `X-Wait0-Discovered-By`, `X-Wait0-Revalidate-At`, `X-Wait0-Revalidate-Entropy`, and `X-Wait0-Cache-Variant-Key`.
184
203
- Omit `logging.debug_headers` to enable all diagnostics; set `debug_headers: []` to disable all of them.
0 commit comments