perf(iceberg): cold/floor program — parallel catalog resolution, persisted catalog cache, 429 hardening (PR-8)#1491
Open
aaj3f wants to merge 4 commits into
Open
Conversation
…PR-8 slice 1) A multi-table virtual-dataset query resolved each table's Iceberg `loadTable` SERIALLY — the fused aggregate-over-join scanned fact + each dim one at a time, and the generic multi-TriplesMap loop did the same — so a k-table query paid k sequential ~1.7s REST round-trips before any data moved. This adds a best-effort `prefetch_tables` that warms a query's per-table catalog contexts CONCURRENTLY (`buffered(8)`, request-order-preserving per 0ade90c) before the serial scan loops, so the per-table GETs overlap instead of summing. It is purely additive: errors are swallowed and the real scan re-resolves and surfaces them, it dedupes against tables already resolved in the session pin, and it is gated by `FLUREE_R2RML_PARALLEL_CATALOG` (off => today's serial behavior). Folded in — the missing half of the session-pin design: the per-query `IcebergCatalogSession` pinned the `loadTable` RESPONSE but not the `S3IcebergStorage` client built FROM it, so EVERY scan rebuilt the AWS SDK client (`aws_config` load + S3 + HTTP client). That is a pre-existing defect on its own (a correlated join re-scanning a dim paid it per re-scan), and it made naive prefetch a net loss (warm + scan = a double build that canceled the parallelism gain). The session now caches `Arc<S3IcebergStorage>` keyed like the pin and reuses it; `store_load_table` invalidates it on any fresh loadTable so a credential refresh rebuilds the client and a stale-creds client is never served (unit-tested: store_load_table_invalidates_cached_storage_on_creds_refresh). Also lands the PR-8 cold-floor decomposition sub-spans (`r2rml.read_metadata`, `r2rml.count_manifest_read`) and the slice-1 engagement span (`r2rml.prefetch`, with `requested`/`warmed`), all allowlisted + callsite-verified in fluree-bench-virtual, plus the PR-8 design doc. Validated in the WARM-DISK state (fresh process = catalog caches cold, on-disk artifact cache warm = fast scan), where the catalog slice is the wall rather than being drowned in ~40s of cold S3 fetch. q008 (fact + 2 dims), median of 3 reps: prefetch ON 3200 ms (reps 4475 / 3190 / 3200) prefetch OFF 6051 ms (reps 6051 / 8779 / 5685) -> 2.85s / 47%, non-overlapping Engagement proven: `r2rml.prefetch` n=1 in every ON rep, absent in every OFF rep. `r2rml.scan_table` span-sum collapses ON 0.65s vs OFF 5.7s (scans hit the warm pins + reused client), while OAuth stays a single exchange even fully concurrent (buffered's cooperative polling builds the client once). Parity: all six ON/OFF result hashes identical to the blessed native oracle — warming caches and reusing a client cannot change rows. Cold walls stay data-fetch-dominated for every corpus fact query, so the cold A/B cannot resolve the catalog slice (±4s S3 noise > the ~3s effect) — the warm-disk A/B is slice-1's DoD gate; see 11-pr8-cold-floor.md sect 7. Cross-pattern non-fused joins resolve tables in separate operators the join pulls serially and are out of an in-operator prefetch's reach; deferred as a documented follow-up.
…he (PR-8 slice 2)
Every cold process re-read a table's parsed `TableMetadata`, its manifest-derived
scan file list, and (for a `COUNT(*)`) its manifest record-count stats from S3 —
even though all three are IMMUTABLE for a given Iceberg snapshot. This persists
them to a small on-disk cache so a cold process with a warm catalog dir serves
them from local disk instead. It does NOT persist any credential or token (ruling
ii): a cold process still issues one `loadTable` GET for fresh vended creds — the
irreducible floor — this only removes the metadata + manifest S3 round-trips that
follow it.
No TTL, no pointer, no invalidation logic. The key is the `metadata_location`, a
content-addressed S3 path: a table commit yields a NEW location = a NEW key = a
clean miss, and a given key's value can never go stale. That is the whole design —
three content-addressed stores keyed by `metadata_location`:
- `TableMetadata` -> removes the `r2rml.read_metadata` S3 GET
- `scan_files` (unfiltered list) -> removes the scan path's `iceberg.scan_plan`
- COUNT(*) manifest stats -> removes `r2rml.count_manifest_read`
The scan_files entry is the UNFILTERED full file list; the in-memory cache is
already bypassed when a pushdown filter prunes, so it is immutable per snapshot.
New module `disk_catalog_cache.rs`, wired at the three read sites in `r2rml.rs`
(in-memory miss -> disk -> S3), in a DEDICATED dir sibling to the Parquet/binary
artifact cache (`<dir>-catalog`) so the cold benchmark protocol clears data while
keeping catalog persistence. Switch `FLUREE_ICEBERG_CATALOG_DISK_CACHE` (default
on). Durability: a versioned `{format_version, payload}` envelope (mismatch or any
deserialize failure = miss + drop the file, never a surfaced error), atomic
temp-file + rename writes (a mid-write crash can't leave a torn file a read
trusts), and an mtime-LRU prune to 512 MiB once per process. `DataFile` (+
`FileFormat`, `PartitionData`) gained `Serialize/Deserialize` in fluree-db-iceberg
to persist the file lists.
Validated with the COLD-DATA / WARM-CATALOG gate (two `--cold` runs sharing one
`--cache-dir`; run 2 clears the data artifacts but reads the warm catalog
sibling):
q036 (COUNT) run1 wall 4274ms -> run2 2060ms
r2rml.read_metadata n=1 (298ms) -> n=0
r2rml.count_manifest_read n=1 (986ms) -> n=0
r2rml.load_table n=1 -> n=1 (the irreducible GET stays)
q001 (scan) run1 wall 3964ms -> run2 3723ms
r2rml.read_metadata n=1 (154ms) -> n=0
iceberg.scan_plan n=1 (113ms) -> n=0
Parity: run1 == run2 == blessed native oracle on both (compare: 0 mismatches).
Persisting immutable, secret-free data can't change rows. Hermetic tests cover the
serde round-trip, version-mismatch-as-miss, corrupt-file-as-miss, and the disabled
switch; 99/99 graph_source lib tests regression-clean. See 11-pr8-cold-floor.md
sect 8.
… (PR-8 slice 3)
The REST catalog client had no rate-limit handling at all: `request_with_retry`
retried a 401 (token refresh) once and treated every other non-2xx as terminal,
and nothing bounded how many catalog requests could be in flight at once. PR-2's
raised scan concurrency, the slice-1 prefetch fan-out, and the wildcard crawl can
each issue a burst of `loadTable`/list calls, so a `429 Too Many Requests` from
Snowflake Horizon would surface as a hard query error. This adds the two missing
pieces — safety hardening, not a latency lever.
429/503 backoff: `request_with_retry` now loops. A `429` or `503` is retried
honoring a `Retry-After` header when present, else exponential backoff with full
jitter (base 250ms, doubling, capped at 8s), bounded to
`FLUREE_ICEBERG_CATALOG_MAX_RETRIES` (default 4) attempts, then the error is
surfaced cleanly (a `Catalog` error naming the status). The existing
401-refresh-and-retry is preserved and drops its semaphore permit before the
recursive call so a small cap can't self-deadlock.
Catalog semaphore: every catalog request now acquires a permit from a
process-wide `Semaphore` (sized by `FLUREE_ICEBERG_CATALOG_CONCURRENCY`, default
8) before its round-trip, held across the backoff so a throttled request keeps
its slot while it waits. The permit is a shared `Arc` cloned into every client,
so all clients bound one pool; it is injectable per-client for tests. Because the
acquire lives in `request_with_retry`, the slice-1 prefetch's `buffered(8)`
fan-out and any crawl fan-out are transparently bounded by it — the slice-1 TODO
is retired. This is independent of the S3 data-scan concurrency (which is not
catalog traffic).
Behavioral gate (wiremock, all deterministic and fast):
- retries_on_429_then_succeeds: 2×429 (Retry-After: 0) then 200 → Ok, 3 requests.
- gives_up_after_max_retries_and_surfaces_the_error: persistent 429 → a Catalog
error naming the status after 1 + DEFAULT_CATALOG_MAX_RETRIES requests.
- retry_after_parses_delta_seconds… : header parse + backoff bounded by the cap.
- semaphore_bounds_concurrent_catalog_requests: 6 delayed requests through a
2-permit client serialize into ≥3 waves; a 6-permit client is faster.
The tests use an injected per-client semaphore (the global OnceLock would make a
bounded-pool test order-dependent) and a plain reqwest client (the production one
hardens against loopback for SSRF, which a loopback mock can't satisfy).
All switches default on; off ⇒ today's single-shot, unbounded behavior. See
11-pr8-cold-floor.md sect 9.
…sh-collision guard) The disk catalog cache names entries by a 64-bit DefaultHasher of the key (metadata_location / lt_key), so two distinct keys can collide onto one filename and silently serve another table's payload — a strictly worse failure than the content-addressed entries' wrong-snapshot, and #1503's pointer entry keys by table. Store the FULL key in the Envelope and verify equality on read: a mismatch is a clean miss, so hash quality is irrelevant to correctness. Bump CACHE_FORMAT_VERSION 1->2 (old keyless entries miss cleanly via the existing version check). Forced-collision hermetic test. Covers the metadata/scanfiles/ countstats entries; the #1503 pointer rides the same Envelope + version.
aaj3f
force-pushed
the
perf/r2rml-pr8-cold-floor
branch
from
July 15, 2026 16:32
c4a9b79 to
2bcc258
Compare
aaj3f
added a commit
that referenced
this pull request
Jul 15, 2026
…ilder symptom doc, F19 memo-key rationale, fmt - lazy_storage.rs: |res| res.is_ok() -> Result::is_ok (workspace-denied redundant_closure_for_method_calls in test code). - r2rml.rs: name send_parquet.rs's tokio::spawn as the site a future '&a narrowing refactor would fail at, so the '\''static owned-Arc capture reads as load-bearing. - operator.rs: mirror the store-disambiguation rationale at R2rmlParentMemoKey so the F19 with_graph_ref memo-share (why graph_source_id can't be dropped) is documented where an editor would remove the component. - cargo fmt (own-delta hunks). C2 pointer hardening (full lt_key stored in the Envelope + verified on read) is delivered by get/put_metadata_location adopting slice-2's (key, suffix) read/write signatures during the restack onto the #1491 CACHE_FORMAT_VERSION=2 chain.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap PR-8 (
docs/audit/2026-07-virtual-dataset-perf/ROADMAP.md; design + measurement in11-pr8-cold-floor.md): the cold/first-touch floor. The floor was decomposed with dedicated sub-spans before any lever was built: per table cold = loadTable GET ~1.5–2.0s (irreducible — it carries the vended credentials, and per the security ruling no tokens/credentials are ever persisted to disk) + metadata/manifest reads 0.3–1.1s (persistable) + OAuth ~0.35s (once per process). Multi-table queries paid the GETs serially.Three slices (three commits)
Slice 1 — parallelize catalog resolution + session storage-cache (
7fef307b8, switchFLUREE_R2RML_PARALLEL_CATALOG): best-effortprefetch_tableswarms the per-query session pin concurrently (buffered(8), deduped against already-pinned tables, engagement-provable via ther2rml.prefetchspan); and the session now caches the constructedArc<S3IcebergStorage>keyed like the loadTable pin (creds-refresh invalidated, unit-tested) — fixing a pre-existing defect where every scan (and every correlated-join re-scan) rebuilt the AWS SDK client. First implementation failed its own A/B (the double SDK build cancelled the gain) — the fold is what made it net-positive by construction. Warm-disk A/B: q008 ON 3.20s vs OFF 6.05s median (−47%), non-overlapping ranges, hashes == oracle.Slice 2 — DiskCatalogCache (
9dd2655db, switchFLUREE_ICEBERG_CATALOG_DISK_CACHE): persists three content-addressed layers keyed by the immutablemetadata_location—TableMetadata, unfilteredscan_files, and the COUNT-path manifest stats. No TTL, no pointer, zero secrets: a table change ⇒ new location ⇒ clean miss. Versioned envelope with deserialize-failure-as-miss, temp-file+atomic-rename writes, 512 MiB oldest-first prune at startup; dedicated sibling dir so the cold protocol clears data and catalog caches independently. Gate (cold data, warm catalog): q036 4.2s → 1.6s withread_metadataandcount_manifest_readcollapsed to n=0; q001read_metadata+scan_plan→ n=0; parity identical. (Manifest slice varies 450–990 ms with Snowflake load — recorded as a range.)Slice 3 — 429 hardening (
c4a9b799e,FLUREE_ICEBERG_CATALOG_MAX_RETRIES/FLUREE_ICEBERG_CATALOG_CONCURRENCY):request_with_retrynow retries 429/503 honoringRetry-After(else exponential backoff + full jitter, base 250ms cap 8s, default 4 retries) and every catalog round-trip passes a process-wide semaphore (default 8) — transparently bounding the slice-1 prefetch fan-out. 401-refresh preserved (permit dropped before recursing — no single-permit self-deadlock). Behavioral tests via wiremock: 429-then-200 retry counting, give-up-after-N, Retry-After honored, semaphore wave-serialization. Live smoke: warm-disk q008 median 3.08s — zero added latency.Cold floor, before → after (full three-slice binary)
The remaining cold residual is the credentials-bearing loadTable GET (+ the data fetch on full scans — PR-7 pruning territory, scoped out). BSBM deferred to an idle-machine run before merge-to-main.
Battery (all eight phases green)
iceberg (aws) / api (iceberg) / query suites exit 0 · W3C suite green · native corpus 54/54, 0 hash mismatches (2 perf violations, environmental per protocol) · cold subset 0-mismatch · kill-switch fidelity: parity clean with each new switch off/tuned. Measurement instrumentation (catalog sub-spans) ships in slice 1 as permanent counters.
Stacks on #1490 (PR-6) ← #1487 ← #1485 ← #1484 ← #1482 ← #1478 ← #1476 ← #1475 ← #1450.