Iceberg on operator-owned S3: use inline metadata, fail closed on vended downgrade, hydrate secret refs, hedge the disk-cache key

[aaj3f]

Summary

Four engine-side changes needed to support reading Snowflake-managed Iceberg tables whose data files live in an operator-owned S3 bucket, via either credential path — while holding the line that performance must not degrade under any credential configuration.

Two of the four are unconditional wins that stand on their own merits, independent of any customer:

1. Use the inline metadata the catalog already hands us — saves ~1 S3 round-trip per table, in every configuration, on the first cold touch.
2. Fail closed on the silent vended→ambient downgrade — a security fix that also establishes the invariant the disk cache's safety argument depends on.
3. ConfigValue::SecretRef + hydrate() — the engine half of the credential-rotation fix.
4. A scope discriminant on the disk cache key — a ~10-line hedge that makes per-graph-source identity (mechanism B) a one-liner later instead of a re-architecture.

All anchors verified at a8de70e0f64454e4e3c736f203a9de4499dece45 (the fully-optimized head, PR-5 tip).

---

1. Use load_response.metadata instead of re-fetching it from S3

We are issuing an S3 GET per table for bytes the catalog already gave us.

LoadTableResult.metadata is a spec field returned inline on a successful loadTable, and the engine already parses and retains it — fluree-db-iceberg/src/catalog/rest.rs:377-383, catalog/mod.rs:31-36:

"so callers can read the full table schema/snapshot without a second S3 fetch"

But the query path never reads it. r2rml.rs:1343-1390 goes in-memory cache → disk cache → storage.read(metadata_location) — a fresh S3 GET. The intended consumers are named at rest.rs:377-379 ("metadata preview"); the read path was simply never wired up.

Live-verified against Snowflake Horizon (account ABACYOU-PP85756, customer-owned external volume, 2026-07-14): loadTable returns 3,438 bytes of inline metadata with full schemas, current-snapshot-id, and 2 snapshots. So rest.rs:378's claim ("Snowflake Horizon / Polaris include it") is confirmed live, not just asserted in a comment.

Why this is the purest form of the constraint: it saves ~1 S3 RTT/table (~85–98 ms on the measured link, docs/audit/**/06-per-file-cost.md:66) in every configuration, with no cache, no disk I/O, and no security surface — and it fires on the first cold touch, whereas the disk cache by definition only helps the second.

• Use load_response.metadata when Some; keep the S3 fetch as the None fallback (already handled).
• Retire the TableMetadata disk-cache layer — it caches something we were handed.
• Assert in a live test: load_response.metadata.is_some() and r2rml.read_metadata span count n=0.

Open: exact r2rml.read_metadata cost is UNVERIFIED (the span is new — 11-pr8-cold-floor.md:102). One cold run sizes the win. The fallback makes the change safe regardless.

---

2. Fail closed on the silent vended→ambient downgrade

r2rml.rs:1226 branches on the response, not the config:

if let Some(ref credentials) = load_response.credentials { /* vended */ }
else { /* from_default_chain — ambient */ }

With vended_credentials: true — the default (fluree-db-iceberg/src/config.rs:344-346) — a catalog that authorizes loadTable but returns no credentials causes a silent downgrade to the process's ambient identity, logged only at info! ("Using ambient AWS credentials", r2rml.rs:1249).

In a deployment where the ambient identity is a shared Lambda execution role, that silently grants a catalog-scoped graph source the shared role's entire S3 reach. That is a fail-open.

This was flagged in an earlier audit as a warn!-vs-info! nit. It is more than that: fixing it is what makes "catalog gate == storage gate" true by construction under vended credentials — which is precisely the invariant the disk cache's "secret-free" safety argument rests on (see §4).

• When io.vended_credentials == true and the response carries no credentials, error — do not fall through to ambient.
• Give it a distinct, typed error (see §5) so callers can tell "catalog won't vend" from "storage denied".
• Opting into ambient must be explicit (vended_credentials: false), which is already the BYO-IAM contract.

---

3. ConfigValue::SecretRef + hydrate() — the rotation fix, engine half

Context: solo currently resolves a catalog's secret reference at bind time and bakes the literal into every virtual dataset's graph-source config. Rotating the credential breaks every previously-bound dataset. (Filed separately on the solo side.)

db already models the right design. AuthConfig::OAuth2ClientCredentials { client_secret: ConfigValue } (fluree-db-iceberg/src/auth/mod.rs:77) takes a ConfigValue. And fluree-db-api/src/graph_source/cache.rs:52-60 describes the intended behaviour verbatim:

"keyed by a fingerprint of the raw config JSON. When a Bearer/OAuth secret is sourced from an env var or secret store, that JSON stores the reference, not the secret… A bounded TTL lets a rotated secret self-heal."

The seam exists: r2rml.rs:1100 fingerprints the raw config; create_provider_arc() (which calls client_secret.resolve()) sits inside the cache-miss branch. This is finishing a design that is ~90% built.

• Add ConfigValue::SecretRef { secret_ref: String } alongside Literal / Dynamic (config_value.rs:47-62).
• Add an async hydrate(&resolver) -> AuthConfig pass that rewrites SecretRef → Literal, called at r2rml.rs:1104 immediately before create_provider_arc(). This keeps resolve() sync (config_value.rs:120) and avoids rippling async through create_provider / create_provider_arc — non-trivial in this crate, which has a known async-recursion trap.
• Define the resolver as an injected trait. db must stay tenant-agnostic: the host constructs the resolver per-request with the authenticated tenant captured, and performs its own authorization inside. db never learns what a tenant is.
• Fail closed: a SecretRef with no injected resolver (OSS, CLI, tests) must hard-error with an actionable message.
• Ensure ConfigValue::SecretRef's Serialize is redacting — see fluree info prints OAuth2 client_secret in plaintext — ConfigValue redacts Debug but not Serialize #1497; the existing redacting Debug (config_value.rs:64-68) does not cover the serde path.

⚠️ The ordering trap — please read before implementing

hydrate() MUST run strictly AFTER rest_client_cache_key (r2rml.rs:1100) computes the fingerprint.

The fingerprint is deliberately taken over the raw, reference-bearing JSON. Hydrating first means every rotation silently re-keys the client cache — costing a resolver call and a full OAuth exchange per query, instead of once per 900s per process. It is a one-line ordering mistake that converts a bounded cost into an unbounded one, and it would not fail any test.

Cost

+1 resolver call per REST-client build = once per 900s per process. Zero per query. Zero per table. Against the measured baseline (cold loadTable ≈ 1.3–3 s/table — the dominant cold term; OAuth exchange ≈ 0.5 s once per process), this is ≤5% of the step it already sits behind. No credential configuration becomes second-class.

Follow-up (not required here)

401 → refresh is not self-healing today: create_provider_arc resolves ConfigValue → String once (auth/mod.rs:135-136), and refresh() reuses the stale String (oauth2.rs:124), so rest.rs:212-218 re-exchanges with the dead secret and 401s again. Rotation self-heals only at the 900s TTL. Having the provider hold ref+resolver and re-resolve inside fetch_token turns 900s of staleness into one round-trip, at the cost of one resolver call per 401 only.

---

4. Disk-cache scope discriminant (hedge for mechanism B)

No leak exists today — do not re-key now. DiskCatalogCache is keyed on a hash of metadata_location (disk_catalog_cache.rs:164-166), and that is currently safe by construction:

• IoConfig (fluree-db-iceberg/src/config.rs:329-342) has exactly four fields — vended_credentials, s3_region, s3_endpoint, s3_path_style. There is no role-ARN / assume-role / per-source identity field.
• S3IcebergStorage::from_default_chain (io/storage.rs:268-280) takes only region/endpoint/path_style and calls aws_config::defaults(...) — the process's ambient identity.
• The workspace's only AssumeRole (fluree-db-api/src/vended_credentials.rs:156-176) is Fluree vending credentials out for its own ledger storage proxy — not on the Iceberg consumption path.

So every graph source in a process reads S3 as the same identity. Two graph sources sharing a metadata_location have identical access. Stronger: the cache is only ever populated by a successful read using that shared identity, so anything in it is something any graph source in that process could have fetched itself.

It stops being safe when mechanism B (per-graph-source sts:AssumeRole) lands — which is on the roadmap. At that point graph sources get different S3 identities, and a shared key defeats the isolation that is B's entire purpose.

What is actually gated (only these two layers matter):

• TableMetadata — not a leak in any configuration. The catalog hands it to you inline on a successful loadTable (§1). Catalog authorization discloses the schema by protocol.

• scan_files + count_stats — column lower_bounds/upper_bounds and per-file record_count (disk_catalog_cache.rs:294-297) come from manifests, i.e. S3 objects behind the storage gate. These are the only layers that move data across a gate.

• Add a scope discriminant to path() (disk_catalog_cache.rs:164-168), constant today. Bump CACHE_FORMAT_VERSION (:119). ~10 lines; makes B's arrival a one-liner.

⚠️ When B lands: do NOT key on the credential

The naive (metadata_location, identity) re-key is a performance catastrophe: vended credentials rotate on every loadTable, so keying on the credential yields a new key per process — 0% hit rate under the default configuration.

Key on a stable scope fingerprint (the catalog-auth config — what rest_clients is already keyed on), and only for scan_files/count_stats. Cost is ≈0, because the dominant hit is same graph source, different process ("persists across process restarts", disk_catalog_cache.rs:3) — not cross-source sharing. Re-keying only splits entries between identities that were never legitimately sharing.

Rejected: a per-identity HeadObject access proof. The arithmetic is survivable (1 RTT ≈ 90 ms against ~500–1200 ms saved), but it is dominated — re-keying costs zero RTT where the probe costs one per table per process, forever.

---

5. Typed storage-permission errors

Storage auth failures surface as QueryError::Internal(format!("Failed to create S3 storage: {e}")) — an opaque 500 with the reason only in the string. IcebergError::Credential is never constructed on this path.

This pushes the classification burden onto every caller. Solo currently has to regex \b403\b to guess, which collides with its auth category and misfires — an S3 AccessDenied is a 403, so an admin with a missing bucket grant gets told "authentication failed" and rotates a perfectly good PAT.

• Add a typed permission-denied variant carrying bucket/key/region.
• Distinguish "catalog would not vend" (§2) from "storage denied".
• Do not disturb the 401→refresh-retry path (rest.rs:212-218) — it matches on the catalog auth failure and is a perf mechanism (it avoids failing the query). Adding a storage-side variant must not change catalog-side matching.

---

6. verify_storage_access(&config, table)

Solo needs to verify at configuration time that the stack can actually read a data file, not merely reach the catalog. Doing that host-side means hand-rolling an S3 HeadObject with a different client than the engine uses — which defeats the point: it would validate a path queries don't take.

• Expose an API that resolves the table and proves storage reachability through the engine's own storage construction, so the probe is fidelity-guaranteed by construction rather than by keeping two code paths in sync.
• Return the typed error from §5.
• Reuse the existing caches where possible; this must not become a per-query cost.

---

Non-goals / notes

• s3:ListBucket is not required. There are zero list_objects/ListObjects calls in fluree-db-iceberg/ or fluree-db-api/src/graph_source/. The catalog returns metadata-location; every hop after (manifest list → manifests → Parquet) is an absolute path fetched by exact key. Consumers should grant s3:GetObject only — and can prefix-scope it. Consequence for runbooks: without ListBucket, a GET on a missing key returns 403 AccessDenied, not 404 NoSuchKey.
• Minor, pre-existing: path() uses DefaultHasher (disk_catalog_cache.rs:165). Rust does not guarantee its algorithm across toolchain releases, so a toolchain bump silently re-keys the whole cache directory — a one-time full miss that CACHE_FORMAT_VERSION will not catch. Worth a stable hasher while touching this file.

Definition of done

• Every change measured against the virtual-dataset benchmark corpus; no regression in any band, in either credential configuration.
• §1 verified live: inline metadata used, r2rml.read_metadata n=0.
• §2 verified: vended-configured + no-creds response errors instead of downgrading.
• §3 verified: rotation self-heals within one client TTL; fingerprint unchanged across a rotation (i.e. hydration is provably after the key).
• The live Snowflake rig reproduces both paths (vended and BYO-IAM) against a customer-owned bucket.

[aaj3f]

Consumers:

• §3 (ConfigValue::SecretRef + hydrate()) is the engine half of fluree/solo#782 — a live credential-rotation defect.
• §5 (typed storage-permission errors) and §6 (verify_storage_access) unblock fluree/solo#783's error dispatch and its configuration-time storage probe.
• §3's redaction requirement overlaps fluree info prints OAuth2 client_secret in plaintext — ConfigValue redacts Debug but not Serialize #1497.

§1 and §2 stand alone. Using the inline metadata is a free ~1 S3 RTT/table in every configuration on the first cold touch (live-verified against Horizon: 3,438 bytes returned inline), and failing closed on the vended→ambient downgrade is a security fix in its own right. Neither needs a consumer to justify it.

[aaj3f]

Heads-up for §3 from the solo side (found while grounding fluree/solo#782 against the engine rev solo currently pins, f084a1ee16):

ConfigValue is #[serde(untagged)], and Dynamic's three fields are all Option + #[serde(default)]. Consequence: any new object wire form — e.g. {"secret_ref": "fsec:v1:sm:…"} — already deserializes successfully today as Dynamic { env_var: None, java_property: None, default_val: None } (untagged ignores unknown fields), then fails at resolve() with a misleading env-var-shaped error. A bare-string form has the opposite trap: Literal matches first and the ref string itself becomes the "secret".

So when adding SecretRef:

• if it's an object variant, order it before Dynamic in the enum (serde tries untagged variants in order);
• add a deserialization test asserting the SecretRef JSON does not fall through to Literal/Dynamic;
• note the compat asymmetry in the PR: an old engine reading a new SecretRef config still silently mis-parses into Dynamic — enum ordering only fixes new binaries. Likely acceptable (solo gates rollout by bumping the pin before writing refs), but worth stating.

Solo consumes §3 via a fluree-db-api pin bump (currently f084a1ee16) to a SHA on the db branch. Please publish the intended wire shape + resolver-trait signature under "Interface commitments" in db-progress.md as soon as they're stable — the solo half of #782 is sequenced around them.

[aaj3f]

DB session — grounding complete at eee643572 (feat/external-s3-iceberg, stacked on #1501 head + 3 F20 commits). Every anchor re-verified; all substantive claims hold. Decisions, for the record:

• §1 — implementing as bypass-when-inline, not full layer removal. LoadTableResponse.metadata is only populated by a real REST loadTable in the current call: session-cache/60s-TTL reconstructions (catalog_session.rs:54-71) and Direct mode (r2rml.rs:1300-1305) always carry metadata: None. Full removal of the disk metadata layer would therefore regress Direct-mode cross-process cold starts, which were never "handed" the bytes. Inline path skips disk+S3 entirely; the in-mem → disk → S3 fallback stays for the None cases. Live outcome unchanged: Snowflake vends inline ⇒ r2rml.read_metadata n=0.
• §5 shape: IcebergError::StorageAccessDenied { bucket, key, region, message } classified at the S3 call sites in io/storage.rs; QueryError gains StorageAccessDenied + CatalogCredentialsNotVended (generic — fluree-db-query stays iceberg-agnostic; mapping helper in fluree-db-api); server maps both to HTTP 403 with machine codes STORAGE_ACCESS_DENIED / CATALOG_CREDENTIALS_NOT_VENDED (today an S3 403 surfaces as 400 INVALID_QUERY). The 401→refresh path is status-based on the catalog call (rest.rs:212-218) and is structurally untouched.
• §2 fires at BOTH downgrade sites — the query path (r2rml.rs:1245 else-branch) and build_preview_storage callers (iceberg_catalog.rs:730), so preview/generate can't silently succeed on a path queries won't take.
• §3: SecretRef must be the object shape {"secret_ref": "..."} and the variant must sit between Literal and Dynamic — untagged serde tries variants in declaration order and Dynamic (all-default fields) matches any object, so declared after Dynamic the ref would be silently swallowed. Resolver = #[async_trait] SecretResolver defined in fluree-db-iceberg, injected per-request via a cheap Fluree::with_secret_resolver(...) clone; hydration at the four create_provider_arc sites, strictly after rest_client_cache_key (r2rml.rs:1100/:1104). Resolver failures construct IcebergError::Credential — currently dead code, never built anywhere; this gives it its first real semantics.
• §4: constant scope segment + stable FNV-1a hasher replacing DefaultHasher, CACHE_FORMAT_VERSION 1→2.
• §6 probe reads BOTH prefixes — manifest list (metadata/) + HeadObject on the first data file (data/). With §1, queries no longer GET the metadata JSON, so probing metadata_location alone would be neither necessary nor sufficient under prefix-scoped grants.

Interface DRAFTs (SecretRef wire shape, resolver trait, error codes, verify_storage_access signature) are published in the cross-session progress file; they'll be recorded here as they stabilize with commit shas.