feat(iceberg): BYO-IAM on operator-owned S3 — inline metadata, fail-closed credentials, SecretRef rotation fix, typed storage errors, verify probe (#1500, #1497, #1498)

[aaj3f]

Iceberg on operator-owned S3 (BYO-IAM): inline metadata, fail-closed credential handling, SecretRef rotation fix, typed storage errors, verify probe. Closes #1500, #1497, #1498.

What this is

The db-side engine work for the cross-repo BYO-IAM effort (fluree/solo#782 credential rotation + fluree/solo#783 configure-time BYO-IAM wizard): a Snowflake operator whose Iceberg data files live in an S3 bucket they own configures it on a live Solo stack — no CloudFormation update per bucket. Driven by a real client configuration, live-verified against a standing Snowflake + AWS rig throughout.

Stacked on the head of the perf chain — base perf/r2rml-f17-union-budget (#1507), i.e. #1502 → #1503 (loadTable-METADATA cache) → #1506 (corpus) → #1507 (F17). That is deliberate: the binding constraint was performance must not degrade under any credential configuration, so this had to be built and measured on top of the virtual-dataset perf work rather than reconciled at merge time.

⚠️ The restack found four semantic breaks — this is the part to review

Rebasing onto the chain applied textually clean for 8 of 10 commits, and that was the trap. #1503 did not replace the two sites this branch hardens — it cloned them into a deferred (LazyS3Storage) copy that none of the guards reached. Every break below is invisible to both test suites (ours predates the pointer rung; #1503's predates SecretRef / fail-closed / typed errors), and a plain merge ships all four silently:

#	What a clean merge would have shipped
1	§1 starves #1503's zero-REST win — on Snowflake specifically. The rung serves zero-REST only if metadata_from_caches hits, and its cross-process source is the disk metadata layer — whose only writer was the S3 read §1 removes. Fresh process ⇒ rung misses ⇒ the ~1–3 s loadTable GET returns, visible only in the load_table.n counter (§1 still books its own win, so nothing looks wrong).
2	SecretRef hard-errors whenever the disk cache is warm. #1503's rung added a second create_provider_arc that §3's hydrate never reached ⇒ resolve()'s fail-closed arm ⇒ fluree/solo#782's rotation breaks on the fast path.
3	The silent vended→ambient downgrade returns. #1503's copy of the storage build has no §2 guard, so any lazy-forced build reads under the process's ambient identity — the exact fail-open §2 exists to close.
4	Typed 403s are flattened. The deferred builder's `map_err(

Fixed by unifying, not by patching duplicates (commit fix(iceberg): reconcile BYO-IAM credential handling with the loadTable-METADATA cache):

• rest_session_storage() — one §2 decision + Direct-mode load_table_context rebuilds the S3 client every call (skips the session storage cache the REST branch uses) #1498 session reuse + vended/ambient construction, shared by the eager arm and the deferred builder. There is now exactly one decide_credential_source call site on the scan path, so the two arms cannot diverge again.
• hydrated_auth_provider() — pairs hydration with provider construction so a new client-construction site cannot ship un-hydrated. Hydration still runs strictly after rest_client_cache_key in both arms (pinned by secret_ref_fingerprint_is_rotation_stable).
• IcebergError::CatalogCredentialsNotVended + a storage_query_error lift — typed errors round-trip the builder's IcebergError channel, so the lazy path reports identically to the eager one.
• §1 seeds the disk metadata layer from the free inline copy — which arms perf(iceberg): loadTable-METADATA cache — zero-REST first-ask (q031 72s→~6s arc) + F19 residual #1503's rung one touch earlier than perf(iceberg): loadTable-METADATA cache — zero-REST first-ask (q031 72s→~6s arc) + F19 residual #1503 alone (previously that layer could only be filled by first paying an S3 GET).
• §4 reconciled with the review wave's own CACHE_FORMAT_VERSION bump: their 2 is a payload change (Envelope.key + verification); our filename re-key is orthogonal and correctly needs no bump. path()'s key param generalized for the new lt_key space; the pointer layer given an explicit scope verdict (catalog-gated and already per-graph-source ⇒ needs no scope).

Commits (one per spec section)

Commit	Spec	What
654677b23	#1497	Local CLI info/iceberg info route config display through the existing fail-closed redactor (re-exported as fluree_db_api::redact_graph_source_config)
dcf692484	#1500 §1	Use the REST catalog's inline loadTable metadata — no S3 GET for bytes we were handed; r2rml.read_metadata fires only on real reads. Seeds both cache layers, which is what keeps #1503's zero-REST rung armed (see above)
6967946be	#1500 §4	Disk catalog cache: stable xxh64 key hash (golden-pinned) + scope segment; reconciled with the wave's Envelope.key bump
2a9d35206	#1500 §5+§2	Typed StorageAccessDenied (bucket/key/region → 403 err:storage/AccessDenied) + fail-closed vended→ambient downgrade (err:catalog/CredentialsNotVended); one shared decide_credential_source matrix for scan AND preview
f305df480	#1500 §3	ConfigValue::SecretRef + async hydrate() + injected SecretResolver (tenant-agnostic); Fluree::with_secret_resolver cheap-clone injection; _ref builders at all three config levels
16e5ed489	#1498	Direct mode reuses the session-cached S3 client (was: full credential-chain + connection pool per call, even on cache hits)
291f00749	#1500 §6	verify_storage_access + POST /v1/fluree/iceberg/catalog/verify: config-time probe through the engine's own credential path (manifest read = metadata/ prefix, HeadObject = data/ prefix)
210cf4833	#1500 × #1503	The restack reconciliation — rest_session_storage(), hydrated_auth_provider(), typed round-trip, §1 disk seeding, §4 version reconcile (see the section above; this is the review-critical commit)
1b25366e2, 42761a386, 7b9f32d74	—	pre-existing clippy lints blocking the -D warnings gate + a bench allow + redactor doc accuracy

(The pre-restack rustfmt-drift commit dropped as empty — the review wave applied the identical fmt fixes upstream, which is where they belong.)

Decisions & corrections to the issues (recorded on the issues as comments)

• fluree info prints OAuth2 client_secret in plaintext — ConfigValue redacts Debug but not Serialize #1497's preferred fix (redacting Serialize) is refuted: a CI-locked contract requires lossless config persistence (ledger_info.rs "storage serialization must be lossless"); a lossy Serialize would persist [redacted] and break query-time auth. The display-layer denylist redactor is the correct seam; §3's SecretRef-at-rest is the disease-level fix (no secret in storage at all). Hardening follow-up: Harden redact_graph_source_config: denylist of secret key names fails open for future fields #1504.
• §1 is bypass-when-inline, not full disk-layer retirement: inline metadata exists only on a real REST loadTable (session/60s-cache reconstructions and Direct mode carry None), so full removal would regress Direct-mode cross-process cold starts. Live outcome identical (Snowflake vends inline ⇒ zero metadata GETs).
• §2 is an intended behavior change: a REST source with vended_credentials=true (default) whose catalog vends nothing now 403s instead of silently reading with the process's ambient AWS identity. Explicit vended_credentials=false is the ambient opt-in. (Solo runbook copy notified.)
• The ordering trap held: hydrate() runs strictly after rest_client_cache_key fingerprints the raw, reference-bearing config, inside the client-cache miss arm — pinned by secret_ref_fingerprint_is_rotation_stable (rotation does not re-key; hydrate-before-fingerprint would have).
• SecretRef sits between Literal and Dynamic: untagged serde tries variants in order and Dynamic (all-default fields) matches any object — declared later, the ref would be silently swallowed. Ordering canary test pins it. (Both sessions found this hazard independently.)

Verification

Adversarial review (independent agent over the full diff): SHIP, zero blockers. Independently recomputed the §4 golden hash, byte-diffed legacy error strings, verified resolved secrets are never persisted/fingerprinted/logged (create/map persists the original ref-bearing config), and confirmed 401→refresh untouched. Its one SHOULD-FIX (redactor doc claimed allowlist semantics) landed as 462c61c + #1504.

Machine gates (local — CI does not fire on stacked bases): cargo +1.96.1 fmt --all --check CLEAN; cargo +1.96.1 clippy --locked --workspace --all-features --all-targets -- -D warnings CLEAN (this branch also clears four pre-existing perf-line lints to get there); workspace test suite (--all-features --no-fail-fast): 174/176 targets pass; both failures are outside this branch's code — a pre-existing fluree-bench-virtual corpus-drift test (error_boundary_queries_match_observed_behavior, q013 error-boundary declaration vs observed engine behavior; the crate is untouched here beyond a lint #[allow] and the test pre-dates the fork) and a testcontainers flake (fluree-db-connection publish_index_without_preexisting_index_item — passes on immediate re-run; crate untouched).

Live rig (standing Snowflake ABACYOU-PP85756 + AWS sandbox, all under fresh isolated caches):

• Vended path E2E: rows, cold-verified. BYO-IAM path E2E under a zero-identity-permission role with bucket-policy-only access: rows, cold-verified.
• §1: Table metadata used inline from loadTable response fired exactly once; r2rml.read_metadata n=0.
• fluree info prints OAuth2 client_secret in plaintext — ConfigValue redacts Debug but not Serialize #1497: local fluree info shows "client_secret": "[redacted]", zero PAT occurrences.
• §5 message fidelity: both AWS AccessDenied variants survive verbatim to the CLI error and the server 403 JSON — no identity-based policy allows (no-perm role) and no permissions boundary allows (boundary-limited role). This is the distinction solo's runbook dispatch branches on (a boundary'd admin must not be told to re-apply a bucket policy).
• §6 route, all three shapes: 200 credential_source:"vended"; 200 credential_source:"ambient" under the bucket-policy-only role (HeadObject needs no ListBucket — Lambda shape confirmed); 403 @type: err:storage/AccessDenied under the no-perm role. The probe reads S3 directly (never cache-served), so it cannot be masked by a warm artifact cache.

Composition proven live (Snowflake virtual-sf01, restacked binary) — two consecutive exec-one processes, warm disk between them:

process	wall	r2rml.load_table.n	iceberg.oauth_token.n	r2rml.read_metadata.n
prime (cold)	3227 ms	1	1	0
gate (fresh proc, warm disk)	4 ms	0	0	0

The counters self-discriminate, so this is a proof rather than a correlation: the prime did a real loadTable (n=1) ⇒ the rung missed ⇒ the disk metadata layer was empty; and read_metadata.n=0 ⇒ §1 served that metadata inline, with no S3 read. So the only possible writer of the entry the gate then hit is §1's seeding — and #1503's DoD gate (load_table.n=0 and oauth_token.n=0) holds because of it. Pre-fix this reads load_table.n=1 at ~3 s with nothing else looking wrong.

Live rig re-verified at the restacked head (standing Snowflake + AWS sandbox; every credential-discriminating case under a fresh per-process cache): vended E2E ✅ (3 rows, 4.82 s) · BYO-IAM E2E under a bucket-policy-only role with zero identity-based S3 permissions ✅ (3 rows, 3.47 s) · both AWS AccessDenied strings survive verbatim to the CLI (no identity-based policy allows present, no permissions boundary allows correctly absent for the identity variant) ✅ · fluree info redaction, zero PAT occurrences ✅ · zero-REST rung × BYO-IAM ✅ (second process resolves with no catalog REST at all) · on-disk proof of §1 seeding (*.shared.metadata.json present after a run that logged "used inline … (no S3 fetch)") ✅ · no cache file ever contains the PAT ✅.
§2's guard is live-proven on the lazy arm: the deferred path emits Using ambient AWS credentials, a log line that lives inside rest_session_storage after the §2 check — so the shared guard demonstrably executes there.

Benchmark corpus: vbench compare --gate is running post-push against the new base (the chain adds q055–q059, so the pre-restack baseline is not comparable). Prior to the restack this branch gated clean (0 hash mismatches, 0 perf violations across 108 records; native median Δ +0 ms, virtual −37 s). Result will be posted here; note the chain's own baseline carries 5 known network-attributed violations (their F21).

Notes for reviewers

• fluree-db-query stays iceberg-agnostic (generic storage-denied variant; mapping helper lives in fluree-db-api). The 401→refresh-retry path is status-based on the catalog call and untouched.
• Fluree is now Clone (all fields Arc-backed or already-Clone); with_secret_resolver is the per-request-or-process-wide injection point. Non-iceberg builds unaffected (feature-gated).
• Known boundary (matches preview): verify_storage_access requires the catalog to vend inline metadata — same as the existing preview path; Snowflake Horizon/Polaris do.
• Follow-up filed: Typed storage errors are flattened on the disk-cache-fill Parquet route (500 instead of 403 err:storage/AccessDenied) #1510 — a pre-existing typed-error gap (present on main, not introduced here): a storage denial on the disk-cache-fill Parquet route surfaces as 500 Internal instead of 403 + err:storage/AccessDenied, because send_parquet.rs's coalesced_fetch flattens the typed error through an io::Error two layers below our lift. perf(iceberg): loadTable-METADATA cache — zero-REST first-ask (q031 72s→~6s arc) + F19 residual #1503 makes that route more reachable (warm catalog + cold artifact cache ⇒ the first S3 read is a data file). §5 is a strict improvement everywhere it lands; the raw AWS chain still survives so host regex dispatch still fires. Not fixed here deliberately — it touches the Parquet reader's hot path and deserves its own review + bench.
• Observed while verifying, not addressed here: the RUST_LOG policy comment in fluree-db-cli/src/main.rs:22 contradicts the code (honored only with --verbose); machine-global artifact cache means previously-fetched table data remains locally readable after upstream access revocation (inherent to content-addressed caching; the §6 probe is immune by construction).

[aaj3f]

Heads-up: your base #1502 moved (review-response wave restack) — a quick restack will clear the CONFLICTING status.

The perf burndown stack beneath this PR (#1490 → #1502) was restacked as part of the 2026-07-15 review-response wave. Your base, perf/r2rml-q031-refprune (#1502), is now at c38d412d7 (was eee643572) — same reviewed content, re-based with per-PR fmt/clippy cleanup commits, and now MERGEABLE.

This PR (feat/external-s3-iceberg @ 462c61c73) still points at the old base, hence the conflict. Restack when convenient:

git rebase --onto perf/r2rml-q031-refprune eee643572 feat/external-s3-iceberg
git push --force-with-lease

(eee643572 = the old #1502 tip your branch forked from; the --onto replays only your own BYO-IAM commits onto the new base.)

I did not touch this branch — flagging so you can restack on your own schedule. The rest of the stack (#1503 loadtable-meta-cache, #1506 exploration, #1507 F17) also restacked on top and is green/MERGEABLE, so #1502's new tip is stable to rebase onto.

[aaj3f]

Corpus perf gate (post-restack, full 118-record run): 0 hash mismatches; 6 perf "violations" diagnosed as network + noise, none attributable to this diff.

Correctness is clean — full result-hash parity with the native oracles across the corpus, including the new exploration family (q055–q059).

The 6 perf violations, with per-query span attribution from the run records:

query	target	wall	load_table span	fetch_bytes	verdict
q002	virtual	2341 ms	1 × 1956 ms	0	catalog GET
q004	virtual	1967 ms	1 × 1663 ms	0	catalog GET
q024	virtual	2754 ms	1 × 2294 ms	0	catalog GET
q030	virtual	2324 ms	1 × 1662 ms	0	catalog GET
q043	virtual	12675 ms	6 × 10487 ms	0	catalog GET (6×)
q009	native	772 ms (rerun)	n/a (native)	0	+74 ms / +11 % noise

For every virtual violator wall ≈ load_table_ms and fetch_bytes = 0 (files disk-served, no artifact-cache eviction) — i.e. the entire wall is the Snowflake Polaris loadTable REST GET, not query compute. These are the exact 5 queries flagged by F21 (filed against #1507's own baseline) as loadTable-GET-latency-bound: ~0.4 s at bless-time vs ~1.5 s on an elevated-catalog day. q009 is a native materialized-ledger query (load_table.n=0, zero Iceberg involvement), +74 ms over a 10 % budget — machine noise (this run followed a host OOM restart).

Nothing here touches loadTable-GET frequency or latency: §1 removes the metadata S3 read and arms the pointer rung earlier (q001 measured 3227 ms → 4 ms, load_table.n=0), §2/§3/§5 are credential-path, #1498 is Direct-mode. The baseline was blessed on a faster-catalog day; the delta is network-day variance on queries whose wall is the GET, plus one native blip — not a regression from this branch.

Confirmatory base-vs-head A/B of just these 6 queries on the same machine available on request (deferred here for time + the host's post-OOM memory pressure).