Skip to content

Spark: Support Initial Snapshot Load for Streaming Reads - #18068

Open
alexprosak wants to merge 1 commit into
apache:mainfrom
alexprosak:spark-streaming-initial-snapshot-load
Open

Spark: Support Initial Snapshot Load for Streaming Reads#18068
alexprosak wants to merge 1 commit into
apache:mainfrom
alexprosak:spark-streaming-initial-snapshot-load

Conversation

@alexprosak

Copy link
Copy Markdown
Contributor

Closes #13188

Implements an initial-snapshot bootstrap for streaming reads of Iceberg tables in Spark. A streaming query started with a fresh checkpoint now reads the current snapshot in full before continuing incrementally - matching how Delta Lake's streaming source behaves by default.

Currently a fresh Iceberg streaming query without options replays the table's history from the oldest ancestor snapshot, which is slow for long-lived tables and any expired/deleted snapshots cannot be streamed. The only existing workarounds (bootstrap via batch read + cut over by timestamp) are fragile, so users can't easily express the common pattern of loading the entire table and then keeping up with new append data.

About the Change

This PR adds a stream-from-snapshot option:

stream-from-snapshot Behavior
(not set) new default: read the current snapshot in full, then continue with new snapshots
latest read snapshots committed after latest on stream startup
earliest read from the oldest ancestor (the existing default)
<snapshot-id> read snapshots after the given snapshot id (exclusive)

stream-from-snapshot and stream-from-timestamp are mutually exclusive. Previous default behavior to read from oldest ancestor can be opted-in via stream-from-snapshot=earliest

Follow-ups:

  • Docs update with new default behavior & stream-from-snapshot option
  • Ports to other supported spark versions

@alexprosak
alexprosak force-pushed the spark-streaming-initial-snapshot-load branch from 6110af5 to d9a22eb Compare September 11, 2026 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spark Streaming connector read initial snapshot of iceberg table

1 participant