Revised MDIO I/O - #332
Open
zafarhassan-tgs wants to merge 7 commits into
Open
Conversation
Turn the MDIO<->RSF converters into a fidelity-preserving pipe so that
MDIO -> sfmdioread -> <sf filters> -> sfmdiowrite -> MDIO reproduces the
parent dataset exactly for supported operations, instead of writing a
reduced-schema child.
sfmdioread stamps mdio_* pipe context (source, fingerprint, data
variable, per-axis geometry, contract) onto the RSF history and, for
unit-stride reads, streams amplitudes in whole-chunk blocks.
sfmdiowrite defaults to a parent-clone finalizer: require or recover the
pipe context, verify the parent fingerprint, detect the geometry change
(same-geometry, truncate/slice/integer-decimate, or interpolating
sample-axis resample), clone the parent store and overwrite only the data
variable (with dead-trace fill), recompute statsV1 in one streaming pass,
stamp sample-geometry headers when the sample axis changed, append
provenance, and publish atomically (local or gs://,s3:// URLs). It fails
closed when fidelity cannot be proven.
reduced=y remains the opt-in lossy path and, with no upstream context,
originates a fresh MDIO from the RSF stream alone -- the only way to seed
a parent store from Madagascar. Also fixes write_dim_coords to await its
TensorStore write future so dimension coordinates flush instead of being
left at the NaN fill.
Shared logic lives in the new mdio_pipe.{cc,hh}; mdio2segy.{cc,hh} and the
system/generic build wiring are updated to match.
Self-contained demo under book/rsf/rsf/mdio: build a small synthetic volume, seed a parent MDIO (reduced=y create), run a faithful same-geometry bandpass round-trip through the pipe, read the child back to RSF, and plot input vs filtered. Guarded with WhereIs so builds without the MDIO converters (no mdio-cpp) skip it instead of failing.
The pipe (and the book example) write .mdio Zarr directory stores as build products; ignore them like the existing .rsf/.vpl outputs.
Replace the inline SHA-256 parent-descriptor digest with a lighter FNV-1a fingerprint, consolidate float block read/write paths, and trim verbose comments across the MDIO mains and build wiring.
Expand sfmdioread/sfmdiowrite header comments to Madagascar style, wire panel= back into chunk-aligned block sizing, and move float block I/O helpers into mdio2segy with fuller build documentation.
Route store clone/metadata/publish through kvstore so local and remote behave the same, stage remote Dataset::Open inputs (and remote write temps) on local scratch to work around Zarr-V3 cloud bucket gaps, and keep stamping/cloning from the original s3:// URI.
Mdio lossless pipe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduced MDIO pipe and revised I/O to maintain metadata and improved efficiency