FujiDecompressor: add lossy-compressed RAF decoding support - #985
Open
igoreskub wants to merge 1 commit into
Open
FujiDecompressor: add lossy-compressed RAF decoding support#985igoreskub wants to merge 1 commit into
igoreskub wants to merge 1 commit into
Conversation
FujiDecompressor previously only handled uncompressed and
lossless-compressed Fuji RAF files; lossy-compressed ("compressed"
quality setting) files were rejected at header validation.
Ports the lossy decode path from dnglab/rawler's Rust implementation
(rawler/src/decoders/fuji_decompressor.rs), which already unifies
lossless and lossy decoding in one algorithm, reconciled against
rawspeed's existing lossless-only architecture:
- Parse the per-line quantization base (q_base) array that follows
the block-offset table in lossy files.
- Add FujiQTable, generalizing the single quantization table into
four: a "main" table rebuilt whenever q_base changes between
lines, plus three static tables (fixed q_base 0/1/2) selected
per-pixel by local gradient magnitude.
- Wire table selection into the sample-decode hot path
(fuji_decode_interpolation_even/odd_inner, fuji_decode_sample,
fuji_quant_gradient), gated throughout by header.isLossless() so
lossless-file behavior is unchanged.
Cross-validated pixel-exact against dnglab/rawler's independent
decoder across 449 real-world files (one X-series body, both
lossless- and lossy-compressed modes across a firmware change) at
25 spot-check coordinates per file.
Fixes a related regression found during that testing: passing
q_bases.data() directly into Array1DRef crashed on every lossless
file, since an empty vector's .data() can be null (observed on
libc++) and Array1DRef's invariant check requires non-null even at
size 0. Fixed with a static sentinel byte.
Includes an opt-in pixel spot-check diagnostic (RAWSPEED_FUJI_DIAG
env var, off by default) used to produce the validation results
above; zero overhead when unset. Also includes
src/utilities/dnglab-crossvalidate/, the standalone script (with
README) used to run that cross-validation, so others can reproduce
or extend it.
Closes darktable-org#232
Closes darktable-org#255
Closes darktable-org#332
Closes darktable-org#366
Closes darktable-org#406
Contributor
|
x-ref #972 (comment) |
Author
|
Thanks for the context, @kmilos — no pressure on timeline. This represents real, tested work (449 files cross-validated against an independent decoder, two compression modes, one regression caught along the way), so happy to have it sit until someone has bandwidth. I'm also glad to stick around and maintain it going forward — respond to feedback, fix issues, extend to other bodies if needed. Is there anyone else with merge rights who might take a look, or best to just leave this open for now? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FujiDecompressor previously only handled uncompressed and lossless-compressed Fuji RAF files; lossy-compressed ("compressed" quality setting) files were rejected at header validation.
Ports the lossy decode path from dnglab/rawler's Rust implementation (rawler/src/decoders/fuji_decompressor.rs), which already unifies lossless and lossy decoding in one algorithm, reconciled against rawspeed's existing lossless-only architecture:
Cross-validated pixel-exact against dnglab/rawler's independent decoder across 449 real-world files (one X-series body, both lossless- and lossy-compressed modes across a firmware change) at 25 spot-check coordinates per file.
Fixes a related regression found during that testing: passing q_bases.data() directly into Array1DRef crashed on every lossless file, since an empty vector's .data() can be null (observed on libc++) and Array1DRef's invariant check requires non-null even at size 0. Fixed with a static sentinel byte.
Includes an opt-in pixel spot-check diagnostic (RAWSPEED_FUJI_DIAG env var, off by default) used to produce the validation results above; zero overhead when unset. Also includes
src/utilities/dnglab-crossvalidate/, the standalone script (with README) used to run that cross-validation, so others can reproduce or extend it.
Closes #232
Closes #255
Closes #332
Closes #366
Closes #406