Skip to content

FujiDecompressor: add lossy-compressed RAF decoding support - #985

Open
igoreskub wants to merge 1 commit into
darktable-org:developfrom
igoreskub:fuji-lossy-compressed-raf
Open

FujiDecompressor: add lossy-compressed RAF decoding support#985
igoreskub wants to merge 1 commit into
darktable-org:developfrom
igoreskub:fuji-lossy-compressed-raf

Conversation

@igoreskub

Copy link
Copy Markdown

FujiDecompressor previously only handled uncompressed and lossless-compressed Fuji RAF files; lossy-compressed ("compressed" quality setting) files were rejected at header validation.

Ports the lossy decode path from dnglab/rawler's Rust implementation (rawler/src/decoders/fuji_decompressor.rs), which already unifies lossless and lossy decoding in one algorithm, reconciled against rawspeed's existing lossless-only architecture:

  • Parse the per-line quantization base (q_base) array that follows the block-offset table in lossy files.
  • Add FujiQTable, generalizing the single quantization table into four: a "main" table rebuilt whenever q_base changes between lines, plus three static tables (fixed q_base 0/1/2) selected per-pixel by local gradient magnitude.
  • Wire table selection into the sample-decode hot path (fuji_decode_interpolation_even/odd_inner, fuji_decode_sample, fuji_quant_gradient), gated throughout by header.isLossless() so lossless-file behavior is unchanged.

Cross-validated pixel-exact against dnglab/rawler's independent decoder across 449 real-world files (one X-series body, both lossless- and lossy-compressed modes across a firmware change) at 25 spot-check coordinates per file.

Fixes a related regression found during that testing: passing q_bases.data() directly into Array1DRef crashed on every lossless file, since an empty vector's .data() can be null (observed on libc++) and Array1DRef's invariant check requires non-null even at size 0. Fixed with a static sentinel byte.

Includes an opt-in pixel spot-check diagnostic (RAWSPEED_FUJI_DIAG env var, off by default) used to produce the validation results above; zero overhead when unset. Also includes
src/utilities/dnglab-crossvalidate/, the standalone script (with README) used to run that cross-validation, so others can reproduce or extend it.

Closes #232
Closes #255
Closes #332
Closes #366
Closes #406

FujiDecompressor previously only handled uncompressed and
lossless-compressed Fuji RAF files; lossy-compressed ("compressed"
quality setting) files were rejected at header validation.

Ports the lossy decode path from dnglab/rawler's Rust implementation
(rawler/src/decoders/fuji_decompressor.rs), which already unifies
lossless and lossy decoding in one algorithm, reconciled against
rawspeed's existing lossless-only architecture:

- Parse the per-line quantization base (q_base) array that follows
  the block-offset table in lossy files.
- Add FujiQTable, generalizing the single quantization table into
  four: a "main" table rebuilt whenever q_base changes between
  lines, plus three static tables (fixed q_base 0/1/2) selected
  per-pixel by local gradient magnitude.
- Wire table selection into the sample-decode hot path
  (fuji_decode_interpolation_even/odd_inner, fuji_decode_sample,
  fuji_quant_gradient), gated throughout by header.isLossless() so
  lossless-file behavior is unchanged.

Cross-validated pixel-exact against dnglab/rawler's independent
decoder across 449 real-world files (one X-series body, both
lossless- and lossy-compressed modes across a firmware change) at
25 spot-check coordinates per file.

Fixes a related regression found during that testing: passing
q_bases.data() directly into Array1DRef crashed on every lossless
file, since an empty vector's .data() can be null (observed on
libc++) and Array1DRef's invariant check requires non-null even at
size 0. Fixed with a static sentinel byte.

Includes an opt-in pixel spot-check diagnostic (RAWSPEED_FUJI_DIAG
env var, off by default) used to produce the validation results
above; zero overhead when unset. Also includes
src/utilities/dnglab-crossvalidate/, the standalone script (with
README) used to run that cross-validation, so others can reproduce
or extend it.

Closes darktable-org#232
Closes darktable-org#255
Closes darktable-org#332
Closes darktable-org#366
Closes darktable-org#406
@igoreskub
igoreskub requested a review from LebedevRI as a code owner August 28, 2026 12:27
@igoreskub igoreskub closed this Aug 28, 2026
@igoreskub igoreskub reopened this Aug 28, 2026
@kmilos

kmilos commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

x-ref #972 (comment)

@igoreskub

Copy link
Copy Markdown
Author

Thanks for the context, @kmilos — no pressure on timeline.

This represents real, tested work (449 files cross-validated against an independent decoder, two compression modes, one regression caught along the way), so happy to have it sit until someone has bandwidth. I'm also glad to stick around and maintain it going forward — respond to feedback, fix issues, extend to other bodies if needed.

Is there anyone else with merge rights who might take a look, or best to just leave this open for now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants