Skip to content

RowDiffChar annotation format for SSHash graphs - #679

Draft
adamant-pwn wants to merge 5 commits into
masterfrom
rowdiff-char-sshash
Draft

adamant-pwn wants to merge 5 commits into
masterfrom
rowdiff-char-sshash

Conversation

@adamant-pwn

@adamant-pwn adamant-pwn commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Ports the RowDiffChar annotation format, dropped from #604 when that PR was trimmed to the SSHash mmap load and traverse() fast path, onto current master. It only matters if SSHash graphs are used for annotated indexes.

What it does

  • DeBruijnGraph::trivial_outgoing(node): returns the unique successor when it can be resolved in O(1), otherwise npos (the default). DBGSSHash overrides it using succ_is_next_ / pred_is_prev_. Both imply out-degree 1, so the result always matches the row-diff successor stored in fork_succ.
  • RowDiffMode::CharSucc + RowDiffChar<BRWT> (.row_diff_char_brwt.annodbg): replaces the per-node fork_succ bitvector with a sparse rd_succ_needs_char_ bitvector plus a 2-bit successor character, stored only for nodes that trivial_outgoing() can't resolve. Serialized as v3.0. The existing v2.0 format is unchanged and still loads as before.
  • transform_anno --anno-type row_diff_char_brwt -i <graph> X.row_diff_brwt.annodbg converts via compute_rd_succ_char(). The BRWT diffs are copied unchanged.

Porting notes (vs. 0a79785 on refs/pull/604/head)

  • Successor dispatch moved into master's refactored get_rd_ids(). The RowDiffChar class now sits next to get_column_from_rows().
  • row_diff_char_brwt is excluded from the new "embedded anchor" reformat path (Batch and share row-diff column reconstruction #659-era), which drops -i; the char transform needs the graph.
  • The v3.0 loader gets the same madvise_random_range hint as v2.0 and checks stream state after reading rd_succ_char_.
  • Kept master's sdsl-lite pin, which already contains the commit's d7fffb72. The sshash pin is unchanged.

Tests

  • Unit: trivial_outgoing_is_unique_successor checks trivial_outgoing() against successors found by independent k-mer lookup. It runs in basic mode for all GraphTypes and in canonical mode for CanonicalGraphTypes, and requires SSHash to actually take the shortcut.
  • Integration: TestRowDiffCharSSHash (basic / canonical / primary SSHash, k=21, transcripts_100.fa) builds row_diff_brwt, converts it to row_diff_char_brwt, and asserts equal stats and byte-identical query output in labels, matches and signature modes, with -p 1 and -p 4.

Verification is left to GitHub CI. Locally (g++-15, DNA) Release and Debug builds succeed, and an earlier Release unit_tests_parallel.sh -j 8 run passed 3747/3748. The one failure was the over-strict per-k shortcut assertion, fixed in bbb6822. The integration test has not run locally.

Separate, pre-existing issue found along the way: metagraph build --graph sshash on raw reads with repeated k-mers (e.g. tests/data/transcripts_100.fa) writes 1 GiB sshash.tmp.run_*.minimizers.*.bin files into the CWD without bound (≈1.4 TB before the disk filled). The test therefore builds SSHash from transform --to-fasta contigs, as the unit-test helper does.

Reported impact (spring 2026 benchmarks, not re-measured here)

On a 67.3B-node SSHash graph, 2500 queries, no mmap, RowDiffChar vs. the fork_succ encoding:

  • row-diff step 32% faster, BRWT step 19% faster
  • total annotation time −27%, wall time −13%
  • +2 GB on disk (23 GB vs 21 GB)

At that scale, succinct + row-diff was still faster overall, so this helps only if SSHash is adopted. SSHash v6 integration waits on consolidating jermp/sshash #97–#104 into the ratschlab/sshash fork. Re-measuring on the 67B graph (on Euler) is a separate, optional step.

🤖 Generated with Claude Code

adamant-pwn and others added 5 commits October 28, 2026 02:05
…SHash

Add a new RowDiffChar<BRWT> annotation format (.row_diff_char_brwt.annodbg)
optimized for SSHash graphs. Instead of storing the full fork_succ bitvector,
it exploits SSHash's succ_is_next/pred_is_prev properties to avoid storing
successor information for ~94% of nodes. Only the ~6% non-trivial nodes
get a 2-bit character stored in a sparse array.

Key changes:
- DeBruijnGraph::trivial_outgoing(): virtual method returning the unique
  successor if resolvable in O(1). Default returns npos; SSHash overrides
  using succ_is_next_/pred_is_prev_ bitvectors.
- RowDiffMode::CharSucc + RowDiffChar<BaseMatrix>: new row-diff variant
  using sparse rd_succ_needs_char_ bitvector + rd_succ_char_ int_vector<>.
- Serialization format v3.0 for CharSucc; v2.0 (Standard) unchanged.
- Transform path: row_diff_brwt -> row_diff_char_brwt via compute_rd_succ_char().

Backward compatible: succinct graphs use Standard mode (v2.0) as before.

Ported onto master from refs/pull/604/head (0a79785): resolved
against the get_rd_ids/get_column_from_rows refactors, kept master's
sdsl-lite pin (contains d7fffb72), and excluded row_diff_char_brwt from
the embedded-anchor reformat path since it needs the graph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- trivial_outgoing() must equal the unique successor found by k-mer
  lookup, for all graph types (basic) and canonical-capable ones.
- row_diff_brwt -> row_diff_char_brwt on basic/canonical/primary SSHash
  graphs keeps stats and labels/matches/signature query output identical,
  single- and multi-threaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…contigs

- At k=3 the canonical test graph is too branchy for any SSHash shortcut,
  so require shortcuts across all k instead of for each k.
- Build the SSHash integration-test graphs from deduplicated contigs:
  `build --graph sshash` on the raw transcripts wrote 1 GiB minimizer
  files without bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With -p > 1, `metagraph query` prints results in no fixed order, so the
row_diff_brwt and row_diff_char_brwt outputs only match as sets of lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Protein graphs have no reverse complement, so building them in canonical
mode fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant