Repository navigation
RowDiffChar annotation format for SSHash graphs - #679
Draft
adamant-pwn wants to merge 5 commits into
Draft
adamant-pwn wants to merge 5 commits into
adamant-pwn wants to merge 5 commits into
Conversation
…SHash Add a new RowDiffChar<BRWT> annotation format (.row_diff_char_brwt.annodbg) optimized for SSHash graphs. Instead of storing the full fork_succ bitvector, it exploits SSHash's succ_is_next/pred_is_prev properties to avoid storing successor information for ~94% of nodes. Only the ~6% non-trivial nodes get a 2-bit character stored in a sparse array. Key changes: - DeBruijnGraph::trivial_outgoing(): virtual method returning the unique successor if resolvable in O(1). Default returns npos; SSHash overrides using succ_is_next_/pred_is_prev_ bitvectors. - RowDiffMode::CharSucc + RowDiffChar<BaseMatrix>: new row-diff variant using sparse rd_succ_needs_char_ bitvector + rd_succ_char_ int_vector<>. - Serialization format v3.0 for CharSucc; v2.0 (Standard) unchanged. - Transform path: row_diff_brwt -> row_diff_char_brwt via compute_rd_succ_char(). Backward compatible: succinct graphs use Standard mode (v2.0) as before. Ported onto master from refs/pull/604/head (0a79785): resolved against the get_rd_ids/get_column_from_rows refactors, kept master's sdsl-lite pin (contains d7fffb72), and excluded row_diff_char_brwt from the embedded-anchor reformat path since it needs the graph. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- trivial_outgoing() must equal the unique successor found by k-mer lookup, for all graph types (basic) and canonical-capable ones. - row_diff_brwt -> row_diff_char_brwt on basic/canonical/primary SSHash graphs keeps stats and labels/matches/signature query output identical, single- and multi-threaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…contigs - At k=3 the canonical test graph is too branchy for any SSHash shortcut, so require shortcuts across all k instead of for each k. - Build the SSHash integration-test graphs from deduplicated contigs: `build --graph sshash` on the raw transcripts wrote 1 GiB minimizer files without bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With -p > 1, `metagraph query` prints results in no fixed order, so the row_diff_brwt and row_diff_char_brwt outputs only match as sets of lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Protein graphs have no reverse complement, so building them in canonical mode fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the RowDiffChar annotation format, dropped from #604 when that PR was trimmed to the SSHash mmap load and
traverse()fast path, onto current master. It only matters if SSHash graphs are used for annotated indexes.What it does
DeBruijnGraph::trivial_outgoing(node): returns the unique successor when it can be resolved in O(1), otherwisenpos(the default).DBGSSHashoverrides it usingsucc_is_next_/pred_is_prev_. Both imply out-degree 1, so the result always matches the row-diff successor stored infork_succ.RowDiffMode::CharSucc+RowDiffChar<BRWT>(.row_diff_char_brwt.annodbg): replaces the per-nodefork_succbitvector with a sparserd_succ_needs_char_bitvector plus a 2-bit successor character, stored only for nodes thattrivial_outgoing()can't resolve. Serialized asv3.0. The existingv2.0format is unchanged and still loads as before.transform_anno --anno-type row_diff_char_brwt -i <graph> X.row_diff_brwt.annodbgconverts viacompute_rd_succ_char(). The BRWT diffs are copied unchanged.Porting notes (vs. 0a79785 on
refs/pull/604/head)get_rd_ids(). TheRowDiffCharclass now sits next toget_column_from_rows().row_diff_char_brwtis excluded from the new "embedded anchor" reformat path (Batch and share row-diff column reconstruction #659-era), which drops-i; the char transform needs the graph.v3.0loader gets the samemadvise_random_rangehint asv2.0and checks stream state after readingrd_succ_char_.sdsl-litepin, which already contains the commit'sd7fffb72. Thesshashpin is unchanged.Tests
trivial_outgoing_is_unique_successorcheckstrivial_outgoing()against successors found by independent k-mer lookup. It runs in basic mode for allGraphTypesand in canonical mode forCanonicalGraphTypes, and requires SSHash to actually take the shortcut.TestRowDiffCharSSHash(basic / canonical / primary SSHash, k=21,transcripts_100.fa) buildsrow_diff_brwt, converts it torow_diff_char_brwt, and asserts equal stats and byte-identicalqueryoutput inlabels,matchesandsignaturemodes, with-p 1and-p 4.Verification is left to GitHub CI. Locally (g++-15, DNA) Release and Debug builds succeed, and an earlier Release
unit_tests_parallel.sh -j 8run passed 3747/3748. The one failure was the over-strict per-k shortcut assertion, fixed in bbb6822. The integration test has not run locally.Separate, pre-existing issue found along the way:
metagraph build --graph sshashon raw reads with repeated k-mers (e.g.tests/data/transcripts_100.fa) writes 1 GiBsshash.tmp.run_*.minimizers.*.binfiles into the CWD without bound (≈1.4 TB before the disk filled). The test therefore builds SSHash fromtransform --to-fastacontigs, as the unit-test helper does.Reported impact (spring 2026 benchmarks, not re-measured here)
On a 67.3B-node SSHash graph, 2500 queries, no mmap, RowDiffChar vs. the
fork_succencoding:At that scale, succinct + row-diff was still faster overall, so this helps only if SSHash is adopted. SSHash v6 integration waits on consolidating jermp/sshash #97–#104 into the
ratschlab/sshashfork. Re-measuring on the 67B graph (on Euler) is a separate, optional step.🤖 Generated with Claude Code