Traversal v1 (1/4): label oracle and /resolve support profiling by adamant-pwn · Pull Request #708 · ratschlab/metagraph

adamant-pwn · 2026-11-01T22:12:59Z

First of four stacked PRs that bring labeled traversal onto master, rebuilt in reviewable pieces from Gunnar's gr/labeled-traversal (d5d26baa). Next in the stack: #709 (the walker), #710 (the CLI and server routes) and #711 (the Python client and traverse.md).

This PR adds the backend of /resolve. For a query sequence, it finds which labels carry which runs of its k-mers, and it selects seeds for /traverse. A label is an annotation column or, on an index with coordinates and a header index, one sequence record named by its FASTA header.

LabelOracle maps query k-mers to annotation rows in every graph regime and resolves label names. It refuses a name that is both a column and a header, or the header of several records.
LabelQuery reads which permitted labels are at a set of rows, in batches and with a bounded cache.
LabelQuery does not reuse the aligner's AnnotationBuffer. That buffer keeps whole rows, never evicts and turns coordinates off on canonical and primary graphs. A walk needs bounded memory and header labels on those graphs. Both use AnnotationKeyMapper for orientation.
LabelQuery keeps the rows it decodes (RowDiffCache, at most 64 MiB per request). A walk reads rows of neighbouring nodes, whose row-diff paths overlap, so each path is decoded once instead of at every level. IRowDiff gains get_rows_cached and get_row_tuples_cached for this. It makes /traverse 2.5x faster on the benchmark in Traversal v1 (2/4): the labeled traversal walker #709, with identical responses.
resolve_support profiles explicit labels or discovers them. Discovery ranks labels with the code of /search with signatures (AnnotatedDBG::get_top_label_signatures), so both answer alike: on 10 RefSeq r237 windows they gave the same labels, counts and runs. That function now reports how its list was cut, and it can rank columns on an index with headers.
select_seeds picks seeds by maximum support: repeatedly, the interval inside the most label runs, outside the earlier picks. It lists them best first.
CoordToHeader gains a reverse index from header to record. It stores record ids sorted by header hash, 8 bytes per record: 161 MiB for 20.9M records.

Changed outside traversal: get_header_results throws instead of exiting the process, and CoordToHeader::map_to_local_coords skips columns without sequence records. CanonicalDBG::with_node_first_cache builds the canonical wrapper that align.cpp and LabelOracle now share.

Scope decisions from review:

Kept: explicit labels and discovery, which the MCP tool traverse_resolve uses, and seed selection with max_seeds, min_block_bp and max_labels_per_seed.
Dropped: longest_first and the merging of overlapping picks. Gunnar's spec offers it for a seed spanning the whole gene, carried by fewer labels. No caller uses it. Setting min_block_bp to the query length gives that seed. The policy can come back when a caller needs it.
Dropped: merge_overlapping and min_block_kmers. No caller set them.
Changed from the prototype: seeds stay in rank order. gr/labeled-traversal has the same order since 01c05323b.
Left out of Gunnar's version: the decode budget and read pacing (the decoded-row cache is ported without them), trace support, annotate mode, deadlines and the memory model in /resolve, and the explicit selection policy.

On RefSeq r237 bact_other (mex, 2 threads, warm median of 21 queries of 500 bp), /resolve takes 10 ms with header discovery, 3 ms with column discovery and 3 ms with explicit labels. The prototype takes 8, 2 and 3 ms.

The routes are experimental. #710 serves them only with --experimental-traverse, so any field here can still be removed without breaking a release.

Ported from graph/traversal/{traversal_types,label_oracle,resolve} and annotation/coord_to_header on d5d26baa.

Co-authored-by: Gunnar Rätsch

🤖 Generated with Claude Code

First piece of the labeled-traversal v1 stack, rebuilt from Gunnar's gr/labeled-traversal (d5d26ba) without the budget, pacing, trace and annotate layers: - traversal_types: regimes, label references, arms and end reasons. - LabelOracle: the oriented graph to walk (a per-request CanonicalDBG in the PRIMARY regime), node-to-annotation-key mapping through AnnotationKeyMapper, label resolution (column, then header), and batched get_rows / get_row_tuples. - LabelQuery: membership of a fixed permitted label set per key, column labels from rows, header labels from tuple rows by coordinate range. - resolve_support / select_seeds: per-label k-mer support of a query (explicit labels or discovery by column or header), seed candidates, and the max_support / longest_first seed selection. - CoordToHeader::find_header (a lazily built reverse header index owned by the object) and last_coord. Source paths on d5d26ba: metagraph/src/graph/traversal/{traversal_types,label_oracle,resolve}.{hpp,cpp} metagraph/src/annotation/coord_to_header.{hpp,cpp} metagraph/tests/graph/traversal/{test_label_oracle,test_resolve}.cpp Co-authored-by: Gunnar Rätsch <Gunnar.Ratsch@ratschlab.org> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An LE-v2.0 label encoder is deserialized with the hash buckets its writer stored, which are only valid for the writer's std::hash. An annotation written by another toolchain then decodes its labels but finds none by name: label_exists("562") is false on the published mini RefSeq index although decode() returns "562". Read the stored record whole and rebuild the buckets from the labels. From d5d26ba metagraph/src/annotation/representation/base/annotation.cpp. Co-authored-by: Gunnar Rätsch <Gunnar.Ratsch@ratschlab.org> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

All reported issues were addressed across 11 files

Architecture diagram

sequenceDiagram
    participant Client as Client / Tests
    participant RO as resolve_support / select_seeds
    participant LO as LabelOracle
    participant LQ as LabelQuery
    participant CTH as CoordToHeader
    participant LE as LabelEncoder
    participant Mat as Annotation Matrix (BinaryMatrix / MultiIntMatrix)
    participant DBG as DeBruijnGraph (BASIC / CANONICAL / PRIMARY wrapper)

    Note over Client,DBG: Label Oracle and /resolve support profiling

    Client->>LO: Create LabelOracle(anno_graph)
    LO->>DBG: Determine regime (BASIC / CANONICAL / PRIMARY via CanonicalDBG wrapper)
    alt PRIMARY regime
        LO->>DBG: Create own CanonicalDBG wrapper (local caches)
        LO->>DBG: Add NodeFirstCache extension
    end
    LO->>Mat: get_matrix() (single fetch)
    LO->>CTH: Obtain CoordToHeader (if available)
    Note over LO: Matrix + CoordToHeader cached per oracle

    Client->>RO: resolve_support(query, options)
    RO->>LO: keys_of_sequence(query)
    LO->>DBG: map_to_nodes(query)
    DBG-->>LO: Oriented node keys
    LO-->>RO: Annotation keys (with npos for absent k-mers)
    RO->>RO: Filter present keys, compute graph_runs

    alt Explicit labels
        RO->>LO: resolve_label(name)
        LO->>LE: find_column(name)
        LE-->>LO: Column or not found
        opt Not a column
            LO->>CTH: find_header(name)
            CTH-->>LO: (column, seq_id) or nullopt
        end
        LO-->>RO: LabelRef (COLUMN or HEADER)
        RO->>LQ: Create LabelQuery(labels)
        LQ->>LQ: Build column_labels / header_ranges index
        RO->>LQ: fetch(keys)
        LQ->>LQ: Check cache
        alt Cache miss
            LQ->>Mat: get_rows(rows) or get_row_tuples(rows)
            Mat-->>LQ: Row data
            LQ->>LQ: Extract hits (column or header coordinate lookup)
        end
        LQ-->>RO: NodeHits per key
        RO->>RO: Aggregate support runs per label
    else Discovery mode
        RO->>LO: get_rows / get_row_tuples (batched)
        LO->>Mat: Fetch rows in batches
        Mat-->>LO: Row data
        LO-->>RO: Column/tuple rows
        RO->>RO: Discover labels, count k-mers, rank by support
        opt Over discover_max_labels
            RO->>RO: Truncate, record LabelTruncation
        end
        RO->>LO: column_name / header_name for display
    end

    RO->>RO: Build seed candidates (group identical runs)
    RO-->>Client: SupportProfile

    Client->>RO: select_seeds(profile, query, policy, canonical)
    alt LONGEST_FIRST
        RO->>RO: Sort candidates by interval length desc
        RO->>RO: Pick non-overlapping longest seeds
    else MAX_SUPPORT
        RO->>RO: Find interval with most supporting labels, merge overlaps
    end
    RO->>RO: Compute seed_id via fnv1a64 (canonical orientation, case mapping)
    opt Over max_labels_per_seed
        RO->>RO: Deterministic drop by hash, record population digest
    end
    RO-->>Client: SeedSelection

    Note over CTH: Reverse header index (lazy build, thread-safe)
    Client->>CTH: find_header(name)
    alt First call
        CTH->>CTH: Build HeaderIndex map from headers_ (under mutex)
    end
    CTH-->>Client: (column, seq_id) or null

    Note over LE: Fixed LE-v2.0 loading
    Client->>LE: load(instream)
    LE->>LE: Read saved VectorSet buckets
    LE->>LE: Rebuild buckets from labels (std::hash may differ)
    LE-->>Client: Populated encoder (label_exists works)

_{Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic}

Merging overlapping picks sorted the seeds by position. MAX_SUPPORT picks never overlap, so under the default policy merging only lost the best-first order. Merge LONGEST_FIRST picks only, give a merged pick the rank of its best member, and keep the seeds in rank order. Explicit labels decoded every row of the query in one call. Read them kResolveBatchRows at a time, as discovery does. Reject discover_max_labels = 0, and do not count cache hits for rows that are read again after a cache clear. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 3 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 5 unresolved issues from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic}

Merging overlapping picks sorted the seeds by position. MAX_SUPPORT picks never overlap, so under the default policy merging only replaced the best-first order with position order. Merge LONGEST_FIRST picks only, give a merged pick the rank of its best member, and keep the seeds in rank order. Same change as Traversal v1 (#708, 6961a59); the service's port of select_seeds changes with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A name that is both an annotation column and a sequence header resolved to the column, and a header of several records to its first record, without saying so. Refuse both with std::invalid_argument (a 400 on the server). CoordToHeader marks the headers of several records, can build its reverse index ahead of time (index_headers), and answers lookups outside its lock once the index is built. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 5 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 5 unresolved issues from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic}

Discovery kept every label it saw before keeping the best max_labels, so its memory grew with labels and runs. Refuse a discovery that sees more than 100,000 distinct labels (kMaxDiscoveredLabels). Header discovery mapped the coordinates of every column, and a column without sequence records threw out_of_range, failing the request. Skip such columns: they have no header labels (LabelOracle::has_header_records). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 5 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 5 unresolved issues from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic}

LabelQuery cleared an over-full cache and then stored the whole batch in it, so a batch with more keys than max_cache_size grew the cache past its bound. Answer such a batch from a temporary map. CoordToHeader::load replaced its vectors in place, so a failed load left a half-loaded object. Read into new vectors and swap them in, dropping the header index in the same step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 3 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 2 unresolved issues from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic}

load() rebuilds the hash buckets from the labels (#725), so the buckets saved in each LE-v2.0 record are unused. Dropping them needs a format version that earlier builds cannot read, so it waits for the next change to this format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Adds an internal tested label-oracle and resolve-support module plus a CoordToHeader reverse-header index, not yet user-facing. The dismissed P2 (discover_max_labels=0 underflow) is unverified in the truncated diff, so human review is needed.

View guided diff | Turn on auto-fix | Re-trigger cubic}

No caller sets either. Overlapping longest_first picks are always merged (max_support picks never overlap), and every support run is a seed candidate; select.min_block_bp is the one minimum seed length. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 3 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Adds internal label-oracle/resolve-support modules and a CoordToHeader reverse-header index; not yet user-facing. The dismissed P2 (discover_max_labels==0 underflow) is unverified and has no visible fix, so human review is needed.

View guided diff | Turn on auto-fix | Re-trigger cubic}

The merge tests before it raise max_seeds to 3 and the label-cap check reused the policy, so it found three seeds where it expects one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Adds internal label-oracle and resolve-support modules, a CoordToHeader reverse header index, and tests; not user-facing yet. The dismissed discover_max_labels==0 underflow (P2) is unverified and no fix is visible in the diff.

View guided diff | Turn on auto-fix | Re-trigger cubic}

/resolve discovery ranked labels and built their runs with code of its own, while /search with signatures (AnnotatedDBG::get_top_label_signatures) answers the same question. On 10 RefSeq r237 windows both gave the same labels, k-mer counts and runs (4,375 of 4,375). Discovery now calls get_top_label_signatures, which can report how its list was cut (TopLabelCut, for labels_truncated) and rank columns on an index with headers. The distinct-label cap is gone: ranking keeps one counter per label and bitmasks only for the labels kept. get_header_results now throws instead of exiting the process on an incompatible annotation, and CoordToHeader::map_to_local_coords drops the columns without sequence records, which have no header labels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

All reported issues were addressed across 9 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic}

RowTuples is a folly::fbvector in builds with folly, which std::erase_if does not accept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 3 unresolved issues from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic}

- Leave headers of several records out of discovery: they are ambiguous, as when named explicitly, and their coverage belongs to one record. - get_top_label_signatures takes its get_top_labels shortcut only when no cut is requested and headers may be ranked. - build_anno_graph always constructs a MutableAnnotatedDBG, so that with_coord_to_header downcasts safely, now with dynamic_cast. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 4 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Adds internal label-oracle and resolve-support modules with a CoordToHeader reverse index and tests; the dismissed discover_max_labels==0 underflow and discovery_fraction==1 fast-path bypass are not visibly fixed in the truncated diff.

View guided diff | Turn on auto-fix | Re-trigger cubic}

The reverse header index was a hash map from header to record: 1.25 GiB for the 20.9M records of a RefSeq r237 chunk, measured on mex (about 64 bytes per record). Keep the global record ids sorted by header hash instead, and compare the headers that share a hash. Lookups take a binary search; find_header and is_repeated_header answer as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

All reported issues were addressed across 2 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic}

align.cpp and LabelOracle each wrapped a PRIMARY graph into a CanonicalDBG and added a NodeFirstCache for a DBGSuccinct base. Do it once in CanonicalDBG::with_node_first_cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 4 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved issue from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic}

No caller selects longest_first: only Gunnar's suite and an example in traverse.md use it. It was also the only reason to merge overlapping seeds, since max_support picks never overlap. Seeds are now picked by maximum support only. Bringing longest_first back is a small change once a caller needs whole-gene seeds; meanwhile select.min_block_bp sets a length floor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

All reported issues were addressed across 3 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic}

A walk reads the rows of neighbouring nodes one level at a time, and their row-diff paths share most of their rows. Without a cache every level decoded its path to an anchor again. RowDiffCache keeps the decoded rows of every path read, bounded by an estimate of their bytes (two generations), and a path now stops at the first row it holds. LabelQuery keeps one per request; on a row-diff coordinate annotation it reads tuple rows for column labels too, so that they are served from the same cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

All reported issues were addressed across 6 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic}

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 2 unresolved issues from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic}

With max_support the only policy, seeds never overlap and every pick is selected: overlaps_with was always empty, num_eligible equalled the number of seeds, and num_candidates is the size of the profile's candidates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- The walker's types (Arm, EndReason) move to the walker; kNumEndReasons and is_resource_stop had no callers. - LabelOracle::Counters: nothing reported them. - The CoordToHeader override of LabelOracle: tests now build their index with with_coord_to_header, and header discovery no longer refuses an override. - LabelQuery::labels() and access_path(): no callers. - SupportProfile::candidates: output only, derivable from the labels' runs. - LabelPopulation::included: the size of the seed's labels. - SeedSelection: select_seeds returns the seeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Its only caller was the candidates output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

adamant-pwn mentioned this pull request Nov 1, 2026

Traversal v1 (2/4): the labeled traversal walker #709

Open

adamant-pwn added this pull request to stack #714 November 3, 2026 20:00

adamant-pwn marked this pull request as ready for review November 8, 2026 18:28