Repository navigation
Add the --min-ic information-content floor and the ic column - #70
Conversation
The temporal benchmark has always weighted terms by the marginal CAFA-style IC(t) = -log2(P(t)) over propagated training frequencies. The upcoming --min-ic reporting floor and the exported ic column need the same quantity, and results are compared across the two, so the counting convention moves to src/information_content.py and the benchmark's version becomes a propagating wrapper around it rather than a second implementation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
Parentless terms skip the relative inference (relative_p = 0) and pass on the overall p-value alone, so DAG roots dominate a relative-inference run — 'biological_process' for 9,623 domains on allspecies. Nothing in the paper's Steps 1-3 prevents it: a root has no parental background to be tested against. The IC floor removes that vacuous top of the DAG directly. IC(t) = -log2(P(t)) over the run's own analysable universe, the shared convention from src/information_content.py. The frequencies are estimated from the True-Path-propagated map whenever a hierarchy is in play — an unpropagated P(t) understates non-leaf frequency and inflates mid-level IC — including when --min-ic alone engages it (a throwaway propagated copy; the tested map is untouched). Flat vocabularies use the direct map, which for them is the same thing. Roots have IC 0 by construction, so any positive floor kills them. The floor is applied AFTER the BH correction, exactly like --min-support and for the same reason: it narrows what is reported and changes no q-value, so the hypothesis family is never shrunk by a property of the outcome. Default 0 (off). Every association export carries the term's ic as a new trailing column either way, the manifest thresholds block records min_ic and ic_source, and --min-ic > 0 makes the hierarchy inputs mandatory up front. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
The mean-#ancestors / on-a-chain / roots-present numbers in VALIDATION_PLAN's relative-inference item were computed ad hoc. validation/specificity_metrics.py computes them from any runner association TSV, over the pipeline's own OntologyProcessor so the edge policy matches the measured runs, and sweeps --min-ic floors against the exported ic column — valid as a sweep because the pipeline applies the floor after BH, so a floored view of one unfloored run is exactly the floored run's reported set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
Human GO single domains, paper-parity config, one run swept post hoc (valid because the floor is post-BH): --min-ic 1 removes the three GO roots and the near-universal band above them for 353 of 30,655 associations (1.2%), leaving the ancestor-chain share nearly unchanged (52.7% -> 50.2%) — the floor kills vacuous terms, it does not decorrelate chains. The GO roots sit at IC 0.09-0.17 rather than exactly 0 (each aspect covers most but not all of the universe), so the 'IC 0 by construction' wording is tempered to universal terms where it is exact. Verified at the runner level: --min-ic 1 reproduces the swept counts and records min_ic/ic_source in the manifest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 59 minutes Limit details: You’ve used all 1 included review currently available under your plan. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (18)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The throwaway propagation behind ic_source="propagated" had two defects. It materialised a full propagated copy of the protein→term map — a multi-GB transient at allspecies scale — for nothing but a frequency counter. And it skipped the input cleaning --propagate-annotations performs (alt_id remap to live primary ids, dropping terms the hierarchy does not contain), so the same manifest value named two different estimates depending on which flag engaged the hierarchy. Both fixed by streaming: src/hierarchy.py gains iter_propagated_term_sets, a lazy twin of propagate_annotation_map built on the same closure builder so the two cannot disagree on unknown-term handling, and src/information_content.py gains information_content_from_term_sets, which counts a stream of per-protein closures without ever holding the propagated map. The runner feeds it alt_id-remapped term sets with drop_unknown=True — exactly the --propagate-annotations policy — and the temporal benchmark's IC wrapper streams the same way. A no-op on current data (zero alt_id/unknown annotations in the 2026 human GAF). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
--min-ic dropped a floored term from the significant table, but Stage 5.5 True-Path propagation then re-derived it — a root sits above every surviving term — straight into domain_*_annotations_propagated.tsv, so the one export downstream consumers feed on kept exactly the vacuous rows the floor exists to remove. The propagated annotations are now floored on the annotated term's IC before export (direct rows already passed by construction), the manifest's propagated_annotations count follows, and the help/docs state the deliverable-wide contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
The exported ic was rounded to 4 decimals while the in-run --min-ic floor compares float64, so sweeping the exported column could disagree with a real floored run at floor boundaries — exactly the equivalence validation/specificity_metrics.py advertises. All three exports now write ic with %.10g and the docstring states the guarantee. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
Three failure modes in validation/specificity_metrics.py produced plausible numbers instead of errors. A table measured against the wrong ontology gave every term an empty closure — 0.0% on-chain, no roots, indistinguishable from a perfect result — so zero overlap is now refused outright and under-50% overlap warns. The --domain-type double-equality trick silently binned missing or foreign domain_type values into the supra family; it is now a membership check over the runner's DomainType values that errors on anything else. And the ic column was detected from rows[0], so an empty-but-valid TSV (a zero-association run) crashed the sweep; it is now read from DictReader.fieldnames and sweeps to zero-count rows. main() takes argv so the guards are testable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
The four-flag "this run reads the hierarchy" disjunction (propagation, either hierarchy stage, or the --min-ic floor) was spelled out three times — resolve_inputs, resolve_ic_source, and the manifest's hierarchy_inputs — inviting the fourth copy to drift. It is now one property, RunRequest.engages_hierarchy, read by all three. --min-ic is also validated with math.isfinite: NaN compares False against everything, so it sailed through the >= 0 check, silently disabled the floor, and wrote literal NaN — invalid JSON — into the manifest; infinity would have dropped every association. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
run_dcgo_human.py's domain_go_annotations_propagated.tsv gained an ic column, but validation/ablation.py writes a file of the same name for its True Path rungs with the old six-column header — one filename, two schemas, and a KeyError waiting for any consumer that reads ic by name. The ablation now computes the same annotation-frequency IC over its own analysable universe (the propagated t0 map restricted to the domain∩annotation intersection, matching the runner's convention) and writes the column at the same %.10g precision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
src/information_content.py and validation/specificity_metrics.py were written fully annotated but sat outside the pyproject files list, so CI's strict fence never checked them (and specificity_metrics carried a real strict-mode error — declaring list[Mapping[str, str]] while returning list[dict], fixed with the loader rework). Fence expanded per the CLAUDE.md policy; uv run mypy is clean on all seven modules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
The --min-ic sweep table was re-derived from a fresh paper-parity run whose ic column is written at full precision: all five rows reproduce exactly (30,655 / 30,302 / 28,348 / 26,401 / 18,888 and identical chain shares), so the earlier 4-decimal rounding sat no association on a floor boundary and the table stands unchanged. The same run confirms the streamed IC-estimate cleanup is a no-op on current data: zero alt_id remaps and zero unknown-term drops on the 2026 human GAF. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd
…rk (#72) * Update manuscript Methods to the post-#61-#70 pipeline The Methods described a superseded pipeline. Since the draft was compiled (2026-08-04), relative inference was split from the True Path flag, generalised to every hierarchical ontology, and folded into the inference before BH as max(overall_p, relative_p) with h-score min(overall, relative) over the union-of-direct-parents background; input annotations gained --propagate-annotations with alt_id remap and unknown-term exclusion; GO propagation was restricted to is_a/part_of (dropping 7,799 regulates-family edges — an era boundary for every propagated GO number); the shrinkage step was removed outright; the --min-support/--min-ic post-BH reporting floors and the ic column landed; and run manifests now record provenance including the propagation_relations era marker. Sections rewritten: 2.1 (manifests exist; draft numbers predate them), 2.3 (shrinkage removed, reporting floors), 2.4 (three propagation sites, edge policy, era marker), 2.5 (in-inference relative test, parentless-term defect, named paper-parity configuration), 2.6 (28 registry keys in six kinds, five hierarchy mechanisms, species defect marked resolved). New 2.7 documents the gene-keyed sources and the audited id-mapping policy, with verified per-layer gene coverage and the ZFA-not-ZP honesty note; former 2.7/2.8 renumbered. The draft conventions box gains the pre-regulates-fix era-marking rule. Ledger rows K* and L* referenced here are added in the companion ledger commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd * Mark pre-regulates-fix numbers PROVISIONAL across Results and Limitations Era pass following the is_a/part_of restriction (#67): every result that passed through GO propagation — InterPro2GO coverage (3.2), the temporal benchmark's gate, truth sets and baselines (3.3), the post-hoc relative filter measurement (3.4), the held-out surprise test (3.6) and the breadth table's GO anchor (3.7) — now carries a PROVISIONAL pre-regulates-fix banner pointing at regeneration from post-#67, manifest-carrying runs. The 164,549/165,823 headlines are flagged as manifest-less pre-parity artifacts (the Fisher stage itself does not propagate, so they are era-stale rather than corrupted), with the paper-parity 30,655 stated alongside. Propagation-free figures (dataset scale, surveys, non-GO breadth rows on their own is_a/part_of closures) are deliberately not blanket-marked. Limitations brought up to date: the component ablation has been run (negative for supra-domains and shrinkage; the True Path rung is confounded and must be re-measured), shrinkage is removed rather than optional, bench_A-D are now committed, run manifests exist but cover none of the reported numbers, and the resolved species defect is replaced by the propagation-era regeneration debt. Stale TABLE 3-CORRECTED placeholder dropped: the species-fix recomputation it demanded happened on 2026-08-04 and the table above it is the result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd * Add the ontology-expansion and specificity Results; update Abstract New 3.8 reports the honest state of relative inference: the paper-parity run's 30,655 single-domain associations, the --min-ic sweep (30,655/30,302/28,348/26,401/18,888 with ancestor-chain shares), the roots' 0.09-0.17-bit band and the 1.2% cost of --min-ic 1 — and, foregrounded, what the floor does not fix: the 52.7% -> 50.2% ancestor-chain cascade, with elim-style decorrelation named as the open idea. New 3.9 reports the multi-vocabulary expansion: the 39-source acquisition with its licence/registration boundary (SNOMED CT, MedDRA, OMIM genemap2, MAxO), the seven merged phenotype layers with per-layer significant counts marked PROVISIONAL pending the manifest-carrying production run, the curation-shape reading, and the plain statement that none of these layers is validated yet. Wave-3 layers are named as in-review and excluded from every count. Discussion: the published-dcGO comparison is no longer missing — precision 0.537-0.625 across six pre-declared variants on the shared SCOP-superfamily space, recall not interpretable; relative inference now counted among the restored pieces. Limitation 8 records the all-species background run as direction, not result. Abstract updated: 26 vocabularies beyond GO, method-parity summary, era-flagged headline with the 30,655 parity count, the expansion sentence, and a conclusions line that replaces the removed shrinkage caveat with the unresolved specificity purpose. Open questions 5/10/11/12/13/17 annotated resolved or updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd * Extend the evidence ledger: era annotations and blocks K and L Header gains the update stamp and the era convention: artifacts that passed through pre-#67 GO propagation (regulates-family edges) are pre-regulates-fix era and must be regenerated before quoting; manifests carry propagation_relations as the marker. Block-level era notes on B (both sides propagated via OntologyProcessor; B7-B9 also measured the superseded post-hoc relative filter), C (gate, truth, baselines and transfer all propagate), E (propagated evidence closures over the pre-parity t0 set) and F (GO anchor only; the non-GO closures never traversed regulates edges). Row annotations on A7/A8 (era-stale headlines), A10 (edge count predates the restriction), A12 (19 keys superseded by K1's 28) and G11 (comparison since run, see K9). New block K records the current pipeline: 28 registry keys / 19 with hierarchies, the 7,799 dropped edges, the paper-parity 30,655 and the --min-ic sweep (provisional: prose table, reproducible via validation/specificity_metrics.py), the root band and the 1.2% floor cost, shrinkage removal, the ablation verdicts with the TPR attribution explicitly unsupported, the published-dcGO precision band, the all-species direction, the in-inference relative design, and the manifest machinery. New block L records the expansion: the audited remap policy, gene coverage per layer (recomputed 2026-08-17 by the committed scripts/verify_gene_coverage.py: hpo 98.5%, syngo 99.9%, mp 85.0%, wbphenotype 82.5%, zfa 68.6%, fbcv 35.1%, fbbt 42.0%), the single-gene/single-allele policies, the ZP-not-derivable assessment, per-layer significant counts as provisional pending the production-run manifests under results/production/, the 39-source acquisition record with its licence boundary, and the wave-3 in-review exclusion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd * Fix cross-references left stale by the methods rewrite Five hierarchy forms (not four) in 2.1, pointer to 2.6-2.7 for the non-GO inputs, and the 3.3 restored-pieces paragraph updated: the relative test named as the superseded post-hoc form, and bench_A-D described as committed rather than untracked, with the era caveat. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd * Resolve Table 5 from the production matrix; fold in the merged wave-3 layers The production matrix completed (63 cells, 0 failures, manifests under results/production/ at main commit d166013, every one recording the is_a/part_of edge policy), and wave 3 merged as #71, so 3.9 is rewritten from artifacts instead of pending markers. Table 5 now reads fourteen expansion layers from the production TSVs (domain_type = single), with baseline and paper-parity columns and a caption naming both configurations and the per-family FDR exactly. The baseline values confirm every dev-worktree count; the paper-parity column moves in both directions (sparse re-keyed layers grow under input propagation, rich layers shrink under the intersection-union test), and the hpo collapse 996 -> 38 is flagged as unexplained rather than interpreted. 2.6 and the abstract move to the merged registry (35 keys, 25 hierarchical, 33 substantive vocabularies beyond GO), adding the Mondo re-keys, the expression layers and the CIViC-chained cancer layers as their own kinds. Ledger: L7 resolved against the production paths (values confirmed); L9 marked merged, with the Mondo/EFO qualifier-stripping note making production numbers the only citable ones for those layers; new L10 records the wave-3 parse policies from the merged docstrings; new block M opens with the matrix description (M0), the per-layer table (M7) and the registry counts (M8); K1/K2 annotated superseded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd * Quote the production headline and parity counts; fix the BH-family description 2.3 no longer claims a single pooled BH family: single and supra-domains have been corrected as separate families against their dense counts since #44, which every production manifest records (fdr_families, per-family thresholds) — the 2021 training run predates the split and is said to. The headline paragraph, 3.1 and the abstract now quote the manifest-carrying production numbers: baseline 165,687 (44,453 single / 121,234 supra) over 1,690,803,963 tests, paper-parity 96,419 with 30,302 single-domain under --min-ic 1. The 3.1 era banner becomes a resolved note: the current-release headline is regenerated post-fix (superseding the manifest-less 165,823), while 164,549 stays quoted as the pre-fix training run the temporal evaluations actually used. 2.5 and the Table 4 caption record that the production run reproduces the sweep's floor-1 count exactly. Ledger: new M1/M2 (go_baseline, go_paperparity — values verified against manifest summaries and domain_type-filtered TSVs) and M9 (per-family BH, #44); A8 marked superseded by M1; K4/K5 annotated with the M2 cross-check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd * Report the all-species and experimental-evidence production cells New 3.10 quotes the manifest-carrying GO variants: all-species baseline 2,911,662 significant (535,133 single) over 1,464,355 proteins and parity 5,597,840 (1,051,061 single) — with the direction reversal against human GO (parity grows 1.92x where human shrinks 42%) flagged as an uninvestigated observation — and experimental-evidence baseline 62,426 (18,310 single) / parity 28,250 (10,127 single), about 38% of the manual baseline. Both are presented strictly as training-run records: the all-species evaluative wins stay provisional under K10's pre-fix era, the orthology-inflation caveat travels with the counts, and no evaluation exists for the experimental cells. Limitation 8 now points at these cells as the base for the missing post-fix re-evaluation. Ledger: M3-M6 added, each value verified against the named manifest summary and its domain_type-filtered significant TSV. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd * Scope the manifest-coverage statements to the evaluations The production matrix means "no number is covered by a manifest" is no longer true: 2.1, limitation 19, the section 6 reproducibility status, the draft-conventions era note and the ledger's cross-cutting caveat now say precisely what remains uncovered - the evaluation results of 3.2-3.7, which still rest on pre-manifest runs - while the association counts are regenerated under manifests (M0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd * Replace the hpo-collapse "unexplained" flag with the measured characterisation The 996 -> 38 paper-parity reduction is not the IC floor (production log: 43 kept, 0 dropped) and not degenerate parents (4.1% of 4,000 sampled terms-with-parents match their direct-parent union exactly, 1.1% more cover >=95% - session measurement, seed 1); it is the relative inference governing 13,742,781 of 14,829,673 evaluated pairs (92.7%). Stated in the Table 5 discussion as consistent with the layer's dense, disease-block-derived annotation making the baseline signal overwhelmingly inherited - not a pipeline defect - with the specificity-vs-over-conservatism question left to the pending post-#67 temporal evaluation. Ledger row M7a carries the log and manifest locators for the counts and the inline recipe for the session measurement; M7's flag note updated to point at it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd * Address the merge-blocking review on #72 - Scope the pre-regulates-fix era statement (limitation 17 and the §2.4 methods note) to the evaluation results in §§3.2-3.4, §3.6 and the §3.7 GO anchor; the §§3.9-3.10 production counts are post-fix and manifest-carrying (block M). - Report acquisition as 40 attempted / 39 verified / 1 failed (CellMarker 2.0, HTTP 403) identically in the manuscript and ledger L8, instead of the contradictory "39 fetched" / "39/39 OK". - Qualify the FlyBase unmapped-tail reading as spot-check-based ("may reflect design") rather than asserted. - Back the cofactor/tcdb Discussion claim with counted artifacts: new ledger row M7b (cofactor 567 -> 1,250, tcdb 68 -> 753 single-domain, from the production TSVs) cited instead of stretching [M7]. - Ledger consistency (CodeRabbit): add the 2026-08-18/d166013 snapshot line for block M; mark H12/C5a's untracked-bench wording historical (bench_A-D committed); scope K12's "no number covered by a manifest" to blocks B-F. - Type-annotate scripts/verify_gene_coverage.py via a GeneKeyedSource protocol. CI red was main's unformatted run_production_matrix.py, fixed on main as 7f22434 and merged in. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Dan Bolser <dan.bolser@outsee.co.uk> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The fix for the documented blocker: parentless/near-universal terms skip the relative test and dominate output (GO roots for thousands of domains).
Sweep (human GO single, paper-parity config): --min-ic 1 kills all three roots + the near-universal band (biological regulation, cytoplasm, organelle…) for 353 associations (1.2%); higher floors start deleting legitimate mid-level terms. Recommendation: --min-ic 1 for production runs; default stays off.
Two honest findings: GO roots sit at 0.09–0.17 bits (not exactly 0 — each aspect covers most-but-not-all of the universe), and the floor does NOT fix the mid-DAG chain cascade (52.7% → 50.2% on-chain at IC≥1) — elim-style decorrelation remains the open item, flagged in VALIDATION_PLAN.
782 tests on the pre-rebase base; suite green post-rebase; ruff clean.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HtDbRGLiqQhtDh2ca57RMd