Skip to content

TEST remove tests that other tests fully cover, and merge tests that differ only in an input - #129

Merged
jmschrei merged 21 commits into
mainfrom
test_remove_redundant
Oct 19, 2026
Merged

jmschrei merged 21 commits into
mainfrom
test_remove_redundant

Conversation

@jmschrei

@jmschrei jmschrei commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

This is the third test-only PR from the file-by-file review of the suite. It removes tests that another test fully covers and merges tests that differ only in an input or parameter into parametrized tests. Tests that overlap partially stay. The last PR in the series adds value checks to weak tests.

How each change was checked

  • Removals. For each removed test, the library was broken in a way that test catches. The break was made in a scratch copy, or by monkeypatching from a scratch plugin, never in the repository. The test named as covering it then had to fail too. Where it did not, the test stayed:
    • test_captum_deep_lift_shap_n_shuffles stays, because it is the only test where n_shuffles is smaller than the number of references given.
    • test_predict_flattendense_fp16 stays, because its pinned values fall outside the tolerance of test_predict_flattendense_16bit. It is merged with its bf16 twin instead.
    • test_deep_lift_shap_invalid_target stays, because TEST fix or remove tests that could not fail, and run three that never ran #128 changed the test that used to overlap it.
  • Merges. Each merge kept every assertion of the tests it replaced, for the same inputs. The assertions were listed before and after the edit. Pinned arrays were copied as source text and checked equal by parsing.
    • Where the originals checked different things, the merged test checks both for every case wherever that holds.
    • Several merges therefore gain a check, for example the dtype in both hypothetical-attribution cases and the value check for numpy annotations.
    • In test_ablate_deep_lift_shap_alpha, the beta case now checks that the attributions scale by beta, not only that they differ.
  • Skipped findings. Findings the review rated low confidence are not in this PR.

Removed or merged, by file

  • test_seqlet: fifteen older recursive_seqlets tests are removed, because the brute-force reference, error, tensor and control-threshold tests that UPDATE recursive_seqlets to decode every control threshold in one pass per row #125 added cover them:

    • numpy input;
    • the return_fdr and empty-control errors;
    • the default thresholds;
    • an out-of-range control;
    • an unreachable target;
    • control dtypes;
    • the p-value oracle, own-span p-values and sorting;
    • coordinate invariants;
    • flanks.

    The empty-X and invalid-threshold cases join test_recursive_seqlets_errors with their messages. Four tests fold into the tests that made the same calls.

  • test_deep_lift_shap:

    • The 20 one-model Sequential tests become one table, and so do the 11 max-pool tests (9 1D, 2 2D) and the captum comparisons.
    • Tests that ran the same call once for convergence and once for pinned values (the rule models, the transformer blocks) now run it once.
    • The attention, shared-module and cleanup tests are parametrized over the model.
    • Removed: LayerNorm/RMSNorm (rows of the rules table, with the same pins); positional-is-linear; two _disable_hooks tests; the ConvDense dense wrapper (its attributions are bitwise equal to FlattenDense's); custom_op_overrides_builtin (its loop never passed the custom op).
  • test_pisa:

    • The 20 one-model tests become test_pisa_sequential.
    • The rules convergence and regression tests make one call.
    • test_pisa_references_ignores_random_state is removed.
    • The pisa test named test_deep_lift_shap_shuffle_ordering is renamed.
  • test_predict:

    • The nine fp32 architecture tests and the fp16/bf16 pairs become parametrized tests.
    • The ConvDense batch-size tests compare the outputs by value.
    • _flattendense_fp64 is removed; test_predict_stored_dtype[float64] covers it.
  • test_saturation_mutagenesis: start/end, identity, irregular-column, input-dtype (now exact), span-error and low-precision tests are parametrized.

  • test_io:

    • Parametrized: _interleave_loci on paths and DataFrames, the summit errors (now matched to their messages), and the out_window, alphabet, integer-chromosome, width and count-threshold cases. The thresholds are now checked as inclusive (>=, <=).
    • Removed: a duplicated summits test, drops_sample_columns, seq_first_n, load_signals_values and one_hot_to_fasta_packed.
  • test_utils: the hand-written one_hot_encode cases, alphabet containers, characters cases, PackedSequence dtype and indexing tests, and packed random_one_hot are parametrized. Five one_hot_encode tests that reference comparisons cover are removed.

  • test_match: three GC-window tests are cases of the multiples-of-5 test, so they are removed. The determinism test repeated a call whose output test_extract_matching_loci pins, so it is removed too. The _calculate_char_perc and _extract_and_filter_chrom cases are parametrized, the latter now comparing whole dicts.

  • Smaller files: ablate, product, marginalize, space, variant_effect, saliency, dependency_map, annotate, kmers, plot and design/test_screen merge tests that differ only in the function, dtype, container or keyword.

    • Batch size 68 is clamped to 64. Two shape-only tests that ran at 68 are removed, and the ConvDense test now compares batch size 68 by value.
    • variant_effect's device test now also covers substitution_effect.

Size

Before (#128) After
Test functions 1,363 1,107
Collected cases 3,509 3,405
Lines in test files 28,371 25,436

Checks

  • Full suite on the CPU and one CUDA device: 3,400 passed, 3 skipped, 2 xfailed. The skips are pisa's two existing CUDA TODOs and the block-gzip FASTA test, which needs BioPython.
  • The CUDA cases of every merged file were also run on their own.
  • sphinx -W passes.
  • prepush flags @RULE_PARAMS and @TRANSFORMER_PARAMS, which are removed decorators, as removed credits. It also notes one space-indented line, copied unchanged from a pinned array in the original test.

The six surfaces

  1. Tests: they are the change.
  2. Docs, README: not touched.
  3. Changelog: one bullet under Testing.
  4. Bundled skill: not affected.
  5. Compatibility: no library code changes.

🤖 Generated with Claude Code

jmschrei and others added 21 commits October 19, 2026 15:12
…vered ones

Merged into parametrized tests: _interleave_loci on a path or a
DataFrame (single files, BED10 with and without summits, integer
chromosome names), its invalid inputs and summit errors (now matched to
their messages), extract_loci's out_window, alphabet and integer-chroms
cases, the window widths, the min_counts and max_counts cases (with the
inclusive bounds checked as such), the two missing-chromosome warning
tests, read_meme with and without n_motifs, read_vcf's column and dtype
checks, and packed extraction from a dict at the default and a small
chunk size.

Removed as covered by another test: the second copy of the
interleaved-summits test, test_read_vcf_drops_sample_columns,
test_extract_loci_seq_first_n (its dtype check moves into the n_loci mask
test), test_load_signals_values, and test_one_hot_to_fasta_packed, whose
cases test_one_hot_to_fasta_N and the packed chunk test cover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ered test

The twenty tests that each built one Sequential model and ran pisa under
warnings-as-errors become test_pisa_sequential, parametrized over the
layers and threshold with the old names as ids. The rules convergence and
regression tests made the same call and become one.
test_pisa_model_unchanged_after_call's handles check moves into
test_pisa_preserves_model_state, which made the same call.
test_pisa_references_ignores_random_state is covered by
test_pisa_reference_tensor, and the pisa test named
test_deep_lift_shap_shuffle_ordering is renamed test_pisa_shuffle_ordering.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pothetical or packing

The autograd and closed-form comparisons are parametrized over
gradient_correction (the closed form also over hypothetical, which adds
the uncorrected hypothetical case), the two pinned-value tests over
hypothetical, the int8 input test joins the stored-dtype test as its
no-args case, and the packed error test joins the two one-hot error tests,
gaining their message matches. test_saliency_is_signed is covered by the
autograd comparison, whose reference spans both signs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…overed ones

The non-negativity, receptive-field and off-diagonal tests made the same
call and become one; the packed int8 test joins the packed test as a case,
and the packed error test joins the two one-hot error tests.
test_dependency_map_skips_identity_substitutions checked column 0 of the
brute-force comparison's call, and test_dependency_map_func_saturation_mutagenesis
is covered by the brute-force comparison with that func.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… test into another

Parametrized: attribution_score over the offset, the start/end tests over
(start, end, hypothetical), the identity-recovery tests over the window,
the all-zero and multi-hot column tests, the input dtype tests (now with
exact equality, which holds for every cast), the four invalid-span tests,
and the fp16/bf16 tests over the dtype. test_saturation_mutagenesis_args
checked only the shapes of the two calls that
test_saturation_mutagenesis_args_applied_per_example makes, which now
also checks y0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h size

The fp32 tests that ran one body on nine models become one parametrized
test, and the fp16/bf16 pairs become tests parametrized over the dtype,
expected values and decimal. The ConvDense tests at batch sizes 2, 8 and
64 become one that compares the heads by value, the batch-68 test joins
the scatter test as a value comparison, and the identity and projection
func tests are parametrized. test_predict_flattendense_fp64 is covered by
test_predict_stored_dtype's float64 case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… covered ones

Parametrized: the hand-written one_hot_encode cases, the alphabet
containers, characters on a one-hot matrix, a PWM and a five-letter
alphabet, random_one_hot's invalid shapes, the PackedSequence dtype tests
(now asserting data, runs, decoded dtype and values for every dtype),
the two-axis indexing forms, and the packed random_one_hot cases.
Removed as covered by a reference comparison or a message check:
test_one_hot_encode_dtype, _raises_ignore, _lower_raises,
_rows_lower_case and _ignore. Space-indented continuation lines in three
pinned arrays are tabs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its two kmers assertions repeated test_kmers_packed_chunks, whose
largest chunk size is the default single chunk; it is renamed
test_gapped_kmers_packed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… covered ones

Parametrized: _calculate_char_perc's cases, _extract_and_filter_chrom's
chromosome, window and N-filter cases (now comparing the whole dict), and
extract_matching_loci's N and chromosome-edge cases. The GC-window tests
over even windows, over out_window (unused without a bigWig) and at
window 50 were cases of the test over every multiple of 5, which is
renamed to say so. test_extract_matching_loci_determinism made the call
that test_extract_matching_loci pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…covered one

count_annotations' dim tests and its padded shape test, the spacing
tests with and without symmetric, and the identical error tests of
count_annotations and pairwise_annotations are parametrized.
test_pairwise_annotations_self_loops is covered by
test_pairwise_annotations_small, which checks the same diagonal with and
without unique. Space-indented continuation lines in one pinned array are
tabs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parametrized: insert, substitute and substitute with start over a string
or one-hot motif; the one-hot motif and X error tests over the function,
each function now checked on all three bad inputs; multisubstitute over
spacing and motif form; dinucleotide_shuffle's composition tests over
the input and alphabet, and its identical-shuffles errors over the
sequence; and local_dinucleotide_shuffle over the alphabet size. Its
composition and bin-endpoint tests become one loop that requires the
same first cut to explain both, which is stronger than either. The CUDA
device test covers local_dinucleotide_shuffle, and the dinucleotide
shuffle case of the packed-alphabet test is covered by
test_dinucleotide_shuffle_packed_input.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd merge the rest

#125 added a brute-force reference for recursive_seqlets and tests of
its errors, tensor input and control thresholds. The older tests of
numpy input, the return_fdr and empty-control errors, the default FDR
thresholds, a control outside X's range, an unreachable target, control
dtypes, the p-value oracle, own-span p-values, sorting, coordinate
invariants and flanks are each covered by those (each was checked with a
break that both the old test and its replacement catch), and are
removed, along with the _span_p_values helper they used. The empty-X and
invalid-threshold cases join test_recursive_seqlets_errors with their
messages; the control, ignored-threshold and return-type tests join the
tests that made the same calls; and the two isotonic threshold tests are
one parametrized test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two lists of invalid calls become one parametrized test, dropping the
case both repeated. The deep_lift_shap tests with alpha and with alpha
and beta are parametrized, and the beta case now checks that the
attributions are scaled by beta rather than only that they differ; args
are sliced to the eight sequences attributed. The fp16 and bf16 tests are
one parametrized test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y in the function

The empty-X and empty-args tests of the two functions are parametrized
over the function, and the packed test with predict and with an unmarked
function is one test using check_packed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… covered ones

The numpy-annotations test joins the basic annotations test as a case and
gains its value checks, and the fp16 and bf16 tests are parametrized.
test_marginalize_returns_tuple_not_list is covered by the named-tuple
test. The ConvDense and Scatter tests at batch size 68 checked only
shapes; test_marginalize_convdense now compares batch size 68 with its
first call by value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed ones

The packed test with predict and with an unmarked function is one test,
whose int8 case now runs on the input with unknown characters. The
ConvDense and Scatter tests at batch size 68 checked only shapes;
test_space_convdense now compares batch size 68 with its first call by
value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three empty-X tests are parametrized over the function, all matching
"at least one example", and the X-on-device tests of insertion_effect and
deletion_effect are one test that also covers substitution_effect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e width or tooltip argument

The per-position color tests, the scalar colormap tests with and without
bounds, and interactive_logo's grid, despine, box line width and tooltip
CSS tests are parametrized, each case keeping the assertions of the test
it replaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tests drawing all-A sequences through a func and through
additional_func_kwargs are parametrized. test_screen_random_state_reproducibility
made the call that test_screen pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…move covered ones

The twenty tests that each attributed one Sequential model become
test_deep_lift_shap_sequential, the nine 1D max-pool tests and the two
2D ones become two parametrized tests, and the captum comparisons are
parametrized over model and batch size. Tests that made the same call to
check convergence and pinned values (the rule models, the transformer
blocks) make it once; the hypothetical rules test joins the raw-outputs
test that made both of its calls; the attention batch-size and
independence tests, the attention-pool value tests, the shared-module
tests and the success-path cleanup tests are each one parametrized test,
the last checking every attribute the cleanup removes.

Removed as covered: the LayerNorm and RMSNorm tests (rows of the rules
test with the same pins), the positional-is-linear test (implied by the
transformer-block pins), the ReLU row of the residual activation sweep,
two _disable_hooks tests, the ConvDense dense-wrapper test (bitwise equal
to the FlattenDense test, whose warning threshold rises to the wrapper's
1e-8), test_deep_lift_shap_dilated_conv (now a row of the shared-module
test), and test_deep_lift_shap_custom_op_overrides_builtin, whose
completeness loop never passed the custom op and whose remaining check
the local-IG pins imply. The commented-out captum test is deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jmschrei
jmschrei force-pushed the test_remove_redundant branch from 3942e92 to 6ba785e Compare October 19, 2026 19:12
@jmschrei
jmschrei merged commit 48056d4 into main Oct 19, 2026
4 checks passed
@jmschrei
jmschrei deleted the test_remove_redundant branch October 19, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant