Generate and publish a validation summary

[roshan2004]

What

The validation work is strong but invisible: the suite cross-checks geometry/RMSD/contacts vs MDAnalysis, bonds/chemistry/descriptors vs RDKit, secondary structure vs mkdssp, pocket interactions vs PLIP, plus a tier of physical invariants — yet the results only ever landed in the CI log. This makes them legible.

A small pytest plugin records the pass / skip / fail outcome of every validation check (including whole modules skipped because a reference tool is absent) and, when run with --validation-summary-dir=DIR, writes:

• validation-summary.md — a table of area · reference · passed/skipped/failed, with per-check detail for anything skipped or failed
• validation-summary.json — the same data, machine-readable

uv run pytest tests/validation -v -rs -s --validation-summary-dir=.

Generated, so it can't drift

Every count comes from the run itself — the summary is produced by the tests, not authored alongside them. Only the human-readable area labels live in tests/validation/_summary.py; new *_ref modules are picked up automatically with a fallback label.

Visible in CI

The validation job now:

• runs with --validation-summary-dir=.,
• cats the Markdown into $GITHUB_STEP_SUMMARY so it renders on the workflow run page (Summary tab), and
• uploads both files as the validation-summary artifact (if: always(), so a failing check still publishes — you see exactly which one).

That run, with all reference tools installed, is the authoritative snapshot of what is currently cross-checked.

Example (generated locally)

**121 checks: 112 passed, 9 skipped, 0 failed.**

| Area | Reference | Passed | Skipped | Failed |
| Geometry, RMSD, distances, inertia | MDAnalysis | 10 | 0 | 0 |
| Bond perception from geometry | RDKit | 19 | 0 | 0 |
| Secondary structure (simplified DSSP) | mkdssp | 6 | 0 | 0 |
| ... |

Design notes

• Plugin, not a new dep: hooks pytest_runtest_logreport (per-test pass/fail/skip) and pytest_collectreport (whole modules skipped when a tool is missing). No pytest-json-report dependency.
• Inert by default: without --validation-summary-dir the plugin records nothing user-visible and writes no files, so normal pytest runs are unchanged.
• In docs vs artifact: the always-fresh CI artifact + run-page summary is the source of truth (no committed snapshot to go stale); docs/validation.md gains a "Generated summary" section explaining and linking it.

Tests & docs

• tests/validation/test_summary_generator.py — 6 unit tests for the pure aggregation/formatting (runs in the normal suite; no reference tools needed).
• Generated locally end-to-end (112 passed / 9 skipped) to confirm the plugin and skip-reason capture work.
• Full suite: 932 passed, 21 skipped (no regressions); ruff clean; ci.yml validated.
• Docs: docs/validation.md "Generated summary" section, README "Scientific validation" footer, CHANGELOG.

[read-the-docs-community]

Documentation build overview

📚 molscope | 🛠️ Build #33073004 | 📁 Comparing b204402 against latest (3bf3ea1)

  🔍 Preview build  

1 file changed

± validation/index.html

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!