Skip to content

Package plugin test suites + a [tests] extra so downstream builders can run them (lehrer plugin-regression) #830

Description

@blarghmatey

Summary

Ship each maintained ol-openedx-* plugin's test suite inside its built
distribution
and declare a tests extra bundling its test-only dependencies,
so the tests can be installed and run against a real, already-built Open edX
platform via pip install "<dist>[tests]==<version>".

Why

lehrer added a plugin-regression runner (dagger call platform plugin-regression, lehrer#111)
that runs each installed plugin's own pytest suite inside a built cell image
(the real edx-platform version + the deployment's pinned plugin set + aqueduct
settings). It uses pytest discovery: it runs whatever tests are installed in
the image, resolving packages at runtime.

Today it discovers nothing, because:

  • published plugin wheels/sdists ship no test files (verified: the
    ol-openedx-logging==0.3.5 sdist contains only settings/test.py, no
    tests/), and
  • this monorepo has no per-package tags, so there's no way for a downstream
    builder to fetch "the tests at the exact installed version."

lehrer already installs "<dist>[tests]==<pinned-version>" opportunistically —
which is a safe no-op until the extra exists — so the moment a plugin ships its
tests + a tests extra, its suite starts running in that pipeline with no
lehrer change required
.

What to do (per maintained plugin)

  • Include the plugin's tests/ in the built distribution (package data /
    don't exclude tests), so pip install <dist> lands the tests in
    site-packages where pytest --pyargs <module> can discover them.
  • Declare a tests optional-dependency extra with the test-only deps the
    suite needs (pytest-mock, responses, factory_boy, etc.), matching what
    run_edx_integration_tests.sh installs today.
  • Confirm the suite runs under --ds=lms.envs.test (or the plugin's own
    standalone settings) against an installed edx-platform.

Roll out incrementally — one plugin unblocks real signal for that plugin.

Verify

Once at least one plugin ships the extra, from a lehrer checkout:

lehrer build plugin-regression --cell mit-ol/master/mitxonline \
  --custom-settings ./deployments/mit-ol/settings

should collect and run that plugin's suite (instead of "no tests discovered").

Tracked in the lehrer CI-verification project; companion CI-wiring task is
blocked on this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions