PROJECT: QuantEcon benchmarking programme (code performance & execution)

[mmcky]

Scope

Code benchmarking and execution performance for QuantEcon computational work. This programme is deliberately focused: AI/LLM evaluation benchmarks (e.g. benchmark.translate-zh-cn) are a separate domain and out of scope here (naming conventions for the two families to be settled separately). Early focus is the lecture series — they are user-facing and where implementation choices are most visible — expanding later to library and infrastructure benchmarks.

Motivation

The review thread on QuantEcon/lecture-python.myst#717 (ge_arrow JAX conversion) produced a quantitative evaluation system for deciding whether an acceleration rewrite actually improves a lecture, and demonstrated why our current practice misleads: warm %timeit-style benchmarks showed the JAX version 1.4–4.8× faster, while replaying the lecture's actual call sequence in a fresh process (so compile time counts) showed it ~45× slower. Getting the measurement methodology right changes conversion decisions.

We already have two scoped benchmarking projects — #264 (kernel/backend suite) and #289 (lecture build monitoring) — plus repos created from them. This umbrella issue organises them into one programme with shared methodology, and adds the missing middle layer that #717 surfaced.

Structure — three workstreams

A. Kernel/backend suite — existing, #264. Library-level benchmarks (NumPy / Numba / JAX CPU / JAX GPU / Dask) across problem sizes, producing crossover curves ("GPU wins above size N for operation X"). This is the reference data layer. Repo: benchmarks.

B. Per-lecture implementation evaluation — new, from QuantEcon/lecture-python.myst#717. Given a lecture's actual problem sizes and call pattern, which implementation should it use — and did a conversion PR improve it? Deliverables: the evaluation scripts (being gathered by @xuanguang-li from the #717 thread), a reusable agent skill (working name /eval-py-acceleration) built with him, and the rubric documented in the manual (tracked in QuantEcon/QuantEcon.manual#104). Both the scripts and the skill live in the new QuantEcon/skills repository as a benchmark plugin, so the harness is versioned once and distributed everywhere (see #304 for the marketplace mechanism). This layer consumes workstream A's crossover data. Validation cases: lecture-python.myst#717 and lecture-python.myst#654.

C. Longitudinal monitoring — existing, #289. Per-lecture execution-time tracking across publish-* releases and environment.yml changes, with regression detection (e.g. the annual Anaconda upgrade cycle). Repo: tool-lecture-benchmark.

Shared foundations

• Measurement standard: the as-used, fresh-process methodology from #717 (actual problem sizes, actual call sequence, JIT compile time included) becomes the house standard across all three workstreams; warm/amortised numbers are reported alongside, never alone.
• Environment descriptors: every stored result records hardware, library versions, and runner type, so results are comparable across time and infrastructure changes.
• Results storage & reporting: one shared format/home for benchmark results (relates to Server & build telemetry — operational dashboard for runner health, build performance & cost #328 build telemetry and the Proposal: GitHub Projects as QuantEcon's PM + reporting layer (central dashboard) #332 dashboard direction).

Sequencing

1. Workstream B first (lectures are user-facing): gather the #717 scripts, build and validate the /eval-py-acceleration skill in QuantEcon/skills, document in the manual.
2. Extract the harness patterns from B to seed the workstream A suite.
3. Wire workstream C into CI for release/environment regression tracking.

Tasks

• Attach PROJECT: Performance Benchmarks Suite for Numba, Dask, JAX (CPU/GPU) #264 and Project: Lecture Performance Benchmarking & Monitoring System #289 as sub-issues of this umbrella
• Scaffold QuantEcon/skills as a Claude Code plugin marketplace with a benchmark plugin (skill + scripts)
• Workstream B: collect the evaluation scripts from the lecture-python.myst#717 thread into QuantEcon/skills (benchmark plugin)
• Workstream B: build /eval-py-acceleration and validate against lecture-python.myst#717 and lecture-python.myst#654
• Workstream B: manual documentation (tracked in QuantEcon/QuantEcon.manual#104)
• Roll out marketplace auto-install settings (extraKnownMarketplaces + enabledPlugins in checked-in .claude/settings.json) across the lecture repos once the skill stabilises
• Adopt the as-used measurement standard in a short written note all workstreams reference
• Define the shared result + environment-descriptor schema (commit, environment hash, runner class, hardware, library versions) — first blocking deliverable: telemetry reports, harness results, and kernel-suite results all write to it (extends Proposal: emit a machine-readable execution_status.json for lecture build/execution status #314; see the telemetry design comment below)
• Telemetry: post-build per-cell execution report from .jupyter_cache (nbclient already records metadata.execution by default) + PR-diff screening comment on preview builds
• Prose feature (separable): an {exec-time} role resolving measured cell durations into lecture text (Sphinx extension now, mystmd plugin post-migration)
• Revisit expansion beyond lectures (QuantEcon.py benchmarks, wasm comparison, runner/cost benchmarking)

Related

• [ge_arrow] Update to JAX and compare runtime lecture-python.myst#717 — origin thread (rubric + evaluation report)
• QuantEcon/QuantEcon.manual#104 — documentation tracking
• Proposal: A QuantEcon Claude Code toolkit (agents, skills, plugins) #304 — Claude toolkit / skills distribution (delivered by QuantEcon/skills)
• Repos: QuantEcon/skills, benchmarks, tool-lecture-benchmark

[mmcky]

Design update: an always-on telemetry layer, and how the three workstreams divide

Working through how the /eval-py-acceleration skill should extract and time lecture code surfaced a cheaper, always-on complement that reshapes the programme's architecture. Recording the design here.

Key discovery: per-cell timings are already being captured

nbclient — the executor under MyST-NB / Jupyter Book — records per-cell execution timestamps by default (record_timing = True, verified on nbclient 0.10.2 in our build environment). Every executed cell already carries metadata.execution timestamps (duration ≈ iopub.status.idle − iopub.execute_input), and jupyter-cache stores the executed notebook with this metadata intact. So every CI build is already generating per-cell timing data; nothing currently surfaces it.

The gap is provenance: the notebook only carries kernelspec/language_info — no hardware, runner class, library versions, or jax_enable_x64 state. Provenance must be attached at report time by the build workflow (commit SHA, environment.yml hash, runner class, CPU/GPU model, key library versions), which knows all of it.

Proposed telemetry pipeline

• Capture — already happening via nbclient (no work).
• Persist — a post-build step reads .jupyter_cache and emits a machine-readable execution report: per cell (label, duration) plus the provenance stamp. This is an extension of Proposal: emit a machine-readable execution_status.json for lecture build/execution status #314 from per-document status to per-cell timing — same artifact, richer schema.
• Compare — PR preview builds already execute changed lectures, so diffing cell timings against the last main build gives an automatic screening comment ("cell solve-ex4 went 8 ms → 950 ms") at zero extra execution cost. This would have flagged the regression in [ge_arrow] Update to JAX and compare runtime lecture-python.myst#717 the day the PR was opened.
• Monitor — reports accumulate into build history and execution statistics (this is workstream C's data source): slowest cells per repo, post-merge drift, before/after distributions for environment upgrades (turns the Anaconda-cycle validation sweeps into a query), runner-class cost attribution.
• Prose (future, separable) — a role like {exec-time} resolving a labelled cell's measured duration into lecture prose. {code-cell} :name: already provides the reference identity; MyST-NB precedents exist (nb-exec-table, glue). Needs a small Sphinx extension now and a mystmd plugin post-migration.

Division of labour across the workstreams

• A calibrates — kernel crossover curves on standardized code, stable over time, independent of lecture edits.
• B diagnoses — the skill: episodic, comparative, controlled. Two implementations of one lecture, same host, K replications, float32/float64 toggles, cold/warm decomposition, recompile audits. Telemetry cannot do counterfactuals — a build never runs two versions of a lecture — so the clinical harness stays.
• C observes — continuous, observational, accumulating; n=1 per build on shared runners, so it screens and trends but does not render verdicts.

Telemetry triggers and pre-loads B (a flagged cell hands the skill its as-used numbers for free); B's verdicts annotate C's history (explaining why a trend line stepped).

Sequencing consequence

The shared result + environment-descriptor schema is the first blocking deliverable — telemetry reports, harness results, and kernel-suite results should all write to it so the datasets join. Task list updated accordingly.

Caveats recorded

Telemetry is n=1 per build on shared runners — good for order-of-magnitude regressions, not fine verdicts. jupyter-cache mixes runner vintages for unchanged notebooks, so benchmark-grade comparisons need cache-busted runs or per-notebook provenance. Cell-level timing measures the whole cell, so keeping solves and plots in separate cells becomes measurement hygiene — arguably a style-guide point of its own.

Cheap first probe

A ~50-line script reading a CI .jupyter_cache and emitting the per-cell timing report for one repo, then diffing the lecture-python.myst#717 preview build against main to see whether the known regression falls out.

[mmcky]

project-compute is planning two GPU machines, and one of them is explicitly a benchmarking testbed — so this is an offer of hardware to workstream A, not a proposal about methodology.

What would be available. quantecon-gpu2 is two TITAN RTX cards (Turing sm_75, 24 GB each) that QuantEcon already owns — currently powered down and shipping to Wharton. quantecon-gpu1 is a workstation whose obsolete card is being replaced, with a used 24 GB Ampere recommended. Together that is two architecture generations on hardware we control, which is the shape workstream A's crossover curves want: "GPU wins above size N for operation X" has a different answer per generation, and one data point cannot show that.

Timing, honestly. Neither is available today. gpu2 needs freight and recommissioning; gpu1's card is recommended but unbought, with a platform review on 10 August. So this is "when you reach workstream A", not "now".

The question that would actually help us. Is owned hardware useful to this programme at all, or is the reference data only meaningful measured on the runners we actually build on? We would rather know before treating benchmarking as a reason to buy anything — project-compute has been removing planning that siblings already own, and we do not want to add hardware demand you have not asked for.

Worth noting the environment-descriptor schema already records hardware per result, so multi-generation results should fit the design rather than complicate it.