FIX: repair the GPU install stack and the build cache - #56
Merged
Conversation
Two independent breakages have kept every workflow in this repo red. Neither
was caused by any recent PR.
1. ci.yml installed torch from the cu128 nightly channel, which has been
frozen since April 2026. pip resolved torch to 2.12.0.dev20260408 while the
newest torchvision there pinned torch==2.12.0.dev20260407, a wheel since
pruned, giving ResolutionImpossible. Nothing imports torchvision or
torchaudio -- the only torch consumer is bayes_nonconj.md, using core tensor
ops -- so both are dropped and torch now comes from the stable cu128 index.
2. cache.yml is the sole producer of the build-cache artifact that ci, collab,
linkcheck and publish all download, and it has never produced one. It never
installed torch or pyro-ppl despite bayes_nonconj.md importing both, so its
Build HTML step failed during notebook execution. Both are added here.
Also fixed, and load-bearing:
- numpy<2. anaconda=2024.10 brings numba 0.60.0, which caps at numpy<2.1, and
9 lectures import numba. The unpinned `jax[cuda12-local]` upgrade resolved to
jax 0.11.0 -> numpy 2.5.1 and broke all of them. Pinning jax to 0.7.1 (the
last release accepting numpy>=1.26) is not sufficient on its own: "pymc<6"
resolves to 5.28.5, which requires pytensor>=2.38.2, which requires
numpy>=2.0. Verified with pip --dry-run: {arviz<1, pymc<6, kaleido<1} gives
numpy 2.4.6, while adding numpy<2 backtracks to pymc 5.25.1 / pytensor
2.31.7 / numpy 1.26.4. An assertion after the install makes drift loud.
- include-hidden-files on cache.yml's upload. _config.yml sets
execute_notebooks: "cache", so the execution cache lives in
_build/.jupyter_cache. upload-artifact has excluded hidden files by default
since v4.4, so the artifact would have shipped without the cache it exists
to carry.
- kaleido<1 in collab.yml. That job runs in the Colab container, which none of
the conda pins reach; it ships plotly 5.24.1 and no kaleido, so
back_prop.md's `!pip install kaleido` pulls an incompatible 1.x.
- The three ci.yml report uploads shared the artifact name execution-reports,
so the second collided with the first exactly when a build failed and the
report was most needed.
jax moves from cuda12-local to cuda12 (bundled CUDA), matching what lecture-jax
runs green on this same AMI. Note .github/runs-on.yml pins
ami-09baf66e396fa7cfd, which is lecture-jax's pre-CUDA-13 image -- the siblings
moved to CUDA 13 in November 2025 and we did not, so cu130 and jax[cuda13]
would be wrong here until the AMI is bumped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first smoke test (run 30974637413) got further than anything in a year -- torch, pyro-ppl, jax 0.7.1 and numpyro all installed cleanly, and numpy stayed at 1.26.4 -- then died in the pin step. "numpy<2" was too weak a constraint. arviz pulls xarray, current xarray wants a pandas newer than the conda-provided 2.2.2, pandas 3.x requires numpy>=2, and with numpy pinned below 2 pip had only pandas left to move. It walked 3.0.5 down to 2.1.0, reached an sdist, and the build failed: the numpy Cython headers require Cython>=3 and pandas 2.1.0 pins Cython<3. Fixed by constraining the conda-provided stack instead of just numpy, so pip backtracks on the arviz/xarray side where wheels exist. Verified with pip --dry-run --only-binary=:all: that holding numpy/pandas/scipy at the runner's actual versions (1.26.4 / 2.2.2 / 1.13.1, read out of the failed run's log) resolves wheel-only to arviz 0.23.4, pymc 5.25.1, pytensor 2.31.7, xarray-einstats 0.9.1, kaleido 0.2.1, leaving all three conda packages untouched. The constraints file is generated from the live environment rather than hardcoded, so bumping anaconda in environment.yml cannot silently invalidate it, and it skips any package that is absent rather than failing. --only-binary=:all: turns a repeat of this failure into an immediate clear error instead of a five-minute compile that ends in a Cython traceback. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 30975106893 executed 24 of 25 lectures successfully, including the 70-minute bayes_nonconj and back_prop's kaleido image export. Exactly one failed, and the execution report -- the first ever retrieved from this repo -- gives the cause: AttributeError: module 'wcwidth' has no attribute 'width' prettytable/prettytable.py:3080 in _str_block_width prettytable declares "Requires-Dist: wcwidth" with no lower bound. pip therefore accepts the wcwidth 0.2.x that anaconda ships as already satisfying it, and installs the latest prettytable. prettytable 3.18.0 switched from wcswidth() to wcwidth.width(), which does not exist before wcwidth 0.3.0. Verified the boundary against the wheels: 3.17.0 uses wcswidth(), 3.18.0 uses wcwidth.width(); wcwidth 0.2.14 exports (wcwidth, wcswidth, list_versions) and 0.3.0 adds width(). That is an upstream packaging bug, and it fires on any machine whose wcwidth predates 0.3.0 -- which is why prob_matrix.md and prob_meaning.md have been able to break without either lecture changing. Pinning prettytable rather than upgrading wcwidth deliberately: prettytable is used by two lectures, while wcwidth is shared with prompt_toolkit and IPython. This keeps the blast radius at the package that is actually wrong. Pre-installing it here also makes the notebooks' own `!pip install prettytable` a no-op. Verified the full pin set still resolves wheel-only with the conda stack held. Not addressed here, and deliberately: linearmodels and interpolation are still installed unpinned at notebook time, where the constraints file does not reach them. Both are fine against today's releases. Narrowing that gap is a separate change; over-broad pinning is what broke run 30974637413. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
✅ Deploy Preview for timely-seahorse-68815c ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
Pull request overview
This pull request updates the repository’s GitHub Actions workflows to restore green builds by fixing GPU/JAX/PyTorch installation breakages and ensuring the build cache artifact is correctly produced and reusable across downstream workflows.
Changes:
- Switch PyTorch installation away from the stale nightly CUDA index and remove unused torchvision/torchaudio installs.
- Ensure
cache.ymlinstalls required runtime deps (torch + pyro-ppl) and uploads a usable execution cache by including hidden files in the artifact. - Stabilize the scientific Python stack used during notebook execution via a constraints-based approach, and avoid artifact name collisions in
ci.yml.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| .github/workflows/ci.yml | Moves torch install to stable CUDA index, pins/constraints key scientific deps, and disambiguates execution-report artifact names. |
| .github/workflows/cache.yml | Installs missing notebook runtime deps (torch/pyro), pins/constraints scientific deps, and uploads build-cache including hidden cache dirs. |
| .github/workflows/collab.yml | Pins kaleido<1 in the Colab container to prevent notebook-time installs from pulling incompatible versions. |
| .github/workflows/publish.yml | Aligns publish workflow’s GPU install stack with CI/cache (torch/pyro + pinned JAX + constraints). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
23
to
+27
| - name: Install JAX, Numpyro | ||
| shell: bash -l {0} | ||
| run: | | ||
| pip install --upgrade "jax[cuda12-local]" | ||
| pip install numpyro | ||
| # CUDA 12 stack. .github/runs-on.yml pins ami-09baf66e396fa7cfd, a CUDA 12 | ||
| # AMI, so do not move to cu130 / jax[cuda13] without bumping the AMI first. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repairs the two breakages that have kept every workflow in this repo red, and verifies the fix by running
cache.ymlto green on this branch — the first successful build in the retained history (run 30980194673, 25 lectures executed, 47 MBbuild-cacheartifact produced).The two root causes
1.
ci.ymlinstalled torch from a frozen nightly channel.pip install --pre torch torchvision torchaudio --index-url .../nightly/cu128resolved torch to2.12.0.dev20260408while the newest torchvision there pinnedtorch==2.12.0.dev20260407, a wheel since pruned —ResolutionImpossible. The channel has been stale since April 2026. Nothing in the book imports torchvision or torchaudio; the only torch consumer isbayes_nonconj.mdusing core tensor ops. Dropped both, moved to the stable cu128 index.2.
cache.ymlnever installed torch or pyro-ppl, althoughbayes_nonconj.mdimports both. It is the sole producer of thebuild-cacheartifact thatci,collab,linkcheckandpublishall download, so its failure during notebook execution is why all four have been failing at the download step. Nobuild-cacheartifact has ever existed in this repo.What else had to be fixed to get there
Each of these was found by running the build, not by reading it.
prettytable<3.18. The single lecture that failed on the first green-install run wasprob_matrix.md, withAttributeError: module 'wcwidth' has no attribute 'width'. prettytable declaresRequires-Dist: wcwidthwith no lower bound, so pip accepts anaconda's wcwidth 0.2.x as satisfying it, then installs prettytable 3.18.0 — which switched fromwcswidth()towcwidth.width(), added only in wcwidth 0.3.0. Verified against the wheels: 3.17.0 useswcswidth(), 3.18.0 useswcwidth.width(). Pinned prettytable rather than upgrading wcwidth, because prettytable is used by two lectures while wcwidth is shared with prompt_toolkit and IPython.Holding the conda stack. An earlier attempt pinned only
numpy<2. That was too weak: arviz pulls xarray, current xarray wants a pandas newer than conda's 2.2.2, pandas 3.x requires numpy>=2, so pandas became the only thing pip could move — it walked 3.0.5 down to 2.1.0, hit an sdist, and the build died because the numpy Cython headers need Cython>=3 while pandas 2.1.0 pins Cython<3. Now a constraints file generated from the live environment holds numpy/pandas/scipy, so pip backtracks on the arviz/xarray side where wheels exist.--only-binary=:all:turns any repeat into an immediate clear error rather than a five-minute compile.include-hidden-files: trueon the cache upload._config.ymlsetsexecute_notebooks: "cache", so the execution cache lives in_build/.jupyter_cache— hidden.upload-artifacthas excluded hidden files by default since v4.4. Without this the artifact ships with neither.jupyter_cachenor.doctrees, and every consumer re-executes the whole book. Confirmed present in the produced artifact.jax[cuda12]==0.7.1. Moved offcuda12-local, which expects a system CUDA install, to the bundled-CUDA variant that lecture-jax runs green on this same AMI. Pinned to 0.7.1 because it is the last release acceptingnumpy>=1.26; unpinned it resolves to 0.11.0, which drags numpy to 2.5.1 and breaks numba 0.60.0 — imported by 9 lectures.kaleido<1incollab.yml. That job runs in the Colab container, which none of the conda pins reach; it ships plotly 5.24.1 and no kaleido, soback_prop.md's!pip install kaleidowould pull an incompatible 1.x.Unique artifact names in
ci.yml. Three report uploads shared the nameexecution-reportsin one job, so the second collided with the first exactly when a build failed and the report was most needed. That collision is why the original failure was never diagnosable.A note on CUDA 12 vs 13
.github/runs-on.ymlpinsami-09baf66e396fa7cfd. That is lecture-jax's pre-CUDA-13 AMI — the siblings moved toami-0edec81935264b6d3in November 2025 and this repo did not, it kept the same image name. Socu130andjax[cuda13], which the sibling repos now use, would be wrong here until the AMI is bumped. Comments in the install blocks record this.What this does not fix
ci.ymlandcollab.ymlwill stay red on this PR. They fail at the artifact download, which cannot succeed until a greencache.ymlhas run onmain— the consumers all pinbranch: main, andcache.ymlhas nopushtrigger. The sequence after merge isgh workflow run cache.yml --ref main, then everything downstream unblocks.Two builders remain unexercised, since
cache.ymlonly builds HTML: thesphinx-tojupyternotebook builder andpdflatex. Both were tested offline against the pinned toolchain and pass — pdflatex produces a 385-page PDF with zero sphinx warnings — but neither has run in CI.publish.ymlhas never run at all.🤖 Generated with Claude Code