Skip to content

Pin the model cache hydration image to one that distrusts eMudhra - #561

Merged
negz merged 1 commit into
modelplaneai:mainfrom
negz:pip-pip-hooray
Oct 22, 2026
Merged

negz merged 1 commit into
modelplaneai:mainfrom
negz:pip-pip-hooray

Conversation

@negz

@negz negz commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Description of your changes

The model cache hydration Job ran python:3.11-slim, whose pip 24.0 vendors a 2023 certifi that still includes the roots the compromised eMudhra intermediate CAs chain to. The Job pip installs huggingface_hub from PyPI with the cache's HF_TOKEN in its environment, then writes the weights serving pods load. Anyone able to forge a certificate for PyPI on that path could run code in the Job, taking the token and tampering with the weights.

This PR moves the Job to a python:3.13-slim pinned by digest, whose pip 26.2.2 distrusts the intermediates. The digest matters as much as the version: under a bare tag, a node that had already pulled an older image would keep running it.

I checked both images against the eMudhra test hosts. pip in today's python:3.11-slim completes the TLS handshake with dv-g3b-emudhra and dv-g3b-aaa-emudhra. In the pinned image, pip and hf download both fail with CERTIFICATE_VERIFY_FAILED. I didn't test the Rust hf_xet transfer path separately.

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

The hydration Job ran python:3.11-slim, whose pip 24.0 vendors a 2023
certifi that still trusts the compromised eMudhra intermediate CAs. The
Job pip installs huggingface_hub from PyPI with the cache's HF_TOKEN in
its environment, then writes the weights serving pods load. Anyone able
to forge a certificate for PyPI on that path could run code in the Job,
taking the token and tampering with the weights.

This commit moves the Job to a python:3.13-slim pinned by digest, whose
pip 26.2.2 distrusts the intermediates.

Signed-off-by: Nic Cope <nicc@rk0n.org>
Copilot AI balanced review requested due to automatic review settings October 22, 2026 17:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@negz
negz merged commit 0c236b3 into modelplaneai:main Oct 22, 2026
5 checks passed
@dennis-upbound

Copy link
Copy Markdown
Collaborator

/backport release-0.5

@github-actions

Copy link
Copy Markdown

Backport failed for release-0.5, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin release-0.5
git worktree add -d .worktree/backport-561-to-release-0.5 origin/release-0.5
cd .worktree/backport-561-to-release-0.5
git switch --create backport-561-to-release-0.5
git cherry-pick -x 3b8cba2bee651811df316735104235133615cb43

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants