Repository navigation
Pin the model cache hydration image to one that distrusts eMudhra - #561
Merged
Merged
Conversation
The hydration Job ran python:3.11-slim, whose pip 24.0 vendors a 2023 certifi that still trusts the compromised eMudhra intermediate CAs. The Job pip installs huggingface_hub from PyPI with the cache's HF_TOKEN in its environment, then writes the weights serving pods load. Anyone able to forge a certificate for PyPI on that path could run code in the Job, taking the token and tampering with the weights. This commit moves the Job to a python:3.13-slim pinned by digest, whose pip 26.2.2 distrusts the intermediates. Signed-off-by: Nic Cope <nicc@rk0n.org>
haarchri
approved these changes
Oct 22, 2026
Collaborator
|
/backport release-0.5 |
|
Backport failed for Please cherry-pick the changes locally and resolve any conflicts. git fetch origin release-0.5
git worktree add -d .worktree/backport-561-to-release-0.5 origin/release-0.5
cd .worktree/backport-561-to-release-0.5
git switch --create backport-561-to-release-0.5
git cherry-pick -x 3b8cba2bee651811df316735104235133615cb43 |
Closed
3 of 4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
The model cache hydration Job ran
python:3.11-slim, whose pip 24.0 vendors a 2023 certifi that still includes the roots the compromised eMudhra intermediate CAs chain to. The Job pip installshuggingface_hubfrom PyPI with the cache'sHF_TOKENin its environment, then writes the weights serving pods load. Anyone able to forge a certificate for PyPI on that path could run code in the Job, taking the token and tampering with the weights.This PR moves the Job to a
python:3.13-slimpinned by digest, whose pip 26.2.2 distrusts the intermediates. The digest matters as much as the version: under a bare tag, a node that had already pulled an older image would keep running it.I checked both images against the eMudhra test hosts. pip in today's
python:3.11-slimcompletes the TLS handshake withdv-g3b-emudhraanddv-g3b-aaa-emudhra. In the pinned image, pip andhf downloadboth fail withCERTIFICATE_VERIFY_FAILED. I didn't test the Rusthf_xettransfer path separately.I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.