Skip to content

libstore: forward runtime substituters and trusted keys to the daemon - #11

Merged
domenkozar merged 4 commits into
cachix:devenv-2.35from
EclipseIsDead:remote-store-forward-runtime-substituters
Sep 23, 2026
Merged

domenkozar merged 4 commits into
cachix:devenv-2.35from
EclipseIsDead:remote-store-forward-runtime-substituters

Conversation

@EclipseIsDead

@EclipseIsDead EclipseIsDead commented Sep 10, 2026 •

Copy link
Copy Markdown

What happened

When running a devenv update, I built devenv shell compile devenv's own rust_devenv-* 2.3.1 crates, the 1.98.0 rustc toolchain and devenv-nixpkgs-patched for ~15 min. However, all paths returned HTTP 200 from https://devenv.cachix.org/<hash>.narinfo. devenv -v --no-tui eval cachix.pull showed the daemon rejecting the settings devenv does forward, and nothing at all about substituters:

warning: ignoring the client-specified setting 'always-allow-substitutes', because it is a restricted setting and you are not a trusted user
warning: ignoring the client-specified setting 'http-connections', because it is a restricted setting and you are not a trusted user
warning: ignoring the client-specified setting 'system', because it is a restricted setting and you are not a trusted user

IF the daemon received the substituters it would have printed ignoring untrusted substituter 'https://devenv.cachix.org' (daemon.cc, setSubstituters) but doesn't because the caches never leave the client process.

Verification

  • meson setup plus compiling the affected objects (remote-store.cc, store-api.cc, daemon.cc, nix_api_store.cc, uds-remote-store.cc, unix_daemon.cc, the libstore tests) with clang 21.1.8 on aarch64-darwin; clang-format clean.
  • Not yet run: the full test suite, and the end-to-end check that a devenv built against this branch makes an untrusted daemon print ignoring untrusted substituter 'https://devenv.cachix.org'. Draft until that is done.

🤖 Generated with Claude Code

Store::addSubstituter only appends to the client process's substituter
list, which serves client-side queries. Builds and substitutions run in
the daemon, whose Worker reads the daemon's own list, so a substituter
added at runtime through a RemoteStore never took part in any build.
Store::addTrustedPublicKeys overrode the setting but never re-sent the
options, so the daemon kept the old keys too.

Override both on RemoteStore: update the `substituters` /
`trusted-public-keys` settings and call setOptions() so the daemon
receives them. The daemon applies its usual policy, keeping the
substituter for trusted users or entries in `trusted-substituters` and
warning otherwise, which also gives untrusted users a visible signal.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bool RemoteStore::addSubstituter(const std::string & uri)
{
if (!Store::addSubstituter(uri))
return false;

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return early on dupe for the actual noop case i was hitting

@EclipseIsDead
EclipseIsDead marked this pull request as ready for review September 21, 2026 15:53
Retire pooled connections after runtime substituter or trusted-key
updates so subsequent operations use fresh daemon-side caches. Track a
settings generation to also retire connections returned after an update.
Forward removals and clearing to the settings used by new connections,
including substituter URIs with query parameters.

Add daemon-backed coverage with three pooled connections for cache
addition, removal, clearing, and trusted-key revocation and restoration.

Validation: build, runtime-substituters, binary-cache, remote-store,
and test-libstoreconsumer passed; formatting and diff checks passed.

Assisted-by: OpenAI Codex (GPT-6)
@domenkozar

Copy link
Copy Markdown
Member

@EclipseIsDead Could you review the follow-up fix in a0471df?

It retires stale pooled daemon connections after runtime cache or trusted-key updates, forwards removals and clearing, and adds daemon-backed regression coverage with three connections. The build and the runtime-substituters, binary-cache, remote-store, and test-libstoreconsumer tests passed locally.

This fix and review request were prepared with OpenAI Codex (GPT-6).

Snapshot the forwarded substituter setting per RemoteStore so updating one store cannot change the options sent by another. Remove the reference corresponding to the local substituter that was removed, including when cache URLs differ only by query parameters.

Retire pooled connections without acquiring a new handle, so updates return even when the pool is full. Extend the daemon-backed test to cover separate stores, duplicate references, and full pool capacity.

@EclipseIsDead EclipseIsDead left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left 2 comments, but overall makes sense to me and it's a clear improvement.

@domenkozar
domenkozar merged commit 2a2ff10 into cachix:devenv-2.35 Sep 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants