Official support for Spegel

[awoimbee]

Related to #1963, #3314

What I'd like:

I'd like bottlerocket to completely, officially, support Spegel.

Why

Spegel is currently kinda supported

See comment from Bottlerocket maintainer:

Indeed, it's the compatibility concern with Spegel that's kept this default in place.

Spegel should be supported

Bottlerocket goes hand in hand with karpenter for fast cluster auto-scaling, the exact use-case that Spegel is tailored to.

Currently, using Spegel on bottelrocket is painful

See: spegel-org/spegel#47 (comment)
To make Spegel (and alternatives) work on Bottlerocket, we have to resort to workarounds that can break when we upgrade.

The best workaround (writing to /.bottlerocket/rootfs/etc/containerd/certs.d/_default/hosts.toml) doesn't work on Bottlerocket 1.51. Now we can set container-registry.mirrors but we can't configure capabilities anymore.

Alternatives

Personally I use Trow, which works just like Spegel from containerd's perspective, but it can resolve images, so I can do without capabilities.
Still, the config required is messy:

const setupContainerdMirrorScript = outdent`
  #!/bin/sh
  set -xeuo pipefail
  IMDS_TOKEN="$(curl -s -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 360" "http://[fd00:ec2::254]/latest/api/token")"
  IP="$(curl -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" http://[fd00:ec2::254]/latest/meta-data/${isIpv6Cluster ? "ipv6" : "local-ipv4"})"
  apiclient set --json "{\\"container-registry\\":{\\"mirrors\\":{\\"*\\":[\\"http://\${IP}:${TROW_PORT}\\"]}}}"
`;
userDataSections.push(outdent`
  [settings.bootstrap-containers.containerd-trow-proxy]
  mode = "always"
  essential = true
  user-data = "${Buffer.from(setupContainerdMirrorScript).toString("base64")}"
`);

(the IP thing is necessary on IPv6 clusters as [::1]:${TROW_PORT} -same for 127.0.0.1- isn't routed to the Trow service)

[tim-mcdonald]

Hi team, I wanted to follow up on this and ask for a response regarding the path forward for Spegel (and alternatives).

Currently, the Bottlerocket containerd configuration relies on the old, deprecated method of configuring registry mirrors (registry.mirrors and registry.configs in the main TOML), rather than the modern config_path pattern (using hosts.toml files).

See the deprecation notice and the hosts.toml directory pattern here: https://github.com/containerd/containerd/blob/main/docs/hosts.md https://github.com/containerd/containerd/blob/main/docs/cri/registry.md

Spegel requires the ability to configure a registry mirror (pointing to itself) via the config_path directory (usually /etc/containerd/certs.d), as well as setting the capabilities field to only pull. Because Bottlerocket uses the legacy config method, there is no supported way to configure containerd for Spegel.

In prior releases, I was able to workaround this by using a shell script inside a bootstrap container to modify the configuration at boot. However, with the enforcement of the immutable root file system, this "loophole" is effectively closed, and there is no longer a viable workaround for users to configure this themselves.

Could you please advise if there are plans to migrate Bottlerocket's default containerd configuration to the supported config_path pattern? This would natively enable tools like Spegel without requiring "break-glass" modifications to the OS.

[bcressey]

Could you please advise if there are plans to migrate Bottlerocket's default containerd configuration to the supported config_path pattern?

I have some work in flight to migrate to the new hosts.toml approach. One of the constraints is preserving settings API compatibility so that user-data that includes the current settings.container-registry.* values continues to work. Credentials in particular are a headache.

I don't want to add a conditional fall back to the old approach since hosts.toml is also required for transfer service support, which is required for image verifier plugins to run.

I expect it to be a two-part migration where variants move over to hosts.toml first, and then once we're confident we don't need to roll that back, we can start adding new fields such as capabilities under settings.container-registry.*.

[tim-mcdonald]

Phase 1 has landed (thank you!) and we've upgraded to Bottlerocket 1.56.0. The hosts.toml migration works well and the credential handling is solid.

We're still blocked on Phase 2: configurable capabilities. Our Spegel setup requires capabilities = ["pull"] only, but thar-be-registries currently hardcodes ["pull", "resolve"] for all mirrors. With containerd 2.1's transfer service, resolve capability causes hard failures when Spegel's DHT isn't fully initialized on a new node (rather than the graceful fallback behavior in containerd 2.0).

Is Phase 2 on the near-term roadmap?

[gyanesh-mishra]

Adding a data point: we got Spegel working on EKS 1.34 AL2023 nodes (containerd 2.1.5) but not on Bottlerocket, due to a missing containerd setting.

The key finding: containerd 2.1 defaults to use_local_image_pull = false, which routes CRI image pulls through the transfer service. The transfer service does not read hosts.toml mirror configuration from config_path. Only the CRI local image pull path honors hosts.toml.

Setting use_local_image_pull = true in config.toml fixes it — Spegel immediately starts receiving mirror requests:

[plugins.'io.containerd.cri.v1.images']
use_local_image_pull = true

On AL2023 we sed this into /etc/containerd/config.toml during pre-bootstrap. On Bottlerocket this isn't possible because:

1. /etc/containerd/config.toml is read-only
2. use_local_image_pull is not exposed in [settings.kubernetes] TOML API

Request: Expose use_local_image_pull as a Bottlerocket setting (e.g. settings.kubernetes.use-local-image-pull) so Spegel mirrors can function on Bottlerocket with containerd 2.1+.

[tim-mcdonald]

Following up with some code-level evidence, because I think one detail in the thread above sends us in the wrong direction.

@gyanesh-mishra reported that the transfer service does not read hosts.toml from config_path. I went looking for that in containerd and I do not think it holds. registry.WithHostDir(c.config.Registry.ConfigPath) is present in internal/cri/server/images/image_pull.go at v2.1.0, v2.1.5, v2.1.6 and v2.2.0, so it was there in the 2.1.5 that was tested.

It is also not a dead call. NewOCIRegistry turns it into host options:

hostOptions := config.HostOptions{}
if ropts.hostDir != "" {
    hostOptions.HostDir = config.HostDirFromRoot(ropts.hostDir)
}

MarshalAny ships it across the transfer API as res.HostDir = r.hostDir, and the server side UnmarshalAny rebuilds it:

if s.Resolver.HostDir != "" {
    hostOptions.HostDir = config.HostDirFromRoot(s.Resolver.HostDir)
}

So the transfer service does load certs.d. The observation that setting use_local_image_pull = true makes Spegel start receiving requests is still real, but the mechanism must be something other than config loading, and I suspect it is the error handling difference I described in March: resolve against a Spegel instance whose DHT has not initialized fails hard under the transfer service where containerd 2.0 fell back to the next host.

If that is right, then exposing use_local_image_pull is a workaround that routes around the failure, and configurable capabilities are the actual fix, since capabilities = ["pull"] means containerd never asks Spegel to resolve in the first place. That is what bottlerocket-os/bottlerocket-settings-sdk#134 and bottlerocket-os/bottlerocket-core-kit#904 implement. Both are still open. I have just rebased the core-kit one onto v16.3.0, since it had picked up a conflict with the port inference changes in #976.

I would appreciate a maintainer read on those two when there is time. Both are backward compatible: when capabilities is unset, consumers keep the current ["pull", "resolve"] behavior.

One more thing worth noting, since it bit us and is independent of all of the above. A wildcard mirror silently stops applying to any registry that also has a credentials entry, because write_credentials_toml creates certs.d/<registry>/ with only a credentials.toml in it, and containerd picks a host directory by stat'ing the directory rather than looking for hosts.toml, so _default is never reached. In our case that meant Spegel appeared to work for ghcr.io and quay.io while never serving anything from our private registry. I have filed that separately against bottlerocket-core-kit.

For what it is worth, the extension mechanism for all of this already exists: imports = ["/etc/containerd/config.d/*.toml"] is in the shipped containerd config template. Every ask in this thread is binding a setting to a config value the OS already supports, not new machinery. The old workaround of writing that file from a bootstrap container is closed now that /etc is tmpfs and bootstrap containers are labelled control_t, which is correct, but it does leave users with no path at all until one of these lands.