Skip to content

Publish the v1 container images with each release #13516

Open
7 of 8 issues completed
Open

Description

@cv

Problem

No v1 image is published. Every agent, runtime and proxy image is built locally (docs/build.md), and the examples reference locally built digests (docs/models.md: "refers to a locally built artifact, not a published registry image"). The default agent pin in versions.json is a local name (nc-fabric-owner-20260924@sha256:…). Kubernetes users push agent images to their own registry.

#12638 requires that a user can follow the first-deployment guide without building from source, which needs published images, but it lists only the CLI bundle and the SDK and provider packages. #12016 covers only the agent base images that downstream blueprints derive from.

What changed recently:

Proposal

Publish to ghcr.io/nvidia/nemoclaw/…, where NemoClaw v0 and OpenShell already publish, from a release workflow.

Scope:

Image Bake target Platforms
Agent images agents All ten harnesses on linux/arm64 and linux/amd64 (#13519)
vLLM and Ollama runtimes vllm, ollama linux/arm64 and linux/amd64, each built natively
Ollama proxy ollama-proxy linux/arm64 and linux/amd64
Reference agent, if derived-image testing needs it dummy linux/arm64 and linux/amd64

Build:

  • one job per architecture on native runners (GitHub's arm64 and amd64 runners), running cargo images build for agents and docker buildx bake --push for the others, with packages: write and the workflow's token;
  • a final job joins each image's platform digests into one index with docker buildx imagetools create;
  • only release tags push; pull requests keep building and qualifying without pushing, as images.yml does.

Names: new names that don't collide with v0's (sandbox-base, openclaw-sandbox, …): for example agent-openclaw, vllm, ollama, ollama-proxy, or a v1/ prefix.

Supply chain:

Distribution review: the images redistribute vLLM, Ollama, OpenClaw, Hermes, Fabric and their dependencies. Their notices are retained (image/NOTICE.md, image/fabric/HERMES-NOTICE.md, image/vllm/NOTICE.md, image/ollama/NOTICE.md), but publishing needs a license review. The AGPL recipe is gone (#13506).

Staging before release

Public v1 releases wait for the installer and onboarding, validated by QA and product (#12638). Until then, stage everything privately so the release itself is a final switch:

  1. Push privately. A manually triggered workflow builds each architecture natively, pushes to new ghcr.io/nvidia/nemoclaw/… packages, and joins each image's platform digests into one index. A new container package is private by default.
  2. Draft the release. The workflow creates a draft release listing every image digest, and later the bundle. Only users with push access see drafts.
  3. Test from the draft. Deployments pull the private images with a token through spec.registries on Docker and imagePullSecrets on Kubernetes (Pull images from private registries #12709), locally and on kind.
  4. Release. Publish the draft and make the packages public. Making a package public can't be undone, and the organization's package-creation policy may need an admin to allow public packages.

Checked on 2026-10-20 with a throwaway push from a personal token (ghcr.io/nvidia/nemoclaw/visibility-probe, labeled with org.opencontainers.image.source=https://github.com/NVIDIA/NemoClaw, deleted afterwards):

  • Visibility: the new package was internal, visible to members of NVIDIA's enterprise. Anonymous pulls were refused (403).
  • Repository link: the label didn't link it to the repository (repository: null).

The v0 packages, which Actions workflows published, are linked to this repository and public. Whether a package published by the workflow's own token starts internal or inherits the repository's visibility is still unchecked. The first private dry run will show it, and the workflow should check the package's visibility after its first push.

Decisions

  • Names (decided): ghcr.io/nvidia/nemoclaw/v1/<image>, for example v1/openclaw, v1/vllm, v1/ollama, v1/ollama-proxy. The reference agent isn't published: it exists for tests, and derived images build FROM the real agent images (Publish base agent images per v1 release for derived sandbox images #12016).
  • Tags (decided): semver release tags, starting at v0.1.0-rc1: the v1 branch ships 0.1.x releases (ci: tag v1-branch releases as v0.1.x #13559). Version tags are permanent, because a repository rule forbids deleting them; v1.0.0-rc1 was staged under the earlier scheme and remains, though nothing reacts to it. Pushing a tag runs the staging workflow from the tagged commit; GitHub only runs manually triggered workflows from the default branch, which is v0's main.
  • Still open: how a release references its images. Either digests listed in the release notes only, or also pinned in the bundle, as versions.json pins OpenShell's images. This can wait for the installer.

Follow-ups: #13521 (provenance attestations), #13522 (SBOMs), #13523 (vulnerability scanning), #13524 (license review).

Done when

  • Published: a release publishes each image above as a signed index for its platforms, with an SBOM, on ghcr.io/nvidia/nemoclaw.
  • Documented: the release notes list every image digest; docs/build.md, docs/derived-images.md (Publish base agent images per v1 release for derived sandbox images #12016) and the examples point to the published images, keeping local builds as the alternative.
  • Usable without building: a first deployment from the guide applies with published images only.
  • Default: the default agent pin in versions.json names a published, registry-qualified digest instead of the local nc-fabric-owner-… name.
  • Derivable: a reference derived image built FROM a published base agent image applies by digest (from Publish base agent images per v1 release for derived sandbox images #12016).

Activity

  1. added
    v1NemoClaw v1 branch
    on Oct 20, 2026
  2. cv commented on Oct 20, 2026

    @cv
    CollaboratorAuthor

    Tested v1.0.0-rc1 from the draft on a DGX Spark (linux/arm64), with a CLI bundle built at the tagged commit and the private images pulled from ghcr.io with a token.

    Docker: works.

    • Images: the managed Hermes example ran with v1/ollama as the model server and v1/hermes as the agent.
    • Pulls: NemoClaw's Docker provider pulled the runtime image through spec.registries and the host's docker login. An anonymous pull is refused.
    • Result: Hermes ran uname -m with its terminal tool and replied aarch64.

    Kubernetes (kind): works, with workarounds.

    • Pulls: the first apply failed with ImagePullBackOff until ghcr-credentials existed in the namespace NemoClaw created, as documented. Then the cluster pulled v1/deepagents and the sandbox ran.
    • Prerequisite: Agent Sandbox v0.5.0 had to be installed on the cluster first, as documented.

    Gaps for publishing:

    1. Docker plan doesn't pull agent images. It reads the catalog label from the local engine only, so a first deployment fails with "image runtime metadata is unavailable" until the user runs docker pull on the agent image. Guides built images locally until now, so this never showed. Either plan pulls or reads the remote config, or the docs say to pull first.
    2. Kubernetes metadata bundles need the repository. image.metadata bundles come only from cargo images export-metadata, which isn't in the CLI bundle. It also can't export from a pulled image: docker image save of the pulled index omits the platform config and layers ("image archive lacks a metadata blob"). For the test I assembled the bundle from the registry's exact bytes. A release should publish a metadata bundle per image, or the CLI should build one from the registry.
    3. Kubernetes rejects multi-platform indexes. crates/nemoclaw-fabric/src/image_metadata.rs requires an index to have exactly one Linux child, "until per-platform runtime contracts are modeled". The published v1/<image> indexes cover arm64 and amd64, so a Kubernetes sandbox must reference a platform manifest digest instead. Either the release lists per-platform digests too, or NemoClaw models per-platform contracts.

    Everything was cleaned up afterwards: the deployments, the kind cluster, the pulled images, and the registry login.

  3. 1 remaining item

  4. added
    area: ciCI workflows, checks, release automation, or GitHub Actions
    area: packagingPackages, images, registries, installers, or distribution
    on Oct 27, 2026
  5. added theissue type on Oct 27, 2026
  6. added and removed on Nov 17, 2026
  7. added this to the Roadmap: GTC DC 2026 milestone on Nov 17, 2026
  8. removed this from the Roadmap: GTC DC 2026 milestone on Nov 17, 2026
  9. added this to the Roadmap: GTC DC 2026 milestone on Nov 18, 2026
  10. cv commented on Nov 18, 2026

    @cv
    CollaboratorAuthor

    Status as of 2026-11-18, with updated acceptance criteria. Not ready to close.

    Done

    No longer applies

    Updated acceptance criteria

    • Signed: each published index has a build provenance attestation (Attest build provenance for the published v1 images #13521).
    • Redistributable: the license review in Review licenses for redistributing the v1 images #13524 is complete for every image.
    • Public: a release publishes the draft and makes the v1/… packages public (needs an org admin if the package policy requires it).
    • Documented: the release notes list every image digest; docs/build-deploy/build.md, docs/build-deploy/derived-images.md and the examples point to the published images, keeping local builds as the alternative.
    • Usable without building: a first deployment from the guide applies with published images only, on Docker and kind.
    • Derivable: a derived image built FROM a published agent image applies by digest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area: ciCI workflows, checks, release automation, or GitHub Actionsarea: packagingPackages, images, registries, installers, or distributionv1NemoClaw v1 branch

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions