Native Slurm worker pool with a Flox-prepared runtime

Kubernetes owns the workers. Slurm owns the work. Flox owns what's inside the workers.

This repository is a local reference implementation of that architecture.

Kubernetes provides and replaces the Slurm worker pods. Slinky's Slurm Operator runs the Slurm control plane and a native slurmd worker pool inside Kubernetes. Users continue to submit ordinary Slurm work with sbatch, srun, salloc, job arrays, dependencies and MPI. Flox prepares the worker runtime before slurmd starts, then a small Slurm TaskProlog carries the intended Flox task environment across Slurm's own task boundary.

There is no per-job translation from Slurm to Kubernetes. Kubernetes sees long-lived Slurm infrastructure and worker pods. Slurm sees native compute nodes and owns job scheduling, allocation and task launch on them.

The implementation is intentionally split along three ownership boundaries:

Kubernetes owns worker lifecycle. The compute NodeSet creates, replaces and reports the slurmd pods.
Slurm owns workload lifecycle. Slurm accepts jobs, allocates nodes, launches tasks, propagates credentials and runs MPI in the normal Slurm model.
Flox owns the worker software runtime. The Flox containerd shim realizes and activates a generation-pinned FloxHub environment inside the Slurm worker container. The immutable Slinky image still supplies the base filesystem and Slurm binaries.

The QEMU/KVM cluster, kubeadm installation, NFS server and test identity are local scaffolding for exercising this architecture end to end. They are not requirements of the architecture itself and are the obvious integration points to replace with a customer's existing Kubernetes, storage and identity systems.

Architecture at a glance

        user
         |
         |  ssh: sbatch / srun / salloc
         v
   +-------------+
   |  LoginSet   |  sackd + sshd
   +------+------+
          |
   +------v------+
   |  slurmctld  |  scheduling and allocations
   +------+------+
          |
     +----+----------------------+
     |                           |
+----v-----------+        +------v---------+
| worker Pod     |        | worker Pod     |   Kubernetes-managed capacity
| RuntimeClass:  |        | RuntimeClass:  |
| flox           |        | flox           |
| init: profile  |        | init: profile  |
| slurmd         |        | slurmd         |
|  -> slurmstepd |        |  -> slurmstepd |
|      -> job    |        |      -> job    |   Slurm-managed work
+--------+-------+        +--------+-------+
         |                         |
         +-----------+-------------+
                     |
                  shared /home

Kubernetes provisions and maintains worker capacity. Slurm schedules and executes work inside that capacity. A submitted job is not a Kubernetes Job, and sbatch does not create a pod. Kubernetes sees a stable Slurm service and worker pool; Slurm remains the scheduler and task launcher.

What this implementation demonstrates

The reference implementation is designed to preserve the ordinary Slurm contract while moving worker lifecycle under Kubernetes and worker software composition under Flox. Its acceptance suite exercises the following properties against the local deployment:

ordinary batch scripts remain ordinary Slurm batch scripts; no Kubernetes-specific submission syntax is introduced;
users submit to Slurm, not to Kubernetes;
Slurm jobs do not become Kubernetes Jobs or one-pod-per-job objects;
slurmd runs in Kubernetes-managed worker pods;
the worker pod uses the flox Kubernetes RuntimeClass;
the worker Flox environment is pinned to a published generation;
the base worker image is pinned by digest;
tasks receive the intended Flox PATH contribution and only explicitly allow-listed activation-derived variables;
a missing or invalid Flox task profile fails closed rather than falling back to the plain image environment;
runtime-changing updates drain and replace the entire compatibility cohort before Slurm returns it to service;
ordinary unprivileged Slurm identities can execute on workers without a duplicate local account on every worker;
the native worker nodes are reserved for Slurm capacity rather than general Kubernetes workloads.

These are properties of the tested reference deployment, not a blanket compatibility claim for an arbitrary production Slurm estate.

What this implementation does not yet validate

Before treating this pattern as production-ready for a specific site, validate the parts that the repository cannot establish on its own:

Customer workload compatibility. The customer's existing workload corpus has not been run. Preserving the sbatch/srun interface is necessary, but the real acceptance criterion is successful execution of the customer's scripts and applications.
Production identity. The bundled slurmtest account is acceptance scaffolding. The intended production model is authentication and identity resolution at the login endpoint, typically against the site's directory service; that integration is not implemented here.
REST submission. The user-facing path exercised here is native Slurm CLI submission. slurmrestd submission semantics are not validated by this repository.
Accounting. The reference deployment does not include slurmdbd, so associations, accounting policy and accounting-backed workflows are outside the current evidence.
GPU and RDMA workloads. No GPU or RDMA hardware is part of the reference test environment. Those paths should be treated as untested, not implicitly supported.

Design scope and non-goals

The pattern deliberately stays inside existing Kubernetes, Slinky, Slurm and Flox extension points. It does not add:

a custom scheduler, controller or admission webhook;
a SPANK plugin or forked Slurm worker image;
a translation layer that turns sbatch submissions into Kubernetes Jobs;
per-job Flox environment selection;
per-task Flox activation or network fetches on the job launch path.

Those are different architectures with different operational contracts. This repository keeps the worker runtime a property of the Slurm worker cohort.

Repository entrypoints

The customer-facing entrypoint is:

slurm-operator.sh

Its supported lifecycle is:

./slurm-operator.sh up
./slurm-operator.sh status
./slurm-operator.sh sinfo
./slurm-operator.sh ssh <node-number> [command...]
./slurm-operator.sh down
./slurm-operator.sh destroy
./slurm-operator.sh strip-secrets

Other important files are:

slurm-cluster.sh creates the local QEMU/KVM + kubeadm substrate used by slurm-operator.sh and generates the Slurm Operator values consumed by this deployment.
acceptance.sh exercises the native Slurm/Flox worker contract end to end.
.flox/env/manifest.toml and .flox/env/manifest.lock define the reproducible host-side toolchain.
nssh is a convenience wrapper for SSH access to the local VMs.
scripts/ and conf/ are generated implementation artifacts. The durable definitions live in the shell generators; generated copies can be overwritten by a later run.

slurm-operator.sh copies slurm-cluster.sh into its deployment work directory and invokes that copy as a substrate helper. Operators should normally interact with slurm-operator.sh, not run the helper directly.

Architecture

The default deployment creates three AlmaLinux 10 VMs on plain QEMU/KVM:

Linux host
│
├── Flox host environment
│   └── qemu, ssh, kubectl, helm, GNU userspace
│
└── QEMU/KVM Kubernetes cluster
    │
    ├── node1
    │   ├── Kubernetes control plane
    │   ├── general cluster infrastructure
    │   ├── NFS export for /home
    │   ├── Slinky Slurm Operator control-plane workloads
    │   └── Slurm login endpoint
    │
    ├── node2
    │   ├── Kubernetes worker
    │   ├── Flox containerd shim
    │   ├── shared /home
    │   └── reserved native-Slurm worker capacity
    │
    └── node3
        ├── Kubernetes worker
        ├── Flox containerd shim
        ├── shared /home
        └── reserved native-Slurm worker capacity

The Slurm chart creates:

a slurmctld controller;
a compute NodeSet containing COMPUTE_REPLICAS slurmd worker pods;
Slurm partition all;
one user-facing LoginSet by default.

The worker pods are long-lived Slurm nodes. A submitted Slurm job executes as a Slurm task inside those worker pods. Submitting another job does not cause Kubernetes to create another workload object for that job.

Worker-pool ownership

slurm-operator.sh marks nodes 2 through NODES as dedicated native Slurm capacity:

label: slinky.local/native-worker=true
taint: slinky.local/native-worker=true:NoSchedule

The compute NodeSet selects that label and carries the matching toleration. node1 explicitly does not carry the native-worker label or taint.

This establishes the scheduling boundary:

Kubernetes may place the Slurm worker pods on the reserved pool.
Ordinary Kubernetes pods without the toleration cannot consume those nodes.
Slurm decides how work is allocated across the slurmd workers after the workers exist.

The taint is placement hygiene, not a security boundary. A Kubernetes principal allowed to create workloads with arbitrary tolerations can bypass it. Production isolation between Slurm capacity and other tenants therefore remains a namespace/RBAC/admission-policy concern.

The practical topology needs at least one infrastructure/control-plane node plus one native-worker node. The default is NODES=3 with COMPUTE_REPLICAS=2.

The two Flox environments

There are two independent uses of Flox in this repository. They should not be confused.

Host toolchain

The repository's own .flox environment makes the host-side tooling reproducible across x86_64 Linux distributions. It supplies Bash, QEMU, genisoimage, OpenSSH, curl, GNU core utilities, sed, awk, grep, kubectl and Helm.

Activate it from the repository root:

flox activate

The current lock resolves, among other packages:

Bash 5.3p15
QEMU 11.1.0
OpenSSH 10.5p1
kubectl 1.37.0
Helm 4.2.4

The activation hook also reports when /dev/kvm is absent or not writable.

Worker runtime

WORKER_FLOX_ENV is a separate, published FloxHub environment that defines the software runtime presented to Slurm work.

The repository default is deliberately only a demonstrator:

WORKER_FLOX_ENV=flox/echoip
WORKER_FLOX_GENERATION=2

For customer use, set these to the environment that should define the worker software stack:

WORKER_FLOX_ENV=myorg/hpc \
WORKER_FLOX_GENERATION=17 \
./slurm-operator.sh up

The generation pin is part of the compatibility contract. An unversioned environment name would allow workers created at different times to resolve different contents under the same logical name.

How Flox prepares a Slurm worker

The local substrate installs Flox and containerd-shim-flox-v2 on the Kubernetes nodes and registers the shim as the containerd runtime named flox.

Kubernetes exposes that runtime as:

runtimeClassName: flox

The compute NodeSet carries the Flox annotations that select and configure the published worker environment, including:

flox.dev/environment=<environment>:<generation>
flox.dev/activate-mode=<mode>
flox.dev/skip-containers=<comma-separated helpers>

By default, the small logfile helper remains on the stock runtime. The two containers that define the actual worker runtime are not allowed to skip Flox preparation:

slurmd
flox-task-profile-init

The values generator rejects any configuration that adds either name to WORKER_FLOX_SKIP_CONTAINERS.

The default worker image is also immutable:

ghcr.io/slinkyproject/slurmd:26.05-ubuntu26.04@sha256:f06ab7b1ce18b3d59c54c8c6f6b03698a28973d4d8fb30985bd0dfd673bd6dd9

WORKER_IMAGE must use the form:

repository:tag@sha256:digest

Arbitrary mutable tags are rejected.

The result is a worker whose Slurm daemon comes from known Slinky image bytes and whose software environment is realized from a known FloxHub generation.

Pre-warming large worker environments

When WORKER_FLOX_ENV is explicitly present in the operator process, including when it was loaded from persisted deployment state, slurm-operator.sh attempts to pre-realize the pinned environment on every native-worker guest:

flox pull <environment> --copy --generation <generation>

This reduces first-container startup pressure for large environments. The implementation added this path because a sufficiently large first realization can exceed kubelet's container-runtime request timeout and cause repeated, competing pulls.

Pre-warming is an optimization, not the correctness boundary. A failed pre-warm emits a warning. The worker must still complete Flox preparation successfully when Kubernetes creates it.

How the Flox runtime reaches Slurm tasks

The key boundary in this design is simple:

The Flox shim prepares a container. It does not prepare a Slurm job.

Preparing the slurmd container is therefore necessary but not sufficient. With activate-mode=run, the shim realizes the pinned environment and runs its Flox hook/profile for the worker container, but Slurm constructs each task's environment from its own launch message rather than inheriting the ambient slurmd process environment. In the reference tests, FLOX_ENV was present in the prepared slurmd container but did not reach launched tasks by inheritance, including across the tested Slurm export modes.

The worker runtime must cross that task boundary explicitly. This repository does so once per worker lifetime, then cheaply per task: an init container materializes a sanitized worker task profile, and Slurm TaskProlog injects that already-prepared profile into each launched task.

1. Flox-backed profile initializer

Each worker pod contains an init container named:

flox-task-profile-init

It runs under the same pinned Flox environment as slurmd and must complete before slurmd can start.

The initializer derives a small, sanitized task profile from its own activated environment and writes it to:

/run/flox-task-profile

The profile contains:

the PATH entries rooted in the realized Flox environment, preserving their order;
explicitly allow-listed activation-derived variables that are actually set;
a schema/version record;
the pinned Flox environment reference;
the activation mode;
the immutable worker image reference;
the computed worker runtime revision;
the realized Flox environment path.

It does not dump the init container's complete environment. Kubernetes service variables, daemon-only state and Flox realization credentials are excluded from the task contract.

The revision file is written last and serves as the profile-completion marker. slurmd never starts if profile initialization fails.

Using a pod init container also gives the profile the correct lifecycle: it is worker state, not job state. The profile is ready before the worker can become Slurm capacity, it survives a restart of the slurmd container because the backing emptyDir is pod-scoped, and it is rebuilt automatically when Kubernetes replaces the pod. If profile generation fails, the pod remains in initialization and never advertises an incompletely prepared worker to Slurm.

2. Slurm `TaskProlog`

The generated Slurm configuration sets:

TaskProlog=/etc/slurm/flox-task-prolog/task-prolog.sh

The prolog runs for each Slurm task. It reads the already-produced profile and emits Slurm export NAME=value directives for the task.

It performs no network work, does not resolve the Flox environment again, does not inspect slurmd's process environment and requires no privilege.

It also uses shell builtins only. That matters because Slurm modes such as:

--export=NONE
--export=NIL

can deliberately start the task with an unusable executable PATH. The prolog must still be able to reconstruct the worker runtime in those modes.

PATH policy

WORKER_PATH_PRECEDENCE makes command-resolution policy explicit:

prepend   worker Flox environment wins command resolution (default)
append    submit-side or module PATH entries remain ahead of Flox

With the default prepend policy, a command supplied by the pinned worker environment is selected before a same-named command from the base image or submit-side PATH.

Additional activation-derived variables

A PATH bridge is sufficient only for environments whose runtime contract is entirely executable lookup. Some Flox environments perform meaningful activation work in hooks or profiles and export variables required by the applications they contain.

The reference testing found a concrete example in the GATK environment: activation materializes a conda environment, exports GATK_CONDAENV_PREFIX, and adds that conda environment's bin directory to PATH. Propagating only the top-level Flox package path can make the launcher visible while omitting runtime state needed by downstream GATK tools.

For that reason, the extra environment contract is explicit rather than guessed. WORKER_ENV_EXPORTS is a comma-separated allow-list. It defaults to empty, which means the worker contributes its Flox PATH but no additional activation-derived variables. Most package-only environments need nothing else; environments with activation-derived runtime state should list only the variables their jobs actually require.

Example:

WORKER_ENV_EXPORTS=GATK_CONDAENV_PREFIX,MY_TOOL_HOME \
./slurm-operator.sh up

The profile initializer accepts only ordinary shell variable names and rejects multiline values because the Slurm prolog directive protocol is line-based.

Fail-closed behavior

There is no plain-image fallback for an invalid worker profile.

If the profile directory or any required profile file is missing or malformed, TaskProlog reports an error to the submitting user and refuses to launch the task with an incomplete environment.

Likewise, if Flox cannot realize the requested environment during container creation, the worker does not silently start without Flox preparation.

Shared storage

The local substrate uses one shared filesystem so submit-side and compute-side paths have the same meaning.

By default:

node1 exports /home over NFS;
nodes2..N mount that export at /home;
the Slurm controller, LoginSet and worker pods mount the same guest path into their containers.

The worker mount is controlled by:

WORKER_HOME_PATH=/home

Because /home on a worker guest is already the NFS mount, the pod's hostPath exposes the same shared filesystem rather than creating another storage layer.

Set an explicitly empty value to omit the shared-home mount from the Slurm pods:

WORKER_HOME_PATH= ./slurm-operator.sh up

For the default local deployment, shared /home is also where batch output and cross-worker file tests are expected to live.

The local-path provisioner is installed separately to satisfy Kubernetes PVCs such as slurmctld save-state. It is not the shared user filesystem.

User submission and identity

A user-facing Slurm LoginSet is enabled by default.

The login pod runs sackd and SSH, sees the same shared /home as the workers and submits work to Slurm in the ordinary way. Submitting by kubectl exec into slurmctld is an administrative/debugging path, not the intended user interface.

The local deployment exposes the login endpoint with:

LOGIN_NODEPORT=32222
LOGIN_SSH_PORT=2222

The generated cluster key is accepted for administrative login:

ssh -i operator/id_cluster -p 2222 root@127.0.0.1

Test identity scaffolding

For end-to-end acceptance testing, the repository creates one unprivileged user by default:

user:                 slurmtest
uid:                  4001
gid:                  4001
supplementary group:  sharedgrp
supplementary gid:    4002

That account is deliberately not the identity architecture. It is local scaffolding that lets the acceptance suite prove unprivileged login, primary/supplementary group propagation, shared-home access and cross-worker execution.

The scaffold is constructed from the login image's own /etc/passwd, /etc/group and /etc/shadow, with the test user and group appended. It is mounted only into the login endpoint. The worker containers do not receive a matching local account.

Slurm is configured with:

LaunchParameters=enable_nss_slurm

This allows slurmstepd to service NSS lookups for the launched user from Slurm's job credentials. Slurm also carries the user's supplementary groups with the launch credential. A job can therefore run under the correct UID/GIDs on a worker that has no locally provisioned copy of that user.

The architectural consequence is that identity is primarily a submit-side concern: authenticate and resolve the user at the login endpoint, then let Slurm carry that credential to the compute side. The workers do not need a second copy of the site's directory just to launch an allocated job.

Disable the test scaffold when integrating a real identity source:

TEST_IDENTITY=0 ./slurm-operator.sh up

For production, replace the synthetic account with the site identity service at the login endpoint—for example, SSSD against the customer directory. This repository does not configure that integration.

The test values are overridable with:

TEST_USER
TEST_UID
TEST_GID
TEST_SUPP_GROUP
TEST_SUPP_GID

Whole-cohort runtime revisions

A worker's compatibility is treated as a property of the complete runtime, not just of its Flox environment name.

The values generator computes a SHA-256 runtime revision over every setting that can change what a native Slurm task sees or what capacity the worker advertises:

immutable worker image;
Flox environment;
Flox generation;
Flox activation mode;
Flox skip-container policy;
task-profile schema;
allow-listed environment exports;
PATH precedence;
worker CPU allocation;
worker memory allocation;
shared-home path.

The hash is placed on the NodeSet pod template as:

slinky.local/runtime-revision=sha256:...

The generated task profile records the same revision.

Why the entire cohort moves together

A normal Kubernetes rolling update can temporarily leave one worker on revision A while another is already on revision B. That is not an acceptable compatibility boundary for Slurm because a multi-node allocation could span both revisions.

slurm-operator.sh up therefore establishes a Slurm maintenance barrier before a runtime-changing Helm update can replace any worker.

For a changed runtime revision it:

sets partition all to DOWN, blocking new allocations;
drains every node in the existing compatibility cohort;
waits for active jobs and job steps to finish naturally;
leaves pending jobs queued behind the closed partition;
applies the new Slurm chart/NodeSet revision;
waits until every desired worker replica is both updated and Ready;
verifies that every worker carries the intended Flox environment and runtime revision;
verifies that every worker's generated task profile reports that same runtime revision;
repairs stale dynamic Slurm node records when a CPU or memory change produces a CoreSpec differ registration failure;
waits for the complete expected worker cohort to register with slurmctld;
resumes the validated Slurm nodes;
verifies that the complete cohort is usable and that the login endpoint can reach Slurm;
reopens partition all.

The partition remains DOWN throughout replacement and validation. Slurm cannot admit a new job onto a mixed or only partially validated worker cohort.

Failed and interrupted updates

If an update fails after the maintenance cycle has begun, the implementation leaves the partition DOWN rather than making unvalidated capacity schedulable.

A later up with the same desired runtime revision does not blindly assume there is nothing to do. It first proves that slurmctld is reachable and checks partition state:

revision unchanged + partition UP takes the no-change fast path;
revision unchanged + partition DOWN is treated as recovery from an incomplete maintenance cycle;
unknown controller or partition state is an error, not evidence that the deployment is healthy.

This makes the maintenance boundary durable across an interrupted update even though the desired NodeSet revision may already have been written.

Persisted deployment revision

Explicit worker-runtime settings are persisted in:

operator/worker-revision.env

This prevents a later bare:

./slurm-operator.sh up

from silently reverting an explicitly deployed runtime setting merely because that environment variable is absent from the new shell.

The persisted keys are:

WORKER_FLOX_ENV
WORKER_FLOX_GENERATION
WORKER_FLOX_ACTIVATE_MODE
WORKER_FLOX_SKIP_CONTAINERS
WORKER_ENV_EXPORTS
WORKER_PATH_PRECEDENCE
WORKER_CPUS
WORKER_MEM_MB
WORKER_IMAGE
WORKER_HOME_PATH

An explicitly supplied value wins over the persisted value and is recorded for future runs. Explicit empty values are preserved for settings where empty has meaning.

Built-in defaults that were never explicitly set are not materialized merely for the sake of filling this file.

Current defaults

Local substrate

WORKDIR=$PWD/operator
NODES=3
VCPUS=8
MEM=8192
DISK_SIZE=40G
SHARED_HOME=1
SSH_PORT_BASE=2240
MCAST=230.0.0.12:14712
K8S_API_PORT=6444
POD_CIDR=10.244.0.0/16
SVC_CIDR=10.96.0.0/12

The private VM fabric is 10.20.0.0/24, with node addresses derived from the node number. NODES is limited by the implementation's address/MAC scheme to 1 through 9; this architecture requires at least one native-worker node in addition to node1.

Software

AlmaLinux cloud image   AlmaLinux-10-GenericCloud-10.2-20260817.0.x86_64.qcow2
Kubernetes              1.36.3
flannel                 v0.28.9
Flox RPM                1.15.0
Flox shim installer     flox/containerd-shim-flox-installer
Slinky charts           1.2.2
cert-manager            v1.19.2
local-path-provisioner  v0.0.31
expected Slurm          26.05.4

Slurm worker runtime

COMPUTE_REPLICAS=2
WORKER_FLOX_ENV=flox/echoip
WORKER_FLOX_GENERATION=2
WORKER_FLOX_ACTIVATE_MODE=run
WORKER_FLOX_SKIP_CONTAINERS=logfile
WORKER_ENV_EXPORTS=
WORKER_PATH_PRECEDENCE=prepend
WORKER_CPUS=6
WORKER_MEM_MB=6144
WORKER_HOME_PATH=/home
WORKER_IMAGE=ghcr.io/slinkyproject/slurmd:26.05-ubuntu26.04@sha256:f06ab7b1ce18b3d59c54c8c6f6b03698a28973d4d8fb30985bd0dfd673bd6dd9

The worker CPU and memory limits are deliberately lower than the default VM capacity. Without explicit pod limits, slurmd can detect the whole Kubernetes node and advertise capacity that the pod does not actually own, including resources needed by kubelet, the CNI and other node services. The Slinky worker entrypoint converts the declared container limits into Slurm specialization so Slurm's allocatable CPU and memory reflect the worker container rather than the whole host.

This is also why CPU/memory changes participate in the runtime revision. Slurm dynamic-node records cannot simply absorb every specialization change in place; a changed CPU allocation can leave an old node record in INVALID_REG with CoreSpec differ. The reconciliation path detects that condition, removes the stale dynamic record and lets the replacement worker register cleanly.

Login endpoint

LOGIN=1
LOGIN_NODEPORT=32222
LOGIN_SSH_PORT=2222
TEST_IDENTITY=1

Requirements

The local reference deployment requires:

x86_64 Linux;
Flox installed on the host;
/dev/kvm present and writable by the invoking user;
sufficient CPU, RAM and disk for the requested VMs;
outbound network access for first-time and uncached provisioning.

The host does not need libvirt, a host bridge or root privileges for normal cluster operation. QEMU uses user-mode networking for host access and a loopback-scoped multicast socket network for the private VM fabric.

The guests do use privileged system administration internally. The cloud-init administrator has passwordless sudo, which is appropriate for this local reference environment and should not be mistaken for a production security design.

Quick start

Activate the host environment:

flox activate

For a customer worker environment, set a published environment and generation explicitly on the first deployment:

WORKER_FLOX_ENV=myorg/hpc \
WORKER_FLOX_GENERATION=17 \
./slurm-operator.sh up

A bare demonstrator deployment is also possible:

./slurm-operator.sh up

Inspect Slurm:

./slurm-operator.sh sinfo

Inspect Kubernetes:

export KUBECONFIG="$PWD/operator/k8s/kubeconfig"
kubectl get nodes
kubectl -n slurm get pods
kubectl -n slurm get nodesets

Connect to a guest when debugging the local substrate:

./slurm-operator.sh ssh 1
./slurm-operator.sh ssh 2

With the default test identity enabled, connect to the Slurm login endpoint as the unprivileged acceptance user:

ssh -i operator/id_cluster -p 2222 slurmtest@127.0.0.1

From that login session, use Slurm normally:

sinfo
srun -N1 hostname
sbatch my-existing-script.sh

The job does not need Flox- or Kubernetes-specific syntax. The worker runtime is a property of the Slurm worker cohort.

Lifecycle

Bring up or reconcile

./slurm-operator.sh up

up is intended to be rerunnable. At a high level it:

loads and normalizes persisted worker revision state;
brings up or repairs the local QEMU/Kubernetes substrate;
reserves nodes2..N for native Slurm worker pods;
installs/reconciles local-path storage and cert-manager;
installs/reconciles the Slinky Slurm Operator;
optionally pre-warms the explicitly selected worker Flox environment;
prepares the local test identity when enabled;
establishes a compatibility-cohort maintenance boundary if the runtime revision changed;
installs/reconciles the Slurm cluster chart;
waits for and validates the complete worker cohort;
resumes workers and reopens Slurm only after validation succeeds.

Status

./slurm-operator.sh status

This reports the local VM substrate state and, when the kubeconfig is available, the Slurm and operator pods.

For Slurm's own view:

./slurm-operator.sh sinfo

Stop without deleting disks

./slurm-operator.sh down

The local VMs are powered off in reverse node order so the NFS server on node1 is stopped last. VM disks and deployment state are retained.

Delete VM disks

./slurm-operator.sh destroy

This stops the VMs and removes their overlay disks. Cached/reproducible inputs and credentials are retained unless separately sanitized.

Remove credentials before publication or handoff

First stop the deployment, then run:

./slurm-operator.sh down
./slurm-operator.sh strip-secrets

The sanitizer removes the local deployment's SSH key, Munge key, kubeconfig, cloud-init seeds and VM disks. The next up generates fresh credentials and a new kubeadm PKI.

Changing configuration

Not all settings have the same lifecycle semantics.

Worker-runtime settings

The WORKER_* settings that participate in worker-revision.env are intentionally treated as deployable worker-runtime state. Changing one causes the runtime-revision hash to change and invokes the whole-cohort maintenance flow described above.

For example:

WORKER_FLOX_GENERATION=18 ./slurm-operator.sh up

or:

WORKER_PATH_PRECEDENCE=append ./slurm-operator.sh up

The first changes software contents. The second changes task command-resolution policy. Both are compatibility-relevant and therefore move the worker cohort as one unit.

COMPUTE_REPLICAS is not part of the runtime-revision hash. It controls NodeSet size rather than the software/runtime contract.

Local substrate topology

Treat the VM substrate as create-time state unless the implementation explicitly reconciles the setting.

NODES is embedded into the cloud-init seeds. If the requested node count differs from the count used to create those seeds, the substrate refuses to continue.

To change the local node count:

./slurm-operator.sh destroy
rm -rf operator/seed
NODES=<new-count> ./slurm-operator.sh up

The same recreate-first approach is the safe default for other fundamental local-substrate changes whose old state is baked into VM disks or provisioning markers.

Acceptance suite

Run the default acceptance suite against a healthy deployment with:

./acceptance.sh

Every check is reported as PASS, FAIL or SKIP.

A SKIP is explicitly not a pass. It means the requirement was not exercised in that run.

The default groups exercise:

real Flox shim execution on the actual slurmd CRI sandbox;
generation-pinned Flox realization inside the worker;
successful worker task-profile initialization;
ordinary sbatch execution with no per-job Kubernetes translation;
interactive and multi-step Slurm behavior;
multi-node MPI;
environment and identity behavior;
arrays, dependencies and other Slurm policy features;
resource accounting and process cleanup;
shared storage;
the user-facing login endpoint;
task-environment provenance across Slurm export modes;
worker safety and isolation;
unprivileged identity and supplementary groups end to end;
native worker-pool placement;
failure isolation when a Flox environment cannot be realized;
worker runtime-revision consistency.

Run selected groups by name:

./acceptance.sh T3 T5 TP TI

Several lifecycle tests are intentionally opt-in because they mutate or disrupt the worker deployment:

./acceptance.sh T11R   # rerun/idempotency and incomplete-maintenance recovery
./acceptance.sh T11U   # live whole-cohort runtime update
./acceptance.sh T11C   # CPU-capacity revision and stale node-record cleanup
./acceptance.sh T11F   # failed runtime revision remains unavailable

These tests can roll the worker cohort and then restore the original runtime policy.

Deployment state

The operator deployment keeps its local state below:

operator/

Important paths include:

operator/base/                         cached AlmaLinux base image
operator/nodes/                        VM overlays, QEMU pidfiles and console logs
operator/seed/                         cloud-init NoCloud seeds
operator/k8s/kubeconfig                host-side kubeconfig
operator/flox/                         cached Flox RPM
operator/conf/slurm-operator-values.yaml
                                       generated Slinky values and task-profile logic
operator/scripts/                      generated guest provisioning scripts
operator/id_cluster                    generated SSH private key
operator/id_cluster.pub                generated SSH public key
operator/munge.key                     generated local substrate secret
operator/worker-revision.env           persisted worker runtime settings

The generated Slinky values file is especially useful when debugging what runtime revision the script intended to deploy, but it is generated output rather than the authoritative source definition.

Secrets and publication safety

The local work directory contains credentials and full guest filesystems. At minimum, treat these as sensitive deployment state:

operator/id_cluster
operator/k8s/kubeconfig
operator/munge.key
operator/seed/
operator/nodes/

The source tree's .gitignore reduces accidental Git commits, but it cannot protect tarballs, rsync, filesystem copies or other publication mechanisms.

Run strip-secrets before publishing or handing off a working tree that has been used to launch the cluster.

Recovery behavior

The implementation is written to distinguish partial state from completed state rather than treating file existence as success.

Relevant examples include:

QEMU pidfiles are verified against the process and expected node disk before they are trusted.
Destructive lifecycle operations follow the persisted on-disk topology rather than only the caller's current NODES value.
the Flox containerd runtime is considered configured only after containerd has restarted and its effective configuration resolves the expected shim path;
worker pods do not become usable Slurm capacity until their Flox-backed task profile has initialized;
the complete worker cohort must report one intended runtime revision before the nodes are resumed;
a failed runtime update leaves the Slurm partition closed;
a same-revision retry can recover an incomplete maintenance cycle rather than incorrectly returning success;
CPU/memory revisions that invalidate dynamic Slurm node records are repaired by deleting the stale record and allowing the worker to register again.

After an ordinary interrupted provisioning run, rerun:

./slurm-operator.sh up

If the prior run entered worker maintenance, the operator script treats the live Slurm partition state as part of recovery and does not infer health merely from an unchanged desired revision.

What to replace for a customer deployment

The core pattern is the native Slurm worker pool plus Flox-prepared worker runtime. Several pieces in this repository exist only to make that pattern self-contained on one Linux machine.

A customer integrating this into existing infrastructure would normally retain the architectural core:

Slinky Slurm Operator and Slurm chart;
Kubernetes-reserved native Slurm worker pool;
Flox containerd RuntimeClass on those worker nodes;
immutable worker image;
generation-pinned worker Flox environment;
Flox-backed profile initializer;
Slurm TaskProlog task-environment contract;
whole-cohort runtime-revision maintenance barrier;
fail-closed behavior when Flox preparation is unavailable or invalid.

The local scaffolding is replaceable:

QEMU/KVM VMs can become the customer's existing Kubernetes nodes;
kubeadm/flannel can become the customer's Kubernetes distribution and CNI;
local-path-provisioner can become the customer's StorageClass;
the node1 NFS export can become the customer's shared filesystem;
the synthetic slurmtest account can become the customer's identity service;
localhost NodePort forwarding can become the customer's normal login/network exposure.

Those substitutions do not change the central execution model:

Kubernetes owns the workers. Slurm owns the work. Flox owns the runtime inside the workers.

Name		Name	Last commit message	Last commit date
.flox		.flox
conf		conf
.gitignore		.gitignore
README.md		README.md
acceptance.sh		acceptance.sh
nssh		nssh
slurm-cluster.sh		slurm-cluster.sh
slurm-operator.sh		slurm-operator.sh

Latest commit

History

Folders and files

Repository files navigation