Cluster inventory never syncs: last_synced_at stays null and a no-op PUT does not trigger a rescan

[jgruberf5]

Summary

A registered ROKS cluster shows no pod inventory on the Kubernetes page — Multus reads as 0 pods while 18 are running — and last_synced_at is never set, even 70+ minutes after registration and after repeated no-op PUTs intended to enqueue a rescan.

The cluster is healthy and the workload is correct. Forge simply has no inventory for it.

Environment

• BNK Forge v4.0.0-rc1 at 161.156.198.185:8443
• Cluster id 16, name f5e2e1, ROKS 4.20 / OpenShift, us-east
• detected_platform_profile: roks, cloud_provider: ibm
• Registered by roksbnkctl v1.50.0 via bnkforge register during a cluster-create module

What Forge holds

id              = 16
name            = f5e2e1
status          = active
created_at      = 2026-08-21T14:26:37Z
last_synced_at  = null          <-- never set
platform_capabilities.secondary_networks             = true
platform_capabilities.network_attachment_definitions = true

Capability detection worked — Forge correctly identified that this cluster supports secondary networks and NADs. Only the inventory is missing.

Ground truth on the cluster

Multus is running, 18 pods, all Running:

multus-*                        6/6 Running   86m
multus-additional-cni-plugins-* 6/6 Running   86m
network-metrics-daemon-*        6/6 Running   86m

bnk up declared two NetworkAttachmentDefinitions at 14:28:18:

f5-bnk/ens3-ipvlan-l2   ipvlan,  master ens3,   static 10.10.1.1/24
f5-bnk/macvlan-conf     macvlan, master dummy0, static 192.168.1.100/24

…and all six data-plane pods are attached to both:

f5-tmm-bq756  k8s.v1.cni.cncf.io/networks = f5-bnk/ens3-ipvlan-l2,f5-bnk/macvlan-conf
f5-tmm-d45ds  (same)   ... 6 of 6

So the multi-NIC path is fully functional and invisible to Forge.

Reproduction — the rescan does not record a sync

A no-op PUT is what we understood to enqueue a fresh scan (routes/k8s/clusters.py update_cluster). It returns 200 and moves updated_at, but last_synced_at does not change:

before  last_synced_at=null  updated=2026-08-21T14:33:50Z
PUT /api/k8s/clusters/16  {} -> 200
after   last_synced_at=null  updated=2026-08-21T15:37:30Z

Our harness has been issuing that PUT every 60s for the full life of the cluster (70+ minutes). last_synced_at has been null throughout.

Why this matters beyond cosmetics

Registration is deliberately performed before bnk up — that is the documented reason to register first, so the install can be watched filling in on the Kubernetes page. The NADs above were created 1m41s after registration. So even a scan that fired correctly at registration would capture a pre-install cluster.

Combined with no periodic resync, the Kubernetes page can never show a BNK install on a cluster Forge registered itself. For multi-NIC specifically it reports the opposite of the truth: 0 Multus pods on a cluster where Multus is running and BNK's NADs are attached to every TMM pod.

Possibly related

k8s_sync_enabled and k8s_sync_interval_seconds exist on the project model and in the API schema, but we have never observed a task read them — consistent with last_synced_at never being written.

What I could not determine

No pod-level endpoint appears to exist to check against. All of these 404 on a cluster that is status: active:

/api/k8s/clusters/16/pods            404
/api/k8s/clusters/16/inventory       404
/api/k8s/clusters/16/workloads       404
/api/k8s/clusters/16/resources       404
/api/k8s/clusters/16/network-attachments 404
/api/k8s/clusters/16/daemonsets      404

/api/k8s/clusters/16/namespaces does return live-looking data including f5-bnk, f5-utils and openshift-multus, so some cluster reads clearly work. I could not identify the endpoint backing the pod counters, so I can't say whether the page reads 0 for all workloads or whether the Multus query specifically is wrong. If it's the latter, that's a separate defect.

Suggested

1. Set last_synced_at when a scan completes, so "never scanned" is distinguishable from "scanned and genuinely empty" — right now they are identical from the API.
2. Make the no-op PUT actually enqueue a scan, or expose an explicit rescan endpoint. Clients currently have no reliable way to refresh inventory.
3. Honour k8s_sync_enabled / k8s_sync_interval_seconds, or remove them — a documented setting that nothing reads is worse than an absent one.
4. Re-scan after a project's modules reach applied. Registration-then-install is the intended flow, so a scan at registration is guaranteed to miss the install it was meant to show.

[jgruberf5]

Correction from the reporter — please disregard the Multus framing

I filed this leading with "0 Multus pods", and that was the wrong emphasis. This deployment is BNK 2.3, not 2.4:

MANIFEST_VERSION = 2.3.0-3.2598.3-0.0.170
every blueprint  = bnk_version: 2.3

If the Kubernetes page's Multus/multi-NIC reporting is gated on BNK 2.4, then 0 is the expected reading here and is not evidence of a defect. I have retitled the issue accordingly. I don't know whether that panel is version-gated — if it is, please say so and treat the Multus observation as noise.

What still stands, independent of BNK version

The sync finding does not depend on BNK at all, and I'd ask that it be assessed on its own:

cluster 16 (f5e2e1), registered 2026-08-21T14:26:37Z
last_synced_at = null          70+ minutes later

PUT /api/k8s/clusters/16 {} -> 200
before  last_synced_at=null  updated=2026-08-21T14:33:50Z
after   last_synced_at=null  updated=2026-08-21T15:37:30Z

updated_at moves, last_synced_at does not. Our harness issued that PUT every 60s for the life of the cluster and last_synced_at was null throughout.

The consequence I care about is the one in the original report: "never scanned" and "scanned, genuinely empty" are indistinguishable from the API. That is exactly why I misread this as a Multus problem in the first place — with no last_synced_at, an empty panel gives a caller no way to tell a stale view from a true one. Had that field been populated I would have diagnosed it correctly and probably not filed at all.

The timing point also stands on 2.3: registration is deliberately performed before bnk up so the install can be watched filling in, so a one-shot scan at registration captures a pre-install cluster by construction.

Suggestions 1, 3 and 4 from the original report are the ones I'd stand behind. Suggestion 2 depends on whether the no-op PUT was ever intended to enqueue a scan — that was our inference from routes/k8s/clusters.py update_cluster, and it may simply be wrong.