Explain separate operator and tenant Mosaic instances

docs/examples/multitenancy/operator-values.yaml

@@ -0,0 +1,42 @@
+global:
+  imagePullSecrets: [{name: nvcr-image-pull-secret}]
+agentSandbox:
+  install: false
+llm:
+  mode: external
+  external:
+    baseUrl: https://llm.example.com/v1
+    model: served-model
+    existingSecret: mosaic-inference
+mosaicUi:
+  auth:
+    existingSecret: mosaic-ui-auth
+modules:
+  ui: {enabled: true}
+  execution: {enabled: true}
+  kubernetes: {enabled: true}
+  observability: {enabled: true}
+  grafana: {enabled: true}
+  bcm: {enabled: true}
+  slurm: {enabled: false}
+  diagnostics: {enabled: false}
+  research: {enabled: false}
+  terminal: {enabled: false}
+  clusters: {enabled: true}
+  edit: {enabled: false}
+kubernetes:
+  defaultCluster: local
+  local:
+    enabled: true
+observability:
+  prometheusUrl: http://prometheus.monitoring.svc.cluster.local:9090
+  alertmanagerUrl: http://alertmanager.monitoring.svc.cluster.local:9093
+  grafanaUrl: https://grafana.example.com/grafana
+  grafanaDatasourceUid: prometheus
+  grafanaAuth:
+    existingSecret: mosaic-grafana-auth
+bcmMcp:
+  enabled: true
+  mode: ssh-adapter
+  headHost: operator@bcm-head.example.com
+  hostSshKeySecretName: bcm-host-ssh-key

docs/examples/multitenancy/tenant-reader.yaml

@@ -0,0 +1,33 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: mosaic-job-reader
+  namespace: tenant-a
+automountServiceAccountToken: false
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: mosaic-job-reader
+  namespace: tenant-a
+rules:
+  - apiGroups: [""]
+    resources: [pods, pods/log, events]
+    verbs: [get, list, watch]
+  - apiGroups: [batch]
+    resources: [jobs, cronjobs]
+    verbs: [get, list, watch]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: mosaic-job-reader
+  namespace: tenant-a
+subjects:
+  - kind: ServiceAccount
+    name: mosaic-job-reader
+    namespace: tenant-a
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: mosaic-job-reader

docs/examples/multitenancy/tenant-values.yaml

@@ -0,0 +1,37 @@
+global:
+  imagePullSecrets: [{name: nvcr-image-pull-secret}]
+agentSandbox:
+  install: false
+llm:
+  mode: external
+  external:
+    baseUrl: https://llm.example.com/v1
+    model: served-model
+    existingSecret: mosaic-inference
+mosaicUi:
+  auth:
+    existingSecret: mosaic-ui-auth
+modules:
+  ui: {enabled: true}
+  execution: {enabled: true}
+  kubernetes: {enabled: true}
+  observability: {enabled: false}
+  grafana: {enabled: false}
+  bcm: {enabled: false}
+  slurm: {enabled: false}
+  diagnostics: {enabled: false}
+  research: {enabled: false}
+  terminal: {enabled: false}
+  clusters: {enabled: false}
+  edit: {enabled: false}
+bcmMcp:
+  enabled: false
+kubernetes:
+  defaultCluster: tenant-a
+  local:
+    enabled: false
+  clusters:
+    - name: tenant-a
+      kubeconfigSecretRef:
+        name: tenant-a-kubeconfig
+        key: config

docs/multitenancy.md

@@ -0,0 +1,41 @@
+# Tenant and operator deployments
+
+Deploy separate Mosaic instances for operators and tenants. **Operators focus on hardware and systems. Tenants focus on their jobs.** Each instance is a separate Helm release with its own UI login, agent state, audit storage, and credentials.
+
+![Separate operator and tenant Mosaic instances, each with its own login, state, and credentials. The operator accesses cluster infrastructure; the tenant accesses only its jobs.](images/tenant-operator-instances.svg)
+
+| Instance | Focus | Example question |
+| --- | --- | --- |
+| Operator | Cluster health, hardware inventory, networking, and system telemetry | “Which nodes are unhealthy?” |
+| Tenant | Its own jobs, Pods, events, and logs | “Why is my job pending?” |
+
+Repeat the tenant instance for each tenant. In an enterprise, these audiences can be the infrastructure team and an application team. Within a tenant VM, visibility is limited to resources exposed to that VM or its scoped APIs.
+
+## Example setup
+
+These Kubernetes examples are read-only. The provider installs both releases and reuses an existing compatible agent-sandbox controller.
+
+- **Tenant:** [tenant-values.yaml](examples/multitenancy/tenant-values.yaml) disables infrastructure integrations and the default cluster reader. Apply [tenant-reader.yaml](examples/multitenancy/tenant-reader.yaml) in workload namespace `tenant-a`, provision a kubeconfig for that identity, and store it as Secret `tenant-a-kubeconfig` (key `config`) in `mosaic-tenant-a`.
+- **Operator:** [operator-values.yaml](examples/multitenancy/operator-values.yaml) enables cluster monitoring, observability, and [Base Command Manager](bcm.md). Configure the monitoring URLs, `mosaic-grafana-auth` Secret (`username`, `password`), and `bcm-host-ssh-key` Secret (`id_ecdsa`) for its read-only SSH identity in `mosaic-operator`.
+
+Complete [namespace and registry access setup](installation.md#1-create-the-namespace-and-registry-access) for each release. In each namespace, create a distinct `mosaic-ui-auth` Secret (`username`, `password`, `machineToken`) and `mosaic-inference` Secret (`apiKey`); replace the example inference URL and model with your approved endpoint.
+
+Install with your approved chart version:
+
+```bash
+helm upgrade --install mosaic-tenant-a \
+  oci://nvcr.io/0948643769302270/afoa-release/mosaic-stack \
+  --version "$MOSAIC_CHART_VERSION" --namespace mosaic-tenant-a \
+  -f tenant-values.yaml --atomic --wait --timeout 12m
+
+helm upgrade --install mosaic-operator \
+  oci://nvcr.io/0948643769302270/afoa-release/mosaic-stack \
+  --version "$MOSAIC_CHART_VERSION" --namespace mosaic-operator \
+  -f operator-values.yaml --atomic --wait --timeout 12m
+```
+
+## Access boundaries
+
+Backend credentials and Kubernetes RBAC enforce access. **The chart's default Kubernetes reader is cluster-wide**, even when installed in a tenant namespace; keep it disabled for tenants. Before handover, verify with the tenant kubeconfig that job and log reads work in `tenant-a`, while other namespaces, nodes, Secrets, and job mutations are denied.
+
+Give each audience its own UI address and login. Enforce network isolation with authenticated ingress and provider-managed network policies; namespace separation alone is insufficient. Keep storage and credentials separate, including any added MCP or metrics backend. These are separate deployments, not per-user authorization in a shared UI. Cluster administrators and the runtime controller remain trusted; use separately administered clusters or VMs when required for mutually untrusted tenants.

index.md

@@ -41,6 +41,7 @@ Kind installation <docs/kind_installation>
 :hidden:
 
 Security model <docs/security_model>
+Tenant and operator deployments <docs/multitenancy>
 Use cases <docs/use_cases>
 Slurm root cause analysis <docs/slurm_rca>
 Headless integration <docs/skills/ai-factory-operations-agent-headless/SKILL>