os.mieweb.org integration: architecture for the `mieweb` target on opensource-server

[runleveldev]

Context

The mieweb target (@mieweb/cloud-os) is meant to interoperate with the orchestration in mieweb/opensource-server — LXC containers on Proxmox, fronted by auto-generated nginx, dnsmasq DNS, LDAP/oauth2-proxy auth, ACME TLS, reconciled by a site agent, and driven by the Manager web app (Node/Express/Sequelize + Postgres).

Today there is no coupling between the two repos. cloud-os is a runtime adapter set (D1→libSQL, R2→S3/MinIO, KV/Queues→Valkey, Vectorize→libSQL vectors, DO→in-proc); opensource-server is a provisioning/orchestration control plane. This issue captures the design discussion for how to bridge them.

How Cloudflare frames the problem (the model we're replicating)

Cloudflare splits into two planes:

• Control plane — wrangler deploy reads wrangler.jsonc as a declarative resource manifest, calls the account API to ensure resources exist, and injects scoped binding handles into the Worker's env. State lives in independently-provisioned, separately-scaled managed services.
• Data plane — at request time env.DB / env.BUCKET are live handles that RPC to those services. Compute is stateless/disposable.

@mieweb/cloud faithfully replicated the data plane (binding shapes) and delegates the control plane to wrangler on the cloudflare target. On mieweb there is no control plane — mieweb.jsonc + docker-compose is a hand-rolled substitute. The integration work is making opensource-server play the control-plane role.

Note: env on Cloudflare is not process.env — it's an object of live handles + string vars passed as the 2nd arg to fetch. Any OS env vars we use are host-bootstrap config, never the worker's env.

Binding identity vs. location (the ID format)

Model wrangler's beta automatic provisioning: IDs in wrangler.jsonc short-circuit provisioning; absent IDs get created and written back. We adopt Option A — the committed wrangler.jsonc is the only registry (documentation instructs committing it after deploy to persist IDs).

For IDs to be self-describing without the Manager persisting any driver-specific data, use a URI form in wrangler's native field:

{ "d1_databases":  [{ "binding": "DB",         "database_id": "mieweb+libsql:bluehive-hum" }] }
{ "r2_buckets":    [{ "binding": "RECORDINGS", "bucket_name": "mieweb+s3:bluehive-recordings" }] }
{ "kv_namespaces": [{ "binding": "SESSIONS",   "id": "mieweb+valkey:bluehive/sessions" }] }
{ "vectorize":     [{ "binding": "SEARCH_INDEX","id": "mieweb+libsql-vec:search_index?dim=768" }] }

• mieweb+<driver> — marks it as ours (not a CF UUID) and is the registry-driver key.
• path/query — the resource identity + tuning (?dim=768). Committed to git. Never contains endpoint or credentials.

Resolution = identity (from committed ID) + location/secrets (injected at boot) → the cfg object the existing driver registry already consumes. Boot-time: parseId(id) → {driver, name, opts}; merge with injected driver endpoint/creds; call getDriver(driver)(cfg) — unchanged below the seam (~30 lines in createCloudEnv/config loader). Fail fast in settleEnv if a driver's endpoint is missing.

The isolation requirement that reshapes everything

Many users share a site; one app's credentials must not reach another app's data. This rules out a site-wide shared-credential blob and pushes toward either per-app scoped credentials on shared stores (Manager must mint/custody admin creds) or per-app dedicated services (isolation becomes structural, no shared admin key exists).

Proposed architecture: single converged container per app

1. Read name from wrangler.jsonc (must be a valid DNS label; CLI rejects non-conforming names rather than silently slugifying).
2. Create a container via the Manager API with hostname = name — this enforces uniqueness per site (not truly global).
3. Container is created from a converged template image (e.g. ghcr.io/mieweb/cloud:latest) running MinIO + libsqld + Valkey (+ optional Ollama) locally under a supervisor (systemd units fit the LXC base). The CLI sets service credentials via environmentVars.
4. The app is deployed into the same container — reaching co-located services over localhost (127.0.0.1:9000/8080/6379); only the app itself is exposed publicly via the Manager's services (nginx + TLS). ⚠️ Open question — delivery mechanism:
   • (A) per-app image FROM the converged base → build + registry push + update-container(template=<digest>). Keeps the Manager purely image-based/dumb. Recommended target.
   • (B) converged base fetches app at boot (env-pointed source). POC shortcut; slower/racier starts, needs somewhere to stage the bundle.
   • (C) agent/volume sync — requires Manager/agent changes; avoid.
5. mieweb deploy is idempotent via hostname: list-containers?hostname=<name> → update, else create. Same name → same container; change name → new container.

Consequences of this model

• Isolation is structural (per-container) — no shared stores, no Manager credential-minting, no bleed.
• Manager stays dumb — reuses create/update/list-container, services, environmentVars. Likely one small gap: a persistent volume for the container's now-critical local state (MinIO/libsqld/Valkey data) + a backup story.
• MIEWEB_DRIVERS collapses — endpoints are constant localhost; only per-container secrets vary (or are generated in-container and never leave).
• Every distributed-coordination problem disappears — DO placement, queue multi-consumer, KV list cursors are non-issues because everything is single-node by design.
• Trade-off: no horizontal scale, no HA, heavy per-app baseline. These are scale-out concerns that don't bite at POC scale.

Decision: single-container-per-app (A) vs. shared services + control plane (B)

Dimension	A: Single container per app	B: Shared services + control plane
Tenant isolation	✅ Structural — no shared store or network path	⚠️ Credential-scoped; only as strong as weakest store's ACLs (Valkey leaky)
Manager code changes	✅ ~None (maybe one volume knob)	❌ Significant — provisioning + credential-minting + admin-cred custody
Credential bleed risk	✅ None — creds only open co-located services	❌ Shared admin key is a real bleed vector
MIEWEB_DRIVERS complexity	✅ Collapses (localhost endpoints; per-container secrets)	⚠️ Per-app scoped creds + endpoints minted per deploy
DO placement / queue consumers / KV list	✅ Evaporate — single-node by design	❌ All persist; must be solved
Horizontal scaling	❌ Impossible (2 replicas = 2 MinIOs)	✅ Native, Cloudflare-faithful
Fault tolerance / HA	❌ Container = SPOF for compute and data	✅ Worker crash cheap; services HA independently
Resource efficiency	❌ Heavy per-app baseline even for tiny apps	✅ Shared services amortize; thin workers, dense
Deploy mechanics	⚠️ Per-app image build + push	⚠️ Worker image, but no co-located stack to build
Backup / DR	✅ Simple unit (snapshot one container) / ❌ many targets, state in-container	⚠️ Centralized stores (fewer targets) / entangled per-app data
Cloudflare fidelity (semantic)	❌ Low — abandons stateless/shared-state split	✅ High — mirrors CF control-plane model
Operational / attack surface	✅ Small — only app exposed; data on localhost	❌ Larger — shared services network-reachable
Time-to-first-working	✅ Fast — buildable image + existing API	❌ Slow — blocked on Manager control-plane features
Cost of the hard problems	Pushed into image build (tractable)	Pushed into the Manager (distributed-systems + authz)
Failure blast radius	✅ One app	❌ Wider — shared-store outage hits every app
Fit for opensource-server	✅ High — platform is a container provisioner	⚠️ Medium — platform must grow a managed-services role

The trade in one line

• A trades horizontal scale + efficiency for the disappearance of every hard problem (isolation, credential custody, DO placement, Manager changes). Cost lands in the image build and "one heavy container per app, no HA."
• B trades simplicity for scale + efficiency; it's Cloudflare-faithful and dense, but forces the Manager to become a real control plane and leaves all coordination problems on the table.

Recommendation

• Ship A now for the POC: only option runnable without turning the Manager into a control plane; makes isolation structural. Weaknesses (no scale/HA, heavy per-app) are later, scale-out concerns.
• Treat B as the eventual model when apps get numerous (per-app full stacks wasteful) or an app needs true HA/scale — at which point growing the Manager's control plane is justified.
• Hybrid path: the URI/MIEWEB_DRIVERS design already supports per-driver, per-app endpoint overrides — a binding can point at localhost (A) or an external shared service (B) without a rewrite. Ship A, adopt B selectively later.

Open questions / follow-ups

• Step 4 delivery mechanism — commit to (A) per-app image FROM converged base; define the mieweb deploy build/push/update-container flow.
• Persistent volume + backup for the converged container's local state — does the Manager create-container API support volume specs, or is this the one real Manager gap?
• Converged image build: supervisor choice (systemd units), making Ollama/AI optional to avoid dragging a model server for no-AI apps.
• name normalization/validation rules (DNS-label constraints vs. wrangler name rules); per-site vs. global uniqueness scoping.
• Per-driver parseId/formatId/ensureExists/create contract + the createCloudEnv/config-loader seam.
• Auth/routing ownership: how the worker relates to opensource-server's nginx + oauth2-proxy + LDAP identity headers (X-User, X-Groups).
• Semantic (not just shape) conformance tests: crash-durability, DO identity — mostly moot under A, but relevant if B is adopted.

[horner]

This is really good. I like the hybrid approach and even allowing a tenant user to decide service by service in their cloud config as to weather they want to use a shared service (B) or single source the container (A). We can default to A for interns and projects where we are OODA looping and persistent data is actually a liability.