Skip to content

Commit 5ec8b33

Browse files
ricardo-devis-agulloricardo-devis-agullo
committed
Document registry metadata stores feature and migration
1 parent fac28d7 commit 5ec8b33

4 files changed

Lines changed: 371 additions & 0 deletions

File tree

‎website/docs/registry/registry-configuration.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ For unsuscribing to all [events](#registry-events).
121121
| <sub>storage</sub> | object | no | - | Low-level storage adapter configuration. Use this for non-S3 storage adapters |
122122
| <sub>storage.adapter</sub> | function | yes\* | - | Storage adapter function that returns a StorageAdapter instance (\*required if using storage) |
123123
| <sub>storage.options</sub> | object | yes\* | - | Options passed to the storage adapter, must include `componentsDir` (\*required if using storage) |
124+
| <sub>metadata</sub> | object | no | - | Opt-in [metadata store](/docs/registry/registry-metadata-stores) configuration. When present, the registry uses a pluggable database as the source of truth for the component index instead of scanning storage. Absent → storage-only mode (default) |
124125
| <sub>refreshInterval</sub> | number (seconds) | no | - | Seconds between each refresh of the internal component list cache. Given the data is immutable, this should be high and just for robustness |
125126
| <sub>pollingInterval</sub> | number (seconds) | no | `5` | Seconds between each poll of the storage adapter for changes. Should be low (5-10 seconds) for efficient distribution |
126127
| <sub>tempDir</sub> | string | no | `"./temp/"` | Directory where components' packages are temporarily stored during publishing |

‎website/docs/registry/registry-metadata-stores.md‎

Lines changed: 368 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,368 @@
1+
---
2+
sidebar_position: 3
3+
---
4+
5+
# Registry Metadata Stores
6+
7+
## Overview
8+
9+
By default, an OpenComponents registry keeps its **metadata index** — the list of
10+
which components and versions exist — in two storage files:
11+
`components.json` and `components-details.json`. This index is *derived*: the
12+
registry rebuilds it by scanning the whole storage directory tree
13+
(`componentsDir/<component>/<version>/package.json`) on startup and again after
14+
every publish.
15+
16+
That scan is `O(registry size)`. It also isn't transactional: concurrent
17+
publishes on different nodes each scan and then overwrite `components.json`
18+
(last-writer-wins), which only "works" because the next scan re-derives the truth
19+
from the immutable directory tree. Under heavy, multi-node publishing this drives
20+
CPU and GC pressure and makes publish cost grow with the size of the registry.
21+
22+
A **metadata store** is an opt-in feature that moves this index into a pluggable
23+
database that becomes the source of truth for *what exists*. With it:
24+
25+
- **Publish** becomes an `O(1)` atomic row append instead of a full scan + blob
26+
rewrite.
27+
- **Startup** becomes a single `O(registry size)` query instead of a directory
28+
walk (and steady-state hydration is just one query).
29+
- **Cross-node correctness** is enforced by a `PRIMARY KEY (component_name,
30+
version)` constraint instead of relying on self-healing rescans. Concurrent
31+
publishes of the same version → one wins, the other gets a clear
32+
"already exists" error; different components never contend.
33+
34+
### What it changes — and what it doesn't
35+
36+
The metadata store **only** owns the index of which `name@version` pairs exist.
37+
It does **not** change:
38+
39+
- **Static files** — component bundles still live in the configured storage
40+
adapter (S3, GS, Azure Blob, …). Storage is still required.
41+
- **`package.json` files** — still fetched from storage by `getComponentInfo`.
42+
- **The hot read path** — component reads are still served from OC's in-memory
43+
cache and never hit the database.
44+
45+
The metadata store contains only four fields per version: component name,
46+
version, publish date, and template size — exactly what the in-memory caches
47+
need.
48+
49+
> Metadata mode is **opt-in and non-breaking**. With no `metadata` block in the
50+
> configuration, the registry behaves byte-for-byte as before (storage-only
51+
> mode).
52+
53+
## How it works
54+
55+
The store is a pluggable adapter, injected the same way as the `storage.adapter`.
56+
The registry core takes zero database dependencies. The contract each adapter
57+
implements is exported by the `oc-metadata-adapters-utils` package:
58+
59+
```ts
60+
type ComponentRow = {
61+
name: string;
62+
version: string;
63+
publishDate: number; // unix seconds
64+
templateSize?: number;
65+
};
66+
67+
interface MetadataStore {
68+
adapterType: string;
69+
isValid(): boolean; // synchronous config sanity check
70+
initialise(): Promise<void>; // open pool; ensure/verify schema
71+
getAllComponents(): Promise<ComponentRow[]>; // hydration — feeds the caches
72+
addVersion(row: ComponentRow): Promise<void>; // commit point
73+
close?(): Promise<void>; // optional: release pool on shutdown
74+
}
75+
```
76+
77+
Runtime behavior in metadata mode:
78+
79+
- **Startup** — the registry initialises the store, then hydrates its in-memory
80+
list and details caches from a single `getAllComponents()` call. If the store
81+
cannot be initialised or queried, startup fails (fail-closed).
82+
- **Reads** — served entirely from the in-memory cache; hot component reads never
83+
touch the database.
84+
- **Polling** — the cache is re-hydrated from `getAllComponents()` on the polling
85+
interval. If a poll fails, the registry keeps serving the last good in-memory
86+
cache and retries on the next interval.
87+
- **Publish** uses a reservation state machine so a failed/concurrent publish can
88+
never clobber another:
89+
1. validate the publish
90+
2. write the `package.json`
91+
3. **reserve** the metadata row
92+
4. upload the statics to storage
93+
5. **commit** the metadata row
94+
A duplicate or in-progress reservation surfaces as the usual
95+
"component version already exists" publish error. If the upload or commit
96+
fails, the reservation is best-effort aborted; any orphaned statics are
97+
harmless unreferenced bytes and a re-publish is idempotent.
98+
- **Shutdown** — `registry.close(callback)` closes the HTTP server and then calls
99+
the adapter's optional `close()` hook so it can release its connection pool.
100+
101+
## Configuration
102+
103+
Add a `metadata` block as a sibling to `storage`. Storage remains required.
104+
105+
```js
106+
const azureSqlMetadataAdapter = require("oc-azure-sql-metadata-adapter").default;
107+
const s3StorageAdapter = require("oc-s3-storage-adapter");
108+
109+
registry.configure({
110+
storage: {
111+
adapter: s3StorageAdapter,
112+
options: {
113+
bucket: process.env.OC_STORAGE_BUCKET,
114+
region: process.env.OC_STORAGE_REGION,
115+
componentsDir: "components",
116+
path: process.env.OC_STORAGE_BASE_URL,
117+
},
118+
},
119+
metadata: {
120+
adapter: azureSqlMetadataAdapter,
121+
options: {
122+
connectionString: process.env.OC_METADATA_SQL_CONNECTION_STRING,
123+
},
124+
manageSchema: true,
125+
reconcileFromStorage: false,
126+
exportLegacyFiles: false,
127+
},
128+
});
129+
```
130+
131+
### `metadata` options
132+
133+
| Parameter | Type | Mandatory | Default | Description |
134+
| ------------------------------------ | ---------------- | --------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
135+
| `metadata` | object | no | - | Presence enables metadata mode. Absent → storage-only mode (default). |
136+
| `metadata.adapter` | function | yes\* | - | Metadata adapter factory returning a `MetadataStore` (\*required if using metadata). |
137+
| `metadata.options` | object | yes\* | - | Connection / pool options passed to the adapter. |
138+
| `metadata.manageSchema` | boolean | no | `true` | When `true`, the adapter auto-creates its table/index if missing. Set `false` for locked-down databases where operators manage DDL; the adapter then only verifies the schema. |
139+
| `metadata.reconcileFromStorage` | boolean | no | `false` | Bake-in flag. On startup, scan storage and idempotently insert any `name@version` present in the directory tree but missing from the database (existing rows skipped). |
140+
| `metadata.exportLegacyFiles` | boolean | no | `false` | Bake-in / DR flag. On startup, write database-derived `components.json` and `components-details.json` projections back to storage. |
141+
| `metadata.exportLegacyFilesInterval` | number (seconds) | no | - | When set (and `exportLegacyFiles` is `true`), also refresh those projections on a non-overlapping background timer. Omit to export on startup only. The timer is cleared on shutdown. |
142+
143+
The bake-in flags (`reconcileFromStorage`, `exportLegacyFiles`,
144+
`exportLegacyFilesInterval`) are only needed while migrating; see
145+
[Migrating an existing registry](#migrating-an-existing-registry).
146+
147+
> The legacy file export is **decoupled from the publish path** — a publish never
148+
> triggers a full-registry export, so publishing stays an `O(1)` append. The
149+
> exported files are a **one-directional projection** of the database (DB → files,
150+
> never read back to mutate the DB). They exist for rollback and as a
151+
> cold-start / DR snapshot; they do **not** replace the storage adapter, which
152+
> still holds all component statics.
153+
154+
## Available adapters
155+
156+
### Azure SQL / SQL Server — `oc-azure-sql-metadata-adapter`
157+
158+
A connection-pool-based (`mssql`) adapter.
159+
160+
```sh
161+
npm install oc-azure-sql-metadata-adapter
162+
```
163+
164+
```js
165+
metadata: {
166+
adapter: require("oc-azure-sql-metadata-adapter").default,
167+
options: {
168+
connectionString: process.env.OC_METADATA_SQL_CONNECTION_STRING,
169+
},
170+
}
171+
```
172+
173+
You can also pass object connection settings instead of a connection string
174+
(`server`, `database`, `user`, `password`, nested `options`, etc. — passed
175+
through to `mssql`). When no `connectionString`, `password`, or explicit
176+
`authentication` is provided, the adapter defaults to Microsoft Entra ID
177+
(`azure-active-directory-default`), so it can connect using an ambient managed
178+
identity with **no secret in config** (pass `clientId` for a user-assigned
179+
identity).
180+
181+
Adapter-specific options: `manageSchema` (default `true`), `schemaName` (default
182+
`dbo`), and `tableName` (default `oc_components`). Identifiers must match
183+
`/^[A-Za-z_][A-Za-z0-9_]*$/`.
184+
185+
With `manageSchema: true` the adapter creates (roughly):
186+
187+
```sql
188+
CREATE TABLE [dbo].[oc_components] (
189+
component_name NVARCHAR(255) NOT NULL,
190+
version NVARCHAR(64) NOT NULL,
191+
publish_date BIGINT NOT NULL,
192+
template_size BIGINT NULL,
193+
status NVARCHAR(16) NOT NULL DEFAULT N'committed',
194+
publish_token NVARCHAR(64) NULL,
195+
created_at DATETIME2 NOT NULL DEFAULT SYSUTCDATETIME(),
196+
updated_at DATETIME2 NOT NULL DEFAULT SYSUTCDATETIME(),
197+
PRIMARY KEY (component_name, version)
198+
);
199+
CREATE INDEX ix_oc_components_name ON [dbo].[oc_components] (component_name);
200+
```
201+
202+
The primary key is the concurrency guard: same-version unique violations
203+
(`2627` / `2601`) are mapped to the shared duplicate / in-progress error codes
204+
before any storage upload happens.
205+
206+
### Azure Table Storage — `oc-azure-table-metadata-adapter`
207+
208+
A schemaless, HTTP-based adapter (`@azure/data-tables`). If you already use Azure
209+
Blob Storage for statics, you can reuse the **same storage account** for the
210+
metadata table — no second database to provision. Its `PartitionKey + RowKey`
211+
uniqueness is exactly the concurrency model the metadata store needs.
212+
213+
```sh
214+
npm install oc-azure-table-metadata-adapter
215+
```
216+
217+
```js
218+
metadata: {
219+
adapter: require("oc-azure-table-metadata-adapter").default,
220+
options: {
221+
connectionString: process.env.OC_METADATA_TABLE_CONNECTION_STRING,
222+
},
223+
}
224+
```
225+
226+
Authentication precedence when `connectionString` is absent:
227+
`accountName` + `accountKey` → `sasToken` → explicit `credential` →
228+
`DefaultAzureCredential` (managed identity / workload identity / `az login`), so
229+
the registry can run with no secret. Other options include `endpoint`,
230+
`tableName` (default `occomponents`), `manageSchema` (default `true`), and
231+
`allowInsecureConnection` (for Azurite / local development). The adapter maps a
232+
`409 Conflict` to the shared `VERSION_ALREADY_EXISTS` code.
233+
234+
### Writing a custom adapter
235+
236+
Implement the contract from `oc-metadata-adapters-utils`. Each adapter maps its
237+
driver's unique-violation to the shared error codes (e.g. SQL Server
238+
`2627`/`2601`, Postgres `23505`, MySQL `1062`):
239+
240+
```ts
241+
import type { ComponentRow, MetadataStore } from "oc-metadata-adapters-utils";
242+
import {
243+
VERSION_ALREADY_EXISTS,
244+
VERSION_PUBLISH_IN_PROGRESS,
245+
} from "oc-metadata-adapters-utils";
246+
```
247+
248+
## Migrating an existing registry
249+
250+
Migration is **gradual and lossless** — storage stays authoritative-enough
251+
throughout the window so you can roll back at any point.
252+
253+
### 1. Backfill the database
254+
255+
Before serving traffic in metadata mode, populate the store from your existing
256+
index using the CLI command:
257+
258+
```sh
259+
oc registry migrate-metadata ./registry.config.js
260+
```
261+
262+
The argument is a path to a module exporting the **same config object** you pass
263+
to `registry.configure()`. It must include both `storage` and `metadata` and pass
264+
registry config validation. The module may be CommonJS, an ES module `default`
265+
export, or an **async function** returning the config (useful for resolving
266+
secrets first):
267+
268+
```js
269+
// registry.config.js — CommonJS
270+
const azureSqlMetadataAdapter = require("oc-azure-sql-metadata-adapter").default;
271+
const s3StorageAdapter = require("oc-s3-storage-adapter");
272+
273+
module.exports = {
274+
baseUrl: "http://my-registry.example.com/",
275+
storage: {
276+
adapter: s3StorageAdapter,
277+
options: {
278+
bucket: "my-bucket",
279+
region: "us-east-1",
280+
componentsDir: "components",
281+
path: "https://cdn.example.com/",
282+
},
283+
},
284+
metadata: {
285+
adapter: azureSqlMetadataAdapter,
286+
options: {
287+
connectionString: process.env.OC_METADATA_SQL_CONNECTION_STRING,
288+
},
289+
},
290+
};
291+
```
292+
293+
The command initialises the adapter and backfills rows from
294+
`${componentsDir}/components-details.json` (which already *is* the
295+
`ComponentRow` set). If that file is missing, it falls back to scanning
296+
`${componentsDir}/<component>/<version>/package.json`. Existing rows are skipped,
297+
so the command is **idempotent** and safe to re-run across nodes. It logs
298+
`{ scanned, inserted, skipped }` and closes the adapter pool on exit (even on
299+
failure).
300+
301+
### 2. Cut over
302+
303+
Deploy with the `metadata` block configured. During the migration window enable
304+
the bake-in flags:
305+
306+
```js
307+
metadata: {
308+
adapter: azureSqlMetadataAdapter,
309+
options: { /* ... */ },
310+
reconcileFromStorage: true, // heal anything published by still-storage-mode nodes
311+
exportLegacyFiles: true, // keep components.json fresh for rollback / external consumers
312+
}
313+
```
314+
315+
Nodes now hydrate from the database. A safe rollout order:
316+
317+
1. Deploy the metadata config to a non-serving environment.
318+
2. Run `oc registry migrate-metadata ./registry.config.js`.
319+
3. Start **one** registry instance in metadata mode and verify reads.
320+
4. Roll out metadata mode to the remaining instances.
321+
322+
### 3. Bake-in
323+
324+
Run mixed / observe. While the bake-in flags are on:
325+
326+
- **`reconcileFromStorage`** upserts, on each boot, any `name@version` that exists
327+
in the directory tree but is missing from the database — healing anything a
328+
node still running in storage mode published during the cutover.
329+
- **`exportLegacyFiles`** (optionally on `exportLegacyFilesInterval`) keeps
330+
`components.json` / `components-details.json` fresh in storage, so external
331+
consumers keep working and **rollback to storage mode loses at most one export
332+
interval**.
333+
334+
The directory tree remains authoritative-enough that the reconcile can heal any
335+
miss — the same self-healing principle storage mode relies on, applied
336+
deliberately once at the boundary.
337+
338+
### 4. Steady state
339+
340+
Once you're confident, drop the bake-in scaffolding:
341+
342+
- Set `reconcileFromStorage: false` (the directory scan is now abandoned).
343+
- Optionally keep `exportLegacyFiles: true` permanently as a cheap, one-way DR
344+
snapshot / cold-start fallback so the database is never an absolute single
345+
point of failure for booting.
346+
347+
`components.json` is now a **non-authoritative projection** of the database.
348+
349+
### Rolling back
350+
351+
Because the legacy files are kept fresh during bake-in, rolling back is simply
352+
removing the `metadata` block (or reverting the deploy): the registry returns to
353+
storage mode and reads the projected `components.json`, losing at most one export
354+
interval of updates.
355+
356+
## Failure model
357+
358+
- **Startup, DB down** — readiness fails and the node retries with backoff;
359+
already-running nodes keep serving from cache and the load balancer skips the
360+
not-ready node. The registry never starts silently empty.
361+
- **Poll, DB blip** — fully resilient: keep serving the in-memory cache, log, and
362+
retry next interval. The only effect is that new publishes propagate slightly
363+
later.
364+
- **Publish, DB unreachable** — the publish fails with a clear error; any statics
365+
uploaded are harmless orphans and the client can retry (idempotent). There is
366+
no buffering.
367+
368+
Net: **reads survive any DB blip, and publishes correctly refuse during one.**

‎website/sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ const sidebars = {
6666
items: [
6767
"components/publishing-to-a-registry",
6868
"registry/registry-configuration",
69+
"registry/registry-metadata-stores",
6970
"registry/registry-using-google-storage",
7071
],
7172
},

‎website/static/llms.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ The framework consists of four core components: CLI & Development Tools for comp
3838
## Registry & Infrastructure
3939

4040
- [Registry Configuration](https://opencomponents.github.io/docs/registry/registry-configuration.md): Production registry setup with S3, authentication, and advanced options
41+
- [Metadata Stores](https://opencomponents.github.io/docs/registry/registry-metadata-stores.md): Opt-in pluggable database as source of truth for the component index, plus migration steps for existing registries
4142
- [Google Cloud Storage](https://opencomponents.github.io/docs/registry/registry-using-google-storage.md): Alternative storage configuration
4243
- [Client Setup](https://opencomponents.github.io/docs/consumers/client-setup.md): Consumer application configuration
4344

0 commit comments

Comments
 (0)