The catalog (Phase 1): a queryable, append-only episode registry

CHANGELOG.md

@@ -9,6 +9,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **The catalog (Phase 1) — an append-only, queryable registry of episodes.**
+  Turns Forge from a per-dataset tool into a system of record: a set of
+  append-only Parquet tables (`episodes`, `quality_scores`) written with pyarrow
+  and queried with embedded DuckDB, on a local directory or an `s3://` / `gs://`
+  bucket.
+
+  ```bash
+  pip install "forge-robotics[catalog]"
+  forge catalog init ./forge-catalog
+  forge ingest ./my_dataset --catalog ./forge-catalog
+  forge query "SELECT task, count(*) FROM episodes GROUP BY task" -c ./forge-catalog
+  forge catalog stats --catalog ./forge-catalog
+  ```
+
+  - New module `forge/catalog/` with a `Catalog` class as the single entry
+    point (`from forge.catalog import Catalog`), plus `forge/catalog/ingest.py`.
+  - New CLI commands: `forge catalog init`, `forge catalog stats`,
+    `forge ingest`, `forge query` (SQL over `episodes`, `quality_scores`, and a
+    `v_latest_quality` view; `--format table|json|csv`).
+  - **Ingestion reuses existing internals** — the same format readers behind
+    `forge inspect` for metadata and `QualityAnalyzer.analyze_episode` (the
+    engine behind `forge quality`) for scoring. No metadata extraction or
+    scoring logic was reimplemented, and no existing behavior changed.
+  - **Idempotent & crash-safe.** Episodes are keyed by a content hash, so
+    re-running an ingest over the same sources is a no-op. Each flush commits
+    atomically via a manifest-last protocol; queries read only manifested
+    part-files, so a crash never exposes a partial batch.
+  - New deps behind the `[catalog]` extra: `duckdb`, `xxhash` (and an explicit
+    `pyarrow`). The base CLI never imports them — they load only when a catalog
+    command runs.
+  - See [forge/catalog/README.md](forge/catalog/README.md) and
+    [docs/forge_data_engine_design.md](docs/forge_data_engine_design.md).
+
 - **Cloud storage support (`s3://`, `gs://`).** Every command that accepts a
   dataset path now also accepts Amazon S3 and Google Cloud Storage URIs, in
   addition to local paths and `hf://` URLs. For example:

README.md

@@ -250,6 +250,45 @@ forge dedup ./my_dataset ./deduped --method dhash        # phash (default) | dha
 
 See [forge/dedup/README.md](forge/dedup/README.md) for the algorithm and tuning.
 
+## The catalog
+
+Forge is a per-dataset tool by default. The **catalog** turns it into a *system of record*: an append-only set of Parquet tables that registers every episode you ingest and annotates it with quality scores, all queryable with SQL. It's zero-server (just Parquet + embedded DuckDB), works on a local directory or an `s3://` / `gs://` bucket, and is readable by pandas/Polars/Spark without Forge.
+
+```bash
+pip install "forge-robotics[catalog]"
+
+# 1. Create a catalog (local dir or cloud bucket)
+forge catalog init ./forge-catalog
+
+# 2. Ingest datasets — registers + quality-scores each episode.
+#    Re-running is a no-op (episodes are skipped by content hash).
+forge ingest ./my_dataset --catalog ./forge-catalog
+forge ingest s3://lab-bucket/raw/2026-07-18/ -c ./forge-catalog
+
+# 3. Query with SQL (views: episodes, quality_scores, v_latest_quality)
+forge query "SELECT task, count(*) FROM episodes GROUP BY task" -c ./forge-catalog
+forge query "SELECT e.language_instruction, q.overall_score
+             FROM episodes e JOIN v_latest_quality q USING(episode_id)
+             ORDER BY q.overall_score DESC LIMIT 10" -c ./forge-catalog --format json
+
+# 4. Summary stats
+forge catalog stats --catalog ./forge-catalog
+```
+
+Python API:
+
+```python
+from forge.catalog import Catalog
+from forge.catalog.ingest import ingest
+
+cat = Catalog.init("s3://lab-bucket/forge-catalog")
+ingest(["s3://lab-bucket/raw/2026-07-18/"], cat)
+df = cat.sql("SELECT robot, avg(overall_score) FROM episodes "
+             "JOIN v_latest_quality USING(episode_id) GROUP BY robot").to_pandas()
+```
+
+Ingestion reuses Forge's existing readers (the metadata behind `forge inspect`) and scorer (the engine behind `forge quality`), so the catalog stays consistent with the rest of the toolkit. Writes go through pyarrow; reads through DuckDB; nothing else touches catalog files. See [forge/catalog/README.md](forge/catalog/README.md) for the architecture, storage layout, and commit protocol.
+
 ## Dataset Registry
 
 A curated catalog of 23+ prominent robotics datasets — browse, search, and download by name instead of memorizing URIs. **[Browse the registry online](https://arpitg1304.github.io/forge/registry.html)**