[arpitg1304]

What

Turns Forge from a per-dataset tool into a system of record: an append-only set of Parquet tables — episodes + quality_scores — written with pyarrow and queried with embedded DuckDB, on a local directory or an s3:// / gs:// bucket.

pip install "forge-robotics[catalog]"
forge catalog init ./forge-catalog
forge ingest ./my_dataset --catalog ./forge-catalog          # register + quality-score each episode
forge query "SELECT task, count(*) FROM episodes GROUP BY task" -c ./forge-catalog
forge catalog stats --catalog ./forge-catalog

This is Phase 1 of the Forge Data Engine design (embeddings/dedup/curation/snapshots are Phases 2–4).

New module forge/catalog/

file	role
schema.py	pyarrow schemas, SCHEMA_VERSION, SCORER_VERSION, validate_rows() (fails loudly)
writer.py	the only writer — atomic part-files + manifest-last commit protocol
catalog.py	Catalog class: init/open (version guard), sql(), commit_batch, stats(), views; manifest-driven reads
ingest.py	discover → content-hash → skip-if-exists → register → score
cli.py	forge catalog init/stats, forge ingest, forge query

Design highlights

• Reuses existing internals — the readers behind forge inspect for metadata and QualityAnalyzer.analyze_episode (the engine behind forge quality) for scoring. No metadata/scoring logic reimplemented; no existing behavior changed. (No refactor was needed — both were already cleanly importable.)
• Idempotent — episodes keyed by an xxh3 content hash of their proprio stream; re-running an ingest over the same sources is a no-op.
• Crash-safe — each flush commits atomically (manifest written last); queries read only manifested part-files, so a crash never exposes a partial batch. Recovery is automatic via hash-skip.
• Cloud-native — catalog and/or source can live in S3/GCS; DuckDB reads through the same fsspec client as the writer (works with real S3 and local MinIO). Builds on the cloud I/O from Add S3/GCS cloud storage support via fsspec #29.
• Base-install safe — duckdb/xxhash/pyarrow live behind the [catalog] extra; forge --help never imports them.
• Open & portable — plain partitioned Parquet, readable by pandas/Polars/DuckDB/Spark without Forge.

Quality columns

quality_scores mirrors EpisodeQuality's flat metric names exactly: ldlj, sparc, dead_fraction, gripper_chatter_rate, joint_path_length, overall_saturation, mean_entropy, psd_high_fraction, state_action_consistency, static_fraction, jitter_ratio (+ overall_score, flags), one row per (episode_id, scorer_version).

Testing

• tests/test_catalog.py — 22 tests, all passing: schema validation, manifest-last commit protocol + crash recovery, hash-based idempotent re-ingest, v_latest_quality two-version selection, synthetic-dataset ingest→query→stats integration, memory:// cloud, and CLI.
• Full existing suite: no regressions (pre-existing failures baselined against main are unchanged).
• Verified end-to-end on the real lerobot/droid_100 — local and fully-cloud (S3 source → S3 catalog against MinIO).

Docs & example

• README "The catalog" section; forge/catalog/README.md (architecture, storage layout, commit protocol); CHANGELOG entry.
• forge/catalog/catalog_example_droid_100/ — a small, ready-to-query example catalog built from droid_100, with its own README. Excluded from the built wheel/sdist (verified) but kept in the repo for demos.

🤖 Generated with Claude Code