[arpitg1304]

What

Every command that accepts a dataset path now also accepts s3:// and gs:// URIs, alongside local paths and hf:// URLs:

forge inspect s3://my-bucket/datasets/run_0413
forge convert gs://lab-data/rosbags ./out --format lerobot-v3
forge quality s3://my-bucket/datasets/droid --report report.html

Design

A single utility — forge/io/paths.py — resolves any source to a local path, downloading remote datasets to a temp dir (auto-cleaned at process exit) and handing them to the existing readers unchanged. All file access funnels through the CLI resolver (_resolve_dataset_path) and the quality/segment analyzers, so nothing is scattered.

Why download-to-temp instead of streaming? The format readers are deeply local-path-dependent — PyAV video keyframe seeks, HDF5 chunk reads, rosbag indexes, open()+json.load, and Path-based format detection. Downloading once keeps every reader (and its tests) working identically to local paths. The fsspec plumbing is in place to add true range-read streaming per-format later (noted in CHANGELOG).

Details

• fsspec is now a core dependency; s3fs/gcsfs are optional [s3]/[gcs] extras. Passing s3:// without s3fs fails with the exact pip install command.
• Auth delegates entirely to AWS/GCP default credential chains — no credential handling of our own.
• Writing outputs to cloud URIs is rejected fast with a clear message (writers remain local-only; documented limitation).
• Local path behavior is 100% unchanged.

Testing

• tests/test_io_paths.py: fsspec memory:// unit tests (always run), a moto mock-S3 test, and end-to-end lerobot-v3/zarr resolution + CLI tests. 18 new tests, all passing.
• Full existing suite: no regressions (pre-existing failures baselined against main are unchanged).
• Verified against a real S3-compatible server (local MinIO) with the actual lerobot/droid_100 dataset (464 MB, videos included): inspect, quality --sample, convert --dry-run, and cloud-output rejection all work. See forge/io/TESTING_MINIO.md.

Docs

• README cloud section (extras, examples, auth).
• forge/io/README.md — usage + a guide to diagnosing cloud connectivity issues.
• forge/io/TESTING_MINIO.md — local MinIO setup & testing.
• CHANGELOG.md.

🤖 Generated with Claude Code