Skip to content

Wire user-provided pinhole calibration into the standard input paths - #106

Open
pb-evercoast wants to merge 1 commit into
nv-tlabs:mainfrom
pb-evercoast:evercoast/calibrated-frame-dir-input
Open

pb-evercoast wants to merge 1 commit into
nv-tlabs:mainfrom
pb-evercoast:evercoast/calibrated-frame-dir-input

Conversation

@pb-evercoast

Copy link
Copy Markdown

Evidence, camera model, seeds, results and the reproduction package are in #105. This PR
is the calibrated-input wiring that issue asks for. Closes #56; related to #37 and #103.

pipeline.init.intrinsics=gt is exposed by the typed configuration, and
slam.optimize_intrinsics already resolves to false when it is selected. But neither
vipe infer video.mp4 nor vipe infer --image-dir has a way to hand a calibration to the
pipeline: RawMp4Stream and FrameDirStream never populate frame intrinsics, and
DefaultAnnotationPipeline unconditionally appends GeoCalib and asserts that no intrinsics
are present. Selecting gt from the CLI therefore fails for every user who already knows
their camera — which is the common case for rendered, rig-calibrated or benchmark footage
(#37, #56).

This PR completes both paths with no change to the default behaviour:

  • vipe infer <video.mp4 | VIDEO_DIR | --image-dir DIR> --intrinsics calibration.json
    accepts one pinhole calibration ({"width", "height", "fx", "fy", "cx", "cy"}), validates
    it before any model is loaded, checks it against the frame size of every input it is
    applied to (a directory of videos shares one camera; each video is checked in turn and a
    mismatch names the video), and selects pipeline.init.intrinsics=gt (which disables
    intrinsic optimization through the existing ${neq:...} resolver).
  • RawMp4Stream and FrameDirStream take an optional [fx, fy, cx, cy] tensor, validate
    it (shape, finiteness, positive focal lengths), stamp a per-frame copy plus
    CameraType.PINHOLE on every VideoFrame, and advertise INTRINSICS and CAMERA_TYPE
    in attributes() so the pipeline's attribute check sees them through the cache wrapper.
  • DefaultAnnotationPipeline._add_init_processors branches on init.intrinsics:
    geocalib keeps the current behaviour and assertions unchanged; gt skips GeoCalib and
    refuses a stream that does not carry intrinsics and a camera type, rather than silently
    estimating them; any other value raises. PoseOnlyAnnotationPipeline and
    PoseOnlyLongAnnotationPipeline inherit this method, so the calibrated path is available
    to them without a change of their own.
    Nothing changes for an invocation without --intrinsics: no new flag is required, no new
    file is written, and the RNG state is untouched.

The --seed flag and the resolved-config.json record that the ablation in #105 relied on are
deliberately not in this PR. They are a separate concern (a per-run record of the resolved
configuration, not a determinism claim) and can follow as their own change if wanted.

Tests

tests/test_calibrated_frame_dir_input.py (pytest, same layout as the existing suite):

  • streams, both FrameDirStream and RawMp4Stream: uncalibrated advertises nothing;
    calibrated advertises exactly {INTRINSICS, CAMERA_TYPE} and the advertisement survives
    ProcessedVideoStream; malformed / non-finite / non-positive-focal intrinsics are refused
    with the documented messages; every frame carries its own copy of the intrinsics on CUDA
    (GPU-only);
  • config: init.intrinsics=gt resolves slam.optimize_intrinsics=false, geocalib
    keeps it true;
  • pipeline: gt refuses a stream without intrinsics, adds no GeoCalib processor for a
    calibrated stream, and the geocalib branch still asserts against pre-existing
    intrinsics;
  • CLI: --intrinsics is accepted on the video path as on the frame directory; malformed
    calibration JSON is rejected before any model loads and before the output directory
    exists; a calibration whose dimensions disagree with the frames is rejected (GPU-only);
    a directory of videos is checked video by video and the refusal names the offending file
    (GPU-only).

CUDA-dependent cases are skipif-guarded; the rest run on CPU.

Compatibility

Pure Python; no changes to vipe_ext or the compiled extensions. The v1.2.0 form was
exercised on an A100 against torch 2.13.0+cu130; the upstream CI matrix (cu124 /
cu128) uses only the same basic tensor API. The rebased form keeps upstream's new
behaviour intact: the eager .cache() is still skipped for the long-sequence pipeline, and
the per-video loop is unchanged apart from the calibration check.

What this PR does not do

  • It does not add a --seed flag or write a record of the resolved configuration; see above.
  • It does not add distortion coefficients to the calibration file. VideoFrame.intrinsics
    already documents a (4+D,) layout; a follow-up could accept k1.. for the camera
    models that consume them. This PR is pinhole-only on purpose.
  • It does not change any default or config file. Without --intrinsics every path behaves
    exactly as before.

Tests on hardware

On an A100 (sm_80), from a clean clone at main 8c9f361 with this commit applied by git am and vipe_ext rebuilt: tests/test_calibrated_frame_dir_input.py 28 passed, and the whole tests/ suite on the same tree 108 passed. CUDA-dependent cases are skipif-guarded for CPU CI.

`pipeline.init.intrinsics=gt` is exposed by the typed configuration and
`slam.optimize_intrinsics` already follows it, but neither
`vipe infer video.mp4` nor `vipe infer --image-dir` had a way to hand a
calibration to the pipeline: RawMp4Stream and FrameDirStream never
populated frame intrinsics, and DefaultAnnotationPipeline unconditionally
appended GeoCalib and asserted that no intrinsics were present. Selecting
`gt` from the CLI therefore failed for every user who already knows their
camera.

- `vipe infer <video.mp4 | VIDEO_DIR | --image-dir DIR> --intrinsics calibration.json`
  accepts one pinhole calibration (`width, height, fx, fy, cx, cy`),
  validates it before any model is loaded, checks it against the frame
  size of every input it is applied to (each video of a directory is
  checked in turn and a mismatch names the video), and selects
  `pipeline.init.intrinsics=gt` (which disables intrinsic optimization
  through the existing config resolver).
- RawMp4Stream and FrameDirStream take an optional `[fx, fy, cx, cy]`
  tensor, stamp a per-frame copy plus `CameraType.PINHOLE` on every
  VideoFrame, and advertise INTRINSICS and CAMERA_TYPE in `attributes()`
  so the pipeline's attribute check sees them through the cache wrapper.
- DefaultAnnotationPipeline branches on `init.intrinsics`: `geocalib`
  keeps the current behaviour and assertions; `gt` skips GeoCalib and
  refuses a stream that does not carry intrinsics and a camera type. The
  pose-only pipelines inherit `_add_init_processors`, so they gain the
  same branch without a change of their own.

tests/test_calibrated_frame_dir_input.py covers both streams' validation
and attribute advertisement, the config resolver, the pipeline branch,
and CLI validation on both input paths and on a directory of videos
(CUDA-dependent cases are skipped without a GPU).

Nothing changes for an invocation without --intrinsics.

Closes nv-tlabs#56. Related: nv-tlabs#37 (closed with a suggestion to hand-edit the processor).

Signed-off-by: Peter Blake <peter@evercoast.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run with calibrated videos

1 participant