Request for an Example of the 57D Egocentric Motion Input

[Zhangshy1019]

Hi, thank you for releasing this great project!

I am currently trying to use the Egocentric Motion input, but I am a bit confused about the format of the 57D action representation.

Could you please provide a simple example of the Egocentric Motion input (e.g., a sequence of 57D actions) or explain how each dimension is defined?

In particular, I would like to understand:

What coordinate system is used (camera coordinate, world coordinate, or another convention)?
What is the scale/unit of the translation values (meters, centimeters, normalized values, etc.)?
Are the rotation values represented as Euler angles, axis-angle, quaternions, or another format?
Is any normalization applied to the 57D action before feeding it into the model?

Having a concrete example or a description of the preprocessing would be extremely helpful for reproducing the results.

Thank you very much for your help!

[qianlim]

Hi, while we work on providing an example, here are some details on how to get the 57D hand pose feature.

action = [camera | R_wrist | R_fingers | L_wrist | L_fingers]

Camera pose: 3 (xyz) + 6 (rot6d) = 9D
R wrist pose: 3 (xyz) + 6 (rot6d) = 9D
R fingertips: 5 joints × 3 (xyz) = 15D
L wrist pose: 3 (xyz) + 6 (rot6d) = 9D
L fingertips: 5 joints × 3 (xyz) = 15D
──────────────────────────────────────────
Total: 9 + 9 + 15 + 9 + 15 = 57D

We use the standard 21-joint hand skeleton. Fingertip joint indices: (4, 8, 12, 16, 20), i.e. thumb tip through pinky tip. index 0 = wrist.

Unit: meters.

Rotation representation: 6D rotation, i.e. first two columns of a 3×3 rotation matrix (Zhou et al. CVPR 2019), see this paper.

How to get the features:

• Camera (ego motion): get world-frame camera-to-world pose from data, compute frame-to-frame delta inv(cam_t) @ cam_{t+1}. Why? This encodes head/viewpoint motion in camera's own coordinates at t.
• Wrists: raw wrist pose in camera frame → lifted to world via cam_c2w → compute frame-to-frame delta inv(wrist_world_t) @ wrist_world_{t+1}. Why? world-lift strips out camera ego-motion so wrist signal = pure hand trajectory in space.
• Fingertips: positions kept in camera frame, projected into per-frame wrist-local via inv(wrist_cam_space) @ finger_cam_space. World-lift is not applied here because finger and wrist share camera frame, so wrist-local result is frame-invariant.

[Zhangshy1019]

Hi, thanks for the clarification — this is very helpful. I tried to reproduce the 57D input using camera pose estimated by ViPE and hand pose estimated by WiLoR with the standard 21-joint hand skeleton. However, I still cannot get reasonable results on my side. Would it be possible for you to provide a *direct runnable test example* for this 57D feature extraction pipeline? For example, even a minimal case with: - one short input sequence, - the corresponding camera pose / hand pose inputs, - and the final expected 57D output would be extremely helpful for me to quickly verify whether my implementation is correct. At the moment, I am not sure whether the issue comes from: - the camera pose convention / coordinate system, - the wrist world-lift step, - the wrist-local fingertip projection, - or possible differences between your hand-pose source and WiLoR outputs. If possible, a small reference script or a sample npy/json output would greatly help me debug and validate the pipeline quickly. Thanks a lot! Qianli Ma ***@***.***> 于2026年6月11日周四 13:55写道：

…

*qianlim* left a comment (NVIDIA/cosmos-framework#31) <#31 (comment)> Hi, while we work on providing an example, here are some details on how to get the 57D hand pose feature. action = [camera | R_wrist | R_fingers | L_wrist | L_fingers] Camera pose: 3 (xyz) + 6 (rot6d) = 9D R wrist pose: 3 (xyz) + 6 (rot6d) = 9D R fingertips: 5 joints × 3 (xyz) = 15D L wrist pose: 3 (xyz) + 6 (rot6d) = 9D L fingertips: 5 joints × 3 (xyz) = 15D ────────────────────────────────────────── Total: 9 + 9 + 15 + 9 + 15 = 57D We use the standard 21-joint hand skeleton. Fingertip joint indices: (4, 8, 12, 16, 20), i.e. thumb tip through pinky tip. index 0 = wrist. Unit: meters. Rotation representation: 6D rotation, i.e. first two columns of a 3×3 rotation matrix (Zhou et al. CVPR 2019), see this paper <https://openaccess.thecvf.com/content_CVPR_2019/papers/Zhou_On_the_Continuity_of_Rotation_Representations_in_Neural_Networks_CVPR_2019_paper.pdf> . How to get the features: - Camera (ego motion): get world-frame camera-to-world pose from data, compute frame-to-frame delta inv(cam_t) @ cam_{t+1}. Why? This encodes head/viewpoint motion in camera's own coordinates at t. - Wrists: raw wrist pose in camera frame → lifted to world via cam_c2w → compute frame-to-frame delta inv(wrist_world_t) @ wrist_world_{t+1}. Why? world-lift strips out camera ego-motion so wrist signal = pure hand trajectory in space. - Fingertips: positions kept in camera frame, projected into per-frame wrist-local via inv(wrist_cam_space) @ finger_cam_space. World-lift is not applied here because finger and wrist share camera frame, so wrist-local result is frame-invariant. — Reply to this email directly, view it on GitHub <#31?email_source=notifications&email_token=BFWQA7YWZZV7IFDZUX7MTSL47JCUZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRXG43DEMBWGEZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4677620613>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BFWQA75PIIQKDRVRMTI6UWL47JCUZAVCNFSNUABGKJSXA33TNF2G64TZHMYTENBTG42TOMBYGY5US43TOVSTWNBWGI4TAOBSGQ4THILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BFWQA77RX3362LBWZPEXL7L47JCUZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRXG43DEMBWGEZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/BFWQA756AXXUY5CKP7H3SUT47JCUZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRXG43DEMBWGEZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[qianlim]

Thanks for the detailed report. I re-checked the data path and realized my earlier answer was missing a few important convention details.

The 57D layout I described is the raw action representation. In the Action model data path, the raw 57D action can then be transformed by the dataset wrapper before it is fed to the model. For the current hand-pose Action configs, this means quantile_rot normalization using stats computed on the raw 57D channels, followed by zero-padding to the model action width, which is 64D.

So there are two useful outputs to distinguish:

• raw 57D action: best for debugging geometry and coordinate conventions;
• model action: normalized + padded action, matching what the model consumes.

There is also a wrist-frame convention detail. For one of our source data formats, after constructing the wrist pose in the camera frame, we apply a fixed wrist-frame alignment by post-multiplying the wrist transform:

WRIST_FRAME_ALIGN_EXAMPLE = np.array([
    [ 0, 1, 0, 0],
    [-1, 0, 0, 0],
    [ 0, 0, 1, 0],
    [ 0, 0, 0, 1],
], dtype=np.float32)

wrist_cam = wrist_cam @ WRIST_FRAME_ALIGN_EXAMPLE

Note that this matrix is not universal. It is specific to that source data format. What matters is the target wrist-local anatomical convention:

X = thumb-side to pinky-side direction (for right hand)
Y = outward palm normal
Z = wrist toward fingertips

If your wrist quaternions come from another estimator such as WiLoR, the correct alignment may be identity or a different fixed rotation. The important check is whether the quaternion’s local XYZ axes match the anatomical directions above. The 21 joint positions can be used to estimate those anatomical directions and diagnose whether an extra wrist-frame alignment is needed.

The raw-action input conventions are:

• Input pose sequence length: T + 1
• Output raw action length: T
• camera_position: shape [T+1, 3], meters, camera position in world
• camera_rotation: shape [T+1, 4], xyzw quaternion, camera-to-world rotation
• hand_{left,right}_cam: shape [T+1, 21*3], 21 joints in the current camera frame, meters
• hand_{left,right}_cam_rotation: shape [T+1, 21*4], xyzw quaternions.
• The wrist quaternion, joint 0, is used for the wrist pose.

Quaternion is not a must for raw data; it's just our choice to save space.

I will follow up with a small example package and script that reproduces the conversion from pose arrays to raw 57D action.

[Zhangshy1019]

Thanks for the feedback! Qianli Ma ***@***.***> 于2026年6月27日周六 07:47写道：

…

*qianlim* left a comment (NVIDIA/cosmos-framework#31) <#31 (comment)> Thanks for the detailed report. I re-checked the data path and realized my earlier answer was missing a few important convention details. The 57D layout I described is the raw action representation. In the Action model data path, the raw 57D action can then be transformed by the dataset wrapper before it is fed to the model. For the current hand-pose Action configs, this means quantile_rot normalization using stats computed on the raw 57D channels, followed by zero-padding to the model action width, which is 64D. So there are two useful outputs to distinguish: - raw 57D action: best for debugging geometry and coordinate conventions; - model action: normalized + padded action, matching what the model consumes. There is also a wrist-frame convention detail. For one of our source data formats, after constructing the wrist pose in the camera frame, we apply a fixed wrist-frame alignment by post-multiplying the wrist transform: WRIST_FRAME_ALIGN_EXAMPLE = np.array([ [ 0, 1, 0, 0], [-1, 0, 0, 0], [ 0, 0, 1, 0], [ 0, 0, 0, 1], ], dtype=np.float32) wrist_cam = wrist_cam @ WRIST_FRAME_ALIGN_EXAMPLE Note that this matrix is not universal. It is specific to that source data format. What matters is the target wrist-local anatomical convention: X = thumb-side to pinky-side direction (for right hand) Y = outward palm normal Z = wrist toward fingertips If your wrist quaternions come from another estimator such as WiLoR, the correct alignment may be identity or a different fixed rotation. The important check is whether the quaternion’s local XYZ axes match the anatomical directions above. The 21 joint positions can be used to estimate those anatomical directions and diagnose whether an extra wrist-frame alignment is needed. The raw-action input conventions are: - Input pose sequence length: T + 1 - Output raw action length: T - camera_position: shape [T+1, 3], meters, camera position in world - camera_rotation: shape [T+1, 4], xyzw quaternion, camera-to-world rotation - hand_{left,right}_cam: shape [T+1, 21*3], 21 joints in the current camera frame, meters - hand_{left,right}_cam_rotation: shape [T+1, 21*4], xyzw quaternions. - The wrist quaternion, joint 0, is used for the wrist pose. Quaternion is not a must for raw data; it's just our choice to save space. I will follow up with a small example package and script that reproduces the conversion from pose arrays to raw 57D action. — Reply to this email directly, view it on GitHub <#31?email_source=notifications&email_token=BFWQA74I3NF7IWHQXGI4WID5B4DP7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBRGQYTMMZVGMZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4814163533>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BFWQA72FDFXNN5IBDV2READ5B4DP7AVCNFSNUABGKJSXA33TNF2G64TZHMYTENBTG42TOMBYGY5US43TOVSTWNBWGI4TAOBSGQ4THILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BFWQA72724HCOHZM37DQUP35B4DP7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBRGQYTMMZVGMZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/BFWQA7ZQWDGYHOWYA3IB6XT5B4DP7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBRGQYTMMZVGMZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[qianlim]

@Zhangshy1019 Added examples and codes in NVIDIA/cosmos#246

[Zhangshy1019]

Thank you very much for providing this example—it was extremely helpful! Following your example, I converted the example data into the raw 57D action format and used it as input for Cosmos-Nano inference. However, the generated results don't seem to match the expected behavior. I noticed in the README that, under *Model Action Space*, an action_stats.json file is required for action normalization. I was wondering whether there is a corresponding action_stats.json available for the Cosmos-Nano 57D egocentric action space. If such a file exists, could you please point me to where I can find it? If not, is there a recommended way to generate or estimate it for custom 57D action data? Thank you again for your time and for sharing your excellent work! Qianli Ma ***@***.***> 于2026年6月29日周一 10:36写道：

…

*qianlim* left a comment (NVIDIA/cosmos-framework#31) <#31 (comment)> @Zhangshy1019 <https://github.com/Zhangshy1019> Added examples and codes in NVIDIA/cosmos#246 <NVIDIA/cosmos#246> — Reply to this email directly, view it on GitHub <#31?email_source=notifications&email_token=BFWQA74CZ7X2OSDSY4TWP2T5CHI4BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBSHA2DMOBTGQ42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4828468349>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BFWQA73LJEWBKTYT5YB7MPD5CHI4BAVCNFSNUABGKJSXA33TNF2G64TZHMYTENBTG42TOMBYGY5US43TOVSTWNBWGI4TAOBSGQ4THILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BFWQA76QFJ6LL5Z6JKHY76D5CHI4BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBSHA2DMOBTGQ42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/BFWQA7YHOBQ35QFACR63WLT5CHI4BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBSHA2DMOBTGQ42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[benjiebob]

@qianlim Yes I hit the same wall as @Zhangshy1019. I think we need the action_stats.json to be able to use it for inference.

Any chance you could share please <3

[qianlim]

@Zhangshy1019 @benjiebob the hand pose stats file is now available - could you give this one a try? https://github.com/NVIDIA/cosmos-framework/blob/main/cosmos_framework/data/generator/action/normalizer_stats/human_hand_pose_lerobot_stats.json

[benjiebob]

By the way, this worked great for me. Thank you!

Any chance you folks would consider releasing some more egohand examples for testing?