Add support for Spatial pinch input on visionOS

[oPinon]

Feature

This pull request adds support for Spatial Events from visionOS (when you look at your game and pinch and/or drag).

Here is a demo project using the feature in Godot:
spatial-events.zip

Spatial.Events.test_hevc.mp4

The events map to XRController3D nodes:

XRController3D tracker name	Data	Actions
"left_hand"	Transform of the left hand while pinching; or left PSVR2 controller if held	"trigger_click"
"right_hand"	Transform of the right hand while pinching; or right PSVR2 controller if held	"trigger_click"
"/user/eyes_ext"	Eye ray from the last pinch (either from left or right hand, depending on which was the latest)	none

Pinch events are sent as "trigger_click" actions on "left_hand" and "right_hand".

Ray

The "/user/eyes_ext" starts from your head and goes towards the direction that you are looking at. It only moves during the initial pinch and does not move while you drag. The ray goes towards -Z, relative to the controller.

Pinch

The "left_hand" and "right_hand" follow your hand. You can combine them with the ray above to implement drag gestures.

Compatibility with hand tracking and controller tracking

Note that this is compatible with hand tracking and controller tracking, but does not require them.

This PR shares the same trackers used for PSVR2 controller tracking ("left_hand" and "right_hand"), but dynamically switches between the two during a pinch.

Here is a video showing that controller tracking still works:

Hands.+.controllers.demo_hevc.mp4

Differences with other platforms

This PR tries to map the visionOS API to existing Godot concepts, but it is not a 1-1 match. While visionOS also supports hand tracking and controller tracking, the "spatial events" are the default interaction model, and work a bit differently: the main difference is using eye tracking to target objects in the scene or target the UI, combined with hand tracking to do various gestures.

[BastiaanOlij]

Hmmm, I'm not sure if we need all the new trackers here.

If the user is holding controllers, say the Sony PSVR2 ones, then yes, those use the left_hand and right_hand trackers, but in that situation the spatial pinch inputs aren't usable.

This only applies when the user isn't holding controllers, and in that case we can fully mimic the behavior in OpenXR.

So instead of introducing visionos/left_hand_pinch and visionos/right_hand_pinch, we can just use the left_hand and right_hand trackers, with the pinch indeed mapped to trigger. It just requires some working together with the controller implementation. Either one, or the other, takes (temporary) ownership of the tracker.

With visionos/left_hand_ray and visionos/right_hand_ray, I'm a little less certain what the best way of working is. It seems like here you're trying to combine eye tracking with pinch gestures. The question here is whether the left hand pinch is linked to the left eye, and right hand pinch is linked to the right eye. I am assuming this is not the case, and that we have a single ray based on where the user is focusing, and are repeating the pinch signals here.

The way this works in OpenXR, and what I think we should mimic here, is that we have a single tracker (user/eyes_ext, which we should really rename to something more generic) that purely exposes the gaze direction.

This way game logic can react to the "trigger" signal on the respective hands, use the eye tracker for the ray, and seamlessly switch between a user doing pinch while looking, and using controller while looking, and do so in a way that is platform agnostic.

[oPinon]

@BastiaanOlij what you are describing seems to be more inline with other XR platforms, that lean more on VR controllers, indeed. My PR currently tries to stay as close as possible to the low-level visionOS APIs, while still using the XRController3D types.

Having the design be closer to visionOS allows games to be more similar to the rest of visionOS; but the downside is that it will be more work to port Godot games from other platforms to visionOS.

What do you think of the following solution, that combines the best of both worlds:

• we add trackers that try to emulate the same behavior as other platforms, to make it easy to port games, like you are suggesting
• we keep trackers specific to visionOS, like I currently have, to give fine controls to devs who need it

?

Using "left_hand" and "right_hand"

I did that in a previous version, but it caused issues when combining PSVR2 controllers and pinch events in the same game (I couldn't achieve what I'm showing in the video below):

Controller.conflict.mp4

However, we can probably solve that by having "left_hand" and "right_hand" dynamically switch between the PSVR2 controllers and the "spatial events" when you put the PSVR2 controllers in your hands, like you are suggesting.

Eye rays

The question here is whether the left hand pinch is linked to the left eye, and right hand pinch is linked to the right eye. I am assuming this is not the case, and that we have a single ray based on where the user is focusing, and are repeating the pinch signals here.

Indeed, it's the latter: the ray is for both eyes, but one ray is for the left hand and the other ray is for the right hand.

I had a previous version where I had a single "user/eyes_ext" controller, that was controlled by the last hand that pinched. I'm ok with either approaches. The main difference with having a single ray is that it requires a bit of state tracking to make sure that both hands are not fighting for the same ray when you pinch with both hands at the same time.

Emulating VR controllers with spatial events

I agree that it would be very useful to emulate VR controllers with the visionOS spatial events. It will make it much easier to port games to visionOS and to re-use UI code such as this:
https://github.com/godotengine/godot-demo-projects/blob/master/xr/openxr_composition_layers/handle_pointers.gd

I did that in my sample above, where the emulated controller is the red ray:
https://github.com/user-attachments/assets/be41e280-5b1e-4130-93cc-32523018bccb

Next steps for this PR

If you agree, I'm going to update this PR with:

1. add emulated VR controllers on "left_hand" and "right_hand" (similar to the red ray above), to make it easy to port games to visionOS
2. dynamically switch "{left/right}_hand" between PSVR2 controllers and spatial events, when you start holding controllers, to support both in the same game
3. use a single XRController3D for the ray, "user/eyes_ext", instead of the two rays that I currently have
4. keep "visionos/left_hand_pinch" and "visionos/right_hand_pinch" for developers who want the rotation of the hands. Because if we only do "{left/right}_hand", those will have the orientation of the eyes instead of the hands

What do you think?

[BastiaanOlij]

Hey Olivier,

I think that is where we differ in stance. I find it far more important to have cross platform synergy, especially if the visionOS behavior can still be achieved by simply knowing where the data comes from. We should only have visionOS specific implementations if there is behavior we can't replicate any other way.

So with your 4 conclusions, I'm only 75% on board:

1. Indeed, emulate left hand and right hand controllers when hand tracking is used, but DON'T use the eye ray, use the pinch pose as it's aim pose. You could even grab more information from the hand tracking system and add proper default, grip and palm poses to this to make it as compatible with OpenXR and WebXR as possible.
2. Fully agree with this, if a controller exist, it drives the tracker, if not, we emulate with the hand tracking and spatial pinch data.
3. Fully agree, just expose the ray by creating a users/eyes_ext tracker
4. I would not do this, if you must duplicate the eye ray poses on the hand trackers, introduce a visionOS only pose for the left and right hand trackers.

I don't like 4, because it introduces visionOS only behavior that can easily be reproduced in a cross platform way by reacting to the trigger action and querying the users/eyes_ext tracker (/XRNode3D) which has that same pose.

[dsnopek]

While I broadly agree with Bastiaan, that the left_hand and right_hand trackers should automatically switch between controllers and hand tracking in a way that's compatible with other interfaces, I think it'd be fine to add additional trackers on top of that which add platform-specific functionality.

For example, in OpenXR, we have the /user/eyes_ext based on the XR_EXT_eye_gaze_interaction extension with a single tracker. But, additionally, on Android XR, there's the XR_ANDROID_eye_tracking extension which provides a tracker for each eye (/user/eye_tracker_android/left and /user/eye_tracker_android/right in the implementation in godot_openxr_vendors).

Similarly, in WebXR, there's some odd platform-specific events (for example, the touch events) where we get a 3D position and ray, but it only updates when the event happens. This can't really be mapped to any of our standard trackers, but it's still a decent fit for a tracker (since we get a 3D position and ray), so these are exposed custom trackers.

So, I think it'd be fine for visionOS to add some extra trackers for platform-specific stuff, that doesn't map to things from other platforms.

[oPinon]

Ok, to summarize the plan (@BastiaanOlij if I understood correctly), we will have the following trackers:

XRController3D tracker name	Data
"left_hand"	Transform of the left hand while pinching; or left PSVR2 controller if held
"right_hand"	Transform of the right hand while pinching; or right PSVR2 controller if held
"user/eyes_ext"	Eye ray from the last pinch (either from left or right hand, depending on which was the latest)

I think that this would work, and provide all the needed data. I should be able to rework my sample above to behave the exact same. And we won't really need the extra trackers, since all the data can be derived from the above (unless you prefer the additional trackers, @dsnopek).

My only concern with this plan is that it does not exactly emulate how other VR platforms work: especially the "laser pointer" interaction (from SteamVR or Meta Quest with controllers) where you aim with a laser coming from your VR controllers. With the plan above, if the Godot game is ported unchanged, you would be aiming with your hands instead of your eyes, which will feel strange on visionOS.

[BastiaanOlij]

My only concern with this plan is that it does not exactly emulate how other VR platforms work: especially the "laser pointer" interaction (from SteamVR or Meta Quest with controllers) where you aim with a laser coming from your VR controllers. With the plan above, if the Godot game is ported unchanged, you would be aiming with your hands instead of your eyes, which will feel strange on visionOS.

Thats a mistaken assumption Oliver.

Yes, a game that never thought of using eye tracking, and is using hand tracking with lasers, will thus behave the same on AVP, and not to the expectation of AVP devs. But you have to understand that in this scenario, it is very likely the developer has done many more things around the hands, such as actually visualising the laser pointers, and suddenly moving the hand position to where the players head is, will cause a lot of problems. This is a game breaking behavioral change.

This is also why I mentioned to have the ray positions as the aim pose, while using hand tracking data to simulate grip and palm poses as there is an expectation that those work as well.

Developers who are doing the proper thing, e.g. using the eye tracker with pinch gestures, for them the functionality will be equal and predictable on AVP and other devices. Their game is already designed with the assumption the ray comes from the head, and they do not have the expectation that the position given is near the hand.
That this becomes the default setup we promote with AVP so AVP developers get the behavior they are expecting, that is a given.

@dsnopek I agree with you, if we have functionality that does not fit our more generic solution, introducing extra trackers makes sence. I am taking a bit of leeway here because the eye tracker, is not generic but specific to the eye gaze OpenXR extension. But I feel we should make it generic as this functionality is starting to become more prevalant.

If we do want to center ray pose available on the hands, in this case I wouldn't go for extra trackers, but just adding extra poses to the left and right hand tracker. This way there is no duplication of inputs, and a developer can easily check if the poses are active and available and switch behavior accordingly.

[BastiaanOlij]

Also @oPinon , am I correct in understanding that we do not get an eye ray pose when the user is not pinching?

That would cause a lot of UI issues as well. A dev will want to highlight the object or UI element the user is about the interact with. I definately want that behavior as a user. I have been playing a lot of Augmental puzzles lately, I can't imagine that game working without being able to know which cell I'm about to interact with.

[oPinon]

am I correct in understanding that we do not get an eye ray pose when the user is not pinching?

Yes, you are correct. The app only gets the eye ray pose during a pinch event. And the eye ray does not move while the user is dragging; it's only at the start of a gesture.

There is a way to have hover effects on visionOS, by having the OS do the hit-testing and render the hover effects, rather than the app. The reason why the app does not get the eye ray is to protect the privacy of the user.

For CompositorServices apps, it means providing an index texture with all the objects that should have hover effects:

Color	Depth	Tracking Areas

https://developer.apple.com/documentation/compositorservices/rendering_hover_effects_in_metal_immersive_apps

If the app (or Godot game) provides this data, visionOS does the hit-testing and renders the hover effects. When the user pinches, the app gets the ray as well as the uint64_t identifier of the object that was looked at:
https://developer.apple.com/documentation/swiftui/spatialeventcollection/event/trackingareaidentifier

I have a prototype of integrating this in Godot: it works, but the main downside is that I modified Godot's main rendering code to optionally write to this new index texture, in the fragment shader. I wonder if this could be done in a more modular way, similar to what the developer of CONNECTOME did in Unity:
https://www.uploadvr.com/connectome-a-game-of-points-the-challenges-of-building-for-apple-vision-pro/

Metal apps built in Unity do not provide eye-based hover effects, a foundational part of Apple Vision Pro's user experience. In layman's terms, this is what you see in an eye tracked headset when the cursor moves to whatever icon you are looking at. This is no fault of Apple, who introduced an API for Metal developers at its annual WWDC (Worldwide Developers Conference) in 2025. Instead, the gap here lies with Unity, who has not implemented support for it. For its part, Unity has acknowledged this and confirmed it does not plan to add integration.

Faced with this new obstacle, Hinkson built a custom bridge from scratch, one that drilled down to the native level of Unity's compositor, registered the necessary eye tracking information in each frame, and wrote the texture for visionOS to render the expected hover effect. Unfortunately, this workaround would be wiped out every time a new build of the app was generated as Unity regenerates its visionOS on each build. So a post build patcher had to be built that runs after the Unity export completes.

[BastiaanOlij]

@oPinon hmm, yeah writing to the index buffer I think is going to be an entirely different challenge, especially on a generic engine like Godot that tries to hide the render implementation details away from the user. So there is writing to this buffer, and somehow linking up the values to the actual nodes related to the geometry that the user will understand.

I have some ideas about enhancing shaders so they support outputting to additional buffers in a more generic way but there are a lot of challenges in joining all the dots.

[oPinon]

The transform of the spatial events did not match the existing controllers or the OpenXR convention. I fixed it by applying a corrective transform to each hand.

Before this fix (you can see that -Z points upwards):

After this fix (note that +Y points upwards and that -Z points forward):

Godot spatial events after fix	OpenXR convention

And here is a video showing the Spatial Events and the PSVR2 controllers side by side: you can see that the orientation now matches (aside from a small angular offset):

Spatial.Events.orientation_hevc.mp4

[oPinon]

@BastiaanOlij I have applied all the changes suggested above.

The only part that I'm unsure of is that now, for a game that was made for VR controllers, you have "left_hand" and "right_hand" floating when no VR controller is connected (because it maps to the Spatial Events):

Spatial.Events.Controllers_hevc.mp4

Maybe I can fix it by adding/removing the controllers from the XRServer every time you start/end a pinch?

[BastiaanOlij]

The only part that I'm unsure of is that now, for a game that was made for VR controllers, you have "left_hand" and "right_hand" floating when no VR controller is connected (because it maps to the Spatial Events):

When no pose data is available, you can invalidate the pose, there is a tickbox on the XRController3D node that will only make the node visible, if pose data is available..

That said, this is why I suggested also taking hand tracking data into account when mimicking controllers.

I really need to add some more functionality to the hand tracking demo we have, the whole idea here is to have seamless switching between controller tracking and hand tracking. As an application developer, I shouldn't need to worry about where my data is coming from, as long as I can be consistent in making my application work.

[BastiaanOlij]

Feels like we're very close to where we want to be.

It does feel like switching between controller tracking and spatial event tracking is a little undefined but it's probably workable. I need to properly test this.

Would be interesting to see how reliable this all works with our hand tracking demo.. I'll spend some more time on this later in the week.

[BastiaanOlij]

This should also set the aim pose, and we should either invalidate the grip and palm poses, or obtain those from hand tracking data.

[oPinon]

I changed it to:

		// Updating the hand pose.
		Transform3D pose = p_event.hand_pose * hand->transform_correction;
		for (const String name : { "default", "aim" }) {
			hand->controller->tracker->set_pose(name, pose, Vector3(), Vector3());
		}
		for (const String name : { "grip", "palm" }) {
			hand->controller->tracker->invalidate_pose(name);
		}

[BastiaanOlij]

If this is not active, should we still be setting poses? Or should they be invalidated? Not sure..

[oPinon]

I updated the code to invalidate the poses when no pinch is active.

In visionos_xr_interface.mm‎:

	// Invalidating poses on controllers when not tracked.
	for (const VisionOSSharedController &hand : { left_hand, right_hand }) {
		if (hand.source == VisionOSSharedController::Source::None) {
			for (const String name : { "default", "aim", "grip", "palm" }) {
				hand.tracker->invalidate_pose(name);
			}
		}
	}

[Clancey]

I have used this in a game I shipped to the app store, it made my Vision pickup way better!

[oPinon]

I have just updated the PR to properly invalidate the trackers when neither pinches nor controllers are active.

For example, here is a red sphere on the "right_hand" XRController3D, with "Show when tracked enabled":

The sphere only appears while I pinch:

No.controller.tracking_hevc.mp4

And if the Godot game asks for controller tracking, you can see that the sphere goes to the controller except while you pinch:

With.controller.tracking_hevc.mp4

I also verified that controller interactions still work fine:

Controller.tracking_hevc.mp4