Question for my Research Project – Flight Simulator 2024 with Apple Vision Pro

[it-ibo]

Hello,

I'm currently working on a research project involving Flight Simulator 2024 with the Apple Vision Pro, and I'm trying to implement eye-tracking and hand-tracking.

For hand-tracking, I can see my hands, but I can't seem to interact with or press any buttons or controls in the cockpit. Eye-tracking also doesn't seem to work. I've read a few reports about this issue, but if these features aren't supported, I would like to cite this page as a source.

Your input would be greatly appreciated! I will share the final paper once it's done – it’s a really exciting project! 😊

Thank you <3

[shinyquagsire23]

Eye tracking is an Apple limitation, technically there's extremely limited support but it's a side-channel exploit based on screen recording, requires the RealityKit (40PPD) renderer, and is basically only good for social VR. SteamVR has yet to expose foveated rendering as well unfortunately.

Hand tracking with SteamVR input 2.0 works by having two separate hand tracking controllers (these will have appropriate icons in the next streamer version) which activate based on a hysteresis--if controller inputs are used, the hand tracking controllers are disabled for 5 seconds or so. However, if the game supports SteamVR skeletal input or XR_EXT_hand_tracking then it will work. Maybe also try disabling SteamVR input 2.0 so that skeletons are sent directly to the main controllers. But ultimately it's up to the game to support it.

If it works with other headsets, it could be a bug with the hand tracking skeleton, I already found a few bugs with the basis vector directions so it wouldn't be the first time, and the XR_EXT_hand_tracking basis vectors are specifically designed for interactions, whereas SteamVR's skeletons are designed for animation. So any errors on our part would possibly result in bugs.