kinect

kinect

Real-time Mixed Reality - Proof of concept: Kinect v2 + HTC Vive + Unity

3d ago
SOURCE  

Description

Prototype of real-time mixed reality previews of VR experiences using a depth camera. A live VR experience at exhibitions, large events, demos, etc is typically observed through first person camera or a third person camera on a TV screen. A more complex setup is sometimes used when the floor and walls feature projections of the VR world (similar principle to CAVE - http://goo.gl/i59uKF), or a green-screen setup with real-time compositing (along the lines of the process described meticulously by Kert Gartner - http://goo.gl/bWuHvu and Northways Games for streaming live - http://goo.gl/aJGxeA). The latter two methods require changes to the physical environment - adding projections or green-screen setup - often a costly or visually unattractive option. To provide a real-time mixed reality feed to an audience, retain the physical design of the space and avoid issues with foreground and background compositing, a RGB+Depth camera can be used to insert the user’s point cloud or mesh into the virtual space. Once the matrices of virtual and real cameras are matched, the virtual camera’s feed becomes a mixed reality feed. This technique is similar to Google’s approach for recording artist performances in Tilt Brush for Virtual Art Sessions (https://g.co/VirtualArtSessions), using two Kinects for depth data. We built a quick prototype using one of HTC Vive’s controllers for camera tracking, Microsoft Kinect v2, a Kinect v2 plugin (https://goo.gl/K5B7nQ) and Unity running on 2x machines. The server ran the actual VR scene and the client extracted data from the Kinect, placed the point cloud into the scene, resulting in a the mixed reality feed. The depth threshold was altered dynamically based on the position of the headset. In theory, any RGB+D camera set up to be tracked by Vive (through SteamVR Tracking - https://partner.steamgames.com/vrtracking/), or a RGB+D camera that can accurately ‘understand' the physical space (i.e. Hololens or Tango) can be used as a mixed reality camera. The downside of this technique is the quality level of the point cloud (notice rough edges of the point cloud in the prototype). Another downside is the requirement from the VR game developer to implement mechanism of connecting a depth camera into the game and render the point cloud into a separate scene from the one the user is experiencing, so they don’t see themselves in the scene. Music from Jukedeck - create your own at http://jukedeck.com (AI-driven music compositing) Alex Anpilogov, Timothy Burdairon & Dan Baczynski, FITCH, July 2016 http://alexeverything.net