Integrating Web-Based 3D Modeling with Leap Motion Tracking and Oculus Rift VR

Kyle Davis - SURF 2015

The Objective

Create web-based applications to demonstrate how to integrate 3D and Virtual Reality with the web

Develop tools to enable the use of 3D graphics with the Oculus and Leap Motion

Overview

  • Introduction to X3DOM, why it should be used and what it's good for
  • Introduction to the Leap Motion tracker and how it ties in with web-based 3D graphics
  • Introduction to the Oculus Rift and what we can do with it
  • What I created
  • Conclusion

The Purpose

  • Integrating 3D with the web - takes advantage of web development tools to create web graphics
  • Allows tools designed for web applications to work on graphics
  • Does not require any additional plug-ins or software
    • All you need is a link to the page and it works
    • Manipulates 3D objects within the HTML DOM.

What is the DOM?

  • Short for "Document Object Model"
  • The data structure that a web browser uses to create the web page 
    • Paragraphs, fonts, styles, etc. are all information in the DOM 
  • The technology we use puts graphical objects into the DOM.

Tools Used:

  • X3DOM
  • HTML 5
  • Javascript
  • JQuery/Ajax
  • Python
  • Leap Motion
  • Oculus Rift

Part I

Manipulating and mapping videos in X3DOM for VR

What is X3DOM?

  • An open-source framework and runtime environment for 3D graphics
    • Allows the integration of graphical (X3D) elements into HTML5 DOM tree
  • Can manipulate the 3D content by adding, removing, or changing DOM elements.

Learning Web-Based 3D

Manipulating Models

Mapping Videos onto 3D Objects

Part II - Leap Motion and Virtual Reality

What is Leap  Motion?

  • Infrared camera sits in front of keyboard
  • Provides computer interaction without keyboards or mice - only requires use of your hands' motions
  • Can recognize a series of gestures that can be made with hands/pointers

 Using Leap Motion

Why Use Leap Motion?

  • When Oculus is being used, you can't see your environment (no keyboard/mouse)
    • Needed to interact in a virtual world
  • Interactivity
  • Simplicity
  • Inexpensive

Using Leap Gestures

Difficulties with Leap Motion

  • Less precision
  • Harder to control
  • False positives (tracker will mistake one gesture for another)

Getting Leap Motion to Interact with X3DOM

  • Translating Leap Motion data into something that the
    X3D objects can understand
  • Determining how the two can be used together realistically (no wonky/confusing controls)
  • Switching between normal and Leap Motion controls smoothly

Part III

Oculus Rift

What is the Oculus Rift?

  • Virtual Reality headset with a separate display for each eye
    • With proper graphics, user can see 360 stereo 
  • Reacts to the user's head position & orientation

Oculus and X3DOM In Action

X3DOM and Virtual Reality Tools

X3D/Oculus/Leap Motion Interaction

Tool (I created) to bring in any .x3d file and have it display in stereo with the Oculus rift - objects can be manipulated by sliders & Leap Motion

.X3D Files to Oculus

  • Python script - takes any file with an X3DOM scene and creates a version with user-indicated attributes and Oculus Rift capabilities

Adding Oculus to X3D

Conclusion

  • What did I learn and accomplish?
    • How to use the web browser to manipulate 3D graphics
    • Designed Leap gesture-controlled X3D objects
    • Integrated Oculus Rift with X3D objects
    • JavaScript (Ajax and jQuery)/HTML/Python
    • Created several tools that will be useful for X3DOM developers

Acknowledgements

Thanks to:

  • My mentor, Sandy Ressler
  • My academic advisor, Deborah Knox
  • The NIST SURF Directors and SURF Program Coordinator, Dr. Brandi Toliver
  • NIST, for letting me come and and have a great research experience this summer 

Questions!