RealSense

From TouchDesigner Documentation
Jump to: navigation, search

TouchDesigner acquires multiple data streams from Intel RealSense devices using the RealSense TOP and RealSense CHOP.

There are 2 RealSense SDKs supported in TouchDesigner.

  • librealsense SDK (newer) - Cross platform support for all camera models in the RealSense TOP only, the RealSense CHOP is not supported.
  • Windows RealSense SDK (older) - Windows only and supports older camera models only such as the F200, R200, SR300. Support for RealSense CHOP and RealSense TOP.

NOTE: D415, D435, D435i and T265 cameras are currently disabled on macOS due to bugs in the Intel librealsense API.

RealSense.1.jpg RealSense.2.jpg

RealSense TOP[edit]

The RealSense TOP node outputs color, depth and IR data in various forms. The TOP can use the newer librealsense SDK or the Windows-only Windows SDK (see below for details).

Various data streams can be captured by the RealSense TOP:

  1. Color, which is the video from the RealSense camera color sensor.
  2. Depth, a calculation of the depth of each pixel. 0 means the pixel is 0 meters from the camera and 1 means the pixel is the maximum distance or more from the camera.
  3. Raw Depth, values taken directly from the RealSense camera SDK. Once again 0 means 1 meter from the camera and 1 is the maximum range or more away from the camera.
  4. Visualized Depth, a gray scale image from the RealSense SDK which can help you visualize the depth but it cannot be used to actually determine a pixel’s exact distance from the camera.
  5. Depth To Color UV Map, the UV values from a 32 bit floating RG texture (note, no blue) that are needed to remap the depth image to line up with the color image. You can use the Remap TOP node to align the images to match.
  6. Color To Depth UV Map, the UV values from a 32 bit floating RG texture (note, no blue) that are needed to remap the color image to line up with the depth image. You can use the Remap TOP node to align the two.
  7. Infrared, the raw video from the infrared sensor of the RealSense camera.
  8. Point cloud, literally a cloud of points in 3d space (X, Y, Z coordinates) or data points created by the scanner of the RealSense Camera.
  9. Point Cloud Color UVs which can be used to get each point’s color from the Color image stream.
  10. Segmented Color (with Alpha) which outputs masked color image of the detected person in front of the camera.
  11. IMU/SLAM Tracking With the T265 model, the positional data of where the camera is in space can be retrieved using the Info CHOP.

See the point cloud sample at: http://www.derivative.ca/Forum/viewtopic.php?f=22&t=7977

Supported Devices[edit]

Currently the RealSense TOP supports the following RealSense devices:

  • F200
  • R200
  • ZR300
  • SR300
  • D415
  • D435
  • D435i
  • T265

RealSense CHOP[edit]

The RealSense CHOP connects to old RealSense devices (F200, R200, SR300) and outputs the following data. The CHOP requires the use of the Windows SDK (see below) and is available for Windows only.

  1. Marker Tracking, where you provide it a TOP image and it will do object recognition to find that image in the camera view and output its position and rotation relative to the camera.
  2. Finger Tracking, which detects 10 gesture types.
  3. Face Tracking, which outputs a mesh patching key points of your the face it detects.
  4. Person Tracking for R200 camera. Outputs tracked points on the persons skeleton.

Installing Required Software[edit]

Using the RealSense Cross Platform SDK[edit]

When using the default RealSense Cross Platform SDK, which is the only SDK that supports newer model devices, no drivers need to be installed.

Camera Firmware[edit]

The RealSense Cross Platform SDK does require that the correct firmware is installed on the device. You can do this by installing the RealSense SDK from here: https://github.com/IntelRealSense/librealsense/releases Then use the Camera Viewer tool to update the firmware to the recommended version.

Installing RealSense Software for Windows SDK[edit]

Note: the Windows SDK has been deprecated by Intel and will not work for new devices such as D-series camera and the T265.

The RealSense CHOP connects to RealSense devices and outputs data from Intel's RealSense camera.

  1. Marker Tracking, where you provide it a TOP image and it will do object recognition to find that image in the camera view and output its position and rotation relative to the camera.
  2. Finger Tracking, which detects 10 gesture types.
  3. Face Tracking, which outputs a mesh patching key points of your the face it detects.
  4. Person Tracking for R200 camera. Outputs tracked points on the persons skeleton.

If you are using the Windows SDK API in the RealSense TOP, or using the RealSense CHOP, you need to use the older RealSense SDK and drivers. You will need to install two pieces of software from Intel, The first is the camera driver software (DCM) for the particule model of RealSense camera you have (F200, R200, SR300). The second is the the SDK Runtime Distributable. If you already have the full RealSense SDK installed you don't need to install the Runtime separately.

These downloads are available here.

The current Windows RealSense API support in TouchDesigner is built against the 2016 R3 (11.0.27.1384) version of the SDK. For builds 2019.10000 or later, you should install the 2016 R3 SDK/Runtime (version 11). For builds 2018.20000 and earlier, you should install the 2016 R2 SDK/Runtime (version 10).

The SDK is split into multiple module downloads that need to be installed to enable different features such as hand tracking or face tracking.
Direct links:
Essentials
Hand Cursor
Face Tracking
Hand Tracking
F200 Camera Driver
SR300 Camera Driver

See RealSense CHOP, RealSense TOP

An Operator Family that creates, composites and modifies images, and reads/writes images and movies to/from files and the network. TOPs run on the graphics card's GPU.

An Operator Family which operate on Channels (a series of numbers) which are used for animation, audio, mathematics, simulation, logic, UI construction, and many other applications.

Each SOP has a list of Points. Each point has an XYZ 3D position value plus other optional attributes. Each polygon Primitive is defined by a vertex list, which is list of point numbers.