Realsense Record and Playback

Overview

In addition to streaming video and other data from devices and sensors, the realsense SDK provides the ability to record a live session of streaming to a file. The recorded files can later be loaded by the SDK and to create a device with “read-only” abilities of the recorded device ( we will explain what “read-only” abilities mean later on). The SDK is recording a single device to a single rosbag file, using mostly standard ROS messages. This allows files recorded by the SDK to be replayed using any ROS tools \ application.

For example recording files, please see Sample Data

Quick Start

`rs2::recorder`

:exclamation: If you are not familiar with the basic streaming examples, please follow them before moving on

To enable recording of any device, simply create a rs2::recorder from it and provide a path to the desired output file:

//Create a context and get the first device
rs2::context ctx;
auto devices = ctx.query_devices();
if (devices.size() > 0)
{
    //Create a rs2::recorder from the first device, and desired file name
    //'.bag' is the common extension for rosbag files
    rs2::recorder device("my_file_name.bag", devices[0]); 	
    //recorder "is a" device, so just use it like any other device now
}

A recorder has the same functionality as a “real” device, with additional control for recording, such as pausing and resuming record.

`rs2::playback`

:exclamation: If you are not familiar with the basic streaming examples, please follow them before moving on

Recorded files can be loaded and used to create a playback device by simply loading a file to the context:

//Create a context
rs2::context ctx;
//Load the recorded file to the context
rs2::playback device = ctx.load_device("my_file_name.bag");
//playback "is a" device, so just use it like any other device now

The above code creates a playback device, which can be used as any device, but has the obvious limitation of only playing the recorded streams. Playback devices can be used to query information on the device and it sensors, and can be extended to which ever extension the “real” device could. A playback provides additional functionalities such as seek, pause, resume and playback speed.

Using `rs2::config` with `rs2::pipeline`

The rs2::pipeline can be configured to record or play a streaming session by providing it with a rs2::config of your choice:

Recording to file:

rs2::config cfg;
cfg.enable_record_to_file("path_to_output_file.bag");
rs2::pipeline pipe;
pipe.start(cfg); //File will be opened in write mode at this point
for (int i = 0; i < 30; i++)
{
    auto frames = pipe.wait_for_frames();
    //use frames here
}
pipe.stop(); //File will be closed at this point

Playing from file:

rs2::config cfg;
cfg.enable_device_from_file("path_to_input_file.bag");
rs2::pipeline pipe;
pipe.start(cfg); //File will be opened in read mode at this point
for (int i = 0; i < 30; i++)
{
    rs2::frameset frames;
    if (pipe.poll_for_frames(&frames))
    {
        //use frames here
    }
    //Note that we use poll_for_frames instead of wait_for_frames
    //This is because the file only contains a finite amount of frames
    // and if we use wait_for_frames then after reading the entire file
    // we will fail to wait for the next frame (and get an exception)
}
pipe.stop(); //File will be closed at this point

Playback and Record in RealSense Viewer

Among its many features, the RealSense Viewer allows recording a device, and loading a file to playback.

To record a streaming session, simply click the “bars” icon next to the device name, choose “Record to File…”, and select the destination for the file.

Recording a device

After choosing a file destination, a red dot will appear next to the device’s name, indicating that it is recording. Starting a stream will save its frames to the file, and once all streams are stopped, recording will complete automatically.

To replay a file, click “Add Source”, choose “Load Recorded Sequence” and select the file you want to play. Once you select the file, the Viewer will automatically add it the the list of source, and a popup should appear stating that the file was loaded:

Loading Recorded Sequence

Notice that devices that were loaded from file have a “movie” icon next to their name.

After loading the file, you can start streaming its streams, view its controls (with the values at time of record), pause the playback, choose speed, and use the seek bar to navigate trough frames.

Under the Hood

Basics Terminology

A Device is a container of Sensors with some correlation between them (e.g - all sensors are on a single board, sensors are mounted on a robot and share calibration information, etc.). A Sensor is a data streaming object, that provides one or more Streams. Stream is a sequence of data items of a single data type, which are ordered according to their time of creation or arrival. The Sensor provides the Streams frames to the user.

We call the device’s sensors and stream, the topology of the device.

Devices and Sensors can have Extensions that provide additional functionalities. A Snapshot of an Extension is a snapshot of the data that is available by the extension at some point of time, it is a sort of “read-only” version of the extension. For example, say we have a DigitalClockExtension, that can set and show the time. If we take a snapshot of that extension at noon, then whenever we ask the snapshot to show the time it will show “12:00”, and trying to set its time will fail.

Finally, we will refer to a an actual implementation of devices and sensors as “live” or “real” devices and sensors.

Rosbag

Overview on ROSBAG file structure

The Rosbag file consists of data records, each data that is written to the file is represented by a topic that represents the data content, message that is the data definition, and capture time. Any topic can be coupled with any message, in our file each topic will be represented by a single message. However, we will have different topics with the same message such as property.
A bag file consists of a list of records, each record can be one of the following types:

Bag header. Stores information about the entire bag, such as the offset to the first index data record, and the number of chunks and connections.
Chunk. Stores (possibly compressed) connection and message records.
Connection. Stores the header of a ROS connection, including topic name and full text of the message definition (message structure).
Message data. Stores the serialized message data (which can be zero-length) with the ID of the connection.
Index data. Stores an index of messages in a single connection of the preceding chunk.
Chunk info. Stores information about messages in a chunk.

Each message that we write is stored in a chunk, each chunk consists of a number of messages based on the chunk size. In addition to the message data, the chunk stores also the connections of the messages, the connection describes the topic name and message definition to be able to serialize the message data. The flow to find a message on topic “A”:

Check in all chunk info’s what kind of connections can be found in the chunk.
Access the chunk with the wanted connection, chunk info contains an offset to the paired chunk.
Access the message from the relevant index data.

Dependencies

The SDK depends on ROS packages that are used for recording and playing the bag file. The dependencies are:

rosbag_storage
roscpp_serialization
cpp_common
rostime
boost_system
lz4

Note: The above dependencies are embedded into the source files, there is no need for additional installations.

Topics

The rosbag file requires messages to be associates with Topics

Topics are named buses over which nodes exchange messages. [ROS Wiki\Topics]

The following table depicts the Topics that are supported by RealSense file format:

Topic

Published Name

Message Type

Description

File Version

/file_version

std_msgs/UInt32

Version of file format.
A single message for entire file

Device Information

/device_<device_id>/info

diagnostic_msgs/KeyValue

Device information.
Many messages to a single topic

Sensor Information

/source_<source_id>/info

diagnostic_msgs/KeyValue

Sensor information.
Many messages to a single topic

Stream Information

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/info

realsense_msg/StreamInfo

generic-stream information
A single messages to a single topic

Video Stream Info

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/info/camera_info

sensor_msgs/camera_info

Image information.
A single messages to a single topic

IMU Intrinsics

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/info/imu_intrinsic

realsense_msgs/ImuIntrinsic

Intrinsic of a motion stream.
A single messages to a single topic

Options

/device_<device_id>/sensor_<sensor_id>/option/<option name>/value
/device_<device_id>/sensor_<sensor_id>/option/<option name>/description

std_msgs/Float32
std_msgs/String

Options of a sensor.
values represent the value of the option and has many messages to a single topic
description is a human readable description of the option, with a single messages to a single topic

Image Data

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/image/data

sensor_msgs/Image

The data of a single image. A single messages to a single topic

Image Information

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/image/metadata

diagnostic_msgs/KeyValue

Additional information of a single image. Many message to a single topic

IMU Data

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/imu/data

sensor_msgs/Imu

The data of a single imu frame.
A single messages to a single topic

IMU Information

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/imu/metadata

diagnostic_msgs/KeyValue

Additional information of a single imu frame. Many message to a single topic

Pose Data

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/pose/{transform, accel, twist}/data

pose/transform	geometry_msgs/Transform Message
pose/accel	geometry_msgs/Accel Message
pose/twist	geometry_msgs/Twist Message

The Pose data is split into 3 messages (each under different topic ^[1]):

pose/transform	Translation and rotation of the object
pose/accel	Angular and linear acceleration of the object
pose/twist	Angular and linear velocity of the object

Pose Information

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/pose/metadata

diagnostic_msgs/KeyValue

Additional information of a single pose frame. Many message to a single topic

Occupancy Map Data

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/occupancy_map/data

nav_msgs/OccupancyGrid

Occupancy Map Information

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/occupancy_map/metadata

diagnostic_msgs/KeyValue

Additional

Time Stream

/device_<device_id>/sensor_<sensor_id>/time

sensor_msgs/TimeReference

Log

/log

rosgraph_msgs/Log

Log messages

Stream Exntrinsics

/device_<device_id>/sensor_<sensor_id>/<stream_type>_<stream_id>/tf/<group_index>

geometry_msgs/Transform Message

Extrinsic transformation between some point of reference (indexed by group_index) to the stream in the topic

Additional Info

/additional_info

diagnostic_msgs/KeyValue

Additinal information of any kind. Can be useful for application that require additional metadata on the recorded file (such as program name, version etc...)

Sensor Notification

/device_<device_id>/sensor_<sensor_id>/notification/<rs2_notification_category>

realsense_msgs/Notification

Additinal information of any kind. Can be useful for application that require additional metadata on the recorded file (such as program name, version etc...)

[1] Pose frames

As you might have noticed, pose data is split into three messages, each under a different topic. The reasons for this choice is so that we expose only standard ros messages instead of creating a proprietary message. This choice allows any common tool that uses rosbag to read the content of a pose frame (and most of the rest of the messages we save).

When reading a file using the SDK’s playback, the reader will only iterate over pose/transform messages and for each of those messages, will match the corresponding pose/accel and pose/twist message to form a single pose frame. This matching of messages adds an overhead of searching the file for a single message, each frame. But, this overhead should have minor performance impact even for huge files since it is merely a tree search (See ros/ros_comm: PR #1223 - Performance improvement for lower/upper bound)

Messages

ROS uses a simplified messages description language for describing the data values (aka messages) that ROS nodes publish. This description makes it easy for ROS tools to automatically generate source code for the message type in several target languages. Message descriptions are stored in .msg files. There are two parts to a .msg file: fields and constants. Fields are the data that is sent inside of the message. Constants define useful values that can be used to interpret those fields (e.g. enum-like constants for an integer value).

RealSense Proprietary Messages

In addition to the standard ROS messsages, the SDK writes additional proprietary messages (available in the 3rd party folder) for new data types that are recorded. The following are the new messages created by the SDK:

realsense_msgs::StreamInfo
realsense_msgs::ImuIntrinsic

Stream Info

This message defines meta information for a stream The stream type is expressed in the topic name

Name	Description	Format
is_recommended	Indicates if this stream is recommended by RealSense SDK	bool
fps	Frame per second value	uint32
encoding	Stream's data format. Supported encodings are listed below	string

Supported Encoding

For video streams, the supported encoding types can be found at ros documentation. Additional supported encodings are listed under rs_sensor.h as the rs2_format enumeration. Note that some of the encodings appear in both locations.

Motion Intrinsic

This message defines a Motion device’s intrinsic: scale, bias, and variances

Name

Description

Format

data

Interpret data array values.
3x4 Row-major matrix:

Scale X	Cross Axis	Cross Axis	Bias X
Cross Axis	Scale Y	Cross Axis	Bias Y
Cross Axis	Cross Axis	Scale Z	Cross Axis

float32[12]

noise_variances

Variance of noise for X, Y, and Z axis

float32[3]

bias_variances

Variance of bias for X, Y, and Z axis

float32[3]

----------

Notification

This message defines a notification

Name	Description	Format
timestamp	The time of occurrence	std_msgs/Time
category	Category of the notification, matching an rs2_notification_category	std_msgs/String
severity	Severity of the notification, matching an rs2_log_severity	std_msgs/String
description	Human readable description of the notification	std_msgs/String
serialized_data	JSON string with additional data	std_msgs/String

Versioning

Each bag file recorded using this SDK should contains a version message. Version number is a single integer which indicates the version of topics and messages that were used to record the file. The above messages and topics reflect the current version. Changes from previous versions will appear at the end of this section.

Current file version: 3

Changes from previous version:

Removed:
- Property topic
Added:
- Options topic

Recording

Recording is performed at the Device level, meaning that the device, its sensors, their streams’ data (frames) and all extensions are saved to file. To allow for a seamless experience when switching between a live device and a record or playback device we save the device’s topology and all of the extensions’ snapshots to the file, in addition to the streaming frames.

A record device is like a wrapper around a real device, that delegates actions to the device and sensors while recording new data in between actions. When a record device is created, a record sensor is created per real sensor of the real device. A record sensor will record newly arriving frames for each of its streams, and changes to extensions’ data (snapshots).

Recording related files are:

record/record_device.cpp
record/record_device.h
record/record_sensor.cpp
record/record_sensor.h
ros/ros_writer.h

A librealsense::record_device is constructed with a “live” device and a device_serializer::writer. At the moment the only device_serializer::writer we use is a ros_writer which writes device information to a rosbag file.

When constructing a ros_writer the requested file is created if it does not exist, and then opened for writing. In addition, a single message containing the realsense file format version is written to the file. The ros_writer implements the device_serializer::writer interface which has only 4 functions:

write_device_description
- Used to record the initial state of the device. This includes writing all of the device’s and sensor’s extensions.
write_frame
- Used to record a single frame to file
write_snapshot (2 overloads)
- Used to record a snapshot of an extension to file.

Recording device description

A device description (or device snapshot) is the set of data that is available for any librealsense::device. It contains all of the device’s extensions snapshots, and a collection of each of the device’s sensors’ extensions snapshots. In addition it holds a mapping of extrinsic information for all streams of the device.

A single librealsense::device_snapshot instance is enough to understand the a device’s topology: Which extensions it supports, how many sensors it has, which extensions each sensor supports, which streams each sensor supports and which streams share calibration information (extrinsic).

Recording Frames

Each frame in the SDK implements the librealsense::frame_interface interface. This means that frames are polymorphic and represent all types of data that streams provide. Frames are recorded to file with all of their additional information (such as metadata, timestamp, etc…), and the time that they arrived from the backend to the sensor.

Recording Snapshots

Upon creation, the record device goes over all of the extensions of the real device and its sensors, and saves snapshots of those extensions to the file (This is the data that is passed to write_device_description(..)) . These snapshots will allow the playback device to recreate the topology of the recorded device, and will serve as the initial data of their extensions. To allow record sensors to save changes to extensions’ data over the life time of the program, when a user asks a record sensor for an extension, a record-able extension is provided. A record-able version of an extension holds an action to perform whenever the extension’s data changes. This action is provided by the record device (or sensor), and requires extensions to pass a reference of themselves to the device, which will usually create a snapshot from them and record them to file with the time at which they occurred.

Playback

Playback device is an implementation of device interface which reads from a file to simulate a real device. Playback device holds playback sensors which simulate real sensors.
When creating the playback device, it will read the initial device snapshot from the file in order to map itself and its sensors in matters of functionality and data provided.
When creating each sensor, the device will create a sensor from the sensor’s initial snapshot. Each sensor will hold a single thread for each of the sensor’s streams which is used to raise frames to the user. The playback device holds a single reading thread that reads the next frame in a loop and dispatches the frame to the relevant sensor. The reading of the file, as well as each sensor’s handling of frames, are done in separate threads. All this is managed via a common dispatcher concurrency mechanism: an invoke() call enqueues an action and is dequeued and run from a worker thread.

Sequence Diagram

playback

Note: this flow uses the sensor API; librealsense2 supports playback with the pipeline API as well, which looks similar inside the playback device. *Created using DrawIO

Realsense Record and Playback

Overview

Quick Start

rs2::recorder

rs2::playback

Using rs2::config with rs2::pipeline

Playback and Record in RealSense Viewer

Under the Hood

Basics Terminology

Rosbag

Overview on ROSBAG file structure

Dependencies

Topics

[1] Pose frames

Messages

RealSense Proprietary Messages

Stream Info

Supported Encoding

Motion Intrinsic

Notification

Versioning

Recording

Recording device description

Recording Frames

Recording Snapshots

Playback

Sequence Diagram

`rs2::recorder`

`rs2::playback`

Using `rs2::config` with `rs2::pipeline`