reductstore_agent

ROS2 recorder node that logs selected topics to ReductStore, with YAML-driven pipelines (mcap/raw, compression, splitting, down-sampling, labelling).

README

reductstore_agent

ROS 2 Jazzy & Rolling

reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.

This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.

System Requirements

To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide

This agent is tested with:

  • ROS 2: Jazzy and Rolling

  • OS: Ubuntu 24.04 (Noble)

  • Python: 3.12

Motivation

  • Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.

  • Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.

  • Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.

  • Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.

  • Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.

Structure

The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).

/**/*:
  ros__parameters:
    storage: # local ReductStore instance
      url: "http://localhost:8383"
      api_token: "access_token"
      bucket: "ros_data"
      quota_type: "FIFO"
      quota_size: "200GB"
    pipelines:
      telemetry:
        filename_mode: "timestamp"
        include_topics:
          - "/camera/.*"
        exclude_topics:
          - "/camera/ignore"
        static_labels:
          source: telemetry
          robot: alpha
        split:
          max_duration_s: 3600
          max_size_bytes: 10000

See the Configuration section for details on available parameters.

Installing

This package requires Python dependencies (reduct-py, mcap, mcap-ros2-support) that are not provided by ROS 2. To keep them isolated while still allowing colcon and ros2 (installed system-wide) to work, we create a virtual environment with:

python3 -m venv .venv --system-site-packages

The --system-site-packages flag is important because it lets the venv access ROS’s system-installed Python packages (rclpy, rosbag2_py, etc.). Without it, colcon and ROS Python APIs won’t be visible inside the venv, and ros2 run would fail to find your node.

Build and Run

# 1. Create a workspace and clone the repository
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..

# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y

# 3. Create and activate the virtual environment
python3 -m venv .venv --system-site-packages
source .venv/bin/activate

# 4. Install Python dependencies
pip install -U reduct-py mcap mcap-ros2-support

# 5. Build the package with colcon using the venv Python
python -m colcon build --symlink-install

# 6. Source the workspace and run the node
source install/setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config/params.yml

There is also a Snap package available for easier installation and updates.

Configuration

The configuration file is a YAML file that defines the storage settings and pipelines. The storage section contains ReductStore connection details, and the pipelines section defines the individual pipelines for recording data.

Storage Configuration

The storage section specifies the ReductStore instance to connect to:

  • url: The URL of the ReductStore instance (e.g., http://localhost:8383).

  • api_token: The API token for authentication. This is required to access the ReductStore instance.

  • bucket: The bucket name where the data will be stored.

  • quota_type: The type of quota to apply. Options are:

    • "FIFO": First In, First Out (oldest data is removed first).

    • "HARD": Hard limit (data is not accepted when the quota is reached).

    • "NONE": No quota applied.

  • quota_size: The size of the quota in bytes. This is required if quota_type is set to "FIFO" or "HARD".

  • max_block_size: Maximum size of each block in bytes.

  • max_block_records: Maximum number of records per block.

More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.

Pipeline Parameters

Each pipeline supports the following parameters:

  • split:

    • max_duration_s: Maximum duration (in seconds) for each data segment. Must be between 1 and 3600.

    • max_size_bytes (optional): Maximum size (in bytes) for each segment. Must be between 1KB and 1GB.

  • chunk_size_bytes: Size of each MCAP chunk in bytes. Defaults to 1MB. Must be between 1KB and 10MB.

  • compression: Compression algorithm to use. One of:

    • "none"

    • "lz4"

    • "zstd" (default)

  • enable_crcs: Whether to enable CRC checks. Defaults to true.

  • spool_max_size_bytes: Maximum in-memory spool size before flushing. Defaults to 10MB. Must be between 1KB and 1GB.

  • include_topics: List of topics to include for recording. Supports regular expressions.

  • exclude_topics (optional): List of topics to exclude from recording. Supports regular expressions.

  • static_labels (optional): Fixed key-value labels to attach to each record.

  • filename_mode: Determines how filenames are generated. One of:

    • "timestamp" (default) — Use first topic timestamp for filenames.

    • "incremental" — Use incrementing numbers (0, 1, 2, …) for filenames.

Snap Package

This snap provides the ReductStore Agent ROS 2 node, which records ROS 2 topics and streams them into ReductStore.

Build and Install

# Install snapcraft if not already installed
sudo snap install snapcraft --classic

# Build the snap
snapcraft pack

# Install locally (for testing)
sudo snap install --dangerous reductstore-agent_*.snap

Or install directly from the store (edge channel for now):

sudo snap install reductstore-agent --edge