README
proto2ros
When exposing Protobuf interfaces to ROS 2, proto2ros streamlines message translation and generation, so as to trivially augment
rosidl pipeline
invocations to take Protobuf definitions and output ROS 2 message definitions. Additionally, message conversion APIs are generated to
simplify bridging Protobuf and ROS 2 message code.
Table of contents
Features
Message generation
Protobuf enumeration and message definitions are translated to equivalent ROS 2 message definitions.
Protobuf definitions, including comments, are extracted from Protobuf descriptor sets, as generated by
protoc. More than one ROS 2
message definition may be necessary to represent a given Protobuf definition, as explained in the
map types and one-of fields subsections.
Package mapping
All Protobuf packages to which processed definitions belong are implicitly mapped to the ROS 2 package
that will host their ROS 2 message equivalents. As for the rest, the user may specify a package mapping
in proto2ros configuration. Whether that is necessary or not depends on how message types
are mapped.
Type mapping
Scalar types
The mapping between Protobuf and ROS 2 scalar types is shown below.
Protobuf scalar type name |
ROS 2 scalar type name |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Note that, unlike all others, the Protobuf bytes scalar type maps to a sequence type in ROS 2.
This is convenient but forces special handling of repeated bytes fields.
Messages types
Every Protobuf message maps to a ROS 2 message. For statically typed message (as opposed to dynamically
typed google.protobuf.Any messages), this mapping is derived from the sequential application of the
following rules, on first match wins basis:
A user-defined message mapping in
proto2rosconfiguration matches. Fully qualified message names are used verbatim.A user-defined or implicit package mapping in
proto2rosconfiguration matches. If multiple matches are found, the longest match applies. Fully qualified message names are camel-cased.Passing unknown Protobuf messages is allowed by
proto2rosconfiguration. Thenproto2ros/AnyProtois used.
For google.protobuf.Any messages, the rules for any types apply.
It is through message mappings that ad-hoc equivalences can be established e.g. some Protobuf message
may be equivalent to some standard ROS 2 message (and this is already the case for several core Protobuf
messages, see default proto2ros configuration). It is through package mappings that equivalences
generated by proto2ros in full, for a given ROS 2 package or one of its dependencies, interact with
each other.
For example, given the following configuration overlay:
message_mapping:
third_party.data.Text: std_msgs/String
google.protobuf.Any: custom_msgs/Any
package_mapping:
third_party.data: data_msgs
third_party.data.legacy: data_legacy_msgs
google.protobuf.Any would map to proto2ros/Any, third_party.data.Text would map to std_msgs/String,
third_party.data.Blob would map to data_msgs/Blob, third_party.data.legacy.Image would map to
data_legacy_msgs/Image, and some_package.Data would map to proto2ros/AnyProto if passthrough_unknown
is enabled, raising an error otherwise.
Enum types
As ROS 2 messages lack the notion of enumerated types entirely, a ROS 2 message is generated for each
Protobuf enumeration. This ROS 2 message defines a homonymous integer constant for each enum value and
a single value integer field to bear it. A sample equivalence is shown below.
| Protobuf .proto definition | ROS 2 .msg definition |
// some.proto
enum Status {
STATUS_UNKNOWN = 0;
STATUS_OK = 1;
STATUS_FAILURE = 2;
}
|
# Status.msg
int32 STATUS_UNKNOWN=0
int32 STATUS_OK=1
int32 STATUS_FAILURE=2
int32 value
|
Map types
Over the wire, Protobuf map types are bound to be equivalent to a sequence of key-value pairs (or map entry messages). For ROS 2, the exact same convention is observed. A ROS 2 message is thus generated for each map entry message. A sample equivalence is shown below.
| Protobuf .proto definition | Protobuf equivalent syntax | ROS 2 .msg definition |
// some.proto
message Device {
map<string, string> attributes = 1;
}
|
// some.proto
message Device {
message AttributesEntry {
string key = 1;
string value = 2;
}
repeated AttributesEntry attributes = 1;
}
|
# DeviceAttributesEntry.msg
string key
string value
|
# Device.msg
<ros_package_name>/DeviceAttributesEntry[] attributes
|
Any types
Dynamically typed (i.e. google.protobuf.Any) Protobuf messages are mapped to ROS 2 messages as specified by any expansions.
An any expansion is a type set $T$ that fully characterizes what to expect of a google.protobuf.Any message field.
These are indexed after Protobuf message name and field name. Cardinality $|T|$ always satisfies $|T| > 0$.
Given an applicable any expansion is found:
if $|T| == 1$ and
allow_any_castsis enabled, the corresponding equivalent ROS 2 message type for that sole Protobuf message type, as dictated by message type mapping rules, will be used (as if that statically typed Protobuf message type had been found in place ofgoogle.protobuf.Any);otherwise, a dynamically typed (i.e.
proto2ros/Any) ROS 2 message is used as the equivalent ROS 2 message type, which will bear the equivalent ROS 2 type set (with $|T| > 0$) as dictated by message type mapping rules.
Else, a proto2ros/AnyProto ROS 2 message type is used as the equivalent ROS 2 message type, bearing the unmodified,
serialized Protobuf message.
For example, given the following configuration overlay:
any_expansions:
third_party.data.Storage.params: third_party.data.StorageParams
third_party.data.StorageParams.implementation_specific: [third_party.data.S3Params, third_party.data.PGParams]
allow_any_casts: true
google.protobuf.Any for the params field in the third_party.data.Storage Protobuf message would map to the ROS 2
equivalent, as per message type mapping rules, of the third_party.data.StorageParams Protobuf message, whereas
google.protobuf.Any for the implementation_specific field in the third_party.data.StorageParams Protobuf message
would map to proto2ros/Any. Conversion APIs, however, can use the information provided by these any expansions to
perform the necessary casting in runtime.
Recursive types
Protobuf supports recursive message definitions but ROS 2 does not. To workaround this limitation, a message dependency graph
reflecting the composition relationships between known messages is built and analyzed for cycles. Once a cycle has been identified,
it is broken by the weakest link (i.e. the minimal change set) using proto2ros/Any, functionally type erasing one or more fields.
Field mapping
Optional fields
Optional fields in Protobuf messages, and fields with explicit presence
tracking in general, are conventionally implemented using a bit mask field in ROS 2 messages. As ROS 2 messages lack the notion
of optional fields entirely, an unsigned integer has_field field explicitly conveys which message fields bear meaningful
information. For each optional field f, an unsigned integer constant F_FIELD_SET bit mask is defined. Bitwise binary
operations can then be used to explicitly indicate and check for field presence. A sample equivalence is shown below.
| Protobuf .proto definition | ROS 2 .msg definition |
// some.proto
message Option {
optional string value = 1;
}
|
// Option.msg
uint8 VALUE_FIELD_SET=1
string value
uint8 has_field 255
|
Note that, to match ROS 2 message semantics, the bit mask is fully set by default. That is, all fields are assumed to be present by default.
Implementation note: bit masks can be 8, 16, 32, or 64 bit long, depending on the number of optional fields. Protobuf messages with more than 64 optional fields are therefore not supported.
Repeated fields
Repeated fields in Protobuf messages are mapped to array fields in ROS 2 messages. This applies to all field types except to bytes
fields. This exception is necessary as scalar bytes fields are already mapped to array fields in ROS 2. In this case, scalar type
mapping rules are overridden and repeated bytes fields are mapped to array fields of proto2ros/Bytes ROS 2 message type. A sample
equivalence is shown below.
| Protobuf .proto definition | ROS 2 .msg definition |
// some.proto
message Payload {
repeated int32 keys = 1;
repeated bytes blobs = 2;
bytes checksum = 3;
}
|
# Payload.msg
int32[] keys
proto2ros/Bytes[] blobs
uint8[] checksum
|
One-of fields
As ROS 2 messages lack the notion of one-of fields entirely, a ROS 2 message is generated for each one-of construct in a Protobuf message,
bearing all one-of fields, as well as an integer which field. This ROS 2 message is functionally equivalent to a tagged union.
For each field f in the one-of construct o, an integer constant O_F_SET tag is defined. Assigning the which field to a given tag thus
conveys presence of the corresponding field. In place for each one-of construct, a message field of the corresponding type is defined. A sample
equivalence is shown below.
| Protobuf .proto definition | ROS 2 .msg definition |
// some.proto
message Timestamp {
oneof value {
uint64 seconds_since_epoch = 1;
string datestring = 2;
}
}
|
# Timestamp.msg
<ros_package_name>/TimestampOneOfValue value
|
# TimestampSecondsSinceEpoch.msg
uint64 seconds_since_epoch
|
|
# TimestampDatestring.msg
string datestring
|
|
# TimestampOneOfValue.msg
int8 VALUE_NOT_SET=0
int8 VALUE_SECOND_SINCE_EPOCH_SET=1
int8 VALUE_DATESTRING_SET=2
<ros_package_name>/TimestampSecondsSinceEpoch seconds_since_epoch
<ros_package_name>/TimestampDatestring datestring
int8 value_choice # deprecated
int8 which
|
Implementation note: 8 bit tags are used for one-of constructs. Protobuf messages with more than 256 one-of fields are therefore not supported.
Deprecated fields
Deprecated fields are kept, unless drop_deprecated is enabled. If kept, these fields are annotated with a comment
in the corresponding ROS 2 message definition. A sample equivalence is shown below.
| Protobuf .proto definition | ROS 2 .msg definition (drop_deprecated disabled) |
ROS 2 .msg definition (drop_deprecated enabled) |
// some.proto
message Duration {
int64 seconds = 1;
int64 nanosec = 2 [deprecated = true];
int64 nanoseconds = 3;
}
|
# Duration.msg
int64 seconds
int64 nanosec # deprecated
int64 nanoseconds
|
# Duration.msg
int64 seconds
int64 nanoseconds
|
Reserved fields
Reserved fields are ignored.
| Protobuf .proto definition | ROS 2 .msg definition |
// some.proto
message Goal {
string location = 1;
reserved "time_budget";
}
|
# Goal.msg
string location
|
Code generation
To simplify conversion from Protobuf messages to equivalent ROS 2 messages and back, proto2ros generates conversion code,
nicely wrapped around convert(from, to) function overloads (i.e. type dispatched). Note, however, that conversion code is
only generated for message equivalences that proto2ros itself generated in full. For ad-hoc equivalences, as specified using
message mappings, the user must implement the corresponding overloads. For auxiliary messages underpinning enums, map types,
one-of fields, and the like, no overloads are generated at all (as there is no Protobuf message to convert to/from).
Python APIs
Conversion APIs in Python are exposed on a package basis, as {ros_package_name}.conversions.convert overloads.
For each pair of equivalent ROSMessageT and ProtoMessageT types, proto2ros generates the following overloads:
convert(ros_msg: ROSMessageT, proto_msg: ProtoMessageT) -> Nonefor ROS 2 message to Protobuf message conversionconvert(proto_msg: ProtoMessageT, ros_msg: ROSMessageT) -> Nonefor Protobuf message to ROS 2 message conversion
While convenient, the mechanisms that enable these overloads do not play
along with static analyzers such as mypy. To workaround this limitation, each overload is also made available, fully
type annotated, under a unique name. This name is derived from argument type names as follows:
convert_{ros_package_name}_{ros_message_name}_message_to_{proto_package_name}_{proto_message_name}_protofor ROS 2 message to Protobuf message conversionconvert_{proto_package_name}_{proto_message_name}_proto_to_{ros_package_name}_{ros_message_name}_messagefor Protobuf message to ROS 2 message conversion
All message names above are snake-cased. Note that user-defined overloads for ad-hoc equivalences must follow the same pattern.
Implementation note: all explicit and implicit _pb2 (i.e. Protobuf) Python imports must be available at generation time. This
requirement allows proto2ros to cope with an omission in Protobuf descriptor sets: these do not specify the mapping between fully
qualified Protobuf message names and their Python counterparts. To workaround this limitation, known _pb2 modules are traversed
to reconstruct this mapping.
C++ APIs
Conversion APIs in C++ are exposed on a package basis as {ros_package_name}::conversions::convert overloads, available from
{ros_package_name}/conversions.hpp headers. For each pair of equivalent ROSMessageT and ProtoMessageT types, proto2ros
generates the following overloads:
void {ros_package_name}::conversions::Convert(const ROSMessageT& ros_msg, ProtoMessageT* proto_msg)for ROS 2 message to Protobuf message conversionvoid {ros_package_name}::conversions::Convert(const ProtoMessageT& proto_msg, ROSMessageT* ros_msg)for Protobuf message to ROS 2 message conversion
Note that user-defined overloads for ad-hoc equivalences must follow the same pattern.
Configuration
Both message and code generation are configured by a number of settings, listed below.
Name |
Description |
Default value |
|---|---|---|
|
Whether to drop deprecated fields on conversion or not. If not dropped, deprecated fields are annotated with a comment. |
|
|
Whether to forward Protobuf messages for which no equivalent ROS message is known as a serialized binary blob in a |
|
|
A mapping from fully qualified Protobuf message names to fully qualified ROS message names. This mapping comes first during composite type translation. |
|
|
A mapping from Protobuf package names to ROS package names, to tell where a ROS equivalent for a Protobuf construct will be found. Note that no checks for package existence are performed. This mapping comes second during composite type translation (i.e. when direct message mapping fails). |
|
|
A mapping from fully qualified Protobuf field names (i.e. a fully qualified Protobuf message name followed by a dot “.” followed by the field name) of |
|
|
When a single Protobuf message type is specified in an any expansion, allowing any casts means to allow using the equivalent ROS message type instead of a dynamically typed, |
|
|
A mapping from ROS message names to known message specifications. Necessary to cascade message generation for interdependent packages. |
|
|
Set of C++ headers to be included (as |
|
|
Set of C++ namespaces bearing conversion overloads, for which unqualified lookup ( |
|
|
Set of Python modules to be imported (as |
|
|
Set of Python modules to be imported into module scope (as |
|
|
Whether to skip importing Python modules for known Protobuf and ROS packages in generated conversion modules or not. These known modules are those derived from |
|
These defaults can be replaced entirely via configuration file or overridden one by one via configuration overlays. Configuration overlays are configuration files that update the baseline configuration, default or user-defined. Scalar values are replaced, lists are extended, dictionaries are updated (i.e. shallow merged).
Use cases
Dual Protobuf / ROS 2 package
A package may provide both Protobuf and ROS 2 messages, all generated from Protobuf definitions.
cmake_minimum_required(VERSION 3.12)
project(proto2ros_tests)
find_package(ament_cmake REQUIRED)
find_package(builtin_interfaces REQUIRED)
find_package(rosidl_default_generators REQUIRED)
find_package(proto2ros REQUIRED)
find_package(rclcpp REQUIRED)
find_package(Protobuf REQUIRED)
# Generate Python code for some.proto
protobuf_generate(
LANGUAGE python
OUT_VAR proto_py_sources
PROTOS some.proto
IMPORT_DIRS proto
)
# Generate C++ code for some.proto
protobuf_generate(
LANGUAGE cpp
OUT_VAR proto_cpp_sources
PROTOS some.proto
IMPORT_DIRS proto
)
# Build generated C++ code
add_library(${PROJECT_NAME}_proto SHARED ${proto_cpp_sources})
target_include_directories(${PROJECT_NAME}_proto PUBLIC
"$<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}>"
"$<INSTALL_INTERFACE:include/${PROJECT_NAME}>"
)
target_link_libraries(${PROJECT_NAME}_proto protobuf::libprotobuf)
# Add dependable target for generated Protobuf code
add_custom_target(
${PROJECT_NAME}_proto_gen ALL
DEPENDS ${proto_py_sources} ${proto_cpp_sources}
)
# Generate equivalent ROS 2 messages and conversion sources
proto2ros_generate(
${PROJECT_NAME}_messages_gen
PROTOS proto/some.proto
INTERFACES_OUT_VAR ros_messages
PYTHON_OUT_VAR ros_py_sources
CPP_OUT_VAR cpp_sources
INCLUDE_OUT_VAR cpp_include_dir
APPEND_PYTHONPATH "${CMAKE_CURRENT_BINARY_DIR}"
DEPENDS ${PROJECT_NAME}_proto_gen
)
# Generate ROS 2 message code.
rosidl_generate_interfaces(
${PROJECT_NAME} ${ros_messages}
DEPENDENCIES builtin_interfaces proto2ros
)
add_dependencies(${PROJECT_NAME} ${PROJECT_NAME}_messages_gen)
# Build C++ conversion library
add_library(${PROJECT_NAME}_conversions SHARED ${cpp_sources} src/manual_conversions.cpp)
target_include_directories(${PROJECT_NAME}_conversions PUBLIC
"$<BUILD_INTERFACE:${cpp_include_dir}>"
"$<INSTALL_INTERFACE:include/${PROJECT_NAME}>"
)
rosidl_get_typesupport_target(${PROJECT_NAME}_cpp_msgs ${PROJECT_NAME} "rosidl_typesupport_cpp")
target_link_libraries(${PROJECT_NAME}_conversions
${${PROJECT_NAME}_cpp_msgs} ${PROJECT_NAME}_proto protobuf::libprotobuf)
ament_target_dependencies(${PROJECT_NAME}_conversions builtin_interfaces proto2ros rclcpp)
# Install generated Python _pb2 and conversion code to the
# Python package that is implicitly defined and installed by the
# rosidl pipeline
rosidl_generated_python_package_add(
${PROJECT_NAME}_additional_modules
MODULES ${proto_py_sources} ${py_sources}
PACKAGES ${PROJECT_NAME}
DESTINATION ${PROJECT_NAME}
)
# Install generated C++ .pb.h and conversion headers
set(cpp_headers ${cpp_sources} ${proto_cpp_sources})
list(FILTER cpp_headers INCLUDE REGEX ".*\.hpp$")
install(
FILES ${cpp_headers}
DESTINATION include/${PROJECT_NAME}/${PROJECT_NAME}/
)
# Install C++ Protobuf messages and conversion libraries
install(
TARGETS
${PROJECT_NAME}_proto
${PROJECT_NAME}_conversions
EXPORT ${PROJECT_NAME}
ARCHIVE DESTINATION lib
LIBRARY DESTINATION lib
RUNTIME DESTINATION bin
)
ament_export_dependencies(builtin_interfaces proto2ros rclcpp)
ament_export_targets(${PROJECT_NAME})
ament_package()
proto2ros_tests is a good example of this.
ROS 2 vendored Protobuf messages
Protobuf messages may already be provided by some third-party package, in which case, it is only the equivalent ROS 2 messages that are relevant.
For a third-party package and .proto files that are hosted on public repositories, the FetchContent
module and the proto2ros_vendor_package CMake macro fully address this use case:
cmake_minimum_required(VERSION 3.8)
project(vendored_third_party)
find_package(ament_cmake REQUIRED)
find_package(proto2ros REQUIRED)
# Fetch third party package sources (incl. .proto files)
include(FetchContent)
FetchContent_Declare(
third_party
GIT_REPOSITORY ...
GIT_TAG ..._
)
FetchContent_Populate(third_party)
# Collect third party .proto files
set(${PROJECT_NAME}_PROTO_DIR "${third_party_SOURCE_DIR}/protos")
file(GLOB ${PROJECT_NAME}_PROTOS "${${PROJECT_NAME}_PROTO_DIR}/*.proto")
# Generate ROS 2 messages and code (wraps rosidl)
proto2ros_vendor_package(${PROJECT_NAME}
PROTOS ${${PROJECT_NAME}_PROTOS}
IMPORT_DIRS ${${PROJECT_NAME}_PROTO_DIR}
)
ament_package()
bosdyn_msgs is a good example of this.