Zero-Copy Intra-Process Communications with ROS 2
Preamble
Using ROS, or any middleware for that matter, has many advantages for developers: code reuse, introspection, a standard set of message definitions so unrelated packages actually interoperate, off-the-shelf tooling that works on any topic without you writing a line of it (ros2 topic echo, rosbag2, RViz, tf2), language interoperability between C++ and Python, and the loose coupling that lets you start, stop, replace or relocate a node without touching its neighbours.
Unfortunately this isn’t a free lunch; copying and serialisation of data across boundaries, communication latency, and dropped packets will be all too familiar for anyone who has worked in this space. Yes, you can harden your system against it as best you can, but another approach is to forgo many of the problems entirely by coupling software together into a single process at run-time so as to share memory, removing the overheads from copying and serialisation whilst increasing determinism and reliability.
In ROS speak this is intra-process communication, not to be confused with inter-process communication (post on that to follow…). When set up correctly it can enable zero-copy publish and subscribe between nodes. When set up incorrectly it can be more trouble than it is worth.
Background
The ROS 2 tutorial for zero-copy intra-process communication is a bit misleading…
“This shows that the address of the message being received is the same as the one that was published and that it is not a copy. This is because we’re publishing and subscribing with std::unique_ptrs which allow ownership of a message to be moved around the system safely. You can also publish and subscribe with const & and std::shared_ptr, but zero-copy will not occur in that case.”
The design article is better but I thought it would be useful to share my findings from a users perspective.
Coming from using intra-process communication with Nodelets and PluginLib in ROS 1, I found the enforcement of std::unique_ptr on both sides of the communication strange (I think it is a hangover from before Dashing), especially in fan-out, one-to-many style publication, so I decided to run some tests.
Testing
Methodology
Using rclcpp 32.0.0 (ROS 2 Lyrical) a PointCloud2’s payload lives in msg->data (a std::vector<uint8_t>).
If the subscriber’s msg->data.data() is the same address the publisher held before publishing, the exact same heap buffer was handed over; true zero-copy.
A different address means a deep-copy happened somewhere in the chain.
A global operator new override was also implemented to count bytes allocated during the publish and dispatch window as a cross-check.
- Payload: 1.75 MiB (115k points × 16 byte point_step)
- One process,
use_intra_process_comms(true),SingleThreadedExecutor, QoS depth 10 volatile. - RMW during the run was
rmw_cyclonedds_cpp, but the intra-process path is RMW-independent, so this does not affect any of the intra-process results; it only matters for cases [11]/[12] in the results.
Results
All message contents and results below are byte-identical across repeated runs.
| # | Publish | Subscribers | Outcome | Heap During Publish and Dispatch |
|---|---|---|---|---|
| 1 | std::move(unique) |
1× UniquePtr | same buffer | 0.00 MiB |
| 2 | std::move(unique) |
1× ConstSharedPtr | same buffer | 0.00 MiB |
| 3 | std::move(unique) |
1× mutable SharedPtr | same buffer | 0.00 MiB |
| 4 | by value | 1× UniquePtr | copy | 1.76 MiB |
| 5 | by value | 1× ConstSharedPtr | copy | 1.76 MiB |
| 6 | std::move(unique) |
2× UniquePtr | 1 copy + 1 same | 1.76 MiB |
| 7 | std::move(unique) |
3× ConstSharedPtr | same buffer ×3 | 0.00 MiB |
| 8 | std::move(unique) |
1× Unique + 1× ConstShared | 1 same + 1 copy | 1.76 MiB |
| 9 | std::move(unique) |
1× Unique + 2× ConstShared | 1 same + 2 copies | 1.76 MiB |
| 10 | std::move(unique) |
2× mutable SharedPtr | 1 copy + 1 same | 1.76 MiB |
| 11 | std::move(unique) |
1× UniquePtr + 1 out-of-process | same buffer | 7.02 MiB |
| 12 | std::move(unique) |
1× ConstSharedPtr + 1 out-of-process | same buffer | 5.27 MiB |
Why Though?
Publish side:
std::moveof aunique_ptris the only zero-copy publish.publish(const T&)deep-copies unconditionally when IPC is enabled (publisher.hpp:277-289,publisher.hpp:566-571). There is nopublish(shared_ptr)overload, sopublish(*shared_msg)also copies. Cases [4]/[5].Subscribe side:
UniquePtrandConstSharedPtrare both zero-copy for a single subscriber, but they are not interchangeable. The deciding factor is inany_subscription_callback.hpp:1021-1030: onlyshared_ptr<const T>,const shared_ptr<const T>&andconst T&callbacks are “take_shared”. Everything else, including a mutableshared_ptr<T>, is a take_ownership subscription. Routing is inintra_process_manager.hpp:245-279:
- all take_shared paths get one promotion, same buffer to all; zero copies for any N (case [7])
- any take_ownership paths:
add_owned_msg_to_buffersgivesstd::moveto the last subscription only and explicitly copies for every earlier one (intra_process_manager.hpp:623-635): N−1 copies (cases [6], [10]) - mixed with >1 take_shared paths yields an extra
allocate_shared(*message)for the shared group (case [9])
Delivery itself never copies for an owning subscriber: the unique buffer dequeues by move (buffers/intra_process_buffer.hpp:256-259) and dispatch does callback(std::move(message)) for UniquePtrCallback and SharedPtrCallback alike (any_subscription_callback.hpp:914-919).
- Attaching an out-of-process subscriber costs a full copy per frame on every
UniquePtrsubscriber hop. Cases [11] vs [12] differ by exactly 1.75 MiB. When an inter-process subscriber exists,publish(unique_ptr)takesdo_intra_process_publish_and_return_shared(publisher.hpp:242-248), which mustallocate_shared(*message)if any owning subscription is registered (intra_process_manager.hpp:317-333). With only take_shared subscribers it promotes and copies nothing. Adding a second consumer in parrallel to any otherUniquePtrsubscription silently introduces a per-frame copy!
Implementations
Quick note; intra-process communications must be enabled on every participating node (NodeOptions::use_intra_process_comms(true)), or the path silently falls back to the RMW.
There are two patterns that are worth keeping in mind when using zero-copy:
Pattern A (fan-out / one-to-many)
Publisher make_unique and std::move; multiple subscribers take ConstSharedPtr.
- Zero copies for any number of subscribers (case [7]).
- Immune to a subscriber being added later, and to ros2 bag record / rviz attaching (case [12]).
- Works with transient_local.
- Cost: a consumer that needs to change the data must allocate a fresh output message; one allocation per frame, but still zero copies of the input.
- Use for: anything observed by more than one node, anything recorded or visualised, latched topics, and any node that retains messages past the end of the callback (mergers, accumulators, sync buffers — ConstSharedPtr is cheap to store and shareable).
#include <memory>
#include <utility>
#include <rclcpp/rclcpp.hpp>
#include <sensor_msgs/msg/point_cloud2.hpp>
using PointCloud2 = sensor_msgs::msg::PointCloud2;
// ---- Producer -------------------------------------------------------------
// Allocate fresh, hand ownership to rclcpp, never touch the message again.
class CloudProducer : public rclcpp::Node
{
public:
explicit CloudProducer(const rclcpp::NodeOptions & options)
: rclcpp::Node("cloud_producer", options)
{
pub_ = create_publisher<PointCloud2>("cloud/input", rclcpp::SensorDataQoS());
}
void publish_frame()
{
auto msg = std::make_unique<PointCloud2>();
fill(*msg);
pub_->publish(std::move(msg)); // msg is null from here on
}
private:
rclcpp::Publisher<PointCloud2>::SharedPtr pub_;
};
// ---- Observer -------------------------------------------------------------
// Read-only. N of these cost nothing extra; all see the producer's buffer.
class CloudObserver : public rclcpp::Node
{
public:
explicit CloudObserver(const rclcpp::NodeOptions & options)
: rclcpp::Node("cloud_observer", options)
{
sub_ = create_subscription<PointCloud2>(
"cloud/input", rclcpp::SensorDataQoS(),
// ConstSharedPtr (or `const PointCloud2::ConstSharedPtr &`) is what makes
// this a take_shared subscription. A mutable SharedPtr<T> would not.
[this](PointCloud2::ConstSharedPtr msg) {
RCLCPP_DEBUG(
get_logger(), "buffer @ %p",
static_cast<const void *>(msg->data.data()));
// Retaining past the callback is free and safe under take_shared.
latest_ = std::move(msg);
});
}
private:
rclcpp::Subscription<PointCloud2>::SharedPtr sub_;
PointCloud2::ConstSharedPtr latest_;
};
// ---- Transformer ----------------------------------------------------------
// Needs to change the data, so it allocates its own output. One allocation per
// frame; still zero copies of the input, and the output is itself Pattern A.
class CloudTransformer : public rclcpp::Node
{
public:
explicit CloudTransformer(const rclcpp::NodeOptions & options)
: rclcpp::Node("cloud_transformer", options)
{
pub_ = create_publisher<PointCloud2>("cloud/transformed", rclcpp::SensorDataQoS());
sub_ = create_subscription<PointCloud2>(
"cloud/input", rclcpp::SensorDataQoS(),
[this](PointCloud2::ConstSharedPtr in) {
auto out = std::make_unique<PointCloud2>();
transform(*in, *out); // e.g. a reprojection or a change of point type
pub_->publish(std::move(out));
});
}
private:
rclcpp::Publisher<PointCloud2>::SharedPtr pub_;
rclcpp::Subscription<PointCloud2>::SharedPtr sub_;
};
Pattern B (in-place mutate-and-forward / linear pipeline)
Publisher make_unique and std::move, single subscriber takes UniquePtr, mutates the buffer it now owns, and republishes std::move(msg).
- 0 copies and 0 allocations per hop — the same heap buffer travels the whole chain. This is strictly better than Pattern A, and it is the only way to get it.
- Valid only while each hop has exactly one in-process consumer and no out-of-process subscriber.
// One in, one out, same buffer the whole way through.
class CloudFilter : public rclcpp::Node
{
public:
explicit CloudFilter(const rclcpp::NodeOptions & options)
: rclcpp::Node("cloud_filter", options)
{
pub_ = create_publisher<PointCloud2>("cloud/filtered", rclcpp::SensorDataQoS());
sub_ = create_subscription<PointCloud2>(
"cloud/input", rclcpp::SensorDataQoS(),
// UniquePtr => take_ownership. We now own the producer's heap buffer.
[this](PointCloud2::UniquePtr msg) {
filter_in_place(*msg);
pub_->publish(std::move(msg)); // same buffer continues downstream
});
}
private:
// Compacts in place. Only ever shrinks `data`, so the vector never
// reallocates and `data.data()` is preserved across the whole chain.
static void filter_in_place(PointCloud2 & cloud)
{
const std::size_t stride = cloud.point_step;
auto * base = cloud.data.data();
std::size_t kept = 0;
for (std::size_t i = 0; i < cloud.width * cloud.height; ++i) {
const auto * src = base + i * stride;
if (!keep(src)) { // whatever your rejection criterion is
continue;
}
auto * dst = base + kept * stride;
if (dst != src) {
std::memmove(dst, src, stride);
}
++kept;
}
cloud.data.resize(kept * stride); // shrink only; capacity is retained
cloud.width = static_cast<uint32_t>(kept);
cloud.height = 1;
cloud.row_step = static_cast<uint32_t>(kept * stride);
cloud.is_dense = true;
}
rclcpp::Publisher<PointCloud2>::SharedPtr pub_;
rclcpp::Subscription<PointCloud2>::SharedPtr sub_;
};
The resize-down detail matters: growing data reallocates and you have quietly lost the zero-copy property while every address check still passes upstream of the growth. Pattern B only holds for pipelines that shrink or rewrite in place.
Practical Advice
The following table shows some common cases where defaulting to Pattern B can produce negative side-effects and how to get around it.
| Trigger | What it costs | What to switch to |
|---|---|---|
| A second in-process consumer subscribes to that hop | N−1 copies per publish (case [6]) | Change all consumers to ConstSharedPtr; the mutating one allocates its own output |
| Any out-of-process subscriber (bag record, rviz, metrics) | One extra full copy per publish (case [11] vs [12]) | Change to ConstSharedPtr |
Publisher is transient_local/latched |
Publisher always keeps a buffer and inter-process publishes, so an owning sub forces the copy every time (publisher.hpp:236-256) |
ConstSharedPtr |
| Consumer must retain the message after the callback and share it | Forces the topic onto the ownership side for no gain | ConstSharedPtr again… |
| Consumer changes the message schema or type | In-place mutation is impossible anyway, so ownership buys nothing | You guessed it; ConstSharedPtr; equal cost and safer |
Default to Pattern A
Use Pattern B only on a genuinely linear, single-consumer chain where the per-hop allocation matters, and treat “exactly one consumer” as an invariant you have chosen to maintain, because nothing in rclcpp will tell you when it breaks.
None of This Crosses a Process Boundary
Intra-process comms only apply within one process (thus the name), and publish-side loaning (borrow_loaned_message()) requires a bounded fixed-size type; PointCloud2’s unbounded data sequence disqualifies it on any RMW.
So there is no zero-copy option between processes for clouds; the only solution for the case presented here is keeping the chain inside one process.
Wrapping Up
This has been good to crystalise several things I ran into in the course of my PhD and earlier work but it is very easy to trip up on this and I’m sure I’ll come back to it in the future. Stay frosty.
Resources and Further Reading
Hey you!
Found this useful or interesting?
Consider donating to support.
Any question, comments, corrections or suggestions?
Reach out on the social links below through the buttons.