Zero-Copy Intra-Process Communications with ROS 2

Preamble

Using ROS, or any middleware for that matter, has many advantages for developers: code reuse, introspection, a standard set of message definitions so unrelated packages actually interoperate, off-the-shelf tooling that works on any topic without you writing a line of it (ros2 topic echo, rosbag2, RViz, tf2), language interoperability between C++ and Python, and the loose coupling that lets you start, stop, replace or relocate a node without touching its neighbours.

Unfortunately this isn’t a free lunch; copying and serialisation of data across boundaries, communication latency, and dropped packets will be all too familiar for anyone who has worked in this space. Yes, you can harden your system against it as best you can, but another approach is to forgo many of the problems entirely by coupling software together into a single process at run-time so as to share memory, removing the overheads from copying and serialisation whilst increasing determinism and reliability.

In ROS speak this is intra-process communication, not to be confused with inter-process communication (post on that to follow…). When set up correctly it can enable zero-copy publish and subscribe between nodes. When set up incorrectly it can be more trouble than it is worth.

Background

The ROS 2 tutorial for zero-copy intra-process communication is a bit misleading…

“This shows that the address of the message being received is the same as the one that was published and that it is not a copy. This is because we’re publishing and subscribing with std::unique_ptrs which allow ownership of a message to be moved around the system safely. You can also publish and subscribe with const & and std::shared_ptr, but zero-copy will not occur in that case.”

The design article is better but I thought it would be useful to share my findings from a users perspective.

Coming from using intra-process communication with Nodelets and PluginLib in ROS 1, I found the enforcement of std::unique_ptr on both sides of the communication strange (I think it is a hangover from before Dashing), especially in fan-out, one-to-many style publication, so I decided to run some tests.

Testing

Methodology

Using rclcpp 32.0.0 (ROS 2 Lyrical) a PointCloud2’s payload lives in msg->data (a std::vector<uint8_t>). If the subscriber’s msg->data.data() is the same address the publisher held before publishing, the exact same heap buffer was handed over; true zero-copy. A different address means a deep-copy happened somewhere in the chain. A global operator new override was also implemented to count bytes allocated during the publish and dispatch window as a cross-check.

  • Payload: 1.75 MiB (115k points × 16 byte point_step)
  • One process, use_intra_process_comms(true), SingleThreadedExecutor, QoS depth 10 volatile.
  • RMW during the run was rmw_cyclonedds_cpp, but the intra-process path is RMW-independent, so this does not affect any of the intra-process results; it only matters for cases [11]/[12] in the results.

Results

All message contents and results below are byte-identical across repeated runs.

# Publish Subscribers Outcome Heap During Publish and Dispatch
1 std::move(unique) 1× UniquePtr same buffer 0.00 MiB
2 std::move(unique) 1× ConstSharedPtr same buffer 0.00 MiB
3 std::move(unique) 1× mutable SharedPtr same buffer 0.00 MiB
4 by value 1× UniquePtr copy 1.76 MiB
5 by value 1× ConstSharedPtr copy 1.76 MiB
6 std::move(unique) 2× UniquePtr 1 copy + 1 same 1.76 MiB
7 std::move(unique) 3× ConstSharedPtr same buffer ×3 0.00 MiB
8 std::move(unique) 1× Unique + 1× ConstShared 1 same + 1 copy 1.76 MiB
9 std::move(unique) 1× Unique + 2× ConstShared 1 same + 2 copies 1.76 MiB
10 std::move(unique) 2× mutable SharedPtr 1 copy + 1 same 1.76 MiB
11 std::move(unique) 1× UniquePtr + 1 out-of-process same buffer 7.02 MiB
12 std::move(unique) 1× ConstSharedPtr + 1 out-of-process same buffer 5.27 MiB
13 std::move(unique) const T & same buffer 0.00 MiB
14 std::move(unique) const T & same buffer ×3 0.00 MiB
15 std::move(unique) const T & + 1× ConstSharedPtr same buffer ×2 0.00 MiB
16 by value const T & copy 1.76 MiB
17 std::move(unique) const T & + 1× UniquePtr 1 copy + 1 same 1.76 MiB
18 std::move(unique) const T & + 1 out-of-process same buffer 5.27 MiB
19 std::move(unique) const std::shared_ptr<const T> & same buffer 0.00 MiB
20 std::move(unique) const T & + 1× ConstSharedPtr + 1× const ConstSharedPtr & same buffer ×3 0.00 MiB

Why Though?

  1. Publish side: std::move of a unique_ptr is the only zero-copy publish. publish(const T&) deep-copies unconditionally when IPC is enabled (publisher.hpp:277-289, publisher.hpp:566-571). There is no publish(shared_ptr) overload, so publish(*shared_msg) also copies. Cases [4]/[5]/[16].

  2. Subscribe side: UniquePtr, ConstSharedPtr and const T & are zero-copy for a single subscriber. In any_subscription_callback.hpp:1021-1030: only shared_ptr<const T>, const shared_ptr<const T>& and const T& callbacks are “take_shared”. Everything else, including a mutable shared_ptr<T>, is a take_ownership subscription. All three take_shared forms, cases [2], [13] and [19], interchangeable and identical. Routing is in intra_process_manager.hpp:245-279:

  • all take_shared paths get one promotion, same buffer to all; zero copies for any N (cases [7], [14], [20])
  • any take_ownership paths: add_owned_msg_to_buffers gives std::move to the last subscription only and explicitly copies for every earlier one (intra_process_manager.hpp:623-635): N−1 copies (cases [6], [10])
  • mixed with >1 take_shared paths yields an extra allocate_shared(*message) for the shared group (case [9])

Delivery itself never copies for an owning subscriber: the unique buffer dequeues by move (buffers/intra_process_buffer.hpp:256-259) and dispatch does callback(std::move(message)) for UniquePtrCallback and SharedPtrCallback alike (any_subscription_callback.hpp:914-919).

  1. Attaching an out-of-process subscriber costs a full copy per frame on every UniquePtr subscriber hop. Cases [11] vs [12] differ by exactly 1.75 MiB. When an inter-process subscriber exists, publish(unique_ptr) takes do_intra_process_publish_and_return_shared (publisher.hpp:242-248), which must allocate_shared(*message) if any owning subscription is registered (intra_process_manager.hpp:317-333). With only take_shared subscribers it promotes and copies nothing. Adding a second consumer in parrallel to any other UniquePtr subscription silently introduces a per-frame copy!

Implementations

Quick note; intra-process communications must be enabled on every participating node (NodeOptions::use_intra_process_comms(true)), or the path silently falls back to the RMW.

There are two patterns that are worth keeping in mind when using zero-copy:

Pattern A (fan-out / one-to-many)

Publisher make_unique and std::move; multiple subscribers take a take_shared callback; ConstSharedPtr, const ConstSharedPtr & or const T &. - Zero copies for any number of subscribers (cases [7], [14], [20]). - Immune to a out of process subscriber being added later (cases [12], [18]). - Works with transient_local. - Cost: a consumer that needs to change the data must allocate a fresh output message; one allocation per frame, but still zero copies of the input. - Use for: anything observed by more than one node, anything recorded or visualised, latched topics, and any node that retains messages past the end of the callback. - Which spelling: const T & is the leanest if the callback is self-contained; use ConstSharedPtr when the node needs to keep the message past the end of the callback, since a reference does not survive the callback return.

#include <memory>
#include <utility>

#include <rclcpp/rclcpp.hpp>
#include <sensor_msgs/msg/point_cloud2.hpp>

using PointCloud2 = sensor_msgs::msg::PointCloud2;

// ---- Producer -------------------------------------------------------------
// Allocate fresh, hand ownership to rclcpp, never touch the message again.
class CloudProducer : public rclcpp::Node
{
public:
  explicit CloudProducer(const rclcpp::NodeOptions & options)
  : rclcpp::Node("cloud_producer", options)
  {
    pub_ = create_publisher<PointCloud2>("cloud/input", rclcpp::SensorDataQoS());
  }

  void publish_frame()
  {
    auto msg = std::make_unique<PointCloud2>();
    fill(*msg);
    pub_->publish(std::move(msg));  // msg is null from here on
  }

private:
  rclcpp::Publisher<PointCloud2>::SharedPtr pub_;
};

// ---- Observer -------------------------------------------------------------
// Read-only. N of these cost nothing extra; all see the producer's buffer.
class CloudObserver : public rclcpp::Node
{
public:
  explicit CloudObserver(const rclcpp::NodeOptions & options)
  : rclcpp::Node("cloud_observer", options)
  {
    sub_ = create_subscription<PointCloud2>(
      "cloud/input", rclcpp::SensorDataQoS(),
      // ConstSharedPtr is what makes this a take_shared subscription;
      // `const PointCloud2::ConstSharedPtr &` and `const PointCloud2 &` are
      // exactly equivalent. A mutable SharedPtr<T> would not be.
      // ConstSharedPtr is the right one here only because we retain below.
      [this](PointCloud2::ConstSharedPtr msg) {
        RCLCPP_DEBUG(
          get_logger(), "buffer @ %p",
          static_cast<const void *>(msg->data.data()));

        // Retaining past the callback is free and safe under take_shared.
        // This is the one thing a `const PointCloud2 &` callback cannot do.
        latest_ = std::move(msg);
      });
  }

private:
  rclcpp::Subscription<PointCloud2>::SharedPtr sub_;
  PointCloud2::ConstSharedPtr latest_;
};

// ---- Transformer ----------------------------------------------------------
// Needs to change the data, so it allocates its own output. One allocation per
// frame; still zero copies of the input, and the output is itself Pattern A.
// Nothing is retained past the callback, so a plain `const PointCloud2 &`
// is all this needs.
class CloudTransformer : public rclcpp::Node
{
public:
  explicit CloudTransformer(const rclcpp::NodeOptions & options)
  : rclcpp::Node("cloud_transformer", options)
  {
    pub_ = create_publisher<PointCloud2>("cloud/transformed", rclcpp::SensorDataQoS());
    sub_ = create_subscription<PointCloud2>(
      "cloud/input", rclcpp::SensorDataQoS(),
      [this](const PointCloud2 & in) {
        auto out = std::make_unique<PointCloud2>();
        transform(in, *out);  // e.g. a reprojection or a change of point type
        pub_->publish(std::move(out));
      });
  }

private:
  rclcpp::Publisher<PointCloud2>::SharedPtr pub_;
  rclcpp::Subscription<PointCloud2>::SharedPtr sub_;
};

Pattern B (in-place mutate-and-forward / linear pipeline)

Publisher make_unique and std::move, single subscriber takes UniquePtr, mutates the buffer it now owns, and republishes std::move(msg). - 0 copies and 0 allocations per hop — the same heap buffer travels the whole chain. This is strictly better than Pattern A, and it is the only way to get it. - Valid only while each hop has exactly one in-process consumer and no out-of-process subscriber.

// One in, one out, same buffer the whole way through.
class CloudFilter : public rclcpp::Node
{
public:
  explicit CloudFilter(const rclcpp::NodeOptions & options)
  : rclcpp::Node("cloud_filter", options)
  {
    pub_ = create_publisher<PointCloud2>("cloud/filtered", rclcpp::SensorDataQoS());
    sub_ = create_subscription<PointCloud2>(
      "cloud/input", rclcpp::SensorDataQoS(),
      // UniquePtr => take_ownership. We now own the producer's heap buffer.
      [this](PointCloud2::UniquePtr msg) {
        filter_in_place(*msg);
        pub_->publish(std::move(msg));  // same buffer continues downstream
      });
  }

private:
  // Compacts in place. Only ever shrinks `data`, so the vector never
  // reallocates and `data.data()` is preserved across the whole chain.
  static void filter_in_place(PointCloud2 & cloud)
  {
    const std::size_t stride = cloud.point_step;
    auto * base = cloud.data.data();
    std::size_t kept = 0;

    for (std::size_t i = 0; i < cloud.width * cloud.height; ++i) {
      const auto * src = base + i * stride;
      if (!keep(src)) {  // whatever your rejection criterion is
        continue;
      }
      auto * dst = base + kept * stride;
      if (dst != src) {
        std::memmove(dst, src, stride);
      }
      ++kept;
    }

    cloud.data.resize(kept * stride);  // shrink only; capacity is retained
    cloud.width = static_cast<uint32_t>(kept);
    cloud.height = 1;
    cloud.row_step = static_cast<uint32_t>(kept * stride);
    cloud.is_dense = true;
  }

  rclcpp::Publisher<PointCloud2>::SharedPtr pub_;
  rclcpp::Subscription<PointCloud2>::SharedPtr sub_;
};

The resize-down detail matters: growing data reallocates and you have quietly lost the zero-copy property while every address check still passes upstream of the growth. Pattern B only holds for pipelines that shrink or rewrite in place.

Practical Advice

The following table shows some common cases where defaulting to Pattern B can produce negative side-effects and how to get around it.

Trigger What it costs What to switch to
A second in-process consumer subscribes to that hop N−1 copies per publish (case [6]) Move all consumers onto take_shared callbacks; the mutating one allocates its own output
Any out-of-process subscriber (bag record, rviz, metrics) One extra full copy per publish (case [11] vs [12]) Any take_shared callback (cases [12], [18])
Publisher is transient_local/latched Publisher always keeps a buffer and inter-process publishes, so an owning sub forces the copy every time (publisher.hpp:236-256) Any take_shared callback
Consumer must retain the message after the callback and share it Forces the topic onto the ownership side for no gain ConstSharedPtr specifically — a const T & reference does not survive the callback
Consumer changes the message schema or type In-place mutation is impossible anyway, so ownership buys nothing Any take_shared callback; equal cost and safer

Default to Pattern A

Use Pattern B only on a genuinely linear, single-consumer chain where the per-hop allocation matters, and treat “exactly one consumer” as an invariant you have chosen to maintain, because nothing in rclcpp will tell you when it breaks.

Never Mix Ownership and Shared Subscribers on One Topic

Cases [8]/[9]/[17] are the worst topology from the test matrix: the shared group pays an extra allocate_shared(*message) on top of everything else. If a topic must serve both a mutator and observers, make everyone take_shared and let the mutator allocate; or restructure so the mutator gets its own dedicated hop upstream and the observed topic is downstream-only. Note that with more than one consumer, in-place mutation is semantically impossible regardless, so take_shared everywhere is not a compromise.

Never Use a Mutable shared_ptr<T> Callback

It is [[deprecated]] (any_subscription_callback.hpp:456), and it silently counts as an ownership subscription with none of UniquePtr’s benefits. Note the failure mode is delayed rather than immediate: with exactly one subscriber it is zero-copy (case [3]), and the N−1 copies only show up the day somebody adds a second consumer (case [10]). That is a worse property than being wrong from the start. Just don’t do it.

None of This Crosses a Process Boundary

Intra-process comms only apply within one process (thus the name), and publish-side loaning (borrow_loaned_message()) requires a bounded fixed-size type; PointCloud2’s unbounded data sequence disqualifies it on any RMW. So there is no zero-copy option between processes for clouds; the only solution for the case presented here is keeping the chain inside one process.

Wrapping Up

This has been good to crystalise several things I ran into in the course of my PhD and earlier work but it is very easy to trip up on this and I’m sure I’ll come back to it in the future. I’ve put a PR in to the docs with suggested changes so we’ll see if it makes it. Though a comment from mini-1235 against the PR pointed me to this interesting ROS Discourse post for soem further reading. Stay frosty.

Hey you!

Found this useful or interesting?

Consider donating to support.

Any question, comments, corrections or suggestions?

Reach out on the social links below through the buttons.