The Raspberry Pi is becoming a core piece of hardware for developing simple to complex embedded projects. It is powerful enough to run an OS like Raspberry Pi OS or Ubuntu, and allows easy integration of various sensors, cameras, or displays. It has a wide cost range, from the simple 15€ Raspberry Pi Zero to the latest Raspberry Pi 5 (from 50€ to 327€; 1GB RAM to 16GB RAM). It is thus used in a wide variety of products, from simple home projects, education and research purposes to industry-grade use. And for this same reason, a wide variety of resources are available, allowing fast prototyping.

While today’s Raspberry Pis are more powerful than low-grade laptops, using one may still be considered embedded development. Embedded development often implies knowledge of compilation toolchains and configuration. As it is used by a wide variety of people for various purposes, not all solutions are adapted to the needs of a given person. Additionally, not all projects are started from scratch, and not all of them are able to use modern cross-compilation tools. Not everyone has the time to dive deep into complex cross-compilation research and setup.

Today’s topic will cover a specific case regarding ROS2 projects. It is thus directed towards research students who hold some basic Linux and ROS2 knowledge and wish to improve their development experience with a ready-to-use cross-compilation solution. Some additional knowledge about docker, virtualization and emulation may be necessary for a deep understanding of this article. For this same reason, it will remain focused on this goal and avoid explaining in depth why and how things work. For further information, you may follow some links. Various solutions will be shared: how each one started, what it improved, and what its drawbacks are. We will aim to increase development comfort and reproducibility, allowing any person to work on a ROS2 project from any distribution.

To keep the performance claims honest and reproducible, every build duration in this article was measured against a public reference project, ros2/demos; the exact protocol is described in the Benchmark section at the end.

The Problem

“The Raspberry Pi is super powerful. It uses an ARM Cortex-A, 64-bit architecture, possibly multi-core, and has a lot of RAM! We are freed from all these low-level microcontrollers requiring proprietary software that eats more than 10GB on my machine. I will just ssh into the machine, develop, build and test from here. Nothing beats using the production hardware itself when coding.”

The answer? Yes, do that! If it works and you have no trouble for your current use, there’s no need to add complexity. Keep It Simple, Stupid (KISS principle), as they say.

However, after some time, you may face several problems that will force you to complicate your current development process.

• The Raspberry Pi died. If you did not make any backups, everything is lost forever. That is why you should use git and GitHub/GitLab (and you most likely already are).
• Your program grew to the point where it can no longer be compiled on the Raspberry Pi.
• You need to work on the project without having the Pi available.
• ROS2 is very specific about the Linux distribution it needs (Ubuntu).

Hence, we will try to find a solution to these problems that allows for portability, rather than one that only works on one particular Linux distribution. That is, a Docker image.

Using an emulated multi-arch docker

The first solution is the simplest one: recreate the Ubuntu environment used by the Raspberry Pi inside a Docker container. This implies downloading the packages compiled for aarch64 and emulating them on our x86 machine. Luckily, various arm64-based Docker images already exist and can be found at https://hub.docker.com/r/arm64v8/ros/ . As a lot of versions of ROS2 exist, finding the one used for your project is essential for version matching.

1
2
3
4
5

FROM arm64v8/ros:jazzy-ros-base
RUN apt-get update -y && \
    apt-get install -y cmake meson git libboost-dev liblgpio-dev && \
    apt-get autoremove -y && \
    rm -rf /var/lib/apt/lists/*

This Dockerfile uses a basic ROS2 image (jazzy version) and installs the dependencies of our project. It also removes all the package cache to make the image as light as possible.

To build the image, we first need to teach the Linux kernel how to run arm64 executables on an x86_64 machine. This is done by registering qemu emulators through the kernel’s binfmt_misc mechanism, and the tonistiigi/binfmt image does exactly that. This step is only done once per computer (and again after a reboot).

1

docker run --rm --privileged tonistiigi/binfmt --install arm64

Let’s break this command down.

• --privileged gives the container the right to modify the host kernel’s binfmt_misc registrations; that is the whole point of the command, and also why you should only run it with images you trust.
• --rm removes the container as soon as it exits, since this is a one-shot setup command with nothing worth keeping.
• tonistiigi/binfmt is the image (maintained by a Docker engineer) that ships the qemu emulators.
• --install arm64 registers the arm64 emulator; use --install all if you want every architecture.

And then, we can start building the Docker image.

1

docker build --platform=linux/arm64/v8 --rm -t rpi:latest .

Here, --platform forces the target architecture of the image instead of defaulting to the host’s, -t gives the image a name (a tag), and the final . is the build context: the directory containing the Dockerfile.

Ensure you are in the same directory as the Dockerfile

This will build an image with an arm64 architecture, and not the one used on your machine. It may take a while depending on the number of packages to install.

Finally, once our Docker image is built, we just need to start it and connect to the terminal.

1
2
3

docker run --platform=linux/arm64/v8 --rm \
    -v `pwd`/myproject:/app/myproject \
    -it rpi:latest

The -v argument of Docker allows you to map a directory to the container so you can access it from inside the container. The syntax is as follows: -v MyHostPath:MyDockerPath. As the paths need to be absolute, pwd is used.

After starting the container, you should arrive in a bash terminal. This bash terminal is no longer your machine, but the Docker container. To confirm success, uname -m should output aarch64 (and not x86_64). If this is not the case, one of the steps failed, and you are still running the Docker container on x86_64 (your host architecture) instead of the one used by a Raspberry Pi. From there, you should be able to navigate to your project directory using the MyDockerPath you specified earlier with the -v command. If your project is a ROS2 project using colcon, a simple colcon build should start the compilation as if you were using the Raspberry Pi itself.

This solution is pretty useful, as you may run your executable from the Docker container and test it locally (especially if you don’t use any external sensors or hardware) in the same conditions as the Raspberry Pi. You may then send the install directory (or build) to the Raspberry Pi using scp or rsync.

However, the major drawback of this solution is the performance. We are talking about emulating a full arm64 environment, which adds an additional cost of time and computation to the compilation process. This appears as a 10x overhead (measured at 11-15x in the benchmark at the end of this article). For a project on my laptop, a full build could take up to 7 minutes, and compilation error feedback took between 30 seconds and 2 minutes. This is quite a lot, but the fact that the setup is pretty simple still makes it worthwhile to know.

Using a docker and a sysroot

In the previous step, we realized that arm64 emulation was definitely not the best way to solve the problem. A 10x overhead is a lot of time lost and doesn’t scale well.

In this solution, we aim to stop emulating and turn towards real cross-compilation. While compilation usually implies compiling from host to host (x86 -> x86 or arm64 -> arm64), cross-compiling, in our case, means compiling from x86 to arm64. We no longer need qemu and will be able to use our computer to its full potential.

The plan is the following: a Dockerfile will be used to download all the cross-compilation tools, and a sysroot will be created to recreate the Raspberry Pi architecture and provide the arm64 libraries.

We will start with the sysroot. A sysroot is a directory on your computer that is basically the root of a system.

$ tree -L 2 sysroot
sysroot
├── lib
├── opt
│   └── ros
└── usr
    ├── include
    ├── lib
    ├── local
    └── share

When we cross-compile, we tell the compiler “The root of the system is here,” so that it searches for libraries in that directory instead of our x86_64 root (/).

The fastest way to create a sysroot is to mount your Raspberry Pi SD card onto your computer and copy the above directories into a directory named sysroot at the same level as the Dockerfile. This created sysroot now contains all the libraries and headers useful for our cross-compilation process. Ensure the Raspberry Pi you used has all the libraries you need pre-installed.

The second step is to create the Dockerfile that will host the cross-compilation tools.

1
2
3
4
5
6
7
8
9

FROM ubuntu:noble
RUN apt-get update -y && \
    apt-get install -y \
        build-essential \
        cmake meson git \
        gcc-aarch64-linux-gnu \
        g++-aarch64-linux-gnu && \
    apt-get autoremove -y && \
    rm -rf /var/lib/apt/lists/*

In this case, we no longer need a ROS2-based image, as our sysroot is the one holding all our environment. We thus only focus on installing our toolchain (cmake, colcon or meson) as well as our cross-compilers: aarch64-linux-gnu- for both gcc and g++. These are the tools used to compile for the arm64 architecture instead of our x86_64 host. Note that colcon is not part of Ubuntu’s own repositories: it comes from the ROS2 apt repository, and the tested snippet to install it is shown in the multi-stage Dockerfile of the last chapter.

But for that, we first need to prepare our project and tell our toolchains how to cross-compile. This step is a bit tricky, as each toolchain has its own dedicated configuration file. We will thus quickly cover the cmake and meson toolchain configurations.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

# The sysroot folder should contain all the libraries necessary to compile.
# It should be a mirror of a Raspberry Pi filesystem:
# ${CMAKE_SYSROOT}/lib
# ${CMAKE_SYSROOT}/opt/ros/jazzy
# ${CMAKE_SYSROOT}/usr/include
# ${CMAKE_SYSROOT}/usr/lib
set(CMAKE_SYSROOT /sysroot)
set(CMAKE_C_COMPILER /usr/bin/aarch64-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER /usr/bin/aarch64-linux-gnu-g++)

set(ENV{PKG_CONFIG_SYSROOT_DIR} "${CMAKE_SYSROOT}")
set(ENV{PKG_CONFIG_PATH} "${CMAKE_SYSROOT}/usr/lib/aarch64-linux-gnu/pkgconfig:${CMAKE_SYSROOT}/usr/lib/pkgconfig")

# Where to search for the arm64 libraries and CMake packages (e.g. OpenCV).
list(APPEND CMAKE_PREFIX_PATH "${CMAKE_SYSROOT}/usr/lib/aarch64-linux-gnu"
                              "${CMAKE_SYSROOT}/usr/lib/aarch64-linux-gnu/cmake/opencv4")

set(CMAKE_FIND_ROOT_PATH ${CMAKE_SYSROOT}
                         ${CMAKE_SYSROOT}/opt/ros/jazzy)

# Programs (compilers, python) run on the host;
# libraries, headers and packages only come from the sysroot.
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

Our toolchain.cmake

For further details, check Cross Compiling with CMake

And the variant for meson:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

[binaries]
c = 'aarch64-linux-gnu-gcc'
cpp = 'aarch64-linux-gnu-g++'
ar = 'aarch64-linux-gnu-ar'
strip = 'aarch64-linux-gnu-strip'
pkgconfig = 'aarch64-linux-gnu-pkg-config'
cmake = '/usr/bin/cmake'

[host_machine]
system = 'linux'
cpu_family = 'aarch64'
cpu = 'armv8-a'
endian = 'little'

[properties]
pkg_config_libdir = '/sysroot/usr/lib/aarch64-linux-gnu/pkgconfig/'
sys_root = '/sysroot'

[built-in options]
c_args = ['-isystem', '/sysroot/usr/include',
    '--sysroot=/sysroot']
cpp_args = ['-isystem', '/sysroot/usr/include',
    '--sysroot=/sysroot',
    '-Wno-shadow']
c_link_args = ['--sysroot=/sysroot',
    '-L/sysroot/usr/lib/aarch64-linux-gnu']
cpp_link_args = ['--sysroot=/sysroot',
    '-L/sysroot/usr/lib/aarch64-linux-gnu']

Our arm64_cross_file.txt

For further details, check Cross Compiling with meson

With these files, we are telling our toolchain that this is not an ordinary compilation process from host to host, but a cross-compilation towards an arm64 architecture.

Hence, we need to have it at the root of our project myproject (from where the build commands are executed).

The build process is the following:

docker build -t rpi:latest .

As in the previous step, we launch Docker. This time, we also mount the sysroot alongside it so it can be used by the toolchains.

docker run \
    -v `pwd`/sysroot:/sysroot \
    -v `pwd`/myproject:/app/myproject \
    --user "$(id -u):$(id -g)" -e HOME=/tmp \
    -e PYTHONPATH=/sysroot/opt/ros/jazzy/lib/python3.12/site-packages \
    -e AMENT_PREFIX_PATH=/sysroot/opt/ros/jazzy \
    -it \
    rpi:latest

Three non-obvious options are doing a lot of work here, so let’s quickly cover them.

• PYTHONPATH exposes the sysroot’s pure-Python ROS2 modules to the host’s Python: ament’s CMake scripts import ament_package during configuration, and without it the build fails with ModuleNotFoundError: No module named 'ament_package' (adjust python3.12 to the Python version of your ROS distro).
• AMENT_PREFIX_PATH tells ament where its resource index lives; it is resolved through this environment variable and not through CMake’s search paths, so without it you are greeted by No 'rosidl_typesupport_c' found.
• Finally, --user makes the container write the build/, install/ and log/ directories as your own user instead of root, keeping them editable and removable from the host.

Once again, we should arrive in a bash-like terminal. Building now requires changing directory to /app/myproject and executing the following command.

colcon build \
    --cmake-args \
        -DCMAKE_TOOLCHAIN_FILE=`pwd`/toolchain.cmake

With this solution, we managed to solve the major build drawback of the previous solution. That is, an order-of-magnitude speed increase (~11x on the reference benchmark) by removing the arm64-to-x64 translation step. We now have a build system able to use the full capacity of our host computer. The project used to test this solution saw a bump from 7min30 to just about 30s.

While this solution solves the performance issue, it also brings a maintainability issue with it. Indeed, our sysroot is now the root of the problem. If we were to keep iterating on our project and add further dependencies, we would have to add them to the sysroot as well. A version mismatch between the sysroot and the Raspberry Pi could also happen pretty fast. And finally, we want to be freed from the Raspberry Pi. (And of course, we want to end up with something simple and reproducible.)

The next chapter will suggest further improvements to the current solution.

Using a multi-stage docker

The first solution was fairly simple to set up but low in performance. The second one is a pain to maintain, but the performance obtained is optimal. Could we benefit from both solutions at the same time?

The answer might be yes. And for that, we need to understand that the problem is no longer the compilation. The new goal is to make the solution simple and deterministic. For that, we will have to dive deeper into Docker.

In the first solution, we created a fully emulated arm64 image. We specified all the libraries we needed, just like we would on the Raspberry Pi. And that is exactly what we will use to create our sysroot.

Docker’s Multi Stage Build will help us with that. The first stage will be dedicated to downloading all the required dependencies for arm64, like in the emulated solution. The second stage will remain as in the sysroot solution, to download all the tools to cross-compile our project.

flowchart TD
    subgraph s1 [Stage 1: arm64, emulated once]
        A[ros:jazzy-ros-base] --> B[+ project dependencies]
    end
    subgraph s2 [Stage 2: x86_64, native speed]
        C[(/sysroot)] --> D[aarch64-linux-gnu-gcc<br>colcon + toolchain.cmake]
    end
    B -- "COPY --from=sysroot" --> C
    D --> E[install/: arm64 binaries]
    E -- "scp / rsync / docker pull" --> F([Raspberry Pi])

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

ARG ROS_DISTRO=jazzy

# ---- Stage 1: arm64 sysroot ----
# Installing all build libraries your project needs
FROM --platform=linux/arm64 ros:${ROS_DISTRO}-ros-base AS sysroot
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        libboost-dev liblgpio-dev && \
    rm -rf /var/lib/apt/lists/*

# ---- Stage 2: x86_64 cross-compilation environment ----
FROM ubuntu:24.04
ENV DEBIAN_FRONTEND=noninteractive

COPY --from=sysroot / /sysroot

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        build-essential cmake meson ninja-build git \
        python3 python3-pip \
        gcc-aarch64-linux-gnu g++-aarch64-linux-gnu \
        curl ca-certificates && \
    rm -rf /var/lib/apt/lists/*

# Installing colcon from the ROS2 apt repository
# ("noble" must match the Ubuntu release of the image above)
RUN curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc \
        -o /usr/share/keyrings/ros-archive-keyring.asc && \
    echo "deb [signed-by=/usr/share/keyrings/ros-archive-keyring.asc] http://packages.ros.org/ros2/ubuntu noble main" \
        > /etc/apt/sources.list.d/ros2.list && \
    apt-get update && \
    apt-get install -y --no-install-recommends python3-colcon-common-extensions && \
    rm -rf /var/lib/apt/lists/*

Now that we have our new Dockerfile, we simply need to let the magic happen.

First, we allow emulation for the first stage, with the same binfmt registration as in the emulated solution (nothing to do if you already ran it):

1

docker run --rm --privileged tonistiigi/binfmt --install arm64

Then we can simply start building with:

docker build -t test .

After a while, we will finally get our generated sysroot inside our cross-compilation Docker image. This solution still relies on the same toolchain.cmake and environment variables as the previous chapter; only the way the sysroot is created changed. Running a build now looks like this:

1
2
3
4
5
6

docker run --rm \
    --user "$(id -u):$(id -g)" -e HOME=/tmp \
    -e PYTHONPATH=/sysroot/opt/ros/jazzy/lib/python3.12/site-packages \
    -e AMENT_PREFIX_PATH=/sysroot/opt/ros/jazzy \
    -v `pwd`/myproject:/app/myproject \
    -it test

But now, it is easily maintainable, as you just have to add your new dependency to the Docker image and rebuild. A person joining the project won’t have to spend hours configuring the tools and searching for the dependencies.

Sending the build to the Pi

When building on your host machine, you have to send the files to the target Raspberry Pi. For a ROS2 project, that implies syncing the install/ directory. For that, you have multiple options, such as:

• Using scp: scp PROJECTPATH/install USER@HOST:~/install
• Using rsync: rsync -urltv -e ssh "SRC_DIR" USER@HOST:DST_DIR
• Using a Docker image: you’d need yet another Dockerfile, but this time for installing only the runtime dependencies, and it would include the install directory. Then you may simply have the Raspberry Pi pull the image and run it.

Benchmark

To keep the numbers honest and reproducible, the benchmark uses a public project instead of my own: ros2/demos (branch jazzy, commit 37e25e0). Each measurement is a cold build of the demo_nodes_cpp and image_tools packages (colcon build --packages-up-to demo_nodes_cpp image_tools), repeated 3 times, keeping the median. The Docker-based solutions were measured on a desktop machine (AMD Ryzen 5 7500F @ 5.08GHz, 6 cores / 12 threads, 32GB of RAM) and on a laptop machine (Intel i7-8650U @ 1.90GHz, 8 threads). The Docker + Sysroot solution is not measured separately: it uses the same cross-compilers and an equivalent sysroot as the Multi-Stage solution, so its build duration is identical by construction.

+ is low/simple, - is high/complex; durations are the median of 3 cold builds, with raw samples in parentheses.

The measured emulation overhead is about 11x on the desktop and about 15x on the laptop, compared to cross-compilation. Painful enough to justify the extra setup either way.

Conclusion

That’s it for today. We covered the journey from a simple workaround to a full, modern cross-compilation solution. The solution suggested is just one among many. Yet I believe it is easily accessible without heavily modifying a given project. It allows teams to share the same environment and unlock various uses, such as pipelines and test integration. Yet, running the executable would still require emulation or the Pi.

If you want to push further, here are some natural next steps.

• Build the cross-compilation image in your CI, so that every commit produces ready-to-deploy arm64 binaries.
• Package the install/ directory into a small runtime image (as sketched in the previous section), turning deployment into a simple docker pull on the Pi.
• Add ccache to the cross toolchain to make warm rebuilds even faster.
• Check out arm64 build machines and CI runners (like GitHub Actions or AWS Graviton). Building on an arm64 host deletes the problem we are trying to solve.

And if this solution does not fit your project, others exist. The official ROS2 cross-compilation guide covers the general approach, and the ros-tooling/cross_compile project automated a similar Docker-based pipeline (it is no longer actively maintained, but remains a good source of ideas). When you outgrow a stock Raspberry Pi OS, full image builders like Yocto or Buildroot become the industry-grade path, at the price of a much steeper learning curve.