Introducing the MCAP File Format

A new and simplified way to record your robotics data

At Foxglove, our guiding mission is to accelerate robotics development for everyone. With Foxglove Studio, we tackled data visualization and debugging; with Foxglove Data Platform, we tackled data storage and management.

For the last several months, our team has been working to streamline and standardize another key component of the robotics stack: data recording. We noticed many companies spending valuable in-house resources to develop custom file formats, only to create more future work and complicate third-party tool integrations. That’s why we’re thrilled to introduce MCAP (pronounced “ em-cap”), a standardized container format for storing heterogeneous robotics data. With MCAP, teams are empowered to focus on their important robotics problems and avoid wasting precious time building commodity tools.

Recording robotics data before MCAP

Which format robotics teams use to store their log data today depends largely on their framework. Those using ROS 1 default to the “ bag file” format; those on ROS 2 default to a SQLite-based format. Companies that don’t use ROS at all often employ a custom in-house binary format such as length-delimited protobuf, or store their messages as opaque bytes inside existing file formats such as HDF5.

These existing storage options have several shortcomings. The ROS 1 bag format is difficult to work with outside of the ROS ecosystem, while the ROS 2 SQLite format is not fully self-contained, making it difficult for third-party tools to read. Custom in-house formats lack interoperability and require developing corresponding libraries in multiple languages to read and write files.

We evaluated many existing data storage formats in the industry and identified a clear need for a general-purpose, open source data container format — specifically optimized for robotics use cases. Designing this format would not only solve an industry-wide problem, but also make it easier for teams to leverage third-party tools and share their own tooling.

Benefits of recording with MCAP

As a container format, MCAP can store message data encoded in a variety of serialization formats. While message serialization formats (e.g. Protobuf, MessagePack, JSON, etc.) translate your data into bits for efficient storage, a container format can embed multiple heterogeneous data streams, with corresponding metadata, into a single file.

MCAP is optimized for high-performance writing — because it leverages an append-only structure, data can be streamed to disk or over the network without seeking backwards. This makes it possible to recover partially-written files, even if on-device recording was interrupted by a software crash or power loss. MCAP files are also completely self-contained — message schemas are embedded in the file, and no additional dependencies are required for decoding. They also support the ability to embed attachments containing calibration information, core dumps, or any other application-specific data.

MCAP is also designed to support efficient reading and seeking within files, even over remote connections. Similar to ROS 1 bag files, MCAP supports indexed reading of messages and fast local or remote access to summary data. MCAP readers can efficiently extract data for a given topic and timestamp without having to scan the entire file.

Try MCAP today

MCAP is currently in beta, and we’d love your feedback! Check out our GitHub repo to get started, and share your thoughts in our Slack community to help us build towards v1.0.

If you’re currently using an in-house recording format or storing binary data directly, MCAP could be a great alternative to consider. We provide libraries in C++, Python, Go and TypeScript, as well as a suite of conformance tests, to facilitate interoperability with other tools. We also support visualizing MCAP files in Foxglove Studio, as well as importing them into Foxglove Data Platform.

If you’re currently on ROS, use our mcap CLI tool to convert your .bag (ROS 1) or .db3 files (ROS 2) to self-contained MCAP files that include all the necessary message schema definitions. You can also track progress on our ROS 2 MCAP plugin, coming soon.

Special thanks

We want to extend special thanks to Emerson Knapp, Michael Orlov, and other members of the ROS Tooling Working Group for collaborating with us on the MCAP spec. This work would not have been possible without your feedback and contributions.

Originally published at https://foxglove.dev on February 16, 2022.