[michalpelka]

ROS1 bags without ROS: bag_to_laz / laz_to_bag, ring in sessions, canonical layout

ROS1 bags join the existing MCAP ↔ session tools, and no ROS installation is needed:

Tool	Direction	How
mcap_to_laz (existing)	MCAP → session	now keeps the ring; new IMU options
bag_to_laz (new)	ROS1 bag → session	the same source as mcap_to_laz, with a ROS1 reader (-DHDMAPPING_IMPORT_ROS1_BAG)
laz_to_mcap (existing)	session → MCAP	now writes the ring; --lidar-type canonical; IMU unit option
laz_to_bag (new)	session → ROS1 bag	the same source as laz_to_mcap, with a ROS1 writer (-DHDMAPPING_EXPORT_ROS1_BAG)

The session stays the hub, so any of the four tools can be chained with any other. The "canonical" bag for the benchmark's spinning-lidar branch is one more --lidar-type.

Changes

• rosbags/Ros1Bag:
  • A ROS1 bag 2.0 reader (uncompressed, bz2 and lz4 chunks).
  • A writer (uncompressed chunks with per-chunk index records and the index section).
  • ROS1 PointCloud2 and Imu codecs.
  • Ros1BagFileReader and Ros1BagFileWriter, with the same API as McapFileReader and McapFileWriter. That means the same options, topic auto-detection, McapPoint/McapImuSample structs and layouts.
  • The reader also decodes livox_ros_driver/CustomMsg, with ring = line, like the MCAP reader.
• cdr_serializer.hpp: decodePc2Points() is split out of decodePc2(). The MCAP and ROS1 readers share it, so presets, time-field rules and warnings are identical. decodePc2()'s behaviour is unchanged.
• McapWriter:
  • The layouts are exposed as pointCloudFields(), pointCloudPointStep() and packPointCloud(), so MCAP and ROS1 write byte-identical records. Existing layouts are unchanged; the tests confirm it.
  • New PointCloudLayout::Canonical: x y z intensity (float32), ring (uint16), time (float32 s) and t (uint32 ns), relative to the first point.
  • Why float32 time: FAST-LIO and LIO-SAM Velodyne point types declare float time, and PCL matches fields by name and type. They read the existing Velodyne layout's float64 time as 0.
• Ring in sessions: MandeyeSessionWriter(dir, with_rgb, with_ring) stores McapPoint::ring as a uint16 LAS extra-bytes attribute named ring.
  • user_data stays the lidar id and point_source_ID stays free, so standard LAS semantics are kept. PDAL and CloudCompare show it as a ring field.
  • load_point_cloud() takes an optional std::vector<uint16_t>* rings, filled for exactly the points it keeps; it holds 0s when the file has no attribute. Existing callers are unaffected.
• Import (mcap_to_laz, bag_to_laz):
  • Several inputs: bag_to_laz part0.bag part1.bag … session/ reads them in the order given into one session, for recordings split into parts. Chunks stay on the absolute 20 s grid, so a chunk straddling two files stays one chunk. All inputs must resolve to the same lidar and IMU topics.
  • Sessions are written with the ring.
  • --imu-rotation qx qy qz qw rotates the IMU into the lidar frame, the frame HDMapping assumes.
  • --imu-acc-unit g|ms2 gives the bag's unit. The default g copies values as before; ms2 divides by 9.80665.
• Export (laz_to_mcap, laz_to_bag):
  • --lidar-type canonical, with the ring taken from the session.
  • Points are sorted by time, then ring.
  • --imu-acc-unit g|ms2 gives the output unit. The default g keeps the existing behaviour; ms2 writes m/s² as sensor_msgs/Imu defines it.

Usage: a ROS1 Hesai bag to the canonical bag

bag_to_laz  oxford.bag session/ --lidar-topic /hesai/pandar --imu-topic /alphasense_driver_ros/imu \
            --imu-rotation 0.000644947 -0.001879371 -0.712962475 0.701199373 --imu-acc-unit ms2
laz_to_bag  session/ canonical.bag  --lidar-type canonical --msg_hz 10 --imu-acc-unit ms2 \
            --lidar-topic /lidar/pointcloud --imu-topic /lidar/imu
laz_to_mcap session/ canonical.mcap --lidar-type canonical --msg_hz 10 --imu-acc-unit ms2 \
            --lidar-topic /lidar/pointcloud --imu-topic /lidar/imu

Verification

• Unit tests: hdmapping_rosbags_tests passes: 38 test cases, 29,896 assertions, 0 failures. The new tests cover:
  • ROS1 codecs, including rejecting truncated messages;
  • a writer/reader round trip over 20 chunks;
  • a Ros1BagFileWriter → Ros1BagFileReader canonical round trip, with topics auto-detected and no empty serial-number topic;
  • Livox CustomMsg decoding;
  • the session ring attribute (VLR type and name, values, user_data and point_source_ID untouched);
  • the Canonical MCAP layout.
• Real data: Oxford Spires 2024-03-12-keble-college-02 (Hesai QT64 and Alphasense IMU; 5.1 GB ROS1 bag with bz2 chunks):
  • Several inputs: the canonical bag split in two in the middle of a chunk imports byte-identical (16 LAZ files and 16 IMU csv files) to the unsplit bag.
  • bag_to_laz: 16 chunks, all 182,696,017 points from 3007 messages, 119,865 IMU samples. It took 533 s and 33 MB of memory. The IMU csv matches an independent Python conversion (rotation and g) to float precision.
  • laz_to_bag and laz_to_mcap (canonical): 3001 messages at 10 Hz, all points and IMU samples. Each took about 60 s and 1.6 GB of memory.
  • rosbag info (ROS Noetic) shows standard md5sums and a valid index.
  • Field check in Python (rosbags, mcap-ros2-support):
    • the field table is as specified;
    • time is sorted and starts at 0;
    • t equals time × 10⁹;
    • all 64 rings are present after the session round trip;
    • intensity is 4–255;
    • IMU gravity reads 9.70 m/s² along +z, in the lidar frame;
    • the first MCAP cloud is byte-identical to the first ROS1 cloud.

Known limitations

• IMU lever arm is not applied. --imu-rotation turns the IMU into the lidar frame, but the accelerometer still measures at the IMU's own origin. At the lidar origin, r away, the specific force is f_imu + α × r + ω × (ω × r). For Oxford Spires r is 8.6 cm: about 0.09 m/s² at 1 rad/s, and about 0.4 m/s² during a 5 rad/s² turn. Fixing it needs a lever-arm option and angular acceleration taken from the gyro over a short window, because a raw difference amplifies noise. Gyro values are unaffected.

• No progress output while reading a bag. bag_to_laz and mcap_to_laz log only when a pass ends. On a 5.1 GB bz2 bag that's two silent passes of about 4.5 min each, IMU first and then lidar, and it looks stuck. Both formats know their message counts up front: MCAP from its statistics record, and ROS1 from the chunk-info records of its index, which the reader currently skips. That's enough for a percentage and an ETA. The same import could also be faster: one pass instead of two, and chunks decompressed in parallel.

• IMU units are a flag, not a rule. sensor_msgs/Imu is m/s² by definition and a session is in g, so each direction has exactly one correct conversion. --imu-acc-unit defaults to g (copy as is) on both import and export, as the tools did before: a standard ROS bag imported with defaults is off by 9.81, and laz_to_mcap/laz_to_bag write g into sensor_msgs/Imu. The canonical pipeline passes ms2 on both sides. A fix would always convert, and auto-detect old files in g from |a| at rest.

• Sweep boundaries: the session doesn't keep the sensor's own sweep boundaries, so --msg_hz cuts on an absolute 1/hz grid. For a 10 Hz spinning lidar, each message still covers about 360°. The first message is short if the recording starts partway through a bin.

• Import speed: bag_to_laz reads the bag twice (IMU pass, then lidar pass), as mcap_to_laz does. With bz2 chunks, decompression dominates.

• MCAP on Humble: no Humble image used by the benchmark has rosbag2_storage_mcap, so ros2 bag play there needs that plugin, or a conversion to sqlite3.

• lz4 chunks follow the LZ4 frame format that roslz4 writes. They're tested only indirectly; the real-data run uses bz2. On Windows the lz4 path is compiled out, like the mcap target's LZ4 support.

• Formatting: clang-format wasn't available, so formatting follows the neighbouring files by hand.

🤖 Generated with Claude Code