Binary Serialize, Header-Only C++ 20 Binary Serialization Classes and Functions Using a `std::format` Like Syntax

Unit Test and Documentation Generation Workflow Status

Overview

The binary_serialize functions and classes provide serializing and unserializing of binary data. Serialization provides a way to transform application objects into and out of byte streams that can be sent over a network (or used for file IO). Many serialization libraries transform objects to and from text representations, but this library keeps data in binary formats.

The serialization functionality in this repository is useful when explicit control is needed for every bit and byte. This allows a developer to match an existing wire protocol or encoding scheme or to define his or her own wire protocol. Support is provided for fundamental arithmetic types as well as many C++ vocabulary types such as std::optional. Both big and little endian support is provided.

Full flexibility is provided for sequences such as std::string or std::vector. The number of elements can be specified to take 8 or 16 or 32 (etc) bits, followed by the sequence of chars or array elements. Similar flexibility is provided for vocabulary types such as std::optional, where the boolean flag can be specified as 8 or 16 or 32 (etc) bits, followed by the object (or none, if there is no value in the optional).

This library uses std::format style formatting. For example:

  struct Hike {
    unsigned int  distance;
    int  elevation;
    std::optional<std::string>> name;
    std::vector<int> waypoints;
  };
  // ...
  chops::expandable_buffer<std::endian::big, chops::mutable_shared_buffer> buf;

  chops::binary_serialize(buf, "{32u}{16}{8u}{16u}{16u}{64}", 
                          hike_obj.distance, hike_obj.elevation, hike_obj.name, hike_obj.waypoints);

  // ...
  net_obj.send(buf.get_buf());

The buffer will contain the following (note that truncation or casting will happen between the application object types and the serialized types as needed):

  32 bit unsigned integer containing distance value
  16 bit signed integer containing elevation value
  8 bit unsigned integer corresponding to true or false for the optional
  16 bit unsigned integer for the size of the name string (if optional is true)
  0 - N 8 bit characters for the name string (if optional is true)
  16 bit unsigned integer for the size of the waypoints vector
  0 - N 64 bit signed integers for each waypoint value

The documentation overview provides a comparison with other serialization libraries as well as a rationale for the design decisions.

Inspiration and thanks go to Louis Langholtz, who steered me towards considering the std::format API.

Generated Documentation

The generated Doxygen documentation for binary_serialize is here.

Dependencies

The binary_serialize header files do not have any third-party dependencies. It uses C++ standard library headers only. The unit test code does have dependencies as noted below.

C++ Standard

binary_serialize uses C++ 20 features, including ... (fill in details here) ... the "spaceship" operator (<=>), std::span, and concepts / requires.

Supported Compilers

Continuous integration workflows build and unit test on g++ (through Ubuntu), MSVC (through Windows), and clang (through macOS).

Unit Test Dependencies

The unit test code uses Catch2. If the BINARY_SERIALIZE_BUILD_TESTS flag is provided to Cmake (see commands below) the Cmake configure / generate will download the Catch2 library as appropriate using the CPM.cmake dependency manager. If Catch2 (v3 or greater) is already installed using a different package manager (such as Conan or vcpkg), the CPM_USE_LOCAL_PACKAGES variable can be set which results in find_package being attempted. Note that v3 (or later) of Catch2 is required.

The unit test uses utilities from Connective C++'s utility-rack.

Specific version (or branch) specs for the dependencies are in the test/CMakeLists.txt file, look for the CPMAddPackage commands.

Build and Run Unit Tests

To build and run the unit test program:

First clone the binary-serialize repository, then create a build directory in parallel to the binary-serialize directory (this is called "out of source" builds, which is recommended), then cd (change directory) into the build directory. The CMake commands:

cmake -D BINARY_SERIALIZE_BUILD_TESTS:BOOL=ON ../binary-serialize

cmake --build .

ctest

For additional test output, run the unit test individually, for example:

test/binary_serialize_test -s

The example can be built by adding -D BINARY_SERIALIZE_BUILD_EXAMPLES:BOOL=ON to the CMake configure / generate step.