Implement Binary Communication Protocols in C++

Introduction

This article is about easy and compile-time configurable implementation of binary communication protocols using C++11 programming language, with main focus on embedded systems (including bare-metal ones).

Interested? Then buckle up and read on.

Background

Almost every electronic device/component nowadays has to be able to communicate to other devices, components, or outside world over some I/O link. Such communication is implemented using various communication protocols, which are notorious for requiring a significant amount of boilerplate code to be written. The implementation of these protocols can be a tedious, time consuming and error-prone process. Therefore, there is a growing tendency among developers to use third party code generators for data (de)serialization. Usually such tools receive description of the protocol data structures in separate source file(s) with a custom grammar, and generate appropriate (de)serialization code and necessary abstractions to access the data.

The main problem with the existing tools is that their major purpose is data structures serialization and/or facilitation of remote procedure calls (RPC). The binary data layout and how the transferred data is going to be used is of much lesser importance. Such tools focus on speed of data serialization and I/O link transfer rather than on safe handling of malformed data, compile time customization needed by various embedded systems and significantly reducing amount of boilerplate code, which needs to be written to integrate generated code into the product's code base.

The binary communication protocols, which may serve as an API or control interface of the device, on the other hand, require a different approach. Their specification puts major emphasis on binary data layout, what values are being transferred (data units, scaling factor, special values, etc.) and how the other end is expected to behave on certain values (what values are considered to be valid and how to behave on reception of invalid values). It requires having some extra meta-information attached to described data structures, which needs to propagate and be accessible in the generated code. The existing tools either don't have means to specify such meta-information (other than in comments), or don't know what to do with it when provided. As a result, the developer still has to write a significant amount of boilerplate code in order to integrate the generated serialization focused code to be used in binary communication protocol handling.

Required Features

As an embedded C++ developer, I require the following features to be available to me, at least to some extent.

Polymorphic Interfaces Configuration

Usually every message definition is implemented (or code generated) as a separate class. In many cases, there is a common code that is applicable to all such message classes. The proper implementation would be to introduce polymorphic behavior with virtual functions (such as reading message data, writing it, calculating serialization length, etc.). However, creation of full interface to be polymorphic may be impractical. Every application that might use the protocol definition code is different. For example, in case many messages are uni-directional, one side (client) will require polymorphic write (but not read) for such messages, the one side (server) will require the opposite - polymorphic read (but not write). Most of the virtual functions will end up being included in the final binary / image, even if they are unused. It can result in unnecessary code bloat, which may be a problem for embedded systems (especially bare-metal ones).

There is a need for compile time, or at least code generation time configuration of the polymorphic interfaces that are going to be used. In the best case scenario, such configuration should be done per message class.

Extra Meta Information

There may be a significant amount of extra meta information that comes with the protocol definition. For example, one of the protocol fields in one of the messages needs to report a distance between two points. The protocol designer has decided to report the distance in centimeters. Such information will probably be written in plain text in protocol specification and if some third party code generator is used, then this information might end up being in the comment section describing the field or a message. However, in most cases, such information will not end up being in generated code. The developer that needs to integrate the generated code into its business logic needs to manually write some boilerplate code to do the math of conversion into the different distance units (such as meters) relevant to the application being developed.

It is also not very uncommon for the binary protocol being specified at the same time the application that uses it being developed. Imagine that some specific use case pops up during the development, where distance in centimeters doesn't provide sufficient precision. Thanks to the fact that protocol specification hasn't been finalized yet, the developer decides to change the reported distance units from centimeters to millimeters. It means that other developers that might have written their boilerplate code with assumption of distance units being centimeters must be aware of the change and not forget to modify their code.

Some protocol developers also dislike sending floating point numbers "as-is" over the communication link. In many such cases, floating point numbers are multiplied by some predefined value and sent over the I/O links as integers while the remaining fraction after the decimal point is dropped. The other side does the opposite operation on reception of the message, i.e., dividing the received integer by the same predefined value to get the floating point number. Which predefined value to use for such scaling is also meta information, which is not transferred "on the wire" and should be present in the generated or manually written code of the protocol definition.

Also in many cases, the protocol may define some values with special meaning. Let's assume there is a need to communicate a delay in seconds before some event needs to happen. There should be some special value that indicates infinite duration. Usually it is either 0 or maximum possible value of the unsigned type being used. The developer that integrates the generated code into the application needs to know what value it is. There is a need to write at least one extra piece of boilerplate code that wraps the special value and gives it a name. Wouldn't it be better if the generated code contained such helper function already?

Many protocols specify ranges of valid values and expect certain behavior on invalid values. There is an expectation that generated code would provide an ability to inquire that the received value is valid to avoid manually written boilerplate code that checks this information.

The list of such examples with meta-information that is part of the protocol definition, but not transfered as data of the messages can go on and on. There is a need for the generated code to provide the required functionality to be used by the integration developer without any concern of what to do when the meta-information is modified.

Customization of Data Types

Most of the currently available solutions for the protocol code generation use hard-coded types for some particular data structures, such as std::string for strings and/or std::vector for lists. Also in many cases, the generated functions that perform serialization / deserialization receive their input / output as std::istream or std::ostream. These data structures may be unsuitable for some applications, especially embedded (including bare-metal) ones.

There is a need to be able to substitute the default data structures with some equivalent replacements, ideally at compile time for selected fields / messages, but globally (during code generation for example) may also be an acceptable solution.

Excluding Exceptions and Dynamic Memory Allocation

Many constrained embedded environments make it difficult to use exceptions as well as dynamic memory allocation (especially bare-metal ones). There is a need for an ability to generate protocol definition code that don't use either of them.

Efficient Built-In Mapping of Message ID to Type

Usually, when new encoded message arrives over I/O link, encoded as raw data, it has some transport framing containing numeric message ID. Unfortunately, most (if not all) of the available protocol code generation solution leave the task of mapping numeric ID into the actual message type (or appropriate handling function) to be written by the developer who integrates the generated code into the business logic. Usually, such code is boilerplate, efficiency of which depends on the competence of the developer.

Third Party Protocol Support

Many of the available protocol generation solutions have their own encoding and framing without any ability to modify it. Using of such tools may be acceptable for new protocols being defined from scratch. However, there are heaps of already defined third party protocols, code for which cannot be generated with such tools.

Injecting Custom Code

This requirement complements the Third Party Protocol Support one. Even if a chosen code generation solution supports definition of the third party protocol and its grammar for schema files is very rich, there will always be a protocol containing some small nuance, which cannot be represented correctly using the available grammar. As a result, the generated code may be incorrect and/or incomplete. Proper protocol code generation solution should allow injection of snippets of custom, manually written code.

Working Solution

Unfortunately, I could not find any third party solution that implements most of the required features listed above. I had no other choice but to implement something of my own. I'd like to present to you the CommsChampion Ecosystem that implements all of the mentioned earlier features. It's been in development as my side project for the last several years. Now it has a stable API and is ready to be used by a wider public. Further below, I'll go through the list of the required features again and give examples on how my solution provides the required functionality.

At first, there was a headers-only, cross-platform, and very flexible COMMS library, which allows implementation of the protocol messages, fields and transport framing using simple declarative statements of types and classes definition. Such statements define WHAT needs to be implemented, the library internals handle the HOW part. The internals of the COMMS library is mostly template, highly compile-time configurable classes which use multiple meta-programming techniques. As a result, the C++ compiler itself becomes a code generation tool, which brings in only the functionality required by the application, providing the best code size and speed performance.

After the library, the test tools followed, which allow analysis, debugging, and visualization of the developed protocol, that was implemented using the COMMS library. The test tools are generic and provide a common testing environment for all the protocols. All the applications are plug-in based, i.e., plug-ins are used to define I/O socket, data filters, and the custom protocol itself. Such architecture allows easy assembly of various protocol communication stacks. The tools use Qt5 framework for GUI interfaces as well as loading and managing plug-ins. They are intended to be used on the development PC and are not part COMMS library, although hosted in the same repository.

With time, the COMMS library grew in features and started requiring more cognitive effort to remember things. It became easier to make mistakes and/or implement the protocol in not a very generic way. As a result, the code generator has followed. It allows definition of the protocol using easy XML based domain specific language (DSL) and generates headers-only library of protocol definition (which in turn uses the types and classes from the mentioned earlier COMMS library), as well as code required to build the protocol plugin for the test tools.

Now let's repeat the required features mentioned earlier and see the solution provided by the CommsChampion Ecosystem.

Polymorphic Interfaces Configuration

The protocol implementation library defines a common interface class for all the messages in the following way. Note: All the types and classes from the COMMS library reside in the comms namespace, while the definition of the custom protocol in the examples below will be defined in my_prot namespace, and application code that uses the protocol definition will be in global namespace.

namespace my_prot
{

// Numeric IDs of the protocol messages
enum MsgId : std::uint8_t
{
    MsgId_Message1,
    MsgId_Message2,
    MsgId_Message3,
    ...
};

// Common interface class for the protocol messages
template <typename... TOptions>
using Interface = 
    comms::Message<
        comms::option::MsgIdType<MsgId>,
        TOptions...
    >;

} // namespace my_prot

The default definition passes comms::option::MsgIdType to the comms::Message (class provided by the COMMS library). It is used to define type used for storing message ID. The code above is equivalent to having it defined like this:

namespace my_prot
{

class Interface
{
public:
    // Type used for message ID
    typedef MsgId MsgIdType;
};

} // namespace my_prot

The definition of the common interface class uses variadic template parameters, which are used to pass various default functionality extension options. The internals of the COMMS library parse the provided options and generate extra requested functionality. For example, adding polymorphic retrieval of message ID, required for the custom application, may look like this:

using MyAppInterface = 
    my_prot::Interface<
        comms::option::IdInfoInterface // Add extra polymorphic ID retrieval interface
    >;

The code above is equivalent to having the following definition:

class MyAppInterface
{
public:
    // Type used for message ID (defined by using comms::option::MsgIdType option)
    typedef MsgId MsgIdType;


    // NVI of polymorphic retrieval of ID (defined by using comms::option::IdInfoInterface option)
    MsgIdType getId() const
    {
        return getIdImpl();
    }

protected:
    // Polymorphic retrieval of ID (defined by using comms::option::IdInfoInterface option)
    virtual MsgIdType getIdImpl() const = 0; // implemented in derived class
};

Note that the COMMS library uses Non-Virtual Interface Idiom (NVI) to define polymorphic behavior. The non virtual interface wrapper function is there to check various pre- and post-conditions the virtual function might require.

The COMMS library provides multiple extension options, which allow definition of various other polymorphic functions. The (currently) full list is below:

using MyAppInterface = 
    my_prot::Interface<
        comms::option::IdInfoInterface,                  // Add polymorphic ID retrieval interface
        comms::option::ReadIterator<const std::uint8_t*>,// Add polymorphic read interface
        comms::option::WriteIterator<std::uint8_t*>,     // Add polymorphic write interface
        comms::option::LengthInfoInterface,              // Add polymorphic serialization 
                                                         // length retrieval
        comms::option::ValidCheckInterface,              // Add polymorphic contents validity check
        comms::option::Handler<MyHandler>,               // Add polymorphic dispatch to 
                                                         // handling function interface
        comms::option::NameInterface                     // Add polymorphic retrieval of message name
    >;

The detailed explanation of every listed above option is a bit beyond the scope of this article. The COMMS library has a very detailed documentation that lists and explains every one of them. The magic behind the extension options and how they end up generating extra types and functions is also outside the scope of this article. It is explained in detail in the free e-book I've written called Guide to Implementing Communication Protocols in C++.

The definition of an actual message may look like this:

namespace my_prot
{

// Definition of the fields used by Message1
struct Message1Fields
{
    // 32 bit unsigned integer, initialized to 0
    using F1 = 
        comms::field::IntValue<
            comms::Field<comms::option::BigEndian>, // use big endian serialization
            std::uint32_t
        >;

    // 16 bit signed integer, initialized to 10
    using F2 = 
        comms::field::IntValue<
            comms::Field<comms::option::BigEndian>, // use big endian serialization
            std::int16_t,
            comms::option::DefaultNumValue<10>
        >;       

    // All the message fields bundled in std::tuple.
    using All = std::tuple<F1, F2>;
};

// Definition of the actual Message1.
// Template parameter TMsgBase is common interface class required by the application
template <typename TMsgBase>
class Message1 : public
    comms::MessageBase<
        TMsgBase,
        comms::option::StaticNumIdImpl<MsgId_Message1>, // Set message ID known at compile time
        comms::option::FieldsImpl<Message1Fields::All>, // Define fields of the message
        comms::option::MsgType<SimpleInts<TMsgBase, TOpt> > // Pass the actual message type 
                                                            // being defined
    >
{

public:
    ... // some small irrelevant at this moment code here
};

} // namespace demo1

The definition of the actual custom protocol message my_prot::Message1 is completely generic. It receives the application specific interface definition class as its template parameter, and based on the required polymorphic interface the comms::MessageBase, provided by the COMMS library, does all the magic of determining the required polymorphic interface and implementing the necessary virtual functions, while also inheriting from the provided interface class. How such magic of determining the required polymorphic functionality is implemented is beyond the scope of this article and also explained in details in my free Guide to Implementing Communication Protocols in C++ e-book.

So, the usage of such class in the actual application may look like this:

// Define Message1 relevant to the application
using MyMessage1 = my_prot::Message1<MyAppInterface>;

// Definition of smart pointer holding any protocol message
using MyMessagePtr = std::unique_ptr<MyAppInterface>;

// Holding Message1 by the pointer to its interface
MyMessagePtr msg(new MyMessage1);

// Retrieving serialization length, works only if comms::option::LengthInfoInterface 
// was passed as option to interface definition, if not compilation will fail.
std::size_t msgLen = msg->length();
assert(msgLen == 6U); // 4 byte of f1 + 2 bytes of f2

The code the developer needs to write to integrate protocol definition into the business logic of the application is quite simple. All the complexity is handled by the C++ compiler behind the scenes. However, writing the generic code for the protocol definition may require a bit more of cognitive effort with a bit of knowledge about the COMMS library internals. That's the reason the separate code generator has been developed which takes schema file(s) as an input and generates proper generic protocol definition that is simple to customize and use.

The definition of the Message1 message from the example above in CommsDSL schema can look like this:

<?xml version="1.0" encoding="UTF-8"?>
<schema name="my_prot" endian="big">
    <fields>
        <enum name="MsgId" type="uint8">
            <validValue name="Message1" val="0" />
            <validValue name="Message2" val="1" />
            ...
        </enum>
    </fields>
    <message name="Message1" id="MsgId.Message1">
        <int name="f1" type="uint32" />
        <int name="f2" type="int16" defaultValue="10"/>
    </message>
    ...
</schema>

The protocol definition code generated out such schema which will look similar to the code example above.

Extra Meta Information

The COMMS library has a built-in support for various units and conversion between them. To follow the example from above with distance, such field could be defined as below:

namespace my_prot
{

using Distance = 
    comms::field::IntValue<
        comms::Field<comms::option::BigEndian>,       // Big endian serialization
        std::uint32_t,                                // 4 bytes serialization
        comms::option::UnitsCentimeters               // Contains centimeters
    >;

} // namespace my_prot

When such field is deserialized and its value in meters is needed, the units retrieval functionality, provided by the COMMS library can be used.

my_prot::Distance distanceField = ...; // Some value assigned
double distanceInMeters = comms::units::getMeters<double>(distanceField);

In case the definition of the field needs to be changed to contain millimeters instead, the client code doesn't need to be changed, just recompiled. The compiler will change the generated math operation to convert between units.

The COMMS library also contains compile-time protection against conversion between incompatible units. For example, it is not possible to retrieve seconds out of field containing millimeters, there will be static_assert failure with appropriate error message.

The language of CommsDSL schema also contains an ability to specify units, which will result in usage of appropriate option passed to the generated field definition.

<?xml version="1.0" encoding="UTF-8"?>
<schema name="my_prot" endian="big">
    <fields>
        <int name="Distance" type="uint32" units="cm" />        
    </fields>
    ...
</schema>

Both COMMS library and CommsDSL support scaling floating point values and serializing them as integers. Such fields are defined as integers with scaling ratio option.

namespace my_prot
{

using ScaledField = 
    comms::field::IntValue<
        comms::Field<comms::option::BigEndian>,       // Big endian serialization
        std::int32_t,                                 // 4 bytes serialization
        comms::option::ScalingRatio<1, 1000>          // Divide by 1000 to get floating point value
    >;

} // namespace my_prot    

Get/set of the floating point value from/to such field looks like this:

my_prot::ScaledField scaledField;
scaledField.setScaled(1.23f);
float val = scaledField.getScaled<float>();

The definition of such field in CommsDSL may look like this:

<?xml version="1.0" encoding="UTF-8"?>
<schema name="my_prot" endian="big">
    <fields>
        <int name="ScaledField" type="int32" scaling="1/1000" />        
    </fields>
    ...
</schema>

Combining units with scaling is also possible. Let's define the distance in 1/10 of millimeters.

namespace my_prot
{

using NewDistance = 
    comms::field::IntValue<
        comms::Field<comms::option::BigEndian>, 
        std::uint32_t,                          
        comms::option::UnitsMilliimeters,
        comms::option::ScalingRatio<1, 10>        
    >;

} // namespace my_prot   

The CommsDSL definition of such field looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<schema name="my_prot" endian="big">
    <fields>
        <int name="NewDistance" type="int32" scaling="1/10" units="mm" />        
    </fields>
    ...
</schema>

The retrieval of distance in meters from such field still looks the same, the compiler generates appropriate math operations taking into account both units and scaling ratio.

my_prot::NewDistance distanceField = ...; // Some value assigned
double distanceInMeters = comms::units::getMeters<double>(distanceField);

Specification of special values also supported in CommsDSL schemas. Let's define duration in seconds where value 0 means infinite.

<?xml version="1.0" encoding="UTF-8"?>
<schema name="my_prot" endian="big">
    <fields>
        <int name="Duration" type="uint32" units="sec" >        
            <special name="Infinite" val="0" />
        </int>
    </fields>
    ...
</schema>

The code generated for such field will contain appropriate member functions to help check for the special value.

namespace my_prot
{

struct Duration : public 
    comms::field::IntValue<
        comms::Field<comms::option::BigEndian>, 
        std::uint32_t,                          
        comms::option::UnitsSeconds
    >
{
    static constexpr std::uint32_t valueInfinite() { return 0U; }
    bool isInfinite() const { /* checks that the stored value is 0U */ }
    void setInfinite() { /* assigns 0 to the stored value */ }
};

} // namespace my_prot

Specifying ranges of valid values is also supported. For example:

<?xml version="1.0" encoding="UTF-8"?>
<schema name="my_prot" endian="big">
    <fields>
        <int name="SomeField" type="uint8" >        
            <validRange value="[0, 20]" />
            <validRange value="[40, 60]" />
            <validValue value="70" />                        
        </int>
    </fields>
    ...
</schema>

The example above defines field of 1 byte with unsigned value, where all the values between 0 and 20, between 40 and 60, as well as single value 70 are considered to be valid.

Every field defined by the means of the COMMS library has valid() member function, which can be used to inquire whether the field contains a valid value. The example above will result in proper functionality of the valid() member function. It will return true for any value between 0 and 20, between 40 and 60, and for value 70. All other values will result in return of false.

my_prot::SomeField field = ...; // some value assigned
if (!field.valid()) {
    ... // do something
}

Customization of Data Types

The COMMS library was designed and implemented with a built-in ability to customize default choices of potentially problematic data structures and/or default behavior. Although the default choices for storage types of strings and lists are std::string and std::vector, it is possible to substitute it with something else as part of compile-time configuration. For example, some string field may be defined like this:

namespace my_prot
{

template <typename... TOptions>
using SomeString = 
    comms::field::String<
        comms::Field<comms::option::BigEndian>,
        TOptions... // Extra configuration options
    >;

} // namespace my_prot

Such definition of string field allows passing extra options, which can be used by the application. For example, if the application is being developed for bare-metal platform, which does not allow usage of dynamic memory allocation, it is recommended to pass comms::option::FixedSizeStorage option to substitute usage of std::string by the string container implementation with fixed maximum size, provided by the COMMS library itself (called comms::util::StaticString).

using MyAppString = my_prot::SomeString<comms::option::FixedSizeStorage<32> >;

It is also possible to use any custom third party storage type, that has the same public interface as std::string.

using MyAppString = my_prot::SomeString<comms::option::CustomStorageType<boost::container::string> >;

The code generator is capable (and does it by default) of generating protocol definition code, which allows such compile time customization of various potentially problematic types.

Excluding Exceptions and Dynamic Memory Allocation

The COMMS library was designed specifically for environments with constrained resources. It does not use RTTI and/or exceptions: the violation of pre- and post-conditions are checked using assertions, while runtime errors are reported via returned status codes.

The dynamic memory allocation is used by default when new serialized message arrives via I/O link and appropriate message object is created. However, there is an available compile time configuration option that replaces dynamic memory allocation with "in-place" one (uses placement new on pre-allocated area),

Efficient Built-In Mapping of Message ID to Type

The COMMS library provides multiple built-in options and helper functions which allow efficient mapping of message ID to appropriate type, as well as dispatching the message object to its appropriate handling function. They are described and documented in detail in the available documentation.

Third Party Protocol Support

The COMMS library was designed specifically to provide building blocks and facilitate implementation of the third party protocols. Its architecture allows an easy way to complement the default behavior with protocol specific nuances.

Injecting Custom Code

The available code generator also allows injection of custom code snippets instead of or in addition to the code that might have been generated by default.

Summary

The main intended audience of CommsChampion Ecosystem is embedded C++ developers, who have to implement third party (or their own) protocols to communicate to various sensors, other devices or outside world. The non-embedded C++ developers, who don't have to worry about constrained environment and who don't require sophisticated code customization, may still find the provided solution convenient and useful, thanks to available constructs for reducing amount of integration boilerplate code.

Those developers who already have their own implementation of some protocol(s) integrated into their product, may still use the available code generator as well as test tools to easily test and debug their work.

Where to Start

If the available solution caught your interest, please start your learning process by reading Where to Start section of the CommsChampion Ecosystem page.

Licensing

All the available components of the CommsChampion Ecosystem are free and open source, each one has a separate license. Please refer to the LICENSES page for the details.