DEV Community: JFrog Community

Pick a Winning Go Module for your Go Build

JFrog Community — Mon, 22 Jun 2020 20:21:41 +0000

With a near-endless list of Go Modules, it can be overwhelming trying to decide which is best for your Go build. For new Go developers, it can be difficult to pick a winner for your specific use case.

This phenomenon is nothing new. In fact, it’s one of the reasons why open-source is such a godsend to developers. Oftentimes, when a module is published by a developer, it was likely developed to solve a specific problem that they are facing. If another developer were to use their module to solve the same problem, there may be some performance criteria needed within that module that the current version does not meet. Luckily, there are often many versions of Go Modules to choose from that can solve different use cases.

The more use cases there are for a module, the more complex it becomes when trying to find the one that is best for your specific use case. Even after you feel that you’ve found the correct module for your build, how can you be sure? Let’s discuss some of the parameters you should consider before deciding on a Go Module and show you how to increase the odds of picking the right module for you!

Choosing a Go Module: What’s Important

There are a plethora of great options available when choosing your Go Module and dwindling that list can be a simple task when considering three fundamental parameters:

Module Popularity - How widely used a module is
Module References - The attribution of the module
Module Compliance - The level of security within that module and its dependencies

Each of these factors is heavily important when choosing a Go Module, as each will have a direct effect on the success of your build. For each parameter, we’ll show an example of a module within GoCenter, a free repository for versioned Go Modules.

Module Popularity

When choosing your Go Module, it’s important that the desired module has been adopted and used by others. A module that has been used by many other developers has a higher likelihood of solving the intended issue; there is also a likelihood that developers are still using the module, ensuring that it’s consistently being updated. There are 2 effective ways to check a module's popularity.

One of the most useful ways for developers to measure the popularity of a module is by looking at the amount of Github stars that it has received. Another alternative for checking a module’s popularity is by checking the number of times that it’s been downloaded by others. These are both tested and proven methods for choosing a module for any programming language. Since Go is a relatively new programming language, the number of downloads may not always reflect the popularity of the module especially if the module is new. If this is the case, It’s recommended that you look at the number of contributors as well. A popular module with a high number of Github stars, downloads (if applicable) and contributors means that you’ll have a higher chance that the module is the right fit for your build.

Here, you see an example of a popular Go module, Aws-sdk-go. It has 5,790 stars and 452,277 downloads, meaning that thousands of developers have used this module and can verify that it’s safe, reliable and has a version that has solved their specific use case. It also has 222 contributors, assuring that many people have used and developed other versions of the module to fit their use case:

Module References

Checking a module’s references is another effective way to be sure that you’re choosing the right module for your build. It provides insight into the dependents of that module, which can be used to discover which other modules are using the module that you’re interested in consuming. By referencing the modules that use your desired module, you may notice that large-scale corporations use that module, meaning that it is more likely to be secure and usable.

For example, below we have the dependent information of the AWS module. Taking a glance, there is no negative information found on this module. The “Used By” tab shows which modules are using the module you want to consume. Below, we see that this module is being used by an enterprise, almost ensuring that the module is secure.

Module Compliance

As the consumption of publicly shared code/modules increases, developers and organizations need a way to check if the module in question is safe enough to consume. When we reference safety, it’s all about the security of the modules and the vulnerabilities that are present. The example below is of a module with two security vulnerabilities, meaning that this module can be risky to consume:

The best compliance practices include choosing modules without vulnerabilities. While this is a no-brainer, sometimes finding a module without vulnerabilities can be impossible. If you’re using a module that has a vulnerability or two, check how frequently the module is being updated. If the same security issue persists for the last 4-5 versions, the author of the module may not be making any updates or any attempts at mitigating the issue. This ultimately means that you’ll need to look for another module to use, or you’ll need to build the one that works best for you! I never recommend using a vulnerable module.

Now Take your Pick!

When choosing a Go module, it can be difficult to pick a winner. There are many parameters that ultimately affect the decision-making process and oftentimes, it can depend solely on a matter of preference. Be sure to consistently check the popularity, references and the compliance of the modules that you’re considering before consuming them.

Originally published at https://jfrog.com/blog/how-to-pick-a-winning-go-module/

Data Serialization with Protobuf

JFrog Community — Thu, 18 Jun 2020 20:18:41 +0000

You probably already had to develop a project where you needed to exchange information between processes or even across different machines with different processor architectures. One well-known technique in this scenario is serialization, which is summarized in the translation of data structures or object state into a format that can be stored and retrieved by both sides.

In this blog post, we will discuss the Protobuf (Protocol Buffers), a project that can extend more than a simple library for serialization. The entire example presented here is available on Github.

What is Protobuf?

Protocol Buffers is an open-source project under the BSD 3-Clause license, a popular one developed by Google, to provide a language-neutral, platform-neutral and extensible mechanism for serializing structured data. It supports many popular languages such as C++, C#, Dart, Go, Java and Python. Although there are still other not official add-ons, that support other languages, such as C. You can find the source code on Github, where its popularity reaches almost 32K stars!

The neutral language used by Protobuf allows you to model messages in a structured format through .proto files:

message Person {
  required string name = 1;
  required int32 age = 2;
  optional string email = 3;
}

In the example above we use a structure that represents a person’s information, where it has mandatory attributes, such as name and age, as well as having the optional email data. Mandatory fields, as the name already says, must be filled when a new message is constructed, otherwise, a runtime error will occur.

But Why not XML?

But, why another language and serialization mechanism if we can use something already available like XML? The answer is performance.

Protobuf has many advantages for serialization that go beyond the capacity of XML. It allows you to create a simpler description than using XML. Even for small messages, when requiring multiple nested messages, reading XML starts to get difficult for human eyes.

Another advantage is the size, as the Protobuf format is simplified, the files can reach 10 times smaller compared to XML. But the great benefit is its speed, which can reach 100 times faster than the standard XML serialization, all due to its optimized mechanism. In addition to size and speed, Protobuf has a compiler capable of processing a .proto file to generate multiple supported languages, unlike the traditional method where it is necessary to arrange the same structure in multiple source files.

That Sounds Good, but How do I Use it in Real Life?

So that we can illustrate the use of Protocol Buffers, we will exchange messages through different architectures and opposite languages. We will compile a code in C++ for armv7hf architecture, serialize an object to file, and retrieve through a Python script. An advantageous model for those who need to exchange messages between opposing architectures through IPC techniques, even for embedded systems.
For our example, we will use a message that has the reading of several sensors. The file sensor.proto, which will represent the message, is described below:

syntax = "proto2";
message Sensor {
  required string name = 1;
  required double temperature = 2;
  required int32 humidity = 3;

  enum SwitchLevel {
    CLOSED = 0;
    OPEN = 1;
  }
  required SwitchLevel door = 5;
}

The variable syntax refers to the version of the Protobuf used, which can be proto2 or proto3. Versions 2 and 3 have important differences, but we will only address version 2 in this post. For more information about version 3, see the official documentation. In addition to the declared attributes, and previously highlighted there is the enumerator SwitchLevel, which represents the state of a port. We could still include new messages, or even lists for multiple ports, for example. For a complete description of the syntax used in proto version 2, see the language guide.

The Protobuf serialization mechanism is given through the protoc application, this compiler will parse the .proto file and will generate as output, source files according to the configured language by its arguments, in this case, C++. You can also obtain more information about, reading the section compiler invocation.

$ protoc --cpp_out=. Sensor.proto

The protoc compiler will generate the sensor.pb.h and sensor.pb.cc files, respectively, of which have the getters and setters needed to access the attributes, as well as methods for serializing and parsing. The files work only as a stub, and it is necessary to include the headers distributed by Protobuf. Without this compiler, we would have to describe all the steps of object serialization in our code, and for any new change, it would be needed to update the C++ and Python files.
Now that we have the stubs, we can implement an example to serialize the data collected by a sensor. The file main.cpp will be described below:

#include “sensor.pb.h”

int main() {
    Sensor sensor;
    sensor.set_name("Laboratory");
    sensor.set_temperature(23.4);
    sensor.set_humidity(68);
    sensor.set_door(Sensor_SwitchLevel_OPEN);
}

The Sensor object can be serialized through methods inherited from the Message class. For example, we can serialize to a string by the SerializeAsString method.

Note that this reconstruction can be performed by other languages also supported by Protobuf, in addition to other architectures. In order for the transmission to occur through different processes, it will be necessary to use IPC techniques, for this, Google provides the gRPC project, a universalRPC framework, that supports Protobuf directly. However, our intention in this post is just to talk about Protobuf, so we will use the only text file as a means to exchange messages between processes:

#include <fstream>
#include “sensor.pb.h”

int main() {
    Sensor sensor;
    sensor.set_name("Laboratory");
    sensor.set_temperature(23.4);
    sensor.set_humidity(68);
    sensor.set_door(Sensor_SwitchLevel_OPEN);
    std::ofstream ofs("sensor.data", std::ios_base::out | std::ios_base::binary);
    sensor.SerializeToOstream(&ofs);
}

To perform serialization through a file, we use the SerializeToOstream method.

Building the project

For the next step, we will describe the actions for constructing the project by CMake:

cmake_minimum_required(VERSION 3.1.2)
project(sensor CXX)

include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup(TARGETS)

find_package(Protobuf REQUIRED)

protobuf_generate_cpp(PROTO_SRCS PROTO_HDRS sensor.proto)
add_executable(${PROJECT_NAME} main.cc ${PROTO_SRCS} ${PROTO_HDRS})
target_link_libraries(${PROJECT_NAME} PUBLIC CONAN_PKG::protobuf)
target_include_directories(${PROJECT_NAME} PRIVATE ${CMAKE_BINARY_DIR})

This recipe searches for the modules, libraries, and macros provided by the Protobuf project when calling find_package. Once found and loaded correctly, protobuf_generate macros will be available for use. The protobuf_generate_cpp function is responsible for executing the protoc and populating the PROTO_SRCS and PROTO_HDRS variables with their generated files. Without this functionality, you would need to manually add the protoc command and the required arguments. The subsequent lines follow the most usual of CMake projects. Because the generated files will be in the build directory, you need to include it by target_include_directories so that main.cc can resolve proto.pb.h.

It is also possible to observe that we are using Conan to solve Protobuf as a dependency. The conan_basic_setup function will be in charge of configuring all the necessary variables, besides generating the target CONAN_PKG::protobuf.

In addition, you must also declare the conanfile.txt file with the following dependencies:

[build_requires]
protoc_installer/3.6.1@bincrafters/stable

[requires]
protobuf/3.6.1@bincrafters/stable

[generators]
Cmake

Since Protobuf can be divided into two parts, the protoc installer, and the libraries, there are two separate packages. Thus, it will be possible to install protoc for the same host architecture, and libraries for a target architecture. As we are using CMake for this project, we need to declare the CMake generator.

Now just run the commands to build the project:

mkdir build
cd build
conan install ..
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build .
bin/sensor

So far so good, but how is it done in case of cross-compilation? In this case, it will be necessary to inform the compiler and the target platform:

conan install .. -s arch=armv7hf
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=arm-linux-gnueabihf-g++
cmake --build .

In the above commands, we have installed only the prebuilt Protobuf libraries for armv7hf. The protoc will only hold for amd64 because it ignores arch, making use of only the host architecture by arch_build in your profile. CMake needs to be informed which compiler will be used, so we define it through CMAKE_CXX_COMPILER. Once ready, we can copy our application directly to the target platform.

Parsing with Python

Now we get to the second step, read the file and retrieve the object using Python. For this, we will only update the CMake script, so that it generates the C++ files and also the python stub:

protobuf_generate_python(PROTO_PYS sensor.proto)
add_custom_target(proto_python ALL DEPENDS ${PROTO_PYS})

The protobuf_generate_python function has the same goal as protobuf_generate_cpp but will generate the file sensor_pb2.py. The proto_python virtual target was added to force CMake to call the generator for Python.

The next step is to develop the script that will read the file with the serialized data and parse it through the script generated in the previous step:

from sensor_pb2 import Sensor

if __name__ == "__main__":
    with open("sensor.data", 'rb') as file:
        content = file.read()
        print("Retrieve Sensor object from sensor.data")
        sensor = Sensor()
        sensor.ParseFromString(content)
        print(f"Sensor name: {sensor.name}")
        print(f"Sensor temperature: {sensor.temperature}")
        print(f"Sensor humidity: {sensor.humidity}")
    print("Sensor door: {}".format("Open" if sensor.temperature else "Closed"))

The script is fairly straightforward, just like the code in C++ and can be copied together with the sensor_pb2.py file directly to the target platform.

Conclusion

Transfer data between processes, serializing objects or even storing data are techniques that are widely used in all scenarios, but they require a lot of effort when implemented and are often not the goal of the project under development. Serialization techniques can be solved through several projects available, such as Protobuf, without having to delve into the low level required to process all the data.

The success in using Protobuf is not only in serializing the data, but in the mechanism as a whole, from the neutral language used, flexible and easy to understand, to the compiler with support for multiple languages, and even integration with other products, such as the gRPC, which provides direct communication between processes without much effort.

This post blog was a tutorial to demonstrate how tasks that could take up to hours to complete, with library development, can be solved in a few steps, only using what is ready and without the need to build from the sources.

To read conan.io teams additional C++ blog posts visit Conan.io blog or learn more about Conan C++ package manager on the website. You can also explore the Protobuf C++ package and other C++ packages on Conan Center.

Originally published at https://blog.conan.io/2019/03/06/Serializing-your-data-with-Protobuf.html

Learn about Deterministic Builds with C/C++

JFrog Community — Tue, 16 Jun 2020 18:34:09 +0000

What are Deterministic Builds?

A deterministic build is a process of building the same source code with the same build environment and build instructions producing the same binary in two builds, even if they are made on different machines, build directories and with different names. They are also sometimes called reproducible or hermetic builds if it is guaranteed to produce the same binaries even compiling from different folders.

Deterministic builds are not something that happens naturally. Normal projects do not produce deterministic builds and the reasons that they are not produced can be different for each operating system and compiler.

Deterministic builds should be guaranteed for a given build environment. That means that certain variables such as the operating system, build system versions and target architecture are assumed to remain the same between different builds.

There are lots of efforts coming from different organizations in the past years to achieve deterministic builds such as Chromium, Reproducible builds, or Yocto.

The Importance of Deterministic Builds

There are two main reasons why deterministic builds are important:

Security. Modifying binaries instead of the upstream source code can make the changes invisible for the original authors. This can be fatal in safety-critical environments such as medical, aerospace and automotive. Promising identical results for given inputs allows third parties to come to a consensus on a correct result.
Traceability and binary management. If you want to have a repository to store your binaries you do not want to generate binaries with random checksums from sources at the same revision. That could lead the repository system to store different binaries as different versions when they should be the same. For example, if you are working on Windows or MacOs the most simple library will lead binaries with different checksums because of the timestamps included in the library formats for these Operating Systems.

Binaries Involved in the Building Process in C/C++

There are different types of binaries that are created during the building process in C/C++ depending on the operating system.

Microsoft Windows. The most important files are the ones with .obj, .lib,.dll and .exe extensions. All of them follow the specification of the Portable Executable format (PE). These files can be analyzed with tools such as dumpbin.
Linux. Files with .o, .a,.so and none (for executable binaries) extensions follow the Executable and Linkable Format (ELF). The contents of ELF files can be analyzed by readelf.
Mac OS. Files with .o, .a,.dylib and none (for executable binaries) extensions follow the Mach-O format specification. These files can be inspected with the otool application that is part of the XCode toolchain in MacOs.

Sources of Variation

Many different factors can make your builds non-deterministic. Factors will vary between different operating systems and compilers. Each compiler has specific options to fix the sources of indeterminism. To date, gcc and clang are the ones that incorporate more options to fix the sources of variation. For msvc there are some undocumented options that you can try but in the end, you will probably need to patch the binaries to get deterministic builds.

Timestamps Introduced by the Compiler/Linker

There are two main reasons for that our binaries could end up containing time information that will make them not reproducible:

The use of __DATE__ or __TIME__ macros in the sources.
When the definition of the file format forces to store time information in the object files. This is the case of Portable Executable format in Windows and Mach-O in MacOs. In Linux ELF files do not encode any kind of timestamp.

Let’s put an example of where this information ends with a basic hello world project linking a static library in MacOs.

.
├── CMakeLists.txt
├── hello_world.cpp
├── hello_world.hpp
├── main.cpp
└── run_build.sh`

The library prints a message in the terminal:

include "hello_world.hpp"
include <iostream>
void HelloWorld::PrintMessage(const std::string & message)
{
    std::cout << message << std::endl;
}

And the application will use it to print a “Hello World!” message:

#include <iostream>
#include "hello_world.hpp"
int main(int argc, char** argv)
{
    HelloWorld hello;
    hello.PrintMessage("Hello World!");
    return 0;
}

We will use CMake to build the project:

cmake_minimum_required(VERSION 3.0)
project(HelloWorld)
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
add_library(HelloLibA hello_world.cpp)
add_library(HelloLibB hello_world.cpp)
add_executable(helloA main.cpp)
add_executable(helloB main.cpp)
target_link_libraries(helloA HelloLibA)
target_link_libraries(helloB HelloLibB)`

We build two different libraries with the exact same sources and two binaries with the same sources as well. If we build the project and execute md5sum to show the checksums of all the binaries:

mkdir build && cd build
cmake ..
make
md5sum helloA
md5sum helloB
md5sum CMakeFiles/HelloLibA.dir/hello_world.cpp.o
md5sum CMakeFiles/HelloLibB.dir/hello_world.cpp.o
md5sum libHelloLibA.a
md5sum libHelloLibB.a

We get an output like this:

b5dce09c593658ee348fd0f7fae22c94  helloA
b5dce09c593658ee348fd0f7fae22c94  helloB
0a4a0de3df8cc7f053f2fcb6d8b75e6d  CMakeFiles/HelloLibA.dir/hello_world.cpp.o
0a4a0de3df8cc7f053f2fcb6d8b75e6d  CMakeFiles/HelloLibB.dir/hello_world.cpp.o
adb80234a61bb66bdc5a3b4b7191eac7  libHelloLibA.a
5ac3c70d28d9fdd9c6571e077131545e  libHelloLibB.a

This is interesting because the executables files helloA and helloB have the same checksums as well as the intermediate Mach-O object files hello_world.cpp.o but that is not the case of the .a files. That is because they store the information of the intermediate object files in archive format. The definition of the header of this format includes a field named st_time set by a stat system call. If we inspect the libHelloLibA.a and libHelloLibB.a using otool to show the headers:

> otool -a libHelloLibA.a   
Archive : libHelloLibA.a
0100644 503/20    612 1566927276 #1/20
0100644 503/20  13036 1566927271 #1/28
> otool -a libHelloLibB.a   
Archive : libHelloLibB.a
0100644 503/20    612 1566927277 #1/20
0100644 503/20  13036 1566927272 #1/28

We can see that the file includes several time fields that will make our build non-deterministic. Let’s note that those fields are not propagated to the final executable because they have the same checksum. This problem would also happen if building in Windows with Visual Studio but with the Portable Executable instead of Mach-O.

At this point, we could try to make things even worse and force our binaries to be non-deterministic as well. If we change main.cpp file to include the __TIME__ macro:

#include <iostream>
#include "hello_world.hpp"
int main(int argc, char** argv)
{
    HelloWorld hello;
    hello.PrintMessage("Hello World!");
    std::cout << "At time: " << __TIME__ << std::endl;
    return 0;
}

Getting the checksums of the files again:

625ecc7296e15d41e292f67b57b04f15  helloA
20f92d2771a7d2f9866c002de918c4da  helloB
0a4a0de3df8cc7f053f2fcb6d8b75e6d  CMakeFiles/HelloLibA.dir/hello_world.cpp.o
0a4a0de3df8cc7f053f2fcb6d8b75e6d  CMakeFiles/HelloLibB.dir/hello_world.cpp.o
b7801c60d3bc4f83640cadc1183f43b3  libHelloLibA.a
4ef6cae3657f2a13ed77830953b0aee8  libHelloLibB.a

We see that now we have different binaries as well. We could analyze the executable file with a tool such as diffoscope that shows us the difference between the two binaries:

> diffoscope helloA helloB
--- helloA
+++ helloB
├── otool -arch x86_64 -tdvV {}
│┄ Code for architecture x86_64
│ @@ -16,15 +16,15 @@
│  00000001000018da   jmp 0x1000018df
│  00000001000018df   leaq    -0x30(%rbp), %rdi
│  00000001000018e3   callq   0x100002d54 ## symbol stub for: __ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEED1Ev
│  00000001000018e8   movq    0x1721(%rip), %rdi ## literal pool symbol address: __ZNSt3__14coutE
│  00000001000018ef   leaq    0x162f(%rip), %rsi ## literal pool for: "At time: "
│  00000001000018f6   callq   0x100002d8a ## symbol stub for: __ZNSt3__1lsINS_11char_traitsIcEEEERNS_13basic_ostreamIcT_EES6_PKc
│  00000001000018fb   movq    %rax, %rdi
│ -00000001000018fe   leaq    0x162a(%rip), %rsi ## literal pool for: "19:40:47"
│ +00000001000018fe   leaq    0x162a(%rip), %rsi ## literal pool for: "19:40:48"
│  0000000100001905   callq   0x100002d8a ## symbol stub for: __ZNSt3__1lsINS_11char_traitsIcEEEERNS_13basic_ostreamIcT_EES6_PKc
│  000000010000190a   movq    %rax, %rdi
│  000000010000190d   leaq    __ZNSt3__1L4endlIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_(%rip), %rsi #

That shows that the __TIME__ information was inserted in the binary making it non-deterministic. Let’s see what we could do to avoid this.

Possible Solutions for Microsoft Visual Studio

Microsoft Visual Studio has a linker flag /Brepro that is undocumented by Microsoft. That flag sets the timestamps from the Portable Executable format to a -1 value as can be seen in the image below.

To activate that flag with CMake we will have to add this lines if creating a .exe:

add_link_options("/Brepro")

or this for .lib

set_target_properties(
    TARGET
    PROPERTIES STATIC_LIBRARY_OPTIONS "/Brepro"
)

The problem is that this flag makes the binaries reproducible (regarding timestamps in the file format) in our final binary is a .exe but will not remove all timestamps from the .lib (the same problem that we talked about with the Mach-O object files above). The TimeDateStamp field from the COFF File Header for the .lib files will stay. The only way to remove this information from the .lib binary is by patching the .lib substituting the bytes corresponding to the TimeDateStamp field with any known value.

Possible Solutions for GCC and CLANG

gcc detects the existence of the SOURCE_DATE_EPOCH environment variable. If this variable is set, its value specifies a UNIX timestamp to be used in replacement of the current date and time in the __DATE__ and __TIME__ macros so that the embedded timestamps become reproducible. The value can be set to a known timestamp such as the last modification time of the source or package.
clang makes use of ZERO_AR_DATE that if set, resets the timestamp that is introduced in the archive files setting it to epoch 0. Take into account that this will not fix the __DATE__ or __TIME__ macros. If we want to fix the effect of these macros we should either patch the binaries or fake the system time.

Let’s continue with our example project for MacOs and see what the results are when setting the ZERO_AR_DATE environment variable.

export ZERO_AR_DATE=1

Now, if we build our executable and libraries (omitting the __DATE__ macro in the sources), we get:

b5dce09c593658ee348fd0f7fae22c94  helloA
b5dce09c593658ee348fd0f7fae22c94  helloB
0a4a0de3df8cc7f053f2fcb6d8b75e6d  CMakeFiles/HelloLibA.dir/hello_world.cpp.o
0a4a0de3df8cc7f053f2fcb6d8b75e6d  CMakeFiles/HelloLibB.dir/hello_world.cpp.o
9f9a9af4bb3e220e7a22fb58d708e1e5  libHelloLibA.a
9f9a9af4bb3e220e7a22fb58d708e1e5  libHelloLibB.a

All the checksums are now the same. And analyzing the .a files headers:

> otool -a libHelloLibA.a
Archive : libHelloLibA.a
0100644 503/20    612 0 #1/20
0100644 503/20  13036 0 #1/28
> otool -a libHelloLibB.a
Archive : libHelloLibB.a
0100644 503/20    612 0 #1/20
0100644 503/20  13036 0 #1/28

We can see that the timestamp field of the library header has been set to zero value.

Build Folder Information Propagated to Binaries

If the same sources are compiled in different folders sometimes folder information is propagated to the binaries. This can happen mainly for two reasons:

Use of macros that contain current file information like __FILE__ macro.
Creating debug binaries that store information of where the sources are.

Continuing with our hello world MacOs example let’s separate the sources so we can show the effect over the final binaries. The project structure will be like the one below.

.
├── run_build.sh
├── srcA
│   ├── CMakeLists.txt
│   ├── hello_world.cpp
│   ├── hello_world.hpp
│   └── main.cpp
└── srcB
    ├── CMakeLists.txt
    ├── hello_world.cpp
    ├── hello_world.hpp
    └── main.cpp

If we build our binaries in Debug mode.

cd srcA/build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make
cd .. && cd ..
cd srcB/build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make
cd .. && cd ..
md5sum srcA/build/hello
md5sum srcB/build/hello
md5sum srcA/build/CMakeFiles/HelloLib.dir/hello_world.cpp.o
md5sum srcB/build/CMakeFiles/HelloLib.dir/hello_world.cpp.o
md5sum srcA/build/libHelloLib.a
md5sum srcB/build/libHelloLib.a
We get the following checksums:
3572a95a8699f71803f3e967f92a5040  srcA/build/hello
7ca693295e62de03a1bba14853efa28c  srcB/build/hello
76e0ae7c4ef79ec3be821ccf5752730f  srcA/build/CMakeFiles/HelloLib.dir/hello_world.cpp.o
5ef044e6dcb73359f46d48f29f566ae5  srcB/build/CMakeFiles/HelloLib.dir/hello_world.cpp.o
dc941156608b578c91e38f8ecebfef6d  srcA/build/libHelloLib.a
1f9697ef23bf70b41b39ef3469845f76  srcB/build/libHelloLib.a

The folder information is propagated from the object files to the final executables making our builds non-reproducible. We could show the differences between binaries using diffoscope to see where the folder information is embedded.

> diffoscope helloA helloB
--- srcA/build/hello
+++ srcB/build/hello
@@ -1282,20 +1282,20 @@
...
 00005070: 5f77 6f72 6c64 5f64 6562 7567 2f73 7263  _world_debug/src
-00005080: 412f 006d 6169 6e2e 6370 7000 2f55 7365  A/.main.cpp./Use
+00005080: 422f 006d 6169 6e2e 6370 7000 2f55 7365  B/.main.cpp./Use
 00005090: 7273 2f63 6172 6c6f 732f 446f 6375 6d65  rs/carlos/Docume
 000050a0: 6e74 732f 6465 7665 6c6f 7065 722f 7265  nts/developer/re
 000050b0: 7072 6f64 7563 6962 6c65 2d62 7569 6c64  producible-build
 000050c0: 732f 7361 6e64 626f 782f 6865 6c6c 6f5f  s/sandbox/hello_
-000050d0: 776f 726c 645f 6465 6275 672f 7372 6341  world_debug/srcA
+000050d0: 776f 726c 645f 6465 6275 672f 7372 6342  world_debug/srcB
 000050e0: 2f62 7569 6c64 2f43 4d61 6b65 4669 6c65  /build/CMakeFile
 000050f0: 732f 6865 6c6c 6f2e 6469 722f 6d61 696e  s/hello.dir/main
 00005100: 2e63 7070 2e6f 005f 6d61 696e 005f 5f5a  .cpp.o._main.__Z
...
@@ -1336,15 +1336,15 @@
...
 000053c0: 6962 6c65 2d62 7569 6c64 732f 7361 6e64  ible-builds/sand
 000053d0: 626f 782f 6865 6c6c 6f5f 776f 726c 645f  box/hello_world_
-000053e0: 6465 6275 672f 7372 6341 2f62 7569 6c64  debug/srcA/build
+000053e0: 6465 6275 672f 7372 6342 2f62 7569 6c64  debug/srcB/build
 000053f0: 2f6c 6962 4865 6c6c 6f4c 6962 2e61 2868  /libHelloLib.a(h
 00005400: 656c 6c6f 5f77 6f72 6c64 2e63 7070 2e6f  ello_world.cpp.o
 00005410: 2900 5f5f 5a4e 3130 4865 6c6c 6f57 6f72  ).__ZN10HelloWor
...

Possible Solutions

Again the solutions will depend on the compiler used:

msvc can’t set options to avoid the propagation of this information to the binary files. The only way to get reproducible binaries is again using a patching tool to strip this information in the build step. Note that as we are patching the binaries to achieve reproducible binaries the folders used for different builds should have the same length in characters.
gcc has three compiler flags to work around the issue:
- -fdebug-prefix-map=OLD=NEW can strip directory prefixes from debug info.
- -fmacro-prefix-map=OLD=NEW is available since gcc 8 and addresses irreproducibility due to the use of __FILE__ macro.
- -ffile-prefix-map=OLD=NEW is available sice gcc 8 and is the union of -fdebug-prefix-map and -fmacro-prefix-map *clang supports -fdebug-prefix-map=OLD=NEW from version 3.8 and is working on supporting the other two flags for future versions.

The best way to solve this is by adding the flags to compiler options. If we are using CMake:

target_compile_options(target PUBLIC "-ffile-prefix-map=${CMAKE_SOURCE_DIR}=.")

File Order Feeding to the Build System

File ordering can be a problem if directories are read to list their files. For example, Unix does not have a deterministic order in which readdir() and listdir() should return the contents of a directory, so trusting in these functions to feed the build system could produce non-deterministic builds.

The same problem arises for example if your build system stores the files for the linker in a container (like a regular python dictionary) that can return the elements in a non-deterministic order. This would make that each time files were linked in a different order and produce different binaries.

We can simulate this problem by changing the order of files in CMake. If we modify the previous example to have more than just one source file for the library:

.
├── CMakeLists.txt
├── CMakeListsA.txt
├── CMakeListsB.txt
├── hello_world.cpp
├── hello_world.hpp
├── main.cpp
├── sources0.cpp
├── sources0.hpp
├── sources1.cpp
├── sources1.hpp
├── sources2.cpp
└── sources2.hpp

We can see that the results of the compilation are different if we change the order of files in the CMakeLists.txt:

cmake_minimum_required(VERSION 3.0)
project(HelloWorld)
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
add_library(HelloLib hello_world.cpp 
                     sources0.cpp 
                     sources1.cpp 
                     sources2.cpp)
add_executable(hello main.cpp)
target_link_libraries(hello HelloLib)

If we make two consecutive builds named A and B swapping sources0.cpp and sources1.cpp in the files list the resulting checksums will be:

30ab264d6f8e1784282cd1a415c067f2  helloA
cdf3c9dd968f7363dc9e8b40918d83af  helloB
707c71bc2a8def6885b96fb67b84d79c  hello_worldA.cpp.o
707c71bc2a8def6885b96fb67b84d79c  hello_worldB.cpp.o
694ff3765b688e6faeebf283052629a3  sources0A.cpp.o
694ff3765b688e6faeebf283052629a3  sources0B.cpp.o
0db24dc6a94da1d167c68b96ff319e56  sources1A.cpp.o
0db24dc6a94da1d167c68b96ff319e56  sources1B.cpp.o
fd0754d9a4a44b0fcc4e4f3c66ad187c  sources2A.cpp.o
fd0754d9a4a44b0fcc4e4f3c66ad187c  sources2B.cpp.o
baba9709d69c9e5fd51ad985ee328172  libHelloLibA.a
72641dc6fc4f4db04166255f62803353  libHelloLibB.a

Object files .o are identical but .a libraries and executables are not. That is because the insertion order in the libraries depends on the order the files were listed.

Randomness Created by the Xompiler

This problem arises for example in gcc when Link-Time Optimizations are activated (with the -flto flag). This option introduces randomly generated names in the binary files. The only way to avoid this problem is to use -frandom-seed flag. This option provides a seed that gcc uses when it would otherwise use random numbers. It is used to generate certain symbol names that have to be different in every compiled file. It is also used to place unique stamps in coverage data files and the object files that produce them. This setting has to be different for each source file. One option would be to set it to the checksum of the file so the probability of collision is very low. For example in CMake it could be made with a function like this:

set(LIB_SOURCES
    ./src/source1.cpp
    ./src/source2.cpp
    ./src/source3.cpp)

foreach(_file ${LIB_SOURCES})
    file(SHA1 ${_file} checksum)
    string(SUBSTRING ${checksum} 0 8 checksum)
    set_property(SOURCE ${_file} APPEND_STRING PROPERTY COMPILE_FLAGS "-frandom-seed=0x${checksum}")
endforeach()

Some Tips Using Conan

Conan hooks can help us in the process of making our builds reproducible. This feature makes it possible to customize the client behavior at determined points.

One use of hooks could be setting environment variables in the pre_build step. The example below is calling a function set_environment and then restoring the environment in the post_build step with reset_environment.

def set_environment(self):
    if self._os == "Linux":
        self._old_source_date_epoch = os.environ.get("SOURCE_DATE_EPOCH")
        timestamp = "1564483496"
        os.environ["SOURCE_DATE_EPOCH"] = timestamp
        self._output.info(
            "set SOURCE_DATE_EPOCH: {}".format(timestamp))
    elif self._os == "Macos":
        os.environ["ZERO_AR_DATE"] = "1"
        self._output.info(
            "set ZERO_AR_DATE: {}".format(timestamp))

def reset_environment(self):
    if self._os == "Linux":
        if self._old_source_date_epoch is None:
            del os.environ["SOURCE_DATE_EPOCH"]
        else:
            os.environ["SOURCE_DATE_EPOCH"] = self._old_source_date_epoch
    elif self._os == "Macos":
        del os.environ["ZERO_AR_DATE"]

Hooks can also be useful to patch binaries in the post_build step. There are different binary files analysis and patching tools like ducible, pefile, pe-parse or strip-nondeterminism. An example of a hook for patching a PE binary using ducible could be like this one:

class Patcher(object):
...
    def patch(self):
        if self._os == "Windows" and self._compiler == "Visual Studio":
            for root, _, filenames in os.walk(self._conanfile.build_folder):
                for filename in filenames:
                    filename = os.path.join(root, filename)
                    if ".exe" in filename or ".dll" in filename:
                        self._patch_pe(filename)

    def _patch_pe(self, filename):
        patch_tool_location = "C:/ducible/ducible.exe"
        if os.path.isfile(patch_tool_location):
            self._output.info("Patching {} with md5sum: {}".format(filename,md5sum(filename)))
            self._conanfile.run("{} {}".format(patch_tool_location, filename))
            self._output.info("Patched file: {} with md5sum: {}".format(filename,md5sum(filename)))
...

def pre_build(output, conanfile, **kwargs):
    lib_patcher.init(output, conanfile)
    lib_patcher.set_environment()

def post_build(output, conanfile, **kwargs):
    lib_patcher.patch()
    lib_patcher.reset_environment()

Conclusions

Deterministic builds are a complex problem highly coupled with the operating system and toolchain used. This introduction should have served to understand the most common causes of indeterminism and how to avoid them. For additional C++ blog posts visit Conan.io blog or learn more about Conan package manager for C++.

References

General Info

Tools

Tools for Comparing Binaries

*https://diffoscope.org/
*https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/fc

Tools for Patching Files

Tools for Analyzing Files

Originally published at https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html