DEV Community: Maxim Radugin

Monitor Progress of Data Transfer in Console

Maxim Radugin — Thu, 09 Oct 2025 20:15:06 +0000

Linux and UNIX-based OS'es provide at least two handy command-line tools to monitor data transfer progress: bar and pv.

Both display a one-line progress bar with speed, elapsed or estimated time, and provide customizable formatting.

Examples with bar:

If data size is unknown:

dd if=/dev/zero | bar | dd of=/dev/null
   2.3GB at  581.8MB/s  elapsed:   0:00:04

If data size is known:

dd if=/dev/zero iflag=count_bytes count=10G | bar --size 10G | dd of=/dev/null
   1.2GB at  592.8MB/s  eta:   0:00:15   11% [=====                               ]

Examples with pv:

dd if=/dev/zero iflag=count_bytes count=10G | pv | dd of=/dev/null
2.59GiB 0:00:06 [ 397MiB/s] [      <=>                                            ]

dd if=/dev/zero iflag=count_bytes count=10G | pv --size=10G | dd of=/dev/null
2.59GiB 0:00:06 [ 467MiB/s] [========>                            ] 25% ETA 0:00:17

pv can also count lines instead of bytes, useful for monitoring log or message rates:

sudo dmesg -w | pv --line --rate > /dev/null
[ 496k/s]

On a side note, dd can also report progress at runtime, when status=progress is specified.

dd if=/dev/zero iflag=count_bytes count=10G of=/dev/null status=progress
2791971840 bytes (2.8 GB, 2.6 GiB) copied, 3 s, 931 MB/s

Run Applications in Rosetta on macOS

Maxim Radugin — Thu, 09 Oct 2025 20:05:24 +0000

I needed to run Terminal in x86_64 mode (Rosetta) in parallel with Terminal in arm64 (Apple Silicon native mode), but in the latest versions of macOS, the old way of creating a copy of the Terminal application and selecting Open using Rosetta in Get Info is not available, since Terminal can't be copied anymore due to security restrictions.

To overcome this, you can use the following command to open any application in Rosetta mode from either Terminal or Automator:

open -a Terminal --new --arch x86_64

This will open a new instance of the Terminal application. To verify Intel emulation mode, running the arch command should output i386 as opposed to arm64 when running natively.

To switch to x86_64 in an already running Terminal session, execute the following command. Note that ~/.zprofile and other shell setup scripts will not be sourced, and the new shell running under x86_64 will inherit the initial shell environment.

arch -arch x86_64 $SHELL

Creating and Restoring Disk Images on macOS: A Guide to Using dd and diskutil

Maxim Radugin — Thu, 09 Oct 2025 19:49:05 +0000

🚀 Introduction

Many IoT and embedded devices, including single-board computers like Raspberry Pi, Radxa, and Orange Pi, use removable SD, CFast cards, or similar media for storing firmware, OS, and user data.
It is desirable to make copies of these media devices before installing updates for backup purposes, or when a new copy of media is needed to be installed in another device.

This article will guide you through the process of working directly with physical disks and raw disk images on macOS from the console:

🔍 Listing physical disks
⚙️ Preparing for backup or restore operations
💾 Saving and restoring disk images
✅ Verifying disk images are properly saved or restored
📦 Optimizing space used by disk images

On macOS, diskutil and dd command-line utilities will be used to accomplish the above tasks.

diskutil is macOS's built-in utility to manage local disks and volumes, while dd is a UNIX utility for working with files, including special device files that represent physical disks in the system.

⚠️ Special care should be taken when working with these utilities, as in most cases operations are irreversible. If used improperly, they can result in data damage or loss.

⚡ TL;DR

If you understand the risks and know what you are doing, you can use commands from this section to save and restore disk images.

If you are not sure or want to understand the details, proceed with the sections below.

💾 Save Disk to an Image File

Identify disk:

  diskutil list

Replace <X> in disk<X> and /dev/rdisk<X> below with the numeric identifier of the desired disk, and replace <disk-image-name> with the desired name of the disk image file.

Unmount volumes:

  diskutil unmountDisk disk<X>

Save disk to a compressed image file:

  sudo dd if=/dev/rdisk<X> bs=1M status=progress | xz --compress -2 --stdout --threads=0 > <disk-image-name>.img.xz

Prepare for device removal:

  diskutil eject disk<X>

🔄 Restore Disk from an Image File

Identify disk:

  diskutil list

Replace <X> in disk<X> and /dev/rdisk<X> below with the numeric identifier of the desired disk, and replace <disk-image-name> with the name of the disk image file.

Unmount volumes:

  diskutil unmountDisk disk<X>

Restore image from a compressed image file:

  xz --decompress --stdout <disk-image-name>.img.xz | sudo dd of=/dev/rdisk<X> bs=1M status=progress

Prepare for device removal:

  diskutil eject disk<X>

⚠️ macOS Unreadable Disk Warning Dialog

When attaching disks or inserting media that contain only partition types that are not supported by macOS, it will issue the warning dialog below.
It is safe to press the Ignore button and continue, or after writing the image to the disk, you can press Eject and skip ejecting
the disk from the console with the diskutil eject command.

🔍 Identify Physical Disk

The first and most important step is to list and identify the disk of interest using diskutil.

Note: Perform the steps below every time you turn on, restart, or wake up your computer, plug/unplug external devices, or insert/remove media, as disk identifiers and device nodes are volatile and may change between these events! Do not hard-code device identifiers in scripts!

diskutil list

This will list all the disks and their partitions that are currently connected to the computer. Stripped output:

/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk0
   1:             Apple_APFS_ISC Container disk1         524.3 MB   disk0s1
   2:                 Apple_APFS Container disk3         994.7 GB   disk0s2
   3:        Apple_APFS_Recovery Container disk2         5.4 GB     disk0s3

/dev/disk3 (synthesized):
   ...

/dev/disk4 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *8.0 GB     disk4
   1:                      Linux                         8.0 GB     disk4s1

/dev/disk6 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *16.0 GB    disk6
   1:                      Linux                         134.2 MB   disk6s1
   2:                      Linux                         536.9 MB   disk6s2
                    (free space)                         15.3 GB    -

Now, identify the disk of interest. Always skip disks that contain Apple partitions or are of synthesized type. In the example above, I have two external disks connected - an SD card inserted into the MacBook's built-in SD card reader, and another - an external USB CFast reader. In the above output, they are listed as disk4 and disk6.

If unsure which one is which, use disk utility to get more information about the disks:

diskutil info disk4

Stripped output:

   Device Identifier:         disk4
   ...
   Device / Media Name:       Built In SDXC Reader
   ...
   Protocol:                  Secure Digital
   ...
   Disk Size:                 8.0 GB (7956594688 Bytes) (exactly 15540224 512-Byte-Units)
   ...
   Device Location:           Internal
   ...

diskutil info disk6

Stripped output:

   Device Identifier:         disk6
   ...
   Device / Media Name:       Transcend
   ...
   Protocol:                  USB
   ...
   Disk Size:                 16.0 GB (16013942784 Bytes) (exactly 31277232 512-Byte-Units)
   ...
   Device Location:           External
   ...

In the above listings, look for Device Identifier, Protocol, Disk Size, Device / Media Name, and Device Location fields to identify the device of interest.

In my case, disk4 is the built-in SD card reader with an 8.0 GB card, and disk6 is the USB CFast reader with a 16.0 GB card.
If unsure which disk corresponds to which physical device, try unplugging and removing unrelated devices from the system and performing the above steps again. If still unsure, stop here to avoid data loss.
In further steps, I will be performing operations on disk6.

⚙️ Prepare Disk for Direct IO Operations

Before performing any direct IO operations on the disk, all partitions and volumes should be unmounted! Unmounting will ensure that any pending write operations on the volume level are finished and the filesystem is in a consistent state.

diskutil unmountDisk disk6

Look for the output below. If you get an error, close user applications that might still be using volumes of the selected disk and try unmounting again. If no success - stop here to avoid data loss.

Unmount of all volumes on disk6 was successful

💾 Backup Disk to an Image File

Now it is time to use the dd utility.

The command below will copy the whole disk6 content (device file /dev/rdisk6) into cfast-disk.img file in the current directory, while reading and writing data in 1 MB chunks, and displaying progress information.

Please note that the r in /dev/rdisk6 is not a typo - this device file corresponds to the raw disk, and operations on this device are much faster than on /dev/disk6.
In most cases, the device block size is 512 bytes, but it is advised to perform read and write operations in larger chunks to improve I/O performance.
Don't worry if the disk size is not a multiple of the selected buffer size - in our case, 1 MB, as dd will handle remainders properly.

Accessing the raw disk device file requires administrator privileges, hence sudo is used when invoking dd, which will ask you to grant permissions.

sudo dd if=/dev/rdisk6 of=cfast-disk.img bs=1M status=progress

Output:

  15972958208 bytes (16 GB, 15 GiB) transferred 72.004s, 222 MB/s   
15272+1 records in
15272+1 records out
16013942784 bytes transferred in 74.675073 secs (214448304 bytes/sec)

🔄 Restore Disk from an Image File

To restore the image, just swap if (input file) and of (output file):

sudo dd if=cfast-disk.img of=/dev/rdisk6 bs=1M status=progress

Output:

  16002318336 bytes (16 GB, 15 GiB) transferred 308.004s, 52 MB/s   
15272+1 records in
15272+1 records out
16013942784 bytes transferred in 308.249739 secs (51951197 bytes/sec)

Run eject disk to make sure all write operations are complete before unplugging the device or removing the media:

diskutil eject disk6

Wait for the output below:

Disk disk6 ejected

Now the device can be safely unplugged or the media removed from the reader.

📦 Optimize Space Used by Disk Images

When the disk image is saved as a regular file, it will take the same amount of space as the original disk size.
In many cases, disks are not fully packed with useful data and may contain unallocated or empty spaces.
Below are two methods to reduce the space taken by disk images.

🕳️ Use of Sparse Files

Sparse files are special files that hold only non-zero data and metadata about empty areas (holes) in the file.
This makes them very efficient for disk images that have large amounts of empty space.

The command below will create a sparse disk image file. It will read data in 1 MB chunks and write in 512 bytes chunks, marking any 512 bytes chunks filled with zeroes as being empty, i.e., "holes".

sudo dd if=/dev/rdisk6 of=cfast-disk-sparse.img ibs=1M obs=512 conv=sparse status=progress

Output:

  15890120704 bytes (16 GB, 15 GiB) transferred 71.002s, 224 MB/s   
15272+1 records in
31277232+0 records out
16013942784 bytes transferred in 71.548794 secs (223818487 bytes/sec)

Check the resulting file sizes using ls:

ls -al cfast-disk.img cfast-disk-sparse.img

ls shows file sizes taking into account sparse file "holes", hence both files appear as having the same size:

-rw-r--r--  1 root  staff  16013942784 May  4 13:56 cfast-disk-sparse.img
-rw-r--r--  1 root  staff  16013942784 May  4 12:34 cfast-disk.img

Now compare the actual space used by sparse and regular disk images using the du utility:

du -hs cfast-disk.img cfast-disk-sparse.img

In my case, the sparse disk image uses less than 2% of the total disk size! And that is expected - if you look at the output of diskutil list above,
disk6 contains two partitions of 134.2 MB and 536.9 MB, which, I also know, are not fully taken.

 15G    cfast-disk.img
282M    cfast-disk-sparse.img

To verify that the content of files is equal, SHA1 hash calculation can be used:

sha1sum cfast-disk.img cfast-disk-sparse.img

7a7d26c342dc44d9e814de00d1de8a6d2b26e17e  cfast-disk.img
7a7d26c342dc44d9e814de00d1de8a6d2b26e17e  cfast-disk-sparse.img

Note on copying sparse files: not all filesystems and file copying utilities preserve sparse files and might replace "holes" with zeroes.

Generally, the second method of using compressed disk images is more preferred over this one, as it produces even smaller disk image files
with negligible performance drop when saving and restoring images. And these files can be safely copied across different systems without worrying about sparse file support.

🗜️ Compress Disk Images

I prefer xz-compressed images since they provide great performance in terms of compression ratio and decompression speed.

sudo dd if=/dev/rdisk6 bs=1M status=progress | xz --compress -2 --stdout --threads=0 > cfast-disk.img.xz

  15922626560 bytes (16 GB, 15 GiB) transferred 73.004s, 218 MB/s   
15272+1 records in
15268+9 records out
16013942784 bytes transferred in 73.422991 secs (218105291 bytes/sec)

In this particular case, the output image size was only 80 MB. And the read + compression speed reached 218 MB/s, which is comparable to 224 MB/s when no compression was used.

To uncompress:

xz --decompress --stdout cfast-disk.img.xz | sudo dd of=/dev/rdisk6 bs=1M status=progress

On a side note - by default, xz produces sparse uncompressed files. Moreover, xz-uncompressed files take even less space compared to ones
produced by dd, since dd operates on fixed block boundaries, while xz looks for continuous empty regions in uncompressed data.

This was verified with:

xz --decompress cfast-disk.img.xz --stdout > cfast-disk-unxz.img
du -sh cfast-disk-sparse.img cfast-disk-unxz.img

282M    cfast-disk-sparse.img
272M    cfast-disk-unxz.img

✅ Verify Disk Image

To ensure the integrity of your disk images, you can calculate and verify SHA1 hashes at different stages of the process.

Creating and Verifying Disk Image

When creating a disk image, you can calculate its SHA1 hash while writing it to a file using process substitution:

sudo dd if=/dev/rdisk6 bs=1M status=progress | tee >(sha1sum) | xz --compress -2 --stdout --threads=0 > cfast-disk.img.xz

This command will:

Read from the disk using dd
Split the output using tee:
- One stream goes to sha1sum for hash calculation
- Another stream goes to xz for compression
Save the compressed output to cfast-disk.img.xz

The output will look like:

  16002318336 bytes (16 GB, 15 GiB) transferred 79.003s, 203 MB/s   
15272+1 records in
15263+19 records out
16013942784 bytes transferred in 79.060099 secs (202554044 bytes/sec)
7a7d26c342dc44d9e814de00d1de8a6d2b26e17e  -

Note: The difference in record counts (15272+1 records in vs 15263+19 records out) is normal and occurs because:

dd reads data in 1MB blocks (as specified by bs=1M)
When the data is piped through multiple commands (tee, xz), the blocks might be split or combined differently
The total number of bytes transferred remains the same (16013942784 bytes)
The +1 and +19 indicate partial blocks at the end of the transfer

This difference doesn't affect data integrity, as the SHA1 hash verifies the actual content.

Save the hash value (7a7d26c342dc44d9e814de00d1de8a6d2b26e17e) for later verification.

Verifying Restored Disk Image

To verify that the disk image was correctly written to a disk, you can calculate the hash of the target disk and compare it with the original:

sudo dd if=/dev/rdisk6 bs=1M status=progress | sha1sum

If the hash matches the one you saved earlier, the disk image was written correctly.

⚠️ This is only true when the image file size exactly matches the target disk size. If the physical disk is larger, the hash calculation will include data from the area after the disk image, resulting in a mismatched hash.

To overcome this limitation, we need to specify the exact size of the image when calculating the hash. However, the macOS version of dd only allows limiting data to read by an integer number of blocks.

My disk image size is not evenly divisible by 1 MB, also using a block size of 512 bytes is impractical as it would result in very slow disk read operations.
Instead, still very impractical, I calculated that my image consists of 20686 blocks of 774144 bytes each,
and with below command I have verified that image was successfully written on the disk of a larger size.

sudo dd if=/dev/rdisk6 bs=774144 count=20686 status=progress | sha1sum

Verifying Compressed Disk Image

To verify the integrity of a compressed disk image without decompressing it to a file:

xz --decompress --stdout cfast-disk.img.xz | sha1sum

This will decompress the image in memory and calculate its hash, which should match the original hash.

🎯 Conclusion

macOS provides powerful command-line utilities for working with disk images, making it possible to create reliable backups of removable media and restore them when needed. The combination of diskutil and dd offers a robust solution for disk imaging tasks, while tools like xz compression and sparse files help optimize storage space.

Key takeaways from this guide:

🔍 Always identify the correct disk using diskutil list and diskutil info before performing any operations
⚙️ Unmount all volumes before working with raw disk devices
💾 Use /dev/rdiskX for faster raw disk access
📦 Consider using compression to save storage space
✅ Always verify disk images using checksums
⏏️ Eject disks properly after operations are complete

Remember that working with raw disk devices requires administrator privileges and carries risks of data loss if not done carefully. Always double-check disk identifiers and ensure you're working with the correct device before proceeding with any operations.

With these tools and precautions in mind, you can confidently create and restore disk images for your IoT devices, embedded systems, and other removable media.

Enter ROS2 Development Container

Maxim Radugin — Thu, 17 Apr 2025 18:32:42 +0000

Introduction

The robotics world has witnessed a significant shift in recent years with the growing adoption of ROS2 (Robot Operating System 2). As the successor to the original ROS, this framework brings improved performance, better security, and enhanced support for real-time systems. However, despite its increasing popularity, ROS2 development has faced a persistent challenge: the lack of an official, convenient development environment where both C++ and Python nodes can be seamlessly developed and debugged. These challenges can significantly slow down development cycles and create barriers for newcomers to the ecosystem.

The Devcontainer Advantage

Meanwhile, development containers (devcontainers) and their support in VS Code have been around for several years now (since 2019), which makes them an ideal environment for ROS2 project development. The technology has matured significantly, offering robust features for containerized development workflows that perfectly complement ROS2's architectural needs.

The ROS2 Growing Pains

Surprisingly, the official ROS2 documentation only briefly touches on the topic of using devcontainers and VS Code for ROS2 development. It provides only a minimalistic guide without an actual starting point workspace, leaving developers to piece together their own solutions. The process is time-consuming and requires numerous trial and error cycles, including:

Managing complex dependencies across different operating systems
Configuring build systems that work efficiently with both C++ and Python
Setting up debugging tools that function properly across languages
Creating clean project structure

Enter ros2-devcontainer

The ros2-devcontainer project aims to address these pain points head-on and leverages Visual Studio Code's development container capabilities to provide a clean, consistent, yet powerful environment for ROS2 development. It also integrates essential VS Code extensions for C++, Python, and ROS2, creating a comprehensive development experience with syntax highlighting, code completion, and debugging for ROS2 applications.

Key Features

The ros2-devcontainer project offers several advantages that streamline the ROS2 development workflow:

1. Containerized Development Environment
By utilizing VS Code's devcontainer support, the project ensures that every developer works with identical dependencies and configurations, eliminating the classic "it works on my machine" problem.

2. Language-Agnostic Development
The environment seamlessly supports both C++ and Python development, allowing developers to choose the right language for each component without sacrificing development experience.

3. Optimized Build System
The project includes a carefully configured build system that maximizes compilation speed while maintaining proper dependency handling between packages.

4. Integrated Debugging Experience
Perhaps most importantly, ros2-devcontainer provides a unified debugging experience across languages. Developers can set breakpoints, inspect variables, and step through code regardless of whether they're working with C++ or Python nodes.

5. Clean Project Structure
The repository offers a logical, well-organized project structure that makes it easy to add new packages and components while maintaining code clarity.

Real-World Benefits

For teams working on complex robotics applications, the benefits of adopting ros2-devcontainer can be substantial:

Reduced onboarding time for new team members who can start contributing quickly without spending days configuring their development environment
Improved collaboration through consistent tooling and environments across the team
Faster debugging cycles with integrated tools that work across languages

Getting Started

Getting started with ros2-devcontainer is straightforward. With VS Code and Docker installed, developers can clone the repository and have a fully functional development environment running within minutes. The project's documentation provides clear instructions for both new and experienced ROS2 developers.

Further Improvements

While the ros2-devcontainer project provides a solid foundation for starting ROS2 development, it currently maintains a focused scope. The implementation is intentionally streamlined and ready for expansion with additional components that would enhance its functionality. Currently, visualization tools like RViz2 or Gazebo aren't included, likewise examples for unit testing ROS2 nodes are yet to be added.

If you find the project useful and want to help expand its capabilities, consider forking the repository and contributing these missing pieces. The project welcomes contributions that could help make ROS2 development more accessible to everyone.

Conclusion

As ROS2 continues to mature and gain adoption, it is important to have a workspace, like ros2-devcontainer provides, that can jump-start development of a new project, enabling developers to focus on what matters most: building innovative robotics applications.

While community-driven solutions like ros2-devcontainer provide excellent alternatives, it would be beneficial to see similar environment readily available as part of the official ROS2 project and tutorials. Having standardized development environments officially supported by the ROS2 team would:

Provide newcomers with a consistent starting point
Ensure compatibility with each ROS2 release
Reduce fragmentation in development approaches
Establish best practices for containerized ROS2 development

Official support for devcontainer configurations would significantly lower the barrier to entry for robotics developers and help consolidate the ecosystem around proven development patterns. This would be particularly valuable for educational institutions and companies introducing new developers to ROS2.

GitLab CI: Tips and Tricks

Maxim Radugin — Tue, 22 Oct 2024 07:53:32 +0000

Introduction

GitLab is a second most-popular DevOps platform that provides a complete solution for source version control, project management, issue tracking, and powerful CI/CD system.

GitLab is constantly working to extend and improve CI/CD functionality, and provides comprehensive documentation for all the new features.

I've been using GitLab for over a decade now for professional and personal projects. In this article I'd like to share some of my findings about useful features and workarounds for some limitations I know of.

Features

Referencing Scripts from Other Jobs

When writing CI/CD pipelines sometimes it is handy to reuse commands between different jobs. GitLab features the !reference custom YAML tag to accomplish this. Below commands from script of .configure-and-build are reused in build-project and build-sub-project jobs.

.configure-and-build:
    script:
        - cmake -G Xcode -B build
        - cmake --build build

build-project:
    script:
        - !reference [.configure-and-build, script]

build-sub-project:
    script:
        - cd sub-project
        - !reference [.configure-and-build, script]

Using Job Scripts as Functions

The above approach can be extended to use variables as "arguments" to referenced script commands. In below example GENERATOR and BUILD_DIR variables are used to customize behavior of .configure-and-build script. In build-project-macos job .configure-and-build script is invoked multiple times with different GENERATOR and BUILD_DIR values.


.configure-and-build:
    script:
        - cmake -G ${GENERATOR} -B build
        - cmake --build ${BUILD_DIR}

build-project-windows:
    ...
    variables:
        GENERATOR: "Visual Studio 16 2019"
        BUILD_DIR: "build"
    script:
        - !reference [.configure-and-build, script]

build-project-macos:
    ...
    script:
        - export GENERATOR="Xcode"
        - export BUILD_DIR="build_xcode"
        - !reference [.configure-and-build, script]
        - export GENERATOR="Ninja"
        - export BUILD_DIR="build_ninja"
        - !reference [.configure-and-build, script]
        - export GENERATOR="Unix Makefiles"
        - export BUILD_DIR="build_make"
        - !reference [.configure-and-build, script]

Multi-line Command Invocation

This is not strictly GitLab CI/CD feature, but rather feature of yaml language which is used to describe CI/CD pipelines in GitLab.

Sometimes it is desired to split long command into multiple lines for readability, but unfortunately bash and powershell use different characters to split multiline command - \ and ` respectively. Therefore, it is not possible to re-use commands for both platforms.
Consider following example, where both windows and macos build jobs have to be duplicated:


build-macos:
    ...
    script: |
        cmake -G Xcode \
            -B build \
            -DOPTION1=YES \
            -DANOTHER_OPTION=Test \
            -DYET_ANOTHER_OPTION=1 \
            -DAND_ANOTHER_OPTION=ON 

build-windows:
    ...
    script: |
        cmake -G "Visual Studio 16 2019" `
            -B build `
            -DOPTION1=YES `
            -DANOTHER_OPTION=Test `
            -DYET_ANOTHER_OPTION=1 `
            -DAND_ANOTHER_OPTION=ON

To reduce duplication a "Folded Block Scalar" > can be used to folds newlines to spaces, which unlocks the ability to write multi-line commands that are compatible with both bash and powershell.

.build:
    ...
    script: 
        - > 
            cmake -G "${GENERATOR}"
            -B build
            -DOPTION1=YES
            -DANOTHER_OPTION=Test
            -DYET_ANOTHER_OPTION=1
            -DAND_ANOTHER_OPTION=ON

build-macos:
    ...
    extends:
        - .build 
    variables:
        GENERATOR: "Xcode"

build-windows:
    ...
    extends:
        - .build 
    variables:
        GENERATOR: "Visual Studio 16 2019"

Exclusive Use of Runners

Sometimes it is desirable to restrict concurrent use of specific runner for a specific job. For example, you might have a runner that can execute unit-tests concurrently, but due to limitations of the underlying framework can only run one instance of end-to-end test application.

Limiting the number of concurrent jobs for the whole runner is suboptimal, since unit-tests can safely be executed in parallel. To resolve this GitLab resource_group keyword can be used to disable execution of a specific job from different pipelines concurrently.

Depending on runner concurrency configuration, multiple unit-test instances can be executed in parallel, but only one instance of test-e2e will be executed concurrently due to presence of resource_group key.

unit-test:
    ...
    tags:
        - test-machine-tag
    script:
        - echo "running unit-tests non-exclusively"
        - sleep 60

test-e2e:
    ...
    tags:
        - test-machine-tag
    resource_group: "test-e2e"
    script:
        - echo "running e2e tests exclusively"
        - sleep 60

Customizing Job Execution Using Rules

GitLab provides powerful rules keyword that is used to customize "when" job is run. Additionally rules supports specifying value for variables on specific conditions that can help to customize "how" job is being executed.

Consider the example below where variable SIGN will be set to YES only when build job is executed for tagged commit, in other cases it will be NO.

build-macos:
    variables:
        SIGN: "NO"
    rules:
        - if: "$CI_COMMIT_TAG"
          variables:
            SIGN: "YES"
        - when: always
    script: |
        echo "Executables will be signed: ${SIGN}"

Clearing Working Directory for Specific Job

Recently GitLab added option to clean build directory before downloading artifacts and executing job scripts. To achieve this, GIT_STRATEGY variable should be set to empty. Per GitLab documentation:

Use the empty Git strategy when:

You do not need the repository data to be present.

You want a clean, controlled, or customized starting state every time a job runs.

Example:

job:
    ...
    variables:
        GIT_STRATEGY: empty
    script:
        - ls -al

Workarounds

Spaces in Variables

Consider a scenario where you want to have a variable that has list of file paths to be passed to some script and file paths may contain spaces. Example below contains list of files to be passed to signing script:

job:
    variables:
        FILES_FOR_SIGNING: "'Test App 1/Test App 1' 'Test App 2/Test App 2'"
    script:
        - echo $FILES_FOR_SIGNING
        - ./sign.sh $FILES_FOR_SIGNING

sign.sh script:

#!/bin/sh

if [ $# -eq 0 ]; then
    echo "No arguments provided"
    exit 1
fi

i=1
for arg in "$@"; do
    echo "$i: $arg"
    i=$((i + 1))
done

Output of pipeline execution will look like below - with arguments split by spaces, without quotes being taken into account.

$ echo $FILES_FOR_SIGNING
'Test App 1/Test App 1' 'Test App 2/Test App 2'
$ ./sign.sh $FILES_FOR_SIGNING
1: 'Test
2: App
3: 1/Test
4: App
5: 1'
6: 'Test
7: App
8: 2/Test
9: App
10: 2'

I have tried different combinations of quotes and escaped quotes, but the result was the same all the time. But, if you have noticed, echo $FILES_FOR_SIGNING produces string with single quotes preserved. Combining it with eval solves the problem!

Replacing ./sign.sh $FILES_FOR_SIGNING with eval $(echo ./sign.sh $FILES_FOR_SIGNING) solved the issue and the output was:

$ eval $(echo ./sign.sh $FILES_FOR_SIGNING)
1: Test App 1/Test App 1
2: Test App 2/Test App 2

A note on security: using eval in certain conditions is considered unsafe, even with current example FILES_FOR_SIGNING can be easily constructed so that it can execute arbitrary commands, use with caution!

Colon+Space in Values

I have stumbled upon this issue several times and each time it took me a while to figure out what is wrong. In short - if you want to use colon and space : in the value, whole value string must be quoted.

Obviously, this will work fine:

job:
    variables:
        ANIMALS: "Animals: dog, cat, mouse"
    script:
        - echo $ANIMALS

But this will not, as it is not compliant with above rule:

job:
    script:
        - echo "Animals: dog, cat, mouse"

To make it compliant, whole line should be quoted, no matter it is dictionary or a list value, like this:

job:
    script:
        - 'echo "Animals: dog, cat, mouse"'

Conclusion

GitLab is a powerful and extensive tool that offers a wide range of features for creating advanced CI/CD pipelines. Throughout this article, we've explored several techniques and workarounds that can help you create more efficient and flexible pipelines, streamline your development process and accelerate your software delivery.
Remember that GitLab is constantly evolving, so it's worth staying up-to-date with the latest features and best practices.

Related Materials

Cleaning up Obsolete Cloudflare Page Deployments

Maxim Radugin — Sat, 31 Aug 2024 18:21:15 +0000

Introduction

In the previous post I have created a Hugo blog which is being deployed to Cloudflare pages. After spending some time playing around and tuning it, I have generated quite a few deployments to Cloudflare pages. Now all of them are visible in Cloudflare dashboard under Workers & Pages section for each site. Why would I need all of them and how to clean all that mess?

Value of Deployment History

For production deployments - useful if latest deployments have problems, then it is easy and straightforward to rollback to the previous version. This can be done from Cloudflare dashboard by finding deployment to rollback to and selecting Rollback to this deployment in the "..." menu.

For preview deployments - while a new feature or content is being created it is good to test how a site performs in different environments after specific changes, track overall feature development progress, and share with other people for review. But after active development is finished, content ready, and everything is deployed in production and tested, I see no much value in keeping deployment history of work-in-progress changes. Compared to git, regular practice is to squash commits and delete source branches when merging features or bugfixes to the default branch.

Cleaning up Deployment History

As stated above, I see value in:

keeping couple of latest production deployments, just in case
keeping preview deployments while actively working on them

Let's now see how this can be achieved.

Cleaning from Cloudflare

Deployments can be deleted manually from the Cloudflare dashboard under Workers & Pages section for each site. I've tried this approach, and if only few deployments are to be deleted then it is fine and failsafe, but deleting many deployments - very tedious and annoying process, because:

deployments can only be deleted one by one, no bulk delete
each delete operation has to be confirmed
if deployment has an alias, alias name should be typed-in into edit field to confirm deletion

Automating Cleanup

I was hoping that the wrangler tool would have something to manage existing deployments, but not... it can only list deployments in human-readable format, but not any standard machine-friendly format.

Fortunately, Cloudflare has Pages APIs that provide full control over deployments, including listing and deleting. Official example are here.
I will use these APIs to create a script to do obsolete deployment cleanup that are older than certain number of days, while always keeping specific number of latest deployments in case rollback would be needed.

With ChatGPT doing most of typing, I've created following python script:

usage: cleanup-deployments.py [-h] --environment {production,preview} --count COUNT --days DAYS [--dry-run]

Fetch and delete obsolete page deployments

options:
  -h, --help            show this help message and exit
  --environment {production,preview}
                        deployment environment
  --count COUNT         number of deployments to keep
  --days DAYS           number of days to keep
  --dry-run             perform a dry run

Apart from command line arguments it also expects certain environment variables to be defined, for authentication - CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID are used, to select pages project - CLOUDFLARE_PROJECT_NAME. Names were chosen to match variables that are already available in my CI/CD environment in GitLab and used by deployment jobs and the wrangler tool.

Integrating into CI/CD

I took .gitlab-ci.yml from previous article and made following modifications to:

define deployment cleanup jobs for production and preview environment with their specific options
make cleanup jobs runnable on schedule
exclude build and deployment jobs from scheduled pipelines

stages:
  - build
  - deploy
  - cleanup

workflow:
  rules:
    ...
      # Always run on schedule
    - if: $CI_PIPELINE_SOURCE == "schedule"

.build:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
    ... 

deploy:preview:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
    ...

deploy:production:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
    ...

.cleanup:
  image: python:3.11-alpine
  stage: cleanup
  variables:
    CLEANUP_KEEP_DAYS: 7
  before_script:
    - pip install requests python-dateutil
  script:
    - python scripts/cleanup-deployments.py --environment ${CLEANUP_ENVIRONMENT} --days ${CLEANUP_KEEP_DAYS} --count ${CLEANUP_KEEP_COUNT}
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

cleanup:preview:
  extends: [.cleanup]
  variables:
    CLEANUP_ENVIRONMENT: "preview"
    CLEANUP_KEEP_COUNT: 0

cleanup:production:
  extends: [.cleanup]
  variables:
    CLEANUP_ENVIRONMENT: "production"
    CLEANUP_KEEP_COUNT: 2

Complete .gitlab-ci.yml and cleanup-deployments.py script source code are available on GitHub.

Next, I have configured scheduled pipeline in my GitLab project under Build -> Pipeline schedules menu:

Now, it is all set and you have to wait for the next run to happen or press the "Play" button to create the scheduled pipeline manually.
This is how successful scheduled pipeline should look like:

And runner log:

...
Installing collected packages: urllib3, six, idna, charset-normalizer, certifi, requests, python-dateutil
Successfully installed certifi-2024.2.2 charset-normalizer-3.3.2 idna-3.7 python-dateutil-2.9.0.post0 requests-2.31.0 six-1.16.0 urllib3-2.2.1
$ python scripts/cleanup-deployments.py --environment ${CLEANUP_ENVIRONMENT} --days ${CLEANUP_KEEP_DAYS} --count ${CLEANUP_KEEP_COUNT}
Fetching all production page deployments...
Found 2 production page deployments.
Deleting obsolete production page deployments older than 7 days, while keeping 2 latest...
0 obsolete production page deployments have been deleted.
Cleaning up project directory and file based variables
00:01
Job succeeded

Conclusion

Cloudflare is a great service for hosting static content, it provides complete history of page deployments
Rolling back production deployment can be done from Cloudflare dashboard
Cleaning deployments from Cloudflare dashboard is tedious task
Cloudflare provides flexible API which allow listing and cleaning up deployments
Scripting skills are required to make use of Cloudflare API
Cleanup job can be added to GitLab CI/CD and configured to run on schedule

Deploying Hugo from Self-Hosted GitLab to Cloudflare Pages

Maxim Radugin — Sat, 31 Aug 2024 18:15:08 +0000

Introduction

As you may have noticed my main blog site is built using Hugo and Papermod theme.
After playing around with Hugo and getting comfortable with it, I wanted to push the initial version of my blog site to the git repository and set up an automatic deployment pipeline to be able to easily publish updates to the web.

Since I am the lucky owner of a self-hosted GitLab instance which I've been using for more than a decade for my projects, it was an obvious decision to push my blog site there too. Same story with Cloudflare - I use their DNS services, and since they also provide static content hosting services - Pages, decided to give it a try.

Fast googling revealed that Cloudflare supports automatic deployments of Hugo-based sites from GitHub and GitLab..., but only for cloud-based SaaS version.

Luckily Cloudflare has Wrangler - command-line interface to manage Worker projects. You may ask what workers have to do with static pages? Well, they are managed by the same tool, see pages command documentation of Wrangler tool.

Below I will explain how to tie Wrangler together with GitLab CI/CD to perform deployment to Cloudflare Pages. Further steps assume you have knowledge and experience of using Cloudflare services, git, Hugo and GitLab CI/CD. I will not go into details for some of the steps.

Setting and Testing Locally

Below are the steps to prepare project, repository and test everything locally:

Create Hugo project
Change into project directory
Login with Wrangler CLI into your Cloudflare account: npx wrangler login, follow instructions
Create new Pages project, I will call mine pages-for-article, make sure your project git default branch matches with what you pass in --production-branch argument. Cloudflare will use it later to determine if you are publishing to production or preview environments: npx wrangler pages project create pages-for-article --production-branch main
Initialize git repository, do this before deploying to Pages, as Wrangler uses git metadata for deployments
Build Hugo project: hugo
Now, run deployment: npx wrangler pages deploy public --project-name pages-for-article
This is what you should see from Cloudflare dashboard Workers & Pages section, notice main branch and Production environment in the screenshot:
Add public/ to .gitignore
Commit and push your project to git

Now we are all set to move to the GitLab CI/CD configuration.

Setting GitLab CI/CD

Requirements

Before going into configuration details, I like to define my requirements for CI/CD pipeline:

I want to have two environments - production and preview (staging)
I want to deploy changes from default branch, tags or merge requests only
I want to be able to see draft content in the preview environment
I want to be able to deploy any changes except tags to preview environment
I want to be able to deploy to production environment changes only from default branch or tags

GitLab Project Configuration

Wrangler requires certain authentication environment variables to be present to operate from CI/CD. See Run Wrangler in CI/CD guide. Add these variables in the GitLab project under Settings -> CI/CD -> Variables. Make variables masked, unprotected and non-expandable.

Make sure that the project has at least one runner with a docker execution environment under Settings -> CI/CD -> Runners.

Pipeline Definition

GitLab uses .gitlab-ci.yml to describe jobs that make up the CI/CD pipeline. I will not go into detail of every aspect of that file, but rather highlight parts that are made up to fulfill above requirements. Complete .gitlab-ci.yml file is available at GitHub.

workflow:rules describes the requirement to create and run a pipeline only for the default branch, merge requests commits and tags, omitting commits to branches without merge requests.

Because of the requirement to have draft content available in the preview environment, two distinct build jobs are needed - build:staging and build:production, first one will build a site with drafts, second - without, see script section. Both jobs inherit from common .build template, which describes common build configuration - using alpine image for runtime environment and installing hugo using apk package manager, collecting build artifacts from public folder.

The most interesting part of .gitlab-ci.yml is the .deploy template. What it does - it runs Wrangler deploy command with additional arguments - project name, branch, commit hash, and commit message. Values for the latter three arguments are taken from corresponding variables defined in the same template, which in turn reference pre-defined GitLab CI/CD variables by default.

variables:
  CLOUDFLARE_BRANCH: "$CI_COMMIT_BRANCH"
  CLOUDFLARE_COMMIT_HASH: "$CI_COMMIT_SHA"
  CLOUDFLARE_COMMIT_MESSAGE: "$CI_COMMIT_MESSAGE"

You may ask - why is it so complex? As mentioned earlier, Wrangler can extract this information from git itself. Yes, indeed, it can, but Wrangler also makes a decision on where to deploy - production or preview environment based on the branch name. Default branch always goes to production, period, you can't affect it, this basically breaks requirements 4, 5.

Luckily, Wrangler has these arguments and you can pass whatever you want making it possible to workaround behavior of always publishing default branch changes to production. And this is what is being done in deploy:staging and deploy:production which inherit from the .deploy template.

Here is a closer look at deploy:staging variables and rules section:

  variables:
    CLOUDFLARE_BRANCH: "$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME"
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' 
      when: manual
      variables:
        CLOUDFLARE_BRANCH: "$CI_COMMIT_BRANCH-preview"
    - if: '$CI_COMMIT_TAG' 
      when: never
    - when: manual

First rule makes it possible to deploy default branch changes to preview environment by adding -preview postfix to the default branch name, this way Wrangler will treat it as a non-default branch.
Second rule disallows deployment of tags to preview, since I will use tags as release milestones for this project.
For all other cases, i.e. merge request commits - merge request source branch name will be used, as defined in variables section.

Please note that preview deployments are manual actions and require launching these jobs from pipeline view in the GitLab by hand. Also for pipelines not to appear as "blocked" in GitLab, deploy:staging has allow_failure: true set.

Cool thing about Cloudflare Pages is that preview deployments can be accessed via <branch-name>.pages-for-article.pages.dev address. For default branch preview for this project it is https://main-preview.pages-for-article.pages.dev.

These are deploy:production rules:

  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: manual
    - if: '$CI_COMMIT_TAG' 
      when: always
      variables:
        CLOUDFLARE_BRANCH: "$CI_DEFAULT_BRANCH"
        CLOUDFLARE_COMMIT_MESSAGE: "$CI_COMMIT_TAG - $CI_COMMIT_TAG_MESSAGE"
    - when: never

First rule allows manually deploying anything from the default branch to the production environment; this normally is not used, but sometimes may come in handy.
Second rule - automatically deploys to production environment when tag is created. Since Wrangler has no support for tags, passing actual tag name into branch name would result in tag being deployed into preview environment instead of production, to workaround this - default branch name is fed into branch name, and actual tag name along with tag message is set in commit message.

That's it, these rules satisfy my requirements set above, and I can use GitLab to track and publish my blog changes to Cloudflare pages.

Further Improvements

I am using bare alpine and node docker images which download hugo and wrangler for each job invocation. This negatively affects pipeline speed and generates excess traffic, a better approach would be to use pre-built images containing these tools. On the other hand, impact is negligible, as whole pipeline takes around 30 seconds to build and deploy.
Add a separate deployment job to the preview environment without draft content to make it identical to what is going to be deployed to the production environment.

Conclusion

GitLab and Cloudflare are great services with lots of useful features and customization options. With some exploration and trial and error approach I was able to create CI/CD pipeline configuration that meets my requirements for publishing Hugo blog to Cloudflare Pages from a self-hosted GitLab instance.

Complete .gitlab-ci.yml file is available on GitHub.

Bridging Jira and GitLab: Automating CI/CD Pipelines for Releases

Maxim Radugin — Sun, 25 Aug 2024 10:32:34 +0000

Introduction

Jira Cloud and GitLab are popular productivity and DevOps platforms widely used by many organizations and software development teams. While the integration between these platforms has significantly improved in recent years, there are still areas in common software development tasks that require manual effort or custom integration solutions.

One such area is software releases. In this post, I will demonstrate how to streamline and automate this process, it will be useful for teams who:

Plan and manage tasks in Jira;
Use Jira versions for planning product milestones and releases;
Keep their source code in GitLab;
Use GitLab's CI/CD pipelines for deploying and releasing products.

User Story

As a Release Manager, I want GitLab to automatically trigger a release CI/CD pipeline when I change a Jira version's status to 'Released' and its name matches a specific format (e.g., 'v1.2.3'), so that release information is immediately synchronized between Jira and GitLab, reducing manual work and potential errors.

Implementation

To make this happen, we need to:

Set up Jira to trigger an action when a version is released.
Use the GitLab API to create a matching tag in git repository.
Configure GitLab CI/CD to run a release or deploy job when this tag is created.

We'll go through these steps in detail for both Jira and GitLab. This guide uses GitLab's cloud version, but the process is similar for self-hosted GitLab instances.

GitLab Configuration

Create Personal Access Token

You can skip this step if you already have a Personal Access Token (PAT) with the api scope that you can reuse. If not, follow these steps:

In GitLab, navigate to your preferences by clicking on your avatar icon, then Preferences -> Access tokens
Click Add new token, enter a token name (e.g., Jira Release Automation), extend the expiration date, and select the api scope
Press Create personal access token and note down the token value

Modify CI/CD Configuration

In your GitLab project, add the following rule to the release job in your .gitlab-ci.yml file. This ensures the job is triggered only for git tags with a vX.Y.Z format, where X, Y, Z are non-negative integers:

rules:
  - if: '$CI_COMMIT_TAG =~ /^v([0-9]+)\.([0-9]+)\.([0-9]+)$/'

You can find a full example of the .gitlab-ci.yml file here.

Jira Project Configuration

We'll use Jira's built-in Automation to trigger the release pipeline by creating "release" tag in the desired GitLab project:

From the project sidebar, select Automation -> Create rule.
In the rule editor, add a trigger: Version released. Expand More options and set Version name filter to ^v([0-9]+)\.([0-9]+)\.([0-9]+)$.
Press Next, then on the workflow press Add component. Select THEN: Add an action, and add the Send web request component.
Configure the Send web request component:
- In Web request URL, enter https://gitlab.com/api/v4/projects/<group>%2F<project name>/repository/tags, replacing <group> with your GitLab group and <project name> with your actual project name. If you wonder what %2F is - it is url-encoded value of / character. In my case final URL is https://gitlab.com/api/v4/projects/radugin.com%2Fjira-gitlab-release-automation/repository/tags;
- Leave HTTP method as POST;
- Set Web request body to Custom data;
- Set Custom data to:
```
{
    "tag_name": {{version.name.asJsonString}},
    "ref": "main",
    "message": {{version.description.asJsonString}}
}
```
  This request will create a new tag in specified GitLab repository, Jira version name will be used as a tag name, tag will be created from branch specified in ref (usually main or master), tag message will be populated from Jira version description field.
- In Headers, add a hidden PRIVATE-TOKEN with your GitLab PAT value.
Press Next, then Turn on rule. Enter a Rule name (e.g., GitLab Release Automation), and click Turn on rule again.

Verifying Automation

From Jira

Create a version, for example, v1.0.0, add version description.
Perform a version release.
Navigate to Automation -> Audit log and verify that the rule execution was successful.

From GitLab

Navigate to the Tags section and observe the newly created tag.
Click on the ✅ next to the tag to open the pipeline view, and verify that the release job (in current example - deploy job) was created.

Conclusion

This integration represents a step towards automating and streamlining the release management process for teams using GitLab and Jira, as it bridges an important gap in the DevOps workflow.
The benefits of this automation include:

Reduced manual effort in managing releases;
Improved synchronization of release information between Jira and GitLab;
Decreased likelihood of errors in the release process;
Enhanced traceability of releases from project management to code deployment.

While the setup requires some initial configuration, the long-term benefits in efficiency and reliability make it a valuable addition to any team's automation toolset.

Related Materials

Improve Productivity with CMake and Compiler Cache Integration

Maxim Radugin — Sun, 14 Jul 2024 22:18:22 +0000

Introduction

Build times can significantly impact developers' productivity. While what constitutes a slow or fast build is subjective, it is clear that slow builds introduce longer feedback loops and lead to more context switches, resulting in decreased productivity.

C++ compilers are slow, and the use of the standard C++ library and other popular libraries, especially header-only ones, exacerbates this. Here, you can explore the cost in terms of build time for including certain headers.

A study shows that even modest improvements in build time can positively affect developers' productivity.

Proven Way of Optimizing Build Times

One proven way to optimize build times is by using compiler cache programs such as ccache or sccache. These programs cache compilation results and reuse them, omitting subsequent compiler calls if the inputs have not changed.

According to performance measurements conducted by the ccache team, build time improvements on subsequent compilations can range from 5x to 145x.

From my personal experience, realistic improvements can range from 1.5x to 10x, depending on multiple factors such as the build machine hardware, OS, compiler, and project source code.

So, What's the Deal?

If compiler cache programs can bring such improvements, why aren't they used everywhere for every C++ project? One possible answer is the difficulty of integration. There are multiple build systems and compiler types, each with its own peculiarities when integrating compiler cache programs. The cross-platform nature of modern software adds even more complexity to the integration process.

Nowadays, for many cross-platform C/C++ software projects, CMake is the go-to build system. According to the Stack Overflow Developer Survey 2023, 14.34% of developers use CMake, with only Make being more widely used as a cross-platform C++ build system. However, if you look at the sccache README, there's a lengthy section on how to start using a compiler cache and make it somewhat usable with CMake. This section does not cover all the possible combinations of OSes, compilers, and generators supported by CMake.

Solution for CMake

To solve this problem, the cmake-findccache module was created. As the name suggests, it finds a suitable compiler cache program and configures the CMake project to enable the use of the compiler cache.

It supports Xcode, Ninja, Unix Makefiles, and Visual Studio generators, and has been tested with gcc, clang, and MSVC compilers on Linux, macOS, and Windows.

As shown in the example project, in most cases, only a couple of lines are needed to enable the use of ccache:

list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/..")

find_package(ccache)

By default, the module will only look for ccache. To try other compiler cache programs, define the CCACHE_PROGRAMS variable with the list of desired compiler cache programs in order of preference, for example:

set(CCACHE_PROGRAMS "ccache;sccache;buildcache" CACHE STRING _ FORCE)

If a ccache program is found and the generator is supported, the CMake configuration phase log should contain lines similar to the following:

-- Found ccache: /usr/bin/ccache
-- Using compiler cache: YES

After the first successful build of the project, inspect the ccache statistics by running ccache --show-stats. The result should be similar to the output below. Cacheable calls statistics should be non-zero, and after the first build, most calls will result in cache misses due to an empty cache.

Cacheable calls:      2 /   2 (100.0%)
  Hits:               0 /   2 ( 0.00%)
    Direct:           0
    Preprocessed:     0
  Misses:             2 /   2 (100.0%)
Local storage:
  Cache size (GiB): 0.0 / 5.0 ( 0.00%)
  Hits:               0 /   2 ( 0.00%)
  Misses:             2 /   2 (100.0%)

On subsequent builds, cacheable call statistics should continue to increment, along with the number of hits. For example:

Cacheable calls:      4 /   4 (100.0%)
  Hits:               2 /   4 (50.00%)
    Direct:           2 /   2 (100.0%)
    Preprocessed:     0 /   2 ( 0.00%)
  Misses:             2 /   4 (50.00%)
Local storage:
  Cache size (GiB): 0.0 / 5.0 ( 0.00%)
  Hits:               2 /   4 (50.00%)
  Misses:             2 /   4 (50.00%)

This is when compilation time should decrease, thanks to using cached results instead of invoking the compiler!

But... Still Experimental and Fragile

After examining more combinations of OSes, generators, and compilers, it turned out that in some combinations, the compiler cache does not work as expected:

On Windows with MSYS2, neither mingw64 nor clang64 works with the Ninja generator, resulting in a ccache: error: Could not find compiler "D:\a\_temp\msys64\clang64\bin\clang++.exe" in PATH error during compilation.
On Windows GitHub Runner (Windows Server 2019) with the CL compiler and Visual Studio 17 2022 generator, ccache behaves oddly. It caches compilation results from the first run of the build, but for the second run, compilations are not accounted for in the statistics at all. However, on Windows 10 Pro, a similar setup works just fine.

Further Improvements

Fix or find workarounds for the above issues.
Add support for more compilers, such as clang-cl.
Test with other cache tools, like sccache.

Feel free to clone and submit pull requests to the cmake-findccache project to improve compiler cache support in CMake!

Conclusion

Compiler cache support can bring significant build time improvements and increase productivity. However, even with the improved integration between CMake and compiler cache programs provided by cmake-findccache, it remains an experimental or unavailable feature on some platforms and configurations.

Extensible Control of Reaper via OSC and Scripts

Maxim Radugin — Sun, 07 Jul 2024 21:33:33 +0000

Introduction

REAPER is a powerful Digital Audio Workstation (DAW) with enormous customization possibilities. Its scripting support, external control capabilities, support for many DAW plugin formats, and compatibility with MacOS and Windows make it an obvious choice for building all sorts of integrations and automation. At Sonarworks, we use REAPER as a plugin host as part of our DAW plugin test automation framework.

Despite its great functionality, REAPER features minimal documentation, which makes it challenging to unleash its full potential.

In this article, I'll provide a complete guide on how to extend REAPER's control possibilities when interacting with it via the Open Sound Control (OSC) interface.

Out-of-the-Box Functionality

Out of the box, REAPER supports a limited set of functions via the OSC interface, which are defined in .ReaperOSC files located in ~/Library/Application\ Support/REAPER/OSC/ on MacOS. You can modify the address patterns of OSC messages REAPER will react to and call pre-defined functions, but you can't define your own functions directly.

Defining and Calling Custom Actions

Fortunately, REAPER supports calling custom actions via /action/_COMMAND_ID OSC messages.

Below, I'll demonstrate the setup. Please note that I'll be modifying some of REAPER's configuration files directly, as certain modifications via REAPER itself are not possible or are very unhandy. Always backup your files before making any modifications to avoid losing your work or breaking your existing setup.

Custom actions are stored in ~/Library/Application\ Support/REAPER/reaper-kb.ini on MacOS. The file format description can be found here.

For my use case, I only need my custom actions, so I started with an empty file and added the following line:

SCR 4 0 MY_COMPANY_TEST_SIMPLE_OSC "MyCompany: test simple OSC trigger" "MyCompany/test_simple_osc.lua"

This defines a custom action. For the meaning of SCR 4 0, see the reaper-kb.ini documentation. _MY_COMPANY_TEST_SIMPLE_OSC (note the underscore) is the Command ID of the test_simple_osc.lua script, which should be placed in the ~/Library/Application\ Support/REAPER/Scripts/MyCompany/ directory.

Content of test_simple_osc.lua:

reaper.ShowMessageBox("This is a simple script that can be invoked via OSC", "Info", 0)

After manually editing files, REAPER must be restarted for changes to take effect!

To verify the script and reaper-kb.ini are correct, run REAPER, and from the Actions dialog (Shift + ?), find the MyCompany: test simple OSC trigger action and run it. You should see a message box with the text "This is a simple script that can be invoked via OSC".

If not done already, from REAPER Preferences (Command + ,), navigate to the Control/OSC/web section and add a new OSC (Open Sound Control) control surface. Leave the Default pattern config, select Local port mode, and note the Local listen port. Other settings can remain at their default values for now, as shown in the screenshot.

Now you can send an OSC message to trigger the custom action. For testing purposes, you can use sendosc (which can be installed on MacOS using brew install yoggy/tap/sendosc).

From the terminal, run: sendosc localhost 8000 /action/_MY_COMPANY_TEST_SIMPLE_OSC

If everything worked correctly, you should see the same message box with the text "This is a simple script that can be invoked via OSC".

Troubleshooting OSC Communication

To ensure OSC messages are reaching REAPER, navigate to Preferences (Command + ,), then the Control/OSC/web section. Select the existing OSC control from the list and press Edit.

From the Control Surface Settings window, press Listen..., and observe incoming OSC messages. Each line represents a single received OSC message, for example: /test_message_with_multi_arg [sf] "test string" 123.449997, where:

/test_message_with_multi_arg is the message address;
[sf] describes argument types, s - string, f - float number, i - integer number. In this example, two arguments were passed: the first of type string, the second a float;
followed by argument values "test string" 123.449997.

Passing Arguments to Custom Actions

The above method works well for many use cases, but it has one downside: it's impossible to pass any arguments when calling actions. If the variance of possible argument values is low, one can create multiple actions. For example, to enable or disable a certain option: /action/_ENABLE_OPTION, /action/_DISABLE_OPTION. This is fine, but such an approach is not suitable for passing numeric or string arguments to an action.

Is there a way to pass an argument to an action? Yes, as REAPER supports binding OSC messages to trigger actions. When an action is invoked this way, a single (first) OSC message argument of string or float type can be retrieved from the script.

To start, make sure binding OSC messages to actions is allowed: In REAPER, navigate to Preferences (Command + ,), then the Control/OSC/web section. Select the existing OSC control from the list and press Edit.
Check the Allow binding messages to REAPER actions and FX learn option and press Ok.

Create osc.lua in ~/Library/Application\ Support/REAPER/Scripts/MyCompany/ with the following content:

-- osc.lua

-- Utility functions to get and parse OSC message and argument from REAPER action context

local osc = {}

function osc.parse(context)
    local msg = {}

    -- Extract the message
    msg.address = context:match("^osc:/([^:[]+)")

    if msg.address == nil then
        return nil
    end

    -- Extract float or string value
    local value_type, value = context:match(":([fs])=([^%]]+)")

    if value_type == "f" then
        msg.arg = tonumber(value)
    elseif value_type == "s" then
        msg.arg = value
    end

    return msg
end

function osc.get()
    local is_new, name, sec, cmd, rel, res, val, ctx = reaper.get_action_context()
    if ctx == nil or ctx == '' then
        return nil
    end

    return osc.parse(ctx)
end

return osc

Next to it, create test_osc_with_arg.lua:

-- Retrieve the directory of the current script.
local script_path = debug.getinfo(1, "S").source:match("@?(.*/)")
-- Set the package path to include the other scripts in the directory
package.path = package.path .. ';' .. script_path .. '?.lua'
-- Require the osc module
local osc = require('osc')

local msg = osc.get()
if msg then
    reaper.ShowMessageBox("OSC address: " .. msg.address .. ", argument: " .. (msg.arg and msg.arg or "(nil)"), "Info", 0)
else
    reaper.ShowMessageBox("Invalid or no OSC message", "Error", 0)
end

Add the following line to ~/Library/Application\ Support/REAPER/reaper-kb.ini:

SCR 4 0 MY_COMPANY_TEST_OSC_WITH_ARG "MyCompany: test OSC trigger with argument" "MyCompany/test_osc_with_arg.lua"

Finally, the newly created action can be mapped to OSC messages. This can be done in two ways:

By editing the ~/Library/Application\ Support/REAPER/reaper-osc-actions.ini file;
Or by creating a shortcut from the Actions dialog by listening for an incoming OSC message.

I prefer the former method. Add the following line to ~/Library/Application\ Support/REAPER/reaper-osc-actions.ini:

"/test_message_with_arg" 0 0 _MY_COMPANY_TEST_OSC_WITH_ARG

This instructs REAPER to invoke the _MY_COMPANY_TEST_OSC_WITH_ARG action when a "/test_message_with_arg" OSC message is received.

Restart REAPER for changes to take effect.

To test that everything works, from the Terminal run: sendosc localhost 8000 /test_message_with_arg s "test string". This should invoke the test_osc_with_arg.lua script in REAPER and display a message box with the text "OSC address: test_message_with_arg, argument: test string".

Running sendosc localhost 8000 /test_message_with_arg f 123.456 will display the message "OSC address: test_message_with_arg, argument: 123.456001".

Running sendosc localhost 8000 /test_message_with_arg will display the message "OSC address: test_message_with_arg, argument: (nil)".

Limitations

REAPER has the following limitations when it comes to working with OSC messages:

Scripts can only receive the first argument of the OSC message. As a workaround, a string argument can be used to encapsulate any number of arguments.
Only string and float argument types are supported. Again, a string can be used to hold any required datatype that can be represented as a string.
Sending OSC messages from scripts is not supported. As a workaround, sendosc or a similar utility can be invoked using os.execute(), or third-party REAPER extensions can be used to accomplish this task.

Conclusion

REAPER is a powerful DAW offering scripting and extensible external control capabilities, which makes it suitable for use as part of various automation applications, including DAW plugin test automation frameworks. Unfortunately, its minimalistic documentation makes feature discovery problematic and requires additional effort to make things work. Some basic functionality is still missing and requires workarounds.

DEV Community: Maxim Radugin

Monitor Progress of Data Transfer in Console

Run Applications in Rosetta on macOS

Creating and Restoring Disk Images on macOS: A Guide to Using dd and diskutil

🚀 Introduction

⚡ TL;DR

💾 Save Disk to an Image File

🔄 Restore Disk from an Image File

⚠️ macOS Unreadable Disk Warning Dialog

🔍 Identify Physical Disk

⚙️ Prepare Disk for Direct IO Operations

💾 Backup Disk to an Image File

🔄 Restore Disk from an Image File

📦 Optimize Space Used by Disk Images

🕳️ Use of Sparse Files

🗜️ Compress Disk Images

✅ Verify Disk Image

Creating and Verifying Disk Image

Verifying Restored Disk Image

Verifying Compressed Disk Image

🎯 Conclusion

Enter ROS2 Development Container

Introduction

The Devcontainer Advantage

The ROS2 Growing Pains

Enter ros2-devcontainer

Key Features

Real-World Benefits

Getting Started

Further Improvements

Conclusion

Links

GitLab CI: Tips and Tricks

Introduction

Features

Referencing Scripts from Other Jobs

Using Job Scripts as Functions

Multi-line Command Invocation

Exclusive Use of Runners

Customizing Job Execution Using Rules

Clearing Working Directory for Specific Job

Workarounds

Spaces in Variables

Colon+Space in Values

Conclusion

Related Materials

Cleaning up Obsolete Cloudflare Page Deployments

Introduction

Value of Deployment History

Cleaning up Deployment History

Cleaning from Cloudflare

Automating Cleanup

Integrating into CI/CD

Conclusion

Deploying Hugo from Self-Hosted GitLab to Cloudflare Pages

Introduction

Setting and Testing Locally

Setting GitLab CI/CD

Requirements

GitLab Project Configuration

Pipeline Definition

Further Improvements

Conclusion

Bridging Jira and GitLab: Automating CI/CD Pipelines for Releases

Introduction

User Story

Implementation

GitLab Configuration

Create Personal Access Token

Modify CI/CD Configuration

Jira Project Configuration

Verifying Automation

From Jira

From GitLab

Conclusion

Related Materials

Improve Productivity with CMake and Compiler Cache Integration

Introduction

Proven Way of Optimizing Build Times

So, What's the Deal?