DEV Community: Dmitry Linov

Submodules That Don’t Suck

Dmitry Linov — Thu, 17 Feb 2022 15:23:49 +0000

Git X-Modules lets you enjoy the speed, scale and simple sharing of Git submodules without the bit that drives you to delirious madness

Look out: Git submodules are a trap

It starts out so innocently.

You have two applications. They share a lot of code. The last thing you want is to waste time developing different versions of that code. You want a way to work on everything together.

Well, that’s just what Git submodules are for, right? The shared code has its own repository, which sits inside a parent repository for every application that uses it.

You don’t even have to install anything. It’s already in the tool. You type one command and it’s up and running. Easy peasy.

That might have been as much as you thought about it. After all, this wasn’t the bit that really excited you. Your mind was on what you wanted to build.

It slowly dawns on you that Git is really complicated

And now you – and your whole team – have no choice but to be really good at it.

Much of the difficulty comes down to this: Git submodules don’t just sit inside parent repositories. Every version of the parent repository is tied to a particular version of the submodule.

So, if a developer updates a submodule used in five other repositories, the developer has to update every single one of them.

If that sounds tedious, that’s because it is. But it has to be managed meticulously or projects can break and hours of work can just vanish.

Other times, different sets of changes from different developers wind up in different versions of the submodule, across different instances of the repository. Putting this all together for code review becomes a massive headache.

You might mutter to yourself that the entire point of having version control was to avoid this exact situation. But here you are, juggling multiple versions of the same files.

Remembering the good old days when we all knew what to type

Suddenly, all the commands you used to know must be done differently and with more of them. Here’s just a taste:

The familiar way

Clone

git clone git://example.org/repos.git

Commit

git commit

Push

git push

Create branch

git checkout -b branch

The submodule way

Clone

git clone git://example.org/repos.git
git submodule init
git submodule update

git clone --recurse -submodules

Commit

git commit /*in each submodule*/
git add /*in the parent repository for each module*/ 
git commit /*in the parent repository*/

Push

git push /*in each submodule*/
git push /*in the parent repository*/

Create branch

/*Create a branch in each submodule and in the parent repository*/

This all has to be done precisely, and in the precise order too, or else the problems multiply.

It’s so mundane and repetitive that it feels obvious to just write some quick scripts for and be done with it. Except you’ll never really be done with it. You just have a growing list of scripts that need regular attention as projects grow and change.

There’s no getting around the fact that submodules are a very involved way of thinking about projects, which your whole team has to engage with all the time.

And if you’d rather use a graphical tool like Github Desktop? You might be out of luck. Some tools don’t support submodules well. Others don’t support them at all.

Using Git submodules, it doesn’t take a big external shock to derail projects

All you really need is for someone to be distracted or have a bad day.

It’s an expensive way to be miserable

The simple way to start tracking your losses is to add up all the time spent dealing with version control and fixing mistakes. And remember: everyone is losing time to this.

Then ask yourself: what is this doing to your team’s focus and momentum? Their morale and pride in their work?

That’s harder to measure, but it’s still very real.

The pity is that a modular approach should be amazing

There’s so much to like about the idea of separating different projects into their own repository.

It’s fast

Commands take milliseconds, not minutes.
Developers can search and edit a whole repository on their local machine.

It scales

It grows with you from 3 developers to 3,000 and beyond.
There’s no worry about what happens when 100 developers commit to the same repository at the same time.

It’s so easy to share

It makes it simple to work with external contractors without granting access to all your code.
If you want to publish open-source projects without publishing all of them like Richard Stallman, that’s just as simple.

It’s in one place

You always have a single, authoritative version of the code.

These are solid benefits. It’s tough to give them up. So, little wonder that Git submodules still have advocates who will tell you to just deal with it and get used to the headaches and mounting technical debt.

But let’s get real about one thing

Look, it’s not impossible to find developers who get excited by great version control. I mean, we’re some of them.

But most developers aren’t. They’ve worked too hard to build their own areas of focus.

They’ll deal with version control just enough to hold a job down and avoid disaster. But it’s never going to be the thing they’re passionate about, that inspires them to their very best.

It can be thankless too. About the best they can hope for is to do everything perfectly and then see nobody notice because nothing went wrong.

And really.. are developers even wrong to want this off their plate?

Why shouldn’t they have all their attention on their code?

Introducing Git X-Modules

An X-Module sits as an ordinary directory inside a regular Git repository. Developers can treat it just like any other directory, without having to think about it any deeper.

Server-side software keeps this X-Module directory in sync with an external repository, pushing changes in both directions automatically.

What if there’s a conflict? X-Modules accepts one set of changes, turning the other into a pull request.

And if you don’t want to synchronise certain files? You can exclude them either individually, or by extension. So it’s easy not to have those 7-gigabyte .iso files clogging up the bandwidth.

Code review made simple

With X-Modules, all the changes are submitted to one place.
Once they’re approved, they’re synchronised across all repositories automatically.

It’s the modular Monorepo

You’re welcome to think of X-Modules as a kind of monorepo. Because there really is one big repository that everything synchronises to.
You’re also welcome to treat them as separate repositories. You still enjoy all the speed, scale and simple sharing of a modular approach.

Your team already knows how to use it

The best thing about X-Modules is that your team doesn’t have to even pay it much notice.

They just interact with it like any other repository, working exactly the same as all the repositories they’ve used before.

All the familiar commands just work. They can use any third-party tools with no special requirements.

So who else wants the server-side solution that syncs your code and stays out of your way?

Git X-Modules is available as a cloud app for GitHub. You can use it for free while it’s still in beta. Support for GitLab and Bitbucket Cloud will be added soon!
There's also an on-premises app for Bitbucket Data Center.

Seriously, you should try it out.

Text by James Mawson

Embedded development project structure with Git X-Modules

Dmitry Linov — Mon, 21 Dec 2020 11:18:10 +0000

Embedded development, just like any other, often depends on shared code components such as libraries for specific hardware. However, there's no de-facto industry standard for managing modular projects in this area. We are going to describe one of the ways to do it on a Git repository level. We will mostly concentrate on project organization topics rather than on particular questions of embedded development.

Infrastructure

As an example, we will create a simple program that blinks the LEDs of STM32F4DISCOVERY board by STMicroelectronics. I assume we have a STM32F4DISCOVERY board with ST32F407VGT6 MCU. I'm using Debian 10.6, but for any other OS, the steps are similar. I have 'stlink-tools' and 'gcc-arm-none-eabi' packages installed. The former is a collection of tools (and 'st-flash' in particular) to work with the Discovery board (e.g. 'st-flash' writes the compiled binary file to the MCU's flash using STLink protocol). The latter is a generic ARM cross-compiler as ST32F407VGT6 is ARM Cortex M4 based MCU.

If you don't have these packages installed do that now:

 sudo apt install stlink-tools gcc-arm-none-eabi

STMicroelectronics provides a standard library for ST32F407VGT6 (STSW-STM32065). But instead of using it (not everybody is happy with its APIs), we will develop our own library by using MCU datasheets directly. Another reason to develop our own library is to demonstrate a typical scenario when libraries are developed together with the main project itself and can be reused in other projects.

Although we will have only one project in this post, we will assume there're other projects reusing the same library. In particular, fixes and updates to the library should get into these projects as well.

We will assume the code to be stored in Git and Atlassian Bitbucket Server/Data Center is used on the server's side. This is not necessary but would give us a nice UI.

## Project Structure ##
Our project will consist of the following major components:

main project source file;
generic library supporting ST32F407VGT6 MCU;
files and compiler options needed to cross-compile the C code to the ARM platform.

To cross-compile a project, one has to use a number of specific compiler options and also a startup and a linker script file. When having several projects for the same MCU, it makes sense to share these files and options across projects. Thus we will have a common component dedicated to these compilation-related files. We will name this component 'common'.

We will name the library 'stm32' component. The main project will be named 'embedded-example'.

Step 1. Create the 'stm32' project

Create 'stm32' Git repositrory with Atlassian Bitbucket Server/Data Center UI

and clone it.

git clone http://example.org/scm/ee/stm32.git stm32/
cd stm32/

Create CMakeLists.txt file there:

cmake_minimum_required(VERSION 3.8 FATAL_ERROR)
project(stm32)

add_library(stm32 pin.c gpio.c util.c)
target_include_directories(stm32 PUBLIC .)

This CMakeLists.txt file describes a project with three C files and tells all the projects that would include it that the include directory is at the project root. So the files structure will be:

├── CMakeLists.txt
├── gpio.c
├── gpio.h
├── pin.c
├── pin.h
├── util.c
└── util.h

Now create these files and their headers:

util.h & util.c - they contain convenience functions to set and get special bits;
gpio.h & gpio.c - they contain a dull and straightforward implementation of ST32F407VGT6 datasheet GPIO specifications;
pin.h & pin.c - they define pin names like hw_pin_PD12 and their convenience equivalents like hw_pin_led_green.

For instance, on STM32F4DISCOVERY PD12 pin of the MCU is connected to the green LED, that's why it's convenient to define:

#define hw_pin_led_green hw_pin_PD12

Now add, commit, and push the changes to 'stm32' library:

git add *.c
git add *.h
git add CMakeLists.txt
git commit -m "Initial."
git push origin master

I would note that as this library is self-contained, one can even compile (but not cross-compile, so far) it:

mkdir build
cd build
cmake ..
make

If everything compiles, it's a good indicator, though a pretty useless property as our target platform is ARM.

Step 2. Create the 'common' project

This project will contain the common configuration that is shared between projects. It's also convenient to put it into a separate Git repository and insert it into each project repository to be included by the project CMakeLists.txt.

Create the 'common' repository with Atlassian Bitbucket Server/Data Center

and clone the newly created repository:

git clone http://example.org/scm/ee/common.git common/
cd common/

Now create vars.cmake:

set(CMAKE_SYSTEM_NAME      Generic)
set(CMAKE_SYSTEM_VERSION   1)
set(CMAKE_SYSTEM_PROCESSOR arm-eabi)

set(CMAKE_C_COMPILER       arm-none-eabi-gcc)
set(CMAKE_CXX_COMPILER     arm-none-eabi-g++)
set(CMAKE_ASM_COMPILER     arm-none-eabi-as)
set(CMAKE_OBJCOPY          arm-none-eabi-objcopy)
set(CMAKE_OBJDUMP          arm-none-eabi-objdump)

set(CMAKE_C_FLAGS "-mthumb -mcpu=cortex-m4 -fno-builtin -Wall -Wno-pointer-to-int-cast -std=gnu99 -fdata-sections -ffunction-sections" CACHE INTERNAL "c compiler flags")
set(CMAKE_CXX_FLAGS "-mthumb -mcpu=cortex-m4 -fno-builtin -Wall -Wno-pointer-to-int-cast -fdata-sections -ffunction-sections" CACHE INTERNAL "cxx compiler flags")
set(CMAKE_ASM_FLAGS "-mthumb -mcpu=cortex-m4" CACHE INTERNAL "asm compiler flags")

set(CMAKE_EXE_LINKER_FLAGS "-nostartfiles -Wl,--gc-sections -mthumb -mcpu=cortex-m4" CACHE INTERNAL "exe link flags")
set(CMAKE_SHARED_LIBRARY_LINK_C_FLAGS CACHE INTERNAL "reset default flags")

set(STM32_STLINK_CLI_EXECUTABLE "st-flash")

This file defines various variables needed for the cross-compilation of sources with ARM as the target platform. It sets compiler flags and a path to the 'st-flash' utility that writes binary files to MCU.

Also add stm32f407vg_flash.ld, startup_stm32f40xx.s, and stm32f4xx.h files. I will not cite them here, but they are quite standard and distributed by STMicroelectronics.

So the repository content becomes:

├── startup_stm32f40xx.s
├── stm32f407vg_flash.ld
├── stm32f4xx.h
└── vars.cmake

Commit and push the changes:

git add vars.cmake stm32f4xx.h startup_stm32f40xx.s stm32f407vg_flash.ld
git commit -m "Initial."
git push origin master

Step 3. The project repository

Now let's create the main project repository. The repository will have the following structure:

├── CMakeLists.txt
├── common   <--- here common.git will be inserted
├── libs
│   └── stm32  <--- here stm32.git will be inserted
│
└── src
    └── main.c

So create an empty Git repository using Atlassian Bitbucket Server/Data Center UI

and clone it:

git clone http://example.org/scm/ee/embedded-example.git embedded-example/
cd embedded-example/

Create CMakeLists.txt there:

cmake_minimum_required(VERSION 3.8 FATAL_ERROR)
include(common/vars.cmake)
project(embedded-example)
enable_language(C ASM)

add_subdirectory(libs/stm32)

set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -T${CMAKE_SOURCE_DIR}/common/stm32f407vg_flash.ld")

add_executable(${PROJECT_NAME}.elf
    ${CMAKE_SOURCE_DIR}/common/startup_stm32f40xx.s
    ${CMAKE_SOURCE_DIR}/src/main.c)
target_include_directories(${PROJECT_NAME}.elf PRIVATE stm32)
target_link_libraries(${PROJECT_NAME}.elf PRIVATE stm32)

add_custom_target(${PROJECT_NAME}.hex DEPENDS ${PROJECT_NAME}.elf COMMAND ${CMAKE_OBJCOPY} -Oihex ${PROJECT_NAME}.elf ${PROJECT_NAME}.hex)
add_custom_target(${PROJECT_NAME}.bin DEPENDS ${PROJECT_NAME}.elf COMMAND ${CMAKE_OBJCOPY} -Obinary ${PROJECT_NAME}.elf ${PROJECT_NAME}.bin)
set(STLINK_CMD ${STM32_STLINK_CLI_EXECUTABLE} write ${CMAKE_BINARY_DIR}/${PROJECT_NAME}.bin 0x8000000)
add_custom_target(write-flash DEPENDS ${PROJECT_NAME}.bin COMMAND ${STLINK_CMD})

It's important to call include() on the file with common variables before project() call otherwise CMake falls into an infinite loop (I would consider this strange behavior as a CMake bug). It's also important to specify the linker script that we've put into the 'common.git' project.

add_subdirectory() call includes the library. We can include several libraries there. The libraries will be inserted in the libs/ directory.

The last line creates a make write-flash target that will not only create a binary file for the MCU but also will write it into its flash using STLink protocol. ${STM32_STLINK_CLI_EXECUTABLE} is defined in the 'common' project.

Now create src/main.c file:

#include <gpio.h>

hw_pin_t leds[] = {
  hw_pin_led_red,
  hw_pin_led_green,
  hw_pin_led_blue,
//  hw_pin_led_orange
};

int leds_count = sizeof(leds) / sizeof(hw_pin_t);

void SystemInit() {
}

int main() {
  hw_gpio_configure(hw_pin_led_blue, hw_gpio_mode_output_pull_push, hw_gpio_speed_2mhz, hw_gpio_alternate_function_none);
  hw_gpio_configure(hw_pin_led_green, hw_gpio_mode_output_pull_push, hw_gpio_speed_2mhz, hw_gpio_alternate_function_none);
  hw_gpio_configure(hw_pin_led_orange, hw_gpio_mode_output_pull_push, hw_gpio_speed_2mhz, hw_gpio_alternate_function_none);
  hw_gpio_configure(hw_pin_led_red, hw_gpio_mode_output_pull_push, hw_gpio_speed_2mhz, hw_gpio_alternate_function_none);

  int current_led = 0;

  while (TRUE) {  
    hw_gpio_set(leds[current_led], FALSE);
    current_led = (current_led + 1) % leds_count;
    hw_gpio_set(leds[current_led], TRUE);

    int delay = 1000000;
    while (delay--) {
    }
  }
}

This file is simple: it initializes pins related to LEDs and blinks with red, green, and blue pins. We've commented out the orange pin as default STM32F4DISCOVERY firmware blinks with all 4 LEDs and we want to differ from that.

Commit and push the changes:

git add src/main.c
git add CMakeLists.txt
git commit -m "Initial."
git push origin master

Step 4. Insert 'common.git' and 'stm32.git' repositories into 'embedded-example.git'

To insert one repository to another we will use Git X-Modules. For Atlassian Bitbucket Server/Data Center there's a dedicated app with a nice UI. For other Git servers use the: Git X-Modules Command-Line Tool.

Make sure you have X-Modules app installed into Atlassian Bitbucket Server/Data Center. If not, visit Administration | Find new apps | Search the [Marketplace](https://marketplace.atlassian.com/apps/1223696/git-x-modules-for-bitbucket-server) and type "X-Modules" from Bitbucket Server/Data Center UI.

Go to the 'embedded-example' Git repository page. When the Git X-Modules app is installed there's a Git X-Modules button on the left panel, click it.

Then click 'Add Module' to add the first module (let it be 'common.git').

Choose 'common' repository and 'master' branch. Make sure "This Repository Path:" is 'common'. It's the path where the repository will be inserted:

Click 'Add Module'. Without applying the changes click 'Add Module' again to add 'stm32' repository as module.

Choose 'stm32' repository and 'master' branch.

Make sure "This Repository Path:" is "libs/stm32", this is the insertion path for 'stm32.git' repository.

Click 'Add Module' and apply the changes.

Now the repositories are inserted as X-Modules.

This means that 'common.git' and 'stm32.git' are synchronized with corresponding directories ('common' and 'libs/stm32' respectively) of 'embedded-example.git'.

Fetch the changes from 'embedded-example':

cd embedded-example/
git pull --rebase

Now the project contains everything:

├── CMakeLists.txt
├── common
│   ├── startup_stm32f40xx.s
│   ├── stm32f407vg_flash.ld
│   ├── stm32f4xx.h
│   └── vars.cmake
├── libs
│   └── stm32
│       ├── CMakeLists.txt
│       ├── gpio.c
│       ├── gpio.h
│       ├── pin.c
│       ├── pin.h
│       ├── util.c
│       └── util.h
└── src
    └── main.c

Check that the project compiles:

mkdir build
cd build
cmake ..
make
make write-flash

At the moment of running make write-flash, the STM32F4DISCOVERY board should be connected. If you do everything correctly, you'll see 3 LEDs blinking.

This repository structure can be reused for any other ST32F407VGT6-based project, it's enough to insert 'common' and 'stm32' at the corresponding places. Moreover, the directories are bi-directionally synchronized with the inserted repositories, so one can change, see the results, and modify 'stm32' directly from the project repository.

Setting up multi-module Python environment with X-Modules and Develop Mode

Dmitry Linov — Wed, 09 Dec 2020 10:42:18 +0000

Quite often a Python project consists not only of an app but also of some libraries that could also be used by other projects.

On one hand, it's very convenient to work with the app repository independently of libraries assuming the libraries are installed on the system (virtual environment).
On the other hand, this creates a painful problem of "change the library" -> "install the library" -> "check the change from the app development setup" cycle.

This post shows, how you can manage this setup with Git X-Modules.

For the Git server, we will use Atlassian Bitbucket Server/Data Center --- one of the most popular self-hosting Git solutions. It's also convenient because there's a special Git X-Modules App for it. If you are on any other Git Server, you may still use Git X-Modules as a command-line tool.

The configuration. As an example, we'll create an app (let's call it 'qsolver') that solves quadratic equations. I will also create a library ('qsolverlib) that performs mathematical calculations, so the 'qsolver' app will use it to output the solution.

Finally, there will be a 'qsolver-project' containing both components. This will help us to pinpoint the version of 'qsolverlib' used by 'qsolver' because otherwise 'qsolver' would use 'qsolverlib' installed on the system that could be of the wrong version. This is especially useful in real life when some problem happens to some older version of the application and it's important to deduce the library version used for that application.

We will create 3 Git repositories: for 'qsolverlib', 'qsolver', and 'qsolver-project'.

Step 1. Create 'qsolverlib' Git repository.
Create 'qsolverlib' Git repository on Atlassian Bitbucket Server/Data Center.

Clone this empty Git repository:

git clone http://example.org/scm/qsolver/qsolverlib.git qsolverlib/
cd qsolverlib/

Inside this Git repository create the 'qsolverlib' subdirectory and create the solve.py file in it with the following content:

"""Quadratic equation solver."""

import math

def solve(a, b, c):
    """Solve equation ax^2 + bx + c = 0, return a root or a list with roots"""
    D = b * b - 4 * a * c
    if D < 0:
        return None
    elif D == 0:
        return -b / (2 * a)
    else:
        return [(-b - math.sqrt(D)) / (2 * a), (-b + math.sqrt(D)) / (2 * a)]

Also create an empty "__init__.py" file in the "qsolverlib" subdirectory:

touch qsolverlib/__init__.py

Finally, create "setup.py" at the root of the Git repository:

"""Setup file for qsolverlib."""

import setuptools

setuptools.setup(
    name="qsolverlib",
    version="0.0.1",
    description="A library solving quadratic equations",
    url="http://example.org/scm/qsolver/qsolverlib.git",
    author="Dmitry Pavlenko",
    author_email="pavlenko@tmatesoft.com",
    license="MIT",
    classifiers=[
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python"
    ],
    packages=setuptools.find_packages(),
    python_requires=">=3.7"
)

So the resulting structure looks as follows:

.
├── qsolverlib
│   ├── __init__.py
│   └── solve.py
└── setup.py

Commit and push the package.

git add .
git commit -m "Initial."
git push origin master

Step 2. Create Python virtual environment and install the package there.

Python virtual environment allows installing python packages locally into some directory without requiring to install them at the system-wide level. To create the environment run (not inside the Git repository but inside it parent directory):

virtualenv .venv/

python3 -m venv .venv/

As result, the ".venv/" directory will be created. This environment should be activated to be used:

source .venv/bin/activate

After that "(.venv)" will be prepended to the command prompt indicating that we're working with the virtual environment.

Now install the 'qsolverlib' package into this environment:

pip3 install qsolverlib/

For me it prints:

...
  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for qsolverlib
...

This can be resolved by installing the 'wheel' package:

pip3 install wheel

Then retrying

pip3 install qsolverlib/

is successful. Now the library can be used by an app if the app is running within the same virtual environment.

The virtual environment can be de-activated with

deactivate

command.

Step 3. Create a 'qsolver' app.

Create 'qsolver' Git repository on Atlassian Bitbucket Server/Data Center.

Clone this empty Git repository:

git clone http://example.org/scm/qsolver/qsolver.git qsolver/
cd qsolver/

Create app.py with the following content:

from qsolverlib import solve

def main():
    print(solve.solve(1, -3, 2))

if __name__ == "__main__":
    main()

I.e. it solves: x^2 - 3x + 2 = 0

Commit and push this file:

git add app.py
git commit -m "Initial."
git push origin master

Check that the app works:

python3 app.py 
[1.0, 2.0]

Indeed, x^2 - 3x + 2 = (x - 1)(x - 2)

Step 4. Develop mode of package.

If we now want to change the "solve()" function, we have to edit solve.py in the 'qsolverlib' repository, install the package, and only then test how it works in app.py in 'qsolver'. This is too inconvenient. "Develop mode" is the solution.

To install 'qsolverlib' in develop mode run.

pip3 install -e qsolverlib/

It logically links qsolverlib installed in the virtual environment with the sources of 'qsolverlib', so any changes to 'qsolverlib' sources become effective for the app immediately.

Step 5. Create a parent project to pinpoint versions of 'qsolverlib' and 'qsolver'.

Create 'qsolver-project' Git repository on Atlassian Bitbucket Server/Data Center.

Now we will use Git X-Modules, make sure you have it installed. To do that, go to Administration | Find new apps | Search theMarketplace and type "X-Modules".

Go to the 'qsolver-project' repository page on Bitbucket. As the app is now installed, there's an X-Modules button, click it.

The repository is empty, so click "Create Default Branch" to create the "master" branch.

Click "Add Module" to add 'qsolverlib' as the first module.

Choose the 'qsolverlib' repository and the master branch.

Make sure "This Repository Path" is 'qsolverlib'. It's a path for the insertion point of the 'qsolverlib' module. Click "Add Module".

Instead of applying changes immediately click "Add Module" again, to add the second module (for the app).

Choose the 'qsolver' repository and the master branch.

Make sure "This Repository Path" is 'qsolver'. Again, this is a path for the insertion point of the 'qsolver' module.

Click "Add Module".

Now apply the changes.

Now 'qsolver-project' repository contains both 'qsolverlib' and 'qsolver' X-Modules. These are regular Git directories that are synchronized with the corresponding repositories.

So now one can clone the 'qsolver-project' repository and both 'qsolverlib' and 'qsolver' components will be there.

Step 6. Quickstart script.

Suppose you have a new colleague who wants to start working on both 'qsolverlib' and 'qsolver' as soon as possible. For this purpose, we will create a "create-virtualenv.sh" script that would set up the development environment.

Clone 'qsolver-project':

git clone http://example.org/scm/qsolver/qsolver-project.git qsolver-project
cd qsolver-project

The current structure of 'qsolver-project' is:

├── qsolver
│   └── app.py
└── qsolverlib
    ├── qsolverlib
    │   ├── __init__.py
    │   └── solve.py
    └── setup.py

It already contains 'qsolverlib' and 'qsolver' components, so one doesn't need to clone them separately.

Create "create-virtualenv.sh" script with the following content:

#!/bin/sh

if [ -z "$1" ]; then
    echo "Path for the virtual environment required, e.g.:"
    echo ""
    echo "    $0 .venv/"
    echo ""
    exit 1
fi

python3 -m venv "$1"
$1/bin/pip3 install wheel
$1/bin/pip3 install -e qsolverlib/

echo "Virtual environment created at $1"

Add, commit, and push this script:

git add create-virtualenv.sh
git commit -m "create-virtualenv.sh script added."
git push origin master

So now for a new colleague, it will be enough to clone 'qsolver-project' and run this script to start working on the project:

sh create-virtualenv.sh .venv/
source .venv/bin/activate

Step 7. An example: make a change to 'qsolverlib'.

Now let's improve our solve() function to handle the case a=0, i.e. to solve bx+c=0 equation.

Edit qsolverlib/qsolverlib/solve.py and qsolver/app.py:

Commit and push the changes with to both 'qsolverlib' and 'qsolver' as a single commit in 'qsolver-project'.

git commit -a -m "solve() improved to handle linear equations as well."
git push origin master

Now you can look at 'qsolverlib' and 'qsolver' histories on Bitbucket: they are updated!

The synchronization is bi-directional, updates of 'qsolverlib' and 'qsolver' also get into 'qsolver-project' automatically.

Multi-component CMake-based project with Git X-Modules

Dmitry Linov — Sun, 06 Dec 2020 14:33:08 +0000

How to build a multi-component project without installing all libraries.

At the time of writing this, there's no default solution for dependency management in the C world. There're several competing tools like Basel, Meson, conan, etc but none of them is recognized as standard de facto like Maven in Java world.

In this post, we will describe a CMake-based setup of a project consisting of several modules, which:

can be built and installed independently;
can be build together, so all the app and library sources can be edited, debugged, and committed simultaneously.

This setup resolves the pain of "fix and build the library" -> "install the library" -> "build the app with the library" -> "test the fix from the app" cycle.

As an example, we will create an app that computes SHA-1 checksum of its argument. To compute SHA-1 checksum we will use an external library by Steve Reid.

Project structure.
Currently, the SHA-1 library in question has only a Makefile to build the project. We want to add a CMake-related build file (CMakeLists.txt) to this library. As we don't have write access to this repository we will fork this project to our Git server. For that, we will use Atlassian Bitbucket Server/Data Center as one of the most popular self-hosting Git solutions. It's also convenient because there's a special Git X-Modules App for it. If you are on any other Git Server, you may still use Git X-Modules as a command-line tool.

The SHA-1 library component will be called 'sha-1'. Then we will create a 'checksum' component that uses 'sha-1' library and produces an executable file. One should be able to build this component independently assuming 'sha-1' library is installed (into /usr/lib and /usr/include or any other OS-specific directory for libraries).

Finally, we will create a 'checksum-project' that includes both 'sha-1' and 'checksum' components and can be built without having to install 'sha-1' library to the system. This will allow us to open, edit, and build all components in an IDE.

From the Git perspective, one can use Git submodules for that. But this solution causes a lot of problems. For example, the Projectile package of Emacs doesn't currently work well with submodules. We will use a better solution: Git X-Modules.

Step 1.
Create 'sha-1' repository with Atlassian Bitbucket Server/Data Center interface.

We want now to fork https://github.com/clibs/sha1 project there. To do that, clone the sha1 project from GitHub and push it to our self-hosted Bitbucket Server repository (we assume it's running on example.org domain):

git clone https://github.com/clibs/sha1.git sha-1
cd sha-1
git remote set-url origin http://example.org/scm/checksum/sha-1.git
git push origin master

Step 2. Add our custom CMakeLists.txt to 'sha-1' project with the following content:

cmake_minimum_required(VERSION 3.8 FATAL_ERROR)
project(sha1)

add_library(sha1 sha1.c)
target_include_directories(sha1 PUBLIC .)
set_target_properties(sha1 PROPERTIES PUBLIC_HEADER "sha1.h")
install(TARGETS sha1 LIBRARY DESTINATION lib ARCHIVE DESTINATION lib PUBLIC_HEADER DESTINATION include)

The install() command tells CMake to install the library to /usr/local/lib and the header to /usr/local/include, so it can be easily included by the app.

The target_include_directories() call is not necessary to build this project but will be useful later for the multi-module project structure.

Commit and push the change:

git add CMakeLists.txt
git commit -m "CMakeLists.txt added."
git push origin master

Make sure CMake can build the project. We can use these old-fashioned commands:

mkdir build
cd build
cmake ..
make

To install the library into the system one can use "sudo make install" command but later we will describe how to create a multi-component structure that doesn't need installation. So we do not install the library.

Step 3. Create a Git repository for our app.

Create a 'checksum' repository using Atlassian Bitbucket Server/Data Center interface.

Clone and "cd" into the repository:

git clone http://example.org/scm/checksum/checksum.git checksum
cd checksum

Create the main.c file with the following content:

#include <sha1.h>
#include <stdio.h>
#include <string.h>

#define SHA1_SIZE_IN_BYTES 20

void print_usage(char* command) {
  fprintf(stderr, "Usage: %s STRING\n", command);
}

void print_hex(char* buffer, int buffer_size) {
  for(int i = 0; i < buffer_size; i++) {
    printf("%02x", buffer[i] & 0xff);
  }
}

int main(int argc, char** argv) {
  if (argc != 2) {
    print_usage(argv[0]);
    return 1;
  }
  char sha1[SHA1_SIZE_IN_BYTES + 1];
  char* str = argv[1];

  SHA1(sha1, str, strlen(str));
  sha1[SHA1_SIZE_IN_BYTES] = '\0';

  print_hex(sha1, SHA1_SIZE_IN_BYTES);
  printf("\n");

  return 0;
}

Then create CMakeLists.txt file with this content:

cmake_minimum_required(VERSION 3.8 FATAL_ERROR)
project(checksum)

add_executable(checksum main.c)

target_include_directories(checksum PRIVATE sha1)
target_link_libraries(checksum PRIVATE sha1)

The main.c is simple: it just calls SHA1() function from 'sha-1' library and CMakeLists.txt adds a dependency on 'sha-1' library. Commit and push the change:

git add main.c CMakeLists.txt
git commit -m "Initial commit."
git push origin master

If 'sha-1' is installed on the system with "sudo make install" command of "Step 2", our checksum app can be build using CMake. But if 'sha-1' is not installed, CMake build command will fail because it can't find the header file for the library:

mkdir build
cd build
cmake ..
make

Scanning dependencies of target checksum
[ 50%] Building C object CMakeFiles/checksum.dir/main.c.o
/home/dmit10/work/blog/02-cmake-post/checksum/main.c:1:10: fatal error: sha1.h: No such file or directory
 #include <sha1.h>
          ^~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/checksum.dir/build.make:63: CMakeFiles/checksum.dir/main.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:73: CMakeFiles/checksum.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

Step 4. Multi-module structure.
Now we want to create a multi-module repository 'checksum-project' with the following structure:

CMakeLists.txt
checksum/    <-- 'checksum' module insertion point
libs/sha-1/  <-- 'sha-1' library insertion point

If you don't have the Git X-Modules app for Bitbucket installed, install it now. To do so, go to Administration | Find new apps | Search the Marketplace and type "X-Modules".

Now create a new Git repository and name it 'checksum-project'.

Once the Git X-Modules app is installed, there will be an X-Modules button on the Git repository page. Click it.

As the repository is empty, click "Create Default Branch".

Then click "Add Module" to add the 'sha-1' library.

Choose the 'sha-1' repository.

Choose the 'master' branch in the 'sha-1' repository.

Make sure the 'sha-1' module is inserted into 'libs/sha-1' so "This Repository Path" should be 'libs/sha-1'.

Click "Add Module". Do not "Apply Changes" yet.

Now add another module to add 'checksum' app to our multi-module structure. To do that click 'Add Module' again:

Then choose the 'checksum' repository.

Choose the 'master' branch in it.

Make sure the 'checksum' repository is inserted into the 'checksum' directory, so "This Repository Path" should be 'checksum'. The directory name is arbitrary (as well as "libs/sha-1"), we will specify it later in add_subdirectory() call.

Click 'Add Module'

Apply the changes.

Now 'checksum-project' repository contains 2 directories: 'checksum' with 'checksum' module and 'libs/sha-1' with 'sha-1' library. They are inserted into 'checksum-project' as normal directories in Git.

Step 5. Create a multi-module CMake configuration in 'checksum-project'.

Clone and 'cd' into the 'checksum-project':

git clone http://example.org/scm/checksum/checksum-project.git checksum-project/
cd checksum-project/

Create CMakeLists.txt with the following content:

cmake_minimum_required(VERSION 3.8 FATAL_ERROR)
project(checksum-project)

add_subdirectory(libs/sha-1)
add_subdirectory(checksum)

Commit and push the file:

git add CMakeLists.txt
git commit -m "CMakeLists.txt configuration added."
git push origin master

That's all!

The project can now be built without installation:

mkdir build
cd build
cmake ..
make

[ 25%] Building C object libs/sha-1/CMakeFiles/sha1.dir/sha1.c.o
[ 50%] Linking C static library libsha1.a
[ 50%] Built target sha1
[ 75%] Building C object checksum/CMakeFiles/checksum.dir/main.c.o
[100%] Linking C executable checksum
[100%] Built target checksum

Now let's try our app:

checksum/checksum foobar
8843d7f92416211de9ebb963ff4ce28125932878

and compare it with standard 'sha1sum' utility:

echo -n foobar | sha1sum
8843d7f92416211de9ebb963ff4ce28125932878  -

All sources are in the same repository as normal Git directories. Now one can modify 'libs/sha-1' and 'checksum/' inside the 'checksum-project' directly and these changes will be synchronized with corresponding 'sha-1' and 'checksum' repositories. And vice versa.

Have fun!

How to develop an Android library together with the apps using it

Dmitry Linov — Wed, 02 Dec 2020 11:46:51 +0000

Imagine that you are developing a taxi service like Uber, with two separate apps (one for passengers, another for drivers). However, they are might be using the same shared libraries (e.g. for network protocols or UI animation), and both teams need access to it. Did you know, that monorepo is not the only way to handle this?

This post describes a way to organize sources of 2 Android apps ('app1' and 'app2') using the same common library ('commonlib'). The configuration will allow one to work on the library sources together with both apps.

Why do you need this?
A common approach would be to put the common library to the Maven and to use it as a binary dependency for each of the apps. This approach makes it difficult to quickly test changes in the library code: to do that one has to

fix the 'commonlib',
build and upload changes to the Maven (often this is done by CI and involves runnings tests),
obtain the updated artifact from the 'app1' or 'app2' project,
and, finally, build and re-run the app to see what's different now.

If the changes do not work as intended, one has to repeat.

Another approach would be to put the 'commonlib', code into the same repository and project as 'app1' or 'app2'. With just one app that would have worked. But with two or more apps it would be an obvious code duplication and therefore not an option.

Finally, there's the "monorepo" approach: put all code - ('app1', 'app2', and 'commonlib') into the same Git repository and manage it together. But, besides other well-known drawbacks of the monorepo approach, it's the best practice to place all code components inside one project directory - yet the 'commonlib' can't be inside the same directory with 'app1' and 'app2' at the same time

The solution sketch.
We will adopt a modification of the second approach because it's natural to the Android Studio. When you create an Android library in the Android Studio, it puts the library as another module into the same app project.

Android app projects are usually built with Gradle. They are organized as Gradle projects containing Gradle modules. One of the modules is the app module itself, while libraries it uses could be added as separate modules (together with their sources if developed simultaneously with the app itself) or as binary Maven dependencies (and then one cannot change their sources on the fly).

In our example, we will create 2 projects: 'app1' and 'app2'. Each of them would contain an app-related module (named 'app' by default) and a 'commonlib' module with the sources of the library. We just need to make sure that the 'commonlib' module of both 'app1' and 'app2' projects uses the same sources.

A possible solution would be to put the 'commonlib' module into a separate Git repository and insert it as a Git submodule into the other two Git repositories: 'app1' and 'app2' (corresponding to Gradle projects with the same names). But you know...

Git submodules have too many disadvantages.

So we will use Git X-Modules instead. This is a drop-in replacement for Git submodules, working at the server's side. From the Git perspective, they are just regular directories.

Infrastructure. For the purpose of this post I will use Atlassian Bitbucket Server/Data Center as Git server software. It's one of the most popular self-hosted Git solutions and Git X-Modules has a dedicated app with a nice UI for it. Yet the same solution would also work for almost any other Git server software - there's a command-line version of it with the same capabilities, just without the GUI.

I'm using Android Studio 4.4.1 and Gradle 6.7.1. Both are the latest versions to date.

Step 1. Create the 'app1' project.

Create a new project, choose "Basic Activity" as the template.

Choose the name of the app (App1), click Finish.

Run the project to make sure everything works smoothly.

Step 2. Create the 'commonlib' library.

Choose File | New | New Module...

Choose "Android library" as the module type.

Choose the module name: 'commonlib'.

Create CommonLib class in the 'commonlib' module with a method.

Add a dependency for the default ('app') module on the 'commonlib' module and use the method in the application.

Run the application to make sure everything works smoothly.

Step 3. Create 'app1' and 'commonlib' Git repositories.

The 'App1' project now has the following structure:

├── app
├── build.gradle
├── commonlib     <--- this should go to commonlib.git
├── gradle
├── gradle.properties
├── gradlew
├── gradlew.bat
├── local.properties
└── settings.gradle

Now we will create the 'commonlib' Git repository and put the 'commonlib' subdirectory there. All other files (except those that shouldn't be in Git at all) will go to the 'app1' Git repository. After that, the 'commonlib' repository will be inserted into the 'app1' Git repository.

Create the 'commonlib' Git repository using the Atlassian Bitbucket Server/Data Center interface.

Copy the Git URL of this newly created repository, e.g.

http://example.org/scm/android/commonlib.git

The simplest way to push 'commonlib' content (except the 'build' directory) to this Git repository is the following:

cd commonlib/
git init .
$ git add src
git add build.gradle 
git add consumer-rules.pro
git add proguard-rules.pro
git commit -m "Initial."
git push http://example.org/scm/android/commonlib.git master

Now create the 'app1' Git repository with Atlassian Bitbucket Server/Data Center.

Suppose its URL is

http://example.org/scm/android/app1.git

Put everything except commonlib, app/build, and local.properties to this repository.

cd ..
git init .
git add app/build.gradle 
git add app/libs
git add app/proguard-rules.pro 
git add app/src
git add gradle
git add gradlew
git add gradlew.bat 
git add gradle.properties 
git add build.gradle
git add settings.gradle 
git commit -m "Initial."
git push http://example.org/scm/android/app1.git master

Step 4. Insert 'commonlib' into the 'app1' repository.

We will be using Git X-Modules. It inserts one repository into another on the server's side as Git trees. So, for the Git client, the module will be just a part of a regular Git tree.

If you don't have the Git X-Modules app installed, go to Administration | Find new apps | Search the Marketplace and type "X-Modules" in the Bitbucket Server/Data Center UI to install this app.

Now go to the 'app1' Git repository page. Click the "Git X-Modules" button on the sidebar.

Now click "Add Module" to add 'commonlib' to the project.

Choose the 'commonlib' repository.

And the 'master' branch.

Make sure "This Repository Path" is 'commonlib'. It's the path in 'app1' where the 'commonlib' repository will be inserted.

Click "Add Module" and apply changes. Now 'app1' Git repository has 'commonlib' directory with 'commonlib' Git repository inserted there.

Now any team member can clone the 'app1' Git repository, create local.properties, and build it with ./gradlew build. Alternatively one could add local.properties.example into the repository to make it simpler to start working with the project.

Step 5. Create the 'app2' repository.

In the same way as with 'App1', create the 'App2' application project.

Create the 'app2' Git repository using Bitbucket Server/Data Center UI.

Put 'App1' project content to the 'app1' Git repository.

cd app2
git init .
git add app/
git add build.gradle
git add gradle/
git add gradle. properties
git add gradlew
git add gradlew.bat
git add settings.gradle
git add .gitignore
git commit -m "Initial."
git remote set-url origin http://example.org/scm/android/app1.git
git push origin master

http://example.org/scm/android/app2.git is the Git URL of the 'app2' repository.

Step 6. Insert 'commonlib' into the 'app2' repository using Git X-Modules.

In a similar way, click the Git X-Modules button on the 'app2' repository page, add the 'commonlib' repository as X-Module to the 'commonlib' directory ("This repository path").

Apply the changes.

Step 7. Add dependency for 'app2' on 'commonlib'.

To fetch the changes, run from 'app2':

git remote set-url origin http://example.org/scm/android/app2.git
git pull --rebase

Change settings.gradle' to include ':commonlib' subdirectory.include ':app' include ':commonlib' rootProject.name = "App2"`

Change app/build.gradle to add:
implementation project(path: ':commonlib')
to the dependencies list.

Commit and push the changes:
git add app/build.gradle git add settings.gradle git commit -m "Add dependency on 'commonlib'." git push origin master

Step 8. Test using 'commonlib' in 'app2'.

Change MainActivity in 'app2' to call CommonLib.helloFromCommonLib().

Run the result in the emulator to see how it works.

Commit and push the changes.

Step 9. Test changing 'commonlib'.

Change 'commonlib' in the 'app2' repository,

push the changes, get the changes in 'app1' and make sure it's updated.

git commit -a -m "'commonlib' updated" git push origin master cd .. rm -rf app1 git clone http://example.org/scm/android/app1.git app1/

Run 'app1' to make sure 'commonlib' is automatically updated.

As you may see, the 'commonlib' is now automatically synchronized between both apps.