DEV Community

Cover image for Setting up multi-module Python environment with X-Modules and Develop Mode
Dmitry Linov
Dmitry Linov

Posted on • Originally published at support.tmatesoft.com

Setting up multi-module Python environment with X-Modules and Develop Mode

Quite often a Python project consists not only of an app but also of some libraries that could also be used by other projects.

  • On one hand, it's very convenient to work with the app repository independently of libraries assuming the libraries are installed on the system (virtual environment).
  • On the other hand, this creates a painful problem of "change the library" -> "install the library" -> "check the change from the app development setup" cycle.

This post shows, how you can manage this setup with Git X-Modules.

For the Git server, we will use Atlassian Bitbucket Server/Data Center --- one of the most popular self-hosting Git solutions. It's also convenient because there's a special Git X-Modules App for it. If you are on any other Git Server, you may still use Git X-Modules as a command-line tool.

The configuration. As an example, we'll create an app (let's call it 'qsolver') that solves quadratic equations. I will also create a library ('qsolverlib) that performs mathematical calculations, so the 'qsolver' app will use it to output the solution.

Finally, there will be a 'qsolver-project' containing both components. This will help us to pinpoint the version of 'qsolverlib' used by 'qsolver' because otherwise 'qsolver' would use 'qsolverlib' installed on the system that could be of the wrong version. This is especially useful in real life when some problem happens to some older version of the application and it's important to deduce the library version used for that application.

01-structure|434x264

We will create 3 Git repositories: for 'qsolverlib', 'qsolver', and 'qsolver-project'.

Step 1. Create 'qsolverlib' Git repository.
Create 'qsolverlib' Git repository on Atlassian Bitbucket Server/Data Center.
02-create-qsolverlib-repository|658x500
Clone this empty Git repository:

git clone http://example.org/scm/qsolver/qsolverlib.git qsolverlib/
cd qsolverlib/
Enter fullscreen mode Exit fullscreen mode

Inside this Git repository create the 'qsolverlib' subdirectory and create the solve.py file in it with the following content:

"""Quadratic equation solver."""

import math

def solve(a, b, c):
    """Solve equation ax^2 + bx + c = 0, return a root or a list with roots"""
    D = b * b - 4 * a * c
    if D < 0:
        return None
    elif D == 0:
        return -b / (2 * a)
    else:
        return [(-b - math.sqrt(D)) / (2 * a), (-b + math.sqrt(D)) / (2 * a)]
Enter fullscreen mode Exit fullscreen mode

Also create an empty "__init__.py" file in the "qsolverlib" subdirectory:

touch qsolverlib/__init__.py
Enter fullscreen mode Exit fullscreen mode

Finally, create "setup.py" at the root of the Git repository:

"""Setup file for qsolverlib."""

import setuptools

setuptools.setup(
    name="qsolverlib",
    version="0.0.1",
    description="A library solving quadratic equations",
    url="http://example.org/scm/qsolver/qsolverlib.git",
    author="Dmitry Pavlenko",
    author_email="pavlenko@tmatesoft.com",
    license="MIT",
    classifiers=[
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python"
    ],
    packages=setuptools.find_packages(),
    python_requires=">=3.7"
)
Enter fullscreen mode Exit fullscreen mode

So the resulting structure looks as follows:

.
├── qsolverlib
│   ├── __init__.py
│   └── solve.py
└── setup.py
Enter fullscreen mode Exit fullscreen mode

Commit and push the package.

git add .
git commit -m "Initial."
git push origin master
Enter fullscreen mode Exit fullscreen mode

03-qsolverlib-repository-toc|560x500

Step 2. Create Python virtual environment and install the package there.

Python virtual environment allows installing python packages locally into some directory without requiring to install them at the system-wide level. To create the environment run (not inside the Git repository but inside it parent directory):

virtualenv .venv/
Enter fullscreen mode Exit fullscreen mode

or

python3 -m venv .venv/
Enter fullscreen mode Exit fullscreen mode

As result, the ".venv/" directory will be created. This environment should be activated to be used:

source .venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

After that "(.venv)" will be prepended to the command prompt indicating that we're working with the virtual environment.

Now install the 'qsolverlib' package into this environment:

pip3 install qsolverlib/
Enter fullscreen mode Exit fullscreen mode

For me it prints:

...
  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for qsolverlib
...
Enter fullscreen mode Exit fullscreen mode

This can be resolved by installing the 'wheel' package:

pip3 install wheel
Enter fullscreen mode Exit fullscreen mode

Then retrying

pip3 install qsolverlib/
Enter fullscreen mode Exit fullscreen mode

is successful. Now the library can be used by an app if the app is running within the same virtual environment.

The virtual environment can be de-activated with

deactivate
Enter fullscreen mode Exit fullscreen mode

command.

Step 3. Create a 'qsolver' app.

Create 'qsolver' Git repository on Atlassian Bitbucket Server/Data Center.
04-create-qsolver-repository|622x500
Clone this empty Git repository:

git clone http://example.org/scm/qsolver/qsolver.git qsolver/
cd qsolver/
Enter fullscreen mode Exit fullscreen mode

Create app.py with the following content:

from qsolverlib import solve

def main():
    print(solve.solve(1, -3, 2))

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

I.e. it solves: x^2 - 3x + 2 = 0

Commit and push this file:

git add app.py
git commit -m "Initial."
git push origin master
Enter fullscreen mode Exit fullscreen mode

Check that the app works:

python3 app.py 
[1.0, 2.0]
Enter fullscreen mode Exit fullscreen mode

Indeed, x^2 - 3x + 2 = (x - 1)(x - 2)

Step 4. Develop mode of package.

If we now want to change the "solve()" function, we have to edit solve.py in the 'qsolverlib' repository, install the package, and only then test how it works in app.py in 'qsolver'. This is too inconvenient. "Develop mode" is the solution.

To install 'qsolverlib' in develop mode run.

pip3 install -e qsolverlib/
Enter fullscreen mode Exit fullscreen mode

It logically links qsolverlib installed in the virtual environment with the sources of 'qsolverlib', so any changes to 'qsolverlib' sources become effective for the app immediately.

Step 5. Create a parent project to pinpoint versions of 'qsolverlib' and 'qsolver'.

Create 'qsolver-project' Git repository on Atlassian Bitbucket Server/Data Center.
06-create-qsolver-project-repository|649x500
Now we will use Git X-Modules, make sure you have it installed. To do that, go to Administration | Find new apps | Search theMarketplace and type "X-Modules".
07-install-x-modules-app|690x203
Go to the 'qsolver-project' repository page on Bitbucket. As the app is now installed, there's an X-Modules button, click it.
09-x-modules-button|690x181
The repository is empty, so click "Create Default Branch" to create the "master" branch.
10-x-modules-create-default-branch|690x285
Click "Add Module" to add 'qsolverlib' as the first module.
11-x-modules-add-first-module|690x275
Choose the 'qsolverlib' repository and the master branch.
11-x-modules-choose-qsolverlib|690x461
Make sure "This Repository Path" is 'qsolverlib'. It's a path for the insertion point of the 'qsolverlib' module. Click "Add Module".
12-x-modules-add-second-module|690x427
Instead of applying changes immediately click "Add Module" again, to add the second module (for the app).

Choose the 'qsolver' repository and the master branch.
13-x-modules-choose-qsolver|690x457
Make sure "This Repository Path" is 'qsolver'. Again, this is a path for the insertion point of the 'qsolver' module.
14-x-modules-second-module-preview|690x465
Click "Add Module".
15-x-modules-apply-changes|690x456
Now apply the changes.

Now 'qsolver-project' repository contains both 'qsolverlib' and 'qsolver' X-Modules. These are regular Git directories that are synchronized with the corresponding repositories.
16-qsolver-project-toc|575x499
So now one can clone the 'qsolver-project' repository and both 'qsolverlib' and 'qsolver' components will be there.

Step 6. Quickstart script.

Suppose you have a new colleague who wants to start working on both 'qsolverlib' and 'qsolver' as soon as possible. For this purpose, we will create a "create-virtualenv.sh" script that would set up the development environment.

Clone 'qsolver-project':

git clone http://example.org/scm/qsolver/qsolver-project.git qsolver-project
cd qsolver-project
Enter fullscreen mode Exit fullscreen mode

The current structure of 'qsolver-project' is:

├── qsolver
│   └── app.py
└── qsolverlib
    ├── qsolverlib
    │   ├── __init__.py
    │   └── solve.py
    └── setup.py
Enter fullscreen mode Exit fullscreen mode

It already contains 'qsolverlib' and 'qsolver' components, so one doesn't need to clone them separately.

Create "create-virtualenv.sh" script with the following content:

#!/bin/sh

if [ -z "$1" ]; then
    echo "Path for the virtual environment required, e.g.:"
    echo ""
    echo "    $0 .venv/"
    echo ""
    exit 1
fi

python3 -m venv "$1"
$1/bin/pip3 install wheel
$1/bin/pip3 install -e qsolverlib/

echo "Virtual environment created at $1"
Enter fullscreen mode Exit fullscreen mode

Add, commit, and push this script:

git add create-virtualenv.sh
git commit -m "create-virtualenv.sh script added."
git push origin master
Enter fullscreen mode Exit fullscreen mode

So now for a new colleague, it will be enough to clone 'qsolver-project' and run this script to start working on the project:

sh create-virtualenv.sh .venv/
source .venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

Step 7. An example: make a change to 'qsolverlib'.

Now let's improve our solve() function to handle the case a=0, i.e. to solve bx+c=0 equation.

Edit qsolverlib/qsolverlib/solve.py and qsolver/app.py:
17-git-diff|656x500
Commit and push the changes with to both 'qsolverlib' and 'qsolver' as a single commit in 'qsolver-project'.

git commit -a -m "solve() improved to handle linear equations as well."
git push origin master
Enter fullscreen mode Exit fullscreen mode

Now you can look at 'qsolverlib' and 'qsolver' histories on Bitbucket: they are updated!
18-qsolverlib-history|690x267 19-qsolver-history|690x265
The synchronization is bi-directional, updates of 'qsolverlib' and 'qsolver' also get into 'qsolver-project' automatically.

Top comments (0)

Read next

abhay_yt_52a8e72b213be229 profile image

Introduction to Docker Container Orchestration: Managing Multi-Container Deployments

Abhay Singh Kathayat -

superbrobenji profile image

How a Netflix Interview question turned into my first NPM package

Benjamin Swanepoel -

abhay_yt_52a8e72b213be229 profile image

Why Python is the Language of Choice for Developers

Abhay Singh Kathayat -

all-time-dev profile image

best os for coding

All-time-dev -