DEV Community

Cover image for Streamlining Project Automation with Makim
Open Science Labs profile image Mfonobong Uyah
Mfonobong Uyah for Open Science Labs

Posted on

Streamlining Project Automation with Makim

#makim #automation #devops #opensource

In software development, where efficiency, consistency, and reliability are paramount, automation tools play a crucial role. Makim, an innovative open-source tool, steps into the spotlight to improve automation workflows. It simplifies script execution, environment management, and task dependencies, positioning itself as a great asset in modern development environments.

Introducing Makim

Makim elevates project automation by offering a structured, yet flexible approach to manage routine tasks, complex task dependencies, and environment configurations. Its design is centered around the .makim.yaml configuration file, allowing developers to orchestrate their workflows with precision and ease. Unlike traditional script execution tools, Makim's Python-based architecture and support for multiple programming languages and shells enhance its versatility and applicability across diverse projects.

Especially suited for DevOps Engineers and Software Developers, Makim eliminates redundancy in automation tasks. Its core functionality extends beyond simple script execution, encompassing:

Argument definition for scripts
Organization of tasks into groups
Advanced dependency management between tasks
Utilization of environment variables and custom variables
Dynamic content generation with Jinja2 templates
Specification of working directories for tasks
Execution flexibility through support for multiple interpreters or shells
Despite its broad capabilities, Makim does not directly support Windows but it can be used on a Windows machine via WSL.

Getting Started with Makim

Installation

Makim can be installed via pip or conda, catering to different setup preferences:

  • To install Makim using pip, run: 
!pip install -q "makim==1.15.2"
Enter fullscreen mode Exit fullscreen mode
  • For those who prefer conda, execute: 
conda install "makim=1.15.2”
Enter fullscreen mode Exit fullscreen mode

Given Makim's active development, pinning to a specific version is recommended to ensure consistency.

For this tutorial, we will disable the output color feature provided by typer, the command-line interface engine used by Makim.

import os
os.environ["NO_COLOR"] = "1"
Enter fullscreen mode Exit fullscreen mode

Configuring .makim.yaml

The .makim.yaml file is the foundation of your Makim configuration. Here's how to start:

  1. Create the .makim.yaml File: Place this file at the root of your project directory.

  2. Define Your Automation Tasks: Configure your tasks, specifying actions, arguments, and dependencies. For example:

%%writefile .makim.yaml
version: 1.0.0
groups:
  clean:
    env-file: .env
    tasks:
      tmp:
        help: Use this task to clean up temporary files
        run: |
          echo "Cleaning up..."
  tests:
    tasks:
     unit:
       help: Build the program
       args:
         clean:
           type: bool
           action: store_true
           help: if not set, the clean dependency will not be triggered.
       dependencies:
         - task: clean.tmp
           if: ${{ args.clean == true }}
       run: |
         echo "Running unit tests..."
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Overwriting .makim.yaml
Enter fullscreen mode Exit fullscreen mode

This setup demonstrates Makim's ability to manage tasks with conditional logic and dependencies.

Exploring Makim's CLI

Makim's CLI provides insights into available commands, arguments, and configurations through the auto-generated help menu:

!makim --help
Enter fullscreen mode Exit fullscreen mode 
OUTPUT


Usage: makim [OPTIONS] COMMAND [ARGS]...                                       

Makim is a tool that helps you to organize and simplify your helper commands.

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --version             -v            Show the version and exit                │
│ --file                        TEXT  Makim config file [default: .makim.yaml] │
│ --dry-run                           Execute the command in dry mode          │
│ --verbose                           Execute the command in verbose mode      │
│ --install-completion                Install completion for the current       │
│                                     shell.                                   │
│ --show-completion                   Show completion for the current shell,   │
│                                     to copy it or customize the              │
│                                     installation.                            │
│ --help                              Show this message and exit.              │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ clean.tmp        Use this task to clean up temporary files                 │
│ tests.unit       Build the program                                           │
╰──────────────────────────────────────────────────────────────────────────────╯

If you have any problem, open an issue at:
https://github.com/osl-incubator/makim
Enter fullscreen mode Exit fullscreen mode

This feature facilitates easy access to Makim's functionalities, enhancing usability and understanding of the tool.

Executing Your First Commands

With your .makim.yaml file set up, you can begin to use makim:

!makim clean.tmp
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Makim file: .makim.yaml
Cleaning up...
Enter fullscreen mode Exit fullscreen mode 
!makim tests.unit
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Makim file: .makim.yaml
Running unit tests...
Enter fullscreen mode Exit fullscreen mode

If you happen to type your command wrongly, Makim will suggest an alternative:

!makim tests.unittest
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Usage: makim [OPTIONS] COMMAND [ARGS]...
Try 'makim --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ No such command 'tests.unittest'.                                            │
╰──────────────────────────────────────────────────────────────────────────────╯
Command tests.unittest not found. Did you mean tests.unit'?
Enter fullscreen mode Exit fullscreen mode

Makim CLI is empowered by Typer, and it allows us to have auto-completion for Makim groups and tasks! If you want to install, you can run the following command:

!makim --install-completion
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

bash completion installed in /home/xmn/.bash_completions/makim.sh
Completion will take effect once you restart the terminal
Enter fullscreen mode Exit fullscreen mode

After this command you will need to restart the terminal in order to use this auto-completion feature.

Advanced Features and Examples

Makim's adaptability is showcased through various features and practical examples:

  • Conditional Dependencies and Arguments: Define complex task dependencies with conditional execution based on passed arguments.
  • Dynamic Configuration with Jinja2: Leverage Jinja2 templates for advanced scripting and dynamic content generation.
  • Environment and Custom Variable Management: Organize and utilize variables effectively across different scopes of your project.
  • Specifying Working Directories: Control the execution context of your tasks by setting working directories. These examples underscore Makim's capability to accommodate intricate automation scenarios, streamlining development workflows.

Exploring Makim Through Examples

Utilizing Various Interpreters

Makim extends its functionality beyond conventional script execution by supporting various interpreters and shell languages, facilitating a versatile development environment. While xonsh is the default interpreter - blending the capabilities of Bash and Python for an enriched command-line experience - Makim's architecture allows for seamless integration with other environments. For developers seeking to leverage this feature, a foundational understanding of xonsh can be beneficial. Comprehensive details and usage guidelines are available in the official xonsh documentation.

This section demonstrates executing straightforward commands across multiple interpreters, showcasing Makim's adaptability to diverse programming contexts.

%%writefile .env
MSG_PREFIX="Running Makim: Hello, World,"
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Overwriting .env
Enter fullscreen mode Exit fullscreen mode 
%%writefile .makim.yaml
version: 1.0
env-file: .env
groups:
  tests:
    tasks:
      node:
        help: Test using nodejs
        shell: node
        run: console.log("${{ env.MSG_PREFIX }} from NodeJS!");
      perl:
        help: Test using perl
        shell: perl
        run: print "${{ env.MSG_PREFIX }} from Perl!\n";

      python:
        help: Test using php
        shell: python
        run: print("${{ env.MSG_PREFIX }} from Python!")

      r:
        help: Test using R
        shell: Rscript
        run: print("${{ env.MSG_PREFIX }} from R!")

      sh:
        help: Test using sh
        shell: sh
        run: echo "${{ env.MSG_PREFIX }} from sh!"

      run-all:
        help: Run tests for all the other tasks
        dependencies:
          - task: node
          - task: perl
          - task: python
          - task: r
          - task: sh
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Overwriting .makim.yaml
Enter fullscreen mode Exit fullscreen mode 
!makim --help
Enter fullscreen mode Exit fullscreen mode 
OUTPUT


 Usage: makim [OPTIONS] COMMAND [ARGS]...                                       

Makim is a tool that helps you to organize and simplify your helper commands.

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --version             -v            Show the version and exit                │
│ --file                        TEXT  Makim config file [default: .makim.yaml] │
│ --dry-run                           Execute the command in dry mode          │
│ --verbose                           Execute the command in verbose mode      │
│ --install-completion                Install completion for the current       │
│                                     shell.                                   │
│ --show-completion                   Show completion for the current shell,   │
│                                     to copy it or customize the              │
│                                     installation.                            │
│ --help                              Show this message and exit.              │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ tests.node             Test using nodejs                                     │
│ tests.perl             Test using perl                                       │
│ tests.python           Test using php                                        │
│ tests.r                Test using R                                          │
│ tests.run-all          Run tests for all the other tasks                   │
│ tests.sh               Test using sh                                         │
╰──────────────────────────────────────────────────────────────────────────────╯

If you have any problem, open an issue at:
https://github.com/osl-incubator/makim
Enter fullscreen mode Exit fullscreen mode

Prior to executing these tasks, it is necessary to install the required dependencies:

!mamba install -q -y perl nodejs r-base sh
Enter fullscreen mode Exit fullscreen mode

Proceed to execute all defined tasks by invoking the run-all task, which encapsulates all other tasks as its dependencies for a sequential execution process:

!makim tests.run-all
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Makim file: .makim.yaml
Running Makim: Hello, World, from NodeJS!
(node:1634785) Warning: The 'NO_COLOR' env is ignored due to the 'FORCE_COLOR' env being set.
(Use `node --trace-warnings ...` to show where the warning was created)
Running Makim: Hello, World, from Perl!
Running Makim: Hello, World, from Python!
[1] "Running Makim: Hello, World, from R!"
Running Makim: Hello, World, from sh!
Enter fullscreen mode Exit fullscreen mode

In scenarios where your chosen interpreter supports debugging - such as Python or Xonsh through the use of breakpoint() - you can introduce a breakpoint within your code. This enables the debugging of your Makim task, allowing for an interactive examination of the execution flow and variable states.

Using Variables (vars)

Makim facilitates the definition of variables within the .makim.yaml configuration, supporting all the YAML data types, including strings, lists, and dictionaries. This feature enhances script configurability and reusability across different tasks and environments.

Consider reviewing the provided example to understand how to effectively leverage variables in your Makim configurations:

%%writefile .makim.yaml
version: 1.0

vars:
  project-name: "my-project"
  dependencies:
    "dep1": "v1"
    "dep2": "v1.1"
    "dep3": "v2.3"
  authorized-users:
    - admin1
    - admin2
    - admin3

groups:
  staging:
    vars:
      env-name: "staging"
      staging-dependencies:
        "dep4": "v4.3"
        "dep5": "v1.1.1"
      staging-authorized-users:
        - staging1
        - staging2
        - staging3
    tasks:
      create-users:
        help: Create users for staging, this example uses jinja2 for loop.
        # each task can also specify their `vars`, but it will not be used in this example
        run: |
          def create_user(username):
              print(f">>> creating user: {username} ... DONE!")

          print("create admin users:")
          {% for user in vars.authorized_users %}
          create_user("${{ user }}")
          {% endfor %}

          print("\ncreate staging users:")
          {% for user in vars.staging_authorized_users %}
          create_user("${{ user }}")
          {% endfor %}

      install:
        help: install deps for staging using native xonsh `for` loop (it could work with Python as well)
        # each task can also specify their `vars`, but it will not be used in this example
        run: |
          def install(package, version):
              print(f">>> installing: {package}@{version} ... DONE!")

          print("install global dependencies:")
          for package, version in ${{ vars.dependencies | safe }}.items():
              install(package, version)

          print("\ninstall staging dependencies:")
          for package, version in ${{ vars.staging_dependencies | safe }}.items():
              install(package, version)
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Overwriting .makim.yaml
Enter fullscreen mode Exit fullscreen mode

Now, let's proceed to create users within the staging environment:

!makim staging.create-users
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Makim file: .makim.yaml
create admin users:
>>> creating user: admin1 ... DONE!
>>> creating user: admin2 ... DONE!
>>> creating user: admin3 ... DONE!

create staging users:
>>> creating user: staging1 ... DONE!
>>> creating user: staging2 ... DONE!
>>> creating user: staging3 ... DONE!
Enter fullscreen mode Exit fullscreen mode

Last but not least, let's run the install task:

!makim staging.install
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Makim file: .makim.yaml
install global dependencies:
>>> installing: dep1@v1 ... DONE!
>>> installing: dep2@v1.1 ... DONE!
>>> installing: dep3@v2.3 ... DONE!

install staging dependencies:
>>> installing: dep4@v4.3 ... DONE!
>>> installing: dep5@v1.1.1 ... DONE!
Enter fullscreen mode Exit fullscreen mode

Defining Arguments

Makim enhances script flexibility by allowing the use of arguments. It enables not only the definition of arguments for tasks but also the passing of arguments to dependencies and the specification of conditions for those dependencies.

Explore this functionality through this example:

%%writefile .makim.yaml
version: 1.0.0
groups:
  print:
    env-file: .env
    tasks:
      name:
        help: Print given name
        args:
          name:
            type: str
            required: true
        run: print("${{ args.name }}")
      list:
        help: Build the program
        args:
          i-am-sure:
            type: bool
        dependencies:
          - task: print.name
            if: ${{ args.i_am_sure == true }}
            args:
              name: Mary
          - task: print.name
            if: ${{ args.i_am_sure == true }}
            args:
              name: John
          - task: print.name
            if: ${{ args.i_am_sure == true }}
            args:
              name: Ellen
          - task: print.name
            if: ${{ args.i_am_sure == true }}
            args:
              name: Marc
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Overwriting .makim.yaml
Enter fullscreen mode Exit fullscreen mode 
!makim print.list
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Makim file: .makim.yaml
Enter fullscreen mode Exit fullscreen mode 
!makim print.list --i-am-sure
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Makim file: .makim.yaml
Mary
John
Ellen
Marc
Enter fullscreen mode Exit fullscreen mode

Utilizing Environment Variables

The previous sections demonstrated the use of environment variables. Here, we'll delve into their application in more detail.

Makim permits the incorporation of environment variables from .env files or directly within the .makim.yaml file, applicable at global, group, and task levels.

Examine an example to understand the implementation:

%%writefile .env
ENV=dev
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Overwriting .env
Enter fullscreen mode Exit fullscreen mode 
%%writefile .makim.yaml
version: 1.0
env-file: .env
env:
  GLOBAL_VAR: "1"
groups:
  global-scope:
    env:
      GROUP_VAR: "2"
    tasks:
      test-var-env-file:
        help: Test env variable defined in the global scope from env-file
        run: |
          import os
          assert str(os.getenv("ENV")) == "dev"

      test-var-env:
        help: Test env variable defined in the global scope in `env` section
        env:
          TASK_VAR: "3"
        run: |
          import os
          # you can get an environment variable directly with xonsh/python
          assert str(os.getenv("GLOBAL_VAR")) == "1"
          # or you can get an environment variable using jinja2 tag
          assert "${{ env.GROUP_VAR }}" == "2"
          assert "${{ env.get("TASK_VAR") }}" == "3"
          assert "${{ env.get("UNKNOWN_TASK_VAR", "4") }}" == "4"
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Overwriting .makim.yaml
Enter fullscreen mode Exit fullscreen mode 
!makim global-scope.test-var-env-file
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Makim file: .makim.yaml
Enter fullscreen mode Exit fullscreen mode 
!makim global-scope.test-var-env
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Makim file: .makim.yaml
Enter fullscreen mode Exit fullscreen mode

Specifying the Working Directory

Makim provides the capability to set a specific working directory for tasks at any scope: global, group, or task.

Review a straightforward example to learn how to apply this feature:

`%%writefile .makim.yaml
version: 1.0
working-directory: "/tmp"

groups:
  check-wd:
    tasks:
      is-tmp:
        help: Test if working directory is `tmp`
        run: |
          import os
          print(os.getcwd())
          assert os.getcwd() == "/tmp"
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Overwriting .makim.yaml
Enter fullscreen mode Exit fullscreen mode 
!makim check-wd.is-tmp
Enter fullscreen mode Exit fullscreen mode 
OUTPUT

Makim file: .makim.yaml
/tmp
Enter fullscreen mode Exit fullscreen mode

This tutorial concludes with a showcase of Makim's key features. While this overview covers the essentials, diving deeper into Makim will reveal more advanced and intriguing ways to leverage its capabilities.

Contributing to Makim

Makim's growth is propelled by its community. Contributions, whether through code, documentation, or feedback, are welcome. Explore the GitHub repository and consider contributing to foster Makim's development.

Tools Similar to Makim

Make:

Make decides which part of a large scale program needs to be recompiled. It is often used in C and C++ languages.

Taskfile:

Taskfile is described as a task runner/build tool that aims to be simpler and easier to use. It is often used in the Go language.

Nox:

Nox is a command line tool that automates testing in multiple Python environments. It uses a standard Python file for configuration.

Ansible:

Ansible is an IT automation engine that automates provisioning, configuration, management, app deployment, orchestration, and many other IT processes.

Conclusion

Makim stands out as a transformative tool in project automation, bridging the gap between simplicity and complexity. Its comprehensive feature set, coupled with the flexibility of its configuration, makes Makim a quintessential tool for developers and DevOps engineers alike. As you incorporate Makim into your projects, its impact on enhancing productivity and consistency will become evident, marking it as an indispensable part of your development toolkit.

Dive deeper into Makim's functionalities by visiting the official documentation. Try it and let us know your thoughts about it!

Top comments (0)

Read next

abhay_yt_52a8e72b213be229 profile image

Understanding Docker Images: Building, Managing, and Optimizing Containers

Abhay Singh Kathayat -

s3cloudhub profile image

Connecting a Virtual Machine to Windows OS: A Step-by-Step Guide

S3CloudHub -

philipfong profile image

Sharding Jest tests. Harder than it should be?

Phil -

samchon profile image

I made OpenAPI and LLM schema definitions

Jeongho Nam -