DEV Community

Cover image for From Zero to Green Build: How I Built a Jenkins + DevPod CI Pipeline with Mise (+ 10 Real Errors That Taught Me Everything)
Alan Varghese
Alan Varghese

Posted on

From Zero to Green Build: How I Built a Jenkins + DevPod CI Pipeline with Mise (+ 10 Real Errors That Taught Me Everything)

#devops #cicd #jenkins #linux

A complete hands-on guide to building an isolated, reproducible CI pipeline with Jenkins Controller, SSH-connected DevPod agents, and Mise runtime management.

Continuous Integration (CI) setups frequently suffer from a classic problem: the "Works on My Machine" dilemma. Local development happens in nicely customized containers or modern toolchains, while CI build servers either drift into giant "snowflakes" full of globally installed packages, or run bloated custom VMs that take forever to configure and maintain.

In this project, I set out to build a modern, isolated, and fully reproducible CI lab environment:

  • Jenkins Controller handles scheduling and orchestration.
  • DevPod (Containers via SSH) provides isolated, reproducible build agents.
  • Mise acts as a declarative polyglot toolchain manager (mise.toml) to deterministically manage runtimes (Python, Java 21, Node.js, and Jenkins CLI) without polluting global paths.

Along the way, things did not work on the first try. I hit subtle shell differences, monorepo path traps, SSH agent quirks, and environment activation edge cases.

Here is the complete walkthrough of how the architecture works, how every file is configured, and the 10 real-world errors that taught me foundational DevOps lessons.

🏗️ Architecture & High-Level Flow

The goal was simple: separate the orchestrator from the execution environment. The Jenkins Controller should never execute heavy compilation or test tasks directly; it should delegate build execution to an ephemeral DevPod container agent over SSH.

flowchart TD
    subgraph GitHub["GitHub Repository (alanvarghese-dev/lab)"]
        Repo["Source Code & Jenkinsfile"]
    end

    subgraph Jenkins_Master["Jenkins Controller"]
        JC["Jenkins Controller\n(Scheduler & Orchestrator)"]
    end

    subgraph DevPod_Agent["DevPod Build Agent (Container)"]
        Agent["Jenkins Agent Node\n(Label: 'devpod')"]
        Mise["Mise Runtime Manager\n(mise.toml)"]
        Python["Python Runtime\n(Latest via Mise)"]
        Pytest["pytest\n(Test Suite Runner)"]
        App["Application Build\n(app.py / test_app.py)"]
    end

    Repo -->|SCM Trigger / Poll| JC
    JC -->|SSH Connection| Agent
    Agent --> Mise
    Mise --> Python
    Python --> Pytest
    Pytest --> App

Complete End-to-End Flow 

GitHub (alanvarghese-dev/lab/jenkins-pipeline)
   ↓
Jenkins Controller (Orchestrates Job & evaluates Jenkinsfile)
   ↓ [SSH]
DevPod Jenkins Agent (Container with label: 'devpod')
   ↓
Mise (Resolves toolchain from mise.toml)
   ↓
Python & pip (Installs dependencies from requirements.txt)
   ↓
pytest (Executes unit test suite)
   ↓
Build Status: SUCCESS ✅
Enter fullscreen mode Exit fullscreen mode

🛠️ The Tech Stack

Component Technology Purpose
CI Orchestrator Jenkins Job orchestration, pipeline stage execution, reporting
Build Agent DevPod (Docker) Isolated, disposable containerized build execution environment
Base Image mcr.microsoft.com/devcontainers/base:ubuntu-24.04 Clean, standardized Linux environment
Tool Version Manager Mise Polyglot runtime versioning (Python, Java 21, Node.js, Jenkins CLI)
App Runtime & Tests Python 3 + pytest Application code (app.py) & unit tests (test_app.py)
Transport SSH Controller-to-Agent communication channel

📁 Repository Structure

The project lives in a monorepo setup (alanvarghese-dev/lab) designed to house multiple DevOps learning projects side by side:

lab/
└── jenkins-pipeline/
    ├── .devcontainer.json      # Devcontainer build definition
    ├── Dockerfile              # DevPod agent image with Mise
    ├── Jenkinsfile             # Declarative Jenkins CI pipeline
    ├── mise.toml               # Tool versions (Java, Python, Node, CLI)
    ├── app.py                  # Sample Python application
    ├── test_app.py             # Pytest test suite
    ├── requirements.txt        # Python dependencies (pytest)
    └── README.md               # Project documentation
Enter fullscreen mode Exit fullscreen mode

⚙️ Step-by-Step Configuration

1. Devcontainer Agent Image (Dockerfile & .devcontainer.json)

To build our DevPod agent with Mise pre-baked, we copy the binary directly from the official Mise multi-stage Docker image:

# Dockerfile
FROM mcr.microsoft.com/devcontainers/base:ubuntu-24.04

COPY --from=jdxcode/mise /usr/local/bin/mise /usr/local/bin/

# Activate mise in interactive shells
RUN echo 'eval "$(mise activate bash)"' >> /home/vscode/.bashrc && \
    echo 'eval "$(mise activate zsh)"' >> /home/vscode/.zshrc
Enter fullscreen mode Exit fullscreen mode

And configure the .devcontainer.json:

{
  "build": {
    "context": "..",
    "dockerfile": "Dockerfile"
  }
}
Enter fullscreen mode Exit fullscreen mode

2. Declarative Toolchain (mise.toml)

Instead of installing packages globally via apt-get or installing random pythons across system directories, we declare all required tooling in mise.toml:

[tools]
"aqua:jenkins-zh/jenkins-cli" = "latest"
python = "latest"
java = "21"
node = "latest"
Enter fullscreen mode Exit fullscreen mode

Because this file is checked into version control, any developer opening the container with DevPod or any CI agent running Jenkins gets the exact same runtime versions.

3. The Application & Pytest Suite

Keep the application minimal so you can focus 100% on pipeline predictability:

# app.py
def hello():
    return "Hello from Jenkins + Devpod!"

if __name__ == "__main__":
    print(hello())
Enter fullscreen mode Exit fullscreen mode 
# test_app.py
from app import hello

def test_hello():
    assert hello() == "Hello from Jenkins + Devpod!"
Enter fullscreen mode Exit fullscreen mode 
# requirements.txt
pytest
Enter fullscreen mode Exit fullscreen mode

4. The Declarative Jenkins Pipeline (Jenkinsfile)

Here is the complete Jenkinsfile. Notice two critical patterns:

  1. dir('jenkins-pipeline'): Because this is a monorepo, we explicitly change into the project directory so Mise can find mise.toml.
  2. mise exec python -- ...: We avoid brittle shell activation scripts and let Mise directly wrap command execution. 
pipeline {
    agent {
        label 'devpod'
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Setup Python') {
            steps {
                dir('jenkins-pipeline') {
                    sh '''
                        set -e
                        echo "Project directory:" && pwd
                        echo "Mise configuration:" && ls -la mise.toml
                        mise install
                        echo "Python version:"
                        mise exec python -- python --version
                    '''
                }
            }
        }

        stage('Install Dependencies') {
            steps {
                dir('jenkins-pipeline') {
                    sh '''
                        set -e
                        mise exec python -- python -m pip install -r requirements.txt
                    '''
                }
            }
        }

        stage('Test') {
            steps {
                dir('jenkins-pipeline') {
                    sh '''
                        set -e
                        mise exec python -- python -m pytest
                    '''
                }
            }
        }
    }

    post {
        success { echo 'Python CI build successful! 🚀' }
        failure { echo 'Python CI build failed! ❌' }
    }
}
Enter fullscreen mode Exit fullscreen mode

💥 10 Real Errors Encountered & What They Taught Me

Building CI pipelines is 10% writing code and 90% debugging subtle environment mismatches. Here are the 10 errors I hit during this lab and the lessons behind them:

1. DevPod Container Created, but Features Failed

  • Symptom: The container was created in Docker, but DevPod reported connection and tunneling errors.
  • Lesson: Container creation and DevPod connectivity are distinct stages. Always ensure: Container Running ➔ DevPod SSH Access ➔ Shell Available before troubleshooting build tools.

2. "No SSH session / exited"

  • Symptom: DevPod SSH tunneling abruptly dropped.
  • Lesson: This was an infrastructure/connection issue, not a Jenkins or Python failure. Once devpod ssh succeeded reliably from the terminal, the underlying transport was proven stable.

3. Dotfiles Were Not Applied

  • Symptom: The agent container spun up without customized shell dotfiles.
  • Lesson: DevPod’s automated dotfiles mechanism requires setup scripts to be located in conventional locations.

4. "Setup not found"

  • Symptom: DevPod searched for bootstrap files (install.sh, bootstrap.sh, setup.sh) and couldn't find them.
  • Lesson: Automation tools depend on strict conventions. Always verify standard entry points when integrating dotfiles repositories into devcontainers.

5. GitHub SSH Authentication Confusion

  • Symptom: Output showed: Hi username! You've successfully authenticated, but GitHub does not provide shell access.
  • Lesson: This is actually a success message from GitHub confirming that your SSH public key is valid. GitHub explicitly denies interactive PTY shells, so don't mistake this for an authentication failure!

6. java: command not found (The Non-Interactive Shell Trap)

  • Symptom: Java 21 was installed inside DevPod, but Jenkins reported java: command not found.
  • Lesson: Installed on disk ≠ Available to the Jenkins process. When you log in via devpod ssh, your .zshrc / .bashrc loads your PATH. Jenkins launches a non-interactive /bin/sh shell that skips interactive rc files. 
Interactive Login:   .zshrc / .bashrc loaded ➔ PATH configured ➔ Tools work
Jenkins Non-Login:   /bin/sh launched        ➔ Minimal PATH    ➔ Command not found
Enter fullscreen mode Exit fullscreen mode

7. "Python is installed but not activated"

  • Symptom: Mise reported that Python was installed on the machine, but not active.
  • Lesson: Mise relies on discovering a valid mise.toml in the current directory or parent tree. If you are in a directory without a config, Mise will not activate the tool by default.

8. Syntax error: "(" unexpected on eval "$(mise activate bash)"

  • Symptom: Attempting to evaluate Mise activation inside a Jenkins sh step blew up with a syntax error.
  • Lesson: Jenkins defaults to /bin/sh (often Debian/Ubuntu dash), which does not support certain Bash/Zsh process substitution or subshell syntax.
  • Solution: Instead of hacking shell startup scripts, use mise exec <tool> -- <command> directly in pipeline steps!

9. Mise Could Not Find Python in Monorepo Root

  • Symptom: mise exec -- python --version failed in Jenkins.
  • Root Cause: Jenkins checked out the entire monorepo (lab/), and executed commands at the root. The mise.toml was inside lab/jenkins-pipeline/.
  • Fix: Use Jenkins' dir('jenkins-pipeline') { ... } block so the working directory matches where mise.toml lives.

10. Git "Does not have a commit checked out"

  • Symptom: Git commands failed unexpectedly inside the agent.
  • Root Cause: Commands ran from /workspaces instead of /workspaces/jenkins-pipeline.
  • Lesson: Always verify your working directory before running Git operations: pwd && git status.

🔍 The 5-Step CI Troubleshooting Checklist

Whenever your Jenkins build fails with a "command not found" or tool version issue, run through this checklist before touching code:

flowchart TD
    A["1. Is the tool installed?\n(mise ls)"] -->|Yes| B["2. Where is the binary?\n(mise which <tool>)"]
    B --> C["3. Is mise.toml in the directory?\n(ls -la mise.toml)"]
    C --> D["4. Is Jenkins in the right folder?\n(pwd)"]
    D --> E["5. Can Mise execute it?\n(mise exec <tool> -- <cmd>)"]
    A -->|No| F["Run: mise install"]
  1. Is the tool installed? 
   mise ls
Enter fullscreen mode Exit fullscreen mode
  1. Where is the binary located? 
   mise which python
   mise which java
Enter fullscreen mode Exit fullscreen mode
  1. Is mise.toml present in the current folder? 
   ls -la mise.toml
Enter fullscreen mode Exit fullscreen mode
  1. What is Jenkins' current working directory? 
   pwd
Enter fullscreen mode Exit fullscreen mode
  1. Can Mise execute the tool directly? 
   mise exec python -- python --version
Enter fullscreen mode Exit fullscreen mode

💡 Key Architectural Takeaways

  1. Infrastructure First, Application Second: Get your container running, SSH passing, and agent labeled before spending time configuring application test suites.
  2. Separate Orchestration from Execution: Keep the Jenkins Controller lean. Run builds inside disposable agents.
  3. Embrace Deterministic Toolchains: Using mise.toml ensures that both local devcontainer sessions and remote CI builds use the exact same runtime versions.
  4. Never Rely on Interactive Shell Environments in CI: Avoid complex .bashrc / .zshrc sourcing in CI. Use explicit CLI wrappers like mise exec to execute binaries cleanly.

🗺️ What's Next on the DevOps Roadmap?

This Jenkins + DevPod CI lab is the foundation of my broader homelab DevOps monorepo (alanvarghese-dev/lab):

lab/
├── jenkins-pipeline/    # ✅ Jenkins + DevPod SSH CI Pipeline (Completed!)
├── docker/              # 🔜 Multi-stage builds & container optimization
├── kubernetes/          # 🔜 K8s deployment manifests & Helm charts
├── terraform/           # 🔜 Infrastructure as Code (IaC)
└── gitops/              # 🔜 GitOps automated delivery with ArgoCD / Flux
Enter fullscreen mode Exit fullscreen mode

💬 Discussion

Have you tried using DevPod or containerized SSH agents with Jenkins? How do you handle tool versioning in your CI runners? Let me know in the comments below!

[github]

[linkedin]

Top comments (0)