DEV Community

Sp1983
Sp1983

Posted on

DSCI pipeline to build c++ project using cmake

#ai #dsci #cpp #automation

📁 Repository layout 

.
├── .dsci/
│   ├── jobs.yaml                     # pipeline definition
│   └── build_job/                    # ── Job 1 – Build & package
│       ├── config.yaml               # default job parameters
│       ├── job.bash                  # job script – orchestrates tasks
│       └── tasks/
│           ├── configure/
│           │   └── task.bash        # cmake configuration
│           ├── build/
│           │   └── task.bash        # compile
│           ├── test/
│           │   └── task.bash        # run CTest
│           └── package/
│               └── task.bash        # create artifact & export state
│
└── deploy_job/                       # ── Job 2 – Deploy (demo)
    ├── job.bash                      # job script – runs the deploy task
    └── tasks/
        └── deploy/
            └── task.py               # Python – reads state & “deploys”
Enter fullscreen mode Exit fullscreen mode

All scripts are executable (chmod +x …).

The ~/artifacts/ directory is automatically mounted by DSCI and is the place where jobs share files (artifacts) and state between each other.

🗂️ jobs.yaml – the only YAML the engine sees 

# .dsci/jobs.yaml
jobs:
  - id: build_job
    path: .dsci/build_job/
    # you can override any default parameter here, e.g.
    # params:
    #   build_type: Debug

  - id: deploy_job
    path: deploy_job/
Enter fullscreen mode Exit fullscreen mode

No matrix, no extra options – just a flat list of jobs.

🛠️ Job 1 – build_job (Bash job)

config.yaml – default parameters (can be overridden from jobs.yaml) 

# .dsci/build_job/config.yaml
source_dir: src               # where CMakeLists.txt lives
build_dir: build              # where CMake will create the out‑of‑source tree
cmake_generator: "Unix Makefiles"
build_type: Release
Enter fullscreen mode Exit fullscreen mode

job.bash – orchestrates the four tasks 

# .dsci/build_job/job.bash
#!/usr/bin/env bash
set -euo pipefail

# Run tasks in the required order
run_task configure
run_task build
run_task test
run_task package
Enter fullscreen mode Exit fullscreen mode

Tasks (all Bash – simple, clear, no extra SDK imports)

1️⃣ configure/task.bash 

# .dsci/build_job/tasks/configure/task.bash
#!/usr/bin/env bash
set -euo pipefail

# Read job parameters
src_dir=$(config source_dir)
build_dir=$(config build_dir)
generator=$(config cmake_generator)
build_type=$(config build_type)

echo "🔧 Configuring CMake …"
mkdir -p "$HOME/$build_dir"
cd "$HOME/$build_dir"

cmake -G "$generator" -DCMAKE_BUILD_TYPE="$build_type" "$HOME/$src_dir"
Enter fullscreen mode Exit fullscreen mode

2️⃣ build/task.bash 

# .dsci/build_job/tasks/build/task.bash
#!/usr/bin/env bash
set -euo pipefail

build_dir=$(config build_dir)

echo "🚧 Building …"
cd "$HOME/$build_dir"
cmake --build . -- -j$(nproc)
Enter fullscreen mode Exit fullscreen mode

3️⃣ test/task.bash 

# .dsci/build_job/tasks/test/task.bash
#!/usr/bin/env bash
set -euo pipefail

build_dir=$(config build_dir)

echo "✅ Running tests …"
cd "$HOME/$build_dir"
ctest --output-on-failure
Enter fullscreen mode Exit fullscreen mode

4️⃣ package/task.bash 

# .dsci/build_job/tasks/package/task.bash
#!/usr/bin/env bash
set -euo pipefail

build_dir=$(config build_dir)

echo "📦 Packaging artifact …"
cd "$HOME/$build_dir"

# Assume the final binary is called `myapp`
# (adjust the name according to your real target)
BINARY="myapp"

if [[ ! -f "$BINARY" ]]; then
    echo "❌ Expected binary '$BINARY' not found!"
    exit 1
fi

# Create a tar.gz in the shared artifacts directory
ARTIFACT_PATH="$HOME/artifacts/${BINARY}.tar.gz"
tar -czf "$ARTIFACT_PATH" "$BINARY"

echo "🗂️ Artifact stored at $ARTIFACT_PATH"

# Export the artifact location as state so downstream jobs can read it
# (single JSON object – see the “state export pitfalls” rule)
update_state '{ "artifact_path": "'"$ARTIFACT_PATH"'" }'
Enter fullscreen mode Exit fullscreen mode

State export note – we use a single JSON object (update_state '{ "artifact_path": "..."}') so no previous state is lost.

🚀 Job 2 – deploy_job (demo of consuming state with Python)

This job shows how a downstream job can read the state (artifact_path) produced by build_job.

job.bash – only runs the single deploy task 

# deploy_job/job.bash
#!/usr/bin/env bash
set -euo pipefail

run_task deploy
Enter fullscreen mode Exit fullscreen mode

deploy/task.py – reads the shared state and pretends to deploy 

#!/usr/bin/env python3
# deploy_job/tasks/deploy/task.py

# The SDK injects `config()` – it returns a dict that contains the whole
# pipeline configuration, including the `_dsci_` block with states from
# previous jobs.
cfg = config()

# Pull the artifact path that `build_job` stored in its state
artifact_path = cfg['_dsci_']['build_job']['artifact_path']

print(f"🚢 Deploy job – received artifact: {artifact_path}")

# ---- Demo deployment -------------------------------------------------
# In a real pipeline you could, for example, upload the artifact to a
# package repository, copy it to a server, etc.  Here we only print a
# message to keep the example self‑contained.

print("✅ Deployment simulated – nothing really happened.")
Enter fullscreen mode Exit fullscreen mode

No extra config.yaml is required for deploy_job (it uses only the state).

📦 How the pipeline works (high‑level)

  1. build_job

    Configure → Build → Test → Package

    – The package task creates ~/artifacts/myapp.tar.gz and publishes its location via update_state.

  2. deploy_job

    Reads the state entry ['_dsci_']['build_job']['artifact_path'] through config() and pretends to deploy the binary.

All heavy‑lifting (CMake, compilation, testing) is done with plain Bash; the only Python piece demonstrates state sharing between jobs.

🛡️ Why this satisfies the constraints

Constraint How it’s met
Prefer Bash/Python All tasks use Bash; the only Python script shows state usage.
jobs.yaml only lists jobs No extra keys, no matrices, no plugins – just id & path.
Parameters & defaults config.yaml provides defaults; jobs.yaml can override via params: (example commented).
Access parameters Bash tasks use $(config name); Python task uses config().
State export / import package/task.bash exports a single JSON object; deploy/task.py reads it via config()['_dsci_'].
Artifacts sharing Artifact is written to ~/artifacts/ – automatically visible to the next job.
Clear file separation Every file is shown in its own markdown code block with the exact path.
No third‑party actions / complex YAML The YAML is minimal; all logic lives in scripts.
Real solution The scripts would actually configure, build, test, package and “deploy” a CMake‑based C++ project.

🎉 Ready to run

Place the tree exactly as shown, make the *.bash and *.py files executable, and let DSCI execute the pipeline. The build artefact will appear under ~/artifacts/ and the second job will confirm it received the correct path.

Top comments (0)