DEV Community: Thu Kha Kyawe

Day 19: Complete a Production DVC Pipeline with SeaweedFS Remote

Thu Kha Kyawe — Sun, 19 Jul 2026 05:10:53 +0000

Lab Information

Complete the xFusionCorp Industries fraud-detection production DVC pipeline. Three stages are already wired in dvc.yaml, two remain, and the pipeline must finish as a reproducible, SeaweedFS-backed, v1.0-tagged release.

A project exists at /root/code/ml-pipeline/ with Git and DVC initialised. The params.yaml is in place and the .dvc/config is pre-configured to push to the SeaweedFS bucket dvc-storage at http://localhost:8333.

The ingest, validate, and preprocess stages are already declared in dvc.yaml, but one of them is misconfigured and prevents dvc repro from completing — run dvc repro to see it fail. The two scripts for the remaining stages are pre-staged at /root/code/ml-pipeline/scripts-staging/train.py and scripts-staging/evaluate.py, and belong in scripts/.

Acceptance criteria:

The misconfigured existing stage is corrected so dvc repro can complete.
Two further stages are declared in dvc.yaml:
    train – Depends on the preprocessed dataset and scripts/train.py; reads n_estimators, max_depth, test_size, and random_seed from params.yaml; outputs models/model.pkl and data/processed/test_split.csv; declares metrics.json as a DVC metric with cache: false.
    evaluate – Depends on models/model.pkl, data/processed/test_split.csv, and scripts/evaluate.py; outputs reports/evaluation.json declared with cache: false.
The full pipeline has been reproduced, the cache pushed to the SeaweedFS remote, and the current state tagged v1.0.
Every change is committed to Git so the release is fully captured.

Open the SeaweedFS Filer button at the top of the lab and navigate to /buckets/dvc-storage/ to confirm that the bucket holds the pushed artefacts under the files/md5/... layout.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

Step 1: Enter the project

cd /root/code/ml-pipeline

Step 2: Run the pipeline once

The lab tells you one stage is intentionally broken.

Run:

dvc repro

Don't fix anything yet.

Read the error carefully.

Examples:

Missing dependency
Output mismatch
Script not found
Wrong filename

The error will tell you exactly which existing stage is misconfigured.

Step 3: Inspect the existing pipeline

cat dvc.yaml

Output:

root@controlplane ml-pipeline on  main [?] ✖ cat dvc.yaml
stages:
  ingest:
    cmd: python3 scripts/ingest.py
    deps:
      - scripts/ingest.py
      - data/raw/data.csv

  validate:
    cmd: python3 scripts/validate.py
    deps:
      - data/raw/data.csv
      - scripts/validate.py
    outs:
      - reports/validation.json:
          cache: false

  preprocess:
    cmd: python3 scripts/preprocess.py
    deps:
      - data/raw/data.csv
      - scripts/preprocess.py
    outs:
      - data/processed/cleaned.csv

Step 4: Check what preprocess.py actually created

Run:

ls -l data/processed

Output:

root@controlplane ml-pipeline on  main [?] ➜  ls -l data/processed
total 4
-rw-r--r-- 1 root root 702 Jul 19 00:15 clean.csv

Step 5: Fix the preprocess output path.

Open dvc.yaml in the VS Code editor. In the preprocess stage, change the last line from:

  - data/processed/cleaned.csv

to:

  - data/processed/clean.csv

Save the file (Ctrl+S).

Step 6: Copy the remaining scripts into scripts/.

cp scripts-staging/train.py    scripts/train.py
cp scripts-staging/evaluate.py scripts/evaluate.py

Add the train and evaluate stages to dvc.yaml.

Step 7: Declare the train and evaluate stages with dvc stage add.

Train stage:

dvc stage add -n train \
  -d data/processed/clean.csv -d scripts/train.py \
  -p n_estimators,max_depth,test_size,random_seed \
  -o models/model.pkl -o data/processed/test_split.csv \
  -M metrics.json \
  python3 scripts/train.py

Evaluate stage:

dvc stage add -n evaluate \
  -d models/model.pkl -d data/processed/test_split.csv -d scripts/evaluate.py \
  -O reports/evaluation.json \
  python3 scripts/evaluate.py

Run:

dvc repro

Push to SeaweedFS:

dvc push

Commit everything:

git add .
git commit -m "Complete production DVC pipeline"

Tag the release:

git tag v1.0

Step 8: Verify the SeaweedFS bucket

🧠 Part 2: Simple Step-by-Step Explanation (Beginner Friendly)

In this lab, the ingest and validate stages completed successfully, but the pipeline stopped during the preprocess stage.

When you ran:

dvc repro

DVC executed the preprocess.py script successfully, and the script printed:

Preprocessed: 20 clean rows

However, DVC then displayed the following error:

ERROR: failed to reproduce 'preprocess':
output 'data/processed/cleaned.csv' does not exist

This tells us that the Python script itself did not fail. Instead, DVC could not find the output file that was defined in dvc.yaml.

To investigate, we opened the pipeline configuration:

cat dvc.yaml

Inside the preprocess stage, the output was configured as:

outs:

data/processed/cleaned.csv

Next, we checked the actual contents of the data/processed directory:

ls -l data/processed

The output showed:

clean.csv

This revealed the problem: the script created clean.csv, but dvc.yaml expected cleaned.csv.

Because DVC verifies that every declared output exists after a stage finishes, it stopped the pipeline when it couldn't find cleaned.csv.

To fix the issue, we updated the outs section of the preprocess stage so that it matched the file actually produced by the script:

preprocess:
cmd: python3 scripts/preprocess.py
deps:
- data/raw/data.csv
- scripts/preprocess.py
outs:
- data/processed/clean.csv

After saving the change, running:

dvc repro

again allows DVC to locate the correct output file and continue executing the remaining stages of the pipeline.

Key Takeaway

A very common DVC error is:

output '' does not exist

When you see this message, the first thing to check is whether the output filename in dvc.yaml exactly matches the filename created by the script. Even a small difference—such as clean.csv versus cleaned.csv—is enough to cause the pipeline to fail. Always verify the actual output file (for example, using ls) before updating the pipeline configuration.

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

Day 18: Version Datasets and Models Across Git Branches

Thu Kha Kyawe — Sun, 19 Jul 2026 05:04:56 +0000

Lab Information

The xFusionCorp Industries MLOps team versions datasets and models on separate Git branches so it can reproduce and roll between versions cleanly. Tag the current state as v1.0, create a v2-improved branch built on a newer dataset (which retrains the model), and confirm that switching back restores the original data and model.

A project exists at /root/code/fraud-detection/ with a working DVC pipeline (it processes the data and trains a model) and the baseline data/raw/transactions.csv already tracked.

An improved dataset has been pre-staged at /root/code/fraud-detection/data/raw/transactions_v2.csv and is visible in the file explorer. Do not delete this file.

Acceptance criteria:

On the main branch, the current state is tagged v1.0.
A branch named v2-improved holds the v2 state: the tracked dataset carries the contents of the v2 file (re-tracked with DVC), the pipeline has been re-run so models/model.pkl is retrained and versioned alongside the dataset, and the changes are committed.
Back on the main branch, the v1 dataset and model are restored on disk, matching the hashes recorded by the v1.0 tag.

The DVC extension's DVC TRACKED section in the EXPLORER panel reflects the tracked dataset and model for the branch you currently have checked out. To compare the exact hashes recorded on each branch, use git show <ref>:dvc.lock or dvc status.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

This is one of the most realistic DVC labs. You'll combine Git branches + DVC + pipeline reproducibility.

Step 1: Go to the project

cd /root/code/fraud-detection

Step 2: Make sure everything is committed

git status

Expected:

On branch main
nothing to commit, working tree clean

If not clean, commit your changes before continuing.

Step 3: Tag the current version

The lab requires the current main branch to be tagged as v1.0.

git tag v1.0

Verify:

git tag

Expected:

v1.0

Step 4: Create the new branch

git checkout -b v2-improved

Verify:

git branch

Expected:

v2-improved main

Step 5: Replace the tracked dataset with the new version

The improved dataset already exists:

data/raw/transactions_v2.csv

Replace the original dataset:

cp data/raw/transactions_v2.csv data/raw/transactions.csv

Step 6: Update DVC tracking

Since the tracked file changed, update the DVC metadata:

dvc add data/raw/transactions.csv

This updates:

transactions.csv.dvc
DVC cache

Step 7: Retrain the pipeline

Run:

dvc repro

This should rerun:

process_data
split_data
train

and regenerate:

models/model.pkl
metrics.json
dvc.lock

Step 8: Check Git status

git status

You should see changes similar to:

modified: data/raw/transactions.csv.dvc
modified: dvc.lock
modified: metrics.json
modified: models/model.pkl.dvc (if model is DVC tracked)

Note: The exact files may differ depending on how the lab repository is set up.

Step 9: Commit the v2 state

Stage everything:

git add .

Commit:

git commit -m "Update dataset to v2 and retrain model"

Step 10: Return to main

git checkout main

Step 11: Restore the DVC-tracked files

After switching branches, restore the correct dataset and model:

dvc checkout

Step 12: Verify

Check the dataset:

md5sum data/raw/transactions.csv

Check the model:

ls -l models/model.pkl

Verify DVC:

dvc status

Expected:

Data and pipelines are up to date.

🧠 Part 2: Simple Step-by-Step Explanation (Beginner Friendly)

Why do we create a Git tag?

The command:

git tag v1.0

creates a permanent label for the current project state.

Think of it as taking a snapshot:

main
│
├── v1.0 ← Snapshot

You can always return to exactly this version later.

Why create a new branch?

Instead of changing the main branch directly, we create:

v2-improved

This allows us to safely experiment with a new dataset.

The structure becomes:

main
│
└── v2-improved

Why run dvc add again?

The file name stays the same:

transactions.csv

but its contents change because you copied in the improved dataset.

DVC tracks file contents (using hashes), not just filenames.

Running:

dvc add data/raw/transactions.csv

updates the .dvc pointer to reference the new dataset version.

Why rerun the pipeline?

The dataset changed.

That means:

transactions.csv
│
▼
process_data
▼
split_data
▼
train
▼
model.pkl

Every downstream stage depends on the dataset, so DVC reruns the pipeline to produce a model trained on the new data.

Why use dvc checkout?

When you switch back to:

git checkout main

Git restores:

dvc.yaml
.dvc files
dvc.lock

However, Git does not restore the actual dataset or model, because DVC manages those files.

Running:

dvc checkout

reads the .dvc files and restores the correct versions of:

transactions.csv
model.pkl

from the DVC cache.

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

Day 17: Run and Compare DVC Experiments

Thu Kha Kyawe — Sun, 19 Jul 2026 05:01:39 +0000

Lab Information

The xFusionCorp Industries MLOps team needs every model training run to be reproducible, automatically tracked, and easy to compare so a chosen configuration can be promoted into version control. The fraud-detection pipeline is parameterized by max_depth, currently set shallow enough to underfit. Using DVC experiments, run three tracked experiments over different max_depth values, compare their recorded f1_score on the held-out test set, and promote the best-scoring run so its parameters, metrics, and model become the tracked workspace state.

A project exists at /root/code/fraud-detection/ with a parameterised DVC pipeline already in place. params.yaml declares n_estimators: 100 and max_depth: 4, and the baseline pipeline has been run once. src/models/train.py reads both parameters, trains the model, and evaluates it on the held-out test set, writing the real accuracy and f1_score to metrics.json. Do not modify the Python files.

Acceptance criteria:

Three DVC experiments have been run, each with a different value for max_depth across a reasonable range (for example 2, 6, and 12); each experiment retrains the model and produces a fresh metrics.json.
The experiment with the highest f1_score is applied to the workspace, so its max_depth, metrics.json, and models/model.pkl become the tracked state.

The DVC extension's EXPERIMENTS view (open the DVC panel from the Activity Bar) lists every experiment alongside its parameters and metrics, which is a convenient way to compare runs at a glance.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

This lab introduces DVC Experiments (dvc exp), which let you test different parameter values without committing changes to Git.

Step 1: Move into the project

cd /root/code/fraud-detection

Step 2: Verify the current parameter

cat params.yaml

Expected:

n_estimators: 100
max_depth: 4

Step 3: Run the first experiment (max_depth=2)

dvc exp run -S max_depth=2

This will:

Update max_depth to 2 for the experiment.
Retrain the model.
Generate a new metrics.json.
Save the experiment in DVC (without changing your Git history).

Step 4: Run the second experiment (max_depth=6)

dvc exp run -S max_depth=6

Step 5: Run the third experiment (max_depth=12)

dvc exp run -S max_depth=12

Now you have three experiments, each using a different max_depth.

Step 6: Compare the experiments

Run:

apt update
apt install -y less
dvc exp show

root@controlplane fraud-detection on  main [!] ➜ dvc exp show
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────>
Experiment Created accuracy f1_score n_estimators max_depth data/processed/clean_transactions.csv data/processed/>
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────>
workspace - 0.75 0.6795 100 12 34832f8da536af18bb25236981b31c04 1d9e91ebeec0d17>
main 09:37 AM 0.71 0.5735 100 4 34832f8da536af18bb25236981b31c04 1d9e91ebeec0d17>
├── f75ec1a [lying-zigs] 09:46 AM 0.75 0.6795 100 12 34832f8da536af18bb25236981b31c04 1d9e91ebeec0d17>
├── fad90d2 [erect-flap] 09:45 AM 0.745 0.6483 100 6 34832f8da536af18bb25236981b31c04 1d9e91ebeec0d17>
└── 422a560 [stone-leas] 09:45 AM 0.6 0.2453 100 2 34832f8da536af18bb25236981b31c04 1d9e91ebeec0d17>
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────>

Step 7: Identify the best experiment

Look at the f1_score column.

Find the experiment with the highest f1_score.

For example:

exp-f75ec1a has the highest f1_score of 0.6795.

Step 8: Apply the best experiment

Apply it to your workspace:

dvc exp apply <experiment-name>

Example:

root@controlplane fraud-detection on  main [!] ➜  dvc exp apply f75ec1a
Building workspace index                                                                                             |7.00 [00:00, 1.46kentry/s]
Comparing indexes                                                                                                    |8.00 [00:00, 4.61kentry/s]
Applying changes                                                                                                      |0.00 [00:00,     ?file/s]
Changes for experiment 'f75ec1a' have been applied to your current workspace.

This updates:

params.yaml
metrics.json
models/model.pkl

to match the selected experiment.

Step 9: Verify the applied parameter

cat params.yaml

Example:

root@controlplane fraud-detection on  main [!] ➜  cat params.yaml
n_estimators: 100
max_depth: 12

Step 10: Verify the metrics

dvc metrics show

The displayed metrics should match the winning experiment.

Example:

root@controlplane fraud-detection on  main [!] ➜  dvc metrics show
Path          accuracy    f1_score
metrics.json  0.75        0.6795

Step 11: Verify the model

ls -l models/model.pkl

The model file should have been regenerated for the applied experiment.

🧠 Part 2: Simple Step-by-Step Explanation (Beginner Friendly)

What is a DVC Experiment?

Normally, if you want to test a new model configuration, you would:

Edit params.yaml.
Run dvc repro.
Record the results.
Repeat.

This becomes tedious and makes it hard to compare experiments.

DVC Experiments automate this process.

Why run multiple experiments?

Different values of max_depth can affect model performance.

For example:

max_depth = 2

A very shallow tree may underfit the data.

max_depth = 6

May provide a good balance.

max_depth = 12

A deeper tree may overfit.

The goal is to find the value that gives the best f1_score.

What does dvc exp run do?

When you run:

dvc exp run -S max_depth=6

DVC temporarily changes:

max_depth: 6

runs the pipeline, records the metrics, and saves the results as a named experiment.

Your Git history remains unchanged.

Why use dvc exp show?

Instead of opening each metrics.json manually, DVC displays all experiments together.

Example:

Experiment max_depth accuracy f1_score

exp-A 2 0.88 0.84
exp-B 6 0.94 0.92
exp-C 12 0.93 0.90

This makes it easy to compare different parameter values.

What does dvc exp apply do?

Once you've found the experiment with the highest f1_score, apply it:

dvc exp apply exp-B

This updates your workspace so that it reflects the best experiment:

params.yaml uses the winning max_depth.
metrics.json contains the winning metrics.
models/model.pkl is the model trained with those parameters.

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

Day 16: Track ML Metrics with DVC

Thu Kha Kyawe — Sun, 19 Jul 2026 04:57:28 +0000

Lab Information

After training a model, the xFusionCorp Industries ML team requires DVC to surface model metrics through dvc metrics show. Although the fraud-detection pipeline successfully trains a model and generates a metrics.json file, DVC currently does not recognize this file as a metric. Ensure that the metrics.json file is properly configured to be recognized by DVC.

A project exists at /root/code/fraud-detection/ with a three-stage DVC pipeline (process_data, split_data, train). The train stage runs src/models/train.py, which writes the model to models/model.pkl and metrics to metrics.json. Do not modify the Python files.

Acceptance criteria:

The train stage in dvc.yaml declares metrics.json as a DVC metric output rather than a regular file output, with cache: false so the JSON lives in Git for diff history rather than in the DVC cache.
The pipeline has been reproduced so the metric registration takes effect, and dvc metrics show reports the accuracy and f1_score values from metrics.json.

Tip: once the metric is registered, dvc metrics diff compares its values across Git commits, which is useful when iterating on the model.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

Step 1: Move into the project

cd /root/code/fraud-detection

Step 2: Inspect the current train stage

Open dvc.yaml:

cat dvc.yaml

Notice that metrics.json is missing.

Step 3: Modify the train stage

Add a metrics section.

stages:
  process_data:
    cmd: python3 src/data/process_data.py
    deps:
      - data/raw/transactions.csv
      - src/data/process_data.py
    outs:
      - data/processed/clean_transactions.csv

  split_data:
    cmd: python3 src/data/split_data.py
    deps:
      - data/processed/clean_transactions.csv
      - src/data/split_data.py
    outs:
      - data/processed/train.csv
      - data/processed/test.csv

  train:
    cmd: python3 src/models/train.py
    deps:
      - data/processed/train.csv
      - src/models/train.py
    params:
      - n_estimators
    outs:
      - models/model.pkl
    metrics:
      - metrics.json:
          cache: false

Important: Do not put metrics.json under outs:. The lab specifically requires it to be declared as a metric with cache: false.

Step 4: Save dvc.yaml and add params.yaml if it doesn't exist.

Verify the file:

cat dvc.yaml
cat > params.yaml <<EOF
n_estimators: 100
EOF

Step 5: Reproduce the pipeline

Run:

dvc repro

This regenerates:

models/model.pkl
metrics.json
dvc.lock

Step 6: Verify the metric

Run:

dvc metrics show

Expected output will look similar to:

Path accuracy f1_score
metrics.json 1.0 1.0

The exact values will vary.

Step 7: Verify the metric file

cat metrics.json

Example:

{
"accuracy": 1.0,
"f1_score": 1.0
}

Step 8: Verify dvc.lock

(Optional)

cat dvc.lock

You should see metrics.json recorded under the train stage.

🧠 Part 2: Simple Step-by-Step Explanation (Beginner Friendly)

What is metrics.json?

After training the model, the Python script creates a file called:

metrics.json

It contains information about how well the model performed.

For example:

{
"accuracy": 1.0,
"f1_score": 1.0
}

These numbers help you evaluate the model.

Why doesn't DVC recognize it automatically?

DVC only knows about files that are declared in dvc.yaml.

Currently, DVC knows about:

input files (deps)
output files (outs)
parameters (params)

It doesn't know that metrics.json contains evaluation metrics.

Why use metrics: instead of outs:?

Regular outputs:

outs:

models/model.pkl

are treated as artifacts (files produced by the pipeline).

Metrics are different—they are values you want to compare between experiments.

So we use:

metrics:

metrics.json: cache: false

This tells DVC:

metrics.json contains evaluation metrics.
Keep it in Git (cache: false) instead of storing it in the DVC cache.
Allow commands like dvc metrics show and dvc metrics diff to read it.

Why cache: false?

Normally, DVC stores outputs in its cache.

For metrics, we want the JSON file to stay in the Git repository because:

it's small,
it's easy to compare between commits,
Git can track its history.

That's why the lab requires:

cache: false

What does dvc metrics show do?

After running:

dvc repro

you can execute:

dvc metrics show

Instead of opening metrics.json manually, DVC displays the important values in a table, for example:

Path accuracy f1_score
metrics.json 1.0 1.0

What is dvc metrics diff?

As you experiment with different model parameters (such as n_estimators), the metrics may change.

Running:

dvc metrics diff

compares the metric values across Git commits, making it easy to see whether the new model performed better or worse.

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

Day 15: Parameterize a DVC Pipeline

Thu Kha Kyawe — Sun, 19 Jul 2026 04:54:04 +0000

Lab Information

The xFusionCorp Industries ML team manages model hyperparameters using params.yaml, enabling experiments to be conducted without altering the code. In the fraud-detection project, the train stage retrieves the n_estimators parameter from params.yaml, but this parameter is not declared to DVC, which means that changing its value does not initiate retraining. Integrate the parameter into the pipeline and illustrate the concept of parameter-driven reproducibility.

A project exists at /root/code/fraud-detection/ with a three-stage DVC pipeline (process_data, split_data, train) and a params.yaml declaring n_estimators: 100. src/models/train.py already reads n_estimators from params.yaml. Do not modify the Python files.

The train stage in dvc.yaml has no params: section, so DVC does not track n_estimators — changing it would not re-run the stage. Add a params: entry to the train stage that lists n_estimators (edit dvc.yaml, or use dvc stage add --force … -p n_estimators …).

Run the full pipeline with dvc repro.

Demonstrate parameter-driven retraining: change n_estimators to a different value (for example 200) and run dvc repro again. Only the train stage should re-execute, the new value must be recorded in dvc.lock, and models/model.pkl must be regenerated.

dvc params diff reports changes to the tracked parameter values across Git commits, which is useful when comparing experiments.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

Step 1: Move into the repository

cd /root/code/fraud-detection

Step 2: Inspect the current pipeline

Open dvc.yaml:

cat dvc.yaml

Locate the train stage.

It will look similar to:

stages:
  process_data:
    cmd: python3 src/data/process_data.py
    deps:
      - data/raw/transactions.csv
      - src/data/process_data.py
    outs:
      - data/processed/clean_transactions.csv

  split_data:
    cmd: python3 src/data/split_data.py
    deps:
      - data/processed/clean_transactions.csv
      - src/data/split_data.py
    outs:
      - data/processed/train.csv
      - data/processed/test.csv

  train:
    cmd: python3 src/models/train.py
    deps:
      - data/processed/train.csv
      - src/models/train.py
    outs:
      - models/model.pkl

Step 3: Verify params.yaml

cat params.yaml

Expected:

n_estimators: 100

Step 4: Add the parameter to the train stage

Edit dvc.yaml:

vi dvc.yaml

Add the highlighted section:

train:
  cmd: python3 src/models/train.py

  deps:
    - data/processed/train.csv
    - src/models/train.py

  params:
    - n_estimators

  outs:
    - models/model.pkl

Save the file.

Step 5: Run the pipeline

Execute:

dvc repro

Expected:

root@controlplane fraud-detection on  main ➜  dvc repro
Running stage 'process_data':                                                  
> python3 src/data/process_data.py
Processed 15 rows
Generating lock file 'dvc.lock'                                                
Updating lock file 'dvc.lock'

Running stage 'split_data':                                                    
> python3 src/data/split_data.py
Train: 12 rows, Test: 3 rows
Updating lock file 'dvc.lock'                                                  

Running stage 'train':                                                         
> python3 src/models/train.py
Trained RandomForestClassifier with n_estimators=100
Updating lock file 'dvc.lock'                                                  

To track the changes with git, run:

        git add data/processed/.gitignore dvc.lock models/.gitignore

To enable auto staging, run:

        dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.

Step 6: Verify dvc.lock

Search for the parameter:

grep -A5 params dvc.lock

You should see something like:

root@controlplane fraud-detection on  main [!?] ➜  grep -A5 params dvc.lock
    params:
      params.yaml:
        n_estimators: 100
    outs:
    - path: models/model.pkl
      hash: md5
      md5: ba2187225d8642cbbcd4b127310ced91

This confirms DVC is now tracking the parameter.

Step 7: Change the parameter

Edit:

vi params.yaml

Change:

n_estimators: 100

n_estimators: 200

Save the file.

Step 8: Run the pipeline again

dvc repro

Expected:

root@controlplane fraud-detection on  main [!?] ➜  dvc repro
Stage 'process_data' didn't change, skipping                                   
Stage 'split_data' didn't change, skipping                                     
Running stage 'train':                                                         
> python3 src/models/train.py
Trained RandomForestClassifier with n_estimators=200
Updating lock file 'dvc.lock'                                                  

To track the changes with git, run:

        git add dvc.lock

To enable auto staging, run:

        dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.

Step 9: Verify dvc.lock

grep -A5 params dvc.lock

Expected:

    params:
      params.yaml:
        n_estimators: 200
    outs:
    - path: models/model.pkl
      hash: md5
      md5: c74b4c9de3ba10fc6f8c3789964ec187

Step 11: View parameter differences

Run:

dvc params diff

Expected:

root@controlplane fraud-detection on  main [!?] ➜  dvc params diff
Path         Param         HEAD    workspace
params.yaml  n_estimators  100     200

🧠 Part 2: Simple Step-by-Step Explanation (Beginner Friendly)

What is params.yaml?

Instead of hardcoding values like:

n_estimators = 100

the project stores them in a separate file:

n_estimators: 100

The training script reads this value automatically.

This makes it easy to try different settings without changing the Python code.

Why isn't changing params.yaml enough?

Even though train.py reads params.yaml, DVC doesn't automatically know that the train stage depends on it.

If the params: section is missing from dvc.yaml, DVC thinks:

"Nothing important changed."

So running:

dvc repro

may skip the training stage.

What does the params: section do?

Adding:

params:

n_estimators

tells DVC:

"The train stage depends on the value of n_estimators."

Now, whenever n_estimators changes, DVC knows the model needs to be retrained.

What happens after changing the parameter?

Initially:

n_estimators: 100

After editing:

n_estimators: 200

When you run:

dvc repro

DVC compares the tracked parameter values and notices that n_estimators has changed.

Since only the training configuration changed:

✅ train runs again.
✅ process_data is skipped.
✅ split_data is skipped.

This saves time by only rerunning the stage that is affected.

What is dvc.lock?

After each successful run, DVC records the parameter value used.

Example:

params:
params.yaml:
n_estimators: 200

This allows anyone to reproduce the exact experiment later using the same parameter values.

What does dvc params diff do?

The command:

dvc params diff

shows how parameter values changed between Git commits.

For example:

Path Param Old New
params.yaml n_estimators 100 200

This is useful for comparing different experiments and understanding what changed between model versions.

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

5.Jenkins Scheduled Jobs

Thu Kha Kyawe — Thu, 09 Jul 2026 11:30:00 +0000

Lab Information

The devops team of xFusionCorp Industries is working on to setup centralised logging management system to maintain and analyse server logs easily. Since it will take some time to implement, they wanted to gather some server logs on a regular basis. At least one of the app servers is having issues with the Apache server. The team needs Apache logs so that they can identify and troubleshoot the issues easily if they arise. So they decided to create a Jenkins job to collect logs from the server. Please create/configure a Jenkins job as per details mentioned below:

Click on the Jenkins button on the top bar to access the Jenkins UI. Login using username admin and password Adm!n321

Create a Jenkins jobs named copy-logs.
Configure it to periodically build every 5 minutes to copy the Apache logs (both access_log and error_log) from App Server 1 (stapp01) from the default logs location to location /usr/src/itadmin on the Storage Server.
Build the job at least once so that the logs are copied and can be verified.

Note:

You might need to install some plugins and restart Jenkins. We recommend selecting Restart Jenkins when installation is complete and no jobs are running in the update centre. Refresh the page if the UI gets stuck after a restart.
Define the cron expression as required (e.g. */10 * * * * to run every 10 minutes).
For scenarios that require web UI changes, take screenshots or record your work (e.g. using loom.com) so you can share it for review if the task is marked incomplete.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

Add SSH Keys

Step 1: Open the Lab Terminal

This cannot be done from the Jenkins UI.

Open the Terminal provided by the KodeKloud lab (usually connected to the jump-host).

Step 2: SSH to the Jenkins Server

ssh jenkins@jenkins

Password:

j@rv!s

Step 3: Generate an SSH Key (if one doesn't exist)

Check first:

ls -la ~/.ssh

If you don't see id_rsa and id_rsa.pub, run:

ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa

Step 4: Trust and Copy the Key to stapp01

ssh-copy-id tony@stapp01

Password:

Ir0nM@n

When prompted:

Are you sure you want to continue connecting (yes/no)?

Type:

yes

Step 5: Verify

ssh tony@stapp01 hostname

Expected:

stapp01

without asking for a password.

Step 6: Copy the Key to the Storage Server

From the Jenkins server:

ssh-copy-id natasha@ststor01

Password:

Bl@kW

Again, answer yes if asked about the host key.

Step 7: Verify

ssh natasha@ststor01 hostname

Expected:

ststor01

Step 1: Log in to Jenkins
Click the Jenkins button.
Login using:

Username:

admin

Password:

Adm!n321

Step 2: Create the Jenkins Job
Click New Item
Enter:
copy-logs
Select:
Freestyle project
Click OK

Step 3: Configure Build Trigger

Scroll to Build Triggers

Enable:

✔ Build periodically

Enter exactly:

*/5 * * * *

⚠️ Do not use:

H/5 * * * *

The validator checks the exact cron expression.

Step 4: Configure Build Step

Scroll to:

Build

Click:

Add build step
→ Execute shell

Paste:

mkdir -p /tmp/apache-logs

scp -o StrictHostKeyChecking=no \
tony@stapp01:/var/log/httpd/access_log \
/tmp/apache-logs/

scp -o StrictHostKeyChecking=no \
tony@stapp01:/var/log/httpd/error_log \
/tmp/apache-logs/

scp -o StrictHostKeyChecking=no \
/tmp/apache-logs/* \
natasha@ststor01:/usr/src/itadmin/

rm -rf /tmp/apache-logs

If your app server uses /var/log/apache2/, replace /var/log/httpd/ with /var/log/apache2/ in the script.

Click Save

Step 5: Build the Job

Open:

copy-logs

Click:

Build Now

Wait for the build to complete.

Step 6: Verify Successful Build

Open:

Build History
→ Latest Build
→ Console Output

Verify the last line shows:

Finished: SUCCESS

There should be no SCP or SSH errors.

Step 7: Validate the Lab

Before clicking Validate, verify:

Job name is correct.
Cron schedule is correct.
Build completed successfully.
Both Apache log files were copied to:
/usr/src/itadmin

on ststor01.

🧠 Part 2: Simple Step-by-Step Explanations (Beginner Friendly)

What are we trying to accomplish?

Apache generates log files whenever users access the website or when errors occur.

Instead of manually logging into the server to collect logs, Jenkins will automatically copy them to the storage server every 5 minutes.

Step 1: Create a Temporary Folder
mkdir -p /tmp/apache-logs
What does it do?

Creates a temporary directory on the Jenkins server.

Example:

/tmp/apache-logs

This folder temporarily stores the log files before sending them to the storage server.

Step 2: Copy the Apache Access Log
scp -o StrictHostKeyChecking=no \
tony@stapp01:/var/log/httpd/access_log \
/tmp/apache-logs/
What is access_log?

The Apache access log records every request made to the web server.

Example entries include:

Client IP address
Date and time
Requested URL
HTTP status code (200, 404, 500, etc.)

This helps administrators analyze website traffic and user requests.

Step 3: Copy the Apache Error Log
scp -o StrictHostKeyChecking=no \
tony@stapp01:/var/log/httpd/error_log \
/tmp/apache-logs/
What is error_log?

The Apache error log records problems such as:

Website errors
PHP errors
Missing files
Apache startup failures

These logs are essential for troubleshooting.

Step 4: Copy Logs to the Storage Server
scp -o StrictHostKeyChecking=no \
/tmp/apache-logs/* \
natasha@ststor01:/usr/src/itadmin/
What does this do?

It transfers both log files from the Jenkins server to the Storage Server.

Destination:

/usr/src/itadmin

Using a storage server keeps logs centralized and prevents them from being lost if the application server has issues.

Step 5: Remove Temporary Files
rm -rf /tmp/apache-logs
Why?

Once the logs have been copied successfully, the temporary folder is no longer needed.

Deleting it:

Frees disk space.
Keeps the Jenkins server clean.
Step 6: Automatic Scheduling

The cron expression:

*/5 * * * *

means:

Field Meaning
*/5 Every 5 minutes

Every hour
Every day
Every month
Every weekday

Jenkins automatically runs the job every 5 minutes.

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

4.Jenkins Database Backup Job

Thu Kha Kyawe — Thu, 09 Jul 2026 08:30:00 +0000

Lab Information

There is a requirement to create a Jenkins job to automate the database backup. Below you can find more details to accomplish this task:

Click on the Jenkins button on the top bar to access the Jenkins UI. Login using username admin and password Adm!n321.

Create a Jenkins job named database-backup.

Configure it to take a database dump of the kodekloud_db01 database present on the App server (stapp01) in Stratos Datacenter, the database user is kodekloud_roy and password is asdfgdsd.

The dump should be named in db_$(date +%F).sql format, where date +%F is the current date.

Copy the db_$(date +%F).sql dump to the Storage server (ststor01) under location /home/natasha/db_backups.

Further, schedule this job to run periodically at */10 * * * * (please use this exact schedule format).

Note:

You might need to install some plugins and restart Jenkins service. So, we recommend clicking on Restart Jenkins when installation is complete and no jobs are running on plugin installation/update page i.e update centre. Also, Jenkins UI sometimes gets stuck when Jenkins service restarts in the back end. In this case please make sure to refresh the UI page.

Please make sure to define you cron expression like this */10 * * * * (this is just an example to run job every 10 minutes).

For these kind of scenarios requiring changes to be done in a web UI, please take screenshots so that you can share it with us for review in case your task is marked incomplete. You may also consider using a screen recording software such as loom.com to record and share your work.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

Add SSH Keys

Step 1: Open the Lab Terminal

This cannot be done from the Jenkins UI.

Open the Terminal provided by the KodeKloud lab (usually connected to the jump-host).

Step 2: SSH to the Jenkins Server

ssh jenkins@jenkins

Password:

j@rv!s

Step 3: Generate an SSH Key (if one doesn't exist)

Check first:

ls -la ~/.ssh

If you don't see id_rsa and id_rsa.pub, run:

ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa

Step 4: Trust and Copy the Key to stapp01

ssh-copy-id tony@stapp01

Password:

Ir0nM@n

When prompted:

Are you sure you want to continue connecting (yes/no)?

Type:

yes

Step 5: Verify

ssh tony@stapp01 hostname

Expected:

stapp01

without asking for a password.

Step 6: Copy the Key to the Storage Server

From the Jenkins server:

ssh-copy-id natasha@ststor01

Password:

Bl@kW

Again, answer yes if asked about the host key.

Step 7: Verify

ssh natasha@ststor01 hostname

Expected:

ststor01

Step 1: Log in to Jenkins
Click the Jenkins button.
Login using:
Username: admin
Password: Adm!n321

Step 2: Create the Jenkins Job
Click New Item.
Enter:
database-backup
Select:
Freestyle project
Click OK.

Step 3: Configure the Build Trigger

Scroll to Build Triggers.

Enable:

✔ Build periodically

Enter exactly:

*/10 * * * *

⚠️ Do not use

H/10 * * * *

The validator checks for the exact cron expression.

Step 4: Configure the Build Step

Scroll to Build.

Click:

Add build step
→ Execute shell

Paste the following script:

mkdir -p /tmp/db-backup

ssh tony@stapp01 "mysqldump -u kodekloud_roy -p'asdfgdsd' kodekloud_db01" \
> /tmp/db-backup/db_$(date +%F).sql

scp -o StrictHostKeyChecking=no \
/tmp/db-backup/*.sql \
natasha@ststor01:/home/natasha/db_backups/

rm -rf /tmp/db-backup

Click Save.

Step 5: Build the Job

Open:

database-backup

Click:

Build Now

Wait until the build finishes.

Step 6: Verify the Build

Open:

Build History
→ Latest Build
→ Console Output

The last line should show:

Finished: SUCCESS

There should be no errors related to:

SSH
mysqldump
SCP

Step 7: Validate the Lab

Before clicking Validate, verify:

Job name is correct.
Cron schedule is correct.
Build completed successfully.
SQL file was copied to:
/home/natasha/db_backups

🧠 Part 2: Simple Step-by-Step Explanations (Beginner Friendly)

What are we trying to accomplish?

Instead of manually backing up the database every time, Jenkins will automate the entire backup process.

Every time the job runs, Jenkins will:

Connect to the application server.
Export the database.
Save it as a SQL file.
Copy it to the storage server.
Remove temporary files.
Step 1: Create a Temporary Folder
mkdir -p /tmp/db-backup
What does it do?

Creates a temporary folder on the Jenkins server.

Example:

/tmp/db-backup
Why?

We need a temporary location to store the SQL backup before sending it to the storage server.

The option:

-p

means:

Create the directory only if it doesn't already exist.

Step 2: Create the Database Backup
ssh tony@stapp01 "mysqldump -u kodekloud_roy -p'asdfgdsd' kodekloud_db01" \

/tmp/db-backup/db_$(date +%F).sql
What happens?

Jenkins connects to:

stapp01

and runs:

mysqldump

This exports the entire database.

The output is redirected to a file on the Jenkins server:

db_YYYY-MM-DD.sql

Example:

db_2026-07-06.sql
Understanding the mysqldump command
mysqldump

Creates a backup of a MySQL database.

-u

Specifies the database username.

-p'asdfgdsd'

Specifies the database password.

kodekloud_db01

Is the database name to back up.

Step 3: Copy the Backup
scp -o StrictHostKeyChecking=no \
/tmp/db-backup/*.sql \
natasha@ststor01:/home/natasha/db_backups/
What happens?

The backup file is copied from the Jenkins server to:

ststor01

Destination folder:

/home/natasha/db_backups
What does scp mean?
scp

stands for:

Secure Copy

It copies files securely over SSH.

What does this option do?
-o StrictHostKeyChecking=no

It prevents SSH from asking:

Do you trust this server?

Since Jenkins jobs run automatically, they cannot answer interactive questions.

Step 4: Delete Temporary Files
rm -rf /tmp/db-backup
Why?

The SQL backup has already been copied to the storage server.

Removing the temporary folder:

Saves disk space.
Keeps the Jenkins server clean.
Step 5: Automatic Scheduling

The cron expression:

*/10 * * * *

means:

Field Meaning
*/10 Every 10 minutes

Every hour
Every day
Every month
Every weekday

So Jenkins automatically performs the backup every 10 minutes.

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

3.Jenkins Workspaces

Thu Kha Kyawe — Thu, 09 Jul 2026 05:30:00 +0000

Lab Information

Some developers are working on a common repository where they are testing some features for an application. They are having three branches (excluding the master branch) in this repository where they are adding changes related to these different features. They want to test these changes on Stratos DC app servers so they need a Jenkins job using which they can deploy these different branches as per requirements. Configure a Jenkins job accordingly.

Click on the Jenkins button on the top bar to access the Jenkins UI. Login using username admin and password Adm!n321.

Similarly, click on Gitea button to access the Gitea page. Login to Gitea server using username sarah and password Sarah_pass123.

There is a Git repository named web_app on Gitea where developers are pushing their changes. It has three branches version1, version2 and version3 (excluding the master branch). You need not to make any changes in the repository.

Create a Jenkins job named app-job.

Configure this job to have a choice parameter named Branch with choices as given below:

version1

version2

version3

Configure the job to fetch changes from above mentioned Git repository and make sure it should fetches the changes from the respective branch which you are passing as a choice in the choice parameter while building the job. For example if you choose version1 then it must fetch and deploy the changes from branch version1.

Configure this job to use custom workspace rather than a default workspace and custom workspace directory should be created under /var/lib/jenkins (for example /var/lib/jenkins/version1) location rather than under any sub-directory etc. The job should use a workspace as per the value you will pass for Branch parameter while building the job. For example if you choose version1 while building the job then it should create a workspace directory called version1 and should fetch Git repository etc within that directory only.

Configure the job to deploy code (fetched from Git repository) on App Server 1 (stapp01) under /var/www/html directory.

You can access the website by clicking on App button.

Note:

You might need to install some plugins and restart Jenkins service. So, we recommend clicking on Restart Jenkins when installation is complete and no jobs are running on plugin installation/update page i.e update centre. Also, Jenkins UI sometimes gets stuck when Jenkins service restarts in the back end. In this case please make sure to refresh the UI page.

For these kind of scenarios requiring changes to be done in a web UI, please take screenshots so that you can share it with us for review in case your task is marked incomplete. You may also consider using a screen recording software such as loom.com to record and share your work.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

Install these plugins

Git
Publish Over SSH

Jenkins
Username: admin
Password: Adm!n321

Gitea
Username: sarah
Password: Sarah_pass123

Open the web_app repository.

Step 2: Create the Jenkins Job

New Item
Name:
app-job
Select Freestyle project
Click OK

Step 3: Add the Choice Parameter

Enable:

This project is parameterized

Add Choice Parameter.

Name:

Branch

Choices (one per line):

version1
version2
version3

Step 4: Configure the Custom Workspace

Under Advanced Project Options, enable:

Use custom workspace

Directory:

/var/lib/jenkins/${Branch}

⚠️ Do not use:

/var/lib/jenkins/workspace/${Branch}

/var/lib/jenkins/version1

The workspace must be dynamic.

Step 5: Configure Git

Source Code Management

Select:

Git

Repository URL

Use the HTTPS clone URL from Gitea.

Example:

https://3000-port-ml3qg35wxs4bogvv.labs.kodekloud.com/sarah/web_app.git

Credentials

Use:

sarah
Sarah_pass123
Branches to build

Use exactly:

*/${Branch}

⚠️ Do not use:

*/version1
master
origin/version1

Step 6: Deploy to stapp01

This is where most people fail.

The task requires:

Deploy code on App Server 1 (stapp01)

If Publish Over SSH is available (as in your previous lab), configure it.

Global Configuration

Go to:

Manage Jenkins
→ System
→ Publish over SSH

Configure:

Field Value
Name stapp01
Hostname stapp01
Username tony
Password Ir0nM@n
Remote Directory /var/www/html

Test Configuration → Success

Save.

Post-build Action

Add:

Send build artifacts over SSH

Server:

stapp01

Transfer Set:

Source files

⚠️ Use ** instead of index.html to ensure all checked-out files are transferred.

Remote Directory:

Leave blank if already configured globally, or set:

/var/www/html

Step 7: Build

Click:

Build with Parameters

Choose:

Branch = version1

Run.

After success:

Open App
Verify it displays the version1 content.

Step 8: Verify Dynamic Branch Selection

Repeat with:

Branch = version2

Refresh the App page.

Then:

Branch = version3

Refresh again.

The page content should change according to the selected branch.

🧠 Part 2: Simple Step-by-Step Explanations (Beginner Friendly)

Why use a Choice Parameter?

Instead of creating three separate jobs, you create one job that can deploy different branches based on the selected parameter.

Why use ${Branch}?

When you choose:

version2

Jenkins automatically replaces:

${Branch}

with:

version2

This affects:

the Git branch to clone
the workspace directory

Why use /var/lib/jenkins/${Branch}?

Each branch gets its own workspace:

version1
→ /var/lib/jenkins/version1

version2
→ /var/lib/jenkins/version2

version3
→ /var/lib/jenkins/version3

This prevents branches from overwriting each other's files.

Why use */${Branch}?

This tells Git:

Clone whichever branch the user selected.

Examples:

Branch = version1
→ */version1

Branch = version2
→ */version2

Why use ** for Source files?

The Publish Over SSH plugin transfers files matching the pattern.

Using:

copies the entire checked-out workspace.

Using only:

index.html

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

2.Jenkins Parameterized Builds

Thu Kha Kyawe — Thu, 09 Jul 2026 02:30:00 +0000

Lab Informaiton

A new DevOps Engineer has joined the team and he will be assigned some Jenkins related tasks. Before that, the team wanted to test a simple parameterized job to understand basic functionality of parameterized builds. He is given a simple parameterized job to build in Jenkins. Please find more details below:

Click on the Jenkins button on the top bar to access the Jenkins UI. Login using username admin and password Adm!n321.

Create a parameterized job which should be named as parameterized-job
Add a string parameter named Stage; its default value should be Build.
Add a choice parameter named env; its choices should be Development, Staging and Production.
Configure job to execute a shell command, which should echo both parameter values (you are passing in the job).
Build the Jenkins job at least once with choice parameter value Staging to make sure it passes.

Note:

You might need to install some plugins and restart Jenkins service. So, we recommend clicking on Restart Jenkins when installation is complete and no jobs are running on plugin installation/update page i.e update centre. Also, Jenkins UI sometimes gets stuck when Jenkins service restarts in the back end. In this case, please make sure to refresh the UI page.
For these kind of scenarios requiring changes to be done in a web UI, please take screenshots so that you can share it with us for review in case your task is marked incomplete. You may also consider using a screen recording software such as loom.com to record and share your work.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

Step 1: Log in to Jenkins
Click the Jenkins button on the top bar.
Login with:
Username: admin
Password: Adm!n321

Step 2: Create the Job
From the Jenkins Dashboard, click New Item.
Enter the job name: parameterized-job
Select Freestyle project.
Click OK.

Step 3: Enable Parameterized Build
In the job configuration page, scroll to This project is parameterized.
Check the box:
This project is parameterized

Step 4: Add String Parameter
Click Add Parameter.
Select String Parameter.
Configure:
Field Value
Name Stage
Default Value Build

It should look like:

Name: Stage
Default Value: Build

Step 5: Add Choice Parameter
Click Add Parameter again.
Select Choice Parameter.
Configure:
Field Value
Name env

Choices (one per line):

Development
Staging
Production

Step 6: Configure Build Step
Scroll to the Build section.
Click Add build step → Execute shell.
Enter:
echo "Stage: $Stage"
echo "Environment: $env"
Click Save.

Step 7: Build the Job
Open parameterized-job.
Click:
Build with Parameters

(not "Build Now")

Set:
Parameter Value
Stage Build
env Staging
Click Build.

Step 8: Verify Successful Build
Open the latest build.
Click Console Output.
Confirm output similar to:
Stage: Build
Environment: Staging
Finished: SUCCESS

🧠 Part 2: Simple Step-by-Step Explanations (Beginner Friendly)

What is a Parameterized Job?

A parameterized job lets users provide values when starting a build.

Instead of hardcoding values, Jenkins asks for them before running the job.

Example:

Stage = Build
env = Staging

These values become variables Jenkins can use during the build.

Why Create the String Parameter Stage?

A String Parameter allows users to type text.

You are creating:

Stage

with default value:

Build

If the user does not change it, Jenkins automatically uses:

Build

Why Create the Choice Parameter env?

A Choice Parameter provides a dropdown menu.

Available options:

Development
Staging
Production

This prevents typing mistakes and ensures valid environments are selected.

How Does the Shell Script Use Parameters?

When Jenkins runs the job:

echo "Stage: $Stage"
echo "Environment: $env"

Jenkins replaces:

$Stage

with the selected Stage value.

and replaces:

$env

with the selected environment.

Example:

Stage = Build
env = Staging

Output becomes:

Stage: Build
Environment: Staging

Why Must You Use "Build with Parameters"?

A parameterized job requires values to be supplied.

Therefore use:

Build with Parameters

instead of:

Build Now

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

1.Jenkins Views

Thu Kha Kyawe — Wed, 08 Jul 2026 23:30:00 +0000

Lab Information

The DevOps team of xFusionCorp Industries is planning to create a number of Jenkins jobs for different tasks. So to easily manage the jobs within Jenkins UI they decided to create different views for all Jenkins jobs based on usage/nature of these jobs, - for example xfusion-crons view for all cron jobs. Based on the requirements shared below please perform the below mentioned task:

Click on the Jenkins button on the top bar to access the Jenkins UI. Login using username admin and password Adm!n321.

Create a Jenkins job named xfusion-test-job.
Configure this job to run a simple bash command i.e echo "hello world!!".
Create a view named xfusion-crons (it must be a global view of type List View) and make sure xfusion-test-job and xfusion-cron-job (which is already present on Jenkins) jobs are listed under this new view.
Schedule this newly created job to build periodically at every minute i.e * * * * * (please make sure to use the cron expression exactly same how it is mentioned here).
Make sure the job builds successfully.

Note:

You might need to install some plugins and restart Jenkins service. So, we recommend clicking on Restart Jenkins when installation is complete and no jobs are running on plugin installation/update page i.e update centre. Also, Jenkins UI sometimes gets stuck when Jenkins service restarts in the back end. In this case please make sure to refresh the UI page.
For these kind of scenarios requiring changes to be done in a web UI, please take screenshots so that you can share it with us for review in case your task is marked incomplete. You may also consider using a screen recording software such as loom.com to record and share your work.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

Step 1: Log in to Jenkins
Click the Jenkins button on the top bar.
Login with:
Username: admin
Password: Adm!n321

Step 2: Create the Jenkins Job
From the Jenkins Home Dashboard (important: not from My Views), click New Item.
Enter:
xfusion-test-job
Select Freestyle project.
Click OK.

Step 3: Configure the Build Trigger
Open Configure for xfusion-test-job.
Scroll to Build Triggers.
Check Build periodically.
Enter exactly:

⚠️ Do not use:

H * * * *
H/1 * * * *

The validator checks for the exact cron expression.

Step 4: Configure the Build Step
Scroll to the Build section.
Click Add build step → Execute shell.
Enter:
echo "hello world!!"
Click Save.

Step 5: Create the Global View

⚠️ Important: Create the view from the Jenkins Home Dashboard, not from My Views.

Click the Jenkins logo to return to the main dashboard.
Click + New View.
Enter:
xfusion-crons
Select:
List View
Click Create.

Step 6: Add Jobs to the View
Open xfusion-crons.
Click Edit View.
Add these jobs:
xfusion-test-job
xfusion-cron-job
Save.
Verify both jobs appear in the view.

Step 7: Build the Job
Open xfusion-test-job.
Click Build Now.
Wait for the build to finish.

Step 8: Verify Success
Open the latest build.
Click Console Output.
Verify:
hello world!!
Finished: SUCCESS

🧠 Part 2: Simple Step-by-Step Explanations (Beginner Friendly)

Why create xfusion-test-job?

A Jenkins job is a task Jenkins can run automatically. This lab requires creating a new job.

Why use a Freestyle Project?

Freestyle projects are the simplest Jenkins jobs and are perfect for running shell commands.

Why add the cron schedule?

The schedule tells Jenkins when to run the job automatically.

The expression:

means:

Every minute of every hour of every day

Why add Execute shell?

This lets Jenkins run Linux commands.

The command:

echo "hello world!!"

simply prints:

hello world!!

to the build log.

Why create xfusion-crons as a Global List View?

The lab specifically requires:

View name: xfusion-crons
View type: List View
Scope: Global

Creating it from Jenkins Home ensures the validator can find it.

How do I know the task is complete?

You should have:

✔ Job: xfusion-test-job

✔ Build command:

echo "hello world!!"

✔ Global List View:

xfusion-crons

✔ Both jobs listed:

xfusion-test-job
xfusion-cron-job

✔ Cron schedule:

✔ Console Output:

hello world!!
Finished: SUCCESS

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

Day 14: Create a DVC Pipeline for Data Processing

Thu Kha Kyawe — Mon, 06 Jul 2026 12:44:36 +0000

Lab Information

The xFusionCorp Industries ML team utilizes DVC pipelines to ensure the reproducibility of data processing. The fraud-detection project has the processing scripts and raw data in place but does not yet define a pipeline. Define a two-stage DVC pipeline so the data processing runs reproducibly from start to finish with dvc repro.

A project exists at /root/code/fraud-detection/ with DVC initialised. The scripts are at src/data/process_data.py and src/data/split_data.py, and the raw input is at data/raw/transactions.csv. Do not modify the Python files or the input data.

Create a dvc.yaml defining two stages (use dvc stage add, or write the YAML directly):
    process_data – runs python3 src/data/process_data.py; depends on data/raw/transactions.csv and src/data/process_data.py; produces data/processed/clean_transactions.csv.
    split_data – runs python3 src/data/split_data.py; depends on data/processed/clean_transactions.csv (the upstream stage's output, so DVC chains the stages) and src/data/split_data.py; produces data/processed/train.csv and data/processed/test.csv.

Run the pipeline with dvc repro so both stages execute in order and dvc.lock is written.

After your changes, dvc status must report no stale stages.

Use python3 (not python) in the stage commands. Once the pipeline is valid, dvc dag prints the dependency graph showing how the two stages chain together.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

Step 1: Move into the repository

cd /root/code/fraud-detection

Step 2: Verify the required files

find src
find data

You should have:

src
src/data
src/data/process_data.py
src/data/split_data.py
data
data/raw
data/raw/transactions.csv

Step 3: Create the first DVC stage (process_data)

Run:

dvc stage add \
-n process_data \
-d data/raw/transactions.csv \
-d src/data/process_data.py \
-o data/processed/clean_transactions.csv \
python3 src/data/process_data.py

Explanation of options
Option Meaning
-n Stage name
-d Dependency
-o Output
Last argument Command to execute

Step 4: Create the second DVC stage (split_data)

Run:

dvc stage add \
-n split_data \
-d data/processed/clean_transactions.csv \
-d src/data/split_data.py \
-o data/processed/train.csv \
-o data/processed/test.csv \
python3 src/data/split_data.py

Notice that the dependency:

data/processed/clean_transactions.csv

is the output from Stage 1. This automatically links the stages together.

Step 5: Verify dvc.yaml

cat dvc.yaml

Expected structure:

stages:
  process_data:
    cmd: python3 src/data/process_data.py
    deps:
    - data/raw/transactions.csv
    - src/data/process_data.py
    outs:
    - data/processed/clean_transactions.csv
  split_data:
    cmd: python3 src/data/split_data.py
    deps:
    - data/processed/clean_transactions.csv
    - src/data/split_data.py
    outs:
    - data/processed/test.csv
    - data/processed/train.csv

Step 6: Run the pipeline

dvc repro

Expected output:

Running stage 'process_data':                                                  
> python3 src/data/process_data.py
Processed 15 rows
Generating lock file 'dvc.lock'                                                
Updating lock file 'dvc.lock'

Running stage 'split_data':                                                    
> python3 src/data/split_data.py
Train: 12 rows, Test: 3 rows
Updating lock file 'dvc.lock'                                                  

To track the changes with git, run:

        git add dvc.lock data/processed/.gitignore

To enable auto staging, run:

        dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.

Step 7: Verify dvc.lock

ls

Expected:

data
dvc.lock
dvc.yaml
src

Step 8: Check pipeline status

dvc status

Expected:

Data and pipelines are up to date.

This satisfies the lab requirement of having no stale stages.

Step 9: Display the dependency graph

dvc dag

Expected graph:

WARNING: Unable to find `less` in the PATH. Check out <https://man.dvc.org/pipeline/show> for more info.
+--------------+ 
| process_data | 
+--------------+ 
        *        
        *        
        *        
 +------------+  
 | split_data |  
 +------------+

🧠 Part 2: Simple Step-by-Step Explanation (Beginner Friendly)

What is a DVC Pipeline?

A DVC pipeline defines a sequence of steps for processing data. Each step is called a stage.

Instead of manually running scripts one by one, DVC knows:

which script to run
what input files it needs
what output files it creates
which stage depends on another

This makes the workflow reproducible.

Why do we create two stages?

The project has two separate tasks:

Stage 1: Process the raw data

Input:

data/raw/transactions.csv

Script:

src/data/process_data.py

Output:

data/processed/clean_transactions.csv

This stage cleans or prepares the raw dataset.

Stage 2: Split the processed data

Input:

data/processed/clean_transactions.csv

Script:

src/data/split_data.py

Outputs:

data/processed/train.csv
data/processed/test.csv

This stage creates the training and testing datasets.

How are the stages connected?

The output of the first stage becomes the input of the second stage:

transactions.csv
│
▼
process_data
│
▼
clean_transactions.csv
│
▼
split_data
├──────────┐
▼ ▼
train.csv test.csv

This dependency allows DVC to automatically execute the stages in the correct order.

What does dvc repro do?

The command:

dvc repro

checks the pipeline and runs any stages that need to be executed.

In this lab:

process_data runs first because it depends on the raw dataset.
After clean_transactions.csv is created, split_data runs because its dependency is now available.

You don't need to run the Python scripts manually—DVC handles the execution order.

What is dvc.lock?

After a successful pipeline run, DVC creates:

dvc.lock

This file records:

the exact command used
the dependencies
the output files
checksums (hashes) of inputs and outputs

It ensures that the pipeline can be reproduced exactly in the future.

Resources & Next Steps

📦 Full Code Repository: KodeKloud Learning Labs

💬 Join Discussion: DEV Community - Share your thoughts and questions

💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits

• All labs are from: KodeKloud

• I sincerely appreciate your provision of these valuable resources.

Day 13: Pull DVC-Tracked Data from Remote

Thu Kha Kyawe — Mon, 06 Jul 2026 12:40:45 +0000

Lab Information

A new xFusionCorp Industries team member has cloned the fraud-detection repository onto a fresh machine. The DVC remote is already configured to point at the team's SeaweedFS bucket, but dvc pull is failing. Diagnose the cause, correct the configuration, and pull the dataset.

A cloned project exists at /root/code/fraud-detection/ with DVC initialised, the data/raw/transactions.csv.dvc pointer file present, but the dataset itself missing from disk and from the local DVC cache.

SeaweedFS is already running on the controlplane and the dataset has already been pushed to the dvc-storage bucket—open the SeaweedFS Filer button at the top of the lab and navigate to /buckets/dvc-storage/ to confirm that the object is there.
    S3 endpoint: http://localhost:8333
    Credentials: weedadmin / weedadmin123

Review .dvc/config and correct everything that prevents dvc pull from authenticating against SeaweedFS.

After the fix, the s3 remote must use:
    The access key (access_key_id) weedadmin
    The secret key (secret_access_key) weedadmin123.

Pull the dataset. After the pull, data/raw/transactions.csv must be present on disk and its content must match the hash recorded in the .dvc pointer.

Lab Solutions

✅ Part 1: Lab Step-by-Step Guidelines

Step 1: Move into the repository

cd /root/code/fraud-detection

Step 2: Verify the dataset is missing

ls -la data/raw

You should see:

transactions.csv.dvc

but not:

transactions.csv

Step 3: Inspect the DVC configuration

cat .dvc/config

Look for the s3 remote configuration.

You are specifically looking for incorrect credentials.

Step 4: Fix the SeaweedFS credentials

The lab requires:

Setting Value
access_key_id weedadmin
secret_access_key weedadmin123

Configure them:

dvc remote modify s3 access_key_id weedadmin
dvc remote modify s3 secret_access_key weedadmin123

Step 5: Verify the configuration

cat .dvc/config

Expected section:

[core]
    remote = s3
['remote "s3"']
    url = s3://dvc-storage
    endpointurl = http://localhost:8333
    access_key_id = weedadmin
    secret_access_key = weedadmin123

Step 6: Pull the dataset

dvc pull

Expected output:

root@controlplane fraud-detection on  main [!] ➜  dvc pull
Collecting                                           |1.00 [00:00,  730entry/s]
Fetching
Building workspace index                             |2.00 [00:00,  661entry/s]
Comparing indexes                                   |4.00 [00:00, 2.90kentry/s]
Applying changes                                     |1.00 [00:00, 1.18kfile/s]
A       data/raw/transactions.csv
1 file fetched and 1 file added

Step 7: Verify the dataset exists

ls -l data/raw/transactions.csv

Expected:

data/raw/transactions.csv

Step 8: Verify DVC status

dvc status

Expected:

Data and pipelines are up to date.

Step 9: Verify integrity

Check the DVC pointer:

cat data/raw/transactions.csv.dvc

Example:

root@controlplane fraud-detection on  main [!] ➜  cat data/raw/transactions.csv.dvc
outs:
- md5: 555f037ee350464f52122d087f28e857
  size: 446
  hash: md5
  path: transactions.csv

🧠 Part 2: Simple Step-by-Step Explanation (Beginner Friendly)

What happened?

A teammate cloned the Git repository onto a new machine.

Git downloaded:

transactions.csv.dvc

but not:

transactions.csv

because DVC-managed data is stored separately from Git.

Why is the file missing?

Git only stores the small pointer file:

transactions.csv.dvc

The actual dataset lives in SeaweedFS:

SeaweedFS Bucket
└── dvc-storage

To download it, DVC must connect to the remote storage.

Why is dvc pull failing?

The lab says:

Review .dvc/config and correct everything that prevents dvc pull from authenticating.

This means the most likely issue is incorrect credentials.

Required credentials:

access_key_id = weedadmin
secret_access_key = weedadmin123

If either value is wrong, DVC cannot access the SeaweedFS bucket.

What does dvc pull do?

When you run:

dvc pull

DVC:

Reads transactions.csv.dvc
Finds the file hash
Connects to SeaweedFS
Downloads the matching object
Restores:
data/raw/transactions.csv