DEV Community: lotanna obianefo

Building a Complete CI/CD Pipeline with Node.js, Docker, and K8s.

lotanna obianefo — Tue, 03 Mar 2026 16:53:45 +0000

Modern software delivery demands speed, reliability, and scalability. Continuous Integration and Continuous Deployment (CI/CD) pipelines enable teams to ship code faster while maintaining high quality and operational stability. When combined with Node.js, Docker, and Kubernetes (K8s), CI/CD becomes a powerful system for building cloud-native, production-ready applications.

This article provides a complete technical walkthrough of designing and implementing a CI/CD pipeline.

Before starting, you need to install these tools on your machine.

Required Downloads:

Node.js (v18 or higher) - Current LTS: v20.x
Download from: https://nodejs.org/
Choose the LTS version (20.x as of 2025)
Verify installation: node --version and npm --version
Git - Latest stable version
Download from: https://git-scm.com/downloads
Choose your operating system version
Verify installation: git --version
Docker Desktop - Latest version
Download from: https://www.docker.com/products/docker-desktop/
Install and start Docker Desktop
Verify installation: docker --version and docker-compose --version
GitHub Account
Sign up at: https://github.com
You'll need this for hosting your code and CI/CD pipeline
VS Code
Download: https://code.visualstudio.com/

Verify Everything is Installed

Step 1: Set Up Git for Version Control

Configures Git on your machine so it knows who you are when you make commits and sets up proper project tracking.

One-time Git Configuration

     git config --global user.name "Your Name"
     git config --global user.email "you@example.com"
     git config --global init.defaultBranch main

To displays all Git configuration settings that are currently applied for your environment use the command git config --list

Create and Initialize Project

To create and initialize the project, first create a new directory, navigate into it, and ensure the directory is initialized with a local Git repository using the below commands.

     mkdir my-devops-project
     cd my-devops-project
     git init

Step 2: Build a Node.js Web App

Creates a web application using Node.js that can serve web pages and API endpoints.

Initialize Node.js Project
This command creates a package.json file that describes your project and manages dependencies.

      npm init -y

The -y flag automatically responds “yes” to all confirmation prompts during command execution.

Update package.json

This customizes the package.json with proper scripts and metadata for your DevOps project.

Create/edit package.json:

      {
        "name": "devops-project01",
        "version": "1.0.0",
       "description": "DevOps learning project with Node.js",
        "main": "app.js",
        "scripts": {
          "start": "node app.js",
          "test": "jest",
          "dev": "node app.js",
          "lint": "eslint ."
        },
        "keywords": ["devops", "nodejs", "docker"],
        "author": "lotanna",
        "license": "MIT",
        "engines": {
          "node": ">=18.0.0"
        },
        "devDependencies": {
          "jest": "^29.7.0",
          "eslint": "^8.57.0",
          "supertest": "^7.1.4"
        }
      }

Create Application File

To create the application file, use the touch command to generate app.js, then insert the required code, which defines the following functionalities and outlines.

HTTP server that listens on port 3000
Serves different endpoints (/, /health, /info, /metrics)
Includes security headers and proper error handling
Provides graceful shutdown capability
Exports the server for testing

Create app.js:

      // core modules
      const http = require("http");
      const url = require("url");

      // environment configuration
      const PORT = process.env.PORT || 3000;
      const ENVIRONMENT = process.env.NODE_ENV || "development";

      let requestCount = 0;

      // helper: send JSON responses
      function sendJSON(res, statusCode, data) {
        res.statusCode = statusCode;
        res.setHeader("Content-Type", "application/json");
        res.end(JSON.stringify(data, null, 2));
      }

      // helper: send HTML responses
      function sendHTML(res, statusCode, content) {
        res.statusCode = statusCode;
        res.setHeader("Content-Type", "text/html");
        res.end(content);
      }

      // helper: send Prometheus metrics
      function sendMetrics(res) {
        const mem = process.memoryUsage();
        const metrics = `
      # HELP http_requests_total Total HTTP requests
      # TYPE http_requests_total counter
      http_requests_total ${requestCount}

      # HELP app_uptime_seconds Application uptime in seconds
      # TYPE app_uptime_seconds gauge
      app_uptime_seconds ${process.uptime()}

      # HELP nodejs_memory_usage_bytes Node.js memory usage
      # TYPE nodejs_memory_usage_bytes gauge
      nodejs_memory_usage_bytes{type="rss"} ${mem.rss}
      nodejs_memory_usage_bytes{type="heapUsed"} ${mem.heapUsed}
      nodejs_memory_usage_bytes{type="heapTotal"} ${mem.heapTotal}
      nodejs_memory_usage_bytes{type="external"} ${mem.external}
      `;
        res.statusCode = 200;
        res.setHeader("Content-Type", "text/plain");
        res.end(metrics);
      }

      // main server
      const server = http.createServer((req, res) => {
        requestCount++;
        const timestamp = new Date().toISOString();
        const { pathname } = url.parse(req.url, true);

        // logging
        console.log(
          `${timestamp} - ${req.method} ${pathname} - ${
            req.headers["user-agent"] || "Unknown"
          }`
        );

        // CORS headers
        res.setHeader("Access-Control-Allow-Origin", "*");
        res.setHeader("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE");
        res.setHeader("Access-Control-Allow-Headers", "Content-Type");

        // security headers
        res.setHeader("X-Content-Type-Options", "nosniff");
        res.setHeader("X-Frame-Options", "DENY");
        res.setHeader("X-XSS-Protection", "1; mode=block");

        // route handling
        switch (pathname) {
          case "/":
           sendHTML(
              res,
              200,
              `
      <!DOCTYPE html>
      <html>
      <head>
        <title>DevOps Lab 2025</title>
        <style>
          body { font-family: Arial, sans-serif; max-width: 800px; margin: 50px auto; padding: 20px; }
          .header { background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 20px; border-radius: 8px; }
          .endpoint { background: #f8f9fa; padding: 15px; margin: 10px 0; border-radius: 5px; border-left: 4px solid #007bff; }
        </style>
      </head>
      <body>
        <div class="header">
          <h1>DevOps Lab 2025</h1>
          <p>Modern Node.js application with CI/CD pipeline</p>
        </div>
        <h2>Available Endpoints:</h2>
        <div class="endpoint"><strong>GET /</strong> - This welcome page</div>
        <div class="endpoint"><strong>GET /health</strong> - Health check (JSON)</div>
        <div class="endpoint"><strong>GET /info</strong> - System information</div>
        <div class="endpoint"><strong>GET /metrics</strong> - Prometheus metrics</div>
        <p>Environment: <strong>${ENVIRONMENT}</strong></p>
        <p>Server time: <strong>${timestamp}</strong></p>
        <p>Requests served: <strong>${requestCount}</strong> </p>
      </body>
      </html>`
            );
            break;

         case "/health":
             sendJSON(res, 200, {
              status: "healthy",
              timestamp,
              uptime: process.uptime(),
              environment: ENVIRONMENT,
              version: "1.0.0",
              node_version: process.version,
              requests_served: requestCount,
            });
            break;

          case "/info":
            sendJSON(res, 200, {
              platform: process.platform,
              architecture: process.arch,
              node_version: process.version,
              memory_usage: process.memoryUsage(),
              environment: ENVIRONMENT,
              pid: process.pid,
              uptime: process.uptime(),
            });
            break;

          case "/metrics":
            sendMetrics(res);
            break;

          default:
            sendJSON(res, 404, {
              error: "Not Found",
              message: `Route ${pathname} not found`,
              timestamp,
            });
        }
      });

      // graceful shutdown
      function shutdown(signal) {
        console.log(`\nReceived ${signal}, shutting down gracefully...`);
        server.close(() => {
          console.log("Server closed");
          process.exit(0);
        });
      }
      process.on("SIGTERM", () => shutdown("SIGTERM"));
      process.on("SIGINT", () => shutdown("SIGINT"));

      // start server
      server.listen(PORT, () => {
        console.log(`🚀 Server running at http://localhost:${PORT}/`);
        console.log(`Environment: ${ENVIRONMENT}`);
        console.log(`Node.js version: ${process.version}`);
      });

      // export for testing
      module.exports = server;

touch app.js

Install Dependencies

      # Install testing and development tools
      npm install --save-dev jest eslint supertest

      # Install all dependencies (creates node_modules folder)
      npm install

npm install --save-dev jest eslint supertest installs development-only tools for testing (Jest), code quality enforcement (ESLint), and HTTP/API testing (Supertest), and records them under devDependencies.

npm install installs all dependencies defined in package.json (both dependencies and devDependencies), creates the node_modules directory, and ensures version consistency using package-lock.json.

After running the code, you'll see:

A node_modules/ folder with all installed packages
A package-lock.json file that locks dependency versions

Step 3: Create Proper Tests

This step sets up automated testing so you can verify your application works correctly every time you make changes.

Create tests directory and test file

      # Create a folder for your tests
      mkdir tests

      # Create the main test file
      touch tests/app.test.js

Create test file

Copy this code into already created tests/app.test.js:

      const request = require('supertest');
      const server = require('../app');

      describe('App Endpoints', () => {
        afterAll(() => {
          server.close();
        });

        test('GET / should return welcome page', async () => {
          const response = await request(server).get('/');
          expect(response.status).toBe(200);
          expect(response.text).toContain('DevOps Lab 2025');
        });

        test('GET /health should return health status', async () => {
          const response = await request(server).get('/health');
          expect(response.status).toBe(200);
          expect(response.body.status).toBe('healthy');
          expect(response.body.timestamp).toBeDefined();
          expect(typeof response.body.uptime).toBe('number');
        });

        test('GET /info should return system info', async () => {
          const response = await request(server).get('/info');
          expect(response.status).toBe(200);
          expect(response.body.platform).toBeDefined();
          expect(response.body.node_version).toBeDefined();
        });

        test('GET /metrics should return prometheus metrics', async () => {
          const response = await request(server).get('/metrics');
          expect(response.status).toBe(200);
          expect(response.text).toContain('http_requests_total');
          expect(response.text).toContain('app_uptime_seconds');
        });

        test('GET /nonexistent should return 404', async () => {
          const response = await request(server).get('/nonexistent');
          expect(response.status).toBe(404);
          expect(response.body.error).toBe('Not Found');
        });
      });

Create Jest configuration

Create a jest.config.js file, add the required configuration content, and save the file.

      module.exports = {
        testEnvironment: 'node',
        collectCoverage: true,
        coverageDirectory: 'coverage',
        testMatch: ['**/tests/**/*.test.js'],
        verbose: true
      };

Step 4: GitHub Actions CI/CD Pipeline

This Creates an automated pipeline that runs tests and builds Docker images every time you push code to GitHub.

Create workflow directory

The .github/workflows directory is the local repository path where GitHub Actions workflows are defined and executed to manage CI/CD processes.

      # Create the GitHub Actions directory structure
      mkdir -p .github/workflows

Create CI/CD pipeline file

Create .github/workflows/ci.yml:

Now add the required configuration content and save the file.

      name: CI/CD Pipeline

      on:
        push:
          branches: [ main, develop ]
          tags: [ 'v*' ]
        pull_request:
          branches: [ main ]

      env:
        REGISTRY: ghcr.io
        IMAGE_NAME: ${{ github.repository }}

      concurrency:
        group: ${{ github.workflow }}-${{ github.ref }}
        cancel-in-progress: true

      jobs:
        test:
          name: Test & Lint
          runs-on: ubuntu-latest

          strategy:
            matrix:
              node-version: [20, 22]

          steps:
            - name: Checkout code
              uses: actions/checkout@v4

            - name: Setup Node.js ${{ matrix.node-version }}
              uses: actions/setup-node@v4
              with:
                node-version: ${{ matrix.node-version }}
                cache: 'npm'

            - name: Install dependencies
              run: npm ci

            - name: Run linting
              run: npm run lint

            - name: Run tests
              run: npm test

            - name: Security audit
              run: npm audit --audit-level=critical || echo "Audit completed with warnings"

       build:
          name: Build & Push Image
          runs-on: ubuntu-latest
          needs: test
          if: github.event_name == 'push'

          permissions:
            contents: read
            packages: write
            security-events: write

          outputs:
            image-tag: ${{ steps.meta.outputs.tags }}
            image-digest: ${{ steps.build.outputs.digest }}

          steps:
            - name: Checkout code
              uses: actions/checkout@v4

            - name: Set up Docker Buildx
              uses: docker/setup-buildx-action@v3
              with:
                platforms: linux/amd64,linux/arm64

            - name: Log in to Container Registry
              uses: docker/login-action@v3
              with:
                registry: ${{ env.REGISTRY }}
                username: ${{ github.actor }}
                password: ${{ secrets.GITHUB_TOKEN }}

            - name: Extract metadata
              id: meta
              uses: docker/metadata-action@v5
              with:
                images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
                tags: |
                  type=ref,event=branch
                  type=ref,event=pr
                  type=semver,pattern={{version}}
                  type=semver,pattern={{major}}.{{minor}}
                  type=sha,prefix={{branch}}-
                  type=raw,value=${{ github.run_id }}
                  type=raw,value=latest,enable={{is_default_branch}}
                labels: |
                  org.opencontainers.image.title=DevOps Lab 2025
                  org.opencontainers.image.description=Modern Node.js DevOps application

            - name: Build and push Docker image
              id: build
              uses: docker/build-push-action@v5
              with:
                context: .
                platforms: linux/amd64,linux/arm64
                push: true
                tags: ${{ steps.meta.outputs.tags }}
                labels: ${{ steps.meta.outputs.labels }}
                cache-from: type=gha
                cache-to: type=gha,mode=max
                target: production

            - name: Run Trivy vulnerability scanner
              uses: aquasecurity/trivy-action@0.24.0
              with:
                image-ref: ${{ steps.meta.outputs.tags }}
                format: 'sarif'
                output: 'trivy-results.sarif'
                severity: 'CRITICAL,HIGH'
              continue-on-error: true

            - name: Upload Trivy scan results
              uses: github/codeql-action/upload-sarif@v3
              if: always() && hashFiles('trivy-results.sarif') != ''
              with:
                sarif_file: 'trivy-results.sarif'

        deploy-staging:
          name: Deploy to Staging
          runs-on: ubuntu-latest
          needs: build
          if: github.ref == 'refs/heads/develop'
          environment: staging

          steps:
            - name: Deploy to Staging
              run: |
                echo "🚀 Deploying to staging environment..."
                echo "Image: ${{ needs.build.outputs.image-tag }}"
                echo "Digest: ${{ needs.build.outputs.image-digest }}"
                # Add your staging deployment commands here (kubectl, helm, etc.)

        deploy-production:
          name: Deploy to Production
          runs-on: ubuntu-latest
          needs: build
          if: github.ref == 'refs/heads/main'
          environment: production

          steps:
            - name: Deploy to Production
              run: |
                echo "🎯 Deploying to production environment..."
                echo "Image: ${{ needs.build.outputs.image-tag }}"
                echo "Digest: ${{ needs.build.outputs.image-digest }}"
                # Add your production deployment commands here

Step 5: Dockerfile

This step defines the build instructions Docker uses to create a portable container image of the application, ensuring consistent execution across environments.

The Dockerfile performs the following functions:

Uses multi-stage builds for smaller image size
Installs curl for health checks
Creates a non-root user for security
Sets up proper file permissions
Configures health checks

Create Dockerfile:

Input these commands into the Dockerfile

      # Multi-stage build for optimized image
      FROM node:20-alpine AS dependencies

      # Update packages for security
      RUN apk update && apk upgrade --no-cache

      WORKDIR /app

      # Copy package files first for better caching
      COPY package*.json ./

      # Install only production dependencies
      RUN npm ci --only=production && npm cache clean --force

      # Production stage  
      FROM node:20-alpine AS production

      # Update packages and install necessary tools
      RUN apk update && apk upgrade --no-cache && \
          apk add --no-cache curl dumb-init && \
          rm -rf /var/cache/apk/*

      # Create non-root user with proper permissions
      RUN addgroup -g 1001 -S nodejs && \
          adduser -S nodeuser -u 1001 -G nodejs

      WORKDIR /app

      # Copy dependencies from previous stage with proper ownership
      COPY --from=dependencies --chown=nodeuser:nodejs /app/node_modules ./node_modules

      # Copy application code with proper ownership
      COPY --chown=nodeuser:nodejs package*.json ./
      COPY --chown=nodeuser:nodejs app.js ./

      # Switch to non-root user
      USER nodeuser

      # Expose port
      EXPOSE 3000

      # Health check with proper timing for Node.js startup
      HEALTHCHECK --interval=30s --timeout=10s --start-period=15s --retries=3 \
        CMD curl -f http://localhost:3000/health || exit 1

      # Use dumb-init for proper signal handling in containers
      ENTRYPOINT ["dumb-init", "--"]

      # Start application
      CMD ["npm", "start"]

Step 6: Essential Configuration Files

This creates configuration files that tell various tools what to ignore, how to behave, and what settings to use.

Create .dockerignore

Create .dockerignore:

Input these commands into .dockerignore and save.

      # =========================
      # Dependencies & Package Managers
      # =========================
      node_modules
      npm-debug.log*
      coverage
      .nyc_output

      # =========================
      # Environment Variables
      # =========================
      .env
      .env.local
      .env.*.local

      # =========================
      # Version Control
      # =========================
      .git
      .github

      # =========================
      # Logs
      # =========================
      logs
      *.log

      # =========================
      # Editor & IDE Files
      # =========================
      .vscode
      .idea
      *.swp
      *.swo

      # =========================
      # OS Files
      # =========================
      .DS_Store
      Thumbs.db

      # =========================
      # Testing & Tooling
      # =========================
      tests/
      jest.config.js
      .eslintrc*

      # =========================
      # Documentation
      # =========================
      README.md

Create .gitignore

Create .gitignore:

Input these commands into already created .gitignore and save.

         # ===============================
         # Dependencies
         # ===============================
         node_modules/
         npm-debug.log*
         yarn-debug.log*
         yarn-error.log*

         # ===============================
         # Runtime Data
         # ===============================
         pids
         *.pid
         *.seed
         *.pid.lock

         # ===============================
         # Coverage & Test Reports
         # ===============================
         coverage/
         .nyc_output/

         # ===============================
         # Environment Variables
         # ===============================
         .env
         .env.local
         .env.*.local

         # ===============================
         # Logs
         # ===============================
         logs/
         *.log

         # ===============================
         # IDE / Editor
         # ===============================
         .vscode/
         .idea/
         *.swp
         *.swo

         # ===============================
         # OS Files
         # ===============================
         .DS_Store
         Thumbs.db

Create environment template

Create .env.example:

Input these commands into already created .env.example and save

       # ===============================
       # Server Configuration
       # ===============================
       PORT=3000
       NODE_ENV=production

       # ===============================
       # Logging Configuration
       # ===============================
       LOG_LEVEL=info

Create ESLint configuration

Create .eslintrc.js:

Input these commands into already created .eslintrc.js and save.

      module.exports = {
        env: {
          node: true,
          es2021: true,
          jest: true
        },
        extends: ['eslint:recommended'],
        parserOptions: {
          ecmaVersion: 12,
          sourceType: 'module'
        },
        rules: {
          'no-console': 'off',
          'no-unused-vars': ['error', { 'argsIgnorePattern': '^_' }]
        }
      };

Step 7: Docker Compose for Development

This creates a Docker Compose file that makes it easy to run your application and any supporting services with a single command.

Create docker-compose.yml:

      version: '3.8'

      services:
        app:
          build: .
          ports:
            - "3000:3000"
          environment:
            - NODE_ENV=development
            - PORT=3000
          restart: unless-stopped
          healthcheck:
            test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
            interval: 30s
            timeout: 10s
            retries: 3
            start_period: 10s

Step 8: Test Everything Locally

This shows you how to actually run and test your application locally before deploying it.

Install and Test Locally

      # Install all dependencies from package.json
      npm install
      # Run your test suite to make sure everything works
      npm test
      # Start the application server
      npm start

Tests should pass with green checkmarks: ✓ GET / should return welcome page

Server starts and shows: 🚀 Server running at http://localhost:3000/

Test endpoints (in a new terminal window):

      curl http://localhost:3000/         # Homepage
      curl http://localhost:3000/health   # Health check JSON
      curl http://localhost:3000/info     # System info JSON  
      curl http://localhost:3000/metrics  # Prometheus metrics

Docker Commands

      # Build image
      docker build -t my-devops-app:latest .

      # Run container
      docker run -d \
        --name my-devops-container \
        -p 3000:3000 \
        --restart unless-stopped \
        my-devops-app:latest

      # Check container status
      docker ps
      docker logs my-devops-container

      # Test health check
      curl http://localhost:3000/health

      # Stop container
      docker stop my-devops-container
      docker rm my-devops-container

Build image

Run container

Check container status

Test health check

Stop container

Docker Compose Commands
Docker compose is not used to test in production environment but locally.

      # Start all services defined in docker-compose.yml
      docker-compose up -d

      # View real-time logs from all services
      docker-compose logs -f

      # Stop all services and clean up
      docker-compose down

Start all services defined in docker-compose.yml

View real-time logs from all services

Stop all services and clean up

Step 9: Deploy to GitHub
This commits your code to Git and pushes it to GitHub so the automated CI/CD pipeline can start working.

Initial commit

      # Add all files to Git staging area
      git add .

      # Create your first commit with a descriptive message
      git commit -m "Initial commit: Complete DevOps setup with working CI/CD"

Connect to GitHub

Before running these commands:

Go to GitHub.com and create a new repository called my-devops-project
DO NOT initialize it with README, .gitignore, or license (we already have these)
Copy the repository URL from GitHub

Replace YOUR_GITHUB_USERNAME below with your actual GitHub username

   # Set main as the default branch
   git branch -M main

   # Connect to your GitHub repository (UPDATE THIS URL!)
   git remote add origin https://github.com/YOUR_GITHUB_USERNAME/my-devops-project.git

   # Push your code to GitHub for the first time
   git push -u origin main

Your code appears on GitHub, and the CI/CD pipeline starts running automatically.

Step 10: Kubernetes Deployment Configurations
This step creates Kubernetes configuration files that define how your application should run in staging and production environments.

Create directories

       # Create directories for Kubernetes configurations
       mkdir -p k8s/staging k8s/production

Create Staging Deployment
Create k8s/staging/deployment.yml

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: devops-app-staging
        namespace: staging
      spec:
        replicas: 2
        selector:
          matchLabels:
            app: devops-app
            environment: staging
        template:
          metadata:
            labels:
              app: devops-app
              environment: staging
          spec:
            containers:
            - name: app
              image: ghcr.io/YOUR_GITHUB_USERNAME/my-devops-project:develop-latest
              ports:
              - containerPort: 3000
              env:
              - name: NODE_ENV
                value: "staging"
              - name: PORT
                value: "3000"
              livenessProbe:
                httpGet:
                  path: /health
                  port: 3000
                initialDelaySeconds: 30
                periodSeconds: 10
              readinessProbe:
                httpGet:
                  path: /health
                  port: 3000
                initialDelaySeconds: 5
                periodSeconds: 5
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: devops-app-service-staging
        namespace: staging
      spec:
        selector:
          app: devops-app
          environment: staging
        ports:
        - protocol: TCP
          port: 80
          targetPort: 3000
        type: LoadBalancer

Create Production Deployment
Create k8s/production/deployment.yml

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: devops-app-production
        namespace: production
      spec:
        replicas: 3
        selector:
          matchLabels:
            app: devops-app
            environment: production
        template:
          metadata:
            labels:
              app: devops-app
              environment: production
          spec:
            containers:
            - name: app
              image: ghcr.io/YOUR_GITHUB_USERNAME/my-devops-project:latest
              ports:
              - containerPort: 3000
              env:
              - name: NODE_ENV
                value: "production"
              - name: PORT
                value: "3000"
              resources:
                requests:
                  memory: "128Mi"
                  cpu: "100m"
                limits:
                  memory: "256Mi"
                  cpu: "200m"
              livenessProbe:
                httpGet:
                  path: /health
                  port: 3000
                initialDelaySeconds: 30
                periodSeconds: 10
              readinessProbe:
                httpGet:
                  path: /health
                  port: 3000
                initialDelaySeconds: 5
                periodSeconds: 5
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: devops-app-service-production
        namespace: production
      spec:
        selector:
          app: devops-app
          environment: production
        ports:
        - protocol: TCP
          port: 80
          targetPort: 3000
        type: LoadBalancer

Step 11: Complete Deployment Workflow

This step shows you how to use the complete CI/CD pipeline with proper branching strategy for staging and production deployments.

Branch-based Deployment Strategy

develop branch → Automatically deploys to staging environment
main branch → Automatically deploys to production environment
Pull requests → Run tests only (no deployment)

Deploy Changes
Deploy to staging:

      # Create and switch to develop branch
      git checkout -b develop

      # Make your changes, then commit and push
      git add .
      git status
      git commit -m "Add new feature"
      git push origin develop

GitHub Actions automatically runs tests and deploys to staging.

Deploy to production:

      # Switch to main branch
      git checkout main

      # Merge changes from develop
      git merge develop

      # Push to trigger production deployment
      git push origin main

Monitor Deployments

You can execute this command to monitor the GitHub Actions workflow execution and verify the health status of the deployed application.

      # Check GitHub Actions status
      # Visit: https://github.com/YOUR_GITHUB_USERNAME/my-devops-project/actions

      # Check your container registry
      # Visit: https://github.com/YOUR_GITHUB_USERNAME/my-devops-project/pkgs/container/my-devops-project

      # Test your deployed applications (once you have URLs)
      curl https://your-staging-url.com/health      # Staging health check
      curl https://your-production-url.com/health   # Production health check

This project implements a production grade CI/CD pipeline for a containerized Node.js application using GitHub Actions, Docker, and Kubernetes, demonstrating end-to-end DevOps engineering practices. The application exposes health, metrics, and system endpoints with graceful shutdown handling for Kubernetes environments.

Mastering Git for Production: Branching, Merging & Squash Strategies Every Engineer Should Know

lotanna obianefo — Thu, 19 Feb 2026 12:23:27 +0000

Modern software development depends heavily on version control systems to manage code changes, support collaboration, and maintain stability. Git is the industry-standard distributed version control system that enables teams to work concurrently without overwriting each other’s progress.

Core to Git’s power are its branching and history management capabilities specifically branching, merging, squashing, and rebasing. Understanding these concepts is essential for maintaining a clean, traceable, and production ready codebase.

By the end of this article, engineers will understand how to:

Create branches to work safely
Merge branches using different strategies
Fix merge conflicts
Squash many commits into one
Push a complete project to GitHub

PROJECT SETUP -Part 1

1. Create Project Folder

      mkdir git-merge-lab

This creates a new, empty directory named git-merge-lab, establishing an isolated workspace that prevents interference with existing files and ensures a controlled environment for Git merge practice.

2. Enter the Folder

     cd git-merge-lab

This navigates the terminal session into the project directory, setting the folder as the current working environment.

Purpose:
Git operations are context-sensitive and can only be executed within an initialized repository or project directory.

Outcome:
All subsequent files, commits, and version control activities will be contained within this directory, ensuring organized project structure and traceability.

3. Initialize Git

      git init

This Initializes the directory as a Git repository by generating a hidden .git subdirectory that stores metadata, configuration settings, and the complete version history.

Purpose:
Repository initialization is required for Git to begin tracking file changes and managing version control.

Outcome:
The directory is now under Git management, enabling change tracking, staging, committing, and collaboration workflows.

4. Create README File

      echo "# Team Project" > README.md

This creates a README.md file and populates it with initial content.

Purpose:
Git requires at least one tracked file to generate a commit and establish version history within the repository.

Outcome:
The README.md file is present in the working directory but remains untracked until it is staged and committed to the repository.

5. Stage the File

      git add README.md

This stages the file for inclusion in the upcoming commit by adding it to the Git index.

Purpose:
Git only records changes that have been explicitly staged, allowing developers to control which modifications are captured in a commit.

Outcome:
README.md is now in the staging area and prepared to be committed to the repository history.

6. Commit the File

      git commit -m "Initial commit"

This creates a commit that captures a snapshot of the project state within the repository history.

Purpose:
Committing establishes a restore point, enabling version rollback and facilitating traceability if future changes introduce issues.

Outcome:
The repository now contains its initial commit, marking the first recorded version of the project.

7. Rename Branch to main

      git branch -M main

This renames the active branch to main, aligning it with modern repository naming conventions.

Purpose:
Most hosting platforms and enterprise workflows designate main as the default branch, promoting consistency across development environments.

Outcome:
The primary development branch is now labeled main, serving as the central integration point for future changes.

FAST-FORWARD MERGE -Part 2

8. Create Feature Branch

     git checkout -b feature-login

This creates a new Git branch and checks it out as the active working branch.

Purpose:
Feature development is isolated in dedicated branches to prevent instability in the main (or production) branch and to support controlled integration workflows.

Outcome:
The working directory is now tracking the feature-login branch instead of main, ensuring all subsequent commits apply only to this feature stream.

9. Add Feature File

      echo "Login page created" > login.txt

This creates a new source file associated with the login functionality.

Purpose:
This step represents the introduction of new application logic and simulates the implementation of a feature within the codebase.

10. Commit Feature

      git add login.txt
      git commit -m "Add login feature"

Purpose of both commands:
git add stages the modified file by placing it into the index, preparing it for version control tracking.

git commit records the staged changes into the repository history as a new snapshot.

Outcome:
The feature changes are now securely versioned and stored within the feature branch’s commit history.

11. Switch Back to Main

      git checkout main

This switches the working context back to the main branch.

Purpose:
In Git, merge operations are executed into the currently checked-out branch, so you must be on the target branch before initiating a merge.

12. Merge Feature (Fast-Forward)

      git merge feature-login

The main branch advances its pointer to incorporate the commits from the feature branch.

Purpose:
Since main has not diverged and contains no new commits since the branch was created, Git performs a fast-forward merge rather than generating an additional merge commit.

3-WAY MERGE (PARALLEL WORK) -Part 3

13. Create Profile Feature

     git checkout -b feature-profile
     echo "Profile page created" > profile.txt
     git add profile.txt
     git commit -m "Add profile feature"

This establishes an isolated development context that allows a new feature to be implemented independently, without impacting the stability or history of other branches.

Note;
git checkout main switches the working directory to the main branch, ensuring the new work is based on the latest stable code.

git checkout -b feature-settings creates a new branch named feature-settings from main and immediately switches to it. These isolates feature development from the main codebase.

echo "Settings page created" > settings.txt creates a new file called settings.txt and writes initial content to it, representing the implementation of the settings feature.

git add settings.txt stages the newly created file by adding it to the Git index, marking it for inclusion in the next commit.

git commit -m "Add settings feature" creates a commit that records the staged changes in the repository history with a descriptive message, capturing the initial implementation of the settings feature.

14. Create Settings Feature

      git checkout main
      git checkout -b feature-settings
      echo "Settings page created" > settings.txt
      git add settings.txt
      git commit -m "Add settings feature"

At this point, the codebase has diverged, with independent changes introduced on two separate branches, resulting in distinct commit histories.

15. Merge Profile

      git checkout main
      git merge feature-profile

This ensures the main branch is updated first with the profile feature, establishing it as the authoritative baseline before integrating additional feature work.

16. Merge Settings (3-Way)

      git merge feature-settings

Note;
Demonstrates familiarity with the Vim editor by entering a commit message within the Vim interface and exiting correctly using the Esc key followed by :wq to write and quit.

Git generates a merge commit that reconciles and combines the divergent commit histories from both branches.

Outcome:
The main and feature branches have progressed independently, resulting in non-linear histories that require an explicit merge commit to integrate their changes.

MERGE CONFLICT -PART 4

17. Bugfix Changes README

      git checkout -b bugfix-title
      echo "# Team Project Version 2" > README.md
      git add README.md
      git commit -m "Update title in bugfix"

18. Feature Also Changes README

      git checkout main
      git checkout -b feature-title-update
      echo "# Awesome Team Project" > README.md
      git add README.md
      git commit -m "Update title in feature"

19. Trigger Conflict

      git checkout main
      git merge bugfix-title
      git merge feature-title-update

Git is unable to automatically complete the merge process and halts the operation.

Outcome:
Conflicting changes exist, and Git cannot determine which version should be applied, requiring manual conflict resolution by the developer.

20. Resolve Conflict

Open README.md delete the conflict marker and write the following command.

Note: Merge conflicts must be resolved manually by the developer. To do this, open the affected file in the Vim editor using vim README.md, review the conflict markers, and explicitly select or reconcile the appropriate changes.

      # Awesome Team Project Version 2

Then:

      git add README.md
      git commit -m "Resolve merge conflict"

This notifies Git that the merge conflict has been resolved and the finalized version of the file has been saved by staging the corrected file for commit.

SQUASH MERGE -PART 5

21. Create Feature With Many Commits & Squash It

      git checkout -b feature-dashboard

      echo "Dashboard layout" > dashboard.txt
      git add dashboard.txt
      git commit -m "Add dashboard layout"

      echo "Add charts" >> dashboard.txt
      git add dashboard.txt
      git commit -m "Add charts"

      echo "Fix alignment" >> dashboard.txt
      git add dashboard.txt
      git commit -m "Fix alignment"

Now squash:

      git checkout main
      git merge --squash feature-dashboard
      git commit -m "Add dashboard feature"

Squashing is used to consolidate multiple incremental development commits into a single, cohesive commit, ensuring that the main branch history remains clean, concise, and focused on meaningful changes rather than granular development steps.

22. View History

     git log --oneline --graph --all

This is to visually inspect and validate the type of merge performed, ensuring the resulting commit history aligns with the intended integration strategy.

PUSH TO GITHUB -PART 6

23. Create Repo on GitHub

A remote repository must be created and accessible prior to performing a push operation, as Git requires a valid remote endpoint to receive and store the local commits.

DO NOT check: Add README, Add .gitignore

24. Connect Local to GitHub

      git remote add origin https://github.com/YOUR_USERNAME/git-merge-lab.git

This defines the remote endpoint, informing Git of the destination repository to which local commits and branches should be transmitted.

25. Push All Branches

         git push -u origin --all

This Pushes all local branches (such as main, feature, and bugfix) to the remote GitHub repository and establishes upstream tracking relationships, enabling simplified and consistent push and pull operations in subsequent workflows.

Branching, merging, and squashing form the backbone of efficient Git workflows. Branching enables safe experimentation, merging integrates work and squashing keeps history readable. Mastering when and how to use each technique allows engineering teams to scale collaboration while maintaining a stable and maintainable codebase.

For modern DevOps environments where automation, rapid releases, and distributed teams are the norm these practices are not optional, they are foundational to high quality software delivery.

Get Started with Elastic Beanstalk

lotanna obianefo — Sat, 27 Dec 2025 15:01:28 +0000

Amazon Elastic Beanstalk (EB) is a Platform as a Service (PaaS) offering from AWS that simplifies the deployment, management, and scaling of web applications. It allows developers to focus on writing code while AWS handles infrastructure provisioning, load balancing, auto scaling, monitoring, and application health management.

Elastic Beanstalk is ideal for teams that want speed, scalability, and operational simplicity without sacrificing control over the underlying AWS resources.

To create an application

Search for Elastic Beanstalk on the search console and click on create Application.

The console provides a six-step process for creating an application and configuring an environment.

Step1: Configure Environment

In the Environment Tier section, select Web Server Environment.

Provide an Application Name (for example, Fintec_app25).
The system will automatically generate the corresponding Environment Name by appending _env to the application name.
Under the Platform configuration, select Node.js.
The recommended Platform Branch and Version will be automatically populated based on the selection.

Leave all other settings at their default values, then proceed by clicking Next.

Step 2: Configure Service Access

For the Service Role, select Create role if no existing role is available.

During the role creation process, leave all configuration parameters at their default values.
Proceed by clicking Next on both the Select trusted entity and Add permissions steps.
Update the Role name to a unique and descriptive identifier.
Complete the process by clicking Create role.

Just as the previous, For EC2 instance profile select Create role if no existing role is available.

During the role creation process, leave all configuration parameters at their default values.
Proceed by clicking Next on both the Select trusted entity and Add permissions steps
Update the Role name to a unique and descriptive identifier.
Complete the process by clicking Create role.

For the EC2 Key Pair, locate the service using the AWS Management Console search bar.

Create a new key pair by providing a unique and descriptive name.
Leave all other configuration options at their default settings.
Complete the process by selecting Create key pair.

Click Next on Configure service page afterwards.

Step3: Set up networking, database, and tags

Select the default VPC from the available options in the dropdown menu.
Enable Public IP address assignment for the instances.
From the Instance Subnet options, select any available subnet.

Leave all remaining parameters at their default values, then click Next to proceed.

Step4: Configure instance traffic and scaling

Select an existing EC2 Security Group from the dropdown list.
Leave all other configuration parameters at their default values, then click Next to continue.

Step5: Configure updates, monitoring, and logging

Select the appropriate CloudWatch metrics instance and the corresponding CloudWatch metrics environment from their respective fields simultaneously
Leave all other configuration parameters at their default values, then click Next to continue.

Now review the entire steps and click create.

Once the Environment is successfully lunched and the health status shows green and OK. Click the URL link listed for Domain to browse your application.

To access the Elastic Beanstalk–managed EC2 instance shell, navigate to Amazon Elastic Compute Cloud (EC2) and select the relevant Instance ID.
From the Instance Overview page, click Connect, then select Connect again to initiate an in-browser SSH session.

Amazon Elastic Beanstalk strikes a balance between simplicity and control. It abstracts infrastructure management while still allowing developers to customize and scale their applications using familiar AWS services.

For teams seeking fast deployments, built-in scaling, and reduced operational overhead, Elastic Beanstalk remains a powerful and reliable choice.

Prepare your app deployment tools and resources

lotanna obianefo — Tue, 02 Dec 2025 02:40:32 +0000

Many companies are looking for ways to simplify and modernize their DevOps processes. In some cases, their teams already run containerized applications on Azure Kubernetes Service (AKS) but aren’t fully using its advanced features, such as custom service mesh and autoscaling. To reduce complexity and improve efficiency, Azure Container Apps offers a lighter, more scalable, and cost-effective alternative.

By switching to Azure Container Apps, teams can streamline how they deploy and manage containerized applications, cut down on DevOps overhead, and benefit from built-in autoscaling and scale-to-zero features to make better use of resources.

Setup the environment
Install Docker Desktop

Open your web browser and go to the Docker Desktop installation page https://docs.docker.com/desktop/install/windows-install/.

Make sure your computer meets the required system specifications and follow the instructions provided on the website to complete the Docker Desktop installation

Install the .NET Software Development Kit

Open a browser window and then navigate to the .NET download page. https://dotnet.microsoft.com/download.

select the latest Long-Term Support (LTS) version, Double-click the installation file to begin the installation process.
On the .NET SDK Installer window, select Install.

Install Visual Studio Code with Docker and Azure App Service extensions

Open a browser window and then navigate to: https://code.visualstudio.com. In the browser window, select Download for Windows or Mac.

The Download page for Visual Studio Code automatically detects your operating system. It displays the version to download for your operating system, such as Linux, macOS, or Windows.

Wait for the installer file to finish downloading, and then use a file explorer application to navigate to your computer’s downloads folder.
In your file explorer application, select and run the Visual Studio Code installer file.

You can install Visual Studio Code using either the User Installer or System Installer. The User Installer installs Visual Studio Code just for the current user, while the System Installer installs Visual Studio Code for all users. The User Installer is the recommended option for most users.

Select I accept the license agreement, and then continue following the online instructions to complete the installation.
Accept the default options during the remainder of the installation.
Ensure that you have Visual Studio Code open.
On the Activity bar, select Extensions.
In the Search Extensions in Marketplace textbox, enter C#

Entering "C#" filters the list of extensions to show only the extensions that have something to do with C# coding.

In the filtered list of available extensions, select the extension labeled "C# Dev Kit - Official C# extension from Microsoft" that's published by Microsoft.
To install the extension, select Install.

Wait for the installation to complete.

On the EXTENSIONS view, replace C# with docker.
In the filtered list of available extensions, select the extension labeled Docker that's published by Microsoft.
To install the extension, select Install.

Wait for the installation to complete.

On the EXTENSIONS view, replace docker with azure app service.
In the filtered list of available extensions, select the extension labeled Azure App Service that's published by Microsoft.
To install the extension, select Install.

Wait for the installation to complete.

Close Visual Studio Code.

Install Azure CLI and the containerapp extension

Complete the following steps install Azure CLI and the containerapp extension.

Open a browser window, and then navigate to: /cli/azure/install-azure-cli.
In the browser window, follow the instructions for installing/updating Azure CLI for your computer's operating system.

The current version of the Azure CLI is 2.65.0. For information about the latest release, see the release notes. To find your installed version and see if you need to update, run az version. You can run az upgrade to install the latest version.

Open a command line or terminal application, such as Windows Command Prompt.
Sign in to Azure using the az login command.
Follow the prompts to complete the authentication process.
Install the Azure Container Apps extension using the az extension add --name containerapp --upgrade command

Install Microsoft PowerShell

Complete the following steps to install Microsoft PowerShell.

Open a browser window, and then navigate to: /powershell/scripting/install/installing-powershell.
In the browser window, follow the instructions for installing/updating PowerShell for your computer's operating system.

Configure a Resource Group for your Azure resources

Complete the following steps to configure a resource group for your Azure resources.

Open a browser window, and then navigate to the Azure portal: https://portal.azure.com/
Ensure that your Azure account has permission to create resources and assign RBAC permissions. Check the RBAC role(s) assigned to your account before you continue.

The Contributor role isn't able to assign Azure RBAC permissions. We recommend using an account that has been assigned the Owner, Azure account administrator, or Azure co-administrator role for your Azure subscription.

On the top search bar of the Azure portal, in the Search textbox, enter resource group
In the search results, select Resource groups, and then select + Create.
On the Basics tab, configure the resource group as follows:

Subscription: Specify the Azure subscription that you're using

Resource group: Enter RG1

Region: Select Central US.

Select Review + create.

Once validation has passed, select Create.

Configure a Virtual Network and subnets

Complete the following steps to configure a Virtual Network and subnets.

Ensure that you have your Azure portal open in a browser window.
On the top search bar of the Azure portal, in the Search textbox, enter virtual network
In the search results, select Virtual networks.
Select Create virtual network.

On the Basics tab, configure your virtual network as follows:

Subscription: Specify the Azure subscription that you're using
Resource group name: Select RG1
Virtual network name: Enter VNET1
Region: Ensure that Central US is selected.

Select the IP addresses tab.
On the IP addresses tab, under Subnets, select default.

On the Edit subnet page, configure the subnet as follows:

Name: Enter PESubnet
Starting address: Ensure that 10.0.0.0 is specified.
Subnet size: Ensure that /24 (256 addresses) is specified.

Select Save.
On the IP addresses tab, select + Add a subnet.

On the Add a subnet page, configure the subnet as follows:

Name: Enter ACASubnet
Starting address: Ensure that 10.0.4.0 is specified.
Subnet size: Ensure that /23 (512 addresses) is specified.

Select Add.

Select Review + create.
Once validation has passed, select Create.

Wait for the deployment to complete.

Configure Service Bus

Complete the following steps to configure a Service Bus instance.

Ensure that you have your Azure portal open in a browser window.

On the top search bar of the Azure portal, in the Search textbox, enter service bus
In the search results, select Service Bus.
Select Create service bus namespace.
On the Basics tab, configure your Service bus namespace as follows:

Subscription: Ensure that the Azure subscription is selected.
Resource group name: Select RG1
Namespace name: Enter sb-az2003-LT.
Location: Ensure that Central US is selected.
Pricing tier: Select Basic.
Select Review + create.

Once the Validation succeeded message appears, select Create.

Wait for the deployment to complete.

Configure Azure Container Registry

Complete the following steps to configure a Container Registry instance.

Ensure that you have your Azure portal open in a browser window.

On the top search bar of the Azure portal, in the Search textbox, enter container registry
In the search results, select Container registries.
On the Container registries page, select Create container registry or + Create.
On the Basics tab of the Create container registry page, specify the following information:

Subscription: Ensure that the Azure subscription is selected.
Resource group: Select RG1.
Registry name: Enter acraz2003LT2025
Location: Ensure that Central US is selected.
SKU: Select Premium.

The name of your Registry must be unique. Also, the Premium tier is required for private link with private endpoints.

Select Review + create.
Select Create.

After the deployment has completed, open the deployed resource.
On the left-side menu, under Settings, select Networking.
On the Networking page, on the Public access tab, ensure that All networks is selected.

On the left-side menu, under Settings, select Properties.
On the Properties page, select Admin user, and then select Save

Create a WebAPI app and publish to a GitHub repository

Complete the following steps to create a WebAPI app and publish to a GitHub repository.

Open Visual Studio Code.
On the File menu, select Open Folder.
Create a new folder named AZ2003 in a location that is easy to find.

For example, create a folder named AZ2003 on the Windows Desktop.

On the Terminal menu, select New Terminal.

At the terminal command prompt, to create a new ASP.NET Web API project, enter the following command:

       dotnet new webapi --no-https

At the terminal command prompt, run the following dotnet CLI command:

      dotnet build

On the View menu, select Command Palette, and then run the following command:

     .NET: Generate Assets for Build and Debug.

If the command generates an error message, select OK, and then run the command again.

In the root project folder, create a .gitignore file that contains the following information:
```
  [Bb]in/
  [Oo]bj/
```
On the File menu, select Save All.
Open the Source Control view.
Select Publish to GitHub.
If prompted, to enable the GitHub extension to sign in using GitHub, select Allow, and then provide authorization in GitHub.
In Visual Studio Code, select Publish to GitHub public repository.
Ensure that the bin and obj folders are not included in the repository.

Create Docker image and push to Azure Container Registry

Complete the following steps to create a Docker image and push the image to your Azure Container Registry.

Ensure that you have your AZ2003 code project open in Visual Studio Code.
To create a Dockerfile, run the following command in the Command Palette: Docker: Add Docker Files to Workspace.

When prompted, specify the following information:

Application Platform: .NET ASP.NET Core.
Operating System: Linux.
Ports: 5000.
Docker Compose files: No.

At a terminal command prompt, run the following docker CLI command:

      docker build --tag aspnetcorecontainer:latest .

The syntax for the build command is: docker build --tag image name:image tag .

This command builds a container image that is hosted by Docker and accessible using the Docker extension for VS Code.

Wait for the Docker Build command to complete.

Open the Visual Studio Code Command Palette, and then run the following command: Docker Images: Push.

When the command runs, enter the following information:

Select the docker image name that you created: aspnetcorecontainer

Select the image tag that you created: latest

If you see a message stating that no registry is connected, select Connect Registry, and then enter the following information:

Registry provider: Select Azure. Follow the online instructions to verify your Azure account if needed.

Azure subscription: Select the Azure Subscription for this project.

Select your Azure Container Registry resource: acraz2003LT25oct

An image tag is generated: acraz2003LT12oct.azurecr.io/aspnetcorecontainer:latest

To push the image to your Container Registry, press Enter.

The following Docker command is executed:

      docker image push acraz2003cah12oct.azurecr.io/aspnetcorecontainer:latest

Wait for the image to be pushed to your Azure Container Registry.
Open the Source Control view and then Commit and Sync your file updates.

Configure Azure DevOps and a starter Pipeline

Complete the following steps to configure Azure DevOps and a starter Pipeline:

Open the Azure portal.

On the top search bar, in the Search textbox, enter devops
In the search results, select Azure DevOps organizations.
Select My Azure DevOps Organizations.
On the home page of your organization, in the lower-left corner of the page, select Organization settings.
On the left side menu under Security, select Policies.
Ensure that the Allow public projects policy is set to On.
Return to the home page of your organization.
To create a new project, select New project.
If you created a new organization, you may see the Create a project to get started page.

On the Create new project page, specify the following information:

Project name: Project1
Description: AZ-2003 project
Visibility: Public

On the Create new project page, select Create project.
On the left-side menu, select Repos.
Under Import a repository, select Import.
On the Import a Git repository page, enter the URL for the GitHub repository you created for your code project, and then select Import.

Your repository URL should be similar to the following example:

https://github.com/your-account/AZ2003

On the left-side menu, select Pipelines.
Select Create Pipeline.
Select Azure Repos Git.
On the Select a repository page, select Project1.
Select Starter pipeline.
Under Save and Run, select Save, and then select Save.

To rename the pipeline to Pipeline1, complete the following steps:

On the left-side menu, select Pipelines.
To the right of the Project1 pipeline, select More options, and then select Rename/move.
In the Rename/move pipeline dialog, under Name, enter Pipeline1 and then select Save.

Don't run the pipeline now. You will configure this pipeline during the project exercise.

Deploy a self-hosted Windows agent

For an Azure Pipeline to build and deploy Windows, Azure, and other Visual Studio solutions you need at least one Windows agent in the host environment.

Complete the following steps to deploy a self-hosted Windows agent:

Ensure that you're signed-in to Azure DevOps with the user account you're using for your Azure DevOps organization.
From the home page of your organization, open your user settings, and then select Personal access tokens.
To create a personal access token, select + New Token.
Under Name, enter AZ2003.
At the bottom of the Create a new personal access token window, to see the complete list of scopes, select Show all scopes.
For the custom designed scope, select Agent Pools (Read & manage) and Deployment Groups (Read & manage).
Ensure that all the other boxes are cleared.
Select Create.

On the Success page, to copy the token, select Copy to clipboard.

You will use this token when you configure the agent.
Ensure that you're signed into Azure DevOps as the Azure DevOps organization owner.
Select your DevOps organization, and then select Organization settings.
On the left side menu under Pipelines, select Agent pools.
If the Get the agent dialog box opens, skip to the next step.

If a list of Agent pools is displayed, complete the following steps:

To select the default pool, select Default.

If the Default pool doesn't exist, select Add pool, and then enter the following information:
Under Pool type, select Self-hosted.
Under Name, enter default
Select Create.

To open the pool that you just created, select default.
Select Agents, and then select New agent

On the Get the agent dialog box, complete the following steps:

Select the Windows tab.
On the left pane, select the processor architecture of the installed Windows OS version on your machine.
The x64 agent version is intended for 64-bit Windows, whereas the x86 version is intended for 32-bit Windows.
On the right pane, select Download.
Follow the instructions to download the agent.

Use File Explorer to create the following folder location for the agent:
C:\agents

Unpack the agent zip file into the directory you created.
Open PowerShell as an Administrator.
Navigate to the "C:\agents" directory, and then enter the following PowerShell command:
```
       .\config
```

Respond to the configuration prompts as follows:

Enter server URL: enter the URL for your DevOps organization. Enter authentication type (press enter for PAT) >: press Enter.
Enter personal access token >: Paste-in the personal access token that you copied to the clipboard earlier.
Enter agent pool (press enter for default): press Enter.
Enter agent name (press enter for YOUR-PC-NAME), enter az2003-agent
Enter work folder (press enter for _work) >: press Enter.
Enter run agent as service? (Y/N) (press enter for N) >: enter Y
Enter enable SERVICE_SID_TYPE_UNRESTRICTED for agent service (Y/N) (press enter for N) >: enter Y
Enter User account to use for the service (press enter for NT AUTHORITY\NETWORK SERVICE) >: press Enter.
Enter whether to prevent service starting immediately after configuration is finished? (Y/N) (press enter for N) >: press Enter.

A message informing you that the agent started successfully is displayed.

Now you're now ready to begin your project.

Manage revisions in Azure Container Apps

lotanna obianefo — Mon, 01 Dec 2025 02:04:26 +0000

Azure Container Apps includes a built-in revision management system that allows you to track, control, and roll back application updates with ease. Each time you deploy new code or update configuration settings, Azure Container Apps automatically generates a new immutable revision. These revisions enable precise traffic management, safe deployments, blue-green or canary release strategies, and quick rollback in case of failures. By leveraging revision modes, traffic-splitting, and version history, teams can maintain application stability while continuously delivering updates in a controlled and observable manner.

Set revision management to multiple

In the Azure portal, open your container app resource.

On the left side menu, under Application, select Revisions and replicas.
At the top of the Revisions and replicas page, select Choose revision mode.
To switch from single to multi-revision mode, select Confirm.
On the Revisions and replicas page, wait for the Revision Mode setting to update.

The Revision Mode will be set to Multiple after the update. Also, on the left-side menu, the section title changes from Application to Revisions.

Create a new revision with a v2 suffix

In the Azure portal, ensure that you have the Revisions and replicas page of your container app resource open.

At the top of the page, select + Create new revision.

On the Create and deploy new revision page, complete the following steps:

Name / suffix: Enter v2

Under Container image, select your container image. aca-az2003. Select Create. Wait for the deployment to be completed.

Configure labels on the revisions

On the left-side menu, under Settings, select Ingress.
If Ingress isn't enabled, select Enabled.
On the Ingress page, specify the following information:

Ingress traffic: select Accepting traffic from anywhere.

Ingress type: select HTTP.

Client certificate mode: ensure that Ignore is selected.

Transport: ensure that Auto is selected.

Insecure connections: ensure that Allowed is NOT checked.

Target port: enter 5000

IP Security Restrictions Mode: ensure that Allow all traffic is selected.

At the bottom of the Ingress page, select Save, and then wait for the update to complete.

On the left-side menu, under Revisions, select Revisions and replicas.
For the v2 revision, under Label, enter updated
For the other revision, enter current

At the top of the page, select Save.

Configure a traffic percentage on the revisions

Ensure that you have the Revisions and replicas page open.

For the v2 revision, under Traffic, enter 25 as the percentage.
For the other revision, under Traffic, enter 75 as the percentage.

At the top of the page, select Save.

Verify your work

Ensure that you have your Container App open in the Azure portal.

On the left-side menu, under Application, select Revisions and replicas.

Verify that your revisions are configured as follows:

Azure Container Apps uses a revision-based deployment model that creates a new immutable revision each time you update a container app’s configuration or container image. This enables safe, controlled rollouts without disrupting existing traffic. Revisions can be activated, deactivated, or rolled back as needed, allowing teams to test new versions, split traffic for gradual releases, and maintain high availability.

Configure continuous integration by using Azure Pipelines

lotanna obianefo — Thu, 27 Nov 2025 22:38:51 +0000

Continuous Integration (CI) enables development teams to automatically build, test, and validate application changes whenever code is committed to a repository. In Azure environments, Azure Pipelines provides a robust CI service that integrates seamlessly with platforms like GitHub and Azure Repos. By configuring CI with Azure Pipelines, teams can automate the process of building container images, run tests, and securely pushing updated images to Azure Container Registry (ACR). This automated workflow ensures consistent deployments, reduces manual errors, and supports a scalable DevOps lifecycle for applications running on Azure Container Apps.

Configure Pipeline1 to use the self-hosted agent pool

Open a browser window, navigate to https://dev.azure.com, and then open your Azure DevOps organization.
On your Azure DevOps page, to open your DevOps project, select Project1.
In the left-side menu, select Pipelines.
Select Pipeline1, and then select Edit.

To use the self-hosted agent pool, update the azure-pipelines.yml file as shown in the following example:

           trigger:
            - main

           pool:
           name: default

           steps:

Recall that the pool section specifies the agent pool to use for the pipeline. The name property specifies the name of the agent pool. In this case, the name is default, which is the pool you configured as a self-hosted agent pool.

Under Validate and save, select Save without validating.
Enter a commit message, and then select Save.

Configure Pipeline1 with an Azure Container Apps deployment task

Ensure that you have Pipeline1 open for editing.

On the right side under Tasks, in the Search tasks field, enter azure container
In the filtered list of tasks, select Azure Container Apps Deploy
Under Azure Resource Manager connection, select the Subscription you're using, and then select Authorize.

In the Azure portal tab, open your Container App resource, and then open the Containers page.

Use the information on the Containers page to configure the following Pipeline1 Task information:

Docker Image to Deploy: Registry/Image:Image tag
Azure Container App name: Name

Configure the following Pipeline1 Task information:

Azure Resource group name: RG1

On the Azure Container Apps Deploy page, select Add.

The Yaml file for your pipeline should now include the AzureContainerApps tasks as follows:

                 trigger:
                   - main
                 pool:
                   name: default
                 steps:
                   - task: AzureContainerApps@1
                     inputs:
                       azureSubscription: '<Subscription>(<Subscription ID>)'
                       imageToDeploy: '<Registry>/<Image>:<Image tag>' from Container App resource
                       containerAppName: '<Name>' from Container App resource 
                       resourceGroup: '<resource group name>'

Select Validate and save, and then select Save again to commit directly to the main branch.

You most likely have an indentation error. Review and correct the indentation to resolve the issue.

The contents of the YAML file must be formatted correctly, including indentation. If you encounter an error, review the YAML file and correct any indentation issues.

Navigate back to the main page of your pipeline.

Run the Pipeline1 deployment task

Ensure that you have Pipeline1 open in Azure DevOps.

On the Runs tab of the Pipeline1 page, select Run pipeline.
A Run pipeline page opens to display the associated job.
Select Run.

The Jobs section displays job status, which progresses from Queued to Waiting.

It can take a couple minutes for the status to transition from Queued to Waiting.

If a 'Permission needed' message is displayed ("This pipeline needs permission to access 2 resources before this run can continue"), select View and then provide the required permissions.

Monitor the status of the run operation and verify that the run is successful.

Verify your work

In this task, you examine your pipeline and container app to verify successful pipeline runs.

Ensure that you have Project1 open in Azure DevOps.

On the left side menu, select Pipelines, and then select Pipeline1.
The Runs tab displays individual runs that can be selected to review details.
Open your Azure portal, and then open your Container App.
On the left side menu, select Activity Log.
Verify that a Create or Update Container App operation succeeded as a result of running your pipeline.

Notice that the Event initiated by column on the right shows your Project1 as the source.

In summary, configuring continuous integration (CI) with Azure Pipelines enables automated building, testing, and deployment of containerized applications to Azure Container Apps. By integrating your source code repository with Azure Pipelines, each commit or pull request can automatically trigger a pipeline that builds the container image, pushes it to Azure Container Registry (ACR), and deploys the updated image to Azure Container Apps. This setup ensures consistent application delivery, reduces manual overhead, and improves deployment reliability. Using YAML-based pipelines also provides version-controlled build definitions, promoting repeatability and alignment with DevOps best practices.

Create and configure a container app in Azure Container Apps

lotanna obianefo — Thu, 27 Nov 2025 03:44:52 +0000

Creating and configuring a container app in Azure Container Apps involves deploying a containerized application into a fully managed, serverless environment designed for cloud-native workloads. Azure Container Apps abstracts away the need to manage Kubernetes clusters while still providing key capabilities such as autoscaling, revisions, traffic splitting, and secure integration with supporting services like Azure Container Registry.

In this process, you define the container image to run, configure environment settings such as ingress and scaling rules, and connect the app to the required resources and identities. This setup enables reliable, scalable, and efficient execution of microservices and event-driven applications without managing underlying infrastructure.

Create a container app that uses an ACR image

Open your Azure portal.

On the portal menu, select + Create a resource.
On the top search bar, in the Search textbox, enter container app
In the search results under Services, select Container Apps.
Select Create.
On the Basics tab, specify the following:

Subscription: Specify the Azure subscription that you're using.
Resource group: RG1
Container app name: aca-az2003
Region: Select the Region specified for VNET1 (Central US).

The container app needs to be in the same region/location as the virtual network so you can choose VNET1 for the managed environment. For this guided project, keep all of your resources in the region/location specified for your resource group.

Container Apps Environment: Select Create new.

On the Create Container Apps Environment page, select the Networking tab, and then specify the following:

Use your own virtual network: Select Yes.
Virtual network: Select VNET1.
Infrastructure subnet: ACASubnet.

If the ACASubnet subnet is not listed, open your virtual network resource, adjust the subnet address range to 10.0.2.0/23 and retry the steps to create the Container App.

On the Create Container Apps Environment page, select Create.
On the Create Container App page, select the Container tab, and then specify the following:

Use quickstart image: Ensure that this setting is not selected. If it is selected, uncheck this setting.
Name: Enter aca-az2003
Image source: Ensure that Azure Container Registry is selected.
Registry: Select your container registry.: acraz2003cah.azurecr.io
Image: Select aspnetcorecontainer
Image tag: Select latest

Select Review + create.

Once verification has Passed, select Create.

Wait for the deployment to complete.

Configure the container app to authenticate using the user assigned identity

On the Azure portal, open the Container App that you created.
Under Security, select Identity.
Select the tab for User assigned.
Select Add user assigned managed identity.
On the Add user assigned managed identity page, select uai-az2003, and then select Add.

Configure a connection between the container app and Service Bus

On the Azure portal, ensure that you have your Container App open.
Under Settings, select Service Connector (Preview).
Select Connect to your Services.
On the Create connection page, specify the following:

Service type: Select Services Bus.
Client type: Select .NET.

Select Next: Authentication.

On the Authentication tab, select User assigned managed identity.
Ensure that the correct subscription and user assigned managed identity are selected.

Subscription: The Azure subscription that you're using.
User assigned managed identity: uai-az2003

To change tabs, select Review + Create.

Once the Validation passed message appears, select Create.

Wait for the connection to be created.

It can take a minute before the Service Connector page updates with the new connection.

Configure HTTP scale rules

Ensure that your Container App is open in the portal.

On the left-side menu under Application, select Revisions and replicas.

Notice the Name assigned to your active revision.

On the left-side menu under Application, select Containers.
To the right of Based on revision, ensure that your active revision is selected.
At the top of the page, select Edit and deploy.

At the bottom of the page, select Next: Scale.

Configure the Min / max replicas as follows:

Set Min replicas: 0
Set Max replicas: 2
Under Scale rule, select + Add.

On the Add scale rule page, specify the following:

Rule name: Enter scalerule-http
Type: Select HTTP scaling.
Concurrent requests: Set the value to 10,000.

On the Add scale rule page, select Add.

On the Create and deploy new revision page, select Create.

Ensure that your new scale rule is displayed.

Verify your work

Now you have to verify that your configuration meets the specified requirements.

In the Azure portal, ensure that your Container App resource is open.
On the left-side menu, under Settings, select Deployment.
At the top of the page, ensure that the Continuous deployment tab is selected.

Verify that the expected Registry settings are reported:

Repository source: Azure Container Registry
Registry: the name of your Container Registry; acraz2003LT2025
Image: aca-az2003

Open your Container Apps Environment resource.

Verify that your Container App uses the proper subnet as follows:

On the Overview page, verify that Virtual Network is set to VNET1.
On the Overview page, verify that Infrastructure subnet is set to ACASubnet.

Verify that the targetService properties match the specified configuration.

To verify your HTTP scale rule, you would need to run testing software that's able to simulate 10,000 concurrent HTTP requests and ensure that container replicas are created.

Configure Azure Container Registry for a secure connection with Azure Container Apps

lotanna obianefo — Thu, 27 Nov 2025 00:57:38 +0000

Configuring Azure Container Registry (ACR) for a secure connection with Azure Container Apps is a crucial step in ensuring that your containerized applications are deployed safely and efficiently. This process involves setting up permissions and authentication so Azure Container Apps can securely pull container images from ACR without exposing credentials. By integrating ACR with managed identities or workload identities, teams can streamline deployments, improve security, and maintain a clean, automated DevOps workflow.

Configure a user-assigned managed identity

Open your Azure portal.
On the portal menu, select + Create a resource.
On the Create a resource page, in the Search services and marketplace text box, enter managed identity
In the filtered list of resources, select User Assigned Managed Identity.
On the User Assigned Managed Identity page, select Create.
On the Create User Assigned Managed Identity page, specify the following information:

Subscription: Specify the Azure subscription that you're using for this guided project.
Resource group: RG1
Region: Central US
Name: uai-az2003
Select Review + create.
Select Create.

Configure Container Registry with AcrPull permissions for the managed identity

In the Azure portal, open your Container Registry resource that was already create.
On the left-side menu, select Access Control (IAM).
On the Access Control (IAM) page, select Add role assignment.
Search for the AcrPull role, and then select AcrPull.

Note: This configuration can also be applied when assigning the AcrPush role.

Select Next.
On the Members tab, to the right of Assign access to, select Managed identity.
Select + Select members.
On the Select managed identities page, under Managed identity, select User-assigned managed identity, and then select the user-assigned managed identity created for this project.

For example: uai-az2003.

On the Select managed identities page, select Select.
On the Members tab of the Add role assignment page, select Review + assign.
On the Review + assign tab, select Review + assign.

Wait for the role assignment to be added.

Configure Container Registry with a private endpoint connection

Ensure that your Container Registry resource is open in the portal.

Under Settings, select Networking.
On the Private access tab, select + Create a private endpoint connection.
On the Basics tab, under Project details, specify the following information:

Subscription: Specify the Azure subscription that you're using for this project.
Resource group: RG1
Name: pe-acr-az2003
Region: Ensure that Central US is selected.
Select Next: Resource.

On the Resource tab, ensure the following information is displayed:

Subscription: Ensure that the Azure subscription that you're using for this project is selected.
Resource type: Ensure that Microsoft.ContainerRegistry/registries is selected.
Resource: Ensure that the name of your registry is selected.
Target sub-resource: Ensure that registry is selected.
Select Next: Virtual Network.

On the Virtual Network tab, under Networking, ensure the following information is displayed:

Virtual network: Ensure that VNET1 is selected
Subnet: Ensure that PESubnet is selected.
Select Next: DNS.

On the DNS tab, under Private DNS Integration, ensure the following information is displayed:

Integrate with private DNS zone: Ensure that Yes is selected.
Private DNS Zone: Notice that (new) privatelink.azurecr.io is specified.

Select Next: Tags and then Select Next: Review + create.

On the Review + create tab, when you see the Validation passed message, select Create.

Wait for the deployment to complete.

Verify your work
In this task, you verify that your configuration meets the specified requirements.

In the Azure portal, open your Container Registry resource.
On the Access Control (IAM) page, select Role assignments.
Verify that the role assignments list shows the AcrPull role assigned to the User-assigned Managed Identity resource.
On the left-side menu, under Settings, select Networking.
On the Networking page, select the Private access tab.
Under Private endpoint, select the private endpoint that you created.

For example, select per-acr-az2003

On the Private endpoint page, under Settings, select DNS configuration.
Verify the following DNS setting:
Private DNS zone: set to privatelink.azurecr.io.
On the left-side menu, select Overview.
Verify the following setting:
Virtual network/subnet: set to VNET1/PESubnet.

To securely deploy containerized workloads in Azure Container Apps, you must establish a protected connection to Azure Container Registry (ACR), where container images are stored.

This configuration ensures that only authorized resources can pull images from the registry.

Ingest data with a pipeline in Microsoft Fabric

lotanna obianefo — Sat, 04 Oct 2025 20:52:07 +0000

A data lakehouse is a common analytical data store for cloud-scale analytics solutions. One of the core tasks of a data engineer is to implement and manage the ingestion of data from multiple operational data sources into the lakehouse. In Microsoft Fabric, you can implement extract, transform, and load (ETL) or extract, load, and transform (ELT) solutions for data ingestion through the creation of pipelines.

Fabric also supports Apache Spark, enabling you to write and run code to process data at scale. By combining the pipeline and Spark capabilities in Fabric, you can implement complex data ingestion logic that copies data from external sources into the OneLake storage on which the lakehouse is based, and then uses Spark code to perform custom data transformations before loading it into tables for analysis.

Create a workspace

Before working with data in Fabric, create a workspace with the Fabric trial enabled.

Navigate to the Microsoft Fabric home page at https://app.fabric.microsoft.com/home?experience=fabric-developer in a browser and sign in with your Fabric credentials.
In the menu bar on the left, select Workspaces (the icon looks similar to 🗇).
Create a new workspace with a name of your choice, selecting a licensing mode in the Advanced section that includes Fabric capacity. When your new workspace opens, it should be empty.

Create a lakehouse

Now that you have a workspace, it’s time to create a data lakehouse into which you will ingest data.

On the menu bar on the left, select Create. In the New page, under the Data Engineering section, select Lakehouse. Give it a unique name of your choice.
After a minute or so, a new lakehouse with no Tables or Files will be created.
On the Explorer pane on the left, in the … menu for the Files node, select New subfolder and create a subfolder named new_data.

Create a pipeline

A simple way to ingest data is to use a Copy Data activity in a pipeline to extract the data from a source and copy it to a file in the lakehouse.

On the Home page for your lakehouse, select Get data and then select New data pipeline, and create a new data pipeline named Ingest Sales Data.
If the Copy Data wizard doesn’t open automatically, select Copy Data > Use copy assistant in the pipeline editor page.
In the Copy Data wizard, on the Choose data source page, type HTTP in the search bar and then select HTTP in the New sources section.

In the Connect to data source pane, enter the following settings for the connection to your data source:

(i) URL: https://raw.githubusercontent.com/MicrosoftLearning/dp-data/main/sales.csv
(ii) Connection: Create new connection
(iii) Connection name: Specify a unique name
(iv) Data gateway: (none)
(v) Authentication kind: Anonymous

Select Next. Then ensure the following settings are selected:

(i) Relative URL: Leave blank
(ii) Request method: GET
(iii) Additional headers: Leave blank
(iv) Binary copy: Unselected
(v) Request timeout: Leave blank
(vi) Max concurrent connections: Leave blank

Select Next, and wait for the data to be sampled and then ensure that the following settings are selected:

(i) File format: DelimitedText
(ii) Column delimiter: Comma (,)
(iii) Row delimiter: Line feed (\n)
(iv) First row as header: Selected
(v) Compression type: None

Select Preview data to see a sample of the data that will be ingested. Then close the data preview and select Next. On the Connect to data destination page, set the following data destination options, and then select Next:

(i) Root folder: Files
(ii) Folder path name: new_data
(iii) File name: sales.csv
(iv) Copy behavior: None

Set the following file format options and then select Next:

(i) File format: DelimitedText
(ii) Column delimiter: Comma (,)
(iii) Row delimiter: Line feed (\n)
(iv) Add header to file: Selected
(v) Compression type: None

On the Copy summary page, review the details of your copy operation and then select Save + Run.
A new pipeline containing a Copy Data activity is created, as shown here:
When the pipeline starts to run, you can monitor its status in the Output pane under the pipeline designer. Use the ↻ (Refresh) icon to refresh the status, and wait until it has succeeeded.
In the menu bar on the left, select your lakehouse.
On the Home page, in the Explorer pane, expand Files and select the new_data folder to verify that the sales.csv file has been copied.

Create a notebook

On the Home page for your lakehouse, in the Open notebook menu, select New notebook.

After a few seconds, a new notebook containing a single cell will open. Notebooks are made up of one or more cells that can contain code or markdown (formatted text).

Select the existing cell in the notebook, which contains some simple code, and then replace the default code with the following variable declaration.

table_name = "sales"

In the … menu for the cell (at its top-right) select Toggle parameter cell. This configures the cell so that the variables declared in it are treated as parameters when running the notebook from a pipeline.
Under the parameters cell, use the + Code button to add a new code cell. Then add the following code to it:

from pyspark.sql.functions import *

# Read the new sales data
df = spark.read.format("csv").option("header","true").load("Files/new_data/*.csv")

## Add month and year columns
df = df.withColumn("Year", year(col("OrderDate"))).withColumn("Month", month(col("OrderDate")))

# Derive FirstName and LastName columns
df = df.withColumn("FirstName", split(col("CustomerName"), " ").getItem(0)).withColumn("LastName", split(col("CustomerName"), " ").getItem(1))

# Filter and reorder columns
df = df["SalesOrderNumber", "SalesOrderLineNumber", "OrderDate", "Year", "Month", "FirstName", "LastName", "EmailAddress", "Item", "Quantity", "UnitPrice", "TaxAmount"]

# Load the data into a table
df.write.format("delta").mode("append").saveAsTable(table_name)

This code loads the data from the sales.csv file that was ingested by the Copy Data activity, applies some transformation logic, and saves the transformed data as a table - appending the data if the table already exists.

Verify that your notebooks looks similar to this, and then use the ▷ Run all button on the toolbar to run all of the cells it contains.

Since this is the first time you’ve run any Spark code in this session, the Spark pool must be started. This means that the first cell can take a minute or so to complete.

When the notebook run has completed, in the Explorer pane on the left, in the … menu for Tables select Refresh and verify that a sales table has been created.
In the notebook menu bar, use the ⚙️ Settings icon to view the notebook settings. Then set the Name of the notebook to Load Sales and close the settings pane.
In the hub menu bar on the left, select your lakehouse.
In the Explorer pane, refresh the view. Then expand Tables, and select the sales table to see a preview of the data it contains.

Modify the pipeline

Now that you’ve implemented a notebook to transform data and load it into a table, you can incorporate the notebook into a pipeline to create a reusable ETL process.

In the hub menu bar on the left select the Ingest Sales Data pipeline you created previously.
On the Activities tab, in the All activities list, select Delete data. Then position the new Delete data activity to the left of the Copy data activity and connect its On completion output to the Copy data activity, as shown here:
Select the Delete data activity, and in the pane below the design canvas, set the following properties:

General
(i) Name: Delete old files

Source
(i) Connection: Your lakehouse
(ii) File path type: Wildcard file path
(iii) Folder path: Files / new_data
(iv) Wildcard file name: .csv
*(v)** Recursively: Selected

Logging settings
(i) Enable logging: Unselected

These settings will ensure that any existing .csv files are deleted before copying the sales.csv file.

Select the Copy data activity and then connect its On Completion output to the Notebook activity as shown here.
Select the Notebook activity, and then in the pane below the design canvas, set the following properties:

General

(i) Name: Load Sales notebook

Settings
(i) Notebook: Load Sales

Base parameters:
Add a new parameter with the following properties:
(i) Name: table_name

(ii) Type: String

(iii) Value: new_sales

The table_name parameter will be passed to the notebook and override the default value assigned to the table_name variable in the parameters cell.

On the Home tab, use the 🖫 (Save) icon to save the pipeline. Then use the ▷ Run button to run the pipeline, and wait for all of the activities to complete.
In the hub menu bar on the left edge of the portal, select your lakehouse.
In the Explorer pane, expand Tables and select the new_sales table to see a preview of the data it contains. This table was created by the notebook when it was run by the pipeline.

In this project, you learned how to design and implement a data pipeline in Microsoft Fabric. The solution ingests data from an external source into a lakehouse using a pipeline, followed by a Spark notebook that transforms the ingested data and loads it into a structured table for further analysis.

Analyze data with Apache Spark in Fabric

lotanna obianefo — Fri, 03 Oct 2025 13:03:15 +0000

Data-driven organizations rely heavily on the ability to process, transform, and analyze large datasets efficiently. Microsoft Fabric provides a unified platform for analytics, and at its core is Apache Spark, a powerful distributed computing engine. Spark in Fabric enables developers, data engineers, and analysts to analyze massive volumes of structured and unstructured data in real time, all within an integrated environment.

In this project you will ingest data into the Fabric lakehouse and use PySpark to read and analyze the data.

Create a workspace

Before working with data in Fabric, create a workspace in a tenant with the Fabric capacity enabled.

Navigate to the Microsoft Fabric home page at https://app.fabric.microsoft.com/home?experience=fabric-developer in a browser and sign in with your Fabric credentials.
In the menu bar on the left, select Workspaces (the icon looks similar to 🗇). You can also find it on the home page.
Create a new workspace with a name of your choice, selecting a licensing mode in the Advanced section that includes Fabric capacity.

Create a lakehouse and upload files

Now that you have a workspace, it’s time to create a data lakehouse for your data.

On the menu bar on the left, select Create. In the New page, under the Data Engineering section, select Lakehouse. Give it a unique name of your choice. After a minute or so, a new lakehouse will be created: View the new lakehouse, and note that the Lakehouse explorer pane on the left enables you to browse tables and files in the lakehouse:
You can now ingest data into the lakehouse. There are several ways to do this, but for now you’ll download a folder of text files to your local computer (or VM if applicable) and then upload them to your lakehouse.
Download the datafiles from https://github.com/MicrosoftLearning/dp-data/raw/main/orders.zip.
Extract the zipped archive and verify that you have a folder named orders which contains three CSV files: 2019.csv, 2020.csv, and 2021.csv.
Return to your new lakehouse. In the Explorer pane, next to the Files folder select the … menu, and select Upload and Upload folder. Navigate to the orders folder on your local computer (or lab VM if applicable) and select Upload.
After the files have been uploaded, expand Files and select the orders folder. Check that the CSV files have been uploaded, as shown here:

Create a notebook

You can now create a Fabric notebook to work with your data. Notebooks provide an interactive environment where you can write and run code.

On the menu bar on the left, select Create. In the New page, under the Data Engineering section, select Notebook.
Fabric assigns a name to each notebook you create, such as Notebook 1, Notebook 2, etc. Click the name panel above the Home tab on the menu to change the name to something more descriptive. A new notebook named data_Notebook is created and opened.
Select the first cell (which is currently a code cell), and then in the top-right tool bar, use the M↓ button to convert it to a markdown cell. The text contained in the cell will then be displayed as formatted text.
Use the 🖉 (Edit) button to switch the cell to editing mode, then modify the markdown as shown below.

# Sales order data exploration
Use this notebook to explore sales order data

When you have finished, click anywhere in the notebook outside of the cell to stop editing it.

Create a DataFrame

Now that you have created a workspace, a lakehouse, and a notebook you are ready to work with your data. You will use PySpark, which is the default language for Fabric notebooks, and the version of Python that is optimized for Spark.

Fabric notebooks support multiple programming languages including Scala, R, and Spark SQL

Select your new workspace from the left bar. You will see a list of items contained in the workspace including your lakehouse and notebook.
Select the lakehouse to display the Explorer pane, including the orders folder.
From the top menu, select Open notebook, Existing notebook, and then open the notebook you created earlier. The notebook should now be open next to the Explorer pane. Expand Lakehouses, expand the Files list, and select the orders folder. The CSV files that you uploaded are listed next to the notebook editor.
From the … menu for 2019.csv, select Load data > Spark. The following code is automatically generated in a new code cell:
Select ▷ Run cell to the left of the cell to run the code.

You can hide the Explorer panes on the left by using the « icons. This gives more space for the notebook.

The first time you run Spark code, a Spark session is started. This can take a few seconds or longer. Subsequent runs within the same session will be quicker.

When the cell code has completed, review the output below the cell, which should look like this:
The output shows data from the 2019.csv file displayed in columns and rows. Notice that the column headers contain the first line of the data. To correct this, you need to modify the first line of the code as follows:

df = spark.read.format("csv").option("header","false").load("Files/orders/2019.csv")

Run the code again, so that the DataFrame correctly identifies the first row as data. Notice that the column names have now changed to _c0, _c1, etc.
Descriptive column names help you make sense of data. To create meaningful column names, you need to define the schema and data types. You also need to import a standard set of Spark SQL types to define the data types. Replace the existing code with the following:

from pyspark.sql.types import *

orderSchema = StructType([
StructField("SalesOrderNumber", StringType()),
StructField("SalesOrderLineNumber", IntegerType()),
StructField("OrderDate", DateType()),
StructField("CustomerName", StringType()),
StructField("Email", StringType()),
StructField("Item", StringType()),
StructField("Quantity", IntegerType()),
StructField("UnitPrice", FloatType()),
StructField("Tax", FloatType())
])

df = spark.read.format("csv").schema(orderSchema).load("Files/orders/2019.csv")

display(df)

Run the cell and review the output
This DataFrame includes only the data from the 2019.csv file. Modify the code so that the file path uses a * wildcard to read all the files in the orders folder:

from pyspark.sql.types import *

df = spark.read.format("csv").schema(orderSchema).load("Files/orders/*.csv")

display(df)

When you run the modified code, you should see sales for 2019, 2020, and 2021. Only a subset of the rows is displayed, so you may not see rows for every year.

You can hide or show the output of a cell by selecting … next to the result. This makes it easier to work in a notebook.

Explore data in a DataFrame

The DataFrame object provides additional functionality such as the ability to filter, group, and manipulate data.

Filter a DataFrame

Add a code cell by selecting + Code which appears when you hover the mouse above or below the current cell or its output. Alternatively, from the ribbon menu select Edit and + Add code cell below.
The following code filters the data so that only two columns are returned. It also uses count and distinct to summarize the number of records:

customers = df['CustomerName', 'Email']

print(customers.count())
print(customers.distinct().count())

display(customers.distinct())

Run the code, and examine the output:

(i) The code creates a new DataFrame called customers which contains a subset of columns from the original df DataFrame. When performing a DataFrame transformation you do not modify the original DataFrame, but return a new one.

Another way of achieving the same result is to use the select method:

customers = df.select("CustomerName", "Email")

The DataFrame functions count and distinct are used to provide totals for the number of customers and unique customers.

Modify the first line of the code by using select with a where function as follows:

customers = df.select("CustomerName", "Email").where(df['Item']=='Road-250 Red, 52')
print(customers.count())
print(customers.distinct().count())

display(customers.distinct())

Run the modified code to select only the customers who have purchased the Road-250 Red, 52 product. Note that you can “chain” multiple functions together so that the output of one function becomes the input for the next. In this case, the DataFrame created by the select method is the source DataFrame for the where method that is used to apply filtering criteria.

Aggregate and group data in a DataFrame

Add a code cell, and enter the following code:

productSales = df.select("Item", "Quantity").groupBy("Item").sum()

display(productSales)

Run the code. You can see that the results show the sum of order quantities grouped by product. The groupBy method groups the rows by Item, and the subsequent sum aggregate function is applied to the remaining numeric columns - in this case, Quantity.
Add another code cell to the notebook, and enter the following code:

from pyspark.sql.functions import *

yearlySales = df.select(year(col("OrderDate")).alias("Year")).groupBy("Year").count().orderBy("Year")

display(yearlySales)

Run the cell. Examine the output. The results now show the number of sales orders per year:

(i) The import statement enables you to use the Spark SQL library.
(ii) The select method is used with a SQL year function to extract the year component of the OrderDate field.
(iii) The alias method is used to assign a column name to the extracted year value.
(iv) The groupBy method groups the data by the derived Year column.
(v) The count of rows in each group is calculated before the orderBy method is used to sort the resulting DataFrame.

Use Spark to transform data files

A common task for data engineers and data scientists is to transform data for further downstream processing or analysis.

Use DataFrame methods and functions to transform data

Add a code cell to the notebook, and enter the following:

from pyspark.sql.functions import *

# Create Year and Month columns
transformed_df = df.withColumn("Year", year(col("OrderDate"))).withColumn("Month", month(col("OrderDate")))

# Create the new FirstName and LastName fields
transformed_df = transformed_df.withColumn("FirstName", split(col("CustomerName"), " ").getItem(0)).withColumn("LastName", split(col("CustomerName"), " ").getItem(1))

# Filter and reorder columns
transformed_df = transformed_df["SalesOrderNumber", "SalesOrderLineNumber", "OrderDate", "Year", "Month", "FirstName", "LastName", "Email", "Item", "Quantity", "UnitPrice", "Tax"]

# Display the first five orders
display(transformed_df.limit(5))

Run the cell. A new DataFrame is created from the original order data with the following transformations:

(i) Year and Month columns added, based on the OrderDate column.
(ii) FirstName and LastName columns added, based on the CustomerName column.
(iii) The columns are filtered and reordered, and the CustomerName column removed.

Review the output and verify that the transformations have been made to the data.

You can use the Spark SQL library to transform the data by filtering rows, deriving, removing, renaming columns, and applying other data modifications.

Save the transformed data

At this point you might want to save the transformed data so that it can be used for further analysis.

Parquet is a popular data storage format because it stores data efficiently and is supported by most large-scale data analytics systems. Indeed, sometimes the data transformation requirement is to convert data from one format such as CSV, to Parquet.

To save the transformed DataFrame in Parquet format, add a code cell and add the following code:

transformed_df.write.mode("overwrite").parquet('Files/transformed_data/orders')

print ("Transformed data saved!")

Run the cell and wait for the message that the data has been saved. Then, in the Explorer pane on the left, in the … menu for the Files node, select Refresh. Select the transformed_data folder to verify that it contains a new folder named orders, which in turn contains one or more Parquet files.

Add a cell with the following code:

orders_df = spark.read.format("parquet").load("Files/transformed_data/orders")
display(orders_df)

Run the cell. A new DataFrame is created from the parquet files in the transformed_data/orders folder. Verify that the results show the order data that has been loaded from the parquet files.

Save data in partitioned files

When dealing with large volumes of data, partitioning can significantly improve performance and make it easier to filter data.

Add a cell with code to save the dataframe, partitioning the data by Year and Month:

orders_df.write.partitionBy("Year","Month").mode("overwrite").parquet("Files/partitioned_data")

print ("Transformed data saved!")

Run the cell and wait for the message that the data has been saved. Then, in the Lakehouses pane on the left, in the … menu for the Files node, select Refresh and expand the partitioned_data folder to verify that it contains a hierarchy of folders named Year=xxxx, each containing folders named Month=xxxx. Each month folder contains a parquet file with the orders for that month.

Add a new cell with the following code to load a new DataFrame from the orders.parquet file:

orders_2021_df = spark.read.format("parquet").load("Files/partitioned_data/Year=2021/Month=*")

display(orders_2021_df)

Run the cell and verify that the results show the order data for sales in 2021. Notice that the partitioning columns specified in the path (Year and Month) are not included in the DataFrame.

Work with tables and SQL

You’ve now seen how the native methods of the DataFrame object enable you to query and analyze data from a file. However, you may be more comfortable working with tables using SQL syntax. Spark provides a metastore in which you can define relational tables.

The Spark SQL library supports the use of SQL statements to query tables in the metastore. This provides the flexibility of a data lake with the structured data schema and SQL-based queries of a relational data warehouse - hence the term “data lakehouse”.

Create a table

Tables in a Spark metastore are relational abstractions over files in the data lake. Tables can be managed by the metastore, or external and managed independently of the metastore.

Add a code cell to the notebook and enter the following code, which saves the DataFrame of sales order data as a table named salesorders:

# Create a new table
df.write.format("delta").saveAsTable("salesorders")

# Get the table description
spark.sql("DESCRIBE EXTENDED salesorders").show(truncate=False)

Run the code cell and review the output, which describes the definition of the new table
In the Explorer pane, in the … menu for the Tables folder, select Refresh. Then expand the Tables node and verify that the salesorders table has been created.

No explicit path is provided, so the files for the table will be managed by the metastore. Also, the table is saved in delta format which adds relational database capabilities to tables. This includes support for transactions, row versioning, and other useful features. Creating tables in delta format is preferred for data lakehouses in Fabric.

In the … menu for the salesorders table, select Load data **> **Spark. A new code cell is added containing code similar to the following:
Run the new code, which uses the Spark SQL library to embed a SQL query against the salesorder table in PySpark code and load the results of the query into a DataFrame.

df = spark.sql("SELECT * FROM [your_lakehouse].salesorders LIMIT 1000")

display(df)

Run SQL code in a cell

While it’s useful to be able to embed SQL statements into a cell containing PySpark code, data analysts often just want to work directly in SQL.

Add a new code cell to the notebook, and enter the following code:

%%sql
SELECT YEAR(OrderDate) AS OrderYear,
SUM((UnitPrice * Quantity) + Tax) AS GrossRevenue
FROM salesorders
GROUP BY YEAR(OrderDate)
ORDER BY OrderYear;

Run the cell and review the results. Observe that:

(i) The %%sql command at the beginning of the cell (called a magic) changes the language to Spark SQL instead of PySpark.
(ii) The SQL code references the salesorders table that you created previously.
(iii) The output from the SQL query is automatically displayed as the result under the cell.

Visualize data with Spark

Charts help you to see patterns and trends faster than would be possible by scanning thousands of rows of data. Fabric notebooks include a built-in chart view but it is not designed for complex charts. For more control over how charts are created from data in DataFrames, use Python graphics libraries like matplotlib or seaborn.

View results as a chart

Add a new code cell, and enter the following code:

%%sql
SELECT * FROM salesorders

Run the code to display data from the salesorders view you created previously. In the results section beneath the cell, select + New chart. Use the Build my own button at the bottom-right of the results section and set the chart settings:

(i) Chart type: Bar chart
(ii) X-axis: Item
(iii) Y-axis: Quantity
(iv) Series Group: leave blank
(v) Aggregation: Sum
(vi) Missing and NULL values: Display as 0
(vii) Stacked: Unselected

Your chart should look similar to this:

Get started with matplotlib

Add a new code cell, and enter the following code:

sqlQuery = "SELECT CAST(YEAR(OrderDate) AS CHAR(4)) AS OrderYear, \
SUM((UnitPrice * Quantity) + Tax) AS GrossRevenue, \
COUNT(DISTINCT SalesOrderNumber) AS YearlyCounts \
FROM salesorders \
GROUP BY CAST(YEAR(OrderDate) AS CHAR(4)) \
ORDER BY OrderYear"
df_spark = spark.sql(sqlQuery)
df_spark.show()

Run the code. It returns a Spark DataFrame containing the yearly revenue and number of orders. To visualize the data as a chart, we’ll first use the matplotlib Python library. This library is the core plotting library on which many others are based and provides a great deal of flexibility in creating charts.
Add a new code cell, and add the following code:

from matplotlib import pyplot as plt

# matplotlib requires a Pandas dataframe, not a Spark one
df_sales = df_spark.toPandas()

# Create a bar plot of revenue by year
plt.bar(x=df_sales['OrderYear'], height=df_sales['GrossRevenue'])

# Display the plot
plt.show()

Run the cell and review the results, which consist of a column chart with the total gross revenue for each year. Review the code, and notice the following:

(i) The matplotlib library requires a Pandas DataFrame, so you need to convert the Spark DataFrame returned by the Spark SQL query.
(ii) At the core of the matplotlib library is the pyplot object. This is the foundation for most plotting functionality.
(iii) The default settings result in a usable chart, but there’s considerable scope to customize it.

Modify the code to plot the chart as follows:

from matplotlib import pyplot as plt

# Clear the plot area
plt.clf()

# Create a bar plot of revenue by year
plt.bar(x=df_sales['OrderYear'], height=df_sales['GrossRevenue'], color='orange')

# Customize the chart
plt.title('Revenue by Year')
plt.xlabel('Year')
plt.ylabel('Revenue')
plt.grid(color='#95a5a6', linestyle='--', linewidth=2, axis='y', alpha=0.7)
plt.xticks(rotation=45)

# Show the figure
plt.show()

Re-run the code cell and view the results. The chart is now easier to understand.
A plot is contained with a Figure. In the previous examples, the figure was created implicitly but it can be created explicitly. Modify the code to plot the chart as follows:

from matplotlib import pyplot as plt

# Clear the plot area
plt.clf()

# Create a Figure
fig = plt.figure(figsize=(8,3))

# Create a bar plot of revenue by year
plt.bar(x=df_sales['OrderYear'], height=df_sales['GrossRevenue'], color='orange')

# Customize the chart
plt.title('Revenue by Year')
plt.xlabel('Year')
plt.ylabel('Revenue')
plt.grid(color='#95a5a6', linestyle='--', linewidth=2, axis='y', alpha=0.7)
plt.xticks(rotation=45)

# Show the figure
plt.show()

Re-run the code cell and view the results. The figure determines the shape and size of the plot.

A figure can contain multiple subplots, each on its own axis. Modify the code to plot the chart as follows:

from matplotlib import pyplot as plt

# Clear the plot area
plt.clf()

# Create a figure for 2 subplots (1 row, 2 columns)
fig, ax = plt.subplots(1, 2, figsize = (10,4))

# Create a bar plot of revenue by year on the first axis
ax[0].bar(x=df_sales['OrderYear'], height=df_sales['GrossRevenue'], color='orange')
ax[0].set_title('Revenue by Year')

# Create a pie chart of yearly order counts on the second axis
ax[1].pie(df_sales['YearlyCounts'])
ax[1].set_title('Orders per Year')
ax[1].legend(df_sales['OrderYear'])

# Add a title to the Figure
fig.suptitle('Sales Data')

# Show the figure
plt.show()

Re-run the code cell and view the results.

Use the seaborn library

While matplotlib enables you to create different chart types, it can require some complex code to achieve the best results. For this reason, new libraries have been built on matplotlib to abstract its complexity and enhance its capabilities. One such library is seaborn.

Add a new code cell to the notebook, and enter the following code:

import seaborn as sns

# Clear the plot area
plt.clf()

# Create a bar chart
ax = sns.barplot(x="OrderYear", y="GrossRevenue", data=df_sales)

plt.show()

Run the code to display a bar chart created using the seaborn library.
Modify the code again as follows:

import seaborn as sns

# Clear the plot area
plt.clf()

# Create a line chart
ax = sns.lineplot(x="OrderYear", y="GrossRevenue", data=df_sales)

plt.show()

Run the modified code to view the yearly revenue as a line chart.

Now in this project, you’ve learned how to use Spark to work with data in Microsoft Fabric.

Create a Microsoft Fabric Lakehouse

lotanna obianefo — Thu, 02 Oct 2025 01:00:06 +0000

Large-scale data analytics solutions have traditionally been built around a data warehouse, in which data is stored in relational tables and queried using SQL. The growth in “big data” (characterized by high volumes, variety, and velocity of new data assets) together with the availability of low-cost storage and cloud-scale distributed compute technologies has led to an alternative approach to analytical data storage; the data lake. In a data lake, data is stored as files without imposing a fixed schema for storage. Increasingly, data engineers and analysts seek to benefit from the best features of both of these approaches by combining them in a data lakehouse; in which data is stored in files in a data lake and a relational schema is applied to them as a metadata layer so that they can be queried using traditional SQL semantics.

In Microsoft Fabric, a lakehouse provides highly scalable file storage in a OneLake store (built on Azure Data Lake Store Gen2) with a metastore for relational objects such as tables and views based on the open source Delta Lake table format. Delta Lake enables you to define a schema of tables in your lakehouse that you can query using SQL.

Create a workspace

Before working with data in Fabric, create a workspace with the Fabric trial enabled.

Navigate to the Microsoft Fabric home page at https://app.fabric.microsoft.com/home?experience=fabric in a browser, and sign in with your Fabric credentials.
In the menu bar on the left, select Workspaces (the icon looks similar to 🗇).
Create a new workspace with a name of your choice, selecting a licensing mode in the Advanced section that includes Fabric capacity. When your new workspace opens, it should be empty.

Create a lakehouse

Now that you have a workspace, it’s time to create a data lakehouse for your data files.

On the menu bar on the left, select Create. In the New page, under the Data Engineering section, select Lakehouse. Give it a unique name of your choice.

View the new lakehouse, and note that the Lakehouse explorer pane on the left enables you to browse tables and files in the lakehouse:
- The Tables folder contains tables that you can query using SQL semantics. Tables in a Microsoft Fabric lakehouse are based on the open source Delta Lake file format, commonly used in Apache Spark.
- The Files folder contains data files in the OneLake storage for the lakehouse that aren’t associated with managed delta tables. You can also create shortcuts in this folder to reference data that is stored externally.

Currently, there are no tables or files in the lakehouse.

Upload a file

Fabric provides multiple ways to load data into the lakehouse, including built-in support for pipelines that copy data from external sources and data flows (Gen 2) that you can define using visual tools based on Power Query. However one of the simplest ways to ingest small amounts of data is to upload files or folders from your local computer.

Download the sales.csv file from https://raw.githubusercontent.com/MicrosoftLearning/dp-data/main/sales.csv, saving it as sales.csv on your local computer.
Return to the web browser tab containing your lakehouse, and in the … menu for the Files folder in the Explorer pane, select New subfolder, and create a subfolder named data.
In the … menu for the new data folder, select Upload and Upload files, and then upload the sales.csv file from your local computer. After the file has been uploaded, select the Files/data folder and verify that the sales.csv file has been uploaded, as shown here: Select the sales.csv file to see a preview of its contents.

Explore shortcuts

In many scenarios, the data you need to work with in your lakehouse may be stored in some other location. While there are many ways to ingest data into the OneLake storage for your lakehouse, another option is to instead create a shortcut. Shortcuts enable you to include externally sourced data in your analytics solution without the overhead and risk of data inconsistency associated with copying it.

In the … menu for the Files folder, select New shortcut.
View the available data source types for shortcuts. Then close the New shortcut dialog box without creating a shortcut.

Load file data into a table

The sales data you uploaded is in a file, which data analysts and engineers can work with directly by using Apache Spark code. However, in many scenarios you may want to load the data from the file into a table so that you can query it using SQL.

In the Explorer pane, select the Files/data folder so you can see the sales.csv file it contains.
In the … menu for the sales.csv file, select Load to Tables > New table.
In Load to table dialog box, set the table name to sales and confirm the load operation. Then wait for the table to be created and loaded.
In the Explorer pane, select the sales table that has been created to view the data.

If the sales table does not automatically appear, in the … menu for the Tables folder, select Refresh.

In the … menu for the sales table, select View files to see the underlying files for this table. Files for a delta table are stored in Parquet format, and include a subfolder named _delta_log in which details of transactions applied to the table are logged.

Use SQL to query tables

When you create a lakehouse and define tables in it, a SQL endpoint is automatically created through which the tables can be queried using SQL SELECT statements.

At the top-right of the Lakehouse page, switch from Lakehouse to SQL analytics endpoint. Then wait a short time until the SQL analytics endpoint for your lakehouse opens in a visual interface from which you can query its tables.

Use the New SQL query button to open a new query editor, and enter the following SQL query:

SELECT Item, SUM(Quantity * UnitPrice) AS Revenue
FROM sales
GROUP BY Item
ORDER BY Revenue DESC;

Use the ▷ Run button to run the query and view the results, which should show the total revenue for each product.

Create a visual query

While many data professionals are familiar with SQL, data analysts with Power BI experience can apply their Power Query skills to create visual queries.

On the toolbar, expand the New SQL query option and select New visual query.
Drag the sales table to the new visual query editor pane that opens to create a Power Query as shown here:
In the Manage columns menu, select Choose columns. Then select only the SalesOrderNumber and SalesOrderLineNumber columns.
In the Transform menu, select Group by. Then group the data by using the following Basic settings:

Group by: SalesOrderNumber
New column name: LineItems
Operation: Count distinct values
Column: SalesOrderLineNumber

In this project, you have created a lakehouse and imported data into it. You’ve seen how a lakehouse consists of files and tables stored in a OneLake data store. The managed tables can be queried using SQL, and are included in a default semantic model to support data visualizations.

Mastering Linux Commands: Essential Tools for Efficient System Management

lotanna obianefo — Wed, 01 Oct 2025 19:05:01 +0000

Linux, an open-source operating system that powers environments ranging from servers and supercomputers to embedded systems, depends extensively on its command-line interface (CLI) for management and automation. In contrast to graphical user interfaces (GUIs), the Linux terminal provides unmatched efficiency, adaptability, and accuracy. Understanding key Linux commands is crucial for navigating the filesystem, managing processes, and troubleshooting issues.

This article explores several essential Linux commands, grouped by category, with their primary functions and practical examples. Focusing on commands that form the backbone of daily Linux operations. By the end, you'll see how to apply in a real-world case study.

Before initiating development in the Linux terminal, it is considered best practice to operate from the root directory by running this command sudo su. This provides elevated privileges required to update system packages and install dependencies essential for application execution.

sudo su

Execute the command apt-get update to synchronize the local package index with the repositories. This operation refreshes metadata for all available packages, ensuring the system references the latest versions.

apt-get update

At this stage, essential software packages and utilities such as NGINX and Vim can be installed. The command apt install nginx provisions NGINX, a high-performance web server that also functions as a reverse proxy, load balancer, and caching solution to optimize application delivery.
Similarly, apt install vim installs Vim, an advanced and extensible text editor derived from Vi, widely used for efficient code and configuration file editing in Linux environments.

apt install nginx

apt install vim

When the terminal output becomes cluttered with excessive command history, execute the clear command to reset the display buffer and present a clean workspace.

Navigation and Directory Management

These commands facilitate navigation within the Linux filesystem, enabling directory creation, traversal, and hierarchical organization.

Utilize the mkdir command to provision three new directories within the filesystem.

mkdir dir1 dir2 dir3

Execute the ls command to display the directories that were recently created within the current filesystem path.
Now you can see the 3 directories we just created dir1, dir2 and dir3.

Initially, it is necessary to navigate to the target directory using the cd command.

cd dir1

To inspect file system metadata, including both standard and hidden entries, the commands ls -l and ls -la are utilized. The ls -l command outputs detailed attributes of visible files and directories, whereas ls -la extends this listing to include hidden entries.

ls -l

ls -la

Notice the dotfiles along the directory we created with their associated metadata.

When navigating the filesystem, the cd .. command moves the user one level up to the parent directory of the current working path, whereas the cd ~ command resolves directly to the user’s home directory, independent of the current location within the filesystem.

cd ..

Notice we moved a step further to the parent directory of the current directory.

cd ~

To displays the absolute path of the current directory in which the user is working, we will use the command pwd which stands for “print working directory.”.

Files and Directory Operations

Since the newly created directories are empty, the next step is to create a file, duplicate and relocate it into one of the target directories, and then open it in a text editor for further modifications.

The touch filename command is used to create an empty file.

Notice, we have to run the ls command afterwards to list out the files and directories we created.

touch datafile.txt

To duplicate the newly created file into another directory, the cp filename.txt dir2 command is used, which preserves the original file in its current location while placing a copy in the target directory. In contrast, the mv filename.txt dir2 command relocates the file entirely into dir2, removing it from the source path. Functionally, cp behaves like a copy operation, whereas mv is analogous to a cut-and-paste.

cp datafile.txt dir2

mv datafile.txt dir2

After executing the ls command to verify the directory contents, you will observe that the file has been relocated to dir2.

On like the touch command vim filename.txt command not only creates the file (if it does not already exist) but also opens it in the Vim editor for further modifications.

vim datafile.txt

When the command is executed, it opens the file in the Vim editor. Press i to enter Insert mode and make your changes. Once editing is complete, press Esc to return to Normal mode, then type :wq and press Enter to write the changes to disk and exit the editor.

To display the contents of the recently edited file, use the cat filename command.

cat datafile.txt

If a directory is no longer required, it can be removed using the rmdir dir command. However, rmdir only deletes empty directories; to remove a directory along with its contents (files and subdirectories), the rm -r dir command must be used. In contrast, the rm filename command removes individual files.

rm datafile
To delete an individual file

rm -r dir1
To delete a directory with it's content

rmdir dir3
To delete an empty directory

Finally, grep is a robust command-line utility designed for pattern matching and text search within files or command outputs. It functions as the Linux terminal’s built-in search mechanism, enabling efficient filtering and extraction of relevant information from large datasets or command results.

Using delete as the search word.

grep delete newfile.txt

System Information and Management

Among the vast array of Linux commands, those dedicated to system information and management stand out for their ability to deliver real-time insights and control over hardware, processes, and software.

The uname -a command provides a comprehensive summary of system-level information, including kernel name, release, version, and hardware architecture details.

uname -a

To verify the storage capacity of the system, the df -h command is executed. This command reports disk space usage for all mounted filesystems, presenting the information in a human-readable format with size units.

df -h

The free -h command provides a detailed summary of the system’s memory utilization, including both RAM and swap space, in a human-readable format. It displays the total, used, free, and available memory, along with buffer/cache usage.

free -h

The top command is used to display active processes and monitor real-time system resource utilization. It provides a continuously updated, dynamic view of process activity, including CPU and memory consumption. Functionally, top serves as a terminal-based equivalent of a task manager in Linux.

top

Note: To terminate a running process in the terminal, use Ctrl+C, which sends a SIGINT (interrupt signal) to stop execution. Alternatively, Ctrl+Z suspends the process by sending a SIGTSTP signal, pausing its execution and placing it in the background.

Mastering Linux commands is not just about memorization. it’s about building intuition for how the operating system interacts with files, processes, and resources. From navigating the filesystem to monitoring system performance and controlling processes, these commands form the foundation of efficient system administration.

By consistently practicing and combining these tools, administrators and developers can transform Linux from a basic operating system into a powerful, precision-driven environment for managing modern workloads.