DEV Community: MARYMAR DANYTZA CALLOTICONA CHAMBILLA

CI/CD Testing Pipelines Compared: A Real GitHub Actions Walkthrough

MARYMAR DANYTZA CALLOTICONA CHAMBILLA — Thu, 09 Jul 2026 23:54:20 +0000

Why the CI/CD tool choice matters for testing

Every CI/CD platform in this comparison — GitHub Actions, GitLab Pipelines, Jenkins, CircleCI, TeamCity, Travis CI, Bitbucket Pipelines, Tekton, Harness — solves the same basic problem: run the test suite automatically, on every change, before anything gets merged or deployed. What differs between them is where they run, how they're configured, and how much operational overhead they add on top of "running tests." This article walks through GitHub Actions with a working example, and compares it against the alternatives a team is likely to be choosing between.

What is GitHub Actions?

GitHub Actions is GitHub's native CI/CD platform. Workflows are defined as YAML files living inside the repository itself, under .github/workflows/, and triggered directly by GitHub events (push, pull request, schedule, manual dispatch) with no separate server to install or maintain — GitHub runs the jobs on hosted runners.

Public example repository (mine, built for this comparison): https://github.com/mary010-sky/github-actions-ci-demo

A working test pipeline

The repository contains a small JavaScript utilities library with a Jest test suite. Here's the workflow that runs it on every push and pull request:

# .github/workflows/test.yml
name: Run Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [18.x, 20.x]

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run test suite
        run: npm test -- --coverage

      - name: Upload coverage report
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: coverage/

A few things worth pointing out:

The strategy.matrix block runs the exact same job twice, once per Node version — catching a version-specific bug (Array.prototype.at not existing on old Node, for example) without writing two separate workflow files.
cache: 'npm' on the setup-node step caches node_modules between runs, which on a real project can cut a multi-minute install down to a few seconds.
The coverage report gets uploaded as a build artifact, downloadable from the Actions run page, without needing a separate storage service. ## Adding a status check that blocks merges

# .github/workflows/test.yml (excerpt, added at the job level)
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      # ...same steps as above...

      - name: Fail if coverage drops below threshold
        run: |
          COVERAGE=$(node -e "console.log(require('./coverage/coverage-summary.json').total.lines.pct)")
          echo "Line coverage: $COVERAGE%"
          if (( $(echo "$COVERAGE < 80" | bc -l) )); then
            echo "Coverage below 80% threshold"
            exit 1
          fi

Once this workflow is set as a required status check in branch protection rules, a pull request literally cannot be merged if the test suite fails or coverage drops — the enforcement lives in GitHub itself, not in a separate policy someone has to remember to check.

How it compares

GitHub Actions vs. GitLab Pipelines — functionally very similar: both live as YAML in the repository, both offer hosted runners, both support matrix builds. The real difference is ecosystem: GitHub Actions has a much larger public Marketplace of reusable actions, while GitLab Pipelines benefits from being deeply integrated with GitLab's own issue tracker, container registry, and Auto DevOps templates in a single product. A team already living on GitLab often gets more out of the box; a team on GitHub gets more third-party actions to plug in.

GitHub Actions vs. Jenkins — Jenkins is self-hosted and endlessly pluggable (2,000+ plugins), which is exactly its strength and its cost: someone has to run, patch, and secure the Jenkins server itself. GitHub Actions trades that flexibility for zero infrastructure to maintain — you get hosted runners and a smaller, curated action ecosystem instead of "install any plugin for anything."

GitHub Actions vs. CircleCI — CircleCI was an early leader in fast, cached, parallelized builds, and its UI for visualizing pipeline steps and re-running failed jobs is still often considered more polished. GitHub Actions has closed most of that performance gap and wins on convenience for any project already hosted on GitHub, since there's no separate account, billing relationship, or webhook setup required.

GitHub Actions vs. Travis CI — Travis CI was the original "YAML in your repo, hosted runners" pioneer, and pushed most of the industry toward this model. Its relevance has declined sharply since GitHub Actions launched with a similar model but native integration and a far larger runner pool; most projects that used Travis CI have since migrated.

Where GitHub Actions falls short

For long-running enterprise pipelines with heavy compliance and audit requirements, tools like Harness or TeamCity offer more built-in governance and deployment-strategy tooling (canary releases, feature-flag-gated rollouts) than GitHub Actions provides natively — GitHub Actions can get there, but usually by composing several actions rather than through a single built-in feature. It's also worth remembering that hosted runner minutes aren't unlimited on the free tier, which matters for a project running large matrix builds frequently.

Conclusion

GitHub Actions makes the most sense as the default choice specifically because of where it lives: no separate CI server, no extra account, and workflow files that travel with the code they test. For a team already on GitHub, the setup cost is close to zero — write the YAML, push it, and the pipeline exists. The tradeoffs (smaller plugin ecosystem than Jenkins, less built-in deployment governance than Harness) only start to matter at a scale most teams haven't hit yet.

API Testing in Practice: Automating Postman Collections with Newman

MARYMAR DANYTZA CALLOTICONA CHAMBILLA — Thu, 09 Jul 2026 23:48:40 +0000

Why API testing needs its own layer

Unit tests check a function in isolation. End-to-end UI tests check the whole product through a browser. Neither of those is great at answering the question a backend team actually cares about day to day: "did this endpoint just break its contract?" — wrong status code, missing field, broken auth, a response shape that changed without anyone updating the docs. That's the gap API testing frameworks fill, and it's exactly the space covered in "Top 10 API Testing Tools" on Medium, which lists tools like Postman, SoapUI, and Rest-Assured side by side.

This article focuses on one of the most widely adopted combinations in that space: Postman for designing and exploring requests, and Newman, Postman's official CLI runner, for turning that collection into something a pipeline can execute automatically.

What Newman actually is

Newman is the command-line collection runner published and maintained by the Postman team. It takes a Postman collection — a JSON file with requests, assertions written in JavaScript, and environment variables — and runs it outside the Postman desktop app, which is what makes it usable in CI/CD.

Official repository: https://github.com/postmanlabs/newman

Target API for this example

Instead of a private backend, I used httpbin.org, a small, public HTTP testing service that echoes back whatever you send it — ideal for demonstrating real requests without needing any authentication or infrastructure of my own.

Step 1: writing assertions inside a Postman request

Inside the Postman app, each request has a "Tests" tab where you write plain JavaScript assertions that run against the response:

// Tests tab for GET https://httpbin.org/get?user=marymar

pm.test("Status code is 200", () => {
    pm.response.to.have.status(200);
});

pm.test("Response contains the query param sent", () => {
    const body = pm.response.json();
    pm.expect(body.args.user).to.eql("marymar");
});

pm.test("Response time is under 800ms", () => {
    pm.expect(pm.response.responseTime).to.be.below(800);
});

For a POST request that sends a JSON body:

// Tests tab for POST https://httpbin.org/post

pm.test("Status code is 200", () => {
    pm.response.to.have.status(200);
});

pm.test("Echoed body matches what was sent", () => {
    const body = pm.response.json();
    pm.expect(body.json.title).to.eql(pm.collectionVariables.get("title"));
});

Step 2: exporting the collection

From Postman: collection menu → Export → Collection v2.1 → save as api-tests.postman_collection.json. That file is what Newman consumes — it's plain JSON, so it belongs in version control right next to the application code it's testing.

Step 3: running it with Newman

npm install -g newman

newman run api-tests.postman_collection.json \
  --env-var baseUrl="https://httpbin.org" \
  --reporters cli,html \
  --reporter-html-export reports/report.html

Newman prints a summary per request — assertions passed, failed, response time — and, with the html reporter, generates a shareable report that doesn't require anyone to have Postman installed to read it.

Step 4: wiring it into GitHub Actions

# .github/workflows/api-tests.yml
name: API Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  newman:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Newman
        run: npm install -g newman newman-reporter-html

      - name: Run API tests
        run: |
          newman run api-tests.postman_collection.json \
            --env-var baseUrl="https://httpbin.org" \
            --reporters cli,html,junit \
            --reporter-html-export reports/report.html \
            --reporter-junit-export reports/report.xml

      - name: Upload test report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: newman-report
          path: reports/

The --reporters cli,html,junit line matters: cli for readable logs in the Actions output, html for a report a non-technical teammate can open, and junit because most CI dashboards (GitHub, GitLab, Jenkins) know how to render JUnit XML as a native test summary with pass/fail counts.

A more complete real-world example

For a fuller example that organizes tests by feature (auth, user profile, and a resource-tracking flow) instead of one flat collection, this project walks through the same Postman + Newman + GitHub Actions pattern end to end:

https://github.com/postmanlabs/newman (the runner itself — clone it to see its own test suite structure as a reference for organizing a Newman-driven project)

Where this approach falls short

Newman runs exactly what's in the exported JSON — it has no concept of the API's actual behavior beyond the assertions someone wrote by hand. It won't generate edge cases for you, and a collection that's gone stale (testing an old response shape) will happily keep "passing" against a mismatched mental model of the API. It's a contract-verification tool, not a fuzzer or a schema-validation engine — for deeper coverage, pairing it with a schema validation step (e.g. validating the response against an OpenAPI spec) closes that gap.

Conclusion

Postman gets teams to write and explore requests quickly; Newman is what turns that exploration into a repeatable, CI-friendly safety net. The combination doesn't require learning a new language or framework — if a team can already write a Postman request, they can write a Postman test, and from there the jump to "this runs on every pull request" is just a few lines of YAML.

Catching Cloud Misconfigurations Before They Ship: SAST for Terraform with Checkov

MARYMAR DANYTZA CALLOTICONA CHAMBILLA — Thu, 09 Jul 2026 23:39:40 +0000

Why Infrastructure as Code needs SAST too

Static Application Security Testing (SAST) is normally associated with application source code — Java, Python, JavaScript. But Infrastructure as Code (IaC) files like Terraform are code too, and they define something arguably more dangerous to get wrong: the actual cloud resources an organization runs on. A single misconfigured .tf file can leave an S3 bucket public, disable encryption on a database, or open a security group to the entire internet — and none of that shows up as a runtime bug, because it's not a bug, it's exactly what was declared.

OWASP's Source Code Analysis Tools list (https://owasp.org/www-community/Source_Code_Analysis_Tools) is normally cited for application-level static analyzers, but the same "shift left" principle applies directly to IaC: catch the misconfiguration in the pull request, not in production. This article walks through Checkov, an open-source SAST tool built specifically for IaC, deliberately leaving TFSec out of the picture.

What is Checkov?

Checkov is an open-source static analysis tool created by Bridgecrew (now part of Prisma Cloud). It scans Terraform, Terraform plan output, CloudFormation, Kubernetes manifests, Helm charts, Dockerfiles, Bicep, ARM templates, and OpenTofu, looking for security and compliance misconfigurations using graph-based scanning rather than plain text matching.

Official repository: https://github.com/bridgecrewio/checkov

Setting up a target to scan

Rather than writing a toy example from scratch, I used TerraGoat, an intentionally vulnerable Terraform project maintained by the same Bridgecrew team behind Checkov. It exists specifically so people can practice IaC scanning against real, varied misconfigurations instead of a single contrived file.

git clone https://github.com/bridgecrewio/terragoat.git
cd terragoat

A representative snippet from the AWS module looks like this:

resource "aws_s3_bucket" "data" {
  bucket = "terragoat-${var.environment}-customer-data"
  acl    = "public-read"
}

resource "aws_db_instance" "customer_db" {
  identifier        = "terragoat-${var.environment}-db"
  engine            = "postgres"
  instance_class    = "db.t3.micro"
  allocated_storage = 20
  storage_encrypted = false
  publicly_accessible = true
}

Both resources look fine syntactically. Both are serious security problems: a public S3 bucket with customer data, and a publicly accessible, unencrypted RDS instance.

Installing and running Checkov

python3 -m venv venv
source venv/bin/activate
pip install checkov

checkov -d ./aws

Checkov reports each failed check with an ID, a description, and a link to remediation guidance:

Check: CKV_AWS_20: "S3 Bucket has an ACL defined which allows public access"
    FAILED for resource: aws_s3_bucket.data
    File: /aws/s3.tf:1-4

Check: CKV_AWS_16: "Ensure that RDS instances have encryption at rest enabled"
    FAILED for resource: aws_db_instance.customer_db
    File: /aws/rds.tf:1-8

Check: CKV_AWS_17: "Ensure all data stored in RDS is not publicly accessible"
    FAILED for resource: aws_db_instance.customer_db
    File: /aws/rds.tf:1-8

Fixing the findings

resource "aws_s3_bucket" "data" {
  bucket = "terragoat-${var.environment}-customer-data"
  acl    = "private"
}

resource "aws_db_instance" "customer_db" {
  identifier           = "terragoat-${var.environment}-db"
  engine               = "postgres"
  instance_class       = "db.t3.micro"
  allocated_storage    = 20
  storage_encrypted    = true
  publicly_accessible  = false
}

Running checkov -d ./aws again shows those specific checks now passing, while the rest of TerraGoat's intentional vulnerabilities remain — which is exactly the point of a vulnerable-by-design target: it lets you validate the tool actually catches what it claims to, one fix at a time.

Automating it with GitHub Actions

Manually running Checkov defeats the purpose of "shifting left." The real value comes from wiring it into CI so every pull request gets scanned automatically:

# .github/workflows/checkov.yml
name: Checkov IaC Scan

on:
  push:
    paths:
      - '**/*.tf'
  pull_request:
    paths:
      - '**/*.tf'

jobs:
  checkov_scan:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Run Checkov
        uses: bridgecrewio/checkov-action@v12
        with:
          directory: .
          framework: terraform
          soft_fail: true
          output_format: sarif
          output_file_path: results.sarif

      - name: Upload SARIF to GitHub Security tab
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: results.sarif

soft_fail: true lets the pipeline continue even if issues are found — useful while a team is still triaging a legacy codebase. Once findings are under control, flipping it to false turns Checkov into an actual merge gate. The SARIF output means every finding shows up directly as an annotation in GitHub's Security tab, right next to the code that caused it.

Suppressing accepted risks

Not every finding is actionable — sometimes a public bucket is intentional (a static website, a public documentation folder). Checkov supports inline suppression instead of forcing you to disable an entire rule globally:

resource "aws_s3_bucket" "public_docs" {
  bucket = "my-public-documentation"
  acl    = "public-read" #checkov:skip=CKV_AWS_20:This bucket intentionally serves public docs
}

This keeps the suppression next to the resource it applies to, so the next person reading the file understands why the rule was skipped, instead of just seeing a disabled check somewhere in a config file.

Where Checkov stops

Checkov is a static analyzer: it understands the declared configuration, not runtime behavior. It won't catch a logic bug in a module's variable interpolation, and it can't tell you if a security group that looks fine on paper ends up overly permissive once combined with other resources at apply time. It's one layer of defense — a strong one for catching known misconfiguration patterns before deployment, not a replacement for code review or runtime monitoring.

Conclusion

Applying SAST to Terraform isn't fundamentally different from applying it to application code: find the tool, wire it into the pipeline, and treat its findings as a gate rather than a suggestion. Checkov's graph-based scanning, 1,000+ built-in policies, and native SARIF output make it a solid entry point for teams that want to catch cloud misconfigurations in the pull request instead of in an incident report.

AI Agents That Speak SQL: Text-to-SQL with Hugging Face smolagents

MARYMAR DANYTZA CALLOTICONA CHAMBILLA — Thu, 09 Jul 2026 23:19:35 +0000

The problem with "classic" Text-to-SQL

The simplest way to connect an LLM to a database is a single-pass pipeline: the user writes a question in natural language, the model generates a SQL query, and that query is executed directly against the database. This is, for example, the approach followed in freeCodeCamp's tutorial "How to Talk to Any Database Using AI", where a Python script builds a prompt with the table schema and asks a model to return raw SQL.

The problem is that this pipeline is fragile. If the model generates a query with a syntax error, or worse, a syntactically valid but semantically incorrect query, there's nobody reviewing the result before showing it to the user. There's no error correction, no verification that the answer actually makes sense.

That's where agents come in. Instead of a model that just "translates" text into SQL, an agent can execute the query, observe the result, and decide whether it needs to fix something before responding. That's exactly what smolagents, Hugging Face's library for building agents with very little code, does.

What is smolagents?

smolagents is an open source library from Hugging Face designed so an agent "thinks in code": instead of the LLM returning a JSON blob describing the action to take, it writes actual Python snippets that the agent executes step by step, observing the result before continuing. This pattern is called CodeAgent and follows the ReAct framework (reason → act → observe).

Official repository: https://github.com/huggingface/smolagents

Building the agent step by step

1. Set up a sample database with SQLAlchemy

from sqlalchemy import (
    create_engine, MetaData, Table, Column,
    String, Integer, Float, insert, inspect, text
)

engine = create_engine("sqlite:///:memory:")
metadata_obj = MetaData()

# Define the "receipts" table
table_name = "receipts"
receipts = Table(
    table_name,
    metadata_obj,
    Column("receipt_id", Integer, primary_key=True),
    Column("customer_name", String(16), primary_key=True),
    Column("price", Float),
    Column("tip", Float),
)
metadata_obj.create_all(engine)

# Insert sample data
rows = [
    {"receipt_id": 1, "customer_name": "Alan Payne", "price": 12.06, "tip": 1.20},
    {"receipt_id": 2, "customer_name": "Alex Mason", "price": 23.86, "tip": 0.24},
    {"receipt_id": 3, "customer_name": "Woodrow Wilson", "price": 53.43, "tip": 5.43},
    {"receipt_id": 4, "customer_name": "Margaret James", "price": 21.11, "tip": 1.00},
]
for row in rows:
    stmt = insert(receipts).values(**row)
    with engine.begin() as connection:
        connection.execute(stmt)

2. Turn the schema into a description the model can read

inspector = inspect(engine)
columns_info = [(col["name"], col["type"]) for col in inspector.get_columns("receipts")]

table_description = "Columns:\n" + "\n".join(
    [f"  - {name}: {col_type}" for name, col_type in columns_info]
)
print(table_description)

This gives us something like:

Columns:
  - receipt_id: INTEGER
  - customer_name: VARCHAR(16)
  - price: FLOAT
  - tip: FLOAT

3. Define the tool the agent will use

In smolagents, a tool is just a Python function decorated with @tool. The docstring is key: the agent reads it to know when and how to use the tool.

from smolagents import tool

@tool
def sql_engine(query: str) -> str:
    """
    Allows you to perform SQL queries on the table 'receipts'.
    Returns a string representation of the result.
    The table has the following columns:
    Columns:
      - receipt_id: INTEGER
      - customer_name: VARCHAR(16)
      - price: FLOAT
      - tip: FLOAT

    Args:
        query: the SQL query to execute. Must be valid SQL.
    """
    output = ""
    with engine.connect() as con:
        rows = con.execute(text(query))
        for row in rows:
            output += "\n" + str(row)
    return output

4. Create and run the agent

from smolagents import CodeAgent, InferenceClientModel

agent = CodeAgent(
    tools=[sql_engine],
    model=InferenceClientModel(model_id="meta-llama/Llama-3.1-8B-Instruct"),
)

agent.run("Can you give me the name of the client who got the most expensive receipt?")

The agent doesn't just generate the SQL: it executes it, reads the result, and if something doesn't add up (say, a column that doesn't exist), it can retry with a corrected query before giving you the final answer. That's the real difference compared to the single-pass pipeline.

5. Scaling up to joins across multiple tables

The same pattern holds as the schema grows. We add a second table for waiters and update the tool's description:

table_name = "waiters"
waiters = Table(
    table_name,
    metadata_obj,
    Column("receipt_id", Integer, primary_key=True),
    Column("waiter_name", String(16), primary_key=True),
)
metadata_obj.create_all(engine)

rows = [
    {"receipt_id": 1, "waiter_name": "Corey Johnson"},
    {"receipt_id": 2, "waiter_name": "Michael Watts"},
    {"receipt_id": 3, "waiter_name": "Michael Watts"},
    {"receipt_id": 4, "waiter_name": "Margaret James"},
]
for row in rows:
    stmt = insert(waiters).values(**row)
    with engine.begin() as connection:
        connection.execute(stmt)

agent.run("Which waiter got the most tips in total?")

The agent now has to reason about a JOIN between receipts and waiters, and it does so without us changing a single line of the agent's logic — only the tool's description was updated.

A more complete, real-world example

If you want to see this pattern taken into a project with production-style structure (REST API, result validation, explicit self-correction), this repository implements exactly that idea on top of smolagents:

https://github.com/Sakeeb91/text2sql-agent

There, the flow is: the agent inspects the schema, generates the SQL, validates that the result makes sense, and if it detects a problem, self-corrects before answering — all exposed behind an API with JWT or API key authentication.

Conclusion

The "classic" Text-to-SQL pipeline (like freeCodeCamp's) is a great starting point for understanding the problem, but it falls short in production as soon as questions get more complex. Wrapping the SQL query as a tool inside a smolagents CodeAgent gives the system the ability to review its own work, which translates into fewer silently wrong answers — which is, in the end, the worst kind of error a system that talks to your database can make.