DEV Community

Pavan Kumar Sadashiv
Pavan Kumar Sadashiv

Posted on

Building an Automated Engineering Telemetry Pipeline with GitLab CI/CD

#gitlab #cicd #devops #automation

Building an Automated Engineering Telemetry Pipeline with GitLab CI/CD

When you manage dozens of repositories across multiple projects, microservices, and AI engines, keeping track of engineering velocity becomes a full-time headache.

Traditional engineering management often relies on fragmented meetings or manual end-of-day reports. But in modern DevOps environments, your version control platform already holds 100% of the ground truth: commit logs, merge requests, review comments, and pipeline durations.

In this guide, I will walk you through building a fully automated, serverless engineering telemetry pipeline powered entirely by scheduled GitLab CI/CD pipelines and the GitLab REST/GraphQL API.

This system autonomously aggregates commit metrics across 50+ repositories, detects velocity bottlenecks, generates a daily markdown audit log, and publishes a live dashboard with zero human intervention.

1. Architecture: The Serverless Telemetry Hub

Rather than running a dedicated daemon server, we leverage GitLab’s native primitives:

+-------------------------------------------------------------+
|               GitLab CI/CD Scheduled Pipeline               |
|                  (Triggers daily at 00:00 UTC)              |
+-------------------------------------------------------------+
                              |
                              v
+-------------------------------------------------------------+
|                   Telemetry Collector Job                   |
|  - Queries GitLab GraphQL API for multi-repo commits        |
|  - Aggregates lines changed, PR review velocity, test runs  |
|  - Formats daily engineering ledger (DAILY_UPDATE.md)       |
+-------------------------------------------------------------+
                              |
                              v
+-------------------------------------------------------------+
|                     GitLab Pages / Git Bot                  |
|  - Commits aggregated report back to telemetry repo         |
|  - Publishes live interactive dashboard via GitLab Pages    |
+-------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

2. Querying the GitLab GraphQL API for Commit Velocity

The GitLab GraphQL API (https://gitlab.com/api/graphql) is vastly superior to REST when fetching bulk telemetry, because a single request can batch project details, branch states, and merge request statuses without hitting rate limits.

Here is a Python script (collector.py) that queries project activity over the last 24 hours:

import os
import requests
from datetime import datetime, timedelta, timezone

GITLAB_TOKEN = os.environ["GITLAB_API_TOKEN"]
GRAPHQL_URL = "https://gitlab.com/api/graphql"

query = """
query getRecentActivity($groupPath: ID!, $after: Time!) {
  group(fullPath: $groupPath) {
    projects(includeSubgroups: true) {
      nodes {
        name
        webUrl
        repository {
          tree {
            lastCommit {
              sha
              committedDate
              authorName
              title
            }
          }
        }
        mergeRequests(state: merged, mergedAfter: $after) {
          count
        }
      }
    }
  }
}
"""

def fetch_telemetry(group_path: str):
    yesterday = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat()
    headers = {"Authorization": f"Bearer {GITLAB_TOKEN}"}

    response = requests.post(
        GRAPHQL_URL,
        json={"query": query, "variables": {"groupPath": group_path, "after": yesterday}},
        headers=headers,
    )
    response.raise_for_status()
    return response.json()["data"]["group"]["projects"]["nodes"]
Enter fullscreen mode Exit fullscreen mode

3. Formatting the Automated Daily Digest

Once data is gathered, the pipeline converts raw metrics into a clean, human-readable markdown format:

def generate_markdown(projects):
    date_str = datetime.now(timezone.utc).strftime("%Y-%m-%d")
    output = [f"# 🚀 Engineering Telemetry Log — {date_str}\n"]
    output.append("| Project | Last Commit | Author | Merged MRs (24h) |")
    output.append("| :--- | :--- | :--- | :---: |")

    for p in projects:
        commit = p["repository"]["tree"]["lastCommit"] if p["repository"]["tree"] else None
        if commit:
            title = commit["title"][:45]
            author = commit["authorName"]
            mrs = p["mergeRequests"]["count"]
            output.append(f"| [{p['name']}]({p['webUrl']}) | {title} | {author} | **{mrs}** |")

    return "\n".join(output)
Enter fullscreen mode Exit fullscreen mode

4. The .gitlab-ci.yml Pipeline Configuration

Now, configure the GitLab CI pipeline to run automatically every night and push the updated ledger back to the repository:

stages:
  - telemetry
  - deploy

run_telemetry:
  stage: telemetry
  image: python:3.12-slim
  rules:
    # Only run on scheduled cron triggers or manual UI trigger
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_PIPELINE_SOURCE == "web"
  before_script:
    - pip install requests
    - git config --global user.name "GitLab Telemetry Bot"
    - git config --global user.email "bot@gitlab.com"
  script:
    - python collector.py > telemetry_report.md
    - cat telemetry_report.md
    # Push back to repo using CI Project Access Token
    - git remote set-url origin https://oauth2:${PROJECT_ACCESS_TOKEN}@gitlab.com/${CI_PROJECT_PATH}.git
    - git add telemetry_report.md
    - git commit -m "chore(telemetry): automated daily sync [skip ci]" || echo "No changes to commit"
    - git push origin HEAD:${CI_DEFAULT_BRANCH}
  artifacts:
    paths:
      - telemetry_report.md

pages:
  stage: deploy
  needs: [run_telemetry]
  script:
    - mkdir -p public
    - cp telemetry_report.md public/index.html
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
Enter fullscreen mode Exit fullscreen mode

5. Setting Up the Pipeline Schedule in GitLab

  1. In your GitLab repository, go to Build $\rightarrow$ Pipeline schedules.
  2. Click New schedule.
  3. Description: Daily 00:00 UTC Telemetry Run.
  4. Interval Pattern: Select Custom and enter 0 0 * * *.
  5. Target branch: main.
  6. Click Save pipeline schedule.

You now have a fully autonomous telemetry engine executing every single night at midnight!

6. Key Benefits & Takeaways

  1. Zero Infrastructure Cost: No dedicated EC2 instances, cron servers, or third-party SaaS subscriptions; GitLab CI runners handle the entire execution.
  2. Deterministic Audit Trail: Daily logs are committed directly into git history, providing an immutable record of development velocity.
  3. Scales Across Unlimited Repos: By using GraphQL queries scoped to your GitLab group or user profile, new projects are discovered and indexed automatically.

Resources & Links

How does your team track cross-repository activity? Share your favorite CI/CD automation tricks below!

Top comments (2)

Collapse
 
abhinavs1920 profile image
Abhinav Sharma •

Pipeline duration is the most underused signal in the GitLab API: it is already recorded per job, needs no extra instrumentation, and a week of it usually shows which stage actually gates the merge train. Pairing it with review-comment turnaround is what makes the audit log actionable rather than a vanity metric.

Collapse
 
hrlpavan profile image
Pavan Kumar Sadashiv •

Spot on, Abhinav. Pipeline duration is indeed a critical, yet often overlooked, metric for optimizing the software delivery lifecycle. By focusing on the critical path of your build, you can identify the true bottlenecks gating your merge train.

When you correlate this with review-comment turnaround, you start to bridge the gap between technical efficiency and team collaboration, moving from simple observability to actionable continuous improvement. Have you looked into using dynamic DAGs or parallel matrix builds to further optimize those gating stages?