Building an Automated Engineering Telemetry Pipeline with GitLab CI/CD

When you manage dozens of repositories across multiple projects, microservices, and AI engines, keeping track of engineering velocity becomes a full-time headache.

Traditional engineering management often relies on fragmented meetings or manual end-of-day reports. But in modern DevOps environments, your version control platform already holds 100% of the ground truth: commit logs, merge requests, review comments, and pipeline durations.

In this guide, I will walk you through building a fully automated, serverless engineering telemetry pipeline powered entirely by scheduled GitLab CI/CD pipelines and the GitLab REST/GraphQL API.

This system autonomously aggregates commit metrics across 50+ repositories, detects velocity bottlenecks, generates a daily markdown audit log, and publishes a live dashboard with zero human intervention.

---

1. Architecture: The Serverless Telemetry Hub

Rather than running a dedicated daemon server, we leverage GitLab’s native primitives:

+-------------------------------------------------------------+
|               GitLab CI/CD Scheduled Pipeline               |
|                  (Triggers daily at 00:00 UTC)              |
+-------------------------------------------------------------+
                              |
                              v
+-------------------------------------------------------------+
|                   Telemetry Collector Job                   |
|  - Queries GitLab GraphQL API for multi-repo commits        |
|  - Aggregates lines changed, PR review velocity, test runs  |
|  - Formats daily engineering ledger (DAILY_UPDATE.md)       |
+-------------------------------------------------------------+
                              |
                              v
+-------------------------------------------------------------+
|                     GitLab Pages / Git Bot                  |
|  - Commits aggregated report back to telemetry repo         |
|  - Publishes live interactive dashboard via GitLab Pages    |
+-------------------------------------------------------------+

---

2. Querying the GitLab GraphQL API for Commit Velocity

The GitLab GraphQL API (https://gitlab.com/api/graphql) is vastly superior to REST when fetching bulk telemetry, because a single request can batch project details, branch states, and merge request statuses without hitting rate limits.

Here is a Python script (collector.py) that queries project activity over the last 24 hours:

import os
import requests
from datetime import datetime, timedelta, timezone

GITLAB_TOKEN = os.environ["GITLAB_API_TOKEN"]
GRAPHQL_URL = "https://gitlab.com/api/graphql"

query = """
query getRecentActivity($groupPath: ID!, $after: Time!) {
  group(fullPath: $groupPath) {
    projects(includeSubgroups: true) {
      nodes {
        name
        webUrl
        repository {
          tree {
            lastCommit {
              sha
              committedDate
              authorName
              title
            }
          }
        }
        mergeRequests(state: merged, mergedAfter: $after) {
          count
        }
      }
    }
  }
}
"""

def fetch_telemetry(group_path: str):
    yesterday = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat()
    headers = {"Authorization": f"Bearer {GITLAB_TOKEN}"}

    response = requests.post(
        GRAPHQL_URL,
        json={"query": query, "variables": {"groupPath": group_path, "after": yesterday}},
        headers=headers,
    )
    response.raise_for_status()
    return response.json()["data"]["group"]["projects"]["nodes"]

---

3. Formatting the Automated Daily Digest

Once data is gathered, the pipeline converts raw metrics into a clean, human-readable markdown format:

def generate_markdown(projects):
    date_str = datetime.now(timezone.utc).strftime("%Y-%m-%d")
    output = [f"# 🚀 Engineering Telemetry Log — {date_str}\n"]
    output.append("| Project | Last Commit | Author | Merged MRs (24h) |")
    output.append("| :--- | :--- | :--- | :---: |")

    for p in projects:
        commit = p["repository"]["tree"]["lastCommit"] if p["repository"]["tree"] else None
        if commit:
            title = commit["title"][:45]
            author = commit["authorName"]
            mrs = p["mergeRequests"]["count"]
            output.append(f"| [{p['name']}]({p['webUrl']}) | {title} | {author} | **{mrs}** |")

    return "\n".join(output)

---

4. The .gitlab-ci.yml Pipeline Configuration

Now, configure the GitLab CI pipeline to run automatically every night and push the updated ledger back to the repository:

stages:
  - telemetry
  - deploy

run_telemetry:
  stage: telemetry
  image: python:3.12-slim
  rules:
    # Only run on scheduled cron triggers or manual UI trigger
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_PIPELINE_SOURCE == "web"
  before_script:
    - pip install requests
    - git config --global user.name "GitLab Telemetry Bot"
    - git config --global user.email "[e-mail]"
  script:
    - python collector.py > telemetry_report.md
    - cat telemetry_report.md
    # Push back to repo using CI Project Access Token
    - git remote set-url origin https://oauth2:${PROJECT_ACCESS_TOKEN}@gitlab.com/${CI_PROJECT_PATH}.git
    - git add telemetry_report.md
    - git commit -m "chore(telemetry): automated daily sync [skip ci]" || echo "No changes to commit"
    - git push origin HEAD:${CI_DEFAULT_BRANCH}
  artifacts:
    paths:
      - telemetry_report.md

pages:
  stage: deploy
  needs: [run_telemetry]
  script:
    - mkdir -p public
    - cp telemetry_report.md public/index.html
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

---

5. Setting Up the Pipeline Schedule in GitLab

1. In your GitLab repository, go to Build $\rightarrow$ Pipeline schedules.
2. Click New schedule.
3. Description: Daily 00:00 UTC Telemetry Run.
4. Interval Pattern: Select Custom and enter 0 0 * * *.
5. Target branch: main.
6. Click Save pipeline schedule.

You now have a fully autonomous telemetry engine executing every single night at midnight!

---

6. Key Benefits & Takeaways

1. Zero Infrastructure Cost: No dedicated EC2 instances, cron servers, or third-party SaaS subscriptions; GitLab CI runners handle the entire execution.
2. Deterministic Audit Trail: Daily logs are committed directly into git history, providing an immutable record of development velocity.
3. Scales Across Unlimited Repos: By using GraphQL queries scoped to your GitLab group or user profile, new projects are discovered and indexed automatically.

---

Resources & Links

• GitLab Profile: gitlab.com/hrlpavan
• GitLab Contributor Profile: contributors.gitlab.com/users/hrlpavan
• GitHub: @hrlpavan

How does your team track cross-repository activity? Share your favorite CI/CD automation tricks below!