JacivaldoCarvalho

Posted on Oct 31

12 Green CI Jobs, 422 Security Findings: Lessons from Building an Observability Platform

#devops #devsecops #sre #cicd

An engineering journey through Python, React, Docker, GitHub Actions, observability, and software supply chain security.

Introduction

Does a green CI pipeline mean your application is secure?

That question became much more meaningful while developing InfraPulse, a monitoring and observability platform I'm building as a hands-on engineering and technical portfolio project.

Throughout development, I implemented automated tests, infrastructure validation, PostgreSQL integration, end-to-end testing, observability checks, and software supply chain security controls.

Eventually, the project reached 12 successful GitHub Actions jobs, including dependency integrity checks, secret detection, and Docker image validation.

Everything looked promising.

Then I introduced dedicated vulnerability scanning.

The scanners reported 422 findings, including one classified as critical and 95 as high severity.

That doesn't mean InfraPulse has 422 exploitable vulnerabilities. It means the scanners produced 422 findings that still require analysis, correlation, and classification.

That distinction matters.

In this article, I'll share how the project evolved, the engineering decisions behind its architecture, and what I learned about integrating security into the development lifecycle.

1. InfraPulse: From an Application to an Engineering Platform

InfraPulse started with a relatively straightforward goal: build an application capable of monitoring services and endpoints, recording results, and exposing operational information.

However, I didn't want to build just another functional dashboard.

I wanted a project that would allow me to practice engineering approaches commonly used in professional DevOps, SRE, and Platform Engineering environments.

That meant answering questions such as:

How can I prevent backend changes from breaking frontend API contracts?
How can I validate real application behavior against PostgreSQL?
How should observability behave when an infrastructure component fails?
How can I verify that Docker images contain the expected artifacts?
How can I prevent dependencies and base images from changing silently?
How can security be introduced without making CI unreliable?

These questions directly influenced the architecture.

Technology Stack

Layer	Technologies
Backend	Python, FastAPI, SQLAlchemy, Alembic
Frontend	React, TypeScript, Vite, Mantine
Database	PostgreSQL 17
Containers	Docker, Docker Compose, Nginx
CI/CD	GitHub Actions
Testing	Pytest, Vitest, Playwright
Metrics	Prometheus
Distributed tracing	OpenTelemetry, Tempo
Visualization	Grafana
Alerting	Alertmanager
Security	Gitleaks, integrity policies, pip-audit, npm audit, Trivy

An important architectural decision was to keep components organized around clear responsibilities.

This made it possible to validate individual layers independently while also testing their integrated behavior.

2. The First Challenge: Building Trust in CI

Creating a GitHub Actions workflow that runs tests and displays a green checkmark is relatively straightforward.

Making sure that green checkmark represents meaningful verification is much harder.

InfraPulse's CI pipeline evolved incrementally.

I started with backend quality checks and unit tests.

Then I expanded coverage to include frontend validation, PostgreSQL integration, API contracts, end-to-end tests, and observability integration.

The next step was validating the Docker images produced by the project.

Finally, I introduced foundational software supply chain security controls.

The 12-Job CI Pipeline

In the reference remote execution, all 12 jobs passed:

Job	Responsibility
`backend-quality`	Backend static quality checks
`backend-unit`	Unit tests
`api-contract`	OpenAPI contract validation
`frontend-quality`	Frontend quality checks
`frontend-tests`	Automated frontend tests
`frontend-build`	Frontend production build
`backend-postgres`	PostgreSQL integration tests
`playwright-e2e`	End-to-end tests
`observability-integration`	Real observability integration scenarios
`docker-build`	Docker builds and image validation
`security-foundations`	Foundational security controls
`ci-required`	Mandatory aggregate CI gate

The ci-required job plays a particularly important role.

It verifies the results of all mandatory validation jobs.

This prevents the pipeline from being considered successful when a required job fails, is canceled, or is unexpectedly skipped.

A reliable CI gate must validate not only application behavior but also the integrity of the validation process itself.

All 12 jobs passed in the reference execution.

Figure 1 — GitHub Actions CI Results

Figure 1. InfraPulse CI pipeline: 12 successful jobs, covering quality, integration, end-to-end testing, observability, Docker image validation, and foundational security controls. Screenshot from a private repository.

3. Testing Observability Means Introducing Failure

One of the most interesting lessons was realizing that instrumenting an application is not the same as validating its observability.

Adding metrics, logs, and traces is only the beginning.

The real challenge is understanding how these signals behave when infrastructure components become unavailable.

In InfraPulse, I implemented integration scenarios involving the OpenTelemetry Collector.

The tests cover three situations:

Normal operation: The application runs while the Collector is available.

Outage: The Collector becomes unavailable.

Recovery: The Collector returns, and the system's behavior is observed again.

These scenarios help validate how the application and its instrumentation behave under changing operational conditions.

All three scenarios passed during the reference remote execution.

This reinforced an important SRE principle:

Observability should be tested under failure conditions, not merely demonstrated through dashboards.

Of course, three successful scenarios do not prove complete system resilience.

They provide evidence about specific behaviors under controlled conditions.

And that is precisely what good engineering validation should do.

4. Docker: A Successful Build Is Not Enough

Another important milestone was improving Docker image validation.

At first, it might seem sufficient to execute docker build and confirm that the command exits successfully.

But that leaves several questions unanswered.

Does the image contain the correct files?

Does the generated frontend match the expected OpenAPI contract?

Does the worker shut down gracefully?

Can Nginx serve the expected assets?

Are the runtime configurations behaving correctly?

To address these questions, I expanded the pipeline with image inspections and smoke tests.

OpenAPI Contract and Frontend Consistency

The frontend uses TypeScript types generated from the OpenAPI contract.

During image validation, the generated types extracted from the frontend build were compared byte-for-byte against an independently generated reference.

Both used the contract from the same CI execution and a pinned generator version.

The verification did not merely check whether files existed or contained data.

It compared their complete contents.

This reduces the risk of shipping a frontend built with types that differ from the expected API contract.

Runtime Behavior

The image validation process also checked:

The API /health endpoint returning HTTP 200.
Readiness returning HTTP 503 when the database was unavailable, as expected in that test scenario.
Worker startup and graceful shutdown using SIGTERM.
Nginx configuration, health checks, and static assets.
Docker image layer inspection.
Cleanup of temporary validation resources.

These checks do not replace production testing.

However, they provide stronger evidence than a successful image build alone.

5. When DevSecOps Became Part of the Architecture

As the pipeline matured, another concern became increasingly important: the integrity of the components used to build and run the software.

An application can pass every functional test while still depending on vulnerable packages, mutable container images, or insufficiently controlled build tools.

That realization led me to introduce a dedicated software supply chain security phase.

The first step focused on foundational controls.

Secret Detection

I integrated Gitleaks to scan the repository history and working checkout for potentially exposed secrets.

Beyond running the scanner, I added synthetic positive and negative test fixtures, operational failure handling, and sanitized output.

The goal wasn't simply to execute a security tool.

It was to verify that the tool detected known test cases and that operational failures could not silently become successful results.

Python Dependency Integrity

Initially, the project contained 49 pinned Python requirements without package hashes.

Pinning dependency versions improves predictability, but it does not independently verify the integrity of the downloaded artifacts.

I introduced hash-verified installations and regression tests that deliberately reject tampered hashes.

Build and bootstrap tools were also version-pinned.

Docker Images Pinned by Digest

Another improvement involved replacing base-image references that relied only on mutable tags with digest-pinned references.

A tag can point to different content over time.

A digest identifies specific content, helping prevent silent changes to the image base.

This improves reproducibility but introduces an additional responsibility: maintaining controlled updates so security fixes can still be incorporated.

GitHub Actions Security

The workflow security policies also covered:

GitHub Actions pinned to full commit SHAs.
Least-privilege workflow permissions.
Checkout without unnecessary credential persistence.
Automated policies designed to detect regressions in these controls.

This work introduced the mandatory security-foundations job.

The reference CI execution passed all blocking foundational security policies with zero violations.

But one question remained:

Were the dependencies and container images free from known vulnerabilities?

6. The Turning Point: 422 Vulnerability Findings

After establishing foundational security controls, I started a dedicated vulnerability discovery phase.

I used three tools:

pip-audit for Python dependencies.
npm audit for frontend dependencies.
Trivy for Docker image vulnerability scanning.

The results were significant.

Target	Findings
Python — pip-audit 2.10.1	2 advisories
Frontend — npm audit 11.16.0	15 occurrences across 14 packages
Backend image — Trivy 0.76.1	176 occurrences
Nginx image — Trivy	229 occurrences
Total	422 occurrences

Figure 2 — Sanitized Vulnerability Audit Summary

Figure 2. Sanitized local vulnerability discovery summary. The 422 findings are scanner occurrences, not unique CVEs or confirmed exploitable vulnerabilities.

The consolidated severity distribution was:

Reported severity	Occurrences
Critical	1
High	95
Medium	143
Moderate	15
Low	159
Unknown	9

These numbers require careful interpretation.

422 findings do not mean 422 unique CVEs, and they certainly do not prove 422 exploitable vulnerabilities.

The same advisory may appear across multiple images, packages, or scanning contexts.

Different scanners may also use different severity classifications and remediation information.

Furthermore, finding a vulnerable package inside a container image does not automatically demonstrate that the affected functionality is reachable or exploitable through the application.

A Concrete Example

Trivy flagged CVE-2026-111497 in brotli-libs version 1.2.0-r0, reporting 1.2.1-r0 as the fixed version.

The Python audit also reported advisories affecting setuptools, with fixed versions indicated by the scanner.

These findings deserve attention, but they still require technical triage.

Important questions include:

Is the affected component present in the production runtime or only during the build?
Is a fixed version available and compatible?
Has the operating system distribution already applied a security backport?
Is the vulnerable functionality actually exposed?
Could the proposed update introduce regressions?
Is the same underlying issue being counted across multiple targets?

At the time of this analysis, those questions had not yet been fully resolved.

Therefore, the vulnerability discovery phase was initially treated as informational, rather than immediately turning every finding into a blocking CI failure.

This was a deliberate engineering decision.

7. Why Not Block CI Immediately?

It may sound contradictory to identify a critical-severity finding without immediately failing the entire pipeline.

But detecting a potential vulnerability and designing a reliable blocking policy are two different activities.

A useful security policy must account for evidence quality, applicability, remediation availability, and scanner behavior.

Otherwise, CI can become unstable, and teams may start introducing broad exceptions simply to restore a green pipeline.

This does not mean ignoring critical findings.

It means prioritizing their investigation and defining verifiable remediation criteria.

The next stage of InfraPulse will focus on classifying findings, deduplicating occurrences, assessing applicability, and establishing security policies with justified, time-limited exceptions.

It is also important to distinguish between different audit outcomes:

PASS: The check executed successfully and its acceptance criteria were satisfied.

FINDINGS: The audit completed and identified issues requiring evaluation.

SCANNER_ERROR: The scanner failed to execute or collect results correctly.

NOT_RUN: The audit was not executed.

A scanner error must never be interpreted as evidence that no vulnerabilities exist.

8. What I've Learned So Far

Building InfraPulse has reinforced several lessons that I consider especially relevant to DevOps and SRE.

First: software quality is built in layers.

Unit tests, integration tests, E2E tests, image validation, and security checks answer different questions. None completely replaces the others.

Second: pipelines should produce evidence.

A green status is useful, but engineering confidence also depends on knowing what was executed, which artifacts were tested, and which limitations remain.

Third: reproducibility and security are related, but they are not the same thing.

Hashes and digests help control the content being used. They do not independently guarantee that the content is vulnerability-free.

Fourth: observability must include failure scenarios.

A working dashboard under normal conditions is valuable, but outage and recovery tests provide additional evidence about operational behavior.

Fifth: automating detection does not automate engineering judgment.

Scanners identify potential problems. Determining applicability, priority, and remediation requires context.

Perhaps the most important lesson is that security should not be treated as a separate activity performed only at the end of development.

It should evolve alongside architecture, testing, infrastructure, and delivery practices.

Conclusion

InfraPulse began as a monitoring project and evolved into a practical exercise in software engineering, reliability, and security.

Reaching 12 successful GitHub Actions jobs was an important milestone.

But discovering 422 vulnerability findings demonstrated that a mature pipeline does not mean the security work is finished.

Instead, it can provide the foundation needed to identify and investigate additional risks.

The next challenge is not simply making the scanners report zero findings.

It is building a security policy capable of distinguishing meaningful risks, prioritizing remediation, and preserving a reliable delivery process.

In Part 2, I plan to explore vulnerability triage, deduplication, applicability analysis, blocking policies, and auditable exceptions.

A green pipeline is evidence of engineering quality. It is not a certificate of zero risk.

References and Tools

This article documents an ongoing engineering project. Vulnerability scan results represent a specific point-in-time assessment, not a definitive statement about exploitability or overall application security. Package and remediation versions cited in the case study reflect the local scanner output.

InfraPulse is currently maintained in a private repository. The screenshots included in this article are sanitized excerpts of the project's validation results.

DEV Community