DEV Community: Okikiola Ashiru

I Built a Security-Gating CI Pipeline, Then Discovered It Wasn't Actually Gating Anything

Okikiola Ashiru — Thu, 13 Aug 2026 07:22:22 +0000

A security check that fails but doesn't block a merge isn't a security gate. It's a warning label nobody has to read.

I learned that the hard way building pipeline-gatekeeper, a small CI/CD pipeline that wires together three open-source security scanners: Trivy for container vulnerabilities, gitleaks for exposed secrets, and Trivy IaC for insecure Terraform configuration. The goal wasn't to build new scanning tools. It was to prove that wiring existing, trusted tools together correctly actually catches what it's supposed to catch, and to find out where "correctly" breaks down in practice.

The setup

Three parallel jobs run on every pull request:

Container Vulnerability Scan (Trivy): scans the built Docker image for CRITICAL and HIGH severity CVEs in both OS packages and Python dependencies.
Secret Detection (gitleaks): scans the full commit history for hardcoded API keys, tokens, and credentials.
Terraform Policy Scan (Trivy IaC, using tfsec rulesets): scans Terraform configuration for misconfigurations like unencrypted storage or missing public-access restrictions.

Trivy was the natural choice for both container and IaC scanning since it runs as a single stateless binary with no daemon or database, and using one tool for two of the three gates kept the pipeline simpler than adding a separate IaC-specific scanner.

Proving it actually works, not just that it's configured

Design decisions are easy to write down and easy to get wrong in practice, so I tested this the same way I've tested every project before it: by deliberately breaking things and watching what happened.

I opened a pull request with three real violations baked in on purpose:

A dependency (pillow==8.2.0) with a known critical remote-code-execution CVE
A hardcoded GitHub personal access token, formatted exactly like a real one, in a config file
An AWS EBS volume in Terraform with encrypted = false

All three checks failed, exactly as expected.

What I found when I looked closer

This is the part that actually mattered. Two things went wrong that I hadn't planned for, and both were more useful to find than a clean pass would have been.

First: failing checks didn't actually block the merge. GitHub showed all three checks red, and right underneath, it also showed "No conflicts with base branch. Merging can be performed automatically," with an active merge button. The scanners were detecting violations correctly. Nothing was stopping anyone from merging past them anyway. Detection and enforcement turned out to be two completely separate things, and I'd only built the first one. The fix is a branch protection rule requiring these checks to pass before merge, which GitHub Actions doesn't set up for you by default just because a workflow exists.

Second: gitleaks found the secret but couldn't tell me about it where I was looking. It tried to post a summary comment directly on the pull request and failed with a permissions error, Resource not accessible by integration. The scan still worked. It still reported the leak in the job logs and summary. It just couldn't write to the PR itself, because the default GITHUB_TOKEN didn't have pull-requests: write permission. An easy fix once I saw it, but exactly the kind of thing that looks fine in a demo and quietly fails in a way you don't notice until you go looking for the comment that should be there.

Why I'm glad I found these instead of avoiding them

It would have been easy to stop at "all three checks failed as expected" and call the project done. That's what the first version of my own README said, before I looked closely enough at the actual PR to notice the merge button was still active.

The gap between "the check fails" and "the merge is blocked" is exactly the kind of thing that's invisible until someone tests it end to end, and it's a real, common misconfiguration, not a contrived edge case. A pipeline that reports failures nobody has to act on isn't protecting anything. It's just generating logs.

After fixing both issues

What this project didn't try to be

This isn't a production security platform, and three scanners wired into one workflow isn't a complete security posture. It doesn't cover runtime security, dependency license compliance, or SAST for the application code itself. What it does prove, concretely, is the difference between a pipeline that looks like it's enforcing something and one that actually is, and that difference is worth checking for directly instead of assuming.

Repo: github.com/aashiruu/pipeline-gatekeeper

What Happens When "Secure" and "Available" Disagree? Building a Healthcare Access-Control System Taught Me There's No Clean Answer

Okikiola Ashiru — Wed, 05 Aug 2026 12:12:41 +0000

What Happens When "Secure" and "Available" Disagree? Building a Healthcare Access-Control System Taught Me There's No Clean Answer

Fintech's constraint was correctness, never double-charge someone. Chaos engineering's constraint was recovery, does the system actually heal, not just look healed. Healthcare access control turned out to be a different problem again: what do you do when being strict and being available point in opposite directions?

That's the question behind access-sentinel, a simulated patient-records access system I built to work through how healthcare software actually handles this tension, not in theory, but in the specific moments where the "right" answer isn't obvious.

The setup: RBAC is the easy part

Role-based access control itself isn't the hard problem. Doctors get clinical and billing data, nurses get clinical only, billing staff get financial data only, and admins (deliberately) get none of it directly; they manage the platform, not patient records. That's a straightforward policy to write and enforce at the API layer.

The hard problems start at the edges: what happens in an emergency, and what happens when part of the system is down.

Edge case one: break-glass access

Real healthcare systems have a well-known pattern called "break-glass", a clinician can override normal access restrictions in a declared emergency. The tension is obvious once you say it out loud: you can never let "emergency" become an unmonitored backdoor, but you also can never let strict access control block someone from saving a life.

I implemented this as an explicit override: a request can include a break-glass header and a required justification reason. Two things I was careful about:

Only clinical roles can break-glass. A billing or admin account requesting an emergency override gets a flat 403, there's no legitimate emergency-care reason for those roles to need it, so the override surface is deliberately narrow.
A justification is mandatory, not optional. A break-glass request without a reason gets rejected with a 400 before it ever touches patient data. The override isn't free, you have to state why.
Every override is logged and counted separately from normal access, specifically so a spike in break-glass usage is visible as an anomaly signal, not buried in normal traffic.

The mechanism doesn't prevent misuse, nothing purely technical can, because the whole point is that a human is allowed to override the rules. What it does is make misuse expensive to hide and cheap to detect after the fact.

Edge case two: what happens when the audit log is down

This is the design decision I spent the most time on. If every access needs to be logged, what happens to a request when the logging store itself is unreachable?

There are two clean-sounding answers, and they're both wrong on their own:

Fail closed (reject all requests until logging is restored) is the "safe" choice, nothing goes unlogged, but it means a doctor can't pull up a patient's chart during an outage, which is a genuinely dangerous failure mode in an emergency.
Fail open (let everything through, log later) keeps care running, but if "log later" quietly never happens, you've built an access-control system with no actual record of who saw what.

I ended up with a hybrid, split by role: clinical reads (doctor, nurse) fail open, the request succeeds, the response carries a header flagging the system as degraded, and the access event gets buffered in memory to flush to the audit store once it recovers. Non-clinical reads (billing) fail closed, a 503, because there's no emergency-care justification for blocking that traffic, so strict compliance wins by default.

I tested this by actually forcing the audit store offline under sustained load, not just a single request, and watching Grafana through the outage:

The buffered-log counter climbed steadily during the outage, confirming events were queuing rather than silently dropping
Billing requests sat flat at zero (correctly blocked) the whole time
Once the store came back online, the buffered counter dropped back to zero, the queued events actually flushed, not just accumulated forever

That last part mattered most to me. "Fail open" is an easy design decision to state and a much harder one to prove, the difference between "available" and "available and honest about what happened" is exactly the buffered-and-flushed behavior, not just the initial 200 response.

The tension I didn't expect: immutability vs. the right to be forgotten

The audit log is append-only by design, every entry is chained to the previous one with a SHA-256 hash, and any attempt to edit or delete a log entry is rejected outright. That's the right call for audit integrity: if the log can be edited, it's not actually proof of anything.

But that same immutability runs directly into privacy regulations like GDPR's "right to be forgotten," which require that personal data be deletable on request. An access log that can never be edited or deleted is, by construction, incompatible with a legal right to erasure.

Real systems handle this with patterns like pseudonymized lookup tables, the immutable log stores a reference ID, not the person's actual identity, and the identity mapping (which can be deleted) lives separately. I didn't build that layer here, but naming the tension directly, rather than pretending an append-only log is a strictly better design with no trade-offs, felt more honest than leaving it out.

What this project didn't try to be

This isn't HIPAA-compliant software, and I'm not claiming it models every real constraint a production healthcare system would need — real deployments have encryption-at-rest requirements, formal compliance audits, and legal review this project never touched. What it does model honestly is the shape of the actual trade-offs: strict-but-blocking versus available-but-riskier, and the fact that a good system doesn't pick one side of that trade-off globally, it picks differently depending on what's actually at stake for a given request.

Repo: github.com/aashiruu/access-sentinel

Building a Chaos Engineering CLI Taught Me Why "It Recovered" Isn't Enough

Okikiola Ashiru — Sat, 01 Aug 2026 14:03:57 +0000

After building ledger-sentinel, a payment system focused on correctness under concurrent load, I wanted to go one level deeper: not just "does the system behave correctly," but "does the system recover when something actually breaks."

That's the premise behind fault-sentinel, a lightweight Kubernetes-native CLI I built in Go to deliberately inject failures (killing pods, adding network latency, generating CPU load) and observe what actually happens.

Why build your own tool instead of just using an existing one?

There are established chaos engineering tools (Chaos Mesh, LitmusChaos) that do this more comprehensively. I built fault-sentinel anyway for a specific reason: I wanted to understand the mechanics from the inside, not just run someone else's custom resources.

Writing the pod-eviction logic myself using k8s.io/client-go means I actually know what a graceful deletion versus a forced deletion does at the API level, rather than just knowing a CRD exists for it.

Note: This is an experimental learning project built to explore chaos injection concepts and Go platform engineering principles, not a production tool.

The Three Experiments

Pod Termination (kill-pod): Selects a pod by label selector and deletes it to verify whether the deployment's ReplicaSet controller notices and replaces it, and how fast reconciliation occurs.

Network Latency (network-delay): Uses Linux Traffic Control (tc netem) inside the target container's network namespace to inject artificial packet delay, automatically triggering a cleanup after the experiment window.
CPU Stress (stress-cpu): Spins up compute load across a configurable number of cores for a set duration to observe how the workload performs under compute starvation.

Each experiment executes through a Cobra-based CLI:

# Terminate a pod matching a label
./bin/fault-cli kill-pod --namespace default --label-selector app=target-app

# Inject 200ms network delay for 30 seconds
./bin/fault-cli network-delay --namespace default --label-selector app=target-app --delay 200ms --duration 30s

# Apply CPU stress across 2 cores for 30 seconds
./bin/fault-cli stress-cpu --duration 30s --cores 2

The design decision I spent the most time on: how much access does this tool actually need?

A chaos tool is, by definition, something that deletes pods and manipulates network behavior in a cluster. That's a meaningful amount of trust to grant a piece of software, so I wanted the access model to be as narrow as it could be.

Two things came out of that:

RBAC scoping: kill-pod only needs list and delete permissions on pods in the target namespace — not cluster-wide access, not access to secrets or other resources.
No persistent daemon: fault-sentinel doesn't run as a long-lived agent inside the cluster. It uses standard client-go calls and SPDY exec connections for the duration of an experiment, then it's done. There's no always-on process with standing permissions waiting to be exploited, the tool is only as dangerous as the moment you're actively running it.

The trade-off here is availability, not just access: a tool with no daemon can't schedule automatic recurring chaos experiments the way Chaos Mesh can. For a learning project focused on understanding failure mechanics rather than running a continuous chaos program, that trade-off made sense.

What the network-delay injection actually required

This was the most fiddly of the three experiments to get right. Injecting latency via tc netem requires the NET_ADMIN Linux capability inside the target container — without it, the tc command inside the pod simply fails with a permissions error. That means the target application's pod spec needs NET_ADMIN added to its securityContext, which is itself a decision worth thinking about: you're granting the target application's container a capability it doesn't normally need, purely so it can be a valid subject for chaos testing.

That's a real trade-off, not a footnote, running chaos experiments against a workload means loosening that workload's security posture slightly, at least for network-based experiments. In a real environment, you'd want a dedicated, tightly scoped service account and probably a separate sidecar with the capability, rather than granting it to the application container directly.

Telemetry: making sure the experiment itself is observable

An experiment you can't measure isn't really an experiment. fault-sentinel exposes Prometheus metrics on :8080/metrics during every run:

chaos_experiments_total{experiment_type, status} — how many experiments ran, and whether they completed or errored
chaos_injected_faults_total{target_pod, fault_type} — which specific pods were hit, and with what
chaos_experiment_duration_seconds{experiment_type} — how long each experiment actually took

Scraping this mid-experiment:

curl -s http://localhost:8080/metrics | grep chaos_

The chaos_injected_faults_total metric tracks the cumulative count of successful fault injections labeled by fault_type and target target_pod. In the screenshot above, the counter value of 1 reflects a single active 250ms latency experiment targeting pod payment-api-667fdccc65-gwmfl. Successive runs against other pods or repeated injections will increment this value per label combination across the lifetime of the telemetry server instance.

What I actually verified, not just designed

The CI pipeline (GitHub Actions) runs golangci-lint, unit tests with Go's race detector enabled (go test -race), builds a multi-stage Docker image, and runs a Trivy vulnerability scan on the built image for critical/high severities before anything is considered done. The race detector matters specifically here, a chaos tool that has its own concurrency bugs while testing other systems' concurrency behavior would be a bad joke.

What this project didn't try to be

This isn't a replacement for Chaos Mesh or Litmus, and I'm not claiming it should be used against a real production cluster. It doesn't have scheduled/recurring experiments, it doesn't have a web UI, and it doesn't handle multi-cluster orchestration. What it does have is a clear, small surface area that I understand completely, because I built every part of it... which was the actual point.

The lesson that stuck with me most: recovery isn't binary. A pod coming back after being killed isn't the same as the system recovering, you have to actually watch what happens to in-flight requests, latency, and error rates during the gap, not just confirm the pod count went back to normal. That's the difference between "it healed" and "it healed in a way that didn't hurt anyone using it."

Repo: https://github.com/aashiruu/fault-sentinel

What Would a Fintech Payment System's DevOps Setup Actually Look Like?

Okikiola Ashiru — Wed, 29 Jul 2026 14:10:43 +0000

Every industry has different constraints that shape its infrastructure decisions. Banking cares about compliance and audit trails. E-commerce cares about traffic spikes on sale days. Fintech payment systems care about something more specific: money cannot be created or destroyed by accident.

If a request gets retried because of a flaky network connection, a customer should never be charged twice. If two requests hit the database at the same time, the ledger should never end up in an inconsistent state. This one constraint, correctness under concurrency and failure, ends up driving almost every infrastructure decision in a payment system.

I wanted to understand this properly, so I built a small payment gateway simulation called ledger-sentinel to work through the actual engineering problems, not just read about them. Here's what the DevOps/infrastructure thinking behind a system like this looks like, using what I built as a concrete example.

The core problem: idempotency

The first design question in any payment system is: what happens when the same request arrives twice?

This happens constantly in production; a client times out and retries, a load balancer resends a request, a mobile app has a flaky connection and fires the same "pay now" tap twice. If your system isn't built to handle this, you get duplicate charges.

The standard pattern is an idempotency key: the client sends a unique key with each logical transaction, and the server guarantees that request only actually processes once, no matter how many times it arrives.

In ledger-sentinel, I implemented this with:

A Redis distributed lock (SET NX EX) that acts as a fast, short-lived gatekeeper... the first request to arrive with a given key gets the lock, any duplicate arriving while it's still processing gets rejected (HTTP 409) instead of double-processing
A response cache layered on top, so once a request has actually completed, any later retry with the same key gets served the original cached response (HTTP 200) instead of hitting the database again

This is the difference between "the system happened to work in testing" and "the system is provably safe under concurrent retries", and it's the first thing I'd want to verify actually holds before trusting any payment infrastructure.

Why async settlement, not synchronous processing

A naive payment API might try to validate, charge, and update the ledger all within a single HTTP request. This is a mistake at any real scale, because it means the client is waiting on your slowest downstream dependency (usually the database) for every single request, and a slow ledger write becomes a slow, unreliable API.

The fix is decoupling ingestion from settlement:

The API validates the request and idempotency key, then pushes the transaction onto a RabbitMQ queue
A separate worker process consumes the queue and does the actual ledger update
The client gets a fast response (validated and queued), and the ledger settles asynchronously

This also gives you backpressure control, if the worker is falling behind under a traffic burst, messages queue up safely instead of the whole system falling over. I set a consumer prefetch limit (prefetch_count=10) specifically so the worker never grabs more in-flight work than it can actually handle at once.

Why the ledger itself still needs strict locking

Queueing solves the "don't block the client" problem, but it doesn't solve the "two workers processing related transactions at the same time" problem. For that, the actual database work still needs strict consistency guarantees.

I used PostgreSQL row-level locking (SELECT ... FOR UPDATE) so that when a worker is updating a specific ledger balance, no other transaction can read or write that same row until the lock releases. This is slower than an unlocked read, but for a ledger, "slower but always correct" beats "fast but occasionally wrong", the entire point of a ledger is that the numbers are always right.

How I actually verified this wasn't just "should work in theory"

Design decisions are cheap to write down and easy to get wrong in practice. So I stress-tested it with k6, firing over 1,000 concurrent transaction requests at the system, some with duplicate idempotency keys on purpose, and checked:

Zero duplicate charges — verified directly against the database (GROUP BY idempotency_key HAVING COUNT(*) > 1 returned nothing)
p99 latency stayed sub-second even during the burst, and recovered to ~4–8ms once the burst normalized and requests were hitting the cache
100% success rate, zero HTTP 5xx errors, across the full load test

I also watched this fail in interesting ways during development, for example, my first version of the load test accidentally reused the same idempotency key across requests, which meant Redis was serving cached responses without ever reaching RabbitMQ at all. The queue graphs looked completely flat, which looked like a bug until I realized the test script itself was wrong, not the system. That kind of debugging, figuring out whether the system or the test is lying to you, is most of what this work actually is.

What I'd add next if this were a real production system

This project is a simulation, not a production system, and there are things a real fintech payment platform would need that I haven't built here:

Multi-region failover, since payment infra can't have a single point of failure
Real reconciliation against an external payment processor's records, not just internal consistency
Proper secrets management and PCI-DSS-aligned access controls, rather than local .env files
Chaos testing (deliberately killing a worker mid-transaction) rather than just load testing

But the core lesson holds even at small scale: in payment systems, the interesting infrastructure decisions aren't about which cloud provider or which orchestrator you pick. They're about how you handle the moment two things happen at exactly the same time.

Repo: https://github.com/aashiruu/ledger-sentinel

Hi, I'm Okikiola. Cloud & DevOps Engineer working through infrastructure problems in public

Okikiola Ashiru — Wed, 29 Jul 2026 10:05:55 +0000

Hi, I'm Okikiola — a Cloud & DevOps Engineer based in Lagos, Nigeria.

I came into this field a little sideways. My background is in Biochemistry, but somewhere along the way I got pulled toward infrastructure... the kind of problems where something breaks under load and you have to figure out why, or where a system needs to survive a traffic burst without falling over or double-charging someone.

That pull turned into two years of hands-on work: provisioning cloud environments on AWS and OCI, building CI/CD pipelines, and setting up the observability stacks that catch failures before a customer notices them.

What I work with day to day

Infrastructure as Code: Modular Terraform across AWS and OCI, managing VPCs, IAM policies, and multi-node EKS clusters across projects.
CI/CD: GitHub Actions workflows for automated linting, container builds, and deployment.
Observability: Prometheus, Grafana, and Loki for metrics, logs, and alerting.
Automation: Go and Python (Boto3) scripts for the unglamorous but necessary work, cleaning up orphaned cloud resources, enforcing tagging policies, automating backup rotations.

Why I'm writing here

I've started building deeper projects specifically to understand infrastructure problems at the system level, not just to add tools to a resume.

My most recent one, ledger-sentinel, is a simulated fintech payment gateway I built to work through idempotency locking (Redis SET NX EX), async settlement via RabbitMQ, and database consistency under concurrent load. It's the kind of system where "it worked in my test" and "it's actually safe against double-charges" are two very different claims, and I wanted to prove the second one, not just assume it.

I plan to write about this project and others like it, real architecture decisions, load-test numbers, what broke, and what I'd still need to add before calling any of it production-ready. I'd rather be upfront about the gap between a personal project and a production system than pretend it isn't there.

What I'm currently digging into

Right now I'm spending time on distributed lock contention patterns, Redis cache short-circuiting, and queue backpressure mechanics, mostly because ledger-sentinel showed me how much nuance sits behind "just add a distributed lock" once thousands of requests hit the same endpoint at once.

If you work in DevOps, SRE, or platform engineering, or you're navigating a non-traditional path into cloud infrastructure yourself, I'd like to connect. Always glad to talk through Terraform patterns, observability setups, or a race condition you've had to hunt down.

GitHub: github.com/aashiruu
LinkedIn: linkedin.com/in/okikiolaashiru