DEV Community: IRM Consulting & Advisory

What a SOC 2 auditor actually asks about your CI/CD pipeline

IRM Consulting & Advisory — Wed, 09 Sep 2026 19:31:29 +0000

The first SOC 2 audit I sat in on with an engineering team went sideways in about eleven minutes.

Not because the pipeline was bad. It was a good pipeline. Trunk-based, tests gated on merge, deploys on green, everything in Terraform. The kind of setup an engineering lead is quietly proud of.

The auditor asked one question: "Show me that the code deployed to production on 14 March was reviewed by someone other than the person who wrote it."

Long pause. Everyone knew the answer was yes. Nobody could produce it.

That gap, between a control that genuinely works and a control you can prove worked, is where most first-time SOC 2 audits burn six weeks they did not budget for. I have watched it happen at companies with far better engineering hygiene than the average. It has almost nothing to do with how secure you are.

Here is what auditors actually ask, in the order they ask it, and what satisfies them.

1. Who can push to main, and how do you know?

The control is separation of duties. The evidence is your branch protection configuration plus a population of merges over the audit window.

What auditors want to see:

Branch protection on your default branch, with required reviews set to at least one
Force-push and branch deletion disabled
Admin bypass either disabled, or enabled with an alerting trail for every use

That last point is where teams lose. Almost every repo has one or two people with admin override, because at some point production was down at 2am and somebody needed it. That is fine. What is not fine is having no record of when it was used.

Turn on the audit log export before your window starts. On GitHub that is Organization settings, then Audit log, then streaming to your log destination. If your observation window has already started and you did not have it on, you cannot retroactively create it, and you will be writing a management response explaining a gap instead.

2. Can a change reach production without passing your gates?

Auditors are testing whether your pipeline is the only path to production. Almost always, it is not.

The usual culprits:

A manual deploy script that someone still has on their laptop
Direct console or CLI access to the production cluster
A hotfix workflow that skips the test stage by design
Infrastructure changes applied by hand outside Terraform

None of these are automatically findings. A documented emergency change procedure with post-hoc review is an entirely acceptable control. An undocumented one that three engineers know about is a finding.

Write the emergency path down. Say who can invoke it, what approval it needs after the fact, and where the record lives. That one paragraph closes a control most teams fail on.

3. Show me a change from ticket to production

This is the walkthrough, and it is where the eleven-minute failure happens.

The auditor picks a date, usually at random, from your deploy history. You need to link, for that one change:

The ticket or issue describing the change
The pull request implementing it
The approval, by a different human
The passing test run
The deployment record with a timestamp

Five artifacts, four systems. If your ticket IDs are not in your commit messages or PR titles, you are going to be reconstructing this by hand, per sample, for however many samples the auditor pulls.

The fix costs about twenty minutes. Add a commit message convention that includes the ticket ID, and enforce it in a pre-merge check. Every subsequent sample becomes a two-link lookup instead of an archaeology project.

4. What happens when a dependency has a critical CVE?

Auditors are not asking whether you scan. Everyone scans. They are asking what happens next.

The control they are testing is vulnerability management, and it needs three things you probably have not written down:

A severity threshold that triggers action
A remediation window per severity
Evidence that you met the window, or documented why you did not

"We use Dependabot" is not a control. "Critical findings are remediated within 7 days, high within 30, tracked in our issue board with the security label, and reviewed monthly" is a control. The tooling is identical. The difference is entirely in whether you defined the policy before the auditor asked.

The pattern underneath all four

Every one of these is the same shape. The engineering practice is already sound. What is missing is the artifact that proves it was sound on a specific date to someone who was not in the room.

That reframing matters, because it changes what you spend your pre-audit time on. Teams tend to assume they need to build new security controls. Mostly they need to make existing controls legible. Configuration exports, audit log retention, a naming convention, a two-paragraph policy document.

The teams that clear a first SOC 2 in ninety days rather than seven months are almost never the more secure ones. They are the ones who set up their evidence collection before the observation window opened rather than after it closed.

Practical sequencing

If you are three to six months out from your first audit:

Now: turn on audit log streaming and set your log retention to cover the full window plus a margin. This is the only item on this list that cannot be fixed later.

This month: write the four documents. Change management, emergency change, vulnerability management, access review. Two pages each is plenty. Auditors read them for existence and coherence, not literary quality.

Next month: run a self-sample. Pick three random deploys from the last thirty days and try to assemble the five artifacts yourself. Whatever takes more than five minutes is what will cost you weeks in the real audit.

Ongoing: do your quarterly access review and actually date it. Undated access reviews are the single most common finding I see, and the single cheapest to avoid.

I lead the vCISO practice at IRM Consulting & Advisory, where a good share of the work is walking SaaS engineering teams through exactly this sequence before an auditor is in the room. Our approach to embedding security into delivery pipelines is here: https://irmcon.com/devsecops-dso/

If you are specifically scoping a first SOC 2, we publish the readiness detail at SOC 2 readiness for SaaS.

What is the control your team found hardest to evidence rather than hardest to implement? I am genuinely curious whether the pattern above holds outside the companies I have worked with.

Threat modeling for a team of six: the version that actually gets done

IRM Consulting & Advisory — Wed, 09 Sep 2026 19:30:24 +0000

Most threat modeling advice is written for companies with a security team. If you are six engineers shipping a product, that advice reads as a list of reasons not to start.

STRIDE per data flow diagram element. Trust boundary enumeration. A DFD maintained as the architecture changes. Attack trees. It is all legitimate, and none of it survives contact with a team where the person who would run it is also on call this week.

So teams do the honest thing and skip it, and then the security questionnaire from a prospect asks "do you perform threat modeling?" and someone writes yes and hopes.

There is a version that fits in one hour and produces something real. I have run it with teams of four and teams of forty. Here it is.

The setup

One hour. One feature or service, not the whole system. Three people: whoever owns the code, whoever owns the infrastructure it runs on, and one person who did not build it. That third person is the important one and the one teams skip.

A whiteboard or a shared document. No tooling.

Step one: draw it, badly (10 minutes)

Boxes and arrows. What talks to what. Where does data enter the system, where does it rest, where does it leave.

Do not make it pretty and do not make it complete. The purpose of the diagram is to give six eyes the same mental model for the next fifty minutes, not to be an artifact you maintain. If it is wrong, someone will say so, and that correction is itself worth the ten minutes.

Mark three things on it:

Every point where data crosses from something you control to something you do not
Every point where a user-supplied value influences what the code does
Every place credentials or tokens live

That is your trust boundary map. You did not have to know the term to draw it.

Step two: four questions (35 minutes)

This is Adam Shostack's framing and it is the most useful thing in the field. Ask, in order:

What are we working on? You just drew it. Say it out loud in two sentences. If you cannot, the diagram is wrong.

What can go wrong? This is the whole exercise. Go around the diagram element by element. For each one, ask what an attacker who reached this point could do. Not "is this secure." What could they do.

The prompts that consistently produce real findings in small teams:

What happens if this identifier is changed to another tenant's identifier?
What does this component do if the thing it calls returns garbage, or nothing, or takes forever?
Who can read this log, and what is in it?
If this credential leaked today, what is the blast radius and how would we know?
What does this endpoint do before it checks authorization?

That last one finds more real problems than any other single question I have used.

Write every answer down, including the ones you immediately dismiss. Dismissing something out loud with two colleagues present is a different act from never having considered it.

What are we going to do about it? Sort the list into three buckets. Fix now, accept and document, needs more thought. Be ruthless about the middle bucket. Accepting a risk deliberately is a valid engineering decision. Accepting it by forgetting is not.

Did we do a good job? Five minutes at the end. Did we cover the diagram. Did anyone hold back a concern. What would we do differently next time.

Step three: the artifact (15 minutes)

One page. Date, participants, the diagram photographed or pasted, the findings table with three columns: what could go wrong, decision, owner.

That page is the deliverable. It is what you point at when a prospect asks whether you threat model, and it is what a SOC 2 or ISO 27001 auditor will accept as evidence of secure design practice, which is a control most small teams have nothing for.

Do it once per meaningful new service or major architectural change. Not per sprint, not per PR. Perhaps four to six times a year for a team your size.

What this deliberately gives up

It is worth being honest about the trade.

You will not get systematic coverage. Structured STRIDE catches categories of issue that ad hoc questioning misses, particularly repudiation and elevation-of-privilege classes that do not occur to people naturally.

You will not build a maintained model. The diagram is disposable, so the next session starts from scratch.

You will be biased toward what the room already worries about. The outsider in step one is the partial mitigation, and it is partial.

What you get in exchange is that it happens. A threat model that gets run four times a year and finds three real issues beats a rigorous methodology that gets deferred for eighteen months. I would rather argue about the coverage gaps of something that exists.

The finding pattern

Across small-team sessions, the same three categories dominate:

Authorization checked after the expensive thing. The endpoint fetches, computes, then checks whether you were allowed. Timing and error-message differences leak existence of other tenants' objects.

Trust in an internal service that is not internal. Something on the private network is reachable from somewhere it should not be, usually via a path added months after the original design.

Secrets with no rotation story. Not leaked secrets. Secrets where nobody can answer what the recovery procedure would be, which means there is not one.

None of these require a specialist to find. They require an hour and someone asking the second question.

I run the vCISO practice at IRM Consulting & Advisory in Toronto, working with SaaS and scaling teams on exactly this problem: security practices proportionate to the size of the team actually doing them. Our threat modeling approach for small engineering teams is written up at https://irmcon.ca/threat-modeling-tm/

If the constraint is that nobody owns this work rather than that nobody knows how to do it, that is the gap fractional security leadership is meant to fill. Detail here: https://irmcon.ca/virtual-ciso-services-vciso/

Curious what the third category is for other people. If you have run a session like this, what kept coming up that is not on my list?