DEV Community: IRM Consulting & Advisory

Logging for the incident you have not had yet

IRM Consulting & Advisory — Sun, 27 Sep 2026 12:26:59 +0000

Most application logs are written for the developer who is debugging right now. Incident logs are written for a stranger reading them six weeks later under pressure, possibly a customer's auditor, possibly your own future self at 2am. Those are different jobs, and the second one is the one almost nobody designs for until they need it.

Here is the shape of the thing that works, at a size a small team can actually build and maintain.

1. Start from the questions, not from the log lines

An incident is not a search problem, it is an answering problem. There are five questions, and every incident is some arrangement of them:

Who did it? Not the display name, the stable identifier.
What did they touch? Which specific records or resources, not which endpoint.
When, in a timeline you can line up across services?
From where? IP, device, session, API key.
What changed as a result, and is it still that way?

If your logging cannot answer those five for a given account over a given window, the volume of what you are storing is irrelevant. Work backwards from the questions and you get a much smaller, much more useful set of events than "log everything and buy a bigger plan".

2. The minimum event set

For a small SaaS product, this is the list I would want in place before anything else:

Authentication. Success and failure, with the reason for failure. Include MFA challenges issued, passed, and bypassed. Include session creation and, importantly, session termination.
Authorization denials. Successes are noise at first. Denials are the signal, because a compromised account probing for scope generates them in patterns a human never does.
Privilege and role changes. Any change to who can do what, including group membership, including changes made by automation.
Bulk read and export. The single most under-logged event in small products. One record read is a user working. Ten thousand records read is either a report or an exfiltration, and you want to be able to tell which.
Configuration and secret changes. Rotations, new API keys, changed webhook targets, changed SSO settings, new OAuth grants.
Integration activity. Which token acted, on whose behalf, against which tenant. Third party integrations are where small products quietly get very large blast radiuses.
Admin actions taken on behalf of a customer. Support impersonation is a normal feature and a genuine risk. Log it as its own event type, not as the customer's own activity.

That list is deliberately boring. Boring is what you want. Compare it against your own product tonight and you will probably find two or three of these are missing entirely.

3. Fields that make logs joinable

A log that cannot be joined to another log is an anecdote. Four fields do most of the work:

A stable actor identifier that never changes, alongside the human readable one. Emails get changed, users get renamed, and your six week old logs become unsearchable.
A request or trace identifier propagated across every service and job that handles the request, including async workers. This is the difference between a timeline and a pile.
A tenant identifier on every event in a multi tenant system. You will need to answer "what happened in this one customer's account" and you will need to answer it without reading anyone else's data.
UTC timestamps in a single format, ideally RFC 3339. Mixed timezones and mixed formats are how a reconstruction quietly goes wrong rather than loudly failing.

Add the outcome of the action, not just the attempt. "Delete requested" and "delete completed" are different facts.

4. What to keep out

This is the part that gets skipped, and it is the part that turns a log store into a liability.

Logs inherit the sensitivity of whatever you put in them. Full request and response bodies will eventually carry personal data, health data, payment data, or a token. Once that is in a system with wide read access and long retention, you have built a second copy of your most sensitive data with none of the controls you put around the first copy.

Practical rules:

Redact at the point of emission, not in the pipeline. A redaction step downstream means the raw value existed somewhere, and somewhere is usually a disk.
Log identifiers, not values. card_id not the card. user_id not the email. document_id not the contents.
Allowlist what gets logged from a payload rather than denylisting what does not. Denylists fail silently the moment someone adds a field.
Treat secrets as a build time problem. Scan for token shaped strings in log output in CI, the same way you scan for them in commits.
Say in writing which categories of personal data are permitted in logs. If nobody has written it down, the answer in practice is "all of them".

5. Retention is a decision, not a default

Pick the number on purpose. Two horizons, two different reasons:

Hot, searchable. Long enough to investigate something you notice. Thirty days is short for this in practice, because breaches are routinely discovered long after they start.
Cold, retrievable. Long enough to satisfy the obligation you actually have. That comes from your contracts, your regulator, and your customers' expectations, not from a blog post.

Write down both numbers and the reason for each. An auditor asking about log retention is not really asking for the number. They are asking whether anyone decided it.

6. Integrity, or why the attacker cleans up after themselves

If the same credentials that run your application can delete your logs, then the logs describe only the incidents your attacker did not care about.

Cheap version of getting this right, which is within reach for a team of five:

Ship logs out of the environment that produced them, to a separate account or project with its own access boundary.
Make the destination append only, with object lock or the equivalent on the archive.
Make deletion a separate, logged, alerted privilege held by a small number of named people.
Alert on the audit log going quiet. Silence is an event.

7. Test it before you need it

Here is a two hour exercise worth more than another dashboard.

Pick a plausible scenario. An employee laptop is compromised and their session token is stolen on a Friday night. On Monday, open only your log tooling, no code reading, no database queries, and try to answer the five questions from the first section. Time yourself and write down each point where you had to guess.

The gaps you find will not be exotic. They will be things like bulk exports not being logged as their own event, or the support impersonation path writing events under the customer's identity, or worker jobs dropping the trace identifier. Every one of those is a small ticket now and a bad week later.

Do this once a quarter, change the scenario, and let it drive your backlog. That single habit is most of the difference between a logging setup that is expensive and one that is useful. It also happens to produce, almost as a by product, the evidence that the SOC 2 common criteria on monitoring and evaluating security events expect you to have.

8. Two references worth the time

NIST SP 800-92, Guide to Computer Security Log Management. Old, still the clearest articulation of the problem.
The OWASP Logging Cheat Sheet, for what to log and what never to.

I work with small SaaS teams at IRM Consulting & Advisory, and the logging conversation above is one I have had more times than any other. The companion conversation, about what should never end up in a log in the first place, usually turns into data security and privacy reviews instead. Flagging the affiliation so you can weigh the advice accordingly.

I am curious about one thing in particular. For those of you who have actually run the exercise in the last section, what was the gap you did not expect to find?

What a SOC 2 auditor actually asks about your CI/CD pipeline

IRM Consulting & Advisory — Wed, 09 Sep 2026 19:31:29 +0000

The first SOC 2 audit I sat in on with an engineering team went sideways in about eleven minutes.

Not because the pipeline was bad. It was a good pipeline. Trunk-based, tests gated on merge, deploys on green, everything in Terraform. The kind of setup an engineering lead is quietly proud of.

The auditor asked one question: "Show me that the code deployed to production on 14 March was reviewed by someone other than the person who wrote it."

Long pause. Everyone knew the answer was yes. Nobody could produce it.

That gap, between a control that genuinely works and a control you can prove worked, is where most first-time SOC 2 audits burn six weeks they did not budget for. I have watched it happen at companies with far better engineering hygiene than the average. It has almost nothing to do with how secure you are.

Here is what auditors actually ask, in the order they ask it, and what satisfies them.

1. Who can push to main, and how do you know?

The control is separation of duties. The evidence is your branch protection configuration plus a population of merges over the audit window.

What auditors want to see:

Branch protection on your default branch, with required reviews set to at least one
Force-push and branch deletion disabled
Admin bypass either disabled, or enabled with an alerting trail for every use

That last point is where teams lose. Almost every repo has one or two people with admin override, because at some point production was down at 2am and somebody needed it. That is fine. What is not fine is having no record of when it was used.

Turn on the audit log export before your window starts. On GitHub that is Organization settings, then Audit log, then streaming to your log destination. If your observation window has already started and you did not have it on, you cannot retroactively create it, and you will be writing a management response explaining a gap instead.

2. Can a change reach production without passing your gates?

Auditors are testing whether your pipeline is the only path to production. Almost always, it is not.

The usual culprits:

A manual deploy script that someone still has on their laptop
Direct console or CLI access to the production cluster
A hotfix workflow that skips the test stage by design
Infrastructure changes applied by hand outside Terraform

None of these are automatically findings. A documented emergency change procedure with post-hoc review is an entirely acceptable control. An undocumented one that three engineers know about is a finding.

Write the emergency path down. Say who can invoke it, what approval it needs after the fact, and where the record lives. That one paragraph closes a control most teams fail on.

3. Show me a change from ticket to production

This is the walkthrough, and it is where the eleven-minute failure happens.

The auditor picks a date, usually at random, from your deploy history. You need to link, for that one change:

The ticket or issue describing the change
The pull request implementing it
The approval, by a different human
The passing test run
The deployment record with a timestamp

Five artifacts, four systems. If your ticket IDs are not in your commit messages or PR titles, you are going to be reconstructing this by hand, per sample, for however many samples the auditor pulls.

The fix costs about twenty minutes. Add a commit message convention that includes the ticket ID, and enforce it in a pre-merge check. Every subsequent sample becomes a two-link lookup instead of an archaeology project.

4. What happens when a dependency has a critical CVE?

Auditors are not asking whether you scan. Everyone scans. They are asking what happens next.

The control they are testing is vulnerability management, and it needs three things you probably have not written down:

A severity threshold that triggers action
A remediation window per severity
Evidence that you met the window, or documented why you did not

"We use Dependabot" is not a control. "Critical findings are remediated within 7 days, high within 30, tracked in our issue board with the security label, and reviewed monthly" is a control. The tooling is identical. The difference is entirely in whether you defined the policy before the auditor asked.

The pattern underneath all four

Every one of these is the same shape. The engineering practice is already sound. What is missing is the artifact that proves it was sound on a specific date to someone who was not in the room.

That reframing matters, because it changes what you spend your pre-audit time on. Teams tend to assume they need to build new security controls. Mostly they need to make existing controls legible. Configuration exports, audit log retention, a naming convention, a two-paragraph policy document.

The teams that clear a first SOC 2 in ninety days rather than seven months are almost never the more secure ones. They are the ones who set up their evidence collection before the observation window opened rather than after it closed.

Practical sequencing

If you are three to six months out from your first audit:

Now: turn on audit log streaming and set your log retention to cover the full window plus a margin. This is the only item on this list that cannot be fixed later.

This month: write the four documents. Change management, emergency change, vulnerability management, access review. Two pages each is plenty. Auditors read them for existence and coherence, not literary quality.

Next month: run a self-sample. Pick three random deploys from the last thirty days and try to assemble the five artifacts yourself. Whatever takes more than five minutes is what will cost you weeks in the real audit.

Ongoing: do your quarterly access review and actually date it. Undated access reviews are the single most common finding I see, and the single cheapest to avoid.

I lead the vCISO practice at IRM Consulting & Advisory, where a good share of the work is walking SaaS engineering teams through exactly this sequence before an auditor is in the room. Our approach to embedding security into delivery pipelines is here: https://irmcon.com/devsecops-dso/

If you are specifically scoping a first SOC 2, we publish the readiness detail at SOC 2 readiness for SaaS.

What is the control your team found hardest to evidence rather than hardest to implement? I am genuinely curious whether the pattern above holds outside the companies I have worked with.

Threat modeling for a team of six: the version that actually gets done

IRM Consulting & Advisory — Wed, 09 Sep 2026 19:30:24 +0000

Most threat modeling advice is written for companies with a security team. If you are six engineers shipping a product, that advice reads as a list of reasons not to start.

STRIDE per data flow diagram element. Trust boundary enumeration. A DFD maintained as the architecture changes. Attack trees. It is all legitimate, and none of it survives contact with a team where the person who would run it is also on call this week.

So teams do the honest thing and skip it, and then the security questionnaire from a prospect asks "do you perform threat modeling?" and someone writes yes and hopes.

There is a version that fits in one hour and produces something real. I have run it with teams of four and teams of forty. Here it is.

The setup

One hour. One feature or service, not the whole system. Three people: whoever owns the code, whoever owns the infrastructure it runs on, and one person who did not build it. That third person is the important one and the one teams skip.

A whiteboard or a shared document. No tooling.

Step one: draw it, badly (10 minutes)

Boxes and arrows. What talks to what. Where does data enter the system, where does it rest, where does it leave.

Do not make it pretty and do not make it complete. The purpose of the diagram is to give six eyes the same mental model for the next fifty minutes, not to be an artifact you maintain. If it is wrong, someone will say so, and that correction is itself worth the ten minutes.

Mark three things on it:

Every point where data crosses from something you control to something you do not
Every point where a user-supplied value influences what the code does
Every place credentials or tokens live

That is your trust boundary map. You did not have to know the term to draw it.

Step two: four questions (35 minutes)

This is Adam Shostack's framing and it is the most useful thing in the field. Ask, in order:

What are we working on? You just drew it. Say it out loud in two sentences. If you cannot, the diagram is wrong.

What can go wrong? This is the whole exercise. Go around the diagram element by element. For each one, ask what an attacker who reached this point could do. Not "is this secure." What could they do.

The prompts that consistently produce real findings in small teams:

What happens if this identifier is changed to another tenant's identifier?
What does this component do if the thing it calls returns garbage, or nothing, or takes forever?
Who can read this log, and what is in it?
If this credential leaked today, what is the blast radius and how would we know?
What does this endpoint do before it checks authorization?

That last one finds more real problems than any other single question I have used.

Write every answer down, including the ones you immediately dismiss. Dismissing something out loud with two colleagues present is a different act from never having considered it.

What are we going to do about it? Sort the list into three buckets. Fix now, accept and document, needs more thought. Be ruthless about the middle bucket. Accepting a risk deliberately is a valid engineering decision. Accepting it by forgetting is not.

Did we do a good job? Five minutes at the end. Did we cover the diagram. Did anyone hold back a concern. What would we do differently next time.

Step three: the artifact (15 minutes)

One page. Date, participants, the diagram photographed or pasted, the findings table with three columns: what could go wrong, decision, owner.

That page is the deliverable. It is what you point at when a prospect asks whether you threat model, and it is what a SOC 2 or ISO 27001 auditor will accept as evidence of secure design practice, which is a control most small teams have nothing for.

Do it once per meaningful new service or major architectural change. Not per sprint, not per PR. Perhaps four to six times a year for a team your size.

What this deliberately gives up

It is worth being honest about the trade.

You will not get systematic coverage. Structured STRIDE catches categories of issue that ad hoc questioning misses, particularly repudiation and elevation-of-privilege classes that do not occur to people naturally.

You will not build a maintained model. The diagram is disposable, so the next session starts from scratch.

You will be biased toward what the room already worries about. The outsider in step one is the partial mitigation, and it is partial.

What you get in exchange is that it happens. A threat model that gets run four times a year and finds three real issues beats a rigorous methodology that gets deferred for eighteen months. I would rather argue about the coverage gaps of something that exists.

The finding pattern

Across small-team sessions, the same three categories dominate:

Authorization checked after the expensive thing. The endpoint fetches, computes, then checks whether you were allowed. Timing and error-message differences leak existence of other tenants' objects.

Trust in an internal service that is not internal. Something on the private network is reachable from somewhere it should not be, usually via a path added months after the original design.

Secrets with no rotation story. Not leaked secrets. Secrets where nobody can answer what the recovery procedure would be, which means there is not one.

None of these require a specialist to find. They require an hour and someone asking the second question.

I run the vCISO practice at IRM Consulting & Advisory in Toronto, working with SaaS and scaling teams on exactly this problem: security practices proportionate to the size of the team actually doing them. Our threat modeling approach for small engineering teams is written up at https://irmcon.ca/threat-modeling-tm/

If the constraint is that nobody owns this work rather than that nobody knows how to do it, that is the gap fractional security leadership is meant to fill. Detail here: https://irmcon.ca/virtual-ciso-services-vciso/

Curious what the third category is for other people. If you have run a session like this, what kept coming up that is not on my list?