Building a SOC 2 Evidence Collector: A Small-Team Alternative to Manual Audit Prep

By Suman Shashikumar

Introduction

SOC 2 compliance has become a near-mandatory requirement for any B2B SaaS company selling into the mid-market or enterprise — but for a small engineering team without dedicated compliance staff, the evidence-gathering process is still often manual: screenshotting IAM console pages, exporting access logs, and documenting change-management approvals by hand ahead of an audit.

Commercial platforms like Vanta, Drata, and Secureframe solve this well for companies that can afford them, offering hundreds of integrations and continuous, cross-framework monitoring. But for an early-stage team not yet ready for that spend, the alternative is usually a spreadsheet and a few stressful days before the auditor's kickoff call.

I built the SOC 2 Evidence Collector as a personal, open-source project to explore that gap: a lightweight, read-only tool that pulls access-control, logging, and change-management evidence directly from AWS and GitHub, and maps each finding to the specific SOC 2 Trust Service Criterion it satisfies. It isn't a competitor to the established platforms in this space — it's a scoped, free starting point for a small team's first audit cycle, and a project that let me work through the practical engineering problems (API permission models, evidence normalization, control mapping) that sit underneath that whole product category.

The Problem

For a small team preparing for a first SOC 2 audit, the friction isn't understanding what the auditor wants — the Trust Service Criteria are publicly documented — it's the manual labor of proving it:

• Which IAM users have MFA enabled, and which don't?
• Is CloudTrail actually logging, and across all regions?
• Are any S3 buckets accidentally publicly accessible?
• Which GitHub repositories require a pull request review before merging to their default branch?

Each of these is a five-minute manual check in isolation. Across a real environment with a dozen-plus IAM users and repositories, it becomes hours of repetitive, error-prone work — and it has to be redone every audit cycle unless someone builds tooling around it.

Approach

The tool is a FastAPI backend with two read-only collectors:

• AWS collector — checks IAM user MFA status and active access keys (CC6.1), CloudTrail logging and multi-region configuration (CC7.2), and S3 bucket public-access-block settings (CC6.6), via boto3.
• GitHub collector — checks repository collaborator/admin counts (CC6.1) and default-branch protection status, including required PR reviews (CC8.1), via the GitHub REST API.

Each collector returns structured, per-item findings with a pass/fail status and maps directly back to the specific SOC 2 control it evidences, rather than returning raw, unlabeled API output. A simple frontend dashboard triggers each collection on demand and renders the results as readable tables instead of raw JSON.

One practical lesson came from making the tool work against a real personal GitHub account rather than only an organization: GitHub's API treats personal accounts and Organizations as different resource types with different endpoints, and its branch-protection endpoint silently returns "not found" for a token lacking sufficient scope — indistinguishable, from the API response alone, from a genuinely unprotected branch. Both are the kind of small, real-world edge cases that don't show up until you test against actual infrastructure rather than a clean, idealized demo environment.

Results

I ran the tool against two real, personal environments — an AWS account and a GitHub account with 14 repositories accumulated over several years of coursework and side projects — to validate it end-to-end, not just against a synthetic test case.

Initial AWS findings:

Control	Result
IAM MFA enforcement (CC6.1)	0 of 16 users had MFA enabled
CloudTrail logging (CC7.2)	1 trail evaluated — logging enabled, multi-region, log file validation on — PASS
S3 public access block (CC6.6)	0 of 6 buckets had public access blocked

Initial GitHub findings:

Control	Result
Repository access control (CC6.1)	14 repositories evaluated
Branch protection (CC8.1)	0 of 14 repositories required PR review on their default branch

These findings were real, not staged — largely from older accounts used for certification study and coursework, which is a realistic picture of how access hygiene tends to degrade over time without active management. That's arguably the more useful validation: a tool that only ever demonstrates against a pristine test environment hasn't really proven anything.

I then used the tool's own findings to remediate two specific gaps — enabling MFA on one IAM user and adding a branch-protection rule to one repository — and re-ran the collection to confirm the tool correctly detected the fix:

Control	Before	After
IAM MFA — target user	FAIL	PASS
Branch protection — target repository	FAIL	PASS

All other findings were left unchanged, both to preserve a realistic baseline and to confirm the tool wasn't simply reporting a false "all clear."

Lessons Learned

A clean demo proves less than a messy one. Testing against an old, semi-abandoned account with over a dozen stale users was more useful than testing against a freshly created, idealized environment — it surfaced real permission edge cases (like the GitHub org-vs-user endpoint difference) that a synthetic test wouldn't have caught.

API error ambiguity is a real design problem, not an edge case. When an API returns the same "not found" response for both "this doesn't exist" and "you don't have permission to see this," a tool built on top of it needs to account for that ambiguity explicitly rather than assuming a clean signal.

Scoped and honest beats broad and vague. The commercial platforms in this space are extensive for good reason — multi-framework support, hundreds of integrations, and continuous monitoring are hard, valuable engineering problems. A small open-source project's value isn't in claiming to compete with that; it's in being a genuinely useful, transparent starting point for a team that isn't there yet.

Conclusion

Building this tool end-to-end — including debugging it against messy, real infrastructure rather than a synthetic demo — was a useful way to work through the practical engineering problems underneath the SOC 2 compliance-automation category: evidence normalization, control mapping, and the gap between an API's documented behavior and its actual behavior under real permission constraints. The project remains open-source and actively maintained, with a public roadmap for additional Trust Service Criteria and cloud providers.

---

Suman Shashikumar is a Site Reliability Engineer with 10+ years of software engineering experience, currently at J.P. Morgan. This project was built entirely independently, outside the scope of his employment.