DEV Community: InstaDevOps

SOC 2 for Startups: The DevOps Work Nobody Warns You About

InstaDevOps — Thu, 27 Aug 2026 13:47:04 +0000

SOC 2 for Startups: The DevOps Work Nobody Warns You About

The pattern is always the same. A deal is moving, the champion is enthusiastic, and then procurement forwards a 200-row security questionnaire with a cell that says "SOC 2 Type II report attached?" The deal does not die. It just stops moving until you answer.

At that point most founders start shopping for compliance software, because that is what the ads are for. The software is useful. It is also the smaller half of the job. The larger half is infrastructure work that has to be done by someone who can log into your AWS account and change things.

Here is what SOC 2 actually asks of your systems, what is engineering versus paperwork, and how long it really takes.

What SOC 2 is, stated correctly

SOC 2 is not a certification and there is no such thing as being "SOC 2 certified." It is an attestation: a licensed CPA firm examines your controls and issues a report with their opinion. You do not get a badge from a standards body, you get a PDF from an audit firm that your customer's security team reads.

The controls are organized under the AICPA Trust Services Criteria. Security (the common criteria) is required in every SOC 2 engagement. Availability, Confidentiality, Processing Integrity, and Privacy are optional categories you include based on what you promise customers. Most startups scope Security only for the first report, sometimes adding Availability and Confidentiality if contracts require it. Every category you add expands the evidence you have to produce, so scope deliberately.

There are two report types:

Type I describes your controls and whether they are suitably designed as of a specific date. It is a point-in-time snapshot.
Type II tests whether those controls actually operated effectively across an observation window, commonly 3 to 12 months.

Enterprise buyers almost always want Type II. Type I is a reasonable interim answer that shows the program exists, and some buyers will accept it with a commitment to Type II. Ask your champion which one actually unblocks the deal before you spend money.

One more thing worth being clear-eyed about: an auditor does not scan your infrastructure. They ask you for evidence, then test samples of it. If you pulled 12 offboarding tickets, they might test 5. That cuts both ways. It means you cannot be caught by an automated scan you never ran, and it means a single sampled item with no evidence behind it becomes an exception in your report.

Which criteria map to which infrastructure control

Most of the common criteria that touch engineering come down to a handful of concrete things.

Trust Services Criteria theme	Concrete DevOps control	Evidence the auditor asks for
Logical access provisioning and removal	SSO with a single identity provider, no local IAM users with console access, group-based role assignment	Termination tickets matched to access-removal timestamps
Least privilege for production	Role assumption with MFA, time-bound elevation, no shared accounts	IAM policy exports, role trust policies, sample access reviews
Change management	Pull requests with required review, protected branches, CI checks before deploy	PR history for a sampled set of production deploys
System monitoring	Centralized logs, CloudTrail in all regions, alerting with a named on-call owner	Alert history plus proof someone responded
Encryption in transit and at rest	TLS enforced at the edge and between services, KMS on volumes, buckets, and databases	Config rule results, bucket policies, RDS settings
Backup and recovery	Automated backups with defined retention, plus a documented restore test	Backup config and a dated restore test record
Vulnerability management	Dependency and image scanning, patch SLAs by severity, tracked remediation	Scan output and tickets showing fixes inside the SLA
Vendor management	Inventory of subprocessors with their own reports reviewed	Vendor list, review dates, signed DPAs

Read the right-hand column again. Almost every row asks for a record produced over time, not a screenshot of a settings page. That is the thing that surprises people.

Turning on the audit trail properly

CloudTrail is the most valuable single piece of evidence infrastructure in an AWS account, and the default setup most startups have is not sufficient. You want a multi-region trail, log file validation enabled, an immutable destination bucket, and KMS encryption.

resource "aws_s3_bucket" "audit_logs" {
  bucket = "acme-audit-logs-prod"
}

resource "aws_s3_bucket_versioning" "audit_logs" {
  bucket = aws_s3_bucket.audit_logs.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_public_access_block" "audit_logs" {
  bucket                  = aws_s3_bucket.audit_logs.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_server_side_encryption_configuration" "audit_logs" {
  bucket = aws_s3_bucket.audit_logs.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.audit.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "audit_logs" {
  bucket = aws_s3_bucket.audit_logs.id
  rule {
    id     = "retain-13-months"
    status = "Enabled"
    filter {}
    expiration {
      days = 400
    }
  }
}

resource "aws_cloudtrail" "org" {
  name                          = "acme-org-trail"
  s3_bucket_name                = aws_s3_bucket.audit_logs.id
  kms_key_id                    = aws_kms_key.audit.arn
  is_multi_region_trail         = true
  include_global_service_events = true
  enable_log_file_validation    = true
  enable_logging                = true

  depends_on = [aws_s3_bucket_policy.audit_logs]
}

The trail needs a bucket policy allowing the CloudTrail service to write, scoped with a source ARN condition so another account cannot drop logs into your bucket:

data "aws_iam_policy_document" "audit_logs" {
  statement {
    sid     = "AWSCloudTrailAclCheck"
    actions = ["s3:GetBucketAcl"]
    resources = [aws_s3_bucket.audit_logs.arn]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = ["arn:aws:cloudtrail:${var.region}:${var.account_id}:trail/acme-org-trail"]
    }
  }

  statement {
    sid       = "AWSCloudTrailWrite"
    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.audit_logs.arn}/AWSLogs/${var.account_id}/*"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"
      values   = ["bucket-owner-full-control"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = ["arn:aws:cloudtrail:${var.region}:${var.account_id}:trail/acme-org-trail"]
    }
  }

  statement {
    sid       = "DenyInsecureTransport"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [aws_s3_bucket.audit_logs.arn, "${aws_s3_bucket.audit_logs.arn}/*"]
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

Retention matters here. If your observation window is 6 months and your log retention is 30 days, you cannot produce evidence for month one. Set retention to cover the window plus the audit fieldwork, and set it before the window opens.

Change management is a GitHub configuration problem

The change management criteria are the easiest to satisfy and the most commonly failed, because teams satisfy them informally. Everyone reviews each other's PRs, but nothing enforces it, so when the auditor samples 25 production deploys and two of them were direct pushes to main at 2am during an incident, those are exceptions.

Enforce it in configuration so the evidence generates itself:

resource "github_branch_protection" "main" {
  repository_id                   = github_repository.api.node_id
  pattern                         = "main"
  enforce_admins                  = true
  require_conversation_resolution = true
  required_linear_history         = true
  allows_force_pushes             = false
  allows_deletions                = false

  required_pull_request_reviews {
    required_approving_review_count = 1
    dismiss_stale_reviews           = true
    require_code_owner_reviews      = true
  }

  required_status_checks {
    strict   = true
    contexts = ["build", "test", "security-scan"]
  }
}

Two notes on this. First, enforce_admins = true is the line people quietly remove when it gets inconvenient, and removing it is exactly what the auditor's sample will surface. Decide now whether you can live with it. Second, if you have an emergency break-glass path, document it as a policy with an after-the-fact review requirement rather than pretending it does not exist. Auditors are far more comfortable with a documented exception process than with an undocumented one they discover in the logs.

The parts that are genuinely engineering

Strip out everything a template can produce and this is the actual work list:

Identity consolidation. Getting every human out of long-lived IAM users and into SSO with role assumption. This is the single biggest lift for most startups because it touches every tool, and because there is always one legacy service account nobody wants to rotate.

Offboarding that leaves a trail. Not just removing access, but producing a record showing access was removed within your stated SLA. If offboarding is a Slack message to whoever is around, you have no evidence. Wire it to a ticket and to your identity provider's audit log.

Encryption sweep. Unencrypted EBS volumes, RDS instances created before you cared, S3 buckets without default encryption, internal traffic on plain HTTP. Each is a small fix and there are always more than you expected. AWS Config with managed rules like ENCRYPTED_VOLUMES, RDS_STORAGE_ENCRYPTED, and S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED will find them and, more usefully, keep producing dated compliance results as evidence.

Restore testing. Backups being configured is not the control. Restoring from them is. You need a dated record of an actual restore, ideally into a scratch environment, with the outcome written down. This is the control I see missing most often, and it is also the one that would save the company in a real incident.

Alerting with an owner. An alert that fires into a channel nobody reads fails the criteria, and the auditor will find it by asking for the response to a specific alert. You need a defined on-call rotation and a record of triage.

Vulnerability management with an SLA. Scanning is easy. Having written severity-based remediation windows and evidence you met them is the part that takes discipline.

Everything else (the information security policy, the incident response plan, the risk assessment, the vendor register, the security awareness training) is documentation. Real, necessary documentation that someone has to write and management has to approve, but not engineering.

What compliance vendors do and do not do

Compliance platforms are worth buying. They give you policy templates, they collect evidence continuously through read-only integrations, they track control status, and they shorten the auditor's fieldwork considerably. If your alternative is a folder of screenshots, buy the tool.

What they do not do is fix your infrastructure. The platform will tell you that 14 EBS volumes are unencrypted. It will not encrypt them. It will tell you that three people still have IAM access keys. It will not migrate you to SSO. It will flag that no restore test is recorded. It will not run one.

That gap is the entire project. The dashboard turning green requires someone to plan a volume-by-volume encryption migration with downtime windows, rebuild the access model, and change how deploys work. Budget for that person, whether they are on your team or not.

A realistic timeline

For a startup with a normal AWS setup and no prior compliance work, remediation runs 4 to 10 weeks of focused engineering time. That is the part you control and the part that is usually underestimated.

After remediation you either take a Type I quickly (weeks) or start your Type II observation window. The window itself is calendar time you cannot compress: a 3 month window takes 3 months. Then auditor fieldwork and report issuance adds a few more weeks.

So "SOC 2 in 90 days" is achievable if it means a Type I plus an open Type II window, and it is not achievable if it means a Type II report in hand. Say that clearly to your customer. Most will accept a Type I and a dated commitment. The ones who will not were probably never going to close this quarter.

The five failure modes

No offboarding trail. Access was removed, but nothing recorded when or by whom.
Shared production credentials. A root key or a shared login in a password manager, which makes every action in the audit log unattributable.
Backups that have never been restored. The config exists, the test does not.
Alerts nobody acts on. Monitoring is in place, response evidence is not.
Starting the observation window before remediating. Every day of the window with a broken control is a day of failing evidence, and you cannot retroactively fix it.

The order matters. Remediate, verify the controls actually produce evidence, then open the window.

If you want help with the infrastructure half

The policies and the evidence platform you can handle in-house or with a compliance vendor. The infrastructure remediation is where teams stall, because it is real engineering on production systems while you are also shipping product. We do that part for clients: identity consolidation, encryption sweeps, change management enforcement, logging and retention, backup restore testing, and the runbooks that keep the controls operating after the auditor leaves. If your deal is stalled on a questionnaire, a short call is usually enough to tell you which of the items above actually apply to your stack.

Signs Your Engineering Team Needs DevOps Help

InstaDevOps — Wed, 26 Aug 2026 13:47:00 +0000

Signs Your Engineering Team Needs DevOps Help

Most teams do not decide one morning that they have a DevOps problem. It is quieter. Releases slip by a day, then a week. One person gets pulled into every infrastructure question. An incident gets fixed, and three months later something suspiciously similar takes the site down again.

Each of those looks like a local problem with a local cause. Together they are usually one thing: nobody owns the path from a laptop to production, so that path decays a little every sprint.

Below are seven signs, each with a diagnostic to confirm it, why it worsens if ignored, and a first fix you can do this week without buying anything. Where useful I refer to DORA's four key metrics (deployment frequency, lead time for changes, change failure rate, time to restore service), which give shared language for problems otherwise argued about on vibes.

Sign 1: Deploys are batched, scheduled, and slightly scary

You recognise this when releases happen on a fixed day, someone is nominated release captain, and nobody ships on Friday afternoon.

Measure it. If you tag releases, count them per month and count the commits riding along in each:

# Releases per month over the last year
git log --tags --simplify-by-decoration --date=format:'%Y-%m' \
  --pretty='%ad' --since='1 year ago' | sort | uniq -c

# How big was the last batch? (commits between the previous tag and HEAD)
git rev-list --count "$(git describe --tags --abbrev=0 HEAD^)"..HEAD

# No tags? Merges to main are a decent proxy for deploys
git log --merges --first-parent main --since='6 months ago' \
  --date=format:'%Y-%m' --pretty='%ad' | sort | uniq -c

This compounds because batch size and fear feed each other. Big releases fail more often, failures make people release less often, and the next batch is bigger still. You also lose bisection: with forty changes in one release, finding the guilty one takes an afternoon.

First fix: do not rebuild the pipeline, shrink one batch. Take the lowest risk service you own, deploy it on every merge to main, and put anything half-finished behind a flag. Then have someone who did not write the rollback procedure run it against staging while you time them. If it takes over five minutes or needs one specific person, fix that first. Cheap rollback is what makes frequent deploys safe.

Sign 2: One person is the only route to production

The tell is social, not technical. Infra questions in Slack end up with the same name, releases get planned around one person's holidays, and one laptop has the only working Terraform credentials.

Confirm it with history:

# Who has actually touched infrastructure code in the last year?
git log --since='1 year ago' --pretty='%an' -- \
  infra/ terraform/ helm/ .github/workflows/ | sort | uniq -c | sort -rn

If one name has ten times the commits of the next, that is your bus factor. Then ask in a retro: if this person vanished for two weeks, what could we not do? If the answers include deploying, rotating a secret, or restoring a backup, that is the finding.

This compounds unfairly. The person who knows everything is interrupted constantly, so they never get quiet time to write it down or automate it. Then they burn out or take a better offer, and it leaves with them.

First fix: take the three riskiest operations from that list. For each, someone other than the expert writes the runbook, and someone else again runs it against a non-production environment while the expert watches without touching the keyboard. Every place the runbook is wrong is knowledge that was about to walk out the door.

Sign 3: The same incident keeps coming back with a new name

You have a postmortem that references an earlier postmortem. Disk fills up again, the queue backs up again, and on-call has a folder of alerts they know how to silence but not how to fix.

Self-check: put six months of incidents in a table with four columns: trigger, contributing cause, agreed action items, and whether those shipped. The last column is the one that matters. Track change failure rate next to it, meaning the share of deploys needing a hotfix or rollback soon after.

This compounds because unfinished remediation is a promise the team stops believing. Postmortems become ritual, and the alert that has fired five times gets ignored a little faster each time until the real one is missed.

First fix: freeze new action items until the open ones are closed or explicitly dropped, and dropping is a legitimate recorded choice. Give remediation the same tracking, owner, and deadline as feature work rather than a wiki page. Then convert one recurring alert into a check that runs before deploy, so the failure becomes a build error instead of a page at 2am.

Sign 4: "Works on staging" stopped meaning anything

Staging passes and production breaks. Instance sizes differ, someone fixed something in the cloud console months ago and never wrote it into code, and production has environment variables that exist nowhere in the repo.

Drift has a direct read. Run a refresh-only plan against production with nothing pending, then compare workloads across environments:

# Any diff here is a change someone made outside of code
terraform plan -refresh-only -no-color | tee drift.txt

# Compare running images and replica counts between environments
for ns in staging production; do
  kubectl -n "$ns" get deploy -o \
    custom-columns='NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image,REPLICAS:.spec.replicas'
done

# Pods with at least one container running with no resource limits
kubectl get pods -A -o json | jq -r '
  .items[]
  | select(any(.spec.containers[]; .resources.limits == null))
  | "\(.metadata.namespace)/\(.metadata.name)"'

This compounds because trust is the product staging sells. Once it has lied a few times, people stop treating a green run as evidence and real testing quietly migrates to production, while you keep paying for the environment.

First fix: eliminate one difference completely. Usually that is the artifact: build the image once, promote that exact image through environments, and inject everything environment specific as configuration. Then run the refresh-only plan nightly and post the output to a channel, so drift surfaces in a day rather than during an incident.

Sign 5: The cloud bill grows faster than traffic

The bill rises every month and the explanation is always "we're growing". Finance asks which team spent what, and engineering estimates.

Self-check: divide monthly spend by a business number you track, such as requests served or active users, and plot the ratio over twelve months. Growth raises the total and keeps the ratio flat; waste makes it climb. Then look at ownership, because unowned resources are where waste hides:

# Resources with no tags at all are usually resources nobody owns
aws resourcegroupstaggingapi get-resources --region eu-west-1 \
  --query 'ResourceTagMappingList[?length(Tags)==`0`].ResourceARN' --output text

# Unattached EBS volumes still bill every hour
aws ec2 describe-volumes --filters Name=status,Values=available \
  --query 'Volumes[].{ID:VolumeId,GiB:Size,Created:CreateTime}' --output table

This compounds because unowned infrastructure never gets deleted. Nobody removes a resource they cannot attribute, in case something depends on it, so waste becomes a permanent floor under your costs until someone mandates a percentage cut under time pressure.

First fix: require an owner tag on everything created from now on and enforce it in CI for your infrastructure code, so the rule cannot rot. Clean up what those commands found, after checking with whoever created them. Set a budget alert at a threshold that would genuinely surprise you.

Sign 6: Onboarding an engineer to infrastructure takes weeks

New hires ship application code in their first few days, then wait a month before anyone lets them near deployment. The setup docs are out of date and the real instructions live in a Slack thread from last year.

Self-check: give your next new engineer a scripted exercise. From a fresh machine and written docs only, get the app running locally and deploy a trivial change to staging. Nobody helps unless they are fully stuck, and every blocker gets recorded. Do not judge the person: their list is a ranked backlog of everything broken in your setup path.

This compounds because it caps what hiring buys you: if each new engineer costs weeks of senior time, growing the team makes your seniors less productive before it makes them more productive.

First fix: take the top three blockers and collapse them into one command, whether that is a make target, a script, or a devcontainer. The goal is one documented command from clean checkout to running environment. Re-run the exercise with the next hire and see if the time drops.

Sign 7: Nobody can answer "what changed?" during an incident

The first twenty minutes of every incident go on asking who deployed what, whether the migration ran, and if anyone touched the load balancer. Nothing shows deploys, config changes, feature flag flips, infrastructure applies, and vendor status on one timeline.

Self-check: take your most recent incident and, using data alone with no memories and no asking colleagues, reconstruct every change in the twenty-four hours before it started. Time yourself. More than fifteen minutes, or an incomplete answer, means you have no change visibility, and that delay is added to time to restore service on every incident.

This compounds because it slows recovery exactly when speed matters. It also pushes teams toward change freezes, which feel safe and are not: the eventual unfreeze is a large batch, which takes you back to sign 1.

First fix: build one change feed. Every deploy emits an event with service, version, commit SHA, actor, and timestamp into one place everyone can see, such as a channel, dashboard annotations, or a small table. Add infrastructure applies and flag changes to the same feed.

The seven signs at a glance

Sign	What it actually costs	Severity	First fix
Batched, scary deploys	Slow lead time, high change failure rate	High	Shrink one batch, rehearse rollback
Single point of knowledge	Total exposure to one person leaving	Critical	Runbooks written and executed by other people
Repeating incidents	Senior time burnt twice, alert fatigue	High	Close open remediation before opening new items
Environment drift	Testing that proves nothing	Medium	Promote one immutable artifact, check drift nightly
Bill outgrowing traffic	Permanent cost floor, panic cuts later	Medium	Owner tags, delete idle resources, budget alerts
Slow infra onboarding	Hiring stops adding throughput	Medium	One command from checkout to running environment
No answer to "what changed?"	Longer time to restore, freeze reflex	High	One change feed for deploys, applies, and flags

Severity means how fast the problem stops being recoverable on your own.

Hire, partner, or fix the process?

If several of these are familiar, the question is what kind of help you need.

Fix the process yourself. Right call when the signs you recognised are about discipline rather than missing skills: unclosed remediation, no change feed, no rehearsed rollback. Those are days of work, not months. The tradeoff is opportunity cost, and the risk is a migration half done then abandoned when a deadline appears. Only do this if someone gets protected time.

Hire a platform or DevOps engineer. This fits when infrastructure work is continuous, already takes more than half of someone's week, and you can describe the role concretely enough to interview for it. Hiring takes months, seniors are expensive, and you need someone who can technically assess candidates. Watch for the trap in sign 2: one hire into a team with no other infrastructure knowledge recreates the single point of failure under a new name, and lone platform engineers with nobody to review their work tend not to stay long.

Bring in a fractional or outsourced partner. This fits when you need senior judgement across several areas at once (pipeline, Kubernetes, cost, observability) but not forty hours a week of it, or when the scope is bounded and you need it sooner than hiring could deliver. The tradeoffs are context and dependency: any outsider spends time learning your system, and if nothing is written down while they work, you rented a solution rather than acquired one. Guard against that in the contract. Infrastructure code lives in your repositories, runbooks are written for your engineers, and there is a named internal owner from day one.

There is a fourth answer nobody sells you: do nothing on purpose. If you are pre-product-market-fit and the system is two services and a database, slow manual deploys are a defensible tradeoff. Just make it a decision you revisit on a date rather than a thing that happened to you.

If you worked through the self-checks and the answer is a partner, that is what we do at InstaDevOps: senior DevOps on a monthly retainer, with everything we build living in your repositories. Either way, run the diagnostics first. They cost an afternoon and tell you more than any sales conversation.

The First 90 Days With a DevOps Partner: A Realistic Timeline

InstaDevOps — Tue, 25 Aug 2026 13:46:56 +0000

The First 90 Days With a DevOps Partner: A Realistic Timeline

Most writing about hiring a DevOps partner stops at the sales conversation. What decides whether the money was well spent is the ninety days after the contract is signed, and almost nobody writes that part down honestly.

This describes what a competent partner does in the first quarter, in what order and why, plus what your side has to provide. That second half matters: the most common reason a good engagement starts badly has nothing to do with the engineers. It is that nobody could grant access, or nobody was empowered to decide.

None of it is a guarantee. A three person startup on one AWS account moves faster than a fifty person company with four environments nobody fully understands. Treat it as the shape of the work, not a schedule you can hold anyone to.

What has to be true before day one

Five things. Miss any of them and the engagement starts slow, and you blame the wrong party.

A named decision maker. One person who can approve a change without convening a committee. When a partner finds an unencrypted database snapshot in a public bucket on day three, somebody says yes that afternoon.

Access, in writing, with a path. Read-only on day one, write access to non-production by the end of week one, production change rights when the real work starts. Access delays are the single most common cause of a slow start. If issuing a role takes your security team three weeks, fine, but start before the kickoff call, not after.

Context on the product. Not documentation, which you probably do not have. An hour with whoever knows why the system is shaped as it is: which service is fragile, what broke last quarter, which queue nobody wants to touch. That saves weeks of archaeology.

About an hour a week from your engineers. Answering questions, reviewing pull requests against your own infrastructure, confirming a change is safe. A partner who needs ten hours a week is not reducing your load.

Willingness to freeze change briefly. Somewhere in weeks five to eight, deploys pause for a few hours while a pipeline or network path is cut over. Agree in advance when that window can happen. Teams that refuse any freeze drag the cutover across a month of half-finished states, which is riskier than the freeze.

Week 1: access, discovery, and a read-only audit

Nothing changes in week one. That is deliberate. A partner who starts editing production on day two either got lucky or is about to cause an incident.

The work is inventory. Which accounts exist, what runs where, what is managed by code and what was clicked into existence, where the data lives, what the backup situation actually is rather than what the dashboard claims, who holds standing production access and whether any of them have left.

Access for this is read-only, scoped and time-bound. On AWS that means a role assumed from an external account with an external ID, carrying the managed read-only and security audit policies plus an explicit deny on secret material:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenySecretMaterialDuringAudit",
      "Effect": "Deny",
      "Action": [
        "secretsmanager:GetSecretValue",
        "ssm:GetParameter",
        "ssm:GetParameters",
        "ssm:GetParametersByPath",
        "kms:Decrypt",
        "lambda:GetFunction",
        "dynamodb:GetItem",
        "dynamodb:Query",
        "dynamodb:Scan",
        "s3:GetObject"
      ],
      "Resource": "*"
    }
  ]
}

Attach that alongside arn:aws:iam::aws:policy/ReadOnlyAccess and arn:aws:iam::aws:policy/SecurityAudit. An explicit deny beats the allows in a broad managed policy, so the auditor sees that a secret exists, who can reach it, and when it was last rotated, without reading its value or the contents of your buckets and tables. It also settles whether an outside party can read your customer data.

Week one ends with a written risk list ordered by what would hurt most if it happened tomorrow. Expect discomfort. Common entries: backups never restored, one person who is the only one able to deploy, no alert on the thing that actually pages, credentials in a repository, a certificate expiring soon, one instance holding state nobody replicated.

Done looks like: an inventory you did not have, a ranked risk list, agreement on the first three fixes.

Weeks 2 to 4: stop the bleeding

This phase is not architecture. It removes the failure modes that turn a normal Tuesday into a very bad one.

Backups that are proven, not configured. A backup you have never restored is a hypothesis. Verifying one is boring and it is the highest value hour of the quarter. For a custom-format Postgres dump, on a scratch instance, never against production:

#!/usr/bin/env bash
set -euo pipefail

SNAPSHOT="${1:?usage: verify-restore.sh <dump-file>}"
SCRATCH_DB="restore_check_$(date +%Y%m%d%H%M%S)"

createdb "$SCRATCH_DB"
trap 'dropdb --if-exists "$SCRATCH_DB"' EXIT

START=$(date +%s)
pg_restore --dbname="$SCRATCH_DB" --no-owner --exit-on-error --jobs=4 "$SNAPSHOT"
ELAPSED=$(( $(date +%s) - START ))

echo "restore completed in ${ELAPSED}s"

psql --dbname="$SCRATCH_DB" --tuples-only --command "
  SELECT 'rows_in_orders=' || count(*) FROM orders;
  SELECT 'newest_order=' || max(created_at) FROM orders;
"

That answers the two questions a green backup dashboard cannot: how long a restore takes, and how much data you would lose. If the first answer is six hours and your recovery objective is one hour, you found a real problem in week two instead of during an outage.

Alerting that corresponds to reality. Most teams arrive with either no alerts or two hundred that everyone mutes. Cut them to a small set reflecting user-visible failure, then add the two or three that were missing. Each should mean something and have an owner.

Access cleanup. Remove departed staff, replace long-lived keys with roles, put multi-factor on accounts that can spend money or delete data, move root credentials off one person's laptop.

The obvious cost items. Unattached volumes, idle load balancers, old snapshots, oversized non-production instances, logs kept forever at premium tier, environments running full size overnight. None of it needs an architectural change. Deeper savings (right-sizing under real load, commitment purchases, storage class changes) come later, once there is data to decide responsibly. Nobody can tell you the percentage in advance, and anyone who does is guessing.

Done looks like: a timed restore you watched succeed, alerts the on-call person trusts, no orphaned credentials, easy waste removed.

Weeks 5 to 8: the structural work

Now the slower work starts and progress becomes less visible. Warn stakeholders: week six feels quieter than week three.

Infrastructure as code coverage. The goal is not rewriting everything in Terraform. It is that anything you could not survive losing is described in code and can be recreated. Usually that means importing existing resources rather than replacing them, starting with networking, IAM, and data stores. Whatever is still clicked into the console is recorded as a known gap, not quietly ignored.

Pipeline hardening. Reproducible builds, pinned dependencies, no secrets in build logs, a staging deploy before production, and a rollback someone has actually executed. Rollback is the part most often skipped, and a pipeline without a tested one is a one-way door.

Environment parity. Not identical environments, which is expensive and rarely worth it. Structural parity: same deployment mechanism, same configuration shape, same runtime versions, differing only in size and data. The point is that a change tested in staging tells you something true about production.

This is where deeper problems surface. If the application keeps state on local disk, cannot run two copies at once, or opens a database connection per request with no pooling, infrastructure work does not fix it. A good partner says so and scopes it as application work instead of quietly building around it.

Done looks like: critical infrastructure reproducible from code, a tested rollback, staging that predicts production, a written list of what is still manual.

Weeks 9 to 12: handover, documentation, and proof

An engagement that leaves your team unable to operate the system alone has failed, whatever the dashboards say.

Runbooks for the incidents that actually occur. Not a generic wiki. One page per realistic failure, written so somebody woken at 3am can follow it:

title: API latency above SLO
severity: sev2
owner: platform
last_reviewed: 2026-08-12
last_tested: 2026-08-12

symptoms:
  - p95 latency above 800ms for 5 minutes
  - checkout error rate rising

first_checks:
  - dashboard: grafana/api-overview (RPS, p95, error rate)
  - recent deploys: last 60 minutes, api and worker
  - database: active connections vs max_connections

likely_causes:
  - id: bad-deploy
    check: a deploy landed within 30 minutes of the alert
    action: roll back to previous release, then investigate
  - id: connection-pool-exhaustion
    check: active connections at or near max_connections
    action: restart the worker pool, then raise limits per runbook db-pool
  - id: upstream-provider
    check: payment provider status page
    action: enable degraded checkout mode, notify support

escalate_if:
  - no improvement after 15 minutes
  - customer data may be affected
escalate_to: engineering lead, then CTO

after:
  - incident note within 24 hours
  - add the check that would have caught this earlier

The last_tested field is the one that matters. A runbook nobody has walked through is fiction.

On-call readiness. A rotation with named people, an escalation path, agreed severity levels, and a short post-incident habit. It does not need to be sophisticated, it needs to exist.

A real handover. Your engineers perform a deploy, a rollback, and a restore with the partner watching, not driving. If they cannot, the work is not finished.

Done looks like: tested runbooks, a working rotation, your team running the critical operations unaided.

The 90 day timeline at a glance

Phase	Focus	Client involvement	What "done" looks like
Week 1	Access, discovery, read-only audit, risk triage	Decision maker named, read-only access issued, an hour of product context	Inventory, ranked risk list, agreed top three fixes
Weeks 2 to 4	Backups verified, alerting fixed, access cleanup, obvious cost waste	Approvals within a day or two, write access to non-production	A timed restore observed, trusted alerts, no orphaned credentials
Weeks 5 to 8	IaC coverage, pipeline hardening, environment parity	An hour a week, code review, one agreed freeze window	Critical infrastructure in code, tested rollback, staging that predicts production
Weeks 9 to 12	Runbooks, on-call, documentation, measurement	Engineers available for handover exercises	Your team deploys, rolls back, and restores unaided

How to tell whether it is working

Judge it on observable things, not on how busy the reports look.

Deploy frequency and lead time. How often you ship, and how long a merged change waits before reaching users. Both come from git and pipeline history, so they are hard to spin. Improvement usually appears in weeks five to eight, not before.

Time to restore. How long from "something is broken" to "it works again", from real incidents where possible, drills where not.

Repeat incidents. The same failure twice means the first fix addressed a symptom. A falling repeat rate beats a falling incident count, which is easy to influence by counting fewer things.

Cost trend, with context. Absolute spend is noisy while you are growing. Look at cost per unit of what you sell, and whether spend moves with usage.

Bus factor. How many people can deploy, restore the database, and rotate a credential. If that is still one at day ninety, the engagement missed its main job.

Baseline all of these in week one. Without a baseline, every later conversation about progress is opinion.

When this does not work

Some engagements fail, and the reasons repeat.

Access never fully arrives. Read-only in week two, write access in week six, production access never. Everything stays advisory. If your process cannot grant a scoped external role, resolve that before signing, or accept that you are buying consulting rather than execution.

No decision maker. Recommendations pile up awaiting approval from someone permanently in another meeting. Ninety days pass, the audit was accurate, almost nothing shipped.

Ownership is unclear internally. Two teams each believe the other owns deployment. A partner cannot arbitrate that, only you can.

The real problem is the application. If the system cannot scale horizontally, or one database is doing work that needs a queue, infrastructure work reduces symptoms without removing the cause. An honest partner says this in week two. Rearchitecting a product is a different engagement with a different timeline.

Expectations were never written down. "Improve our DevOps" is not a scope. Three named outcomes, each with a definition of done, is. Vague briefs produce disappointing quarters even when the work was good.

Change is not permitted. A team that will not pause deploys, accept a pipeline change, or adopt new tooling has bought an auditor. That is a legitimate purchase, but it should be a conscious one.

The short version

A partner controls perhaps two thirds of the first ninety days. The rest is your access, your decisions, and an hour a week.

If you are evaluating a partner now, the useful question is not what they will build. It is what they expect from you in week one, and what they do when the answer is not more infrastructure. If you want to talk through what your own first ninety days would realistically look like, we are happy to map it out before anything is signed.

DevOps for Series A Startups: What to Build First

InstaDevOps — Mon, 24 Aug 2026 17:11:51 +0000

DevOps for Series A Startups: What to Build First

A Series A changes the shape of your engineering problems before it changes the size of your team. The product still works. The same three people still know how everything fits together. But now there is a board deck with a hiring plan in it, a couple of enterprise deals that come with security questionnaires, and a runway long enough that "we will fix it later" has quietly become a decision rather than an accident.

The pressure to "do DevOps properly" usually arrives as a shopping list: Kubernetes, a service mesh, multi-region, an internal developer platform, a platform team. Almost none of that is the right first move. What follows is what actually matters between roughly 8 and 30 engineers, what to postpone, and how to think about hiring versus outsourcing versus doing nothing yet.

What actually breaks at this stage

The failures are boring and predictable, and they are almost never about scale.

Deployment becomes contested. At five engineers, one person deploys when the tests look fine. At fifteen, two teams want to ship on the same afternoon, someone reverts someone else's migration, and the fix is a Slack thread instead of a process.

One person is the runbook. There is a founding engineer who knows why the NAT gateway is configured that way, which environment variable the worker actually reads, and how to restart the thing that gets stuck on Sundays. That person is now a manager, or interviewing candidates, or on a plane.

Environments drift. Staging was created by hand in the console eighteen months ago. Production was created by hand too, differently. Nobody can say with confidence what the difference is, so staging stops being evidence of anything.

Credentials sprawl. A shared AWS IAM user with an access key pasted into three laptops and a CI provider. A database password in a pinned Slack message. Nobody has rotated anything, and nobody can tell you who has access to customer data. This is the item that blocks enterprise deals, not your uptime numbers.

Nobody has restored a backup. Automated snapshots are enabled, which is not the same as knowing you can bring the database back. Untested backups are a belief, not a control.

Alerting is either silent or useless. Either you find out about outages from customers, or you have 40 alerts a day and everyone has muted the channel.

Each of those is fixable in days or weeks. None of them requires Kubernetes.

The minimum viable platform

Six things. In this order.

1. One repeatable path to production

Every deploy goes through the same pipeline, triggered by a merge to the main branch, with no human running commands on their laptop. The pipeline should use short-lived cloud credentials, not a stored access key. On AWS and GitHub, that means OIDC.

name: deploy
on:
  push:
    branches: [main]

permissions:
  id-token: write   # required for OIDC
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/github-deploy
          aws-region: eu-west-1

      - id: ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build, push, deploy
        env:
          REGISTRY: ${{ steps.ecr.outputs.registry }}
          TAG: ${{ github.sha }}
        run: |
          set -euo pipefail
          IMAGE="$REGISTRY/api:$TAG"
          docker build -t "$IMAGE" .
          docker push "$IMAGE"

          NEW_TD=$(aws ecs register-task-definition \
            --cli-input-json "$(jq --arg img "$IMAGE" \
              '.containerDefinitions[0].image = $img' task-def.json)" \
            --query 'taskDefinition.taskDefinitionArn' --output text)

          aws ecs update-service --cluster prod --service api \
            --task-definition "$NEW_TD"
          aws ecs wait services-stable --cluster prod --services api

Two details matter more than the platform choice. The image tag is the commit SHA, so any running container maps back to exact source. And the job waits for the service to stabilise, so a red pipeline means a failed deploy rather than a successful upload.

2. Infrastructure as code, starting with the parts that hurt

You do not need to import every hand-made resource on day one. Codify the things that are painful to recreate and dangerous to change: networking, IAM, databases, DNS, and the CI role itself. Leave the S3 bucket someone made in 2024 for later.

terraform {
  backend "s3" {
    bucket       = "acme-tfstate"
    key          = "prod/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true   # S3-native locking, no DynamoDB table needed
  }
}

resource "aws_iam_openid_connect_provider" "github" {
  url            = "https://token.actions.githubusercontent.com"
  client_id_list = ["sts.amazonaws.com"]
}

data "aws_iam_policy_document" "github_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    # Scope to one repo and one branch. Without this condition any
    # GitHub repository in the world can assume the role.
    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:acme/api:ref:refs/heads/main"]
    }
  }
}

resource "aws_iam_role" "github_deploy" {
  name               = "github-deploy"
  assume_role_policy = data.aws_iam_policy_document.github_assume.json
}

The sub condition is the one people get wrong. A trust policy that only checks the audience is effectively public.

3. Observability that answers three questions

Not dashboards. Three questions: is it broken, what changed, and where is the time going. In practice that is structured logs with a request ID, error tracking wired to a real channel, and latency and error rate per endpoint. A hosted tool is the right answer here. Running your own metrics and log storage at this size costs more engineering time than it saves in licence fees, and it fails at the worst moment because nobody owns it.

4. On-call that a person can survive

Two or three people in rotation, a written escalation path, and a rule that every page must be actionable. If an alert fires and the response is "yeah, that happens", either fix the thing or delete the alert. A rotation of one is not a rotation, it is a single point of failure with a phone.

5. Backups you have actually restored

Snapshots enabled is step one. Step two is a restore rehearsal you run on a schedule, ideally in CI, so it cannot rot.

set -euo pipefail

SNAPSHOT=$(aws rds describe-db-snapshots \
  --db-instance-identifier prod-postgres \
  --snapshot-type automated \
  --query 'reverse(sort_by(DBSnapshots, &SnapshotCreateTime))[0].DBSnapshotIdentifier' \
  --output text)

aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier restore-test \
  --db-snapshot-identifier "$SNAPSHOT" \
  --db-instance-class db.t4g.medium \
  --no-publicly-accessible

aws rds wait db-instance-available --db-instance-identifier restore-test

# Prove the data is real, not just that the instance booted.
psql "$RESTORE_URL" -c "select count(*), max(created_at) from orders;"

aws rds delete-db-instance --db-instance-identifier restore-test \
  --skip-final-snapshot

Write down how long that took. That number is your recovery time objective, and it is the honest answer when a customer asks.

6. Access control you can explain

SSO for the cloud console, individual identities instead of shared logins, a secrets manager instead of environment variables copied between people, and no long-lived access keys in CI. This is unglamorous and it is the work that turns a security questionnaire from a two-week scramble into a form-filling exercise.

What is premature right now

Thing	Why teams want it	Why it can wait
Kubernetes	It is what "real" infrastructure looks like	Unless you already run many services or need its scheduling, ECS Fargate, Cloud Run, or a managed PaaS gets you there with a fraction of the operational surface
Service mesh	Observability and mTLS between services	You probably have three services. A load balancer and good logging cover it
Multi-region active-active	An enterprise prospect asked	It multiplies data consistency problems and cost. A tested restore and a documented recovery objective answer the actual question
Internal developer platform	Deploys feel manual	A platform is an abstraction over repeated pain. Have the repetition first
Dedicated platform team	The org chart says so	Pulling two engineers off product to run a platform for twelve people is usually net negative until roughly 25 to 40 engineers
Splitting the monolith	It feels crowded	Service boundaries chosen before you understand the domain become distributed versions of the same coupling, with network calls added

The pattern: these are all solutions to coordination problems that appear when many teams share infrastructure. At Series A you usually have one or two teams. Buying the solution before you have the problem means paying the operating cost without the benefit.

A realistic order of operations

Roughly a quarter of work, mostly sequential because each step makes the next one cheaper.

Weeks 1 to 2. Access control and secrets. Kill shared credentials and long-lived keys. This is also the highest-value item for sales.
Weeks 2 to 4. CI/CD to production for the main service. One path, commit-tagged images, automatic rollback or at least a one-command revert.
Weeks 4 to 7. Infrastructure as code for networking, IAM, data stores, DNS. Rebuild staging from that code to prove it works.
Weeks 6 to 9. Observability and alerting. Error tracking, structured logs, a small set of alerts tied to customer-visible symptoms.
Weeks 9 to 11. Backups, restore rehearsal, written recovery objectives.
Weeks 11 to 13. On-call rotation, runbooks for the five things that actually break, a short incident review habit.

Cost control belongs somewhere in there too, but as a habit rather than a project: tagging, a budget alarm, and a monthly look at the top five line items.

Hire, outsource, or defer

There is no universally correct answer, and the honest framing is about what you are buying.

Option	Fits when	Real cost	Where it goes wrong
Hire a platform or DevOps engineer	Infrastructure work is continuous, not a project. You have a technical hiring loop and someone senior to manage them	Salary plus recruiting time, typically 2 to 4 months from opening the role to productive output	You hire one person, they become the single point of failure you were trying to remove, and they get bored if the work is 80% maintenance
Outsource to a service or fractional team	You have a known gap to close in a defined window, and product engineers are the constraint	Monthly fee, plus your own time on context transfer	Nobody on your side owns the outcome. Without a named internal counterpart, you get artefacts you cannot maintain
Defer	Deploys are not blocked, no customer is asking for security evidence, fewer than about eight engineers	Compounding interest. The cleanup gets more expensive as more is built on the shortcut	It stops being a choice. You notice when an outage or a deal forces the work at the worst possible time

A pattern that works well: outsource the build, own the operation. Bring in help to set up the pipeline, the Terraform, the alerting and the on-call structure, then have your own engineers run it day to day with the option to call for help on the hard parts. That avoids the two common failure modes, which are hiring a specialist to do six weeks of setup and then two years of waiting, and handing your infrastructure to an outside party with no internal understanding of it.

When not to buy this from us

Straightforwardly: if you have fewer than about eight engineers and your app is a single service on a managed platform, you probably do not need us yet. Spend two engineer-weeks on CI/CD, a secrets manager, and a restore test, and get back to product.

If you already have a strong infrastructure-minded engineer with time to spend, you do not need us either. Give them the list above and a quarter.

And if what you actually want is someone to hand a problem to permanently, with no internal owner, that arrangement does not work well with anyone, us included. External help is good at building and unblocking. It is a poor substitute for someone inside the company who cares whether the system stays healthy.

If none of those apply, and you are staring at a quarter of infrastructure work that would come straight out of your product roadmap, that is the case where bringing in a fractional DevOps team is the cheaper trade. We are happy to look at what you have and tell you which parts of the list you can skip, including the parts we would not charge you for.

What a DevOps Audit Actually Covers (and What It Finds)

InstaDevOps — Mon, 24 Aug 2026 15:53:36 +0000

What a DevOps Audit Actually Covers (and What It Finds)

An infrastructure audit is an awkward purchase. The deliverable is a document, the value is judgement rather than hours, and you cannot tell a good one from a bad one until it lands on your desk. Meanwhile your team already has a list of things they know are wrong, so the obvious question is whether you are paying someone to write that list down again.

Here is what a serious audit inspects, how it is run, what it costs you in your own team's attention, and when to skip it.

What "audit" should mean here

An audit is a bounded, read-only assessment of your infrastructure, delivery process, and operational practice, ending in a prioritised set of findings with severity, effort, and business consequence attached to each one.

Three words there carry weight. Bounded: a defined scope and end date, usually one to three weeks, not an open engagement. Read-only: the auditor changes nothing, so the assessment cannot itself cause an incident. Prioritised: the output ranks findings, which is what separates an audit from a scanner report. If a proposal lacks all three, you are buying something else.

The domains a real audit covers

Domain	Evidence examined
Cost	Spend by service and tag, Savings Plan and reserved instance coverage, unattached volumes, idle load balancers, cross-AZ and egress transfer, non-production running outside working hours
Security posture	Ingress rules open to the internet, encryption at rest and in transit, where secrets live and who can read them, image and dependency scan results, patch levels, findings sitting unread in the provider's own security services
Reliability and single points of failure	Availability zone spread, replica counts, database failover config, health check and timeout settings, shared dependencies every service touches, six months of incident tickets
CI/CD maturity	Pipeline definitions, time from merge to production, pipeline failure and rerun rate, approval gates, whether a rollback path exists and has been used, build reproducibility
IaC coverage and drift	Share of live resources actually managed by Terraform or CloudFormation, refresh-only plan results, console changes visible in audit logs, module and provider version pinning
Observability	Metric, log, and trace coverage per service, retention settings and their cost, the alert inventory, ratio of alerts fired to incidents declared, which dashboards were opened during the last outage
Access control	IAM policies with wildcard actions or resources, long-lived static keys, separation of human and machine identities, break-glass procedure, evidence that leavers were actually removed
Backup and DR	Schedules and retention, date of the last successful restore test, stated RTO and RPO versus what the setup can achieve, cross-region and cross-account copies
Documentation and bus factor	Runbook coverage for the top ten alerts, realistic onboarding time for a new engineer, count of systems only one person can deploy or debug

Cost and security are the domains buyers ask for. They are also the ones scanners handle best, which makes them the least valuable part of a human audit. An experienced reviewer earns the fee on reliability, CI/CD, drift, and bus factor, because all four require reading your actual code and talking to your actual engineers.

Reliability work is a hunt for single points of failure nobody has written down: the one NAT gateway, the single-writer database with a replica nobody has ever promoted, the shared Redis that four services treat as optional and one treats as mandatory, the certificate renewed by hand each year by someone who left in March.

CI/CD maturity is measured, not described. How long from merge to production, honestly, including waiting for a human to click approve? What fraction of pipeline runs fail for reasons unrelated to the change? Can you roll back, and when did you last do it? A team with a fast, boring pipeline can fix almost anything else. A team without one keeps regenerating the same problems.

Drift is the gap between your Terraform and your reality. Most teams over-report their infrastructure-as-code coverage, because they count the resources in state and not the resources in the account.

How a good audit is actually run

Four inputs, in roughly this order.

Read-only access. A dedicated IAM role with a managed read-only policy, ideally with an external ID and an expiry date, plus read access to repositories and observability tooling. If a provider asks for admin, ask why. The usual reason is laziness on their side.

Interviews. Forty-five minutes each with the people who carry the pager, plus the engineering lead and, if cost is in scope, whoever owns the bill. Engineers tell you in ten minutes what a scanner cannot find in a week: which deploy everyone dreads, which alert gets muted, which service nobody wants to touch.

Tooling scans. Provider-native tools first, since you already pay for them, then whatever the auditor brings. Fast and largely automated, which is why it should not be the bulk of the fee.

Manual review. Reading Terraform modules, pipeline definitions, Kubernetes manifests, and incident write-ups. This is where the fee goes, and it cannot be shortened without hollowing out the result.

A clean sequence: access and scans in the first two or three days, interviews in the first week, manual review through the second, a draft walkthrough, then the written deliverable. The walkthrough matters. Findings written without a chance to say "we know, that is deliberate, here is why" produce documents that get ignored.

What it costs you in your team's time

The invoice is not the whole price. Budget six to ten hours of engineering time: an hour to provision access, three to five hours of interviews across two or three people, an hour or two of follow-up questions, and an hour for the walkthrough. A provider who needs zero time from your team is producing a scanner dump.

On the invoice side, the drivers are scope (how many domains), environment and account count, whether Kubernetes is involved, how much of the infrastructure is code you can read versus clicked-together resources, and depth (a survey versus a review that reads every module). Review work scales with surface area rather than headcount, so a multi-account setup with several clusters costs considerably more than one account with a handful of services.

Checks you can run before you pay anyone

Run these first. If they come back clean, an audit will find less than you hope. If they come back ugly, you have a scope.

# --- AWS: cost and encryption basics (all read-only) ---

# Unencrypted EBS volumes
aws ec2 describe-volumes --filters Name=encrypted,Values=false \
  --query 'Volumes[].{ID:VolumeId,Size:Size,AZ:AvailabilityZone,State:State}' \
  --output table

# Volumes detached from everything and still billed every month
aws ec2 describe-volumes --filters Name=status,Values=available \
  --query 'Volumes[].{ID:VolumeId,Size:Size,Created:CreateTime}' --output table

# Elastic IPs not associated with anything (charged hourly while idle)
aws ec2 describe-addresses \
  --query 'Addresses[?AssociationId==`null`].{IP:PublicIp,Alloc:AllocationId}' \
  --output table

# Security group rules open to the whole internet
aws ec2 describe-security-groups --query \
  'SecurityGroups[?IpPermissions[?IpRanges[?CidrIp==`0.0.0.0/0`]]].{ID:GroupId,Name:GroupName,VPC:VpcId}' \
  --output table

# Buckets with no public access block configured at all
for b in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do
  aws s3api get-public-access-block --bucket "$b" >/dev/null 2>&1 \
    || echo "NO PUBLIC ACCESS BLOCK: $b"
done

# Credential hygiene: key age, MFA, unused passwords, in one report
aws iam generate-credential-report >/dev/null
aws iam get-credential-report --query Content --output text \
  | base64 --decode | column -s, -t     # macOS: use `base64 -D`

The Kubernetes checks cover the reliability side. Missing resource limits cause noisy-neighbour incidents, and single-replica deployments survive precisely until the next node rotation.

# --- Kubernetes: reliability and hygiene (read-only) ---

# Containers with no memory limit set
kubectl get pods --all-namespaces -o json | jq -r '
  .items[] as $p | $p.spec.containers[]
  | select(.resources.limits.memory == null)
  | "\($p.metadata.namespace)/\($p.metadata.name)  container=\(.name)"'

# Deployments running exactly one replica
kubectl get deploy --all-namespaces --no-headers \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,REPLICAS:.spec.replicas' \
  | awk '$3 == 1 {print $1"/"$2}'

# Namespaces with workloads but no PodDisruptionBudget
comm -23 \
  <(kubectl get deploy -A --no-headers | awk '{print $1}' | sort -u) \
  <(kubectl get pdb -A --no-headers 2>/dev/null | awk '{print $1}' | sort -u)

# Images pinned to a moving tag
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{range .spec.containers[*]}{.image}{" "}{end}{"\n"}{end}' \
  | grep ':latest'

# Pods not required to run as a non-root user
kubectl get pods -A -o json | jq -r '
  .items[] | select((.spec.securityContext.runAsNonRoot // false) != true)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Then the drift check, which is usually the most uncomfortable of the three. A refresh-only plan compares state to reality without proposing or applying any change.

# Detect drift without changing anything.
# Exit code 0 = state matches reality, 2 = drift found, 1 = error.
terraform plan -refresh-only -detailed-exitcode

# How much of the account is actually managed here?
terraform state list | grep -c '^aws_instance\.'
aws ec2 describe-instances \
  --filters Name=instance-state-name,Values=running \
  --query 'length(Reservations[].Instances[])'

If the second number is much larger than the first, your infrastructure-as-code coverage is a story rather than a fact. Repeat for security groups, RDS instances, and IAM roles.

What the deliverable should look like

A finding is useful only when it carries four things: what is wrong, what happens if it stays wrong, how much work the fix is, and what to do first. A usable format is one page per finding, with a severity, an effort estimate in engineer-days, the business consequence in plain language, and the specific resource or file it applies to.

Ahead of the findings, expect a short executive summary an engineering leader can forward without editing, and a sequenced remediation plan: this week, this quarter, this year, with dependencies noted (you cannot enforce pipeline policy before the pipeline exists).

Do not accept a PDF export of a scanner, five hundred unranked medium-severity rows, or recommendations that all point toward the auditor's own retainer.

What audits commonly find

These are patterns, not measurements, and every environment differs. Commonly:

Infrastructure-as-code coverage below what the team believed, with the gap concentrated in resources created during incidents.
Non-production environments sized like production and running around the clock.
Backups configured and never restored, so the recovery time objective is a guess.
One person who is the only one who can deploy a particular service.
An alert set nobody trusts, the noisiest ones routed to a muted channel.
Long-lived access keys belonging to former employees or to a service replaced two years ago.
A pipeline that deploys but cannot roll back, so every bad release becomes a forward-fix under pressure.
Log and metric retention left at the default and never revisited, quietly becoming a top line on the bill.

None of these are exotic. They accumulate because they are individually small and collectively nobody's job.

Telling a genuine audit from a sales exercise

Signals it is real: read-only access with a defined expiry, a written scope naming the domains covered, interviews with your engineers, a draft walkthrough before the final document, findings that include things to stop doing as well as start, and several recommendations you can implement yourself with no outside help.

Signals it is a sales exercise: a fixed-length report promised before anyone has looked at your environment, no engineer interviews, generic best-practice findings that never name your own resources, severity ratings that are all high, and a remediation plan whose every step requires the auditor.

The strongest test: ask what the audit would say if your environment turned out to be in good shape. A provider who cannot describe that outcome is not selling an assessment.

When you do not need one

If your team can already name the top five problems, agrees on their order, and knows how to fix them, an audit will mostly confirm the list. That is a capacity problem, and the money is better spent on the fixing than on the finding. Buying an audit here is often really about getting an outside voice to repeat what your engineers have been saying. That is sometimes a legitimate need, but call it what it is and buy less of it.

Skip it too if you are pre-launch with a couple of services and one environment, if you are mid-migration and the target architecture is already decided, or if you audited within the last year and have shipped nothing structural since.

The case is strongest when you inherited the environment, when the people who built it have gone, when the spend moved and nobody can explain why, when a compliance review or due diligence is coming, or when incidents are rising and no one can say which risk to spend the next quarter on.

If you want an outside read on any of that, we run scoped, read-only infrastructure reviews and hand back prioritised findings you can act on with or without us. A short call is enough to work out whether an audit is the right purchase for where you are, or whether you should skip straight to the fixing.

Terraform Cloud Workspaces: Team Collaboration & Remote State

InstaDevOps — Thu, 20 Aug 2026 13:47:43 +0000

Terraform Cloud and Workspaces: Remote Execution, Policy Enforcement, and Team Workflows

Terraform Cloud transforms Terraform from a CLI tool that engineers run on their laptops into a collaborative platform with remote state, remote execution, policy enforcement, and audit trails. Running terraform apply locally works for a solo engineer, but breaks down with teams - who ran the last apply, what version of the code was used, were the right variables set, did anyone review the plan? Terraform Cloud solves these problems with a centralized execution environment.

Workspaces in Terraform Cloud map to environments or components. A typical structure uses separate workspaces for each environment (app-production, app-staging, app-development) sharing the same Terraform code but with different variable values. VCS-driven workspaces automatically trigger plans when code is pushed to a connected Git branch - push to main triggers the production workspace, push to develop triggers staging. Run triggers chain workspaces together: when the networking workspace applies successfully, it triggers the compute workspace that depends on it.

Policy enforcement with Sentinel (or OPA for the free tier) is the governance layer that distinguishes Terraform Cloud from running the CLI with a remote backend. Policies check plans before apply - reject any plan that creates public S3 buckets, require specific tags on all resources, limit instance sizes in non-production environments, and mandate encryption on all storage. The approval workflow adds human review: plans require approval from designated team members before apply, with the full plan output visible in the UI. For teams, this creates a self-service infrastructure workflow where developers can deploy within guardrails without needing infrastructure team involvement for every change.

Need help setting up Terraform Cloud? InstaDevOps implements Terraform Cloud workflows with proper governance for growing engineering teams. Book a free consultation.

Python for DevOps: Automation Scripts, APIs & CLI Tools

InstaDevOps — Wed, 19 Aug 2026 13:47:38 +0000

Python for DevOps: Automating AWS, Kubernetes, and CI/CD with Boto3 and Fabric

Python has become the default scripting language for DevOps automation. Bash scripts work for simple tasks, but once you need error handling, API integrations, data processing, or testable code, Python's ecosystem is unmatched. Boto3 provides complete AWS API coverage, the official Kubernetes Python client manages clusters programmatically, and libraries like Fabric, Paramiko, and Invoke handle SSH and local command execution.

Boto3 is the most commonly used DevOps Python library. Automating AWS operations - cleaning up unattached EBS volumes, rotating IAM access keys older than 90 days, generating cost reports across accounts, or managing EC2 instances based on tags - becomes straightforward with Boto3's resource and client interfaces. Use sessions with assumed roles for cross-account automation, paginators for listing large result sets, and waiters for operations that take time (instance launch, stack creation). For Kubernetes, the official Python client mirrors kubectl functionality: list pods, scale deployments, apply manifests, and watch for resource changes programmatically.

The key to maintainable DevOps Python is treating scripts like production code. Use Click or Typer for CLI argument parsing instead of sys.argv. Structure code in functions and classes, not monolithic scripts. Write unit tests with pytest and mock AWS calls with moto. Use virtual environments and pin dependencies. For recurring tasks, package your scripts as Python packages with proper entry points rather than running raw .py files. Fabric excels for SSH-based automation when you need to run commands on remote servers - deploying to legacy infrastructure, collecting diagnostics, or managing services that are not yet containerized.

Need DevOps automation expertise? InstaDevOps builds custom automation tooling for AWS, Kubernetes, and CI/CD workflows. Book a free consultation.

Feature Flags & Progressive Delivery: Ship Faster, Safer

InstaDevOps — Tue, 18 Aug 2026 13:47:34 +0000

Feature Flags and Progressive Delivery: LaunchDarkly, Unleash, and Flagsmith

Feature flags decouple deployment from release. You deploy code to production with new features hidden behind flags, then gradually enable them for specific users, percentages of traffic, or entire cohorts. This eliminates the high-stakes big-bang release - if a feature causes problems, you toggle it off instantly without rolling back a deployment. Progressive delivery extends this pattern with automated, metrics-driven rollout: start at 1% of traffic, monitor error rates and latency, automatically increase to 10%, then 50%, then 100% if all metrics stay healthy.

The three major platforms serve different segments. LaunchDarkly is the enterprise standard with real-time flag evaluation, sophisticated targeting rules, and audit logging - but it is expensive at scale. Unleash is open-source and self-hosted, offering feature toggles, gradual rollouts, and A/B testing with full control over your data. Flagsmith provides a middle ground with both cloud-hosted and self-hosted options, a clean API, and built-in analytics. For most startups, Unleash gives you 90% of the functionality at a fraction of the cost.

The implementation pattern matters as much as the tool. Wrap flag evaluations in a thin abstraction layer so you can swap providers without changing application code. Use flag naming conventions that indicate lifecycle (e.g., release-new-checkout, experiment-pricing-page, ops-circuit-breaker-payments). Set expiration dates on temporary release flags and enforce cleanup - stale flags accumulate quickly and become technical debt. For progressive delivery, integrate your flag platform with your observability stack: the flag system reads metrics from Prometheus or Datadog and automatically advances or rolls back the rollout based on error budgets.

Need help with deployment strategies? InstaDevOps implements progressive delivery pipelines with feature flags and automated rollouts. Book a free consultation.

Managed DevOps Services: What's Actually Included (and What to Watch For)

InstaDevOps — Mon, 17 Aug 2026 13:47:30 +0000

Managed DevOps Services: What's Actually Included (and What to Watch For)

'Managed DevOps services' sounds precise but means wildly different things depending on who is selling it. One provider means a full production operations partnership; another means they will set up a pipeline once and disappear. Before you commit budget, know what a genuinely complete managed service covers and where the gaps hide in the fine print.

What it should include. A complete engagement covers the full lifecycle: cloud infrastructure defined in Infrastructure as Code (usually Terraform), CI/CD pipelines with safe deployment and rollbacks, monitoring and alerting (typically Prometheus and Grafana), ongoing cloud cost optimization, security hardening with least-privilege IAM and secrets management, and incident support. A narrow provider who only does pipelines will send you shopping again the moment you need observability.

What to watch for in the fine print. Incident support can mean 24/7 paged response, or 'we will look at it during business hours.' These are radically different products, so get specific about hours and response times. Watch for vendor lock-in where configuration lives on the provider's systems instead of your cloud account. Watch for undefined scope, where 'includes CI/CD support' has no volume limit and your work quietly gets deprioritized. And watch for hourly billing that turns a predictable managed service into an unpredictable invoice.

Define capacity clearly. Models that state throughput are easier to reason about. For example, our plans are structured around active requests: the Startup tier at $2,999/month handles one request at a time and the Business tier at $4,999/month handles two, both with unlimited queued requests, no hourly billing, and roughly 48-hour turnaround. That tells you exactly what you are buying.

Confirm who holds the pager. A managed service is only as good as the people behind it. Make sure senior engineers, not rotating junior contractors, are responsible for your environment, with real AWS depth alongside Kubernetes, CI/CD, Terraform, monitoring, and security.

A checklist before you sign: Is the full lifecycle covered? What does incident support include and during what hours? Does everything live in your account and repositories? Is pricing flat? Is capacity defined? Who are the actual engineers? How do you leave cleanly? Answer those and you will know whether a managed service is a real operations partnership or a thin wrapper around a one-time setup.

At InstaDevOps we deliver managed DevOps for startups with senior, AWS-focused engineers on flat monthly plans from $2,999/month, with roughly 48-hour turnaround and everything built in your own cloud account. Book a free 15-minute consultation.

DevOps Engineer Salary vs Retainer: The Real 2026 Cost Comparison

InstaDevOps — Sun, 16 Aug 2026 13:47:27 +0000

DevOps Engineer Salary vs Retainer: The Real 2026 Cost Comparison

When founders price out DevOps, they anchor on one number: the salary. But salary is the smallest part of what a full-time hire actually costs, and it tells you almost nothing about whether hiring is the right move at your stage. Here is an honest, numbers-first comparison for 2026.

The salary is just the sticker price. In 2026, a mid-to-senior DevOps engineer in the US commands a base salary of roughly $140,000 to $185,000. That is the number founders quote to each other, and it is also the number that hides the most.

The fully loaded cost is far higher. Add payroll taxes and benefits (25 to 40 percent on top of base), recruiting fees (15 to 25 percent of first-year salary), equipment, and management overhead, and a $160,000 base becomes roughly $210,000 to $260,000 in year one. Then add the hidden costs: two to four months to hire, weeks of ramp time at full salary, single-point-of-failure risk when your one engineer takes vacation or quits, and underutilization during the quiet stretches between infrastructure projects.

What a retainer actually costs. A DevOps retainer flips the model. For reference, our plans are $2,999/month for the Startup tier (one active request at a time) and $4,999/month for the Business tier (two active requests), both with unlimited queued requests, no hourly billing, and pause or resume. That is a fully loaded annual cost of roughly $36,000 to $60,000 with zero recruiting fees, zero benefits overhead, and no ramp time.

When hiring full-time still wins. A full-time engineer makes sense when infrastructure is your core product, when you have enough continuous deep work to keep a specialist busy every single week, or when compliance requires employees. For most early and growth-stage teams, the math favors flexible external capacity until the volume genuinely justifies a dedicated headcount.

Run the numbers honestly. Take the base salary, add 30 to 40 percent for benefits and taxes, add amortized recruiting cost, and add the opportunity cost of months without a solution. Compare that to twelve months of a retainer at your expected workload, then ask whether you truly have 40-plus hours of DevOps work every week or whether it comes in bursts. That question decides it for most founders.

At InstaDevOps we give startups senior, AWS-focused DevOps on a flat monthly retainer from $2,999/month, with no recruiting fees and no ramp time. Book a free 15-minute consultation.

How to Choose a DevOps as a Service Provider: A 2026 Buyer's Checklist

InstaDevOps — Sat, 15 Aug 2026 13:47:23 +0000

How to Choose a DevOps as a Service Provider: A 2026 Buyer's Checklist

Hiring a full-time DevOps engineer takes months and costs a small fortune. DevOps as a Service has become the practical alternative for startups and scale-ups that need production-grade infrastructure without building an internal platform team. But the market is crowded and quality varies wildly, so this checklist walks through exactly what to evaluate before you sign.

Start with the problem, not the provider. Write down what you actually need done in the next 90 days: a CI/CD pipeline that keeps breaking, an AWS bill that has doubled, a Kubernetes cluster no one wants to touch, or a founding engineer who has quietly become the accidental infrastructure owner. Being specific helps you filter broad 'digital transformation' consultants from the providers who ship real infrastructure changes every week.

Check who actually does the work. Many agencies sell you a senior architect in the pitch, then hand execution to junior contractors. You want senior engineers doing the hands-on work, with real depth in your cloud provider. AWS experience matters most for the majority of startups, ideally backed by Kubernetes, CI/CD, Terraform, monitoring, and security skills on the same team.

Prefer flat pricing over hourly. Hourly billing rewards slow work. A flat monthly price aligns incentives and makes budgeting predictable. For reference, our own plans run $2,999/month for the Startup tier (one active request at a time) and $4,999/month for the Business tier (two active requests), both with unlimited queued requests, no hourly billing, and roughly 48-hour turnaround.

Insist on no lock-in. Everything a provider builds should live in your repositories and your cloud account, described in readable Infrastructure as Code and runbooks. If a provider keeps configuration on their own machines or refuses to hand over Terraform state, walk away. A confident provider makes leaving a non-event.

Red flags to watch for: vague deliverables, no named engineers, proprietary tooling lock-in, and pressure to sign a long contract before you have seen any output. Confidence looks like month-to-month; insecurity looks like a 12-month commitment up front.

At InstaDevOps we provide senior, AWS-focused DevOps as a Service on flat monthly plans from $2,999/month, with no hourly billing and pause or resume whenever your needs change. Book a free 15-minute consultation.

When to Hire Your First DevOps Engineer (and What to Do Before You Do)

InstaDevOps — Fri, 14 Aug 2026 13:47:19 +0000

Most teams hire their first DevOps engineer too early or too late

There is no magic headcount number that says it is time. Some 5-person teams genuinely need a dedicated infrastructure person; some 30-person teams get by fine with developers who share the load. The right question is not how big are we but how much operational work exists, who is doing it now, and what is it costing us. This guide walks through the signals that actually matter, what the role should own, what it costs, and the cheaper options worth trying first.

The signals that you genuinely need one

Watch for these patterns. One alone is rarely decisive; three or more together usually means the workload is real.

Deploys are scary. Releases happen rarely because they are manual, fragile, or only one person understands them.
Your best developers are drowning in infrastructure. Senior engineers spend hours per week on CI failures, cloud config, and firefighting instead of product work.
Incidents have no owner. When production breaks, it is a scramble because nobody clearly owns monitoring, alerting, or on-call.
Cloud spend is climbing and nobody can explain it. The AWS bill grows every month and no one has time to audit it.
Compliance is knocking. A customer wants SOC 2, or you need auditable access controls, and there is no system for it.
Onboarding a new environment takes days. Spinning up staging or a new region is a manual, error-prone ordeal.

If you nodded at most of these, the operational load has outgrown ad-hoc ownership. That does not automatically mean a full-time hire, but it does mean something has to change.

What the role should actually own

Before you write a job description, get clear on scope. A first DevOps engineer is a generalist who typically owns:

CI/CD pipelines: fast, reliable, automated build, test, and deploy
Infrastructure as code: reproducible environments in Terraform, Pulumi, or CloudFormation
Observability: metrics, logs, tracing, and alerting that catch problems before customers do
Cloud architecture and cost: right-sizing, networking, security groups, and spend control
Security baseline: secrets management, least-privilege access, patching
Incident response: runbooks, on-call structure, and post-incident reviews

That is a wide surface area. Be honest that one person cannot do all of it deeply at once. Rank these by pain and let the first hire focus there.

What it costs

A dedicated DevOps or SRE engineer is one of the more expensive engineering hires. In the US, total compensation for a mid-to-senior engineer commonly lands between 130,000 and 200,000 USD per year, plus benefits, equipment, and recruiting cost. Add the hidden costs:

Hiring lead time: senior DevOps roles often take 2 to 4 months to fill
Ramp time: even a strong hire needs weeks to learn your stack
The bus factor: with one person, vacations and departures are real risk

None of this means do not hire. It means the decision should be deliberate, because the fully loaded cost of a first DevOps engineer easily exceeds 180,000 USD per year.

Cheaper alternatives worth trying first

If the workload is real but not yet a full 40 hours a week, or you cannot afford the ramp time, consider these before committing to a permanent hire.

1. Fractional or part-time senior help

A senior engineer for part of their time can set up your pipelines, harden security, and get cloud costs under control without a six-figure commitment. This works especially well for the initial setup phase, where you need senior judgment more than 40 weekly hours. Our fractional DevOps engineer page covers how that arrangement works.

2. DevOps as a service on a retainer

If the need is ongoing operations rather than a one-time setup, a monthly retainer with an external team gives you continuous coverage at a predictable cost, often less than half a full-time salary. See our DevOps monthly retainer breakdown and the broader case in our alternative to hiring DevOps guide.

3. Upskill a developer plus better tooling

Sometimes the honest answer is that you do not need a specialist yet. A motivated developer plus managed platforms (managed Kubernetes, managed databases, a good CI provider, and a platform-as-a-service host) can carry a small team a surprisingly long way. Managed services trade money for reduced operational burden, which can defer the hire entirely.

When you should just hire in-house

Be clear about when the alternatives stop making sense. Hire full-time when:

DevOps work is clearly a full-time load and will only grow
Infrastructure is core to your product, not a supporting function
You need someone deeply embedded in daily engineering decisions
Compliance or customer requirements demand a dedicated internal owner
You are past the setup phase and into continuous scaling

At that point, a permanent hire gives you the deep context and availability that outside help cannot fully match. Many teams sequence it: use fractional or retainer help to get the foundation right, then hire in-house once the workload is steady and well understood. Our hire a DevOps engineer page covers what to look for when you reach that point.

A simple framework

Measure the load. For two weeks, track hours your team spends on infra, CI, incidents, and cloud. That number tells you if it is a full-time job.
Match the model to the load. Under 15 hours a week and mostly setup: fractional. Steady ongoing ops under a full role: retainer. A clear full-time load that is core to the product: hire.
Protect the knowledge. Whatever you choose, insist on infrastructure as code, documentation, and cloud accounts your company owns.

If you want help figuring out which bucket you are in, InstaDevOps offers senior DevOps on a monthly retainer as one option: Startup at 2,999 USD per month, Business at 4,999 USD per month, roughly 48-hour turnaround, pause anytime. It is one path, and we will tell you honestly if a full-time hire or a short fractional engagement fits you better. Book a free 15-minute call at calendly.com/instadevops/15min.