DEV Community: Kunwar Harshit

Your integration is running as an admin. Nobody remembers approving that.

Kunwar Harshit — Tue, 15 Sep 2026 10:35:32 +0000

It happens in about forty seconds, usually on a Thursday.

You are wiring a sync in a sandbox. The write fails on a permission you did not expect. You have three other things to do that day, so you grant the integration user a broader role to get past it, make a mental note to narrow it later, and ship.

Nobody narrows it later. Narrowing it has no deadline, no ticket, and no owner, and the integration works, which is the whole problem. Working software generates no pressure to change.

Two years on, that account can read every record in the system, write to objects the sync has never touched, and in a few cases delete. It has no password rotation, no expiry, and a name like api_user_2.

The part that is actually new

Over-privileged service accounts are an old problem, and the old advice has been available for decades. What changed is the volume and the attack surface.

CyberArk's 2025 Identity Security Landscape, a survey of 2,600 security decision-makers, put machine identities at 82 for every human identity, with 42% of them holding privileged or sensitive access. In the same survey, 88% of organisations still define "privileged user" as a human being. So the majority of privileged accounts in a typical company sit outside the definition that governs privileged accounts.

Then agents arrived, and they differ from a cron job in one way that matters here: their instructions come from content they read at runtime. A scheduled sync does the same thing every night. An agent summarising an inbox is executing partly on the contents of that inbox, and anyone who can get text in front of it gets a vote. That is not a hypothetical risk class, it is the first entry in the OWASP LLM Top 10.

An over-privileged cron job is a bug waiting for an edge case. An over-privileged agent is a bug waiting for someone to write it a message.

Four things that break, in order of how quietly they break

Blast radius. The sync was supposed to update one field on one object. With a broad role, a bad loop can touch anything the account can reach. The permission set is the only thing standing between a logic error and your whole database, and you removed it on a Thursday.

Attribution disappears. If three workflows share api_user_2, your audit log records that api_user_2 changed a close date. It cannot tell you which workflow, which run, or what triggered it. You are now debugging by correlating timestamps, which is the worst kind of Tuesday.

You cannot revoke one thing. Shared credentials mean revocation is all or nothing. Something misbehaves, you check what else uses that account, discover it is four things including one finance cares about, and you do not revoke it. You open a Slack thread instead.

Credentials outlive their purpose. The workflow gets deprecated. The key does not. It sits there, valid, attached to a system nobody runs, until an audit finds it or something worse does.

The fix is one identity per workflow

Not one per integration, and definitely not one per company. One per workflow, scoped to exactly the objects and fields that workflow touches.

  # Before: one account, everything, forever
- scopes: [ crm.objects.contacts.*, crm.objects.deals.*,
-            crm.objects.companies.*, tickets.*, files.*, settings.* ]
- owner: (unset)
- expires: never

  # After: one per workflow, scoped to what it actually does
+ workflow: post-call-crm-update
+ scopes:  [ crm.objects.deals.read,
+            crm.objects.deals.write:close_date,
+            crm.objects.deals.write:stage ]
+ owner:   platform-team
+ expires: 90d

The mechanics exist in every platform worth integrating with. Salesforce gives you permission sets and field-level security, so an integration user can be granted edit on two fields rather than an object. HubSpot private apps take granular scopes rather than a role. Most OAuth providers let you request narrow scopes and simply are not asked to.

Three rules make it stick:

Scope to fields, not objects, wherever the platform allows it. "Can edit Opportunity" and "can edit Opportunity.CloseDate" are different blast radii, and the second one is usually what you meant.

Give every identity a human owner and an expiry. An account nobody owns is an account nobody revokes. An expiry forces a five-minute review you would otherwise never schedule, and if renewing it is annoying, that is the system telling you something true.

Put the workflow name in the identity. post-call-crm-update in an audit log answers the question. api_user_2 starts an investigation.

The test

Pick your most recently shipped integration and answer this: can you turn it off, right now, without a meeting?

If revoking one workflow's access means checking what else breaks, you do not have scoped identities. You have a shared admin account with extra steps, and every agent you point at it inherits the whole thing.

SailPoint's 2025 survey found only 44% of organisations have any policy for securing the AI agents they already run. The gap is not usually knowledge. It is that scoping permissions produces nothing you can demo, and Thursday you had three other things to do.

The longer version of this argument, including how per-workflow identity makes writes attributable and reversible rather than just contained, is written up here: giving each agent its own scoped identity. The vocabulary for the account type itself is non-human identity, which is worth knowing because it is what the tooling and the compliance frameworks call it.

I build approval and audit layers over agents that write to production systems at Mindlyft.

What is the broadest permission you have found on an account nobody could explain? Mine was a reporting integration with delete.

Your integration returned 200. The record didn't change.

Kunwar Harshit — Tue, 15 Sep 2026 10:23:50 +0000

Every cross-system integration I have debugged in the last two years failed the same way, and it was never the way the logs suggested.

The logs said success. The API returned 200. The retry counter was zero. And the field in the target system still held the old value.

This is not an edge case. It is the default behaviour of every major CRM and ticketing API, and if your pipeline treats a 2xx as proof of a write, you have a silent data-loss bug you have not found yet.

A 2xx means "accepted", not "stored"

That distinction sounds pedantic until it costs you a quarter of pipeline data. Here are the five mechanisms I keep hitting, roughly in order of how often they bite.

1. Partial success inside a successful response. Bulk and composite endpoints are the worst offenders. Salesforce's composite API with allOrNone: false returns 200 OK with a body containing per-record failures. If you check the HTTP status and move on, you have just discarded the error.

# This is the bug.
r = requests.post(composite_url, json=payload, headers=h)
r.raise_for_status()          # 200. Looks fine.
log.info("synced %d records", len(payload["records"]))

# The failures were in here all along.
for result in r.json()["results"]:
    if not result["success"]:
        log.error("record %s: %s", result.get("id"), result["errors"])

2. Downstream automation overwrites you milliseconds later. Your write lands. Then a workflow rule, a Flow, or a HubSpot workflow fires on that same record change and sets the field back, or to something else entirely. Your write succeeded and was immediately undone. Nothing in your logs will show it, because from your side nothing went wrong.

3. Duplicate and merge rules move the record out from under you. You write to record A. A dedupe rule merges A into B. Your value is now either on a record you were not targeting or gone. The API still told you 200, because at the moment it answered, it was true.

4. Field-level permissions drop fields silently. The integration user lacks edit access on one field in a twelve-field update. Depending on the endpoint, you get a 200 and eleven fields written. The twelfth is simply absent from the result, not reported as an error.

5. Eventual consistency means read-your-write is not guaranteed. Read back too fast and you get the pre-write value, which sends you chasing a bug that does not exist. This one produces false alarms rather than silent failures, which is why it is the least dangerous and the most annoying.

The fix is boring: read it back

def write_and_verify(client, record_id, fields, *, attempts=3, delay=0.5):
    """Write, then confirm the target system actually holds the values."""
    client.update(record_id, fields)

    for attempt in range(attempts):
        time.sleep(delay * (2 ** attempt))        # back off for #5
        actual = client.get(record_id, fields.keys())
        drift = {k: (v, actual.get(k)) for k, v in fields.items()
                 if actual.get(k) != v}
        if not drift:
            return Verified(record_id, fields)

    # Do NOT retry the write. Something is actively rejecting or
    # overwriting it, and hammering the endpoint will not change that.
    raise VerificationFailed(record_id, drift)

Three things matter more than the code.

Verify the fields you care about, not all of them. A full record comparison will drift constantly on system-managed fields like LastModifiedDate and produce alerts nobody reads. Pick the fields whose wrongness would actually cost something.

Never auto-retry a failed verification. A failed verify means something is rejecting or overwriting your write. Retrying makes that happen again, faster. Escalate to a human with the diff: expected, actual, record, timestamp. A three-line diff is a five-minute fix. A retry loop is a Tuesday.

Log the diff, not the outcome. verification failed tells you nothing at 2am. close_date expected 2026-03-31, actual 2026-06-30, record 0061x…, 340ms after write tells you a Flow is overwriting close dates, which is a completely different problem than a permissions error.

Why the verify step is the one that gets skipped

Because it is the only stage with no visible output when it works. Trigger, join, transform and write all produce something you can point at in a demo. Verification produces silence, and silence does not demo well.

It is also the stage that separates an automation from a system. A zap moves a record. A system proves the record arrived and tells you when it did not. That gap is where most "our data is a mess" complaints actually originate, long after anyone remembers which integration caused it.

If you want the wider version of this argument, the discipline of building revenue workflows this way now has a name and a job market attached to it: GTM engineering. The write-verify pattern above is stage five of five in how those workflows get built, and it is the stage almost every tutorial leaves out.

I build these systems at Mindlyft. Half of what we ship is not the automation. It is the proof that the automation did what it said.

What is the worst silent-write failure you have found? I am collecting the mechanisms, and I suspect five is not the complete list.

The approval gate: a pattern for AI agents that write to production systems

Kunwar Harshit — Wed, 09 Sep 2026 20:34:38 +0000

Most AI agent demos end at the moment the agent decides something. The interesting engineering starts right after that, when the agent writes to a system other people depend on.

I build automation that writes to CRMs, ticketing systems and inboxes for revenue teams. Those are systems of record. A wrong write is not a bad answer you can regenerate. It is a support ticket, a corrupted forecast, or an email a customer has already read.

Here is the pattern that survived contact with production.

The failure mode

The naive design gives the agent credentials and lets it call the API directly.

This works most of the time. Most of the time is the problem. At 95 percent action accuracy, one in twenty writes is wrong, and those writes compound into a database humans use to make decisions. The cleanup cost per bad write is much higher than the time saved per good write, so the automation goes net negative while still looking successful in your metrics.

The second order failure is worse. Once users find two or three wrong records they stop trusting the whole dataset and go back to their spreadsheets. Now you have added a system nobody trusts.

The pattern: propose, diff, approve, execute, log

Split the decision from the side effect. The agent never touches the target API. It emits a proposed action, and a separate executor applies it only after approval.

{
  "action_id": "act_01H8Z...",
  "type": "crm.opportunity.update",
  "target": { "system": "salesforce", "object": "Opportunity", "id": "0061..." },
  "diff": {
    "StageName": { "from": "Discovery", "to": "Proposal" },
    "CloseDate": { "from": "2026-10-31", "to": "2026-11-30" }
  },
  "evidence": { "source": "call_2291", "quote": "let us aim to get paperwork out next month" },
  "status": "pending_approval",
  "idempotency_key": "call_2291:opp_0061:stage"
}

Four things make this work in practice.

A diff, not a payload. Show the reviewer from and to. A human can approve or reject a diff in about two seconds. Nobody actually reviews a raw JSON payload, so if you show a payload you have built an approval step that everyone rubber stamps, which is the same as having no gate at all.

Evidence attached to the claim. Every proposed change carries the quote or source that produced it. That is what makes review fast, and what makes the audit log useful six weeks later when someone asks why a field changed.

Idempotency keys. Approval is asynchronous and humans double click. Derive the key from the source event plus the target field so a replay is a no-op instead of a duplicate ticket.

Reversibility. Store the from values, not just the to. Every applied action gets an inverse. Undo turns a scary automation into one people will actually leave switched on, and it is nearly free to implement once you have captured the diff.

Preflight validation, before a human ever sees it

Do not spend human attention on actions that were going to fail anyway. Before an action enters the approval queue, validate it against the target system. Does the field exist, is the picklist value legal, does the record still exist, does this user have write permission on it.

Most bad proposals are not judgment errors. They are schema errors, and schema errors are cheap to catch in code.

What to log

Log the whole envelope, not just the outcome. Proposal, evidence, reviewer, decision, timestamp, applied result, and the inverse. If you can answer "who approved this, based on what, and how do I undo it" from a single table, you can pass a security review and you can debug production.

Do not log the raw customer content that produced the action beyond the specific evidence quote. You want interaction patterns and decisions, not a shadow copy of your customers' data.

The tradeoff

This is slower per action than full autonomy, and that is fine. Reviewing a diff is cheap. Recovering trust after a bad automated write is expensive, and often you do not get the chance.

Autonomy is the part that demos well. The gate is the part that decides whether the system is still switched on in six months.

I build this pattern for revenue teams at Mindlyft, where agents draft the post-call CRM updates, tickets and follow ups, and a human approves before anything ships.