How to Build an AI Agent Pipeline That Won't Delete Your Database

#ai #agents #safety #production

An AI agent in production can delete your database. Not because the model is malicious. Because it doesn't know what it doesn't know. It sees a modal it can't parse, guesses the wrong action, and that guess gets executed against a live system.

I've seen this pattern in enough projects to know it's not a theoretical risk. The question is not whether your agent will make a mistake. It's whether your architecture can survive that mistake.

Here's what I've learned building production LLM pipelines, browser automation agents, and RAG systems. The safety patterns that actually work.

Sandboxed Execution Is Not Optional

Every AI agent action should run in a context that cannot reach production data directly. This sounds obvious. I've seen teams skip it because "the model is just generating text, it can't do anything dangerous."

A model generating text can call functions. Functions can hit APIs. APIs can delete rows.

On a job platform I built, the LLM scoring pipeline processes listings at scale. The pipeline has a strict architecture: the model never touches the database. It receives structured inputs, returns structured outputs, and a middleware layer validates every field before it touches a row.

// The agent never gets direct DB access
// It talks to a validation middleware
async function processAgentAction(action: AgentAction): Promise<ActionResult> {
  // Step 1: Validate the action against allowed operations
  const validation = validateAction(action, ALLOWED_ACTIONS);
  if (!validation.valid) {
    return { status: 'rejected', reason: validation.error };
  }

  // Step 2: Execute in a sandboxed transaction
  // The agent can only perform READ operations
  // Any WRITE requires explicit human approval
  if (action.type === 'write') {
    return { status: 'pending_approval', action };
  }

  // Step 3: Log everything for audit
  await auditLog.create({
    agent: action.agentId,
    action: action.type,
    timestamp: new Date(),
    approved: action.type === 'read' // reads auto-approved
  });

  return executeRead(action);
}

The rule is simple: the agent proposes, the system disposes. No write path exists without an approval gate.

Human in the Loop Means Real Approval, Not a Formality

Suppose you're building an autonomous job application module. The agent browses listings, fills forms, and submits. Full automation sounds great until the agent applies to the wrong job with the wrong resume.

What works is per action approval. The candidate sees each match, reviews a summary, and explicitly approves before the agent acts. This is not a checkbox they click once and forget. It's a deliberate decision each time.

The pattern matters because approval fatigue is real. If you make someone approve 100 actions, they stop reading. They tap through. That's worse than no approval at all because it creates a false sense of safety.

What works is batching approvals at a granularity that matches human attention span. Show 5-10 matches at a time with clear previews of what the agent will do. The candidate reads the job title, company, and a short summary. They approve. The agent handles the rest.

For higher risk actions like payment or account changes, the approval gate needs to be explicit. A separate confirmation modal. A required text input. Something that forces the user to actually engage.

Idempotency Is Your Safety Net

The most dangerous AI agent bug I've seen is not the one that does the wrong thing. It's the one that does the wrong thing repeatedly.

An idempotent action produces the same result no matter how many times you call it. If your agent sends a "create lead" API call, and the API creates a duplicate lead each time, a retry loop or a confused agent can flood your system with garbage.

On a job platform, every pipeline step is idempotent by design:

// Idempotent job processing
// Running this 10 times produces the same result as running it once
async function processJobListing(jobId: string, source: string): Promise<void> {
  // Use a unique constraint to prevent duplicates
  // The database rejects a second insert with the same externalId
  await db.jobListing.upsert({
    where: { externalId: `${source}_${jobId}` },
    create: {
      externalId: `${source}_${jobId}`,
      title: rawData.title,
      company: rawData.company,
      // ... other fields
    },
    update: {
      // Only update fields that should change
      lastSeenAt: new Date(),
      status: rawData.status,
    }
  });
}

The upsert pattern is your friend. It says "insert if new, update if exists." No duplicates. No matter how many times the pipeline runs.

For browser automation agents, idempotency means checking state before acting. "Is this button already clicked? Is this form already submitted? Is this modal already dismissed?" The agent should read first, act second.

Rollback Strategies That Actually Work

Every AI agent action should be reversible or logged well enough to reverse manually.

For an autonomous apply module, that means a proxy email inbox. Every application goes through a unique email address. If something goes wrong, the candidate can see exactly what was sent, to whom, and when. They can follow up manually. The system logs every field that was submitted.

For database actions, rollback means transactions. If an agent writes to the database, it should do so inside a transaction that can be rolled back if the next validation step fails.

// Wrap agent writes in a transaction
// If any step fails, everything rolls back
async function executeAgentActionWithRollback(action: AgentAction): Promise<void> {
  const transaction = await db.$transaction(async (tx) => {
    // Step 1: Execute the action
    const result = await tx[action.table].create({
      data: action.data
    });

    // Step 2: Validate the result with a separate process
    const validation = await validateResult(result);
    if (!validation.passed) {
      // Throw to trigger rollback
      throw new Error(`Validation failed: ${validation.reason}`);
    }

    return result;
  });

  // Only committed if validation passed
  return transaction;
}

The transaction boundary creates a clean line. If the agent hallucinates, the database stays clean. You retry with a corrected prompt, not with a data cleanup script.

The Architecture That Survives Production

Here's the pattern I use now for every AI agent pipeline I build:

Isolation layer: The agent never touches production data directly. It reads through a restricted API, writes through a validation middleware.
Per action approval: Every write action requires explicit human confirmation. Reads are auto-approved but logged.
Idempotent operations: Every action produces the same result on retry. Upserts, not inserts. State checks before state changes.
Transaction boundaries: All writes happen inside transactions. Validation gates sit between the write and the commit.
Full audit trail: Every action is logged with agent ID, timestamp, input, output, and approval status. You can replay any session.
Kill switch: One API call stops all agent activity. Not a rate limit. A hard stop.

These patterns apply whether you're building an LLM scoring pipeline, a browser automation agent, or a RAG system. The specifics change. The principles don't.

If your team is evaluating AI agent integration and wondering how to ship fast without shipping dangerous, that's the kind of thing I help with. Happy to compare notes on what's worked and what hasn't.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.