DEV Community: Abdul Rehman

Your AI Agent Is Only as Safe as Your Data Model

Abdul Rehman — Wed, 05 Aug 2026 09:02:27 +0000

The AI Agent Promise, and What It Hides

Every week there's a new story about an AI agent that booked a double appointment, misfiled a customer record, or replied to a client with nonsense that sounded confident. The headlines make it sound like the problem is the AI itself. In my experience, the AI is rarely the problem.

The problem is the data underneath.

I've spent years building software and automation for growing businesses, and I've seen the same pattern repeat: a team hears "we can automate that with an AI agent," gets excited, and plugs an agent into a system that was never designed to be machine-readable. Then the agent does exactly what it was told, and the business pays for it.

If you run a clinic, a booking platform, a staffing agency, or any business where a mistake touches a customer directly, this should worry you. Not because AI is dangerous, but because your team probably doesn't know what your data is capable of. And an AI mistake can cost you a client's booking, leak a patient record, or break a compliance promise you made years ago.

The good news: this is preventable. But not the way most vendors will tell you.

Safety Isn't a Prompt Tweak Away

The first thing most people reach for is a better prompt. Tell the agent to be careful. Add guardrails to the instructions. Make it double-check its work.

That's a reasonable instinct, but it misses the real failure mode.

An AI agent doesn't fail because it misunderstood your instructions. It fails because the system it's working with gives it contradictory information. Suppose your booking flow has two different records for the same client, one from the website, one from the front desk. The agent looks at both, can't tell which is current, and books the appointment against the wrong one. No prompt in the world fixes that, because the agent was never told which record is authoritative. That's not an AI problem. That's a data-model problem.

The same logic applies to patient records, inventory counts, candidate profiles, and every other piece of information a business runs on. If your data is messy, your AI will be messy. If your data is contradictory, your AI will be contradictory. The agent will faithfully carry the confusion into production, and it will do it at scale.

So before you let an AI agent touch anything customer-facing, you have to know what your data can and can't do. That's a business question, not a technical one.

The Dental Group That Unified Before It Automated

A few years ago, I worked with a dental group running multiple locations. Their staff spent hours every day switching between several disconnected internal tools, scheduling in one place, patient notes in another, billing in a third. Re-entering the same data across systems was routine, and it wasn't just slow. It was a risk. Every time a record was re-entered by hand, there was a chance it was entered wrong.

The group's leaders came to me with a problem, not a feature request. They wanted their staff to stop juggling tools and to have one place where everything lived. So I built a desktop app that unified their internal tools into a single interface. After adoption, the group reported a 50% productivity boost, staff stopped re-entering data, and the information they trusted was the information in front of them.

Here's the part that matters for this conversation: we did this before any AI was involved. The point was to give the business one authoritative version of the truth. Only once that existed could they safely think about automation. If we'd plugged an AI agent into their old mess of disconnected tools, it would have inherited every inconsistency and every duplicate record. Instead, the agent, when it eventually arrives, will work from a clean, consistent foundation.

That's the order that keeps production safe: unify first, automate second. You can't delegate decisions to a machine when the machine can't even tell which record is the real one.

What to Audit Before an AI Agent Touches Anything

If you're a business owner or operations lead, you don't need to understand the internals of language models. You need a practical checklist. Here's the one I use with every client before we let an AI agent near a customer-facing system, the same discipline I bring to every automation project.

1. Who owns the record? For every customer, patient, or client, there should be exactly one system of record. If more than one system can create or edit that record, you have a conflict waiting to happen. Decide which system is authoritative, and make everything else read from it.

2. Can the system distinguish two records? If two customers share a name, or two patients have the same address, can your system tell them apart? If not, an agent will merge them, and you'll lose history. This is a data-quality problem, and it's the most common reason agents misfile.

3. What are the failure conditions? Before you automate, write down what "going wrong" looks like. A double booking. A deleted history. A reply sent to the wrong client. Then build the system so those failure conditions are impossible, not just unlikely. That's a design decision, not a prompt.

4. Can you see what the agent did? Every action an agent takes should leave a trace. If you can't answer "what did it do, and when?" within minutes, you're flying blind. Audit trails aren't a nice-to-have; they're how you catch a mistake before it reaches a customer.

5. What happens when it goes wrong? Hope is not a rollback plan. Before you turn an agent loose, know exactly how you'll undo its work. That might mean a backup, a staging environment, or a human approval step for high-risk actions. The question isn't whether something will go wrong, it's whether you can recover.

These five questions are business questions. They're about ownership, clarity, and accountability. Any AI vendor who can't answer them with you should give you pause.

The Question That Matters Most

I understand the appeal of AI agents. They promise to take real work off your team's plate, respond to customers in minutes instead of days, and handle the repetitive tasks that drain your staff. That's a future worth building toward.

But the question I hear from owners isn't "will it be fast?" It's "will this lose my client's booking or leak a patient record?" That's the right question to ask, and the honest answer is: an AI agent is only as safe as the data model it's plugged into. If your systems are disconnected, your records are duplicated, and your business rules live in someone's head, no agent should touch them yet.

That's why I encourage every business to audit their data before they automate, it's the core of how I partner with growing businesses. It's not glamorous. It doesn't make a good demo. But it's the difference between an AI that quietly makes your business easier and one that quietly breaks your most important customer promises.

If you're starting to feel this friction, if your team is re-entering data across systems, or you've been told an AI agent can "just handle it", the first step isn't a new tool. It's a clear, honest picture of what your data can and can't do. Get that right, and the automation will take care of itself.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at Abdul Rehman.

Why Your Booking Flow Is Losing Customers (And How to Fix It Without a Full Platform Rebuild)

Abdul Rehman — Sat, 01 Aug 2026 09:02:29 +0000

The hidden cost of a booking flow that doesn’t work for anyone

Every service business I work with has a moment when they realise their booking process is costing them more than they thought. For Owner Omar, it’s the Friday afternoon call from his admin lead: “Another customer said they booked online, but we never got it. They’re blaming us, and now they’re going to a competitor.” For Proprietor Priya, it’s the stack of manual intake forms her team re-enters into three different systems every morning, hours lost before the first appointment even starts.

The problem isn’t that customers don’t want to book. It’s that the process makes it harder than it should be. A booking flow with too many fields, unclear availability, or a broken confirmation step quietly pushes people away. You don’t get a complaint, you get silence and a lost booking. And because the friction is spread across small interactions, it’s easy to dismiss as “just how it is.”

But the operational cost adds up. Customers who give up mid-booking rarely come back. Staff spend time manually reconciling bookings, chasing no-shows, and re-entering data that should have flowed automatically. The business feels busy but isn’t growing, because every digital interaction is a small leak in the bucket.

The three friction points that quietly kill your conversion

When I look at a booking or intake flow for a service business, I’m not looking at the technology first. I’m looking for the moments where a customer or an employee has to do something that feels like unnecessary work. Those moments are where the leaks are. There are three that show up again and again.

Form drop-offs. The customer reaches the booking page, starts filling in details, and leaves before finishing. Maybe the form asks for too much upfront, address, insurance details, preferred contact time, before showing availability. Maybe the mobile experience is slow or the calendar widget doesn’t work. Whatever the reason, each field you add is a chance for someone to abandon. A hotel booking platform I worked with was losing guests mid-booking because the flow felt disjointed and the front-end wasn’t reliable. Customers couldn’t trust that their selection would stick, so they left.

No-shows from weak reminders. A booking confirmation email that lands in spam, or a text message that arrives three days early and gets buried, is the same as no reminder at all. Staff then spend time calling to confirm, or worse, dealing with empty slots that could have been filled. The friction here is invisible, you don’t see the customer who forgot, you only see the gap in your schedule.

Double data entry for your team. When a booking comes in through an online form but doesn’t sync with your internal system, someone has to copy it over. That’s time, and it’s error-prone. Staff re-entering data is a sign that your systems don’t talk to each other. It’s not a technology problem, it’s a friction problem that makes your operations harder to run and your team less effective.

How a targeted fix changed the numbers for one platform

Let me share a real example that shows what’s possible when you stop trying to rebuild everything and instead fix the specific friction points.

A hotel booking platform was losing guests mid-booking. The flow existed, but it wasn’t reliable, customers would select dates, see a price, then hit a page that didn’t load properly or lost their selection. The team knew they needed to improve, but a full platform rebuild wasn’t on the table. It would take months, cost too much, and risk breaking things that already worked.

Instead, we focused on the booking engine itself. I improved the front-end reliability and cleaned up the flow so that each step felt predictable. The calendar showed accurate availability, the price didn’t change unexpectedly, and the confirmation step worked every time. It was a scoped fix, no new backend, no new database, no migration. Just a better experience for the customer at the point where the decision to book happens.

The result? The project delivered on deadline, and the client told me: “Good quality and respected the defined deadlines.” More importantly, the booking flow stopped losing guests. The business didn’t need a new platform, it needed the existing one to work properly. That’s the kind of fix that pays for itself in recovered bookings within weeks, without the risk or cost of a full rebuild.

What a booking flow fix looks like in practice

A targeted improvement doesn’t mean small thinking. It means diagnosing exactly where the friction is and addressing it with a clear outcome in mind: fewer abandoned bookings, fewer no-shows, less manual work for your team.

Here’s what that looks like in practice for a typical service business:

Simplify the form. Cut unnecessary fields. Show availability before asking for personal details. Use smart defaults, if the customer is booking for themselves, don’t ask for a separate “guest name.”
Add live availability. If your booking system shows a calendar that’s not in sync with your actual schedule, customers will book slots you can’t fill, or skip because they think you’re full. A real-time connection to your calendar removes that uncertainty.
Automate reminders. Send a confirmation immediately after booking, then a reminder 24 hours and 2 hours before. Use SMS or email, whichever your customers actually read. A good reminder flow can cut no-shows by a meaningful margin.
Connect booking to your operations. When a booking is made, it should appear in your internal system automatically. No copy-paste, no double entry. This is often the highest-impact change for your team’s productivity, they stop being data entry clerks and start serving customers.

Each of these changes is small on its own. Together, they transform the experience for customers and staff. And because they’re targeted, you can implement them in weeks, not months, without affecting your existing platform.

The business case for a scoped fix instead of a rebuild

I understand the temptation to think, “Maybe we just need a whole new booking system.” I’ve seen businesses spend months evaluating platforms, then months more migrating, only to discover the new system has its own friction points.

The better path is to fix what’s actually broken. A scoped engagement, focused on the booking flow, the reminder system, or the integration between your booking and your CRM, is faster, cheaper, and less risky. It also gives you proof that the investment works before you commit to a larger change.

In my experience, the businesses that get the most value from this approach are the ones where the friction is clear but the solution feels out of reach. They know customers are dropping off. They know staff are overwhelmed. But they don’t have the internal expertise to diagnose the problem or the confidence to trust a partner who can deliver a fix without breaking everything else.

That’s where a senior technology partner who thinks in business outcomes first makes the difference. When you work with someone who removes digital friction, you get a clear diagnosis, a scoped plan, and a delivery that respects your deadlines and your budget. You don’t get a new platform, you get a booking flow that works.

From digital friction to digital confidence

A friction-free booking experience doesn’t just mean more bookings. It means your team spends less time fixing problems and more time delivering the service you’re known for. Your customers trust that when they book, it’s done. Your operations run without the daily patchwork of workarounds.

That’s the shift I help business owners make. Not by selling them a new platform, but by removing the specific friction that’s holding their current one back. If you’re reading this and recognising the feeling of losing customers to a competitor with a smoother booking process, or watching your team spend hours on manual data entry that should be automatic, I’d welcome a conversation.

No sales pitch, just a diagnostic look at where the friction is and what a targeted fix would take. I partner with growing businesses to remove digital friction, and sometimes the most valuable thing I can do is help you see the problem clearly. From there, the solution is straightforward.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at Abdul Rehman.

AI Agent Disasters Are a Wake-Up Call: Automate Safely, Not Blindly

Abdul Rehman — Thu, 30 Jul 2026 09:27:10 +0000

You've seen the headlines.

An AI agent deletes a production database. Another spends a marketing budget in hours instead of months. A third publishes something damaging because its instructions were too vague.

These stories circulate widely, and they're unsettling. If you're a business owner who has been cautiously exploring AI automation, these cautionary tales probably made you pause. Maybe they made you think: This isn't ready for my business. I'll wait until it's safer.

That's a reasonable instinct. But it's also a missed opportunity.

The Real Problem Isn't AI. It's Uncontrolled Experiments.

Every one of those cautionary tales shares a common root cause: the AI was given too much autonomy, too little context, and no human oversight. Someone treated a large language model like a junior employee who could be trusted to figure things out on their own, without guardrails, without supervision, and without a clear definition of what "done" looks like.

That's not responsible AI implementation. That's an experiment.

The difference between a disaster and a reliable automation is not the technology itself. It's how you design the system around it. It's the boundaries you set, the approvals you require, and the human checkpoints you build in.

For a growing business, the goal isn't to hand over the keys. It's to put AI to work on specific, well-defined tasks where it can remove real friction, while keeping you in control.

Controlled Automation: How AI Should Work in a Real Business

The safest approach to AI automation is one where the AI operates within a clearly defined scope, and a human approves every meaningful action before it takes effect.

Think of it this way: you wouldn't give a new hire the company credit card and say "go spend whatever you think is right." You'd give them a budget, a list of approved vendors, and a process for getting purchases signed off. AI agents need the same structure.

For a recruitment business, I built AI-driven workflows that handled resume tailoring and outreach automation. The AI generated drafts and suggested candidates. But every message was reviewed before it went out. The result was a significant increase in sales, because the team could focus on closing deals instead of writing dozens of individual emails. The AI didn't replace the human judgment. It multiplied the human's capacity.

That's the model that works. The AI does the heavy lifting, the repetitive work, the pattern matching. The human makes the decisions.

Where AI Automation Delivers Without the Risk

There are several categories of work where AI can be deployed safely, with minimal risk and maximum return. These are tasks that are repetitive, time-consuming, and bounded, where the cost of a mistake is low, and the cost of manual effort is high.

Content production is a good example. One marketing team I worked with was spending three days a week manually creating social media posts. We built an AI pipeline that generated multi-frame content with templates and scheduling, integrated with their existing CMS. Creation time dropped dramatically, and publish frequency increased significantly. The team reviewed and approved everything before it went live. No rogue posts. No brand disasters. Just faster output with the same human oversight.

Document analysis is another safe entry point. Contract review is slow and expensive, but the stakes are high enough that you want a human in the loop. I built a legal document analyzer where all parsing happens client-side in the browser, and only the extracted text goes to the LLM for clause-by-clause review. The AI flags clauses as present, missing, or ambiguous across multiple contract sections. The human lawyer makes the final call. It's faster, but not unsupervised.

Meeting assistance is almost zero-risk. An AI that captures screen, audio, and transcription, then generates coaching summaries, doesn't make decisions. It just observes and reports. The risk is low, and the time saved on note-taking is substantial.

The Guardrails That Protect Your Business

If you're considering AI automation, here are the guardrails I build into every implementation:

Define the scope precisely. The AI should know exactly what it is and isn't allowed to do. Vague instructions are the enemy of safe automation. Every prompt, every workflow, every integration should have clear boundaries written into its instructions.

Require human approval for actions. Any action that has financial, legal, or reputational consequences should require a human to click "approve" before it executes. This is non-negotiable.

Log everything. Every AI decision, every generated output, every action taken should be logged. If something goes wrong, you need to be able to trace exactly what happened and why.

Start small. Pick one workflow that is low-risk, well-understood, and currently costing your team significant time. Automate that. Measure the result. Then expand.

Test with real data before going live. Run the AI on historical data. Compare its outputs to what your team actually did. Validate that it works correctly before you let it touch live operations.

The Opportunity You Don't Want to Miss

The businesses that figure out AI automation now, safely and responsibly, are going to have a real advantage. They'll serve more customers with the same team. They'll respond faster. They'll make fewer manual errors. They'll free their people to focus on the work that actually requires human judgment.

The disasters you've read about are real, but they're also avoidable. They come from treating AI like a magic black box instead of a tool that needs careful design, clear boundaries, and human partnership.

If you're curious about what safe AI automation could look like in your business, I'd be glad to talk through where it fits and where it doesn't. The goal isn't to automate everything. It's to remove the friction that's slowing you down, one controlled step at a time.

For more on how I approach problems like this, see how I help businesses remove this kind of friction.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at Abdul Rehman.

Build a Real AI Feature for Your SaaS in 6 to 12 Weeks

Abdul Rehman — Tue, 28 Jul 2026 20:01:33 +0000

The Problem with AI Features That Never Ship

Every week I talk to founders who want to add an AI feature to their SaaS. They’ve seen the demos, read the hype, and know their competitors are moving. But when I ask what’s actually shipping, most admit nothing has made it to production yet.

That pattern is surprisingly common across the industry. The technology works fine in isolation. The failure comes from how the project is approached: starting with a shiny capability instead of a painful problem, underestimating the gap between a demo and a production system, or trying to build too much too soon.

The result is wasted investment, frustrated teams, and another AI initiative quietly shelved. The good news is that this pattern is avoidable. With the right focus and structure, a real AI feature can ship in six to twelve weeks.

Start with the Pain, Not the Technology

The single most common mistake is choosing an AI feature because it’s technically interesting. “Let’s add a chatbot” or “We should do AI-powered recommendations” often starts with the tool, not the problem.

A better approach is to ask: what is the most painful manual task your users or your team face every day? That friction is where AI creates real value.

I worked with a recruiting SaaS whose team spent hours tailoring resumes and writing outreach messages for each candidate. It was tedious, inconsistent, and limited how many placements they could handle. The business problem was clear: manual effort was a bottleneck to growth.

We built AI-driven workflows that automated resume tailoring and outreach, using OpenAI and enrichment APIs. The outcome was a 70% increase in sales after the workflows went live. The technology was secondary. The focus was on removing a specific, painful friction.

When you define the problem first, the AI feature has a clear purpose and a measurable impact. You also avoid the trap of building something nobody asked for.

From Demo to Production in 6 to 12 Weeks

Once you’ve identified the pain, the next challenge is moving from a proof of concept to a live feature without getting stuck. The key is to scope a thin slice that delivers real value, then iterate.

Here’s a typical structure I follow:

Week 1–2: Audit and define. Map the current workflow, identify where AI can remove friction, and define success criteria. This is not about writing code. It’s about understanding the business context.
Week 3–6: Build the core loop. Build the smallest end-to-end pipeline that solves the problem. No bells, no whistles. For example, a job discovery platform I worked on replaced a fragile manual scraping workflow with an AI pipeline that automatically ingests and scores over 10,000 listings daily. That core loop took a focused development cycle.
Week 7–9: Production hardening. Add error handling, monitoring, caching, and integration with existing systems. This is where most demos die. A real production AI feature must handle bad inputs, rate limits, and scale.
Week 10–12: Launch and measure. Ship to a subset of users, gather feedback, and refine. The goal is to get real usage data before investing in additional features.

That timeline works because it forces hard decisions early. You cannot build everything. You build the one thing that matters most, and you get it live fast.

Avoiding the Common Pitfalls

Even with a clear plan, a few traps consistently derail AI projects. I’ve seen each of them more than once across different projects.

Data privacy. If your SaaS handles sensitive user data, you cannot send everything to an external API without thought. For a legal document analyzer I built, all document parsing happens client-side in the browser. Only extracted text goes to the LLM for analysis. That architecture removed the trust barrier. Think about where your data lives and what your users expect.

Integration complexity. An AI feature that doesn’t fit into existing workflows is a feature nobody uses. The recruiting workflows I mentioned integrated directly with the SaaS’s existing candidate pipeline. No new logins, no extra steps. The AI was invisible. Plan for integration from day one.

Scope creep. The moment you start building, someone will suggest “let’s also add sentiment analysis” or “what if it could generate reports too?” Stick to the original pain. You can always add more later. In my experience, the features that made it to production and stayed useful were the ones that shipped narrow and reliable first.

The Partnership That Makes It Work

Building a production AI feature in weeks requires more than technical skill. It requires someone who can ask the right questions, push back on bad ideas, and own the entire delivery from start to finish.

That’s why I approach every project as a partner, not a hired developer. I start with an audit of your current systems and workflows. I recommend what creates the greatest business value, even if that means advising against a feature you thought you wanted. I communicate clearly about timelines, risks, and trade-offs.

One client, a staffing agency, put it this way: “We interview developers for every project, yet we’ve gone back to Abdul for the third time. What’s different is his communication. He is always responsive and sets expectations clear. He asks thoughtful questions and provides his insights and recommendations.”

That’s the kind of relationship that turns an AI project from a gamble into a reliable investment. I’ve written more about how I help businesses remove this kind of friction in a detailed guide.

Is Your SaaS Ready for a Real AI Feature?

If you’re spending hours on manual processes that could be automated, if your users are asking for smarter features, or if you’ve tried AI before and it didn’t ship, you’re not alone. The difference between a failed experiment and a valuable feature is the approach.

Start with the pain, scope a thin slice, and partner with someone who will own the outcome. Six to twelve weeks is realistic when you focus on what matters.

If that sounds like the kind of project you need help with, let’s talk. I’ll start with an honest assessment of where AI can actually make a difference for your business.

Written by Abdul Rehman, a trusted technology partner who helps growing businesses remove digital friction through modern software and intelligent automation. More at Abdul Rehman.

Technical Debt in Property Management: Why Your Old System Blocks AI

Abdul Rehman — Tue, 28 Jul 2026 19:14:20 +0000

What Technical Debt Looks Like in Property Management

Technical debt is a term most business owners hear from their IT teams, but rarely get explained in plain language. In property management, it's the cost of shortcuts taken in your software over time, a custom integration that was never updated, a manual workflow that was supposed to be temporary, or a system that was built quickly to meet a deadline and never revisited.

The problem isn't the original decision. Every business makes trade-offs to get things done. The problem is that these small compromises accumulate. What starts as a "quick fix" becomes a permanent part of your daily operations. Before long, your team is working around the software rather than through it.

Consider a property management business that relies on systems stitched together over years. One company might have its main property management software, a separate tool for tenant screening, a spreadsheet for maintenance tracking, and a different system for rent collection. Staff spend hours each week re-entering data between these tools. That's technical debt in action, invisible but expensive.

Three Ways Technical Debt Costs You Every Day

First, it costs time. Every time an employee has to manually copy tenant information from one system to another, or search through multiple platforms to find a lease agreement, that's lost productivity. I've seen this pattern in a multi-location business where staff had to jump between several disconnected internal tools every day, switching and re-entering data that ate hours across the group. After we built a unified desktop app that brought everything into one place, the group reported a 50% productivity improvement. That's not unusual, it's what happens when you remove the friction of disconnected systems.

Second, it costs reliability. Quick fixes tend to break. A custom integration that depends on a specific browser extension or an outdated API can fail without warning, and then you're scrambling to restore service. The fragility of these systems creates a constant low-level risk that your team has to manage.

Third, and most importantly for the future, technical debt blocks your ability to adopt new technology, especially AI. You can't add an intelligent assistant that automates lease renewals or predicts maintenance issues if your data is scattered across incompatible systems and your core platform doesn't expose clean, structured data through APIs. The AI needs a solid foundation, and technical debt undermines that foundation.

Why AI Can't Simply Be Added to an Old System

It's a natural question: "Can I just buy an AI tool and plug it into my existing property management software?" The answer is almost always no, unless your system is modern, well-integrated, and has clean data.

AI tools work best when they have access to reliable, structured data through stable APIs. If your system was built with quick fixes, it likely lacks proper APIs, or the data is inconsistent (e.g., tenant names stored in different formats across modules). An AI model trained on messy data will produce unreliable results. Worse, if the underlying system is fragile, adding an AI layer on top can introduce new failure points.

I've seen this pattern in other industries. One business I worked with relied on a fragile, manual workflow to gather data from external sources, a process that was one update away from breaking completely. The team spent more time maintaining the workaround than using it productively. When we replaced it with a proper pipeline that automatically ingests and scores data using AI, the team went from constant maintenance to a system that processes thousands of records daily without manual effort. The lesson is the same for property management: you can't build a modern AI layer on top of a legacy foundation.

How to Assess Your System's Technical Debt

You don't need a technical audit to know if your property management system has technical debt. Here are a few questions to ask yourself and your team:

Do you have to enter the same data in multiple places?
Are there manual steps that feel like they should be automated?
Does adding a new feature (like online rent payments or tenant portals) require a major project or a custom build?
Have you had to "work around" your software to get things done?
Is your team's productivity held back by how slow or clunky the system feels?

If you answered yes to any of these, you have technical debt. It's not a failure, it's a common outcome of running a growing business. The important thing is to recognize it and decide what to do.

I once worked with a business that tracked everything in spreadsheets: cost, production, sales, payroll. It worked, but it was manual and error-prone. We replaced that workflow with a dedicated operations platform, and the manual spreadsheet process was eliminated entirely. The same approach can apply to property management: identify the highest-friction areas first, and fix them one at a time.

The Path Forward: From Debt to AI-Ready

The good news is that fixing technical debt doesn't mean throwing everything out and starting over. That's rarely the right answer for a growing business. Instead, it's about strategic improvements: unifying data sources, building stable integrations, and modernizing the parts of your system that create the most friction.

In my experience, the most effective approach is to start with a single, high-impact workflow, tenant onboarding, maintenance requests, or lease renewals, and build a clean, integrated solution for that. Once that works, you expand. This phased approach reduces risk and gives you confidence that the investment is paying off.

Removing technical debt is the prerequisite for AI adoption. Once your systems are clean, stable, and connected, adding intelligent automation becomes straightforward. The AI can then actually help, by automating routine tasks, surfacing insights from your data, and improving the experience for both tenants and staff.

If you're wondering whether your property management system is holding you back from using AI, start by looking at the daily friction your team experiences. Those small frustrations are the real cost of technical debt. And they're the first place to begin the work of removing it.

I've written more about this specific challenge in real estate and how I help businesses remove this kind of friction, you can find that article here. If the questions above resonated with you, that's a good sign that a conversation about your system's readiness might be worthwhile.

Written by Abdul Rehman, a trusted technology partner who helps growing businesses remove digital friction through modern software and intelligent automation. More at Abdul Rehman.

How to Build a Production AI Agent That Won't Delete Your Database

Abdul Rehman — Mon, 20 Jul 2026 09:05:18 +0000

Suppose an agent is asked to "clean up duplicate entries." It could interpret that as "delete everything where the title field matches more than one record," which might match records with a common word in the title. A cascade delete in under four seconds, and a database with no point-in-time recovery. That's a bad day.

I've seen close calls like that. Not because the agent was malicious. Because the prompt and the agent's threshold for "stale" differed from the developer's intent. Since then I've built agents for job platforms, real estate matching, legal document analysis, and document intelligence systems. Every project reinforced the same patterns. Here's what I've learned about keeping AI agents from destroying your business.

The Three Failure Modes That Will Bite You

Production agents fail in predictable ways. The first is tool misuse. Give an agent a delete function and it will eventually call it on the wrong data. Not because it's malicious. Because the prompt said "remove stale entries" and the agent's threshold for "stale" differs from yours.

The second is hallucination cascades. An agent generates a plausible intermediate result, then acts on that result as if it were verified fact. One wrong assumption compounds into a chain of bad decisions.

The third is permission creep. You start with a read-only agent. Then someone adds a write capability for a specific use case. Then another. Six months later your agent has database admin credentials and nobody remembers why.

Every production incident I've dealt with or observed traces back to one of these three.

Human in the Loop Is Not Optional

I scoped an autonomous apply system for a job platform. The agent would find matching jobs, fill out applications, and submit them. The client wanted it fully automated. Instead, every application goes through a swipe interface. The agent prepares the application, screenshots the filled forms, and presents them to the user for approval. The user swipes right to send, left to discard. The agent never touches the submit button without a human signal.

This approach cost development time. It also prevents scenarios like the agent applying to dozens of jobs with the wrong resume template, which can happen when a prompt misreads "senior engineer" as "entry level."

The pattern is simple: destructive actions require human confirmation. Define destructive broadly. Sending emails. Deleting data. Making payments. Updating production records. If the action has irreversible consequences, a human needs to approve it.

// The pattern I use for human-in-the-loop approval
interface AgentAction {
  id: string;
  type: 'delete' | 'update' | 'create' | 'send';
  target: string;
  payload: unknown;
  risk: 'low' | 'medium' | 'high';
  requiresApproval: boolean;
  approvedBy?: string;
  approvedAt?: Date;
}

class ActionQueue {
  private pending: AgentAction[] = [];

  async propose(action: Omit<AgentAction, 'id'>): Promise<string> {
    const id = crypto.randomUUID();
    this.pending.push({ ...action, id, requiresApproval: action.risk !== 'low' });
    if (action.risk === 'high') {
      await this.notifyHuman(id, action);
    }
    return id;
  }

  async approve(id: string, userId: string): Promise<void> {
    const action = this.pending.find(a => a.id === id);
    if (!action) throw new Error('Action not found');
    action.approvedBy = userId;
    action.approvedAt = new Date();
    await this.execute(action);
  }
}

The queue buys you time. It forces the agent to articulate what it wants to do before doing it. That alone catches most bad decisions.

Sandbox Everything, Trust Nothing

Every agent I build runs in a sandboxed environment. The database connection uses a read-replica for any exploratory queries. Write operations go through a restricted API layer that validates every mutation against a schema.

For browser automation agents, I use headless Chrome in a disposable container. The agent gets a fresh browser profile every session. No cookies, no localStorage, no saved credentials. It starts clean and ends clean.

The job platform's autonomous apply module uses this exact pattern. The agent logs into a test account, navigates the application flow, and screenshots each step. The user reviews the screenshots before the agent can access the real production credentials. Even then, the agent never stores credentials. They're injected at runtime and revoked after the session.

// Sandboxed database access pattern
class AgentDatabase {
  private connection: Pool;

  constructor(readOnly: boolean = true) {
    this.connection = new Pool({
      // Read replica for exploration
      connectionString: readOnly 
        ? process.env.READ_REPLICA_URL 
        : process.env.WRITE_ENDPOINT_URL,
      max: 5,
    });
  }

  async query(sql: string, params: unknown[]): Promise<unknown[]> {
    // Block dangerous operations at the application layer
    const normalized = sql.trim().toLowerCase();
    if (normalized.startsWith('drop') || normalized.startsWith('truncate')) {
      throw new Error('Destructive operations require human approval');
    }
    // Rate limit aggressive queries
    if (this.isHeavyQuery(normalized)) {
      await this.queueForReview(sql);
      return [];
    }
    return this.connection.query(sql, params);
  }
}

The read-replica pattern alone is worth it. An agent can accidentally run an unindexed query on a large collection. The read replica handles it. The production primary never feels a thing.

Idempotency Is Your Safety Net

Idempotent operations are the single most underrated pattern in agent design. If an agent can retry an operation safely, you've eliminated an entire class of failure.

Every write operation in my agent systems uses idempotency keys. The agent generates a unique key before making any mutation. If the operation succeeds but the agent times out waiting for confirmation, it retries with the same key. The system sees the duplicate key and returns the original result without re-executing.

This matters more than you think. Agents retry. They retry aggressively. Without idempotency, a single timeout becomes a double charge, a duplicate entry, or a corrupted state.

// Idempotent mutation wrapper
async function executeWithIdempotency<T>(
  operation: () => Promise<T>,
  key: string
): Promise<T> {
  const existing = await cache.get(key);
  if (existing) return existing as T;

  const result = await operation();
  await cache.set(key, result, { ttl: 3600 }); // 1 hour window
  return result;
}

// Agent side
const idempotencyKey = `${sessionId}:${Date.now()}:${crypto.randomUUID()}`;
await executeWithIdempotency(
  () => api.createApplication(formData),
  idempotencyKey
);

The cache doesn't need to be permanent. An hour window covers the vast majority of retry scenarios. After that, if the agent retries, it's a genuinely new attempt and should execute fresh.

Rollback Strategies That Actually Work

Every agent operation should be reversible. Not theoretically. Practically.

For database mutations, I wrap agent actions in database transactions. If the agent's next step fails, the entire chain rolls back. For external API calls (sending emails, creating tickets), I maintain a compensation log. Each action gets a compensating action that undoes it.

The compensation log is a simple table: action_id, action_type, payload, compensating_action, status. When something goes wrong, I replay the compensating actions in reverse order. It's not perfect for every scenario, but it covers the common cases.

For the legal document analysis system I built, every document parse and clause extraction is logged with a version hash. If the agent generates a flawed risk assessment, the system can roll back to the previous version and flag the document for manual review. The agent never modifies the original document, it works on a copy until the human signs off.

The One Rule I Never Break

Don't give agents access to anything they haven't explicitly asked for.

No blanket database permissions. No "you can access any table" instructions. No API keys with wildcard scopes. Every capability the agent has must be justified by a specific, documented use case.

This sounds obvious. I've seen teams violate it under pressure to ship faster. The result is the same every time: a cascade delete, a mass email sent to the wrong list, a production data corruption. The fix is always the same too. Restrict permissions to the minimum needed for each specific task. Approve every escalation. Log every action.

If your team is building agents that touch production data and you're shipping slower because of safety concerns, that's the kind of thing I help with, happy to compare notes on what guardrails actually scale.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.

Production-Ready AI Agents: How to Deploy Without Losing Your Database

Abdul Rehman — Sun, 19 Jul 2026 09:01:59 +0000

I watched an AI agent send 200 emails to the wrong recipients because I forgot one validation check. The emails were well written. The offers were real. The recipients were just... not our leads.

That was early. I learned fast. Every agent I build now has three layers of guardrails before it touches a database or an API. Here's exactly what those layers look like and why they're non-negotiable for production.

Input Validation: Your Prompt Is Not a Schema

The first mistake people make is trusting the LLM to produce valid output. It won't. Not reliably. I've seen GPT-4 return a JSON key called "emial" instead of "email" in a critical pipeline. One typo, and the whole record is garbage.

The fix is a strict validation layer that runs before any data reaches your system. In my AI resume tailor, I use a JSON schema with conditional presence flags. Every field that must be real has a has_* boolean guard. If the LLM tries to fabricate a phone number, the schema rejects it.

const resumeSchema = z.object({
  contact: z.object({
    email: z.string().email(),
    phone: z.string().optional(),
    has_phone: z.boolean()
  }).refine(data => {
    // If phone is present, the guard must be true
    return data.phone ? data.has_phone : !data.has_phone
  }, "Phone number present but has_phone flag is false")
})

This pattern catches hallucinations before they corrupt your database. The schema is the contract. The LLM is just a suggestion engine.

Permission Scoping: Give Agents the Minimum They Need

An agent should never have write access to tables it doesn't need. That sounds obvious, but I've seen production systems where a job description rewriting agent had full CRUD access to the user table.

When I built the LLM scoring pipeline for a job board platform, I created separate database roles. The scoring agent only had SELECT on the job listings table and INSERT on a scoring results table. It never touched users, applications, or configuration. Even if the prompt was hijacked, the damage was contained.

For a real-time meeting assistant I built, the transcription agent could write to an analysis results table but had no access to user profiles or billing data. A separate service handled the database write after the agent's output passed validation.

The rule: scope permissions to the narrowest possible set. Then scope again. Then test what happens when the agent goes rogue.

Human-in-the-Loop: Destructive Actions Need Approval

Some actions are too dangerous to automate. Deleting records, sending bulk communications, writing to production databases. These need a human in the loop.

In a social media automation tool I built, the agent could generate posts and schedule them, but publishing required a manual approval step. The system showed a preview. The user clicked "Approve" or "Reject". No auto-publish without consent.

The autonomous apply module I'm building for a client uses a swipe interface. The agent finds matching jobs, but the user swipes right to approve each application. The agent never applies without confirmation.

This pattern is simple. It saves you from the "I accidentally sent 5000 emails" panic. It also forces your users to stay engaged with the agent's output rather than blindly trusting it.

Rate Limiting and Cost Controls

AI agents can run up costs fast. An infinite loop calling GPT-4 can burn through your API budget in minutes. I learned this the hard way when the job board's AI rewrite pipeline had to be shut down because costs spiraled.

Now every agent I build has a token budget and a rate limit. The social media tool uses Groq API with load balancing across 16 models, but each model has a per-minute cap. If the cap is hit, the agent pauses and retries later.

For the resume tailor, I use GPT-4o-mini for bulk processing and GPT-4o only for complex tasks. The system tracks token usage per session and alerts me if it exceeds a threshold.

Rate limiting isn't just about cost. It's about stability. An agent that fires 1000 requests per second can bring down your database. A simple circuit breaker pattern prevents that.

Rollback Strategies: Undo What the Agent Did

No matter how many guardrails you have, something will slip through. The question is: can you undo it?

Every agent I build writes to an audit log before making a change. The log captures the original state, the proposed change, and the agent's ID. If something goes wrong, I can replay the log and reverse the actions.

For database operations, I use transactions. The agent's write is wrapped in a transaction that can be rolled back if the change doesn't pass validation. This is standard practice for any production system, but it's often forgotten when building AI features.

The ATS middleware I built for a client used idempotent sync logic. If the same job listing was processed twice, it wouldn't create a duplicate. Safe retry mechanisms meant a failed sync could be replayed without corrupting data.

Idempotency is your best friend. Design every agent operation so it can be safely retried. That way, a rollback is just a replay of the correct state.

Closing

AI agents are powerful. They can automate entire workflows, write content, and engage with users. But they're also unpredictable. Every agent I've deployed has done something I didn't expect.

The guardrails I've described are not optional. They're the difference between a tool that makes your team more productive and a liability that costs you a weekend of firefighting.

If your team is shipping AI features and worrying about what happens when they go rogue, that's the kind of thing I help with. Happy to compare notes.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.

Production RAG at Scale: Lessons from Processing 10,000+ Listings Daily

Abdul Rehman — Wed, 15 Jul 2026 09:05:08 +0000

I spent time debugging a RAG pipeline that looked correct in staging and behaved differently under real load. The job board I built processes thousands of listings every day, scoring each one against candidate profiles using LLM function calls. Getting that pipeline stable and affordable took more than a good vector store choice. It took understanding where the system actually breaks.

Here's what I learned about chunking, embeddings, vector stores, cost control, and observability when RAG stops being a demo and starts being a production service.

Chunking Strategy: The Decision That Determines Everything

Most tutorials treat chunking as a parameter you tune later. In production, your chunking strategy determines your embedding quality, your retrieval accuracy, and your cost structure from day one.

For job listings, I tried three approaches before landing on one that worked:

Fixed-size chunks (512 tokens). Simple but terrible for this use case. A job description is a structured document with sections: responsibilities, qualifications, benefits. Fixed chunks split those sections arbitrarily. You lose the semantic boundary between "must have 5 years Python" and "nice to have React experience." Retrieval becomes noisy.

Semantic chunking with section headers. Better. I split on markdown headers and paragraph breaks. Each chunk preserved a complete thought. But some sections were too long (qualifications could run 800 tokens) and others too short (benefits might be two lines). Uneven chunk sizes meant wasted embedding calls on tiny sections and missed context on large ones.

Recursive character splitting with overlap. This is what stayed. I split on double newlines first, then single newlines, then sentences. Each chunk targets roughly 400 tokens with some overlap. The overlap is critical: it catches the boundary cases where a sentence spans two chunks. For job listings specifically, I added a pre-processing step that normalizes the raw ATS output into a consistent structure before chunking. Greenhouse and Lever both return different formats. Normalizing first means the chunker sees clean input.

async function chunkJobListing(text: string): Promise<string[]> {
  const normalized = normalizeListing(text) // strip HTML, unify line endings
  const splitter = new RecursiveCharacterTextSplitter({
    chunkSize: 400,
    chunkOverlap: 50,
    separators: ['\n\n', '\n', '. ', ' '],
  })
  return splitter.splitText(normalized)
}

The key lesson: design your chunking around your document type, not around a generic token budget. Job listings have a natural structure. Legal documents have a different one. Product descriptions have another. Match the chunker to the content.

Embedding Model: OpenAI vs Local

I started with text-embedding-3-small. It works, it's reliable, and the API is trivial to call. At thousands of listings per day, each listing producing several chunks, I was generating tens of thousands of embeddings daily. The cost was manageable but not trivial.

I tested a self-hosted alternative: Llama 3.1 via Ollama on the same AWS EC2 instance running the application. The embedding quality was noticeably worse for this use case. Job descriptions contain domain-specific language: "ATS-compliant resume," "Greenhouse integration," "equity compensation." The local model blurred these distinctions. Two listings that required very different skill sets ended up with similar embeddings.

The tradeoff was clear. OpenAI's embeddings cost money but returned accurate matches. The self-hosted model was free but produced noisy retrieval that required more LLM calls to correct downstream. I stayed with OpenAI.

One optimization that helped: I batch embedding requests. Instead of calling the API once per chunk, I send arrays of up to 100 chunks in a single call. OpenAI charges per token, not per call, so batching saves no token cost. But it reduces latency from N sequential calls to one parallel call, and it keeps the pipeline fast enough to process the daily volume within the scraping window.

Vector Store: Pinecone vs pgvector

I evaluated both and ended up with a hybrid that surprised me.

Pinecone is easy to set up and fast at query time. I had a working PoC in an afternoon. But at production scale, the cost added up fast. A Pinecone pod index with enough capacity for the vector volume I needed cost more than I wanted to pay per month. And I needed retention beyond a short window for historical matching.

pgvector inside PostgreSQL was more work upfront. I had to write the index setup, the query logic, and the maintenance scripts myself. But the savings were dramatic. My existing PostgreSQL instance on the same EC2 box handled the vector workload with no additional infrastructure cost. Query performance was close enough for approximate nearest neighbor search with the right index configuration (IVFFlat with appropriate lists and probes).

CREATE INDEX idx_job_embeddings ON job_chunks 
USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

SELECT * FROM job_chunks 
ORDER BY embedding <=> $1::vector 
LIMIT 20;

The real win was transactional consistency. With Pinecone, I had to manage sync between PostgreSQL and the vector store. A listing could be in one but not the other during failures. With pgvector, the embedding lives in the same database as the listing data. One transaction, one source of truth. No reconciliation scripts.

I use Pinecone for prototyping new RAG features. I use pgvector for production. That pattern has held across multiple projects.

LLM Scoring at Scale: Function Calls and Cost Control

The embedding search returns candidate chunks. Then I need an LLM to score each job listing's relevance to a specific candidate profile. This is the expensive part.

I use GPT-4o with function calling for scoring. The function schema is strict: it outputs a relevance score (0-100), a brief justification, and a list of matching skills. The schema enforces that the model cannot fabricate matches. Every skill in the output must be present in the listing text.

const scoreFunction: ChatCompletionTool = {
  type: 'function',
  function: {
    name: 'score_job_relevance',
    parameters: {
      type: 'object',
      properties: {
        relevance_score: { type: 'number', minimum: 0, maximum: 100 },
        justification: { type: 'string', maxLength: 200 },
        matching_skills: { type: 'array', items: { type: 'string' } },
      },
      required: ['relevance_score', 'justification', 'matching_skills'],
    },
  },
}

At thousands of listings per day, scoring each one with GPT-4o would cost far more than is sustainable.

Three things brought it down:

OpenAI Batch API. I send scoring jobs in batches of several hundred at a time. The Batch API gives a significant discount and completes within a few hours. Since the scoring pipeline runs overnight during the scraping window, latency isn't a problem. The batch completes before the morning traffic spike.

Caching. If a listing has been scored for a similar candidate profile before, I return the cached result. The cache key is a hash of the listing ID plus the candidate's skill vector. The hit rate is meaningful for repeat candidates.

Model tiering. Not every listing needs GPT-4o. Listings for common roles (software engineer, sales representative) score well enough with GPT-4o-mini. I reserve GPT-4o for niche roles where accuracy matters more (compliance officer, biomedical engineer). The routing logic is simple: if the job title matches a list of common titles, use the cheaper model.

Total daily LLM cost dropped substantially. The Batch API alone accounted for much of the savings.

Observability: What Broke and How I Caught It

The pipeline ran silently for a period before I discovered it was failing on a noticeable percentage of listings. The errors were non-fatal: a malformed ATS response would cause the chunker to produce empty chunks, which the embedding step would skip silently, and the listing would never appear in search results. No crashes, no alerts.

I added Sentry for error tracking and LogRocket for session replay on the frontend. But the real win was adding structured logging with a correlation ID that traces a single listing through the entire pipeline: ingestion, normalization, chunking, embedding, storage, scoring.

logger.info('Listing processed', {
  listingId: id,
  source: 'greenhouse',
  chunks: chunkCount,
  embeddingLatency: ms,
  score: relevanceScore,
})

This let me build a dashboard showing pipeline health per source. Some ATS sources processed successfully at a high rate. Others failed more often because their API returns HTML in description fields that the normalizer didn't handle. I fixed the normalizer, and the failure rate dropped significantly.

Without the correlation ID, I would have found that bug much later, if at all.

What I'd Do Differently

If I started this pipeline today, I'd skip Pinecone entirely and go straight to pgvector. The setup cost is higher, but the operational simplicity of having one database is worth it. I'd also build the observability layer before the pipeline, not after.

The biggest surprise was how much of the work was data normalization, not AI. Most of the bugs came from inconsistent ATS output formats, not from the LLM or the vector store. The AI parts are surprisingly reliable once you handle the data plumbing correctly.

If your team is building a production RAG pipeline and hitting the same kind of silent failure modes or cost surprises, that's the kind of thing I help with. Happy to compare notes on what worked and what didn't.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.

How to Build AI Agents That Won't Delete Your Database

Abdul Rehman — Tue, 14 Jul 2026 09:02:08 +0000

I watched a test agent try to delete a PostgreSQL table. It had the credentials. It had the intent. The only thing stopping it was a single line in the system prompt that said "you are a read-only assistant."

That line held. But I've seen enough close calls to know that a prompt is not a safety net.

Every week there's a new horror story. An agent that wiped a production database. An agent that mailed 10,000 customers the wrong message. An agent that ran up a $50,000 API bill in an hour. The common thread is not malicious intent. It's architecture that treats the LLM as a trusted operator instead of a powerful but unreliable intern.

The answer is not "don't use agents." The answer is "use agents with the right guardrails." I've built systems that use LLMs for scoring, generation, and structured extraction at production scale. Here are the patterns that keep agents from doing damage.

Default to Read-Only

Every agent I build starts life as a read-only system. It can observe, analyze, and report. It cannot write, delete, or execute.

This is not a feature you add later. It is the default state. You promote the agent to write access only when you have proven that it needs it and that you can control what it writes.

The implementation is straightforward. Give the agent a database connection with read-only credentials. Wrap all write operations behind a separate service that requires explicit approval. Never let the agent call a mutation API directly.

Here is what that looks like in practice with OpenAI function calling:

const readOnlyFunctions = [
  {
    name: "query_database",
    description: "Execute a SELECT query against the database. Read only.",
    parameters: {
      type: "object",
      properties: {
        sql: { type: "string", description: "The SELECT query to execute" },
      },
      required: ["sql"],
    },
  },
  {
    name: "search_records",
    description: "Search for records matching criteria.",
    parameters: { /* ... */ },
  },
];

// Write operations live in a separate system, not exposed to the agent.
// The agent can REQUEST a write, but a human must approve it.
const writeFunctions = [
  {
    name: "request_write",
    description: "Request a write operation. Requires human approval.",
    parameters: {
      type: "object",
      properties: {
        action: { type: "string", enum: ["update", "delete", "insert"] },
        target: { type: "string" },
        reason: { type: "string" },
      },
      required: ["action", "target", "reason"],
    },
  },
];

The agent can ask for a write. It cannot perform one. This forces a human-in-the-loop gate at the architectural level, not just the prompt level.

Human-in-the-Loop Gates That Actually Work

The classic "human in the loop" is a yes/no confirmation dialog. It works until users get fatigue and start clicking "yes" without reading. I've seen it happen.

The better pattern is to require the human to re-express the intent in their own words. Instead of "approve this action," show a summary and ask the user to type or confirm the specific change. For example, rather than a button that says "Delete user 1234," show the user a preview of what will happen and require them to type "delete user 1234" to proceed.

For high-risk actions, add a cooldown period. The agent cannot execute the same action twice within a window. This prevents a runaway loop where the agent floods the approval queue with identical requests.

Consider a social media engagement agent that autonomously replies to conversations. Suppose you give it the ability to generate replies but not to post them. A human reviews each generated reply, edits if needed, and clicks publish. That extra step catches hallucinations, tone mismatches, and the kind of inappropriate joke an LLM thinks is funny. The agent never touches the posting API at all. That separation is what saves you.

Idempotent Actions and Safe Retries

Agents fail. They time out. They retry. The worst thing you can do is let an agent retry a non-idempotent action.

Every mutation your agent performs should be safe to call twice. Use idempotency keys. Generate a unique key for each action, and if the agent retries with the same key, the system deduplicates it.

Here is a pattern I use for job application agents that need to submit applications:

async function submitApplication(application: Application, idempotencyKey: string) {
  // Check if this key was already processed
  const existing = await db.applications.findUnique({
    where: { idempotencyKey },
  });
  if (existing) {
    return existing; // Already submitted, return the result
  }

  // Submit the application
  const result = await atsApi.submit(application);

  // Store the result with the key
  await db.applications.create({
    data: {
      idempotencyKey,
      status: result.status,
      submittedAt: new Date(),
    },
  });

  return result;
}

The agent always passes an idempotency key. If the API call times out and the agent retries, the second call is a no-op. No duplicate applications, no double charges, no double deletions.

Sandbox the Agent's Environment

The agent should never have direct access to your production database or any production API. This is non-negotiable.

Create a proxy layer that sits between the agent and every external system. This proxy validates every request against a policy. The policy defines what the agent is allowed to do, what data it can see, and what it cannot touch.

For databases, I run agents against a replica that is read-only and lag-tolerant. The agent can see near-real-time data but cannot write to it. If the agent needs to write, it goes through the proxy which logs every mutation and enforces rate limits.

For external APIs, the proxy strips dangerous capabilities. If the agent is calling a CRM API, the proxy removes the DELETE endpoint from the available routes. The agent never sees the full API surface. It sees only what the proxy exposes.

Error Recovery Without Cascading

When an agent's action fails, the default behavior should be to stop, not to retry blindly. A circuit breaker pattern works well here: suppose the agent gets three errors in a row from the same service. The circuit opens. The agent cannot call that service again until a human resets it.

This prevents a single misbehaving agent from hammering a downstream system during an outage. It also prevents the agent from making things worse by trying to "fix" a failed operation with a destructive alternative.

I also log every action the agent takes, including the full reasoning chain. When something goes wrong, I can replay the agent's decision-making process and see exactly where it went off the rails. This is invaluable for debugging and for improving the guardrails.

The Architecture That Scales

The pattern that works across all agent systems I've designed is this: the agent is a proposal engine, not an execution engine. It generates ideas, arguments, and plans. A separate, tightly controlled execution layer decides whether to carry them out.

The agent never touches the production database. It never calls a mutation API directly. It never has admin credentials. It is given the minimum permissions needed to do its job, and those permissions are enforced at the infrastructure level, not the prompt level.

If your team is wrestling with AI agent safety and shipping slower because you're afraid of what the agent might do, that is the kind of thing I help with. I build production AI systems that are powerful enough to be useful and safe enough to ship without fear. Happy to compare notes.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.

How to Build an AI Agent Pipeline That Won't Delete Your Database

Abdul Rehman — Mon, 13 Jul 2026 09:02:51 +0000

An AI agent in production can delete your database. Not because the model is malicious. Because it doesn't know what it doesn't know. It sees a modal it can't parse, guesses the wrong action, and that guess gets executed against a live system.

I've seen this pattern in enough projects to know it's not a theoretical risk. The question is not whether your agent will make a mistake. It's whether your architecture can survive that mistake.

Here's what I've learned building production LLM pipelines, browser automation agents, and RAG systems. The safety patterns that actually work.

Sandboxed Execution Is Not Optional

Every AI agent action should run in a context that cannot reach production data directly. This sounds obvious. I've seen teams skip it because "the model is just generating text, it can't do anything dangerous."

A model generating text can call functions. Functions can hit APIs. APIs can delete rows.

On a job platform I built, the LLM scoring pipeline processes listings at scale. The pipeline has a strict architecture: the model never touches the database. It receives structured inputs, returns structured outputs, and a middleware layer validates every field before it touches a row.

// The agent never gets direct DB access
// It talks to a validation middleware
async function processAgentAction(action: AgentAction): Promise<ActionResult> {
  // Step 1: Validate the action against allowed operations
  const validation = validateAction(action, ALLOWED_ACTIONS);
  if (!validation.valid) {
    return { status: 'rejected', reason: validation.error };
  }

  // Step 2: Execute in a sandboxed transaction
  // The agent can only perform READ operations
  // Any WRITE requires explicit human approval
  if (action.type === 'write') {
    return { status: 'pending_approval', action };
  }

  // Step 3: Log everything for audit
  await auditLog.create({
    agent: action.agentId,
    action: action.type,
    timestamp: new Date(),
    approved: action.type === 'read' // reads auto-approved
  });

  return executeRead(action);
}

The rule is simple: the agent proposes, the system disposes. No write path exists without an approval gate.

Human in the Loop Means Real Approval, Not a Formality

Suppose you're building an autonomous job application module. The agent browses listings, fills forms, and submits. Full automation sounds great until the agent applies to the wrong job with the wrong resume.

What works is per action approval. The candidate sees each match, reviews a summary, and explicitly approves before the agent acts. This is not a checkbox they click once and forget. It's a deliberate decision each time.

The pattern matters because approval fatigue is real. If you make someone approve 100 actions, they stop reading. They tap through. That's worse than no approval at all because it creates a false sense of safety.

What works is batching approvals at a granularity that matches human attention span. Show 5-10 matches at a time with clear previews of what the agent will do. The candidate reads the job title, company, and a short summary. They approve. The agent handles the rest.

For higher risk actions like payment or account changes, the approval gate needs to be explicit. A separate confirmation modal. A required text input. Something that forces the user to actually engage.

Idempotency Is Your Safety Net

The most dangerous AI agent bug I've seen is not the one that does the wrong thing. It's the one that does the wrong thing repeatedly.

An idempotent action produces the same result no matter how many times you call it. If your agent sends a "create lead" API call, and the API creates a duplicate lead each time, a retry loop or a confused agent can flood your system with garbage.

On a job platform, every pipeline step is idempotent by design:

// Idempotent job processing
// Running this 10 times produces the same result as running it once
async function processJobListing(jobId: string, source: string): Promise<void> {
  // Use a unique constraint to prevent duplicates
  // The database rejects a second insert with the same externalId
  await db.jobListing.upsert({
    where: { externalId: `${source}_${jobId}` },
    create: {
      externalId: `${source}_${jobId}`,
      title: rawData.title,
      company: rawData.company,
      // ... other fields
    },
    update: {
      // Only update fields that should change
      lastSeenAt: new Date(),
      status: rawData.status,
    }
  });
}

The upsert pattern is your friend. It says "insert if new, update if exists." No duplicates. No matter how many times the pipeline runs.

For browser automation agents, idempotency means checking state before acting. "Is this button already clicked? Is this form already submitted? Is this modal already dismissed?" The agent should read first, act second.

Rollback Strategies That Actually Work

Every AI agent action should be reversible or logged well enough to reverse manually.

For an autonomous apply module, that means a proxy email inbox. Every application goes through a unique email address. If something goes wrong, the candidate can see exactly what was sent, to whom, and when. They can follow up manually. The system logs every field that was submitted.

For database actions, rollback means transactions. If an agent writes to the database, it should do so inside a transaction that can be rolled back if the next validation step fails.

// Wrap agent writes in a transaction
// If any step fails, everything rolls back
async function executeAgentActionWithRollback(action: AgentAction): Promise<void> {
  const transaction = await db.$transaction(async (tx) => {
    // Step 1: Execute the action
    const result = await tx[action.table].create({
      data: action.data
    });

    // Step 2: Validate the result with a separate process
    const validation = await validateResult(result);
    if (!validation.passed) {
      // Throw to trigger rollback
      throw new Error(`Validation failed: ${validation.reason}`);
    }

    return result;
  });

  // Only committed if validation passed
  return transaction;
}

The transaction boundary creates a clean line. If the agent hallucinates, the database stays clean. You retry with a corrected prompt, not with a data cleanup script.

The Architecture That Survives Production

Here's the pattern I use now for every AI agent pipeline I build:

Isolation layer: The agent never touches production data directly. It reads through a restricted API, writes through a validation middleware.
Per action approval: Every write action requires explicit human confirmation. Reads are auto-approved but logged.
Idempotent operations: Every action produces the same result on retry. Upserts, not inserts. State checks before state changes.
Transaction boundaries: All writes happen inside transactions. Validation gates sit between the write and the commit.
Full audit trail: Every action is logged with agent ID, timestamp, input, output, and approval status. You can replay any session.
Kill switch: One API call stops all agent activity. Not a rate limit. A hard stop.

These patterns apply whether you're building an LLM scoring pipeline, a browser automation agent, or a RAG system. The specifics change. The principles don't.

If your team is evaluating AI agent integration and wondering how to ship fast without shipping dangerous, that's the kind of thing I help with. Happy to compare notes on what's worked and what hasn't.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.

How to Build AI Agents That Don't Delete Your Production Database

Abdul Rehman — Sun, 12 Jul 2026 09:02:50 +0000

Suppose you give an AI agent a database connection string with write permissions and a tool to execute raw SQL. It will do exactly what you asked. That's the problem.

I've seen teams make this mistake. They treat agent safety as an afterthought, something to add after something breaks. But by then the damage is done. The guardrails I rely on came from studying those failures and building systems that prevent them from happening in the first place.

Read-Only Connections Are Your First Line of Defense

The single most effective safety measure I've implemented costs nothing and takes five minutes. Give your agent a read-only database connection by default. Only promote to write access when a human explicitly approves the operation.

// Database connection factory with role-based permissions
function getDbConnection(operation: 'read' | 'write'): Pool {
  if (operation === 'read') {
    return new Pool({
      connectionString: process.env.DB_READONLY_URL,
      // This user has SELECT only on all tables
    });
  }

  return new Pool({
    connectionString: process.env.DB_WRITE_URL,
    // This user has INSERT, UPDATE, DELETE on specific tables
  });
}

// Agent tool definition
const queryDatabase = {
  name: 'query_database',
  execute: async (sql: string, context: AgentContext) => {
    const connection = getDbConnection(
      context.hasWriteApproval ? 'write' : 'read'
    );
    return connection.query(sql);
  }
};

This pattern alone prevents an entire category of catastrophic failures. The agent can explore data, analyze patterns, and generate insights without ever touching production records. When it genuinely needs to write, the flow goes through a human approval step.

Consider an agent that parses natural language buyer requirements and matches them against property opportunities in a PostgreSQL database. The agent should never have write access. It should only query and return results. That constraint forces you to design a separate ingestion pipeline for new data, which turns out to be the right architecture anyway.

Human-in-the-Loop Isn't Optional, It's Architecture

The teams that skip human approval are the ones that deal with the consequences. I've learned to treat human approval as a design constraint from day one, not a feature to add later.

My pattern looks like this: every write operation generates a preview. The preview shows exactly what will change, in plain language and in structured data. A human reviews it. If they approve, the change executes. If they reject it, the agent gets feedback and tries a different approach.

// Approval queue for write operations
interface WriteOperation {
  id: string;
  agentId: string;
  operation: 'insert' | 'update' | 'delete';
  targetTable: string;
  preview: {
    affectedRows: number;
    beforeState?: Record<string, unknown>;
    afterState: Record<string, unknown>;
  };
  status: 'pending' | 'approved' | 'rejected';
  createdAt: Date;
}

async function requestWriteApproval(
  operation: WriteOperation
): Promise<boolean> {
  // Store in approval queue
  await db.insert(operationQueue).values(operation);

  // Notify human via Slack/email
  await notifyApprovalChannel({
    agentId: operation.agentId,
    preview: operation.preview,
    approvalUrl: `${APP_URL}/approvals/${operation.id}`,
  });

  // Wait for human response (with timeout)
  return pollForApproval(operation.id, TIMEOUT_30_MINUTES);
}

Think about an agent tasked with updating job listing statuses. It finds a bug in a partner feed that marks all listings as "closed." Without human approval, it would archive thousands of active listings in one pass. The human reviewer catches the anomaly in the preview, rejects the operation, and the feed gets fixed instead. That's the value of a review step.

Output Validation Is a Contract, Not a Suggestion

LLMs are probabilistic. They will occasionally output malformed JSON, invent fields, or return data that doesn't match your schema. If your agent passes that output directly to your database or API, you're asking for trouble.

I validate every agent output against a strict schema before it touches any production system. If the output doesn't match, the agent retries or the operation fails safely.

import { z } from 'zod';

// Schema for agent output that will be written to the database
const JobUpdateSchema = z.object({
  listingId: z.string().uuid(),
  status: z.enum(['active', 'paused', 'closed', 'draft']),
  salaryRange: z.object({
    min: z.number().positive(),
    max: z.number().positive(),
  }).refine(data => data.max >= data.min, {
    message: 'Max salary must be >= min salary',
  }),
  // Anti-hallucination guard: must match existing listing IDs
  hasValidListingId: z.literal(true),
});

function validateAgentOutput<T>(
  output: unknown,
  schema: z.ZodSchema<T>
): T {
  const result = schema.safeParse(output);

  if (!result.success) {
    // Log the failure for debugging
    logger.error('Agent output validation failed', {
      errors: result.error.flatten(),
      rawOutput: output,
    });

    // Don't proceed. Fail safely.
    throw new AgentValidationError(
      'Agent output did not pass validation',
      result.error
    );
  }

  return result.data;
}

The hasValidListingId field is a pattern I use frequently. It forces the LLM to explicitly confirm it found a valid reference before proceeding. If the model hallucinates a listing ID that doesn't exist, this check catches it.

Sandboxed Execution Environments Contain the Damage

Even with read-only connections and output validation, your agent might still need to run code. Maybe it's generating SQL, executing API calls, or running Python scripts for data analysis. Never run that code in your main application process.

I run all agent-generated code in isolated sandboxes. For simple operations, I use Docker containers with no network access and a read-only filesystem. For more complex scenarios, I spin up ephemeral environments that are destroyed after each execution.

// Execute agent-generated code in a sandboxed Docker container
async function executeInSandbox(code: string): Promise<SandboxResult> {
  const container = await docker.createContainer({
    Image: 'node:20-slim',
    Cmd: ['node', '-e', code],
    NetworkDisabled: true,        // No network access
    ReadonlyRootfs: true,         // Read-only filesystem
    HostConfig: {
      Memory: 256 * 1024 * 1024, // 256MB memory limit
      CpuPeriod: 100000,
      CpuQuota: 50000,            // 0.5 CPU limit
    },
  });

  const output = await container.wait();
  const logs = await container.logs();
  await container.remove();

  return {
    exitCode: output.StatusCode,
    stdout: logs.stdout,
    stderr: logs.stderr,
  };
}

Suppose an agent is extracting text from PDFs and structuring it. One run produces an infinite loop. In the sandbox, it times out and returns an error. The main application keeps running. The sandbox absorbed the failure, and the only cost was a few seconds of compute.

Log Everything, Especially Failures

You can't fix what you can't see. Every agent interaction should produce structured logs that capture the input, the output, the decisions made, and any errors encountered. This isn't just for debugging. It's for understanding how your agent behaves in production so you can improve its safety over time.

// Structured agent logging
interface AgentLog {
  agentId: string;
  sessionId: string;
  timestamp: Date;
  input: {
    query: string;
    context: Record<string, unknown>;
    tools: string[];
  };
  output: {
    response: string;
    toolsCalled: Array<{
      toolName: string;
      args: Record<string, unknown>;
      result: unknown;
    }>;
  };
  safety: {
    validationPassed: boolean;
    sandboxUsed: boolean;
    humanApprovalRequired: boolean;
    approvalStatus?: 'approved' | 'rejected';
  };
  errors?: Array<{
    type: string;
    message: string;
    stack?: string;
  }>;
}

I send these logs to Sentry for error tracking and to a dedicated analytics database for pattern analysis. When something goes wrong, I can replay the exact sequence of events that led to the failure. When something goes right, I can extract the patterns that made it work.

Reliability Is a Design Choice

The teams that ship reliable AI agents don't get lucky. They make deliberate architectural decisions that constrain what the agent can do, validate what it produces, and contain the damage when something goes wrong.

If your team is wrestling with agent reliability and shipping slower because of it, that's the kind of thing I help with. Happy to compare notes.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.

Production AI Scoring: Processing 10,000+ Job Listings Daily with GPT-4

Abdul Rehman — Sat, 11 Jul 2026 09:01:59 +0000

I spent months building an AI scoring pipeline that processes over 10,000 job listings every day. The first version was a mess. Slow, expensive, and unreliable. The second version worked. Here's what I learned about the architecture decisions that actually matter when you put LLMs in a production data pipeline.

Most tutorials show you how to call an API and get a response. They don't show you what happens when you need to do that 10,000 times a day without burning your budget or losing accuracy. I had to figure that out the hard way.

The Problem With Naive AI Scoring

My first approach was embarrassingly simple. Take a job listing, dump the full text into a GPT prompt, and ask for a relevance score. It worked on three test listings. It fell apart at 500.

Three problems emerged immediately.

First, latency. Each call took 3 to 8 seconds. At 10,000 listings, that's 8 to 22 hours of sequential processing. Even with parallel batching, the wall clock time was unacceptable for a system that needed to surface fresh listings within minutes.

Second, cost. GPT-4 token counts were all over the place. Some job descriptions were 200 words. Others were 2,000. I was paying for the long ones the same way as the short ones, with no control over how much context the model consumed.

Third, inconsistency. The same listing scored differently depending on how I phrased the prompt. Minor wording changes produced wildly different relevance scores. That's fine for a prototype. It's a disaster for a production system where recruiters depend on consistent ranking.

I needed structured output, predictable token usage, and a pipeline that could scale horizontally without breaking the bank.

Why Function Calling Changed the Game

GPT-4 function calling gave me a way to enforce structure on the output. Instead of asking for a score and hoping the model returned a number, I defined a schema that the model had to follow.

const scoringSchema = {
  name: "score_job_listing",
  description: "Score a job listing for relevance to a candidate profile",
  parameters: {
    type: "object",
    properties: {
      relevance_score: {
        type: "number",
        description: "Relevance score from 0 to 100",
        minimum: 0,
        maximum: 100
      },
      match_reasons: {
        type: "array",
        items: { type: "string" },
        description: "Top 3 reasons for this score"
      },
      skill_match_count: {
        type: "integer",
        description: "Number of candidate skills found in the listing"
      },
      seniority_level: {
        type: "string",
        enum: ["junior", "mid", "senior", "lead", "executive"]
      }
    },
    required: ["relevance_score", "match_reasons", "skill_match_count", "seniority_level"]
  }
};

This forced the model to return a predictable JSON object every time. No more parsing free text. No more guessing whether a score was 75 or 0.75. The schema acted as a contract between the pipeline and the LLM.

The real win was downstream. With structured output, I could store scores directly in PostgreSQL, build SQL queries that filtered by score thresholds, and feed the results into a REST API without any transformation layer. The pipeline became predictable.

The Architecture That Survived Production

The final architecture had four stages, each with a specific job.

Stage one was ingestion. Listings arrived from multiple ATS sources via their public APIs. I normalized them into a common schema: title, description, company, location, skills. No LLM calls at this stage. Just data cleaning and deduplication.

Stage two was pre-filtering. Before any AI call, I ran a fast keyword and rule-based filter. Listings that clearly didn't match the candidate's criteria were dropped without ever touching the LLM. This eliminated about 40% of listings and saved a significant chunk of API costs.

Stage three was the scoring pipeline. This ran in batches of 50 listings per worker, with each batch sent to GPT-4 via function calling. I used a simple queue system with retry logic. Failed calls were retried up to three times with exponential backoff. Listings that failed all retries were logged and sent to a dead-letter queue for manual review.

Stage four was delivery. Scored listings were written to the database with their relevance score and match reasons. The REST API served them to the frontend and to external integrations. The entire pipeline from ingestion to API availability was under 5 minutes for most listings.

Cost Control: The Real Engineering Challenge

The biggest mistake I see teams make is treating LLM calls like regular API calls. They're not. A single GPT-4 call can cost ten times more than another depending on input length and output structure.

I implemented three cost controls that made the difference between viable and unviable.

First, I used a shorter model for pre-filtering. Before sending a listing to GPT-4 for scoring, I ran a lightweight GPT-4o-mini call that classified listings into three buckets: likely match, possible match, and clear miss. Only the first two buckets went to GPT-4. This cut GPT-4 usage by about 60%.

Second, I optimized prompt length aggressively. Every word in the system prompt cost money on every call. I trimmed candidate profiles to the essential signals: skills, experience level, preferred locations. No biography, no career narrative, no fluff.

Third, I cached aggressively. Listings that were scored once and hadn't changed were never rescored. The same candidate profile scored the same listing only once. This sounds obvious, but I've seen production systems that rescored everything on every pipeline run because nobody added a cache check.

Total cost per listing landed well under one cent. At 10,000 listings daily, that's a manageable operational expense. Without these controls, it would have been five to ten times higher.

What I'd Do Differently

If I were building this today, I'd make three changes.

I'd use a batch processing strategy from day one instead of the streaming approach I started with. The OpenAI Batch API offers 50% cost reduction for asynchronous workloads. My pipeline was mostly batch-friendly, but I designed it for real-time processing first and had to retrofit.

I'd invest more in the pre-filtering stage. The keyword and rule-based filter was effective, but a lightweight embedding similarity search would have caught more false positives before they reached the LLM. I added this later, but it should have been in the initial design.

I'd build better observability into the scoring pipeline. I had basic logging, but I didn't track score drift over time. I only noticed that scores were trending downward after a prompt change when a client complained. A simple dashboard tracking average scores per day would have caught this immediately.

Production AI pipelines have more in common with ETL systems than with chat applications. The same engineering practices that make data pipelines reliable apply here: idempotency, retry logic, dead-letter queues, monitoring, and cost tracking. The LLM is just one component in a larger system.

If your team is building this kind of pipeline and wondering how to keep accuracy high while managing costs at scale, that's the kind of thing I help with. Happy to compare notes on what's worked and what hasn't.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.