The problem: 30 minutes per application, 400 times a day
On paper, broker back-office work sounds easy. A broker sends an email with a few PDFs attached. Someone on the back-office team opens it, reads every document, checks each one against a list of requirements, then writes back saying what's there and what's missing.
One review took about 30 minutes. The team got roughly 400 applications a day.
Do that math and you get skilled people spending most of their shift copying fields out of PDFs. Nobody enjoyed it, and honestly nobody should have to.
At work I built a multi-agent AI system to take over that flow. Review time dropped from 30 minutes to 2 per application (about 93% less), and the team needed fewer people on that manual task. Here's how it's put together, and a few things I'd do the same way again.
Why I didn't use one big prompt
My first thought, like most people's, was one giant prompt. "Here's an email and some PDFs, tell me if this application meets the requirements."
It doesn't hold up. Pulling data out of a document and judging that data against rules are two separate jobs. They break in different ways and they need different context. Cram both into one call and when the answer is wrong, you can't tell which half failed.
So I split the work into agents, built with LangChain and LangGraph on Azure OpenAI (GPT). The document-extraction agent reads the PDF attachments and pulls out what the review needs. The requirement-validation agent takes that output and checks it against the criteria. There are a few more specialized agents too, each owning one small step of the review.
The payoff is mostly in debugging. If a reply comes out wrong, I can look at the extraction output first. Was the data wrong, or was the judgment wrong? That one question saves a lot of time.
The pipeline, end to end
Here's what happens between a broker hitting send and getting a reply.
Intake from Zoho CRM
The pipeline hooks into Zoho CRM. When a new broker email lands, a Zoho webhook tells the pipeline right away. No polling loop sitting there checking every few minutes. The system grabs the application and its PDF attachments.
Reading the documents
Broker documents are a mess, and a lot of them are scans. The extraction agent uses a GPT vision model, plus OCR for the scanned PDFs, to read each file and turn it into structured data the later steps can actually use.
Checking against the real rules (RAG over Pinecone)
The requirement criteria sit in a Pinecone vector database. I chunk the criteria documents before embedding them, so a lookup brings back the specific rules that apply instead of a whole document.
For each application, the validation step retrieves the relevant criteria and hands them to the model along with the extracted data. That changes the question. The model isn't asked "what do you think the requirements are?" It's asked "here are the requirements, does this application meet them?"
I'd call this the part that makes people trust it. Without retrieval, the model is guessing from memory, and it guesses confidently.
Writing the reply
When matching is finished, the system drafts the validation and match-result email back to the user: what matched, what didn't.
How the pieces are wired
Each workflow (extraction, validation and the rest) is built in Langflow. Being able to see and tweak a flow visually was genuinely useful while requirements kept shifting. LangGraph sits on top as the orchestrator. It routes each application through those workflows and keeps state between steps. FastAPI is the service layer around all of it.
The trade-off: you now have two tools to understand instead of one. For us the split was worth it, because Langflow handled the "what does this step do" part and LangGraph handled the "what runs next" part.
Knowing when to stop and ask a human
The biggest design decision on this project had nothing to do with which model to use. It was deciding what the system is not allowed to decide alone.
Every result carries a confidence score. If a case falls below the threshold, or it's an edge case, it goes to a person on the back-office team before anything is sent to the broker. Confident cases move forward on their own.
Why bother? A back-office team will not adopt a tool that quietly sends wrong answers to brokers. Once they know shaky cases come to them, they can stop second-guessing the rest.
Also, some applications are just weird, and they always will be. If I'd tried to automate every last one, the system would have turned brittle. Automating the confident majority and escalating the rest is how you get to 2 minutes on average without risking bad replies.
Results
Review time per application went from 30 minutes to 2, about a 93% cut, across roughly 400 applications a day. The team also needed less manual headcount on the task.
A rough calculation from those two numbers: saving 28 minutes on each of 400 applications comes to about 186 hours of review work a day that nobody has to do by hand anymore.
If you're building something like this
Split the job before you write a single prompt. Extraction and validation really are different problems, and separating them is what made this system debuggable.
Make the model check against retrieved rules, not its own memory. RAG over the actual criteria is what keeps it honest.
Plan the human handoff from the start. I treated it as a core part of the design rather than something to add once things broke, and I think that's the main reason the team accepted the system.
And treat the AI like any other backend code. The agents live inside a FastAPI service with clear interfaces. I used LangSmith for tracing, testing and monitoring, so I could open any agent run, evaluate it, and keep an eye on it in production. When a reply looks off, that trace is the first place I go.
Wrapping up
Getting an LLM to read a PDF was the easy part. The real work was everything around it: CRM intake, small focused agents, validation grounded in retrieved rules, and a clear point where a person takes over. That's what took a 30-minute manual review down to 2 minutes at 400 applications a day.
I'm a Python backend and AI engineer working with FastAPI, Django, PostgreSQL, LangGraph and RAG, and I'm open to mid-level backend and AI engineering roles in India or remote. You can find me on LinkedIn, GitHub, or X.
Top comments (0)