What a year of building an enterprise AI copilot actually taught me
When I started, the goal sounded simple: give employees one place to ask a question and get an answer. No more hunting through a dozen internal apps to find a leave policy, check a project allocation, or raise a request. One assistant, one box, one answer.
It took me the better part of a year, several architectures I'm now slightly embarrassed by, and a long list of failures that each taught me something specific. This is the honest version of that journey — the walls I hit, what I learned at each one, and the rearchitecture that finally made everything click.
(Names, systems, and numbers here are anonymized. This is about the engineering, not the company.)
The problem I set out to solve
The organization had roughly 6,000 employees across about ten countries, and their digital workday was fragmented. Policies lived in one place, HR data in another, tickets in a third, learning records in a fourth. Every routine question meant knowing which system to open and how to phrase things there.
The vision was a single AI assistant embedded in the employee platform: ask in plain language, and it retrieves the right knowledge, reads your live data, and — eventually — takes actions on your behalf. That's easy to say on a slide. The gap between the slide and a system that actually works is where all the learning lives.
Where I started — and the first wall
My first version was the obvious one. I had country-specific policy content indexed for search — thirteen separate vector indexes, one per country. Behind the assistant sat a workflow that received the question, looped through the relevant indexes, called a search agent, then called a language model to generate the answer. Every branch was wired by hand: knowledge path, live-data path, default path.
It worked. It also took about fifteen seconds per answer.
Fifteen seconds is an eternity for someone who just wants to know how many casual leaves they have left. Worse, the whole thing was sequential: an HTTP call, then a loop over indexes, then an agent call, then a model call, each waiting on the last. The workflow engine was doing orchestration work it was never designed for.
Lesson zero: if your response time embarrasses you in a demo, the architecture is telling you something. Don't tune it — question it.
Learning #1: A wrapped search is not a good search
My instinct was to consolidate: take those thirteen indexes and hang them all off one intelligent agent, letting the model pick which to query based on the question and the user's location. Cleaner, in theory.
Then I noticed something that stopped me. When I queried the search index directly, the results were sharp. When I routed the same question through an agent that wrapped the search, the answers got noticeably worse.
The reason turned out to be fundamental, and it's a trap a lot of people fall into. When you query search directly, you control the exact query text and exactly which retrieved chunks reach the model. When an agent sits in front of the search, it silently rewrites your question into its own search query first — and that rewrite loses nuance. "How many casual leaves can I take in [country]?" becomes a generic "casual leave policy," and the country-specific content never surfaces. On top of that, the agent's default retrieval settings rarely match the ones you carefully tuned for direct queries.
Lesson: an abstraction that "handles retrieval for you" is also an abstraction that quietly degrades it. I collapsed the thirteen country indexes into one federated index with a country filter — the total document count was small enough that a single filtered query was both faster and more accurate than looping over thirteen. That one change killed the index loop entirely and reclaimed a big chunk of those fifteen seconds.
Learning #2: Sequential is the enemy — and parallelism has its own traps
With retrieval fixed, the next bottleneck was structural. My workflow had around twenty intent branches — projects, skills, leave, allocations, and so on — evaluated one after another. So I parallelized them: fan every branch out from a single intent-detection step and let them run at once.
This is where I learned that concurrency in a low-code workflow engine has sharp edges nobody warns you about:
- Shared variables aren't concurrency-safe. My branches each appended their result to one shared string variable. Under parallel execution, writes stepped on each other and results came back garbled. The fix was to stop using a mutable shared variable inside parallel branches entirely and give each branch its own scoped output object, then aggregate once at the end with a single atomic write.
- A variable cannot reference itself in its own assignment. I had a fallback that set a variable to a value derived from that same variable — and the platform rejected it outright with a "self-reference is not supported" error. The fix was to wrap the assignment in a condition: only overwrite the variable when the upstream step actually succeeded; otherwise leave it holding whatever it already had.
- Every action name must be globally unique, even across nested scopes. Duplicate an action into a second path and forget to rename it, and the whole definition fails to save.
- Branch containers have hard caps. The switch construct I was leaning on maxes out at twenty-five cases. The moment my tool set grew past that, I had to nest switches inside switches — which is exactly as fun to hand-maintain as it sounds. (More on how I stopped hand-maintaining it below.)
Lesson: parallelism buys you latency but charges you correctness. Every shortcut that was fine in a sequential flow — shared state, casual naming, one big switch — becomes a bug the instant things run at the same time.
Learning #3: Routing is a first-class problem, not an afterthought
Once the assistant could answer fast, it started answering wrong — not factually wrong, but routed to the wrong place. A question like "who is the admin for the [city] office?" got sent to a live-data lookup instead of the knowledge base. "How do I apply for leave?" got answered from policy documents instead of the how-to system manual, because both mention the word "leave."
I'd been routing on keywords. Keywords can't tell the difference between asking about a topic and asking how to do something in that topic.
Two changes fixed this:
First, intent-first routing. Before matching any topic, classify what the user actually wants — a how-to, a policy lookup, a data read, an action. Classify intent, then pick the source. Ambiguous words like "leave" stop being landmines once you've already decided the user wants instructions rather than policy.
Second, a hybrid answer pattern instead of a single winner-take-all route. The knowledge base runs as a default for every question. A specialist path runs only when the router judges it relevant. Then a final model scores both outputs and either picks the stronger one or merges them when both clear a relevance bar. I also added a dedicated query-rewriting step before every search so the query hitting the index was optimized rather than raw.
Lesson: don't make routing an all-or-nothing bet on one classifier. Let a cheap default always run, let specialists run when warranted, and let a synthesizer decide at the end. It's more forgiving of the router being occasionally wrong — which it will be.
Learning #4: Identity is the hard part nobody puts on the roadmap
This one cost me more time than any clever routing logic. The assistant doesn't just need to know what you asked — it needs to know who you are across every system, and every system identifies you differently.
The core HR system keyed people one way. The skills and project systems used a different internal integer ID. The engagement platform used yet another. Passing the wrong identifier to the wrong API didn't throw a clean error — it silently returned someone else's data, or nothing, or results from years ago.
I learned to treat identity as its own resolution step. Resolve the user once, up front, using a universal lookup key (email worked as the fallback that every system understood), map that to each system's internal ID, and then inject the correct ID into every downstream call. I also established one firm rule: there is exactly one canonical cross-system employee identifier, and local, system-specific IDs never leave their own system — no matter how tempting the parameter name makes it look.
Lesson: in any multi-system integration, identity mapping is not plumbing you can bolt on later. It's foundational. Get it wrong and every feature built on top inherits a silent data-correctness bug.
And a related one that kept biting me: field-name mismatches fail silently. An API expects emp_id, you send employee_id, and instead of an error you get an empty result you'll misread as "no data." I started defending against this with fallback chains over the likely field names — but the real fix was always to stop guessing and verify the actual field names from run history.
Learning #5: When the definition has 54 tools, stop editing it by hand
By this point the assistant had grown from a handful of branches into a genuine orchestrator exposing dozens of tools — read-only lookups and read-write actions merged into a single definition with more than fifty tools and nested switches several layers deep.
You cannot hand-edit a file like that reliably. One duplicated action name, one branch that points to a step that no longer exists, one switch that quietly grew past its case limit, and the whole thing refuses to deploy — usually with an error that points nowhere near the actual cause.
So I stopped writing the definition by hand and started generating it with a script that validated as it built. Every build ran the same checks before producing output: no duplicate action names anywhere in the tree, no branch referencing a non-existent predecessor, no switch over its case cap, no variable referencing itself. Structural errors got caught at build time, at my desk, instead of at deploy time in front of everyone.
Lesson: past a certain complexity, your configuration is code, and it deserves the same treatment — generated, validated, and tested before it ships. Catching a structural error in a build script is cheap. Catching it in production is not.
The rearchitecture that finally clicked
Every fix above made the system better, but they were all improvements to the same fundamental shape: a big hand-orchestrated workflow with an intermediate agent standing between the user and the tools. That intermediate layer was expensive — it consumed tokens on every single turn, it could only really handle one question at a time, and it leaned on data that was up to a day stale because it read from a batch-loaded store.
The unlock was adopting the Model Context Protocol (MCP) and letting the model talk to the tools directly through a standard interface, removing the intermediate orchestrator entirely. Three things fell out of that at once:
- Multi-question support. A user could now ask "what's my leave policy and my current balance?" in one message and get both, because the model could invoke multiple tools in a single turn instead of being funneled through a single-intent pipeline.
- Lower cost. Deleting the intermediate agent deleted its per-turn token overhead.
- Real-time data. Direct tool calls hit live sources instead of a day-old snapshot.
That was the moment the thing stopped feeling like a chatbot bolted onto some search indexes and started feeling like an assistant that could actually reason about which tools to use and combine their results.
Where it landed
The assistant went from a fifteen-second single-answer bot to something most of the workforce uses, resolving a large share of routine queries on its own, in a fraction of the original response time, across every country the organization operates in. It reads live data, answers policy questions grounded in the right regional content, and is moving steadily toward safely taking actions on the user's behalf.
The number I'm proudest of isn't a latency figure. It's that the architecture is now boring in the right way — a clean tool interface, generated and validated configuration, identity resolved once and correctly, routing that fails gracefully. Boring, at this scale, is the achievement.
The lessons that generalize
If you're building something similar, here's what I wish I'd known at the start, in the order the project taught them to me:
- A slow demo is an architecture problem, not a tuning problem. Question the shape before you optimize the parts.
- Convenience abstractions over retrieval quietly degrade retrieval. Know exactly what query is hitting your index and what reaches your model.
- Parallelism trades latency for correctness. Shared state, loose naming, and one giant branch are all fine until things run concurrently — then they're bugs.
- Route on intent, not keywords, and don't bet everything on one classifier. A cheap always-on default plus a scored synthesizer is far more forgiving.
- Identity mapping across systems is foundational, not plumbing. One canonical ID; local IDs never travel; resolve the user once, up front.
- Silent failures are the expensive ones. Empty results from a field-name mismatch will mislead you longer than any exception.
- Past a certain size, your config is code. Generate it, validate it, and catch structural errors before deploy.
- The right abstraction removes a layer instead of adding one. The biggest win of the whole project came from deleting the orchestrator in the middle, not making it smarter.
The learning curve wasn't a smooth ramp. It was a series of walls, and each wall taught me exactly one thing I couldn't have learned any other way. That's the part worth writing down.
Top comments (0)