Ask five vendors how to get your data ready for an AI agent, and you'll get five different answers, and every one of them will happen to be whatever that vendor sells. The graph-database company says knowledge graph. The vector-database company says RAG. The semantic-layer company says semantic model. None of them is lying, exactly. Each one is describing the one tool in their own box.
The honest answer is that there is no single best way to structure data for an AI agent. Five real approaches are in production use today: semantic models, ontologies and knowledge graphs, retrieval-augmented generation (RAG), plain markdown files searched with grep, and open knowledge formats. Each one solves a different failure mode. Pick the wrong one for your question, and the agent doesn't fail loudly. It fails quietly, returning a fluent, confident, wrong answer, because the model was never the bottleneck. The data it was handed was.
This guide walks through all five, with the evidence for when each one works, so you can match the structure to the question instead of the vendor pitch.
What "structuring data for AI agents" means
Structuring data for an AI agent means organizing information so the agent can find the right fact, definition, or document reliably. The alternative is guessing from raw, unlabeled data and getting it wrong in a way that sounds right. Enterprise AI agents don't fail because the underlying model is weak. They fail because nobody told the agent what "active customer" means, or where the answer to a specific question lives.
This is a data problem before it's a prompting problem. A model handed a clean, governed definition of a metric will use it. A model handed a pile of raw tables and no context will invent a plausible-sounding calculation instead, and you won't find out until someone checks the number in a meeting.
The five ways to structure data for an AI agent
1. Semantic models
A semantic model (or semantic layer) defines your business metrics and entities once: "monthly recurring revenue," "active customer," "churn rate," with the exact tables, joins, and filters behind each one. The agent queries the metric by name instead of writing SQL against raw tables from scratch.
This is the highest-leverage fix for a specific, common failure: an agent guessing at a business definition and getting a number that sounds right but isn't. In dbt Labs' 2026 benchmark, agents querying a semantic model hit 98.2%-100% accuracy on a set of business questions, against 84.1%-90.0% for the same models writing raw text-to-SQL against the full schema. The same benchmark also compared today's models to a 2023-era model doing raw text-to-SQL, and found accuracy nearly doubled over that 2.5-year gap, from 32.7% to 64.5%. That's real model progress. But even the best 2026 model still trailed the semantic-model number. That's the more direct evidence: the gap between a wrong query and a right one comes from what the agent is grounded against, not from how capable the underlying model is.
Semantic models are the right call when the question is a recurring business metric that many tools and many agents will ask about repeatedly, where "everyone gets the same number" matters. They're the wrong call for a one-off question about a single document, or for connecting facts across entities the model was never designed to relate.
A semantic model also earns its keep in a way the other four approaches don't automatically match: define the metric once, and any AI agent can query it through MCP (the Model Context Protocol, an open standard for connecting agents to data sources), instead of every team building its own bespoke retrieval or graph layer to answer the same question. That turns the semantic model from a one-off project into shared infrastructure: a governed layer other agents can read from directly, rather than a definition locked inside one team's dashboard.
You don't need a specific vendor to get this. dbt and Cube will both get you a semantic layer. What matters more than which tool you pick is treating the metric definition as the thing worth getting right once.
2. Ontologies and knowledge graphs
A knowledge graph stores entities (people, accounts, products, contracts) and the explicit relationships between them, instead of flat tables or blocks of text. An ontology is the schema that defines what those entities and relationships are allowed to mean, so the graph stays consistent as it grows. Together, they let an agent traverse connections that a keyword or embedding match would miss entirely, such as "which accounts does this contact influence, and which of those accounts churned within six months of a support escalation?"
This is where GraphRAG earns its name: retrieval that queries a knowledge graph, plus generated summaries of clusters of related entities, instead of a flat vector index. Microsoft's 2024 research found GraphRAG held a 72% win rate on comprehensiveness and a 62% win rate on diversity over plain vector RAG. Its top-level summaries also needed over 97% fewer tokens than summarizing the source text directly. A separate, independent 2025 systematic evaluation of RAG versus GraphRAG found that adding selection and integration strategies on top of graph retrieval improved the best baseline by 1.1% to 6.4% on a multi-hop reasoning benchmark. That's a real but more modest gain than the headline GraphRAG numbers alone suggest, and a useful reminder that the retrieval technique matters as much as the graph itself.
Knowledge graphs cost more to build and maintain than the other four approaches. Someone has to model the entities and relationships, and keep the ontology honest as the business changes. That cost is worth paying when the questions genuinely require multi-hop reasoning across connected entities. It's overkill for a question that boils down to "look up this one fact."
3. RAG (retrieval-augmented generation)
RAG retrieves the most relevant chunks of text, usually by embedding similarity in a vector database, and hands them to the model as context before it answers. It's the fastest approach to stand up: chunk your documents, embed them, index them, and you have a working system in an afternoon.
RAG is the right default for fact lookup across a large, loosely structured document set: contracts, support tickets, product docs, meeting notes. It's the wrong tool when the answer requires connecting several facts across different documents (that's the knowledge-graph case above), or when the "fact" is a governed metric that should have one certified answer everywhere (that's the semantic-model case).
There's also a subtler trap: more retrieved context is not automatically better context. 2025 research from Chroma, testing 18 frontier models including GPT-4.1, Claude Opus 4, and Gemini 2.5, found that accuracy degrades non-uniformly as input length grows. Even a single distractor passage measurably hurt performance relative to a cleaner baseline. Models sometimes did worse on logically well-structured long context than on the same information shuffled. The finding has been corroborated widely enough that it's a fair rule of thumb: precision in what you retrieve beats volume, every time.
Some teams get real gains combining approaches: retrieving with a vector index, then using graph relationships to check or expand the answer. AWS's 2024 case study with Lettria reported up to 35% improvement in answer precision when graph-based retrieval was layered on top of vector RAG, evaluated across finance, healthcare, aeronautics, and legal use cases.
4. Plain markdown + grep
The least glamorous option, and the one no infrastructure vendor will ever recommend to you, because none of them sell it: put your data in plain markdown files, organized sensibly, and let the agent search them with grep or another full-text tool. No embeddings, no vector database, no graph to maintain.
This works better than it sounds for a small, well-organized internal corpus: think hundreds of files, not millions. Coding agents already do this constantly. A well-structured repository of markdown docs, read directly by an agent with a file-search tool, is often faster to build and easier to reason about than standing up retrieval infrastructure for a corpus that would fit on a laptop. It's the same instinct behind emerging patterns like llms.txt and skill-file conventions: plain, greppable text, organized by a human who knows the content, beats a half-tuned vector index over the same material.
There's a maintenance upside here that's easy to undervalue. Markdown files live in version control like everything else you write. Every change is a diff a human can read, review, and revert, with no embedding pipeline to rerun, no index to rebuild, no vector database to keep in sync with the source of truth.
The approach breaks down exactly where you'd expect: once the corpus gets large enough, or the phrasing gap between a query and the right document gets wide enough, that keyword matching starts missing things a semantic or graph-based search would catch. That's the signal to graduate to RAG or a knowledge graph, not a reason to skip markdown plus grep as a legitimate starting point.
5. Open knowledge formats
Open knowledge formats, such as schema.org, JSON-LD (JSON for Linking Data), and RDF (Resource Description Framework), structure data using a shared, public vocabulary instead of a format only your own systems understand. Where the first four approaches structure data for your agents to read, open formats matter when the data needs to be readable by agents you don't control: a customer's AI assistant checking your product's pricing, a search engine's AI Overview summarizing your documentation, or a partner's agent pulling structured facts from your public site.
This is a different job from the other four, and it's easy to conflate them. A semantic model, knowledge graph, or RAG index is internal infrastructure serving your own agents. Open formats are a publishing choice, closer to how you already mark up a webpage for search engines, except the audience reading that markup increasingly includes an AI agent instead of only a crawler. If your data only needs to serve your own team's agents, this is the one approach on the list you can reasonably skip.
The five approaches at a glance
| Approach | Best for | Setup cost | Maintenance | Sharpest weakness |
|---|---|---|---|---|
| Semantic model | Governed business metrics, repeated questions | Medium: model the metrics once | Medium: update as metrics evolve | Doesn't help with unstructured documents |
| Ontology / knowledge graph | Multi-hop reasoning across connected entities | High: model entities and relationships | High: keep the ontology honest | Expensive for simple lookup questions |
| RAG (vector search) | Fact lookup across a large, loosely structured corpus | Low: chunk, embed, index | Low to medium: re-index as content changes | More context isn't automatically more accurate |
| Plain markdown + grep | Small, well-organized internal corpus | Very low: organize files sensibly | Low: no infrastructure to run | Breaks down as the corpus grows or gets ambiguous |
| Open formats (schema.org, JSON-LD) | Data read by agents you don't control | Low to medium: markup your public content | Low: mostly set-and-forget | Doesn't structure data for your own internal agents |
Which one should you use?
Match the structure to the question, not the vendor pitch. In practice, that means asking what kind of question the agent needs to answer:
- "What's our [metric]?" a recurring, governed business number. Use a semantic model.
- "How does X connect to Y, and what does that chain imply?" multi-hop reasoning across entities. Use an ontology or knowledge graph, optionally with GraphRAG-style retrieval.
- "What does this document say about...?" fact lookup across a large corpus you didn't write by hand. Use RAG.
- "What's our process for...?" a corpus small enough that you know roughly where the answer lives. Plain markdown plus grep is often faster to build and equally reliable.
- "Can an agent I don't control read this?" a partner's, customer's, or a search engine's AI. Use open formats. Most production systems combine two or three of these rather than picking exactly one: a semantic model for metrics, RAG for contracts and support tickets, and grep-searchable internal docs for team runbooks, all read by the same agent. The mistake isn't picking one approach. It's assuming one approach has to cover every question.
How this plays out in practice
Take a Revenue Operations team building one agent to help with renewals. In a single conversation, that agent might field five different kinds of questions, and each one wants a different structure behind it.
"What's this account's net revenue retention?" is a governed metric with one correct calculation. The agent should query a semantic model, not write its own SQL against raw subscription tables and hope it matches finance's number.
"Who at Acme Corp reports to whom, and which of them also sits on the renewal decision?" is a multi-hop question across people, roles, and accounts. That's a knowledge graph question. A flat table or a text search won't trace the reporting chain.
"What did the contract say about the renewal notice period?" is one fact, buried in one document, out of thousands of contracts. RAG retrieves the relevant clause without anyone having to read every file.
"What's our internal playbook for a renewal at risk?" points to a few dozen well-organized runbook documents the team wrote and knows by heart. Plain markdown plus grep finds the right one in milliseconds, with zero infrastructure to maintain.
"Does our public pricing page make sense to the customer's own AI assistant?" is an agent-you-don't-control question. That's the open-format case.
No single approach on this list answers all five. That's the real argument for treating "how do I structure my data for AI agents" as a portfolio decision, not a one-time architecture choice.
The migration path tends to run in one direction. Teams start with markdown plus grep because it's the fastest thing that works. They move pieces of it to RAG once the corpus outgrows keyword search. They add a knowledge graph only once they hit a specific class of question, usually a multi-hop one, that RAG keeps getting wrong. Few teams start with a knowledge graph on day one, and that's a reasonable default: build the structure your current questions require, and add more only when a real question exposes the gap.
Context engineering vs. prompt engineering
Everything above is an exercise in what's now being called context engineering: deciding what information reaches the model's context window, in what form, and how much of it. That's distinct from prompt engineering, which is how you phrase the instructions in a single message. The distinction matters because a perfectly worded prompt can't fix a model that was handed the wrong facts, or too many of them.
However you structure your data (semantic model, knowledge graph, RAG, or a markdown file) the same discipline applies. Give the agent exactly the facts it needs to answer the question, not everything you have.
Conclusion
Five real approaches, five different jobs. A semantic model for governed metrics. A knowledge graph for multi-hop reasoning. RAG for fact lookup across a big corpus. Plain markdown and grep for a small one. Open formats for the agents you don't control. The vendors selling each piece will tell you theirs is the answer. It usually isn't, not on its own. Start from the question your agent needs to answer, and the right structure follows from that, not from whichever infrastructure happens to be easiest to sell you.
This article was originally published on the RevOS blog.
Top comments (0)