Michael Sommer

Posted on Aug 5 Edited on Aug 12 Originally published at sommercode.gitbook.io

From Model to System

#ai #security #aisecurity

The Foundations of Secure LLM Applications

Anyone looking at an LLM application for the first time usually sees three things: an input field, a model, and an answer. For everyday use, that picture may be good enough. For security, it is unusable.

A real application consists of considerably more: frontend, backend orchestration, system instructions, chat history, data sources, retrieval, embeddings, tools, output processing, logging, and permissions. The language model is prominent within that picture, but it is still only one component.

Week 1 of an AI security learning path must therefore, above all, build a workable picture of the system. Before we talk about prompt injection, data leakage, or tool abuse, we need to understand:

what kind of AI system we're even looking at,
how text is translated into tokens and active context,
what role embeddings and retrieval play,
how an LLM app is built with and without RAG,
and at which trust boundaries data starts to have security-relevant effects.

The most important mental move is this:

Away from the isolated model — toward components, data flows, context sources, and trust boundaries.

ML, Deep Learning, Generative AI, and LLM Are Not the Same Thing

"AI" is convenient as an umbrella term, but from a security standpoint it is often far too blurry. A spam filter, an image generator, and an agentic support assistant can all be called "AI." Their attack surfaces are still fundamentally different.

Figure 1: Machine Learning and Deep Learning primarily describe learning methods; Generative AI describes a generating capability. An LLM typically combines deep learning, generative processing, and language — i.e., tokens.

Machine Learning

Machine Learning (ML) refers to methods that learn patterns from data in order to produce predictions, classifications, or decisions. Not every rule is explicitly programmed.

Typical examples are spam detection, credit risk scoring, anomaly detection, or failure prediction. The output can simply be a class or a score. An ML system doesn't have to generate anything.

From a security standpoint, classic ML systems often raise different questions than chatbots do:

Can manipulated inputs evade classification?
Can training data be poisoned?
Can an attacker extract the model?
How robust is the system against distribution shifts?

Deep Learning

Deep Learning (DL) is a subfield of machine learning. It uses deep neural networks to learn complex patterns from large volumes of data.

Image processing, speech-to-text, and modern language models are frequently based on it. Still, not every ML method is deep learning: decision trees, support vector machines, or random forests also belong to ML.

DL systems come with typical challenges: high data requirements, limited interpretability, sensitivity to distribution shifts, and a particular relevance of adversarial examples.

Generative AI

Generative AI (GenAI) describes systems that create new content: text, images, audio, video, or code.

The term primarily describes the type of task, not automatically a specific architecture. An image generator is GenAI but not an LLM. A classification network can be deep learning without working generatively.

Generative systems bring additional risks:

plausible-sounding but false content,
disclosure of sensitive information,
unsafe model outputs,
prompt injection,
exploitable automation,
dangerous downstream processing of the output.

Large Language Models

A Large Language Model (LLM) is a large model, usually based on deep learning, that processes and generates token sequences. It can continue text, summarize, rephrase, answer questions, or generate code.

An LLM is typically:

a machine learning system,
a deep learning model,
generative,
language- and token-centric,
and in practice, embedded in a larger application.

This last property is decisive for security. A chatbot is not simply "the model." The application supplements the model with rules, context, memory, retrieval, tools, permissions, and a way of using the output.

Term	Main Idea	Typical Output	Example	Typical Security Focus
ML	Learn patterns from data	Class, score, prediction	Spam filter	Evasion, poisoning, model extraction
DL	ML with deep neural networks	Task-dependent	Image recognition	Adversarial examples, robustness
GenAI	Generate new content	Text, image, audio, code	Image generator	Hallucination, unsafe output
LLM	Process language and tokens	Response, text, code	Support assistant	Context manipulation, leakage, tool use

The security payoff of this terminology work is immediate: whoever mis-describes the system type also models the wrong attack surface.

Tokens and Context: What the Model Actually Processes Right Now

Humans read words and sentences. A language model processes a sequence of tokens.

A token is not automatically a word. Words can split into several tokens; punctuation, whitespace, numbers, code fragments, URLs, email addresses, or emojis can form their own, sometimes unusually split, token sequences. That's why visible text length and technical token length are not the same thing.

This isn't merely an implementation detail. Tokenization affects:

how much content fits into one model call,
how encodings and special characters are handled,
whether filters and the model "see" the same structure,
and how strongly different content competes for space and attention.

The Context Window Is the Active Working Space

The context window refers to the limited set of tokens the model can take into account in a single pass. It can simultaneously contain:

system and developer instructions,
user input,
chat history,
retrieved document chunks,
tool results,
intermediate steps,
and already-generated parts of the output.

Figure 2: The context window is a limited, operational working space. Instructions, user data, RAG content, and tool results share the same capacity and compete for effect.

A useful mental model distinguishes three layers:

Model weights: statistical patterns learned during training.
Active context: information and instructions processed in the current pass.
External storage: databases, documents, or chat memory whose content only reaches the context once the application loads it.

This makes four important distinctions visible:

Training is not context. A learned pattern is not the same as currently supplied information.
Storage is not visibility. A file in the system is invisible to the model until its content lands in the request.
Visibility is not priority. Content can sit in the context and still be overridden or crowded out.
Data is not instructions. The application wants to keep these categories separate; the model, however, does not enforce this boundary the way a formal policy engine would.

An LLM Doesn't "Know" Like a Human

An LLM has no human memory, no guaranteed fact retrieval, and no guarantee of consistent understanding. It works with learned patterns, current context, and probabilities for the next token.

Because of that, it can:

sound very convincing and still be wrong,
reproduce information correctly, as long as it's in the context,
appear to "forget" the moment it's no longer available,
react differently given contradictory context.

The statement "the model knows that" is therefore usually too blunt. Better questions are:

Does this information come from trained patterns?
Is it visible in the current context?
What source does it come from?
Is it allowed to be visible to this user?
What competing instructions sit next to it?

Context Limits Create Security Consequences

A finite context window leads to three practical problems:

Crowding out: Long inputs, large retrieval volumes, or tool outputs can push other content out of the usable context.

Fragmentation: Rules and information that belong together can end up far apart, or arrive only partially.

Prioritization: Visibility alone does not guarantee that content is reliably followed.

More context is therefore not automatically better. More content can also mean more noise, more conflicts, higher cost, and additional attack surface. A safeguard that exists only as a single sentence somewhere in the prompt is not a reliable security mechanism.

Embeddings: Similarity in Numbers, Not Truth

An LLM generates language. An embedding model performs a different task.

An embedding is the numerical representation of content as a vector in a high-dimensional space. Semantically similar content is meant to sit closer together in it than dissimilar content, in most cases.

This lets a search query like "How do I reset my password?" find documents that talk about "credential reset" or "account recovery," even though the words aren't identical.

Embeddings are therefore valuable for:

semantic search,
retrieval,
clustering,
duplicate and similarity analysis,
ranking and recommendations.

What an Embedding Does Not Tell You

An embedding is not:

proof of truth,
a trust signal,
an authority marker,
a security judgment,
proof of permission.

Semantic closeness only means that a piece of content appears similar according to the model and search method used. A hit can still be outdated, wrong, manipulated, sensitive, over-privileged, or unsuitable for the specific task.

A retriever therefore does not say: "This is the right and safe source." It says, rather: "These are probably similar candidates."

RAG Supplies Context — Not New Human Knowledge

Retrieval-Augmented Generation (RAG) connects search and generation. Documents are prepared, matching excerpts are found, and then handed to an LLM as additional context.

Figure 3: Embeddings support candidate search. Only the context builder actually brings selected chunks into the model context; the LLM then generates a response.

A typical RAG flow consists of two paths.

The Preparation Path

Document sources are ingested.
Content is extracted and cleaned.
Documents are split into smaller chunks.
An embedding model turns the chunks into vectors.
A vector store, or index, saves these representations.

The Query Path

The user asks a question.
The question is embedded as well.
The search identifies semantically similar chunks.
Filtering and ranking select candidates.
The context builder combines instructions, question, and chunks.
The LLM generates the response on this basis.

This makes an important architectural separation possible:

Building Block	Task	Result
Embedding model	Represent content as a vector	Numerical representation
Vector store	Index representations	Searchable collection
Retriever	Find similar candidates	Selected chunks
Context builder	Assemble the context	Model request
LLM	Process and generate language	Response or text
RAG pipeline	Connect retrieval and generation	Response with added context

If an answer is wrong or unsafe, the diagnosis "the model failed" is not enough. It's possible that:

the wrong document was indexed,
chunking was done clumsily,
an unsuitable hit was ranked too high,
a permission filter wasn't applied,
too much context was inserted,
or a correct context was simply used incorrectly by the model.

Chunking Is Security Design Too

RAG frequently doesn't process complete documents, but chunks: paragraphs, page fragments, FAQ blocks, or feature descriptions.

The chunking strategy influences:

which pieces of information are retrieved together,
whether sensitive and harmless content end up mixed,
whether instructions appear without their explaining context,
how well manipulative text is found,
and how much material occupies the context window.

Chunking is therefore not merely an optimization question. It also decides which content keeps its original meaning or trust boundary — and which loses it.

The LLM Application as a System

A simple LLM app without RAG can consist of user, frontend, backend, prompt assembly, model, and output handling.

The typical flow is:

The user sends an input.
The backend combines it with system instructions and history.
The LLM processes the assembled context.
The response goes back to the application.
The frontend displays it, or a downstream system processes it further.

Even without RAG, the application can contain role logic, memory, moderation, structured output, tool calls, or policy checks. "Without RAG" therefore does not mean "simple."

With RAG, additional components and data flows are added:

document sources,
ingestion and cleaning,
chunking,
embedding model,
vector store,
retriever,
metadata and permission filters,
context builder.

That grows not just the system's capability, but its attack surface as well.

Feature	Without RAG	With RAG
Context sources	System prompt, user input, history	Plus documents and search results
Retrieval pipeline	No	Yes
Embeddings and vector store	Usually not needed	Typically present
External knowledge base	Not directly	Actively pulled into context
Additional risks	Prompting, output, tools	Plus retrieval, chunking, leakage, context poisoning

The decisive insight:

An LLM app is not a model with a user interface, but a system of components, data flows, and context pipelines.

Trust Boundaries: Where Trust Must Be Re-Evaluated

A trust boundary is a boundary at which data, commands, or decisions move from one trust context into another.

At this boundary, the system must never simply assume that incoming content is harmless, correct, current, authorized, or safe to interpret.

Trust boundaries are well known from web, API, and cloud security. In LLM systems, they become especially tricky because so many different things share the same form: text.

Text can simultaneously be:

user input,
a data fragment,
a system rule,
document content,
a tool result,
a model response,
or the basis for a real action.

The trust boundary can therefore run right through the middle of the model context.

Figure 4: Every handoff into a new trust or effect context requires renewed scrutiny. The context builder and the transition from model output to privileged actions are especially critical.

The Most Important Boundaries

User → application: User input is untrusted. It can contain legitimate requests, manipulation, data junk, or attacks on downstream components.

Backend → model context: The orchestrator merges system rules, user input, history, RAG content, and tool output. Different trust levels thereby land in a shared decision space.

Document source → ingestion and retrieval: An internal document is not automatically correct, current, harmless, or cleared for every user.

Retriever → model context: Stored content becomes active context. At this boundary, permission, tenant, sensitivity, and relevance must be checked.

Tool output → model context: API responses, files, and search results are further input. They can contain wrong data, secrets, or hidden instructions.

Model output → user: The response is not automatically true, verified, or action-guiding.

Model output → tool or workflow: Text can become code, an email, a database query, or another privileged action. This boundary can be more security-critical than the chat window itself.

A Practical Trust Model

Component or Flow	Baseline Rating	Why
User input	untrusted	can be manipulated arbitrarily
System instructions	higher trust	set internally, but not infallible
Internal knowledge base	conditionally trusted	possibly sensitive, outdated, or manipulated
Tool output	conditionally trusted	technically generated, but not automatically safe
Model output	untrusted for critical effect	can be wrong or misused

The goal is not to reject everything across the board. The goal is to deliberately check trust at every boundary and to limit the effect that's permitted.

Case Example: An Internal Security Assistant

Let's take an internal assistant for security policies, password resets, and VPN usage.

A weak description would be:

"The bot looks at our documents and answers questions."

A resilient system description reads:

The system is an internal, RAG-based LLM assistant. User requests reach a backend orchestrator through a frontend. This orchestrator combines system instructions, chat history, and retriever-selected document chunks into a bounded model context. The knowledge base is built up via ingestion, chunking, embeddings, and a vector store. The LLM generates a response that the backend processes and displays to the user. Key trust boundaries lie between user and application, document sources and retrieval, retriever and model context, and model output and further use.

This description alone already exposes the key high-level risks:

manipulated or overly broad user input,
access to disallowed documents,
semantically similar but wrong or sensitive chunks,
displaced safety instructions,
disclosure of internal data,
overly trusting handling of the model response,
abusive tool or workflow actions.

The decisive point: the same model API can create a completely different security posture within a different architecture.

The Analysis Framework for Every LLM Application

A first system analysis should cover six areas.

1. System Type

Is it a simple chatbot, a RAG assistant, a tool-using agent, a Q&A system, or an internal document assistant?

2. Main Components

What building blocks exist: user, frontend, backend, prompt builder, LLM, knowledge base, retriever, vector store, tools, and output consumer?

3. Data Flow

How does a request move through the system? What intermediate steps, storage, and external calls exist?

4. Model Context

What content actually reaches the model? This includes not just user input and system prompt, but also history, chunks, tool outputs, and intermediate results.

5. Trust Boundaries

Where does data change its trust level, its permission context, or its possible effect?

6. High-Level Risks

What broad risks follow from the architecture: data leakage, context manipulation, unauthorized retrieval, insecure output use, or tool abuse?

For a practical review, seven short questions additionally help:

Who interacts with the system?
What are the main components?
Where does the context come from?
Is there retrieval?
Are there tools or external APIs?
What data ends up in the model context?
What happens to the model's output?

Template for a Precise System Description

System Type:
What kind of AI/LLM application is this?

Components:
What building blocks, storage, models, and integrations exist?

Data Flow:
How does a request move from input to output or action?

Context Sources:
What instructions, user data, documents, histories, and tool results
reach the model context?

Retrieval:
How are documents ingested, chunked, embedded, filtered, and selected?

Trust Boundaries:
Where does content change trust level, permission, or effect?

Output Use:
Is the model's response only displayed, or further processed and executed?

High-Level Risks:
What protected assets and privileged actions are reachable?

Conclusion

The most important insight of this foundations week is not a single definition. It's a new way of looking.

An LLM processes tokens within a bounded, current context. Embeddings make semantic similarity comparable, but supply neither truth nor trust. RAG doesn't add a human knowledge store to the model, but rather an additional retrieval and context pipeline. Every additional source, every retriever, and every tool creates new trust boundaries.

The compact formula is:

System Type + Components + Data Flows + Context Sources + Trust Boundaries + Effect

Whoever can name these six dimensions no longer sees just "a model, a prompt, and an answer." They see a system — and, for the first time, its real attack surface.

Video

The video is a brief summary of the article.

Further articles in this series