DEV Community: Saurabh

Bloom Filters Explained: A Simple Guide With Real-World Examples and JavaScript

Saurabh — Mon, 19 Oct 2026 10:09:45 +0000

Have you ever wondered how a system can check whether something might exist among millions of records without storing every single record in its memory?

Imagine you are building an application with 10 million registered users.

Whenever somebody enters an email address, your application needs to check whether that email has already been registered.

The straightforward solution is to query your database. But imagine that your application performs millions of such checks every day. Could you avoid some unnecessary database queries?

One interesting solution is a Bloom Filter.

The name might sound complicated, but the basic idea is surprisingly simple. Let's understand it from scratch, explore where it is useful, and build a small working implementation in JavaScript.

1. First, understand the problem

Suppose you maintain a website with the following registered users:

alice@example.com
bob@example.com
charlie@example.com
david@example.com

When a new visitor enters alice@example.com, your application needs to determine whether the email already exists.

You could store all registered emails in a database or an in-memory data structure such as a Set.

That works well. In many applications, an ordinary database or Set is exactly what you should use.

But what happens when your data grows significantly and checking every possible record becomes expensive?

For example, finding that a particular item does not exist in a large database may still require accessing storage or checking an index.

What if we had a much smaller structure whose main job was to quickly tell us when an item definitely does not exist?

That is the problem a Bloom Filter helps solve.

2. What exactly is a Bloom Filter?

A Bloom Filter is a space-efficient data structure that answers a simple question:

Have I probably seen this item before?

It does not store the original items like a normal Set. Instead, it maintains a collection of bits and uses hash functions to mark positions associated with each item.

Its answers have two possible meanings:

Definitely not present: The item was not added to this filter.
Possibly present: The item may have been added, but we cannot be certain.

Notice the word possibly. It is important.

A Bloom Filter can occasionally say that an item might exist even when it was never added. This is called a false positive.

The standard Bloom Filter is designed to avoid false negatives, provided items are inserted and checked correctly using the same filter.

Let's see why.

3. Imagine a board full of tiny lights

Forget the code for a moment.

Imagine a board with 10 small lights. Initially, every light is switched off.

We represent an off light with 0 and an on light with 1.

Our empty board looks like this:

0 0 0 0 0 0 0 0 0 0

Now imagine every email address is associated with three positions on this board.

For example, when we add Alice's email, our hash functions point to positions 1, 4, and 7.

We switch those positions on.

0 1 0 0 1 0 0 1 0 0

That's it. The board now contains information about Alice, but it does not contain her email address.

Next, we add Bob's email. Suppose it maps to positions 2, 4, and 8.

After adding Bob, the board becomes:

0 1 1 0 1 0 0 1 1 0

Notice something interesting: position 4 was already on. Adding Bob did not require any additional information to be stored there.

Now let's check another email address, Eve.

Suppose Eve maps to positions 1, 4, and 8. All three positions are already on.

Our Bloom Filter says that Eve's email might exist.

But we never added Eve!

Alice and Bob happened to switch on all the positions associated with Eve. Our tiny board cannot tell which items originally switched on each light.

This is the key idea behind a Bloom Filter: it remembers which positions have been marked, not the original items that marked them.

In a real implementation, the positions are calculated using hash functions rather than chosen manually.

4. What are hash functions, and why do we need them?

A hash function takes an input and turns it into a number.

For example, a hash function might turn alice@example.com into 471. A different hash function might turn the same email into 1204, and a third into 987.

But our board only has 10 lights, numbered 0 to 9, so we need a valid position. The fix is to take the remainder after dividing by the board size (the % operator):

471 % 10 = 1
1204 % 10 = 4
987 % 10 = 7

That's how Alice ended up at positions 1, 4 and 7 in the previous section.

When adding an item:

Pass the item through each hash function.
Use % size to turn each result into a position in the bit array.
Change those bits from 0 to 1.

When checking an item:

Pass the item through the same hash functions.
Calculate the same positions.
Check whether every one of those bits is on.

If even one bit is off, the item definitely was not added. If every bit is on, the item might have been added, or other items might have switched those bits on.

One rule matters a lot: hash functions must be deterministic. The same input must always give the same positions. If "alice" hashed to 1, 4, 7 when added but to 2, 5, 9 when checked, you would look at the wrong lights and wrongly say "not present". The filter would then have false negatives, which a Bloom filter must never have.

A good hash function also spreads its results evenly across the board. If it favors certain positions, those lights fill up quickly and false positives become more common.

5. What is a false positive?

Let's revisit our example.

Alice and Bob switched on several lights. Later, Eve's email mapped to positions that were already switched on.

The filter returned a positive result even though Eve was never added.

This is a false positive.

It is not a programming bug. It is an expected trade-off of a standard Bloom Filter.

The important thing is that the probability can be controlled.

Several factors affect the likelihood of a false positive:

Size of the filter: More bits generally provide more space to represent the same number of items.
Number of items: Adding more items makes more bits switch on, increasing the chance of overlap.
Number of hash functions (k): Too few, and different items overlap easily. Too many, and each item switches on so many lights that the board fills up quickly. There is a sweet spot in between.

If a filter becomes too full, false positives become more likely. That is why choosing an appropriate capacity and target false-positive rate matters.

For example, a target false-positive rate of 1% means that approximately 1% of queries for items that were not added may return a positive result under the design's expected operating conditions. It does not mean that 1% of all positive answers are necessarily wrong.

That distinction matters when interpreting results.

How big should the filter be?

You don't have to guess. If you know how many items you expect (n) and the false-positive rate you can tolerate (p), two formulas give the best size and number of hash functions:

Bits needed: m = -(n × ln p) / (ln 2)²
Hash functions: k = (m / n) × ln 2

Here are some common targets:

Target false-positive rate	Bits per item	Hash functions
10%	about 4.8	3
1%	about 9.6	7
0.1%	about 14.4	10

So a rough rule is about 10 bits per item and 7 hash functions for a 1% false-positive rate. For 1 million emails, that works out to roughly 9.6 million bits, which is about 1.2 MB. A Set holding a million email strings would need far more.

The same calculation in code:

function optimalSize(n, p) {
  const m = Math.ceil(-(n * Math.log(p)) / Math.log(2) ** 2);
  const k = Math.max(1, Math.round((m / n) * Math.log(2)));
  return { m, k };
}

console.log(optimalSize(1_000_000, 0.01));
// { m: 9585059, k: 7 }

const { m, k } = optimalSize(1_000_000, 0.01);
const filter = new BloomFilter(m, k); // the class we build in the next section

This is why the BloomFilter(1000, 3) we use below is only a toy. Real filters should be sized for the data they will hold.

6. Let's build a Bloom Filter in JavaScript

Theory is useful, but let's put it into practice.

We'll create a small educational implementation using JavaScript. It will accept items, mark positions in a bit array, and tell us whether an item might be present.

The example uses a basic, non-cryptographic hash function to keep the implementation understandable. It is for learning purposes, not a production-ready implementation.

Step 1: Create a hash function

function hashString(value, seed) {
  let hash = seed >>> 0;

  for (let i = 0; i < value.length; i++) {
    hash ^= value.charCodeAt(i);
    hash = Math.imul(hash, 16777619);
  }

  return hash >>> 0;
}

This function converts a string into a 32-bit unsigned integer.

We use two different seeds to derive multiple positions for each item. The code uses a technique called double hashing, which generates several positions from two hash values.

You do not need to memorize the hashing details to understand how the Bloom Filter works. The important part is that the same input consistently produces the same positions.

Step 2: Implement the Bloom Filter

class BloomFilter {
  constructor(size = 1000, hashCount = 3) {
    this.size = size;
    this.hashCount = hashCount;

    // Eight bits are stored in each byte.
    this.bytes = new Uint8Array(Math.ceil(size / 8));
  }

  getIndexes(value) {
    const firstHash = hashString(value, 2166136261);

    const secondHash =
      (hashString(value, 0x9e3779b9) | 1) >>> 0;

    return Array.from(
      { length: this.hashCount },
      (_, i) => (firstHash + i * secondHash) % this.size
    );
  }

  setBit(index) {
    const byteIndex = Math.floor(index / 8);
    const bitPosition = index % 8;

    this.bytes[byteIndex] |= 1 << bitPosition;
  }

  isBitSet(index) {
    const byteIndex = Math.floor(index / 8);
    const bitPosition = index % 8;

    return (
      (this.bytes[byteIndex] & (1 << bitPosition)) !== 0
    );
  }

  add(value) {
    for (const index of this.getIndexes(value)) {
      this.setBit(index);
    }
  }

  mightContain(value) {
    return this.getIndexes(value).every(
      (index) => this.isBitSet(index)
    );
  }
}

Let's break down the important parts.

constructor()

Creates a filter with 1,000 available bit positions by default and uses three hash-derived positions per item.

We use a Uint8Array to store eight bits per byte. This allows the example to use an actual packed bit array rather than reserving an entire byte for every position.

getIndexes()

Calculates the positions associated with an item. The same item produces the same positions as long as the filter parameters and hash implementation remain unchanged.

add()

Marks every calculated position as 1.

mightContain()

Checks every position. If any required bit is 0, it returns false. Otherwise, it returns true, meaning the item might exist.

The method is deliberately called mightContain() rather than contains(). A Bloom Filter cannot guarantee that a positive result is correct.

Step 3: Try it out

Let's add a registered email address and check a few values.

const filter = new BloomFilter(1000, 3);

filter.add("alice@example.com");

console.log(
  filter.mightContain("alice@example.com")
);

console.log(
  filter.mightContain("bob@example.com")
);

Expected output:

true
false

The filter returns true for Alice because we added her email.

It returns false for Bob because at least one of Bob's required positions is still off.

With this filter's size and current contents, Bob's check returns false. However, another absent email could return true if all its positions happened to be switched on.

That is the false-positive trade-off in action.

Step 4: See a false positive happen

So far we've only seen the filter behave well. Let's break it on purpose by giving it far too little space: 20 emails squeezed into just 64 bits.

const tiny = new BloomFilter(64, 3);

// Add 20 registered users
for (let i = 1; i <= 20; i++) {
  tiny.add(`user${i}@example.com`);
}

// Added items are always found
console.log(tiny.mightContain("user7@example.com")); // true

// Now check 200 emails we never added
const falsePositives = [];
for (let i = 21; i <= 220; i++) {
  const email = `user${i}@example.com`;
  if (tiny.mightContain(email)) falsePositives.push(email);
}

console.log(`False positives: ${falsePositives.length} out of 200`);
console.log(falsePositives.slice(0, 3));

Output:

true
False positives: 60 out of 200
[ 'user25@example.com', 'user26@example.com', 'user30@example.com' ]

Look at what happened:

user7@example.com was found, as it always will be. There are no false negatives.
60 of the 200 emails we never added were reported as "might exist". With 37 of the 64 bits switched on, a random absent email has a good chance of landing entirely on lights that other users already switched on.
By the sizing rule from section 5, 20 items at a 1% target need about 192 bits. We gave the filter 64, so it is overcrowded and the false-positive rate is high.

Try changing 64 to 200 and run it again. The false positives should drop sharply. That is the memory-versus-accuracy trade-off in action.

One important limitation: This is a teaching implementation, not a recommendation to write your own production hashing algorithm. Real systems should use a tested Bloom Filter library with appropriate sizing, hash behavior, concurrency handling, and operational monitoring.

7. A real-world example: Checking whether a username exists

Imagine you're building a registration service that handles millions of username availability checks.

A simplified request flow might look like this:

User enters a username
          |
          v
   Check Bloom Filter
          |
     +----+----+
     |         |
   Absent    Possibly
     |         |
     v         v
 Continue    Check database
 registration   |
                v
        Is username registered?

Let's consider both cases.

Case A: The Bloom Filter says the username is definitely absent.

The application can skip a preliminary database lookup whose only purpose was to check whether that username existed.

It can continue with the normal registration process.

The database must still enforce a unique constraint on the username. Otherwise, two users could try registering the same username at the same time.

Case B: The Bloom Filter says the username might exist.

The application queries the database to confirm the result.

If the username exists, it can tell the user that the name is unavailable.

If the database says the name is free, the application can continue registration. The Bloom Filter may simply have produced a false positive.

This approach can help reduce unnecessary database lookups when many queried usernames are absent.

There is an important engineering detail here: the filter must stay consistent with the actual registered-user data. Adding items too early or failing to update the filter after successful registrations can undermine the reliability of the optimization.

A Bloom Filter is an optimization, not a replacement for database correctness.

8. Where else are Bloom Filters useful?

The username example is only one possible application.

A. Databases and storage engines

Databases sometimes need to check whether a key may be present in a particular on-disk data structure.

Checking storage can be more expensive than checking information already available in memory.

A Bloom Filter can help the database avoid some unnecessary storage reads when it determines that a key is definitely absent.

B. Web crawlers

Imagine a web crawler exploring millions of pages.

Before processing a URL, it needs to determine whether that URL may have already been visited.

A Bloom Filter can provide a compact preliminary check. A positive result might require further verification, depending on the crawler's requirements.

This is useful when memory is limited, but a false positive could cause a page to be skipped incorrectly. That trade-off must be considered.

C. Recommendation and advertising systems

A service might want to avoid repeatedly recommending the same item or displaying the same advertisement.

A Bloom Filter can help identify items that may already have been seen.

However, if showing an item twice is unacceptable, the application needs an exact verification mechanism rather than relying solely on the filter.

D. Large-scale data processing

When processing huge datasets, a system may want to avoid performing an expensive lookup for every item.

A Bloom Filter can serve as a compact preliminary check, reducing unnecessary work when a large proportion of the queried items are absent.

The broader pattern is the same: use a small structure to avoid expensive work whenever it can confidently rule an item out.

9. Bloom Filter vs. Set: Which should you use?

A Bloom Filter is not universally better than a Set. Each solves a different problem.

Feature	JavaScript `Set`	Bloom Filter
Stores original items	Yes	No
Memory usage	Depends on stored items and implementation	Usually much smaller for large collections
Absent-item check	Exact	Exact negative result
Present-item check	Exact	May be a false positive
Delete an item	Supported	Not supported by a standard Bloom Filter
Best suited for	When exact membership matters	When memory savings and fast preliminary checks matter

For a small application storing a few hundred usernames, a Set may be simpler and entirely sufficient.

For a large system where memory usage matters and checking absent items is expensive, a Bloom Filter may be worth considering.

The right choice depends on the requirements, not simply on which data structure sounds more advanced.

10. Important limitations to remember

Bloom Filters are useful, but they are not magic.

They cannot recover the original data. The bit array does not tell you which email addresses, URLs, or IDs were inserted.

False positives are possible. A positive result needs verification whenever correctness depends on exact membership.

Standard Bloom Filters do not support deletion. Setting a bit back to zero could accidentally remove information needed by other items. Variants such as counting Bloom Filters can support deletion with additional complexity and storage.

Sizing matters. If you insert substantially more items than the filter was designed for, its false-positive rate can rise.

They are not a security mechanism. The basic implementation presented here is not designed to protect sensitive data, and a Bloom Filter should not replace appropriate security checks.

Understanding these limitations is just as important as understanding how the data structure works.

11. The takeaway

A Bloom Filter is a great example of a practical engineering trade-off.

Instead of storing every original item, it stores a compact representation of information about those items.

In return for saving memory, it accepts uncertainty when the answer is positive.

Remember these three points:

If a standard Bloom Filter says an item is absent, the item was not added, assuming the filter has been used correctly.
If it says an item might be present, the result is not guaranteed to be correct.
It is most useful when ruling out absent items can save significant memory or expensive work.

The next time you encounter a system that handles millions of records, ask yourself: Do we really need to perform an exact lookup every time, or can we cheaply rule out most absent items first?

That is where a Bloom Filter might be worth exploring.

Beyond Unit Tests: How to Test and Evaluate LLM Applications

Saurabh — Tue, 13 Oct 2026 15:12:21 +0000

Imagine deploying an AI-powered chatbot. Your API returns 200 OK, all your unit tests pass, and the application responds within an acceptable time.

But there is one problem: the answer is wrong.

The chatbot confidently provides an incorrect refund policy, invents a step that does not exist in the documentation, or answers a question that it does not have enough information to answer.

From a traditional software testing perspective, everything looks fine. From the user's perspective, the application has failed.

This is one of the biggest challenges when building applications powered by Large Language Models (LLMs).

Unlike traditional functions, LLMs can generate different responses to the same input. A response can be grammatically correct and well-structured while still being factually incorrect.

So, how do we test these applications effectively?

In this article, we'll explore practical approaches to LLM evaluation, hallucination testing, regression testing, and production monitoring, with examples relevant to JavaScript and backend engineers.

1. Why Traditional Unit Tests Are Not Enough

Consider a traditional function:

function calculateDiscount(price, discountPercentage) {
  return price - (price * discountPercentage) / 100;
}

console.log(calculateDiscount(1000, 10)); // 900

For a given input, we can define an expected output and assert that the result matches it.

import assert from "node:assert/strict";

assert.equal(calculateDiscount(1000, 10), 900);

This is a straightforward and deterministic test.

Now consider an AI assistant that answers questions using internal company documentation.

Question:

"What is the refund processing time?"

Expected answer:

"Refunds are processed within five business days."

The LLM might answer:

"Refunds are processed within five business days."
"You should receive your refund within five working days."
"The refund process typically takes about a week."

These answers may not be equivalent in every context. The third answer, for example, is less precise and might introduce ambiguity.

An exact string comparison cannot reliably tell us whether the response preserves the intended meaning.

This is why AI application testing needs more than traditional unit tests.

Traditional testing	AI evaluation
Does the function return the expected value?	Does the answer communicate the correct meaning?
Does the API return a successful status?	Does the response actually solve the user's problem?
Does the output match a fixed structure?	Is the response relevant, accurate, and supported by evidence?
Does the application handle errors?	Does it also behave safely when the answer is unknown?

The solution is not to replace unit tests. It's to combine them with additional evaluation techniques.

2. What Should We Measure in an AI Application?

Before evaluating an AI system, we need to define what a good response looks like.

There is no single metric that can measure every aspect of AI quality. Different applications require different evaluation criteria.

A. Correctness

Does the response provide the correct answer?

For a customer support assistant, correctness might mean that refund periods, eligibility conditions, and other policy details match the approved source.

Possible approaches include:

Comparing answers against reference answers.
Checking required facts or claims.
Having domain experts review a representative sample.
Using an automated evaluator with clearly defined criteria.

B. Relevance

Does the response answer the question that was asked?

For example, if someone asks how to reset a password, a detailed explanation of account registration is not a relevant answer, even if all the information is accurate.

Relevance matters when building documentation assistants, support chatbots, and search applications.

C. Faithfulness and hallucinations

For applications that retrieve documents before generating an answer, faithfulness means that the answer's claims are supported by the retrieved context.

Suppose the source document states:

"Refunds are processed within five business days."

The assistant answers:

"Refunds are processed within five business days, and a confirmation email is always sent within one hour."

If the source says nothing about a confirmation email, the additional claim is unsupported.

This is a useful way to test hallucinations in Retrieval-Augmented Generation (RAG) applications.

However, an answer that is faithful to a document is not automatically correct in the real world. The source itself could be outdated or incorrect, so source quality matters too.

D. Completeness

Does the response cover all the important parts of the question?

If a user asks for three steps and the assistant provides only two, the answer may be accurate but incomplete.

E. Safety

Does the application avoid unsafe or unauthorized behavior?

Examples include refusing to reveal another customer's private information, following access restrictions, and resisting attempts to bypass system instructions.

Safety tests should cover both ordinary usage and adversarial inputs.

The important lesson: Define your evaluation criteria before choosing the metrics. A high score on one metric does not guarantee that the application is good overall.

3. Building a Small Evaluation Dataset

One of the most practical ways to test an LLM application is to maintain a fixed dataset of representative questions and expected behavior.

This dataset can be treated as an evaluation suite and run whenever you change a prompt, model, retrieval configuration, or important application component.

For example:

Test scenario	Expected behavior
Ask for the refund processing time	Return the documented time
Ask about an unsupported policy	Acknowledge that the information is unavailable
Request a short summary	Preserve the key facts without adding claims
Ask a question unrelated to the documentation	Avoid inventing an answer
Request another customer's private data	Do not disclose unauthorized information

A basic dataset might look like this:

const evaluationCases = [
  {
    name: "Refund processing time",
    question: "How long does a refund take?",
    requiredPhrases: ["five business days"],
  },
  {
    name: "Unknown information",
    question: "What is the refund policy for a product not covered by the documentation?",
    uncertaintyPhrases: [
      "not available",
      "not specified",
      "cannot confirm",
      "don't have enough information",
    ],
  },
];

The required phrases and accepted uncertainty phrases are illustrative. They should be aligned with the actual policy and the response conventions used by your application.

As the application grows, include more examples covering different user intents, edge cases, and failure modes. Keep the dataset representative rather than filling it only with easy questions.

4. Writing AI Regression Tests in JavaScript

Let's turn the dataset into a simple automated test suite.

Assume that our application exposes an asynchronous answerQuestion() function that returns an object containing an answer property.

We can use Node.js's built-in test runner, avoiding the need for an additional test framework.

import test from "node:test";
import assert from "node:assert/strict";
import { answerQuestion } from "../src/assistant.js";

const evaluationCases = [
  {
    name: "Refund processing time",
    question: "How long does a refund take?",
    requiredPhrases: ["five business days"],
  },
  {
    name: "Unknown information",
    question: "What is the refund policy for an unsupported product?",
    uncertaintyPhrases: [
      "not available",
      "not specified",
      "cannot confirm",
      "don't have enough information",
    ],
  },
];

for (const scenario of evaluationCases) {
  test(scenario.name, async () => {
    const response = await answerQuestion(scenario.question);

    assert.equal(typeof response.answer, "string");
    assert.ok(response.answer.trim().length > 0);

    const answer = response.answer.toLowerCase();

    if (scenario.requiredPhrases) {
      for (const phrase of scenario.requiredPhrases) {
        assert.ok(
          answer.includes(phrase.toLowerCase()),
          `Expected answer to contain: ${phrase}`
        );
      }
    }

    if (scenario.uncertaintyPhrases) {
      const acknowledgesUncertainty =
        scenario.uncertaintyPhrases.some((phrase) =>
          answer.includes(phrase)
        );

      assert.ok(
        acknowledgesUncertainty,
        "Expected the assistant to acknowledge missing information"
      );
    }
  });
}

Run the tests with:

node --test

This is a simple baseline, not a complete semantic evaluator.

The code checks whether the response contains expected wording. That can catch obvious regressions, but a correct answer that uses different wording might fail. Conversely, a response could include the required phrase and still contain incorrect additional claims.

For production systems, combine these checks with claim-level verification, semantic evaluation, and human review where appropriate.

What happens when the prompt changes?

Imagine that the team modifies the system prompt to make answers shorter.

The change may improve response time and reduce token usage, but it could also remove important policy details.

Running the evaluation dataset again helps determine whether the change improved the system overall or introduced regressions.

This is the central benefit of an AI regression suite: changes are evaluated against a repeatable set of expectations instead of relying only on a few manual experiments.

5. Using an LLM as an Evaluator

Exact phrase matching is useful for specific requirements, but many answers need semantic evaluation.

For example, these two answers communicate approximately the same information:

"The request will be processed within five business days."
"Please allow five working days for processing."

A simple string comparison may mark them as different even when both are acceptable.

One approach is to use a separate LLM to evaluate the generated response. This technique is commonly known as LLM-as-a-judge.

The evaluator can be given:

The user's question.
The relevant source context.
The generated answer.
A rubric describing how the answer should be assessed.

For example, we might ask the evaluator to score each dimension from 1 to 5:

Dimension	Evaluation question
Relevance	Does the answer address the user's question?
Faithfulness	Are the answer's claims supported by the supplied context?
Completeness	Does it cover the required information?
Clarity	Is it understandable and unambiguous?

A simple evaluation instruction could be:

You are evaluating an AI-generated answer.

Question:
{question}

Source context:
{context}

Generated answer:
{answer}

Evaluate relevance, faithfulness, and completeness
on a scale from 1 to 5.

Use the supplied source context to assess faithfulness.
Do not reward an answer merely because it sounds confident.
Identify any unsupported claims and explain the reason
for each score.

Return structured JSON containing the scores,
unsupported claims, and a short explanation.

The evaluator should follow a consistent rubric and produce structured output that can be stored and compared.

However, an LLM evaluator is not a perfect source of truth. It can overlook errors, prefer fluent answers, or make its own incorrect judgments.

To improve reliability:

Validate the evaluator's results against human-reviewed examples.
Keep the evaluation instructions consistent across comparisons.
Use trusted source material where available.
Review borderline cases and disagreements.
Do not rely on one automated score for critical safety decisions.

LLM-as-a-judge is best treated as a scalable evaluation aid, not a replacement for ground truth or human judgment.

6. Measuring Latency, Token Usage, and Cost

Answer quality is only one part of production readiness.

A response can be accurate but take too long to generate. Another response might be acceptable but consume significantly more tokens than expected.

For this reason, quality evaluations should also track operational metrics.

Latency

Measure how long a request takes to complete.

Useful metrics include:

Median latency (p50).
p95 latency.
Timeout rate.
Error rate.

The p95 latency tells us the response time at or below which approximately 95% of measured requests complete. It helps identify slow experiences that an average can hide.

Token usage and cost

When supported by the model provider, collect input and output token counts for every request.

A simplified cost calculation is:

function estimateCost({
  inputTokens,
  outputTokens,
  inputPricePerMillion,
  outputPricePerMillion,
}) {
  return (
    (inputTokens / 1_000_000) * inputPricePerMillion +
    (outputTokens / 1_000_000) * outputPricePerMillion
  );
}

The price values should come from the applicable model's current pricing configuration. This calculation is an estimate and may not include other infrastructure or service costs.

Track cost per request, cost by feature, and cost per successfully completed user task.

The last measure is particularly useful because reducing the cost of each response does not necessarily improve the system if the answers become less useful.

Quality alongside performance

Consider a new model or prompt configuration that reduces average latency by 20% but also increases incorrect answers.

Looking only at latency would suggest an improvement. Evaluating quality and operational metrics together gives the team a more complete picture.

7. Monitoring an AI Application in Production

Offline evaluation helps us test changes before deployment. It cannot predict every real-world input or failure.

Production monitoring is therefore equally important.

A useful monitoring strategy includes:

Technical health

Request volume and error rate.
Model-provider failures and timeouts.
Median and p95 latency.

Cost and usage

Input and output tokens.
Estimated cost by model and feature.
Unexpected usage spikes.

AI quality

Periodic evaluation of sampled responses.
Unsupported claims and retrieval failures.
User feedback and task completion rates.
Changes in quality after a model or prompt update.

For a RAG system, it can also be valuable to monitor retrieval quality separately from answer generation. The retriever may return irrelevant documents even when the LLM is following those documents correctly.

When logging requests, take care to protect customer information. Redact sensitive data, apply appropriate access controls, and define retention rules rather than storing every raw prompt and answer indefinitely.

The goal is not to capture everything. It is to collect the right signals to detect failures and investigate them safely.

8. Putting It All Together: An AI Testing Workflow

A practical workflow for an engineering team could look like this:

Define quality criteria. Decide what a correct, relevant, safe response means for the feature.
Build an evaluation dataset. Include normal scenarios, edge cases, and known failure modes.
Run traditional tests. Verify API contracts, validation, authentication, error handling, and application logic.
Run AI evaluations. Check correctness, relevance, faithfulness, and the behaviors that matter to the feature.
Compare with the previous version. Investigate regressions after prompt, model, or retrieval changes.
Set deployment gates. Use thresholds for critical metrics that reflect the application's actual requirements.
Monitor production. Track latency, errors, cost, feedback, and sampled quality evaluations.
Feed real failures back into the dataset. Turn recurring production problems into regression tests.

This creates a continuous improvement loop: evaluation informs deployment, and production observations inform future evaluation.

Conclusion

Testing AI applications requires a broader approach than checking whether an API responds successfully.

Traditional unit and integration tests remain essential, but they do not establish that an LLM's answer is correct, relevant, complete, or safe.

A stronger strategy combines repeatable evaluation datasets, clearly defined quality criteria, semantic evaluation, regression testing, and production monitoring.

You do not need a complicated evaluation platform to get started. A small dataset, an automated test runner, a consistent evaluation rubric, and basic latency and cost tracking can already make AI development more disciplined.

The key takeaway is simple: don't just test whether your AI application works. Test whether it works well enough for the user and the problem it is designed to solve.