DEV Community: Ruhan Khandakar

2x faster, 7x cheaper: your search router doesn't need a chat model

Ruhan Khandakar — Thu, 29 Oct 2026 12:47:38 +0000

We recently added record search to Sapling RCM, our medical billing app. You press ⌘K, type something, hit Enter, and get results. The query might be a clinic name, patient name, claim number, payment ID, or something else. The results can come from eight different tables.

Names are blurred.

Before hitting the database, we run the search text through a small model. Its job is to figure out what kind of text the user entered, so we can search only the tables that make sense.

I tried two models for this: Jev and Gemini 3.5 Flash-Lite. Then I measured how they compared on speed, cost, and missed results.

Here's what the router does, how I tested it, and what I found.

TL;DR

Jev's median routing call took 466 ms. Gemini took 900 ms.
Jev was about 7x cheaper: roughly $23 per million searches vs. $170.
Jev missed results on 3 of 27 searches. Gemini missed on 6 of 30.

Ask what the text is, not which table to search

One way to build this is to ask the model, "Which tables should I search?"

We don't do that.

Instead, we ask the model what kind of text the user entered, and then map that to tables in our own code.

For example:

Kind of text	Example	Tables we search
person name	a patient or a provider	patients, providers, claims
claim number	`CLM-2026-00847`	claims, payouts
payment ID	`ERA-…`	payments

We have six kinds of text like this.

This ended up being useful for a few reasons:

Each question is simple. "Could this be a person's name?" is easier than "Which of these eight tables should I search?"
The model never needs to know our database schema. If we add another table, we update the mapping in our code instead of changing the prompt.
A table can belong to multiple kinds. Claims, for example, can be found through a patient name or a claim number. The model doesn't need to know that relationship.

Router 1: Jev

Jev, from TypeSafe and served through OpenRouter, isn't a chat model.

You give it some state and a set of typed questions, and it returns a probability for each question. There's no generated text to parse and no reasoning tokens.

const response = await fetch('https://openrouter.ai/api/alpha/decisions', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'typesafe/jev-1.14',
    state: { query },
    questions: {
      claim_number: {
        type: 'noul',
        instructions:
          'Could this search text be all or part of a claim number? Claim numbers look like CLM-2026-00847.',
      },
      // one question per kind
    },
    provider: { zdr: true },
  }),
  signal: AbortSignal.timeout(2_500),
});
// { answers: { claim_number: { noul: 0.97 }, person_name: { noul: 0.02 }, ... } }

A few details in this request turned out to matter:

Put your actual ID formats in the question. Otherwise, the model has to guess what your IDs look like.
Pin the model version. Using something like ~latest means your routing behavior can change without you noticing.
provider: { zdr: true } tells OpenRouter to use providers with zero data retention.

For each kind, we search its tables when the score is 0.3 or higher. If nothing reaches 0.3, we take the highest scoring kind.

The nice thing about getting probabilities is that the threshold is ours.

If we're missing results, we can lower it and search more tables. If we're getting too many irrelevant results, we can raise it. We don't need to change the prompt.

Router 2: Gemini 3.5 Flash-Lite

For Gemini, I gave it the same questions as a list in the system prompt, along with a JSON schema for the response. Temperature is set to 0.

const response = await vertex.models.generateContent({
  model: 'gemini-3.5-flash-lite',
  contents: query,
  config: {
    systemInstruction:
      'Decide which kinds of text this search could be. Return every kind that could apply.\n' +
      '- claim_number: Could this be all or part of a claim number? ...',
    temperature: 0,
    responseMimeType: 'application/json',
    responseJsonSchema: {
      type: 'object',
      properties: {
        kinds: {
          type: 'array',
          items: { type: 'string', enum: ['person_name', 'claim_number', 'payment_id' /* ... */] },
        },
      },
      required: ['kinds'],
    },
    abortSignal: AbortSignal.timeout(2_500),
  },
});
// {"kinds":["claim_number"]}

The schema makes the response easy to parse.

The downside is that Gemini gives us a list rather than scores. There's no equivalent of "use everything above 0.3," so there's no threshold to tune.

Don't let the router break search

The router is an extra network call in the search path, so eventually it's going to fail.

That's why every step has a fallback:

Ask Jev.
If Jev fails for any reason missing API key, timeout, bad status, malformed response, etc. ask Gemini.
If Gemini fails too, skip routing and search every table the user has access to.
Otherwise, search only the tables for the selected kinds, in parallel.
If those searches return nothing, search the remaining tables before showing "No results." The router can be wrong.

The last step is important. Routing is an optimization. It shouldn't change whether search works at all.

Trace every call, but be careful with search text

Every routing call creates an OpenTelemetry span with OpenInference attributes, so Phoenix displays it as an LLM call:

tracer.startActiveSpan(
  'search_routing',
  {
    attributes: {
      'openinference.span.kind': 'LLM',
      'llm.provider': 'openrouter',
      'llm.model_name': 'typesafe/jev-1.14',
      'input.value': JSON.stringify(body),
    },
  },
  async (span) => {
    // ...call the model...
    span.setAttributes({
      'output.value': JSON.stringify(answers),
      'llm.token_count.prompt': usage.input_tokens,
      'llm.token_count.completion': usage.output_tokens,
      'llm.cost.total': usage.cost,
    });
  },
);

These spans made it easy to compare the two routers call by call.

I also ran into two things worth mentioning.

First, Phoenix displays any cost below one cent as <$0.01. A Jev call costs only a few millionths of a dollar, so every call looked identical in the UI. If you care about the actual number, look at the raw llm.cost.total attribute.

Second, our Phoenix instance runs inside our own infrastructure, so the spans currently contain the actual search text. That's convenient for debugging, but it also means that search text can end up in your trace system.

If you're using a SaaS trace provider, think about that before sending the raw input. In a healthcare app, the search text might be a patient name.

Error logs can have the same problem.

Some ORMs, in our case, Drizzle, include query parameters in database error messages. Since our query parameter is the user's search text, that can end up in the logs too.

When a query fails, we log the database error code rather than the full error message.

How I measured the two routers

Speed and cost are straightforward. Accuracy is a little harder because we need to know what the correct answer actually was.

I kept it simple: for every test query, I also searched all eight tables. If a router skipped a table that contained a match, I counted that search as a miss.

The test queries included:

a clinic name
two claim numbers
a payment ID and the bare prefix ERA-
the bare prefix of a batch number, DISB-
a provider ID (NPI)
a short word: pay
a junk string: a_b%
a common surname

Each query ran three times through our real search service.

The surname only went to Gemini. We don't have a BAA with OpenRouter yet, so patient names don't go through it. That's why Jev has 27 calls while Gemini has 30.

Results

	Jev 1.14	Gemini 3.5 Flash-Lite
Routing time, median	466 ms	900 ms
Routing time, p95	648 ms	1,588 ms
Cost per search	$0.000023	$0.00017
Cost per 1M searches	~$23	~$170
Failed calls	0 of 27	1 of 30 (timeout)
Searches that missed a result	3 of 27	6 of 30

For this test, Jev was roughly 2x faster at the median and 2.5x faster at p95. It was also about 7x cheaper and had fewer misses.

Jev costs $0.042 per million input tokens, and output is free. Our questions are short, so each call ends up costing almost nothing.

Gemini's actual cost may be slightly higher than what's shown in the table because our span didn't include thinking tokens.

For our use case, Jev was the better fit of the two.

The biggest difference wasn't just that it was faster and cheaper. The probability output gives us something we can tune. That's useful for a router because we don't really need the model to "answer" the user. We just need a good signal for deciding where to search.

If you need a small classifier in a hot path, that's a pretty good reason not to reach for a chat model by default.

Deploy Django + MySql Application into AWS EC2 instance with nginx and SSL

Ruhan Khandakar — Thu, 13 May 2021 04:11:18 +0000

All the version used in this tutorial

Python 3.8.X
Django 3.2.2
gunicorn 20.1.0
MySQL 8.0.20 (RDS)

EC2 Setup

Create EC2 instance (Screenshots attached below), during this instance creation. we'll have to download pem/cer file. It'll be used later to access this instance from the terminal via SSH
To connect the instance, Follow how to connect to instance guide from the EC2 dashboard.
After login into our instance, run the following commands to update our instance -> sudo apt update; sudo apt upgrade -y.
In order to use pip3 and venv we have to install some packages -> sudo apt install python3-pip python3-venv
Let's create one virutal env -> python3 -m venv env. And activate it -> source env/bin/activate.
Now we have to install gunicorn^details (make sure you're in the virtual environment) -pip3 install gunicorn.

Note: if your project has a MySQL database, you have to install some other packages
sudo apt install python3-dev default-libmysqlclient-dev build-essential libssl-dev

Now let us pull the repository and install all the dependencies. After this do the necessary migrations.
Let us install nginx^details -> sudo apt install nginx -y. now you can check with sudo nginx.
Before starting the nginx setup we have to change our instance's security groups (Check below screenshots)
Now let us do a little test, we're going to start our application by gunicorn (what is it? will discuss it in later.) -gunicorn --bind 0.0.0.0:8000 courier_allocation.wsgi:application, after running this we'll see something like 👇🏻

and now we can access our application from our instance IP address, like this ec2-XX-XX-XX-XX.ap-south-1.compute.amazonaws.com:8000.
But now if we close our terminal then we won't be able to access our application.
Now we have to do some setup so that our application can run in the background. Here, we'll use supervisor^details. To install this package we have to run sudo apt install -y supervisor.
After installation, we have to create one supervisor config file. For this follow the below steps.

First go to cd /etc/supervisor/conf.d/
Here we'll create one config file -> sudo touch gunicorn.conf
In this conf file paste the below code.

    [program:gunicorn]
    directory=/home/ubuntu/PROJECT_FOLDER
    command=/home/ubuntu/env/bin/gunicorn --workers 3 --bind unix:/home/ubuntu/PROJECT_FOLDER/app.sock DJANGO_PROJECT_FOLDER.wsgi:application
    autostart=true
    autorestart=true
    stderr_logfile=/home/ubuntu/logs/gunicorn.err.log
    stdout_logfile=/home/ubuntu/logs/gunicorn.out.log
    [group:guni]
    programs:gunicorn

Now we have to run some commands

sudo supervisorctl reread -> Output - guni: available

sudo supervisorctl update -> Output - guni: added process group

sudo supervisorctl status -> Output - guni:gunicorn RUNNING pid 20199, uptime 0:00:17

Now let's do the nginx^details configuration

First we have to go to - cd /etc/nginx/sites-available/.

Here we need to create one config file - sudo touch django.conf (you can choose any file name). Paste the below code inside your newly created file

    # Nginx Configuration for sample Django application
    ######################################################
    server {
        listen 80;
        server_name ec2-XX-XX-XX-XX.ap-south-1.compute.amazonaws.com;
        location / {
            include proxy_params;
            proxy_pass http://unix:/home/ubuntu/PROJECT_FOLDER/app.sock;
        }
    }

After this run - sudo ln django.conf /etc/nginx/sites-enabled/

now let's test - sudo nginx -t - Output -> nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

Sometime you might get this error -> nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64. To fix this we have to add server_names_hash_bucket_size 128; at the top of your http block (located in /etc/nginx/nginx.conf).

Now we have to restart our nginx service -> sudo service nginx restart.

Our nginx^details is up and running 😁. You can directly access your app from Public IP Address (EC2)

Note: Still a lot of configuration is left, like cache, route multiple ports, load balancer etc. Will discuss it in a different post. 🙂

SSL Setup

* Requirements: domain address which is mapped to your Instance Public IP.

Setup SSL certificate with certbot is quite easy. We just need to install some packages and a little bit of setup

Install packages - sudo apt install certbot python3-certbot-nginx

Before we generate the certificate, we have to add our domain name in our django.conf

Change existing server_name to - server_name test.domain.com; (if you want to map multiple domains then use as space separate like this - server_name test.domain.com testing.domain.com;)

After setup is done, run the below two commands

sudo certbot certonly --nginx

sudo certbot certonly --nginx

Now we have to restart our nginx - sudo service nginx restart
Now you can visit your secure website 😊.

Django Static File issue Fix after deployment (production)

You might notice that in production, CSS/JS/Images/Fonts files are not working for /admin.
This is because of the Static folder issue. In the local development environment Django use Python HTTPServer, and it automatically handles Static files. But in production our server is running through nginx, that's why we have to do little setup. Let's do the setup.

First, we have to create one static folder in our root directory.
Now inside our project main settings.py we just need to add one line of code

STATIC_ROOT = BASE_DIR / 'static/'

Now run this command python3 manage.py collectstatic³. With this command, Django will copy all static assets into your Root Static Folder. In our case, it's the Static folder.
Now one setup is left, which is nginx,

Go to your nginx django.conf file and add the below code after the location code block
location /static/ {
               autoindex on;
               alias /home/ubuntu/PROJECT_FOLDER/static/;
}

Now restart nginx server - sudo service nginx restart.

nginx

Nginx, pronounced like “engine-ex”, is an open-source web server that, since its initial success as a web server, is now also used as a reverse proxy, HTTP cache, and load balancer.

Nginx is built to offer low memory usage and high concurrency. Rather than creating new processes for each web request, Nginx uses an asynchronous, event-driven approach where requests are handled in a single thread.

With Nginx, one master process can control multiple worker processes. The master maintains the worker processes, while the workers do the actual processing. Because Nginx is asynchronous, each request can be executed by the worker concurrently without blocking other requests. more info

gunicorn:

Gunicorn is a stand-alone WSGI web application server that offers a lot of functionality. It natively supports various frameworks with its adapters, making it an extremely easy to use drop-in replacement for many development servers that are used during development.

supervisor:

At some point you'll likely find yourself writing a script that needs to run all the time - a "long running script". These are scripts that shouldn't fail if there's an error or ones that should restart when the system reboots.

To accomplish this, we need something to watch these scripts. Such tools are process watchers. They watch processes and restart them if they fail, and ensure they start on system boot.

And Supervisord is a simple and popular choice for doing these tasks and monitoring our process

collectstatic

Using the collectstatic command, Django looks for all static files in your apps and collects them into a single location that can easily be served in production, i.e. the STATIC_ROOT. In our case, we are telling Django that when we run python3 manage.py collectstatic, gather all static files into a folder called static in our project root directory.

That's it for this article. I hope you found it useful. Feel free to drop your comments below and like the post.