DEV Community: Jake Casto

Signal-driven health monitoring for HNSW indices w/ pgvector

Jake Casto — Mon, 08 Dec 2025 18:47:52 +0000

Vector search in PostgreSQL has come a long way, and pgvector makes it simple:

insert embeddings, create an HNSW index, and run ANN queries.

At least for a while.

As our system scaled across stores, regions, and product volumes, something became clear:

HNSW is not “fire and forget.”

Indexes age. They bloat. They fragment. They silently lose recall.

When they degrade, search results degrade too — sometimes gradually, sometimes catastrophically.

For months, we assumed our approach was fine:

Rebuild every night at 1 AM
Rebuild after imports
Rebuild after new embeddings
Rebuild everything, always

It felt safe.

It was absolutely not.

When Scheduled Rebuilds Turned Into a Problem

Our first approach was periodic instead of condition-based — every night, every tenant, every index.

And here's what happened:

What we saw	Why it was bad
We rebuilt indexes with zero new writes	Massive waste of compute + IO
Large tenants saw daily index decay	Recall dropped, latency increased
Crash mid-rebuild left index invalid but present	PostgreSQL stopped using it entirely
Replicas diverged from primaries	ANN returned different results per region

The worst case was the INVALID index scenario.

A Postgres restart happened during a CONCURRENT index build. During this operation the index is created first as INVALID, then validated after two full table scans.

A crash meant the scans never completed — leaving a corrupt index that still appeared valid.

Result:

An index that looks present but is unusable, silently ruining search until someone notices.

That incident forced a rethink.

We didn’t need scheduled rebuilding.

We needed index health monitoring — rebuild only when necessary.

What Does “Healthy” Mean for an HNSW Index?

B-trees give you bloat %, page churn, dead tuples.

HNSW is different — it’s a navigable multi-layer graph whose health depends on:

how many vectors were replaced without pruning
how many nodes reference embeddings that no longer exist
whether graph layers remain navigable under memory pressure
whether the index fits in cache vs spilling to disk

There is no single fragmentation metric.

So we built one.

The Three-Metric Health Model

We generate one score per index using:

Metric	Detects
Dead tuple ratio	Graph churn, stale nodes, recall degradation
Cache hit ratio	Memory residency vs disk traversal
Bytes per vector	HNSW structural bloat / unnecessary overhead

If an index inflates, falls out of cache, or accumulates dead references — ANN quality drops.

PostgreSQL already exposes everything we need.

📊 Metric 1 — Dead Tuple Ratio

Dead tuples = deleted or overwritten embeddings still referenced inside the graph.

They increase hop count and reduce recall.

dead_tuple_ratio = n_dead_tup / n_live_tup

Threshold: 10–15% is worth rebuilding.

SELECT
    relname AS table_name,
    n_dead_tup,
    n_live_tup,
    ROUND((n_dead_tup / GREATEST(n_live_tup, 1)::float) * 100, 2) AS dead_tuple_ratio_pct
FROM pg_stat_all_tables
WHERE relname = 'your_embedding_table_name';

📊 Metric 2 — Cache Hit Ratio

Most performance issues weren't degraded structure —

they were cache eviction.

If HNSW pages fall out of shared_buffers → traversal becomes disk-bound → planner may stop using the index.

SELECT
    indexrelid::regclass AS index_name,
    idx_blks_hit,
    idx_blks_read,
    ROUND((idx_blks_hit * 100.0 / GREATEST(idx_blks_hit + idx_blks_read, 1)), 2) AS cache_hit_ratio_pct
FROM pg_statio_user_indexes
WHERE indexrelid::regclass::text LIKE '%hnsw%';

Ratio	Status
> 90%	Healthy
90–75%	Yellow
< 75%	Red — likely disk-bound, planner risk

📊 Metric 3 — Bytes Per Vector

Heuristic for structural bloat.

If stored bytes per embedding drift too high → rebuild pays off.

WITH idx AS (
    SELECT c.relname AS index_name, pg_relation_size(c.oid) AS index_bytes
    FROM pg_class c
    JOIN pg_index i ON i.indexrelid = c.oid
    WHERE c.relname LIKE '%hnsw%'
)
SELECT
    index_name,
    pg_size_pretty(index_bytes) AS index_size,
    (
        index_bytes / NULLIF(
            (SELECT reltuples FROM pg_class WHERE relname = 'your_embedding_table_name'),
            0
        )
    ) AS bytes_per_vector
FROM idx;

Example: expected 5.0 KB/vector, observed 9.2 KB/vector → rebuild.

Not perfect, but highly correlated with real-world recall loss.

The Health Score

Instead of binary rebuild vs no rebuild, we compute:

score = (dead_score * 0.4) + (cache_score * 0.4) + (bloat_score * 0.2)

Weighting rationale:

Dead tuples = recall degradation → heaviest signal
Cache residency = latency + planner reliability
Bloat matters, but less across large embeddings

If score < threshold → rebuild just that index.

Result

We stopped rebuilding everything.

We rebuild only when needed.

Smarter. Faster. Cheaper.

And search quality no longer surprises us.

Like the writing? Follow product development at Layers

Laravel Nova: Adding custom buttons to resource toolbars

Jake Casto — Thu, 14 Mar 2019 00:48:15 +0000

If you've seen issue 786 on laravel/nova-issues on Github or attempted to add custom buttons to Resources in Laravel Nova you are probably frustrated. I spent a whole day trying to figure this out. But it's a lot simpler than you think!

Laravel Nova is powered by Vue.JS a very powerful JS microframework. Each resource has a custom component and that component has its own scope. This allows us to override Nova's built-in components and add custom buttons.

Let's start by creating a new Nova Resource Tool:



php artisan nova:resource-tool 0x15f/custom-resource-toolbar

Say yes to all of the prompts...

Now that you've created your resource tool navigate to the nova-components/custom-resource-toolbar/resources/js directory open tool.js in your favorite JS editor and paste the following.



Nova.booting((Vue, router) => {
    Vue.component('custom-detail-toolbar', require('./components/CustomDetailToolbar'));
    Vue.component('quotes-detail-toolbar', require('./components/QuotesDetailToolbar'));
})

Navigate into the components directory and delete Tool.vue. Now create two files, one named CustomDetailToolbar.vue the other named QuotesDetailToolbar.vue. Paste the following into CustomDetailToolbar.vue:



<template>
    <div class="flex w-full justify-end items-center mx-3">
        <component v-if="hasComponent" :is="component" />
    </div>
</template>

<script>
    export default {
        props: ['resourceName', 'resourceId'],
        computed: {
            component(){
                return this.resourceName + '-detail-toolbar';
            },
            hasComponent()
            {
                return this.component in this.$options.components;
            }    
        }
    }
</script>

And paste the following into QuotesDetailToolbar.vue:



<template>
    <div>
        <div class="flex w-full justify-end items-center">
           <a
                class="btn btn-default btn-icon btn-white"
                :href="'/nova-vendor/custom-resource-toolbar/export-quote/' + this.$parent.resourceId"
                style="background-color: var(--primary); color: white;">
                Export as PDF
             </a>
        </div>
    </div>
</template> 

<script>
    export default {
        props: ['resourceName', 'resourceId', 'field'],
        mounted() {
            var elements = document.getElementById('nova').querySelectorAll('h4');
            [].forEach.call(elements, function(element) {
                if(element.innerHTML === 'Custom Detail Toolbar') {
                    element.parentNode.remove();
                }
            });
        }
    }
</script>

You can now build your Resource Tool using npm run watch, add it to your Resources, and open your CustomResourceToolbar.php file and change the component name to custom-detail-toolbar. I'll explain what we did below.

In tool.js we registered two new components custom-detail-toolbar and quotes-detail-toolbar, custom-detail-toolbar is used by Nova to load any components that should be displayed on that toolbar. Nova then looks for the resource's toolbar. You can define your resources toolbar component by registering a component with your resources plural name followed by -detail-toolbar. Within that component, you can add any CSS/JS that should be placed in the toolbar.

If you notice I have a mounted function in my component with the following code:



mounted() {
    var elements = document.getElementById('nova').querySelectorAll('h4');
    [].forEach.call(elements, function(element) {
        if(element.innerHTML === 'Custom Detail Toolbar') {
            element.parentNode.remove();
        }
    });
}

Typically a resource tool has a panel on your resource detail page. This function runs when our component has been mounted to remove the panel that nova adds giving your page a clean feel.

You can find all the code used in this tutorial on github.

Note: I quickly wrote this article, I'll clean it up later.

Using libcurl3 and libcurl4 on Ubuntu 18.04 (Bionic)

Jake Casto — Tue, 02 Oct 2018 15:49:02 +0000

I'm sure many of us have needed to run libcurl3 and libcurl4 on Ubuntu 18 for some reason at any point in time.

On Ubuntu 18.04 (Bionic) libcurl3 and libcurl4 cannot coexist. Many third party packages, libraries, etc require libcurl3 while curl itself requires libcurl4.

Please note, this solution is only for precompiled commands (and potentially binaries, not tested). This was tested on a C# binary.

First you'll need to get a copy of libcurl3 and save it to your /usr/lib directory. You can do that by running:

$ sudo apt-get install libcurl3 -y
$ cp /usr/lib/x86_64-linux-gnu/libcurl.so.3 /usr/lib/

Then remove libcurl3 and reinstall libcurl4:

$ sudo apt-get install libcurl4 libcurl4-openssl-dev -y

Finally prepend env LD_PRELOAD=/usr/lib/libcurl.so.3 to any commands that require libcurl3.

Using Headers & Cookies with PHP's file_get_contents() function

Jake Casto — Thu, 13 Sep 2018 03:00:24 +0000

This solution was originally linked to me by a colleague, all credit goes to this tweet.

Did you know that using stream contexts, you can set headers when making HTTP requests with php's file_get_contents() function?

$context = stream_create_context([
    "http" => [
        "method" => "GET",
        "header" => "Accept-languange: en\r\n" .
        "Cookie: foo=bar\r\n"
    ]
]);

$file = file_get_contents('https://example.com', false, $context);

How do you find potential clients to cold-email?

Jake Casto — Sun, 15 Apr 2018 15:16:28 +0000

I just launched a development studio Lynn Digital. I'll give you a bit of background on my new company. Originally I needed more developers to assist me on larger projects when freelancing but eventually, clients started to dislike developers they didn't know working on their projects so I decided to start a company. Currently, I have four developers that I can subcontract work too, we each bring a different skillset to the table which allows us to work together on nearly any project.

I've sent a handful of custom cold-emails to potential clients I found through another client but I was only able to get 5 - 10 references. I don't want to buy a database of emails and launch a marketing campaign, I want to be able to reach out to potential clients personally.

How can I find potential clients to cold-email?

Configuring PHP-FPM For High Network Traffic

Jake Casto — Tue, 23 Jan 2018 19:50:14 +0000

Maintaining a constant response time on a server with high network traffic while using PHP is probably the hardest & most annoying thing I've done in my career. Switching from Apache to Nginx was a huge performance upgrade... that lasted for a solid half hour before the RPS (requests-per-second) was causing the stack to overflow once again.

I started browsing Stack Overflow & Google (devs best & worst friend/enemy) for answers, I found a few posts that showed how to fix the issue (PHP-FPM conf) but didn't explain anything. As a hands-on developer, I wanted to know how things worked.

Originally my PHP-FPM conf looked something like this



[www]

user = apache
group = apache

listen = 127.0.0.1:9000
listen.allowed_clients = 127.0.0.1

pm = dynamic

pm.max_children = 200

pm.start_servers = 20

pm.min_spare_servers = 10
pm.max_spare_servers = 20

pm.max_requests = 1000

I started reading into what pm = dynamic meant and I found this (in another conf file haha)

dynamic - the number of child processes are set dynamically based on the following directives. With this process management, there will be always at least 1 children.

Hold on... are all 200 children being spawned on startup? Are they always idle?

Yes & Yes (i think). I started viewing and recording memory usage of the apache pool using ps aux |grep apache. No matter how many requests were being processed (0 requests - 1000 requests) there were always 200 children alive. Don't get me wrong, I love kids but 200 at once for no reason is a bit much.

After spending a few hours screwing with my PHP-FPM conf and running stress tests I came up with this



[www]

user = apache
group = apache

listen = 127.0.0.1:9001

listen.allowed_clients = 127.0.0.1

pm = ondemand
pm.max_children = 200
pm.process_idle_timeout = 1s
pm.max_requests = 1000

I ran another stress test: 2000 RPS (requests-per-second) for one min and the average response time went from 1000MS to 120MS.

What happened?

PHP-FPM's PM (Pool Manager) was spawning 200 children on startup, even though those children were idle the PM was using unnecessary resources to manage the children. The switch from dynamic to ondemand allowed children to be spawned when needed and killed them after 1s of inactivity.

Feel free to critique this post, I have little knowledge of how PHP-FPM's pool manager works. I felt that it might be helpful to someone in a bind with PHP-FPM.

EDIT: Some info in this article is incorrect e.g I confused dynamic with static spawning 200 children on startup. However, I still use this same setup on new servers and it performs so much better than any other config I've used.

Hi, I'm Jake Casto

Jake Casto — Sat, 27 May 2017 14:16:27 +0000

I have been coding for 3 years.

I live in Winston-Salem NC.

I'm a freelance developer.

I mostly program in these languages: [PHP, Python, C#/++].

Nice to meet you.