DEV Community: Nishikanta Ray

S3 Glacier Restore Is Not a Storage-Class Migration

Nishikanta Ray — Fri, 30 Oct 2026 19:42:58 +0000

Amazon S3 offers different storage classes to help reduce storage costs. One of them is S3 Glacier, designed for storing data that you rarely need to access.

But there's an important detail that can cause confusion when working with archived objects:

Restoring an S3 Glacier object does not automatically move it back to S3 Standard.

Let's understand why this happens, the different Glacier storage classes, and how to bring an archived object back to regular storage.

1. What is Amazon S3 Glacier?

Amazon S3 Glacier is a set of storage classes designed for long-term data archiving.

Instead of paying higher storage costs for files that are rarely accessed, you can move them into Glacier storage classes.

For example:

Old user-uploaded images.
Historical reports and documents.
Database backups.
Application logs.
Compliance and audit records.

The trade-off is that some Glacier classes have slower retrieval times and additional retrieval costs.

2. Understanding the three Glacier storage classes

Not every Glacier storage class works the same way.

Storage class	Retrieval time	Best suited for
Glacier Instant Retrieval (`GLACIER_IR`)	Milliseconds	Rarely accessed files that still need immediate access
Glacier Flexible Retrieval (`GLACIER`)	Minutes to hours	Backups and disaster recovery
Glacier Deep Archive (`DEEP_ARCHIVE`)	Hours	Long-term archives and compliance data

A few important differences:

Glacier Instant Retrieval

Objects are immediately accessible through normal S3 GET requests.
No restore request is required.
Designed for data accessed occasionally but requiring immediate availability.
Minimum storage duration: 90 days.

Glacier Flexible Retrieval

Objects are archived and cannot be accessed directly.
You must initiate a restore request before accessing them.
Supports Expedited, Standard, and Bulk retrieval.
Minimum storage duration: 90 days.

Glacier Deep Archive

Designed for data that is rarely accessed.
Requires a restore request before accessing objects.
Supports Standard and Bulk retrieval.
Minimum storage duration: 180 days.

The important distinction is that Instant Retrieval provides immediate access, whereas Flexible Retrieval and Deep Archive require restoration.

3. What actually happens when you restore an object?

This is where the confusion starts.

Imagine an application stores an image in S3. Over time, a lifecycle policy moves that image into Glacier Flexible Retrieval to reduce storage costs.

Later, the application needs that image again.

You initiate a restore request.

AWS processes the request and creates a temporary accessible copy of the archived object. You specify how many days that temporary copy should remain available.

Once restoration completes, you can access the object.

However, the original object's storage class remains GLACIER.

For example, a HEAD request can return:

{
  "StorageClass": "GLACIER",
  "Restore": "ongoing-request=\"false\", expiry-date=\"...\""
}

Let's understand these fields:

StorageClass: GLACIER means the object is still in Glacier Flexible Retrieval.
ongoing-request="false" means restoration has completed.
expiry-date indicates when the temporary restored copy expires.

Restoration makes the object accessible. It does not migrate the object to another storage class.

This distinction is critical when an application expects the object to remain available beyond the restore period.

4. How do you permanently move a restored object back to S3 Standard?

If you need the object to remain available without depending on the temporary restore window, you need another operation.

The solution is to:

Restore the archived object.
Wait until restoration completes.
Copy the restored object to the same bucket and key.
Specify STANDARD as the destination storage class.

This creates a new object using S3 Standard storage.

Step 1: Check whether restoration is complete

Using AWS CLI:

BUCKET=only-for-testing
KEY="letsflo-core/dev/_1793103890540/HH0922_166_11.jpg"

aws s3api head-object \
  --bucket "$BUCKET" \
  --key "$KEY" \
  --query '[StorageClass,Restore]'

Make sure the restore status is:

ongoing-request="false"

Do not attempt the copy before restoration completes.

Step 2: Copy the object onto itself

Once restoration is complete:

aws s3 cp \
  "s3://$BUCKET/$KEY" \
  "s3://$BUCKET/$KEY" \
  --storage-class STANDARD \
  --force-glacier-transfer

Why copy the object to itself?

Because the copy operation creates a new object at the destination, even when the source and destination use the same bucket and key.

By specifying:

--storage-class STANDARD

we tell S3 to store the resulting object in Standard storage instead of Glacier.

Step 3: Verify the storage class

Run the HEAD request again:

aws s3api head-object \
  --bucket "$BUCKET" \
  --key "$KEY" \
  --query '[StorageClass,Restore]'

Expected result:

[
  "STANDARD",
  null
]

S3 Standard is the default storage class, so AWS may omit the StorageClass field.

The temporary restore information should no longer be needed for the new Standard object.

5. Performing the same operation using Node.js

In our case, we also needed to perform this operation through the backend's existing AWS configuration.

Using AWS SDK for JavaScript v3, we can use CopyObjectCommand.

const cfg = require("../backend/src/flo/config/dev.js").aws;
const S = require("@aws-sdk/client-s3");

const client = new S.S3Client({
  region: cfg.region || "eu-west-1",
  credentials: {
    accessKeyId: cfg.accessKeyId,
    secretAccessKey: cfg.secretAccessKey
  },
  followRegionRedirects: true
});

const Bucket = cfg.Bucket;
const Key = "letsflo-core/dev/_1793103890540/HH0922_166_11.jpg";

const head = async (label) => {
  const result = await client.send(
    new S.HeadObjectCommand({ Bucket, Key })
  );

  console.log(label, {
    storageClass: result.StorageClass || "STANDARD",
    restore: result.Restore || "no restore"
  });

  return result;
};

(async () => {
  const before = await head("BEFORE");

  if (
    !before.Restore ||
    !before.Restore.includes('ongoing-request="false"')
  ) {
    throw new Error("Object restoration is not complete.");
  }

  await client.send(
    new S.CopyObjectCommand({
      Bucket,
      Key,
      CopySource: encodeURIComponent(`${Bucket}/${Key}`),
      StorageClass: "STANDARD",
      MetadataDirective: "COPY"
    })
  );

  const after = await head("AFTER");

  if (
    after.StorageClass &&
    after.StorageClass !== "STANDARD"
  ) {
    throw new Error("Storage class verification failed.");
  }

  console.log("Object successfully moved to STANDARD.");
})().catch(error => {
  console.error("ERROR:", error.name, error.message);
  process.exitCode = 1;
});

The implementation performs three important operations:

Checks the existing object's restoration status.
Copies the restored object using the Standard storage class.
Verifies the resulting object's storage class.

For production implementations, also consider multipart copy for objects larger than 5 GB, bucket versioning, object tags, encryption, and permissions.

6. Important things to check before doing this in production

Before changing storage classes, consider a few operational details.

Bucket versioning

If versioning is enabled, copying an object to the same key creates a new version. The previous Glacier version can still exist and incur storage charges.

Lifecycle policies

Check whether lifecycle rules automatically move objects back into Glacier. Otherwise, the newly created Standard object might be archived again.

Storage and retrieval costs

Glacier has minimum storage-duration requirements. Moving or deleting objects before those periods expire can result in early deletion charges.

Restoration also incurs retrieval costs, and the temporary restored copy incurs Standard storage charges during its availability period.

Metadata and encryption

Check whether the copy preserves the metadata, tags, encryption settings, and other properties your application depends on. MetadataDirective: "COPY" preserves user-defined metadata, but production implementations should explicitly verify the required properties.

Conclusion

Amazon S3 Glacier is useful for reducing storage costs, but understanding its restoration behavior is equally important.

Remember these three things:

Glacier Instant Retrieval: Access objects immediately without restoring them.
Glacier Flexible Retrieval and Deep Archive: Restore objects before accessing them.
Need permanent Standard storage? Restore first, then copy the object with STANDARD as its storage class.

A successful restore request doesn't necessarily mean your storage migration is complete.

Always verify the actual storage class rather than relying only on the restoration status.

Official AWS references

How to Divide Work Across Threads in JavaScript: CPU Cores, Workload, and Smart Scheduling

Nishikanta Ray — Mon, 26 Oct 2026 17:00:46 +0000

Modern applications often need to process millions of records, generate reports, analyze events, resize images, or perform complex calculations. As workloads grow, running everything on a single JavaScript thread can become a performance bottleneck.

The solution isn't simply to create more threads. The real challenge is deciding how many threads to create, how to divide the work, and how to assign tasks efficiently based on the machine's available resources.

In this guide, we'll explore how JavaScript multithreading works in Node.js, how to divide workloads across CPU cores, and how to design a worker system that can scale efficiently.

1. Understanding JavaScript's Execution Model

JavaScript traditionally executes application code on a single main thread. In Node.js, this thread runs the event loop, handles callbacks, and coordinates asynchronous operations.

Consider a program that processes a large dataset:

function processData(records) {
  for (const record of records) {
    // Perform expensive calculations
  }
}

processData(largeDataset);

If processData() performs CPU-intensive work, the main thread remains occupied until the function finishes. During that time, other JavaScript callbacks cannot execute on the same thread.

As a result, an application may become unresponsive, requests may take longer to complete, and overall throughput can decrease.

Node.js provides the worker_threads module to run JavaScript in separate threads. These workers can execute CPU-intensive tasks in parallel, subject to the machine's available CPU resources.

The important distinction is that asynchronous programming and multithreading solve different problems:

Asynchronous programming allows a thread to continue while waiting for operations such as network or file I/O.
Multithreading allows JavaScript code to execute on multiple threads, making parallel CPU computation possible.

Understanding this distinction is the first step toward building an efficient worker system.

2. How Does a Machine Determine Its Thread Capacity?

Before distributing work, we need to understand the machine running the application.

Imagine a server with the following specifications:

Resource	Specification
Logical CPUs	8
RAM	16 GB
Workload	CPU-intensive data processing
Main thread	Handles coordination and application logic

The eight logical CPUs represent the hardware's reported capacity for simultaneous thread execution. Depending on the processor, logical CPUs may include simultaneous multithreading, such as Intel Hyper-Threading.

However, eight logical CPUs do not automatically mean that eight worker threads will produce the best performance.

Performance depends on several factors:

Whether the task is CPU-intensive or I/O-intensive.
How much memory each worker requires.
How much work is involved in creating and communicating with workers.
Whether other applications are competing for CPU resources.
Whether the application runs inside a container with CPU restrictions.

The operating system schedules threads across available processors. Multiple threads can exist even when fewer processors are available, but excess runnable threads may compete for CPU time.

Therefore, CPU count is a useful starting point for selecting a worker count, not a universal rule for determining the optimal number of workers.

3. Four Strategies for Dividing Work

There is no single strategy that fits every workload. The right approach depends on the shape of the data and the amount of computation involved.

Strategy 1: Divide work based on CPU availability

For CPU-heavy calculations, the number of available logical CPUs is a reasonable starting point.

Node.js provides os.availableParallelism() to estimate how much CPU parallelism the process can use.

const os = require("node:os");

const availableCPUs = os.availableParallelism();

const workerCount = Math.max(1, availableCPUs - 1);

console.log("Available CPU parallelism:", availableCPUs);
console.log("Initial worker count:", workerCount);

On a machine reporting eight available logical CPUs, this example starts with seven workers.

Leaving capacity for the main thread can be useful, but it is not always necessary or optimal. The main thread might perform very little CPU work, or the application might run alongside other services that require substantial resources.

Benchmarking should determine the final worker count.

Strategy 2: Divide work based on data size

Suppose an application needs to process one million records using four workers.

A straightforward approach assigns approximately 250,000 records to each worker.

The general formula is:

chunkSize = ceil(totalItems / workerCount)

This approach is known as static partitioning because the work is divided before processing begins.

It works particularly well when each item requires roughly the same amount of computation.

However, equal numbers of records do not always mean equal amounts of work. For example, processing a record with two fields may take far less time than processing one containing a large nested object.

When processing costs vary significantly, static partitioning can leave some workers idle while others continue working.

Strategy 3: Divide work into smaller tasks

Instead of assigning one large block to each worker, an application can divide the workload into smaller jobs.

For example, rather than assigning 250,000 records to each worker, it could create batches of 10,000 records.

Workers retrieve additional batches as they finish their current tasks.

This is dynamic scheduling.

It is particularly useful when:

Tasks have unpredictable execution times.
Records vary in complexity.
Some jobs occasionally fail or need retries.
The application processes a continuous stream of incoming work.

Dynamic scheduling introduces additional overhead for managing the task queue, so the batches should not be so small that coordination becomes more expensive than the computation.

Strategy 4: Divide work based on resource limits

CPU availability is only one constraint. Memory can become the limiting factor.

Suppose every worker needs 500 MB of additional memory for its data and intermediate calculations. Creating 20 workers could require roughly 10 GB of additional memory, before accounting for the main application, runtime overhead, and other processes.

A practical worker system should consider:

CPU availability.
Memory available to the process.
Input and output data size.
Queue length.
Task duration.
Other application activity.

Resource-aware scheduling is especially important in production services and containerized environments.

4. A Complete Example: Distributing Work Across Node.js Workers

Let's build a working example that calculates the sum of all integers from 1 to 100 million.

The goal is to divide the calculation into separate ranges, assign each range to a worker, and combine the results.

Project structure

parallel-calculation/
├── main.js
└── worker.js

Step 1: Create the worker

Create worker.js:

const {
  parentPort,
  workerData
} = require("node:worker_threads");

const { start, end } = workerData;

let sum = 0;

for (let i = start; i <= end; i++) {
  sum += i;
}

parentPort.postMessage(sum);

Each worker receives a start and end value, calculates the sum of its assigned range, and sends the result back to the main thread.

Step 2: Create the main program

Create main.js:

const os = require("node:os");

const {
  Worker
} = require("node:worker_threads");

const TOTAL = 100_000_000;

const availableCPUs = os.availableParallelism();

// Choose a starting worker count
const workerCount = Math.max(1, availableCPUs - 1);

const chunkSize = Math.ceil(TOTAL / workerCount);

function runWorker(start, end) {
  return new Promise((resolve, reject) => {
    const worker = new Worker("./worker.js", {
      workerData: { start, end }
    });

    worker.once("message", resolve);
    worker.once("error", reject);

    worker.once("exit", (code) => {
      if (code !== 0) {
        reject(
          new Error(`Worker exited with code ${code}`)
        );
      }
    });
  });
}

async function main() {
  console.log("Available CPUs:", availableCPUs);
  console.log("Workers:", workerCount);

  const jobs = [];

  for (let i = 0; i < workerCount; i++) {
    const start = i * chunkSize + 1;

    const end = Math.min(
      (i + 1) * chunkSize,
      TOTAL
    );

    if (start <= end) {
      console.log(`Assigning range ${start}-${end}`);

      jobs.push(runWorker(start, end));
    }
  }

  const results = await Promise.all(jobs);

  const totalSum = results.reduce(
    (total, result) => total + result,
    0
  );

  console.log("Final sum:", totalSum);
}

main().catch(console.error);

Step 3: Run the program

Install Node.js if necessary, then run:

node main.js

The program detects available CPU parallelism, calculates a chunk size, launches workers, and waits for all results.

The expected result is:

Final sum: 5000000050000000

The output also displays the CPU count, worker count, and assigned ranges.

This example demonstrates the essential workflow:

Detect available CPU parallelism.
Determine the worker count.
Divide the input into ranges.
Run the calculations in separate threads.
Collect the results.
Combine the partial results.

For this particular arithmetic problem, a direct mathematical formula would be much faster than either implementation. The example is intended to demonstrate work partitioning, not to suggest that creating workers is always the fastest solution.

5. Static Partitioning vs. Dynamic Scheduling

The previous example uses static partitioning: each worker receives a predetermined range.

Consider a different scenario where an application processes customer analytics events. Some events contain simple properties, while others require expensive transformations.

Assigning the same number of events to every worker might not produce balanced execution times.

A dynamic task queue can solve this problem.

Conceptually, the system works like this:

Incoming tasks
      |
      v
    Queue
   /  |  \
  v   v   v
 W1  W2  W3
  |   |   |
  +---+---+
      |
      v
  Completed results

Whenever a worker finishes its current task, it takes another task from the queue.

This approach is often called work sharing. It can improve utilization when task durations vary, although the queue itself must be designed to avoid becoming a bottleneck.

A production implementation might also include retry policies, cancellation, timeouts, task priorities, and backpressure to prevent incoming work from exceeding processing capacity.

6. Why Worker Pools Are Better Than Creating a Thread for Every Task

Creating a worker involves initialization, communication setup, and resource allocation. If an application creates a new worker for every small task, this overhead can become significant.

A worker pool keeps a fixed number of workers alive and reuses them for multiple tasks.

For example, a pool might maintain four workers:

Worker pool
├── Worker 1 → Task A → Task E
├── Worker 2 → Task B → Task F
├── Worker 3 → Task C → Task G
└── Worker 4 → Task D → Task H

Tasks can wait in a queue until a worker becomes available.

A worker pool can provide:

Controlled concurrency.
Reduced worker initialization overhead.
More predictable resource usage.
Centralized error handling.
Better throughput for repeated CPU-heavy jobs.

In Node.js, worker pools can be implemented using worker_threads or libraries such as Piscina.

The pool size should still be selected according to the workload and measured performance.

7. How to Find the Optimal Number of Workers

Suppose a workload runs on a machine reporting eight available logical CPUs.

You could benchmark it using different worker counts:

Worker count	What to evaluate
1	Baseline performance
2	Benefit from initial parallelism
4	Throughput and CPU utilization
6	Additional scaling
7	Potential balance with main-thread work
8	Whether using all available parallelism helps

Measure the time required to complete the same workload, as well as throughput, CPU utilization, memory usage, and latency.

For a CPU-intensive task, adding workers can reduce execution time until the workload becomes limited by processor capacity, memory bandwidth, synchronization, or other overhead.

Beyond that point, additional workers may offer little improvement or even reduce performance.

Also remember that worker threads are not independent physical CPUs. They share the machine's underlying resources, and CPU scheduling can affect their performance.

The best configuration is the one that meets your performance goals within your resource budget.

8. Common Mistakes to Avoid

Creating too many workers

More workers do not always mean faster execution. Excessive concurrency can increase memory consumption and scheduling overhead.

Assuming every task takes the same time

Equal-sized chunks work best when processing costs are reasonably uniform. For uneven tasks, dynamic scheduling can improve load balancing.

Using workers for ordinary I/O operations

Network requests and many file operations can already be handled asynchronously by Node.js. Workers are most valuable when substantial CPU-intensive JavaScript work would otherwise block the main thread.

Ignoring data-transfer costs

Workers normally communicate using messages, which can involve copying data. Large data transfers can offset the benefits of parallel computation. Transferable objects and shared memory can help in appropriate cases, but they require careful ownership and synchronization.

Ignoring memory limits

Every worker adds runtime overhead and may allocate its own data. A worker count that works on a development laptop might be inappropriate in a production container with less memory or CPU capacity.

9. Final Architecture: A Resource-Aware Worker System

For a production application, a robust design generally combines several of these techniques:

Detect the available CPU parallelism.
Establish a configurable upper limit for workers.
Estimate the memory and processing cost of tasks.
Divide work into appropriately sized batches.
Use a worker pool to reuse threads.
Use dynamic scheduling when tasks vary significantly in duration.
Measure throughput, latency, CPU usage, and memory consumption.
Adjust the configuration based on observed performance.

The result is a system that distributes work according to both the machine's capabilities and the workload's characteristics.

Conclusion

Multithreading is not just about creating more threads. It is about distributing work efficiently across the resources available to your application.

CPU-based partitioning provides a useful starting point. Data partitioning works well for uniform workloads. Dynamic scheduling helps balance uneven tasks, while resource-aware limits prevent excessive CPU and memory consumption.

For JavaScript developers, Node.js worker_threads provides the foundation for parallel CPU-intensive execution. Combining it with chunking, worker pools, and performance measurement can turn a single-threaded bottleneck into a more scalable processing system.

The key principle is simple: choose the worker count based on available resources, divide work according to its computational cost, and let real performance measurements guide optimization.

How Machines Find Meaning: The Math Behind Semantic Search

Nishikanta Ray — Sun, 25 Oct 2026 16:51:16 +0000

Understanding cosine similarity, Euclidean distance, and how machines find relevant text and images.

1. Why do we need semantic search?

Imagine you're building a search feature for your application.

A user searches:

How do I make my application faster?

Your database contains these articles:

How to optimize database queries
Improving API performance with caching
A beginner's guide to JavaScript arrays
How to bake a chocolate cake

Which articles should appear first?

Traditional keyword search looks for matching words. Since the first two articles don't contain the exact phrase "make my application faster," a basic keyword search might rank them poorly.

Semantic search takes a different approach: it searches for meaning, not just matching words.

It can recognize that "make my application faster," "optimize database queries," and "improve API performance" may describe related ideas.

To understand how semantic search works, we need three concepts:

Embeddings: Representing text or images as numbers.
Vectors: Working with those numbers mathematically.
Similarity and distance: Comparing vectors to rank relevant results.

Let's build these concepts from scratch using simple mathematics and JavaScript.

2. What is a vector?

A vector is an ordered list of numbers.

For example:

const vector = [3, 4];

We can imagine this vector as a point on a two-dimensional graph:

3 represents the horizontal coordinate.
4 represents the vertical coordinate.

The point is therefore located at (3, 4).

A vector can have two dimensions, three dimensions, or thousands of dimensions.

For learning purposes, we'll use two-dimensional vectors because they're easy to visualize and calculate by hand.

Important: These examples use illustrative numbers. Real embedding dimensions generally don't have simple, individually interpretable meanings.

3. What is an embedding?

An embedding is a numerical representation of content generated by a machine-learning model.

Imagine an embedding model converts three sentences into vectors:

Sentence	Illustrative embedding
"Make my app faster"	`[0.9, 0.1]`
"Improve application performance"	`[0.8, 0.2]`
"Chocolate cake recipe"	`[0.1, 0.9]`

The first two vectors point in similar directions. The third points in a different direction.

This gives us a way to compare content mathematically.

The embedding model learns useful representations during training. Our search system uses those representations to find and rank related content.

A simplified semantic-search pipeline looks like this:

User query
    |
    v
Embedding model
    |
    v
Query vector
    |
    v
Compare with document vectors
    |
    v
Rank relevant documents

Now let's explore the mathematics behind those comparisons.

4. Vector magnitude: how long is a vector?

Before comparing vectors, let's understand their length, also called their magnitude or L2 norm.

Suppose we have:

a = [3, 4]

The magnitude is calculated using the Pythagorean theorem.

Step 1: Square both coordinates.

‖a‖₂ = √(3² + 4²)

Step 2: Simplify the squares.

‖a‖₂ = √(9 + 16)

Step 3: Add the values and calculate the square root.

‖a‖₂ = √25 = 5

Therefore, the magnitude of vector [3, 4] is 5.

In simple terms, the magnitude tells us how far the point is from the origin, (0, 0).

Implementing magnitude in JavaScript

function magnitude(vector) {
  const sumOfSquares = vector.reduce(
    (sum, value) => sum + value * value,
    0
  );

  return Math.sqrt(sumOfSquares);
}

console.log(magnitude([3, 4])); // 5

Here's what the code does:

Squares every value.
Adds the squared values together.
Takes the square root of the sum.

We'll reuse this function when calculating cosine similarity.

5. Euclidean distance: how far apart are two vectors?

Magnitude measures the distance from the origin. Euclidean distance measures the distance between two points.

Suppose we have two points.

Point A

a = [1, 2]

Point B

b = [4, 6]

How far apart are they?

First, calculate the difference between their coordinates:

Horizontal difference: 4 − 1 = 3
Vertical difference: 6 − 2 = 4

Using the Pythagorean theorem:

d = √(3² + 4²)

d = √(9 + 16)

d = √25 = 5

The Euclidean distance is 5 units.

The general L2 distance formula

For two vectors with the same number of dimensions:

dL2(a, b) = √[Σᵢ₌₁ⁿ (aᵢ − bᵢ)²]

You don't need to memorize the notation immediately. Remember the process:

Subtract corresponding coordinates.
Square each difference.
Add the squared differences.
Take the square root.

Implementing L2 distance in JavaScript

function l2Distance(a, b) {
  if (a.length !== b.length) {
    throw new Error("Vectors must have equal dimensions");
  }

  const sumOfSquares = a.reduce((sum, value, i) => {
    const difference = value - b[i];
    return sum + difference * difference;
  }, 0);

  return Math.sqrt(sumOfSquares);
}

const a = [1, 2];
const b = [4, 6];

console.log(l2Distance(a, b)); // 5

How does this help with semantic search?

Imagine our query has already been converted into a vector:

const query = [2, 3];

We also have two document embeddings:

const documentA = [4, 4];
const documentB = [2, 5];

Let's calculate the distance from the query to each document.

Document A

Distance = √((2 − 4)² + (3 − 4)²)

Distance = √(4 + 1)

Distance ≈ 2.236

Document B

Distance = √((2 − 2)² + (3 − 5)²)

Distance = √(0 + 4)

Distance = 2

Document B is closer to the query because 2 is smaller than 2.236.

When ranking by L2 distance, a smaller distance means the vectors are closer together.

However, Euclidean distance isn't the only way to compare vectors. Another popular metric is cosine similarity.

6. Dot product: multiplying corresponding values

The dot product is an operation used in vector mathematics. It also plays an important role in calculating cosine similarity.

Suppose we have two vectors:

a = [2, 3]

b = [4, 5]

Multiply their corresponding values and add the results:

a · b = (2 × 4) + (3 × 5)

a · b = 8 + 15 = 23

The dot product is 23.

Implementing dot product in JavaScript

function dotProduct(a, b) {
  if (a.length !== b.length) {
    throw new Error("Vectors must have equal dimensions");
  }

  return a.reduce(
    (sum, value, i) => sum + value * b[i],
    0
  );
}

console.log(dotProduct([2, 3], [4, 5])); // 23

The dot product combines information from corresponding dimensions. When divided by the magnitudes of both vectors, it gives us the cosine of the angle between them.

That is the foundation of cosine similarity.

7. Cosine similarity: are the vectors pointing in the same direction?

Euclidean distance asks:

How far apart are two points?

Cosine similarity asks:

How closely do two vectors point in the same direction?

Its formula is:

cosineSimilarity(a, b) = (a · b) / (‖a‖₂ × ‖b‖₂)

Let's work through an example.

Suppose:

a = [1, 2]

b = [2, 4]

Notice that vector B is twice vector A. Both point in exactly the same direction.

Step 1: Calculate the dot product

a · b = (1 × 2) + (2 × 4) = 10

Step 2: Calculate both magnitudes

For vector A:

‖a‖₂ = √(1² + 2²) = √5

For vector B:

‖b‖₂ = √(2² + 4²) = √20

Step 3: Calculate cosine similarity

cosineSimilarity = 10 / (√5 × √20)

Since √5 × √20 = 10:

cosineSimilarity = 10 / 10 = 1

The cosine similarity is 1, meaning the vectors point in exactly the same direction.

Interpreting cosine similarity

Score	Geometric interpretation
`1`	Same direction
`0.8`	Strong positive directional alignment
`0`	Perpendicular directions
`-1`	Opposite directions

These are geometric interpretations, not universal thresholds for semantic relevance. Whether a score indicates similar meaning depends on the embedding model and the data.

Implementing cosine similarity in JavaScript

function magnitude(vector) {
  return Math.sqrt(
    vector.reduce((sum, value) => sum + value * value, 0)
  );
}

function dotProduct(a, b) {
  if (a.length !== b.length) {
    throw new Error("Vectors must have equal dimensions");
  }

  return a.reduce(
    (sum, value, i) => sum + value * b[i],
    0
  );
}

function cosineSimilarity(a, b) {
  const magnitudeA = magnitude(a);
  const magnitudeB = magnitude(b);

  if (magnitudeA === 0 || magnitudeB === 0) {
    throw new Error(
      "Cosine similarity is undefined for zero vectors"
    );
  }

  return dotProduct(a, b) / (magnitudeA * magnitudeB);
}

console.log(cosineSimilarity([1, 2], [2, 4])); // 1
console.log(cosineSimilarity([1, 0], [0, 1])); // 0
console.log(cosineSimilarity([1, 0], [-1, 0])); // -1

Cosine similarity is useful because multiplying a nonzero vector by a positive number doesn't change its direction. Consequently, cosine similarity is unaffected by positive scaling of either vector.

This can be helpful when comparing embeddings whose lengths differ.

8. Cosine distance: turning similarity into distance

Sometimes we want a distance metric where lower values indicate a closer match.

A common definition is:

cosineDistance = 1 − cosineSimilarity

Suppose Document A has a cosine similarity score of 0.9.

Distance A = 1 − 0.9 = 0.1

Document B has a cosine similarity score of 0.6.

Distance B = 1 − 0.6 = 0.4

Document A is the closer match under this metric because its cosine distance is smaller.

Implementing cosine distance

function cosineDistance(a, b) {
  return 1 - cosineSimilarity(a, b);
}

console.log(cosineDistance([1, 2], [2, 4])); // 0
console.log(cosineDistance([1, 0], [0, 1])); // 1

Remember the distinction:

Cosine similarity: Higher is better for directional alignment.
Cosine distance: Lower is better.
L2 distance: Lower means the points are closer.

Implementations and vector databases may define or report metrics differently, so check the documentation for the system you're using.

9. L2 distance vs. cosine similarity

Let's compare two vectors:

a = [1, 1]

b = [10, 10]

These vectors point in exactly the same direction, but their magnitudes differ.

Their cosine similarity is:

cosineSimilarity(a, b) = 1

Their cosine distance is:

cosineDistance = 1 − 1 = 0

Now calculate their Euclidean distance:

dL2 = √((1 − 10)² + (1 − 10)²)

dL2 = √(81 + 81)

dL2 = √162 ≈ 12.728

The vectors have perfect directional alignment, yet their Euclidean distance is greater than 12.

This illustrates the key difference:

Property	L2 distance	Cosine similarity
Measures	Distance between coordinates	Alignment of directions
Affected by vector magnitude?	Yes	Not by positive scaling
Better score	Smaller	Larger
Useful when	Coordinate distance matters	Directional alignment matters

Neither metric is always better. The right choice depends on how the embedding model was trained and how its vectors should be compared.

10. Build a simple semantic-ranking example

Now let's combine our functions to build a small vector-ranking demonstration in JavaScript.

Imagine an embedding model has already converted a search query and three documents into two-dimensional vectors.

const query = [0.8, 0.6];

const documents = [
  {
    title: "API performance optimization",
    embedding: [0.9, 0.4]
  },
  {
    title: "Database query optimization",
    embedding: [0.7, 0.7]
  },
  {
    title: "Chocolate cake recipe",
    embedding: [-0.5, 0.8]
  }
];

These vectors are invented for demonstration. In a real application, an embedding model would generate them.

Let's rank the documents by cosine similarity.

function magnitude(vector) {
  return Math.sqrt(
    vector.reduce((sum, value) => sum + value * value, 0)
  );
}

function cosineSimilarity(a, b) {
  if (a.length !== b.length) {
    throw new Error("Vectors must have equal dimensions");
  }

  const dot = a.reduce(
    (sum, value, i) => sum + value * b[i],
    0
  );

  const normA = magnitude(a);
  const normB = magnitude(b);

  if (normA === 0 || normB === 0) {
    throw new Error("Zero vectors are not supported");
  }

  return dot / (normA * normB);
}

const rankedDocuments = documents
  .map(document => ({
    title: document.title,
    score: cosineSimilarity(query, document.embedding)
  }))
  .sort((a, b) => b.score - a.score);

console.log(rankedDocuments);

The results are approximately:

[
  {
    title: "Database query optimization",
    score: 0.990
  },
  {
    title: "API performance optimization",
    score: 0.975
  },
  {
    title: "Chocolate cake recipe",
    score: 0.085
  }
]

We sort the results in descending order because higher cosine similarity means stronger directional alignment.

Notice something important: our JavaScript code doesn't understand language. It only compares vectors.

The embedding model is responsible for learning useful representations of language. The ranking code uses those representations to order the results.

Our example demonstrates the mechanics of vector search, not a complete semantic search system.

11. Where do real embeddings come from?

So far, we've manually supplied vectors such as [0.8, 0.6].

Real applications use an embedding model to generate vectors from text or images.

A typical workflow looks like this:

Choose an embedding model.
Convert each document into a vector.
Store the vectors alongside document IDs and content.
Convert the user's query into a vector using a compatible model.
Find the nearest document vectors.
Return the corresponding documents in ranked order.

A JavaScript application might call an embedding API like this:

// Illustrative pseudocode.
// Replace this with your embedding provider's SDK.

const queryEmbedding = await embeddingModel.embed(
  "How do I make my application faster?"
);

const documentEmbedding = await embeddingModel.embed(
  "Improve API performance with caching."
);

const score = cosineSimilarity(
  queryEmbedding,
  documentEmbedding
);

console.log(score);

The exact API depends on your embedding provider. The key requirement is that both vectors must come from compatible embedding representations.

For thousands or millions of documents, calculating every distance in application code can become expensive. A vector database or search engine can index embeddings and retrieve nearby candidates more efficiently.

12. Does the same mathematics work for image search?

Yes. The comparison mathematics is the same, but the process that generates the vectors differs.

Imagine you're building an image-search application. A user enters:

A dog playing on a beach.

A compatible text-image model converts the query into an embedding. It also converts the available images into embeddings in a shared vector space.

The application compares the query vector against the image vectors and ranks the images by similarity.

The process looks like this:

Text query
    |
    v
Text encoder
    |
    v
Query embedding
    |
    v
Compare against image embeddings
    |
    v
Rank matching images

This works directly only when the model is designed to produce comparable text and image embeddings.

Why not simply compare raw pixels?

Two photographs of the same dog might have different lighting, camera angles, cropping, or backgrounds. Raw pixel differences may be large even when the images depict similar content.

Learned image embeddings can capture useful visual patterns that raw pixel distance misses.

However, the model can still make mistakes. A high similarity score is a ranking signal, not proof that an image contains every detail in the query.

13. Why normalized vectors matter

Some embedding models produce vectors with a magnitude of 1. These are called L2-normalized vectors.

For unit-normalized vectors, Euclidean distance and cosine similarity are mathematically related:

L2 distance² = 2 × (1 − cosine similarity)

Equivalently:

L2 distance = √[2 × (1 − cosine similarity)]

This means that ranking the same unit-normalized vectors by L2 distance or cosine similarity produces the same ordering:

Higher cosine similarity means lower L2 distance.
Lower cosine similarity means higher L2 distance.

This relationship can be useful when choosing a metric for a vector index.

Don't assume every embedding model automatically normalizes its output. Check the model documentation before relying on this property.

14. What would I use in a real JavaScript application?

For a practical semantic-search feature, I'd separate the system into four responsibilities.

1. Embedding model

Converts text or images into vectors that represent useful features of the content.

2. Vector database

Stores the vectors and retrieves nearby candidates efficiently.

3. Similarity metric

Determines how the system compares vectors, using cosine similarity, Euclidean distance, or another supported metric.

4. Application code

Handles queries, metadata, filtering, ranking, and displaying results to users.

Start with the metric recommended by your embedding model, then test it against real queries and expected results.

For example, if you're building documentation search, create a test set of queries such as "make API faster" and identify which articles should appear first. Measure whether those articles show up in the top five results.

That gives you a practical way to evaluate your search quality rather than choosing a metric based only on intuition.

You may also want hybrid search, which combines vector similarity with keyword search. This can be especially useful when users search for exact identifiers, error messages, version numbers, or technical terms that must match precisely.

Conclusion

Semantic search becomes much easier to understand when we break it into smaller mathematical concepts.

A vector is an ordered list of numbers. An embedding model converts text or images into vectors. Euclidean distance measures how far apart two vectors are, while cosine similarity measures how closely their directions align. Cosine distance converts similarity into a distance-style score.

With a few JavaScript functions, we can implement these calculations and build a simple vector-ranking demonstration.

The most important distinction is this:

Vector mathematics gives us a way to compare representations. The embedding model learns representations that can capture useful relationships in meaning.

Both parts matter when building a useful semantic-search system.

Building InsightTrack with SerpApi for the SerpApi India Hackathon

Nishikanta Ray — Sat, 03 Oct 2026 15:35:40 +0000

A few weeks ago, I was looking at a page whose traffic had suddenly dropped.

The analytics told me exactly what happened:

Traffic was down.

But then I had the question that analytics couldn't answer:

Why?

Did the page lose its Google ranking?
Did a competitor move above me?
Did Google change the search results?
Did an AI Overview start answering the query?

My next step was usually pretty manual. I'd open Google, search the keyword, check my position, look at the competitors, and try to figure out what changed.

That got me thinking:

What if I could connect my traffic data with the search results automatically?

That's how I started building InsightTrack.

Connecting traffic with search

The idea behind InsightTrack is pretty simple.

I already have one side of the story from my analytics:

/guides/email-templates

Last week: 1,240 visits
This week:   719 visits

Now I need the other side.

What happened in Google?

That's where SerpApi comes in.

For the SerpApi India Hackathon, I'm integrating SerpApi with InsightTrack to collect SERP data and track how search visibility changes over time.

For example:

"free email templates"

Last week: #3
This week: #9

Now the two pieces can be connected.

Instead of simply saying:

“Traffic dropped 42%.”

InsightTrack can say:

“Traffic dropped because the page moved from #3 to #9, while other websites moved above it.”

That's a much more useful answer.

I didn't want it to always invent a reason

This became one of the most important parts of the project.

Imagine traffic drops by 30%, but the ranking stays exactly the same.

A system that always needs to produce an explanation might blame search anyway.

I didn't want that.

InsightTrack uses deterministic rules to compare traffic and SERP changes.

If the evidence points to a ranking change, it reports that.

If a competitor moved above you, it can identify that.

But if search didn't really change, it can say:

“Search visibility didn't change enough to explain this. Check your referrers, campaigns, or recent deployment.”

The goal isn't to always have an answer.

The goal is to have an answer that can be explained.

A small example

Suppose we have:

Traffic:       -42%
Ranking:       #3 → #9
Competitor:    moved above us
AI Overview:   appeared

InsightTrack combines those observations and produces a finding instead of making me investigate each piece manually.

Under the hood, the attribution engine is just a pure function:

traffic data + keyword findings → finding

No database or API calls inside that layer.

That makes it easy to test and, more importantly, makes the result reproducible.

Then I added MCP

I also wanted this information to be accessible to AI agents.

So InsightTrack exposes the same functionality through MCP.

Now I can ask:

“Who is ranking above me for ‘free email templates’, and why did they overtake me?”

The Gemini API handles the natural-language interaction, while InsightTrack provides the underlying search and traffic data.

The AI doesn't decide the attribution rules.

It asks the tools for the evidence.

The unexpected challenge: search credits

One thing I underestimated was API usage.

Every SERP check costs credits.

If I have 8 keywords and check them every day, that's roughly:

8 × 30 = 240 searches/month

So the search budget became part of the product design.

Instead of making users calculate everything themselves, InsightTrack can work out a collection schedule based on their available credits.

That was a good reminder that API limits aren't just infrastructure problems. They can affect the product itself.

What I'm building next

I want to make the search history more useful over longer periods, improve competitor tracking, and expand AI Overview monitoring.

Because search is changing.

A page can keep the same traditional ranking while the actual search experience around it changes dramatically.

That's the gap I'm trying to explore with InsightTrack.

Analytics tells you what changed.

SerpApi tells you what changed in search.

InsightTrack connects the two.

Built for the SerpApi India Hackathon.

🔗 insightstrack.dev

GitHub: github.com/NishikantaRay/InsightTrack

Built with: SerpApi · PostgreSQL · DuckDB · Node.js · React · MCP · Gemini API

8 LLMs, 480 Questions, 1 Kaggle Benchmark: Who Can Explain a Traffic Drop?

Nishikanta Ray — Sun, 27 Sep 2026 18:42:53 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

The short version

I build InsightTrack, an open-source web analytics
platform. It has an AI analyst called Pulse. You ask "why did traffic to /pricing drop last
week?", and Pulse picks the right tools, reads the numbers, and answers in plain English.

Pulse needs a language model behind it. I wanted to choose that model with evidence, not vibes, so I
turned Pulse's real job into a benchmark: 480 questions, graded automatically, with answers
that come from InsightTrack's own code.

The problem: an AI analyst can be confidently wrong

An analytics assistant isn't a chatbot for fun facts. People act on what it says: they rewrite pages,
pause campaigns, or stop worrying. When it's wrong, it's wrong in one of two costly ways:

It invents a reason. Traffic moves for a boring reason (a holiday, a newsletter, plain noise), and the assistant confidently blames Google rankings. You spend a day fixing SEO that was never broken.
It misses a real problem. Traffic falls by half, and the assistant says "nothing significant happened". You find out a month later.

Both are worse than saying "I don't know". A useful analyst needs three habits that don't show up on
general AI leaderboards:

Restraint: saying "this is noise" or "the data doesn't show that" when that's the truth.
Rule-following on numbers: applying thresholds exactly. A 15% change on 6 visitors is not a trend.
Reading real tool output: the exact fields, rounding and formats a product actually returns, not textbook examples.

Why general benchmarks don't answer this

Public benchmarks are excellent at telling you which model knows more maths, writes better code, or
calls functions correctly. But choosing a model for a product is a product decision, and it has
three questions they can't answer for you:

Accuracy on my job: my tools, my data formats, my database dialect (DuckDB, not PostgreSQL).
The kind of mistakes: a model that is wrong 10% of the time by saying "not sure" is safer than one that is wrong 5% of the time by making things up.
Cost: is the expensive model worth it, or does a model at a tenth of the price do the same job?

The only way to answer those is to test models on the actual work. That's what this benchmark does.

What's in it

InsightTrack Analyst Bench has 480 auto-graded cases across four tasks. The first three are Pulse's job today; the fourth, SQL, is the step Pulse doesn't take yet: writing a query when no built-in tool fits.

Task	What Pulse must do	The model gets	It's right when
Tool choice	Decide how to answer	A user question + InsightTrack's real 23-tool catalogue	It picks the right tool and arguments, or says "none" when no tool can do it
Data reading	Understand a result	A tool result in InsightTrack's real output format	It gets the number right, or says `not_available` instead of inventing one
Diagnosis	Explain a change	8 weeks of page traffic + Google results before and after	It says whether the change is real, which way it went, and the exact set of causes
SQL	Dig deeper (next on Pulse's roadmap)	InsightTrack's DuckDB `events` and `sessions` tables	Its query, executed read-only, returns the right rows

Each task has 80 standard + 40 hard cases. The hard ones are where careless reasoning shows:

changes that land exactly on a threshold (a 15.0% drop)
several keywords pointing in different directions
pages that stopped ranking entirely
lookalike domains: getsite.com is a different site, but www.site.com is yours

One case, step by step

Here's what a diagnosis question looks like, simplified:

Weekly pageviews: … 495, 517, 347. The last week dropped by 170 views, or 33%.
Google, before: the page ranked #3 for its keyword.
Google, now: it ranks #5. Nothing else changed.
The rules are all in the prompt. A change counts only if it's at least 15% and at least 20 views. A 1–2 place rank move is normal jitter, never a cause.

The correct answer: the drop is real, but search didn't cause it. In InsightTrack's terms,
{"significant": true, "direction": "drop", "causes": ["unexplained_by_serp"]}. The tempting wrong
answer is "your rank dropped". A model that says that sends the user to fix SEO that isn't broken.

That's the heart of the benchmark: can the model tell a real cause from a coincidence?

Why the answers can be trusted

No LLM grades another LLM:

Diagnosis: the answer key is InsightTrack's own correlation engine, ported to Python. A test runs the original JavaScript on every case plus 1,500 random ones and fails on any disagreement.
SQL: answers are graded by executing the model's query, so two different but correct queries both pass.
Everything else: answers are computed from the same data the model sees.
Data: all synthetic (5,940 events, 2,494 sessions), with no real person's data, and the whole set rebuilds byte-for-byte.

Every grader also labels how a model was wrong: invented a cause, called a real change noise,
made up a number, refused an answerable question, used a tool that doesn't exist. That turns a
score into something you can act on.

Who this helps

Developers building AI features on analytics or BI data. You face the same choice I did. These results, and the failure types above, show what to expect from each class of model.
Anyone picking a model on a budget. The results put a price on accuracy: where paying more buys a lot, and where it buys nothing.
People who build and evaluate models. Most tests reward knowing things. This one rewards not claiming things, which is harder to measure and matters in real products.
InsightTrack's users. Pulse gets its model and its safety checks from evidence instead of guesswork.

Models Tested

All models ran through Kaggle's model proxy on the same task version. I picked them in pairs, so each
comparison answers a practical question:

Frontier vs. cheap, same company: Claude Sonnet 5 vs. Claude Haiku 4.5, and Gemini 3.7 Flash vs. Gemini 3.5 Flash-Lite. How much do you lose by saving money?
Reasoning on vs. off, same model: Grok 4.20 with and without reasoning. Does "thinking" change the answer?
The cheapest option: GPT-5.4 nano. Is "good enough" good enough?
OpenAI's flagship: GPT-5.5, as a reference point.

Findings

Model	Tool choice	Data reading	Diagnosis	SQL	Overall	Cost (480 cases)
Claude Sonnet 5	100	100	100	100	100.0	$2.08
GPT-5.5	100	100	96.7	100	99.2	$3.23
Grok 4.20 (reasoning)	100	96.7	100	95.0	97.9	$1.21
Gemini 3.7 Flash	100	100	100	90.8	97.7	$1.07
Gemini 3.5 Flash-Lite	98.3	88.3	49.2	88.3	81.0	$0.20
Claude Haiku 4.5	98.3	82.5	44.2	86.7	77.9	$0.63
Grok 4.20 (no reasoning)	98.3	82.5	35.0	92.5	77.1	$0.40
GPT-5.4 nano	93.3	61.7	20.0	93.3	67.1	$0.07

1. Tool choice is solved. Diagnosis is where models split.

Every model picked the right tool at least 93% of the time. So "can it call tools?" is no longer the
interesting question for an analytics assistant. The real question is whether it understands what the
numbers mean. On diagnosis, the top three models score 100%, GPT-5.5 scores 96.7%, and every cheaper
model scores between 20% and 49%.

2. Same model, reasoning on vs. off: 100% vs. 35%

Grok 4.20 with reasoning got every diagnosis case right. The same model without reasoning got
35%. It lost all 16 multi-cause cases, and 9 of the 12 decoys. Nothing else in the results moved this
much. If your assistant answers "why did X change?", reasoning isn't a nice-to-have.

3. Cheap models fail in two opposite ways

The alarmist. Without reasoning, Grok invented a search cause 34 times. In one case, traffic
rose from 437 to 538 views while the page slipped from #3 to #5 on Google. The prompt says plainly
that a 1–2 place move is normal jitter and never a cause. Grok answered:

{"significant": true, "direction": "spike", "causes": ["rank_drop"]}

It blamed a rank drop for a traffic rise. The right answer was unexplained_by_serp: this change
didn't come from search.

The shrugger. GPT-5.4 nano went the other way, calling real changes noise 73 times. A page fell
from 209 to 110 views (−47%) right after sliding from #14 to #26 on Google. Nano answered:

{"significant": false, "direction": null, "causes": []}

For an analytics product, this is the dangerous one. An alarmist wastes your afternoon. A shrugger
tells you everything is fine while your traffic halves.

4. Mixed signals break every cheaper model: 0 of 8

In the hard cases, one keyword lost rank while another gained, and traffic fell. The correct answer
names only the loss, since a gain can't explain a drop. Flash-Lite, Haiku, Grok without reasoning, and
nano all scored 0 out of 8. Haiku, for instance, blamed the drop on ai_citation_gained, a rule the
prompt says applies only when traffic rises. Real pages rank for many keywords at once, so this is
the everyday case, not an edge case.

5. Reading data: the cheaper models refuse and invent

On data reading, all 7 models correctly said not_available on the 24 standard questions the data
couldn't answer. The traps in the hard set split them:

Made-up numbers: asked for the total unique visitors across a list of pages (impossible, because the same person can visit several pages), nano confidently answered 9,149.
Refused answerable questions: nano did this 13 times, and the other cheaper models 5–7 times each. They answered "not available" when the answer took one calculation.

A model that can't tell "I can't compute this" from "this can't be computed" isn't safe in either direction.

6. The SQL failures were habits, not logic

Gemini 3.7 Flash scored 0/7 on JSON questions, and nearly everything else it got right. Every time,
it wrote:

WHERE properties->>'name' = 'signup'

That's fine in PostgreSQL. In DuckDB, ->> binds more loosely than =, so it parses as
properties ->> ('name' = 'signup') and crashes. One pair of parentheses fixes it. It looks like a
habit carried over from PostgreSQL, which is far more common in examples than DuckDB.

Claude Haiku corrects itself mid-answer. In 9 SQL replies it wrote a query, then "Wait, let me
reconsider…", then a second query. The prompt asks for the query only, so the grader takes the first
one. In 8 of those 9 cases, the second query was right. Graded on its last query, Haiku's SQL score would
be 93.3% instead of 86.7%. That's a real cost of thinking out loud inside a tool call: whatever sits
between your model and your database reads the first answer.

7. GPT-5.5's only misses came from my prompt

GPT-5.5 made exactly four mistakes on diagnosis, and all four are the same one. Each time, a page slid
just past #10 (#10 to #11 twice, #10 to #12 once, #9 to #11 once): it fell off page one, so the
answer is page_one_exit. GPT-5.5 said
unexplained_by_serp every time.

Rereading my prompt, I found two rules that clash:

page_one_exit: "position 1–10 before; now below 10".
"Moves of 1–2 places are normal jitter and are never a cause."

I meant the jitter rule to cover rank_drop and rank_gain. GPT-5.5 took "never" literally. Claude
Sonnet 5, Gemini 3.7 Flash and Grok with reasoning all read the intent. Neither reading is careless.
One model follows the letter of an instruction and others follow its spirit, and that's worth knowing
about any model you give rules to. It's also a reminder that a benchmark's prompt is code too.
The next version will say "never counts as rank_drop or rank_gain", and I'm reporting the
score as it stands rather than quietly fixing it.

8. Value: Gemini 3.7 Flash is the sweet spot

Gemini 3.7 Flash reaches 97.7% at about half the cost of Claude Sonnet 5. GPT-5.5 was the most
expensive run ($3.23) and still finished just behind Sonnet. Below the top four there's a cliff: no model
under $0.65 per run scored above 81%, and all of them fell apart on diagnosis.

So what will power Pulse?

Gemini 3.7 Flash by default. It scored 100% on the three tasks that are about judgement, at half the
price of the most accurate model. Its one weakness is fixable in code: wrap ->> comparisons in
parentheses before running any generated SQL.

Claude Sonnet 5 as the fallback for anything SQL-heavy, since it was perfect everywhere, and
cheaper than GPT-5.5 for the same job.

No cheaper model for diagnosis. An assistant that shrugs at a 47% drop costs more than it saves.

The benchmark also backed a design choice InsightTrack already makes. In the product, Pulse's
explain_traffic_change tool works out significance and causes in code, and the model only
explains the result in plain English. These results show why: left to do that reasoning alone, every
cheaper model got it wrong at least half the time.

What this means if you're building an AI assistant

These lessons carry over to any assistant that works with numbers:

Compute, don't ask. Anything with a threshold (significance, "is this a trend?") belongs in code. Let the model explain the result, not decide it.
Make "nothing happened" and "I can't tell" valid answers, and test them. If your tests only have cases with a cause, you'll never see the shrugger or the alarmist until a user does.
Use reasoning where the answer is a why. It was the biggest single factor in the results: 100% vs. 35% on the same model.
Check generated SQL against your real database dialect. Most failures here were PostgreSQL habits, not wrong logic, and a small rewrite or check catches them.
Don't parse the first thing a model says. Ask for a clean final answer, or read the last one, because models do correct themselves.
Route by task. Tool choice is cheap to get right, so a small model can handle it. Diagnosis isn't.

What I'd measure next

Consistency: run each model three times and count how often the same case flips.
The fixed jitter rule: rerun every model and see whether GPT-5.5 reaches 100%, and whether any other model's score changes.
Real tool calls: multi-turn runs where the model actually calls InsightTrack's tools, instead of naming the one it would call.
Messier evidence: seasonality, like a holiday dip that isn't search, alongside keywords that point in different directions.

Limits, honestly

"Correct" means correct by InsightTrack's rules. Every rule and threshold is written into the prompt, so this tests careful application, not guessing a hidden convention.
Synthetic data, one run per model, all through Kaggle's model proxy.
SQL is graded strictly on the first query. That costs Haiku, as described above.
One rule in the diagnosis prompt is ambiguous (the jitter rule vs. page_one_exit). It accounts for all four of GPT-5.5's misses. Fixed wording is planned for the next version.

My Benchmark

Kaggle benchmark: https://www.kaggle.com/benchmarks/nishikantaray/insighttrack-analyst-bench

Dataset (cases, sample tables, graders): https://www.kaggle.com/datasets/nishikantaray/insighttrack-analyst-bench

The product it's built from: https://github.com/NishikantaRay/InsightTrack

How you can use it

Test another model: the four tasks are public on Kaggle, so anyone can run them against any model in Kaggle's catalogue and compare with the table above.
Reuse the method for your own product: take a rule your code already enforces, turn it into generated test cases with the answer from your code, include cases where the right answer is "nothing", and label the kind of mistake as well as the score.

Built with Kaggle Benchmarks. The task pattern follows
its dataset_evaluation.py example. It covers ground near BFCL and τ-bench (tool calling),
InfiAgent-DABench and DSBench (data analysis), and Spider 2.0 and BIRD (text-to-SQL). It differs by
testing one real product's job end to end, and by scoring whether a model knows when nothing happened.

What If Your Coding Agent Could Remember What It Learned Yesterday?

Nishikanta Ray — Thu, 17 Sep 2026 16:28:47 +0000

I built Attic to help Claude Code and Codex CLI remember what they discover — and now you can save knowledge from your browser too.

You ask your AI coding assistant to investigate a bug.

It searches the repository. Reads five files. Follows a few imports. Runs some commands. Finally, it tells you:

"Found it. The authentication check happens after the desktop sub-window is created."

Great. You fix the bug.

A few days later, you start a new session.

You ask the assistant to investigate another issue in the same area.

It searches the same files.

Reads the same imports.

Runs the same commands.

And eventually discovers the same thing you already knew.

You just paid for the same investigation twice.

Maybe three times.

This is the problem I wanted to solve with Attic — an open-source knowledge persistence tool for AI coding assistants.

But the interesting part isn't simply saving notes.

It's the design behind making those notes useful across sessions, after context compaction, and now even from your browser.

GitHub: https://github.com/NishikantaRay/Attic

The problem: Your AI knows more than your codebase

Let's start with a simple example.

Imagine you're working on a JavaScript monorepo.

You ask Claude Code:

"Why is the cloud package failing to load?"

The assistant investigates and finds:

// packages/cloud/index.js

export default cloudClient;

export default cloudConfig;

Two default exports.

The file throws a syntax error.

The assistant explains the issue. You fix it.

But where does that knowledge go?

If you only keep the conversation, the discovery is trapped inside that session.

If you start a new session, your assistant may have to rediscover the same problem.

The codebase contains the bug fix, but it doesn't necessarily contain the reasoning that helped you find it.

And reasoning is often what you need the next time.

Another example: Documentation versus reality

Suppose your project has a rule:

"All desktop sub-windows must wait for authentication before rendering."

But the actual implementation creates the window before the authentication gate finishes.

Your assistant investigates and discovers the contradiction.

That's valuable knowledge.

Not just:

There is a bug.

But:

The desktop sub-window can be created before the authentication gate
completes, despite the documented requirement.

That tells the next investigation where to look.

It also tells a teammate what assumption is unsafe.

It explains why the code behaves the way it does.

This kind of knowledge is easy to lose.

And it isn't limited to bugs.

It includes:

Architectural decisions.
Repository conventions.
Failed approaches.
Important file locations.
Authentication assumptions.
Performance discoveries.
Why a particular implementation exists.
Things you should never repeat.

Your codebase is a library of implementation.

Your AI assistant's discoveries are the notes in the margins.

The question is: Where do those notes live?

Meet Attic: A memory shelf for your coding agent

Attic is an open-source tool designed to help AI coding assistants persist and reuse knowledge.

It works with:

Claude Code.
Codex CLI.

The idea is straightforward:

Write what you learn to a file. Keep a small index in context. Retrieve the details when you need them.

No need to put every discovery into the assistant's context forever.

No need to rebuild the same understanding from scratch every session.

And no need to turn your entire repository into a giant collection of instructions.

Attic gives the agent a place to store useful discoveries.

Think of it as a small knowledge library inside your project.

The library analogy that makes Attic click

Imagine you have a library.

You have 200 books.

You don't carry all 200 books around with you.

You carry a catalogue.

The catalogue tells you:

Which books exist.
Where they are.
Which ones are important.
What each book is about.

When you need a particular book, you look it up.

Attic follows the same idea.

The `.attic/` folder is the library

A typical Attic knowledge folder contains Markdown files:

.attic/
├── INDEX.md
├── DECISIONS.md
└── items/
    ├── f1.md
    ├── f2.md
    ├── f3.md
    └── ...

The exact files and commands depend on the installed version, but the core concept is consistent:

The knowledge lives on disk. The index helps the agent find it.

`INDEX.md` is the catalogue card

The index is not supposed to contain every detail.

It's a compact representation of what matters.

For example:

# Attic Index

## Pinned

- f3: Authentication must complete before
  desktop sub-windows are created.

## Recent findings

- f12: Cloud package has duplicate default exports.
- f11: API client is initialized in the shared module.
- f10: The docs and implementation disagree
  about the authentication lifecycle.

## Older items

95 older items not shown.
Use /attic-recall <topic>.

This is an illustrative example of the idea, not a claim about the exact generated format.

The agent can see the important information without loading the entire knowledge base into every prompt.

When more detail is needed, it can recall the relevant item.

That's the distinction:

The index is the catalogue. The Markdown files are the books.

Why not just put everything in the system prompt?

This is a reasonable question.

If the assistant needs the information, why not inject all of it?

Because context is limited.

And even when you have enough context, more context isn't automatically better.

Imagine an assistant working on a project with 200 findings.

If every finding is included in every session, the prompt becomes filled with:

Old discoveries.
Superseded decisions.
Unrelated bugs.
Details from previous tasks.
Information that may not matter to the current question.

The agent now has to spend attention deciding what matters.

Attic takes a different approach.

Keep the index small

Attic's runtime includes logic for loading the index and keeping its injected representation compact.

The repository's design notes describe an index that prioritizes pinned findings, recent items, and compact summaries for older knowledge.

The important idea is not a magic token number.

It's this:

Don't force the assistant to carry the entire library when a catalogue will do.

Retrieve the details when needed

Suppose the agent is investigating authentication.

It doesn't need to read every database discovery, UI decision, and deployment note.

It needs the authentication-related knowledge.

That's where recall becomes useful.

The four commands: Put knowledge in, get knowledge out

Attic's commands are organized around a simple workflow.

1. `stash` — save what you learned

After an investigation, save the discovery.

Example:

/attic-stash

The exact command options depend on the installed version.

Conceptually, you are turning an ephemeral chat discovery into a persistent Markdown item.

Instead of:

"I think the auth gate is somewhere in the desktop code."

You save:

# Authentication gate and desktop sub-windows

The desktop sub-window creation path can run
before the authentication gate completes.

When investigating window lifecycle bugs,
check the auth gate before the window creation path.

Tags:
- authentication
- desktop
- lifecycle

Now that discovery has a home.

2. `recall` — find what you already know

Later, you can ask Attic to retrieve relevant knowledge.

Example:

/attic-recall authentication

The assistant can use the stored findings instead of starting from zero.

This is particularly useful when:

A new session starts.
You run /compact.
A teammate clones the project.
You return to an old feature.
You forgot why a particular design exists.

3. `index` — inspect the catalogue

The index helps you understand what the agent can see.

Example:

/attic-index

The idea is to inspect the compact catalogue rather than opening every item.

4. `pin`, `prune`, and `archive` — manage the shelf

Not every discovery deserves permanent prominence.

Some knowledge is important forever.

Some is useful for a week.

Some becomes obsolete after a refactor.

Attic provides commands for managing that lifecycle:

/attic-pin
/attic-prune
/attic-archive

The exact syntax depends on the version and installed commands.

The important design decision is that knowledge needs management.

A memory system that only writes and never cleans up eventually becomes another source of noise.

The part I find most interesting: The index is not the memory

This is the design detail worth understanding.

A common approach to agent memory is:

Store a large amount of information and inject it into the next prompt.

Attic separates storage from injection.

Storage

The detailed findings live in Markdown files.

They can contain context, explanations, file references, and reasoning.

Injection

The assistant receives a compact index.

The index tells it what exists and how to find it.

This separation gives you a useful property:

You can have a large knowledge base without making every prompt equally large.

For example:

Knowledge on disk:

200 findings
│
├── Pinned findings
├── Recent findings
└── Older findings
    └── Available through recall

The assistant doesn't need all 200 findings in context.

It needs enough information to navigate the collection.

This is a familiar pattern in software engineering:

Database versus index.
Filesystem versus directory listing.
Library versus catalogue.
Search engine versus document store.

Attic applies that idea to coding-agent memory.

What happens when you use `/compact`?

This is one of the reasons persistent memory matters.

Imagine a long coding session.

At the beginning, you ask:

"Understand the authentication flow."

The agent investigates.

It discovers:

The auth gate lives in one module.
The desktop window is created in another.
The window creation can happen before auth finishes.
The documentation assumes the opposite order.

You fix the problem.

Then the conversation becomes huge.

You use:

/compact

The context is reduced.

The agent may no longer have every detail from the earlier investigation.

If the key discovery existed only in the conversation, it can be difficult to recover.

But if you saved it to Attic:

.attic/items/f3.md

The knowledge remains on disk.

A later session can recall it.

The discovery has survived the conversation.

This is the central promise of persistent agent memory:

Your knowledge should outlive the context window.

The browser extension: Knowledge doesn't only happen in your terminal

Now let's talk about the feature that makes Attic particularly interesting.

Most coding-agent memory tools focus on what happens inside the terminal.

But developers learn things everywhere.

You might discover a useful fact while reading:

A GitHub issue.
A pull request.
A Stack Overflow answer.
A technical blog.
A framework documentation page.
A browser tab containing an important explanation.
A discussion about a design decision.

You might think:

"This is useful. I should save it."

Then you close the tab.

Or forget.

Or leave it in your browser history.

The information never makes it into your project's knowledge base.

Attic's browser companion is designed to close that gap.

Instead of only remembering what the coding agent discovers, you can save useful knowledge from the browser into the same Attic shelf.

That is a powerful extension of the original idea.

Browser to project memory

Let's say you're investigating a bug in a Node.js application.

You open a GitHub issue discussing a subtle behavior in a dependency.

The issue contains the explanation you need.

You don't want to copy the entire page into your project.

You want to save the useful discovery.

The workflow becomes:

Read useful information in browser
          │
          ▼
       Clip it
          │
          ▼
  Save to Attic knowledge
          │
          ▼
  Recall it in a future session

The browser becomes another input into your project memory.

This is important because developers don't learn only by asking an AI agent questions.

They learn by reading.

And a lot of that reading happens outside the terminal.

What is a clip?

A clip is a way to capture useful browser content and save it as a knowledge item.

Imagine reading this on a documentation page:

"The authentication callback must complete before the desktop window is created."

Instead of copying the entire article into a notes app, you can clip the useful content into Attic.

The saved item might look like:

# Authentication callback lifecycle

Source: Browser research

The authentication callback must complete
before the desktop window is created.

This matters because creating the window
too early can bypass the expected auth gate.

Tags:
- authentication
- desktop
- lifecycle

Now your browser research becomes part of the same knowledge collection used by your coding agent.

The exact clip UI and available options depend on the installed extension version.

Why reuse the CLI writer instead of building a second memory system?

This is one of the most important design calls in Attic's browser companion.

A tempting approach would be to build a separate browser storage system.

Something like:

Browser extension
       │
       ▼
Browser database
       │
       ▼
Browser-only notes

Then the CLI would have its own memory system:

CLI
 │
 ▼
.attic/
 │
 ▼
CLI-only notes

Now you have two systems.

Two storage formats.

Two sets of rules.

Two places to search.

Two systems to maintain.

Attic takes a different approach.

The browser companion reuses the CLI writer.

The repository's design notes explicitly call out that the companion uses the existing CLI writer rather than reimplementing it.

Conceptually:

                 ┌────────────────────┐
                 │  Browser extension │
                 └─────────┬──────────┘
                           │
                           ▼
                 ┌────────────────────┐
                 │   CLI writer       │
                 └─────────┬──────────┘
                           │
                           ▼
                      .attic/
                           │
                 ┌─────────┴──────────┐
                 ▼                    ▼
             INDEX.md             items/*.md

This is a classic software engineering principle:

Reuse the existing source of truth instead of creating a second implementation.

The CLI already knows how to write Attic items.

The browser companion should not need to invent a different way to do the same job.

That keeps the storage model consistent.

It also means the knowledge saved from your browser can be used by the CLI.

And knowledge saved from the CLI can be found in the same collection.

One shelf.

Multiple ways to put books on it.

The subtle distinction: Edit is not stash

This is another design decision worth talking about.

At first glance, these two actions might seem similar.

Stash

You discover something new.

You want to add it to the knowledge base.

New discovery → Add a knowledge item

Edit

You already have a knowledge item.

You want to correct or update it.

Existing item → Replace its content

These are not the same operation.

Suppose you have this finding:

# Authentication lifecycle

The desktop window is created before
the authentication callback completes.

Later, you investigate more deeply and discover that the bug has been fixed.

You edit the existing item:

# Authentication lifecycle

The desktop window now waits for the
authentication callback to complete.

The previous behavior was caused by
the window creation path running too early.

You have updated the knowledge.

You haven't discovered a second, unrelated fact.

Why appending everything is a problem

If editing simply appended another copy, you could end up with:

# Authentication lifecycle

The desktop window is created before auth.

---

# Authentication lifecycle

The desktop window waits for auth.

---

# Authentication lifecycle

The previous behavior was caused by...

Now the agent has to reconcile multiple versions.

That's noise.

The repository's design notes explicitly distinguish editing from stashing:

Editing replaces; stashing appends.

This is a small detail, but it reflects a larger principle:

Memory needs both creation and correction.

A knowledge system that only adds information eventually becomes cluttered with stale information.

Why there is no delete endpoint

This might sound strange.

If a knowledge item is wrong, why not just delete it?

The design notes for Attic's browser companion call out that there is no delete endpoint.

Instead, the system provides management operations such as archive and prune.

There is a reason to be careful here.

Deleting knowledge can be destructive.

Imagine an agent saves an important architectural decision.

A browser clip accidentally overwrites it.

Or a user removes an item because it looks old.

The information may be gone before anyone realizes it was useful.

A safer approach is to separate:

Editing an existing item.
Archiving an item.
Pruning stale knowledge.
Permanently removing data, if supported through an intentional local workflow.

This is a design choice, not a universal rule.

Different tools may reasonably choose different deletion models.

But the principle is worth considering:

Memory should be managed deliberately, not casually destroyed.

The browser companion is also a security boundary

The moment a browser extension can write to your project files, security becomes important.

A browser extension is not the same as a trusted terminal command.

It runs in a different environment.

It may interact with content from arbitrary websites.

It needs a way to communicate with the local Attic server.

That means the design has to answer:

Which origins can connect?
Which requests are allowed?
How is the local service authenticated?
Where can it write?
What happens when a user clips sensitive information?
Can arbitrary websites cause files to be written?

Attic's design notes identify several of these concerns:

Loopback communication.
A token.
A root allowlist.
Origin checks.
A credential scan for clips.

These are important because a browser-to-local-file bridge should not behave like an unrestricted file-writing API.

Loopback: Keep communication local

The companion communicates with a local server.

The goal is to keep the interaction on the developer's machine rather than sending knowledge to a remote cloud service.

The repository's design notes identify loopback communication as part of the implementation.

This is aligned with the project's broader local-first approach.

The knowledge is intended to live in the project's .attic/ directory.

Token authentication

A local service still needs protection.

Just because something runs on localhost doesn't mean every request should be trusted.

Attic's browser server uses a token as part of its request validation.

Conceptually:

Browser extension
       │
       │ request + token
       ▼
Local Attic server
       │
       ▼
Validate request
       │
       ▼
Write knowledge item

The exact request format is an implementation detail.

The important point is that the server isn't designed as an unauthenticated arbitrary file-writing endpoint.

Root allowlist: Where can it write?

This is another useful security concept.

Suppose the browser companion could tell the server:

Write this content to:

/Users/me/anything/on/my/computer

That would be dangerous.

A project memory tool should not be allowed to write to arbitrary locations.

Attic's design notes identify a root allowlist in the server.

The intended principle is:

Restrict file writes to approved project roots.

That limits the scope of what the companion can modify.

Credential scanning: Don't stash secrets

This is particularly important for browser clipping.

Imagine you are reading a page containing:

API_KEY=super-secret-value

You click "Clip."

Should that secret be saved to your project's memory?

No.

Attic's design notes identify a credential scan that refuses a clip containing a credential.

This is a practical safeguard.

But it doesn't mean every secret will be detected.

You should still avoid clipping:

API keys.
Passwords.
Session tokens.
Private access tokens.
Credentials.
Personal secrets.

A browser extension that can save content needs to treat sensitive information as a first-class concern.

The local-first philosophy

One of the things I like about Attic is that its design stays close to ordinary developer tools.

The knowledge is stored as Markdown.

The project can inspect it.

The files can be version-controlled.

And the runtime is designed around local filesystem operations rather than a cloud memory service.

The repository describes the runtime as using Node.js fs and path, with no network calls in the plugin runtime.

That gives the project a straightforward mental model:

Your project
    │
    ├── Source code
    ├── Documentation
    ├── Tests
    └── .attic/
         ├── INDEX.md
         ├── DECISIONS.md
         └── items/

Your agent's memory is another part of the repository.

Not a mysterious remote database.

Not a separate SaaS dashboard.

Not a service you need to query over the internet every time the agent needs to remember something.

But should you commit `.attic/` to Git?

This is an important practical question.

If the knowledge lives in your project, it can potentially be shared with the team.

For example:

Developer A
    │
    ▼
Saves an architectural discovery
    │
    ▼
Commits .attic/
    │
    ▼
Developer B pulls the changes
    │
    ▼
Agent can recall the discovery

The repository notes describe .attic/ as surviving a Git clone only if it is committed rather than ignored.

That distinction matters.

If you want shared project memory, you need to decide:

Should the knowledge be committed?
Should some findings remain local?
How should stale findings be managed?
How should conflicting changes be merged?
What information is appropriate for the whole team?

For a team project, knowledge management becomes part of the repository workflow.

A complete example: From bug investigation to reusable knowledge

Let's walk through a realistic workflow.

Step 1: Start a new investigation

You ask Claude Code:

"Investigate why the desktop sub-window sometimes appears before authentication."

The agent searches the codebase.

It discovers:

packages/desktop/window.js
packages/auth/gate.js
packages/desktop/bootstrap.js

It traces the execution path.

Step 2: The agent finds the root cause

The agent explains:

"The window creation call happens before the authentication callback completes."

You fix the issue.

Step 3: Save the discovery

You stash the important knowledge:

/attic-stash

The saved item contains something like:

# Desktop window authentication lifecycle

The desktop window creation path can run
before the authentication callback completes.

When investigating window lifecycle issues,
check the auth gate before window creation.

Relevant areas:
- packages/desktop/window.js
- packages/auth/gate.js
- packages/desktop/bootstrap.js

Tags:
- desktop
- authentication
- lifecycle

Step 4: Start a new session

A few days later:

"Why is the desktop window lifecycle failing?"

The agent can inspect the Attic index and recall relevant knowledge.

Instead of immediately rediscovering the entire execution path, it has a useful starting point.

Step 5: Update the knowledge after the fix

You discover a more precise explanation.

You edit the existing item.

Now the knowledge stays current.

This is the lifecycle that matters:

Investigate
    ↓
Discover
    ↓
Stash
    ↓
Recall
    ↓
Update
    ↓
Reuse

It's not magic.

It's a disciplined way to make discoveries persistent.

What Attic does not try to do

It's worth being clear about the scope.

Attic is not a replacement for:

Your source code.
Your Git history.
Your issue tracker.
Your documentation.
Your test suite.
Your own engineering judgment.

It's a knowledge persistence layer for coding-agent workflows.

It helps preserve discoveries and make them available again.

It doesn't guarantee that the assistant will always recall the right item.

It doesn't guarantee that every saved finding is correct.

It doesn't eliminate the need to verify facts.

And it doesn't mean the agent should blindly trust old notes.

A stale memory can be just as dangerous as no memory.

That's why editing, pruning, and archive workflows matter.

The bigger idea: Agents need a knowledge lifecycle

When people talk about AI coding assistants, they often focus on:

Better models.
Larger context windows.
Faster inference.
More tool calls.
Better code generation.

Those things matter.

But there is another problem:

What happens to the useful knowledge after the task ends?

A coding agent might discover:

The API client is initialized in a shared module.

That can be valuable.

But if it exists only in the chat, the next session may have to rediscover it.

The same is true for:

Why a dependency was chosen.
Why a workaround exists.
Why a particular file is important.
Which approach failed.
What assumption caused a bug.
What the team should avoid repeating.

These are not necessarily code changes.

They are discoveries about the codebase.

And they deserve a place to live.

Why browser memory is a natural next step

The browser extension expands the idea beyond terminal-based discovery.

Developers constantly learn from external information.

They read a blog post.

Find a GitHub issue.

Understand a framework quirk.

Discover a workaround.

Read a discussion about a bug.

That knowledge is useful even if the AI agent didn't discover it itself.

The browser companion makes it possible to capture useful information and bring it into the same local knowledge system.

This creates a broader workflow:

             ┌───────────────┐
             │   Terminal    │
             │ Claude/Codex  │
             └───────┬───────┘
                     │
                     ▼
              ┌────────────┐
              │   Attic    │
              │  Knowledge │
              └────────────┘
                     ▲
                     │
             ┌───────┴───────┐
             │               │
      ┌──────┴──────┐ ┌──────┴──────┐
      │ CLI stash   │ │ Browser     │
      │             │ │ clip        │
      └─────────────┘ └─────────────┘

One knowledge shelf.

Different ways to add information.

The same place to recall it.

Getting started

You can explore Attic here:

GitHub: https://github.com/NishikantaRay/Attic

The project is designed for Claude Code and Codex CLI.

A typical workflow is:

1. Install Attic
2. Initialize it in your project
3. Investigate a problem
4. Stash useful discoveries
5. Recall them in later sessions
6. Manage the knowledge as it evolves
7. Use the browser companion to capture useful research

For the exact installation and usage commands, follow the repository's current README and release documentation.

Final thoughts

I think the most interesting thing about AI coding assistants isn't just how much code they can write.

It's how much they can discover.

An agent can investigate a codebase for ten minutes and uncover something that would take a developer much longer to understand.

But if that discovery disappears when the session ends, you lose part of the value.

Attic is built around a simple idea:

Don't make your AI coding assistant rediscover what it already learned.

Write the discovery to a file.

Keep a compact index.

Recall the details when needed.

And now, when useful knowledge comes from your browser, clip it into the same shelf.

The future of AI coding assistants may not just be about giving agents more context.

It may also be about giving them better ways to remember.

Try Attic

If you use Claude Code or Codex CLI and you've ever thought:

"I know we already solved this. Why is the agent investigating it again?"

Attic is worth exploring.

GitHub: https://github.com/NishikantaRay/Attic

If you try it, I'd be interested to hear how you manage your agent's knowledge.

What would you want your coding assistant to remember permanently?

ai #claudecode #codex #opensource #developer-tools #codingagents #artificialintelligence #productivity

AI Skills Are Not Just Prompts: A Practical Architecture for Building, Evaluating, Shipping, and Maintaining Agent Skills

Nishikanta Ray — Thu, 03 Sep 2026 19:13:59 +0000

The current generation of AI coding agents makes it surprisingly easy to create a "skill."

Write a Markdown file. Add instructions. Give it a name. Put it inside .claude/skills/. Done.

Except it isn't.

As soon as you build more than a handful of skills, a different set of problems appears:

Which skill should activate?
Why did two skills activate at the same time?
Why did the agent ignore an important instruction?
What belongs in a skill versus a rule, agent, hook, or script?
How much context should the skill load?
How do you test whether the skill actually works?
What happens when the underlying framework changes?
How do you distribute skills?
How do you retire obsolete skills?
How do you prevent dozens of skills from becoming an unmaintainable mess?

At that point, AI skills stop looking like prompts and start looking like software systems.

A production-grade AI skill is not merely a Markdown prompt. It is a versioned, testable, routable, enforceable software component with a defined lifecycle.

1. The Skill Lifecycle

A useful way to understand an AI skill is to look at its entire lifecycle:

Runtime
   ↓
Scope / Fit
   ↓
Triggers
   ↓
Architecture
   ↓
Anatomy
   ↓
Content
   ↓
Enforcement
   ↓
Measurement
   ↓
Shipping
   ↓
Maintenance
   ↓
Portfolio

Each stage answers a different question:

Stage	Core question
Runtime	How does the skill load and execute?
Fit & scope	Should this be a skill at all?
Typing & triggers	When should it activate?
Architecture	How should the workflow operate?
Anatomy	What files make up the skill?
Content	How should instructions be written?
Enforcement	What can be enforced mechanically?
Measurement	How do we know it works?
Shipping	How do users receive it?
Maintenance	How does it survive change?
Portfolio	How do many skills coexist?

The important insight is that skill engineering covers the entire lifecycle, not just the writing of SKILL.md.

2. Chapter 00 — How Skills Load and Run

Before designing a skill, understand the runtime.

A skill may look simple on disk:

.claude/
└── skills/
    └── code-review/
        └── SKILL.md

But conceptually the runtime does something like:

User request
     ↓
Skill discovery
     ↓
Trigger evaluation
     ↓
Skill activation
     ↓
Instruction loading
     ↓
Reference/tool loading
     ↓
Agent execution
     ↓
Result

The important question is:

What does the agent actually see, and when does it see it?

This matters because context is limited.

A skill might contain:

SKILL.md
references/
    database.md
    security.md
    examples.md
    architecture.md
scripts/
    validate.js
    check.sh
templates/
    report.md

You usually don't want every file loaded for every request.

Instead:

Request
   ↓
SKILL.md
   ↓
Determine relevant task
   ↓
Load relevant reference
   ↓
Execute required script

This is progressive context loading.

The main skill file acts as an entry point rather than a giant knowledge dump.

Context Is a Resource

One of the biggest mistakes in AI skill design is treating context as free.

It isn't.

Design A — Everything in one file

SKILL.md
──────────────
3000 lines
50 examples
20 rules
10 workflows
15 edge cases

Design B — Layered knowledge

SKILL.md
   ↓
Routing
   ↓
references/
   ├── workflow.md
   ├── security.md
   └── examples.md

The second design gives you more control over what enters the model's context.

Design skills around context boundaries, not just file boundaries.

3. Chapter 01 — Fit and Scope

The next question is:

Should this behavior be implemented as a skill?

AI development environments often provide multiple primitives:

Skill
Rule
Agent
Hook
Script
Plugin

They are not interchangeable.

Skill

A skill describes how to perform a class of task.

Example:

database-migration

It might define:

1. Inspect schema
2. Inspect migration history
3. Design migration
4. Implement migration
5. Validate migration
6. Test rollback

Rule

A rule is a constraint:

Never modify production data directly.

That is policy, not a workflow.

Agent

An agent is useful when you need a distinct reasoning or execution role:

Main Agent
   ├── Research Agent
   ├── Security Agent
   └── Testing Agent

Hook

A hook responds to an event:

Before tool call
      ↓
Security check

or:

After file modification
      ↓
Formatter

Script

A script performs deterministic computation:

validate-schema.js

The AI may decide when to run it. The script determines how validation works.

4. The Boundary Is More Important Than the Skill

A good skill should explicitly state what it does not do.

For example:

This skill handles:

- PostgreSQL schema migrations
- Migration generation
- Migration validation
- Rollback testing

This skill does not handle:

- Application architecture
- Production deployment
- Database backups
- Infrastructure provisioning

Why?

Because skills tend to grow.

A developer starts with:

database migration

Then adds:

database design
query optimization
backup strategy
production deployment

Eventually the skill becomes:

database-engineering-everything

and becomes difficult to reason about.

Scope is a defense against skill inflation.

5. Chapter 02 — Typing and Triggers

Once you know what a skill does, you need to answer:

When should it activate?

Imagine:

frontend-review
backend-review
security-review
performance-review
database-review

The user says:

"Review my authentication API."

Multiple skills may be relevant.

You now have a routing problem:

                 User request
                      │
        ┌─────────────┼─────────────┐
        ↓             ↓             ↓
    Backend       Security       Performance

If everything activates, the system becomes noisy.

If nothing activates, the skill is useless.

Activation Is a Classification Problem

Think of activation as:

User request
      ↓
Intent classification
      ↓
Skill candidates
      ↓
Relevance evaluation
      ↓
Activation

Example:

Request	Expected skill
"Create a PostgreSQL migration"	Database migration
"Fix this React component"	Frontend
"Audit authentication"	Security
"Why is this query slow?"	Database performance
"Write a technical article"	Documentation

A good skill therefore needs a clear activation signature.

False Positives and False Negatives

False positive

"Write a blog about PostgreSQL"

→ security skill activates

False negative

"Why is my PostgreSQL migration failing?"

→ database migration skill does not activate

Both are important.

Therefore, evaluate activation separately from execution.

6. Chapter 03 — Shape and Architecture

Once a skill activates, what does it actually do?

There are several useful workflow shapes.

Route

A route chooses a path:

                 Request
                    ↓
                 classify
              /     |                   /      |                  Bug    Feature   Refactor
            ↓        ↓         ↓
         Debug    Implement  Refactor

Pipeline

A pipeline is sequential:

Research
   ↓
Analyze
   ↓
Plan
   ↓
Implement
   ↓
Test
   ↓
Review
   ↓
Report

This is particularly useful for engineering skills.

Loop

A loop supports iteration:

Implement
    ↓
Test
    ↓
Failed?
  /    Yes     No
 ↓       ↓
Fix    Complete
 ↓
Test again

For coding agents, this is natural:

Write implementation
        ↓
Run tests
        ↓
Read failure
        ↓
Modify implementation
        ↓
Run tests again

Failure becomes an expected workflow state rather than an exceptional event.

Map

A map splits a problem into independent pieces:

                 Research repository
                        │
          ┌─────────────┼─────────────┐
          ↓             ↓             ↓
       Frontend      Backend       Database
          ↓             ↓             ↓
       Findings      Findings      Findings
          └─────────────┼─────────────┘
                        ↓
                     Synthesis

Combining Shapes

Real skills often combine these patterns:

Route
  ↓
Pipeline
  ↓
Map
  ↓
Loop
  ↓
Final report

This is where AI skills start looking like workflow engines rather than prompts.

7. Chapter 04 — Anatomy

A mature skill might look like:

skills/
└── deep-research/
    │
    ├── SKILL.md
    │
    ├── references/
    │   ├── research-methodology.md
    │   ├── source-evaluation.md
    │   └── evidence.md
    │
    ├── scripts/
    │   ├── validate-sources.js
    │   └── generate-report.js
    │
    ├── templates/
    │   └── report.md
    │
    └── examples/
        ├── example-1.md
        └── example-2.md

Each part has a job.

`SKILL.md` Is the Hub

Think of SKILL.md as the router and operating manual.

It should explain:

What this skill does
When it activates
What it must accomplish
What references to load
What workflow to follow
What constraints apply
How to validate the result

It should not necessarily contain every piece of knowledge.

Hub-and-Spoke Architecture

                   SKILL.md
                  /   |                    /    |                    ↓     ↓     ↓
          References Scripts Templates

The hub provides orchestration.

The spokes provide specialized resources.

Data vs Instructions vs Code

Separate them:

Instructions

SKILL.md

Tell the agent what to do.

Data

references/

Provide knowledge.

Code

scripts/

Perform deterministic operations.

This separation makes skills easier to maintain and test.

8. Chapter 05 — Writing the Content

Now we reach the instruction layer.

Compare:

Weak

You might want to check whether tests pass.

Strong

Run the test suite before declaring the task complete.

Stronger

Do not declare the task complete until the test suite passes.

The difference is binding strength.

Instructions Have Different Authority

Preference

Prefer TypeScript.

Requirement

Use TypeScript for new files.

Hard constraint

Do not create JavaScript files. New implementation files must use TypeScript.

Important rules should be unambiguous.

Present-Tense Instructions

Instead of:

The agent should inspect the repository.

write:

Inspect the repository before making changes.

Instead of:

The agent will run the tests after implementation.

write:

Run the tests after implementation.

Direct instructions are easier to interpret.

Don't Overload the Skill With Philosophy

Avoid thousands of words explaining philosophy.

Prefer operational instructions:

1. Inspect X.
2. Determine Y.
3. Run Z.
4. If Z fails, investigate.
5. Do not proceed until Y is verified.

The skill should be operational.

9. Chapter 06 — Enforcement

This is one of the most important ideas.

Don't rely on the model to enforce something that software can enforce.

Suppose your skill says:

Always run Prettier.

The agent might comply.

But it might also forget.

A stronger design is:

AI modifies file
      ↓
Hook
      ↓
Prettier
      ↓
Formatted file

Formatting no longer depends entirely on model memory.

AI Instructions vs Mechanical Enforcement

Consider:

Never commit secrets.

A prompt can say:

RULE:
Never commit API keys.

But a stronger architecture is:

Agent
  ↓
git commit
  ↓
secret scanner
  ↓
Secret found?
  ├── Yes → Block commit
  └── No  → Continue

Enforcement Hierarchy

A useful model is:

Human judgment
      ↓
AI instruction
      ↓
Automated validation
      ↓
Mechanical enforcement

The further down the stack you go, the less you rely on model compliance.

Examples of good candidates for automation:

Formatting
Linting
Type checking
Schema validation
Tests
Secret detection
File naming
Generated artifacts
SQL safety
Permission boundaries

The AI should focus on tasks requiring judgment.

10. Chapter 07 — Measurement

Now ask:

How do we know the skill works?

A skill should ideally be evaluated, not merely read and trusted.

There are two major dimensions.

Activation Evaluation

Did the correct skill activate?

Example:

Prompt:
"Create a PostgreSQL migration for the users table."

Expected:
database-migration → activated

You can maintain an evaluation set:

┌────────────────────────────────────┐
│ Activation Evaluation              │
├────────────────────────────────────┤
│ Prompt                             │
│ Expected skill                     │
│ Should activate?                   │
│ Actual skill                       │
│ Result                             │
└────────────────────────────────────┘

Behavior Evaluation

Once activated, did the skill behave correctly?

Suppose the migration skill requires:

Inspect schema
Check migration history
Create migration
Validate SQL
Test migration
Test rollback

Behavior evaluation checks those requirements.

                    Skill
                      ↓
             ┌────────┴────────┐
             ↓                 ↓
       Activation          Behavior
             ↓                 ↓
        Correct skill?    Correct workflow?

A skill could have:

90% activation accuracy
40% behavior accuracy

and still be a poor skill.

Measuring Skills Like Software

Useful metrics include:

Activation accuracy

correct activations / total activation tests

False positive rate

incorrect activations / total tests

False negative rate

missed activations / applicable tests

Task success rate

successful executions / total executions

Rule compliance

required behaviors satisfied / required behaviors

Regression rate

previously passing cases now failing

This turns skill development into an engineering discipline.

11. Auditing Against Prior Art

Another useful activity is comparing your skill against existing approaches.

Suppose you create:

deep-research

Before declaring it complete, ask:

What existing research workflows already exist?

What evaluation techniques do they use?

What source-quality rules are common?

What am I missing?

Which practices are unnecessarily complicated?

This is prior-art auditing.

You don't need to reinvent every workflow.

12. Chapter 08 — Shipping

A skill isn't useful if nobody can install it.

A typical flow is:

Development
    ↓
Repository
    ↓
Package / Plugin
    ↓
Distribution
    ↓
Installation
    ↓
Skill available

Plugin Binding

A plugin can act as a distribution container:

Plugin
│
├── Skills
│   ├── research
│   ├── testing
│   └── code-review
│
├── Hooks
├── Commands
└── Configuration

This lets users install a cohesive collection rather than manually copying individual files.

Versioning

Once users depend on a skill, versioning matters:

research-skill@1.0.0
research-skill@1.1.0
research-skill@2.0.0

Changing:

"prefer X"

to:

"must use X"

can change agent behavior significantly.

Therefore skill versions should be treated as meaningful behavioral versions.

Dormancy

A skill may sit unused for months.

Then:

Developer
   ↓
invokes old skill
   ↓
framework changed
   ↓
skill behaves differently

This is skill dormancy.

Skills need lifecycle states and maintenance expectations.

13. Chapter 09 — Maintenance

AI skills exist inside rapidly changing ecosystems.

Things change:

AI models
Agent runtimes
APIs
CLI tools
Frameworks
Repository structures
Tool interfaces
Best practices
Security requirements

Therefore:

A skill is software, and software drifts.

Drift

Suppose:

Tool v1
 ↓
Skill v1

Later:

Tool v2
 ↓
behavior changed

Your skill still assumes the old behavior.

That is drift.

Vendored Knowledge

Suppose your skill contains a copy of external documentation:

external methodology
       ↓
copy
       ↓
your skill

The external source changes.

Your copy doesn't.

Now:

Upstream
   ↓
Version 3

Your skill
   ↓
Version 1

You have a maintenance obligation.

Review Gate

A good update flow is:

Upstream changes
       ↓
Detect change
       ↓
Review
       ↓
Update skill
       ↓
Run evaluation suite
       ↓
Check regressions
       ↓
Release

Not:

Upstream changed
       ↓
blindly copy everything

14. Chapter 10 — The Skill Portfolio

One skill is easy.

Two are manageable.

Ten are interesting.

Fifty become an architecture problem.

You may eventually have:

skills/
├── research/
├── frontend/
├── backend/
├── database/
├── security/
├── testing/
├── performance/
├── deployment/
├── documentation/
├── architecture/
├── debugging/
└── code-review/

Now you need portfolio management.

Skill Collision

Suppose the user says:

"Review my API."

Potential matches:

api-review
backend-review
security-review
performance-review

Which one activates?

This is a collision.

If multiple skills activate, their instructions may conflict.

For example:

Skill A:
"Keep the implementation minimal."

Skill B:
"Add extensive validation."

Skill C:
"Refactor the architecture."

A skill portfolio therefore needs routing and priority policies.

Skill Granularity

How large should a skill be?

Too broad:

software-engineering

with everything inside it.

Too narrow:

read-file
write-file
check-import
check-variable
run-test

You don't want hundreds of microscopic skills.

A better structure might be:

Engineering
│
├── Research
├── Implementation
├── Testing
├── Security
└── Deployment

Each skill owns a meaningful capability.

Router Families

A router can first identify the broad family:

                  User request
                       ↓
                  Main Router
                       ↓
       ┌───────────────┼───────────────┐
       ↓               ↓               ↓
   Research        Engineering      Operations
       ↓               ↓               ↓
   Research        Backend         Deployment
   skill           Security        Monitoring
                   Testing

Instead of asking:

"Which of these 100 skills should run?"

you ask:

Which family?
     ↓
Which subcategory?
     ↓
Which skill?

Retirement

A healthy portfolio needs a retirement policy.

A skill may become obsolete because:

the underlying tool disappeared
the workflow became part of another skill
the framework changed
the skill is redundant
another skill superseded it
nobody uses it anymore

A lifecycle might be:

Experimental
     ↓
Active
     ↓
Stable
     ↓
Deprecated
     ↓
Retired

This prevents the skill directory from becoming a graveyard.

15. The Deeper Architecture: Instructions + Tools + Enforcement + Evaluation

A mature skill can be viewed as four major layers:

┌──────────────────────────────────────┐
│              SKILL                   │
│                                      │
│  ┌────────────────────────────────┐  │
│  │ Instructions                    │  │
│  │ What the agent should do       │  │
│  └────────────────────────────────┘  │
│                  ↓                   │
│  ┌────────────────────────────────┐  │
│  │ Workflow                       │  │
│  │ Route / Pipeline / Loop / Map │  │
│  └────────────────────────────────┘  │
│                  ↓                   │
│  ┌────────────────────────────────┐  │
│  │ Enforcement                    │  │
│  │ Hooks / Scripts / Tests       │  │
│  └────────────────────────────────┘  │
│                  ↓                   │
│  ┌────────────────────────────────┐  │
│  │ Evaluation                     │  │
│  │ Activation / Behavior / QA    │  │
│  └────────────────────────────────┘  │
└──────────────────────────────────────┘

This is the important shift in thinking.

A skill is not merely:

prompt → answer

It is closer to:

intent
  ↓
routing
  ↓
context
  ↓
workflow
  ↓
tools
  ↓
validation
  ↓
feedback
  ↓
result

16. Example: Building a Deep Research Skill

Suppose you want:

deep-research

Its directory could be:

deep-research/
├── SKILL.md
├── references/
│   ├── research-methodology.md
│   ├── source-quality.md
│   ├── evidence-evaluation.md
│   └── synthesis.md
├── scripts/
│   ├── validate-sources.js
│   └── generate-report.js
├── templates/
│   └── research-report.md
└── evals/
    ├── activation.json
    └── behavior.json

Step 1 — Define Scope

This skill performs structured research.

It handles:
- multi-source research
- source evaluation
- evidence synthesis
- contradiction analysis
- structured reporting

It does not handle:
- software implementation
- deployment
- generic writing

Step 2 — Define Activation

Potential activation prompts:

"Research this topic deeply."

"Investigate the current state of..."

"Compare these technologies using external sources."

"Find evidence for and against this claim."

Non-activation examples:

"Fix this React bug."

"Run the tests."

"Format this file."

Step 3 — Define the Workflow

Understand question
       ↓
Decompose question
       ↓
Identify evidence requirements
       ↓
Search
       ↓
Evaluate sources
       ↓
Extract evidence
       ↓
Cross-check claims
       ↓
Synthesize
       ↓
Write report
       ↓
Validate citations

Step 4 — Add a Loop

If two sources disagree:

Source A → Claim X
Source B → Claim Y

then:

Conflict detected
       ↓
Investigate
       ↓
Find additional sources
       ↓
Re-evaluate evidence
       ↓
Resolve / report uncertainty

Now the skill combines:

Pipeline + Loop

Step 5 — Add Mechanical Validation

Instead of only telling the AI:

"Make sure every claim has a citation."

build a validator:

Report
  ↓
Citation validator
  ↓
Missing citation?
  ├── Yes → fail
  └── No  → pass

Step 6 — Evaluate Activation

Example:

Prompt:
"Do a deep investigation into DuckDB vs PostgreSQL for analytics."

Expected:
deep-research → YES

And:

Prompt:
"Fix the DuckDB connection bug."

Expected:
deep-research → NO

Step 7 — Evaluate Behavior

For a research request:

✓ question decomposition
✓ multiple sources
✓ source quality assessment
✓ evidence extraction
✓ conflicting evidence analysis
✓ synthesis
✓ citations
✓ final report

Now regressions can be detected.

17. Applying This to an AI Engineering Harness

This model becomes especially interesting when building a larger AI development harness.

Imagine:

                    AI HARNESS
                        │
                        ↓
                  Intent Router
                        │
        ┌───────────────┼───────────────┐
        ↓               ↓               ↓
     Research       Engineering      Operations
        │               │               │
        ↓               ↓               ↓
     Skills           Skills          Skills
        │               │               │
        └───────────────┼───────────────┘
                        ↓
                     Tools
                        ↓
                  Enforcement
                        ↓
                    Evaluation
                        ↓
                    Reporting

This is much more powerful than simply having a folder full of Markdown files.

18. Skills as a Policy Execution Layer

Traditional software:

Code
 ↓
Execution
 ↓
Result

AI software:

Intent
 ↓
Skill
 ↓
Reasoning
 ↓
Tools
 ↓
Result

But production AI systems need another layer:

Intent
 ↓
Skill
 ↓
Reasoning
 ↓
Tools
 ↓
Policy
 ↓
Validation
 ↓
Result

The skill becomes a bridge between natural-language intent and deterministic engineering systems.

19. The Most Important Design Principle

If there is one idea to take away from all of this, it is:

Use AI for judgment. Use software for certainty.

Let the model handle:

Interpretation
Planning
Hypothesis generation
Trade-offs
Synthesis
Creative reasoning

Let software handle:

Formatting
Validation
Testing
Schema checking
Permissions
Secret detection
Deterministic calculations
Policy enforcement

For example:

AI:
"These three files probably need to change."

Software:
"Does the resulting code compile?"

AI:
"This migration should be safe."

Software:
"Does the migration actually execute successfully?"

AI:
"These sources support the conclusion."

Software:
"Are the required citations present?"

That division produces more reliable systems.

20. Why Skill Engineering Will Become Important

As AI agents become more capable, the bottleneck increasingly shifts away from:

"Can the model write code?"

toward:

"Can we reliably control how the model works?"

That is a different engineering problem.

We need to reason about:

Activation
Context
Permissions
Workflow
Tools
Memory
Policies
Evaluation
Regression
Versioning
Distribution

These are systems problems.

That is why skill engineering starts resembling:

software architecture
        +
prompt engineering
        +
workflow orchestration
        +
testing
        +
policy enforcement
        +
package management

21. Practical Checklist for Building a Skill

Scope

[ ] Is this actually a skill?
[ ] Could this be a rule?
[ ] Could this be a hook?
[ ] Could this be a script?
[ ] What does the skill explicitly refuse to do?

Activation

[ ] When should it activate?
[ ] When should it not activate?
[ ] What are the ambiguous prompts?
[ ] Can it collide with another skill?

Architecture

[ ] Is the workflow a route?
[ ] Pipeline?
[ ] Loop?
[ ] Map?
[ ] Combination of these?

Context

[ ] Is SKILL.md concise?
[ ] Can supporting information be loaded progressively?
[ ] Are references separated from instructions?

Enforcement

[ ] Which rules can be automated?
[ ] Which checks should be hooks?
[ ] Which checks should be scripts?
[ ] Which requirements belong in CI?

Evaluation

[ ] Does the skill activate correctly?
[ ] Does it perform the correct workflow?
[ ] Are there regression tests?
[ ] Are failure cases tested?

Shipping

[ ] Is the skill versioned?
[ ] Can another developer install it?
[ ] Is plugin/package integration defined?

Maintenance

[ ] What external dependencies can drift?
[ ] Are vendored references tracked?
[ ] Is there a review process?
[ ] Can the skill become deprecated?

Portfolio

[ ] Does it overlap with another skill?
[ ] Is the granularity appropriate?
[ ] Does the router know where it belongs?
[ ] What happens when the skill is obsolete?

22. Final Architecture

Putting everything together:

                         USER INTENT
                              │
                              ▼
                     ┌────────────────┐
                     │     ROUTER     │
                     └───────┬────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │      SKILL       │
                    │                  │
                    │ Scope            │
                    │ Instructions     │
                    │ Workflow         │
                    │ References       │
                    └────────┬─────────┘
                             │
                             ▼
                  ┌─────────────────────┐
                  │   AI REASONING      │
                  └──────────┬──────────┘
                             │
                  ┌──────────┴──────────┐
                  ↓                     ↓
             Tools / APIs          Scripts
                  │                     │
                  └──────────┬──────────┘
                             ↓
                    ┌─────────────────┐
                    │   ENFORCEMENT   │
                    │                 │
                    │ Hooks           │
                    │ Policies        │
                    │ Validators      │
                    └────────┬────────┘
                             ↓
                    ┌─────────────────┐
                    │   EVALUATION    │
                    │                 │
                    │ Activation      │
                    │ Behavior        │
                    │ Regression      │
                    └────────┬────────┘
                             ↓
                         RESULT

And around the whole system:

          ┌───────────────────────────────────┐
          │           SKILL LIFECYCLE         │
          │                                   │
          │ Version → Ship → Observe →       │
          │ Maintain → Update → Deprecate    │
          │                                   │
          └───────────────────────────────────┘

Conclusion: From Prompt Files to AI Engineering

The simplest way to build an AI skill is:

Write SKILL.md

The professional way is:

Define scope
     ↓
Define activation
     ↓
Design workflow
     ↓
Structure context
     ↓
Write instructions
     ↓
Add tools
     ↓
Mechanically enforce critical rules
     ↓
Evaluate activation
     ↓
Evaluate behavior
     ↓
Version
     ↓
Ship
     ↓
Monitor drift
     ↓
Maintain
     ↓
Retire when necessary

That is the fundamental shift.

AI skills should be treated less like prompts and more like software components.

A prompt tells an AI what you would like it to do.

A well-engineered skill defines:

when it should act,
what it should do,
what context it should consume,
how it should execute,
which rules are mandatory,
what software can enforce,
how success is measured, and
how the component evolves over time.

Once you start thinking this way, .claude/skills/ stops being a collection of Markdown files.

It becomes an AI-native software architecture layer.

The future of agent engineering is not just better prompts — it is better systems around prompts.

Your Computer Is Busy When You're Not: Building a Desktop Observability Tool with Electron

Nishikanta Ray — Sun, 23 Aug 2026 17:14:54 +0000

You close your laptop.

The screen goes dark.

You assume everything stopped.

But what if it didn't?

Maybe Docker is still transferring data.

Maybe Chrome is still downloading something.

Maybe an application is consuming CPU in the background.

Maybe your computer woke up at 3 AM, performed some background work, and went back to sleep.

Most operating systems already expose pieces of this information.

The interesting part is putting those pieces together.

That's what I wanted to explore:

Can we build an observability layer for a desktop computer using Electron?

Not another CPU/RAM widget.

Something that answers:

What is my computer actually doing?

The Idea

Imagine opening your desktop monitoring application and seeing:

TODAY
────────────────────────────────────

Computer active        7h 42m
Idle                   1h 13m
Sleep                  8h 05m

Network
↓ Download             6.8 GB
↑ Upload               1.2 GB

Top applications
────────────────────────────────────

VS Code                3h 42m
Chrome                 2h 18m
Docker                 1h 31m
Slack                  52m

Network-heavy apps
────────────────────────────────────

Docker                 2.4 GB
Chrome                 1.8 GB
VS Code                920 MB

Now imagine clicking "What happened while I was away?"

18:42  User became idle
18:51  Screen locked
18:53  Docker network activity
19:07  Background sync completed
19:14  System entered sleep

That's much more interesting than a CPU percentage.

The npm Package That Makes This Easier

Instead of writing native system integrations for everything ourselves, we can start with:

npm install systeminformation

systeminformation provides APIs for system information, CPU, memory, battery, filesystem, network, processes, Docker and more. It also supports Linux, macOS and Windows, although individual APIs have platform-specific support.

For example:

const si = require("systeminformation");

async function getSystemStats() {
  const [cpu, memory, battery, network] = await Promise.all([
    si.currentLoad(),
    si.mem(),
    si.battery(),
    si.networkStats()
  ]);

  return {
    cpu: cpu.currentLoad,
    memory: memory.active,
    battery: battery.percent,
    network
  };
}

console.log(await getSystemStats());

Now our Electron main process can periodically collect system information.

Monitoring Network Activity

One of the most useful APIs is:

si.networkStats()

It provides network statistics including received/transmitted bytes and calculated per-second rates. The package calculates rates based on differences between successive calls, so the first call doesn't provide a meaningful rate yet.

A simple monitor:

const si = require("systeminformation");

setInterval(async () => {
  const stats = await si.networkStats();

  for (const network of stats) {
    console.log({
      interface: network.iface,
      download: network.rx_sec,
      upload: network.tx_sec
    });
  }
}, 1000);

Now we have something like:

Wi-Fi
Download: 2.4 MB/s
Upload:   180 KB/s

But this is only the beginning.

Network Connections

We can also inspect active network connections:

const connections = await si.networkConnections();

console.log(connections);

This gives us information about active TCP/UDP connections.

That means we can potentially build:

Active Connections
────────────────────────────

TCP    443    api.example.com
TCP    443    github.com
TCP    443    google.com
UDP    53     DNS

Now the dashboard starts becoming useful for troubleshooting.

Which Application Is Active?

Network traffic alone doesn't tell us who is responsible.

We also want to know which application the user is interacting with.

One option is a native active-window module such as:

npm install @jannchie/active-window

The package exposes information about the currently selected window and user idle time on supported Windows, macOS and Linux/Xorg environments. Platform coverage isn't identical everywhere, so this should be treated as an OS-dependent component.

Conceptually:

const activeWindow = require("@jannchie/active-window");

async function getActiveApp() {
  const window = await activeWindow().getActiveWindow();

  return {
    process: window.windowClass,
    title: window.windowName
  };
}

Then we can record:

10:32:14  VS Code
10:32:15  VS Code
10:32:16  VS Code
...
10:45:02  Chrome
10:45:03  Chrome

Instead of storing every second individually, we can aggregate this into sessions.

VS Code
10:32 → 10:45
Duration: 13 minutes

Sleep and Wake

This is where Electron itself becomes useful.

Electron provides the powerMonitor module for monitoring system power-state changes, including suspend/resume, AC/battery changes, idle state and lock/unlock events on supported platforms.

In the Electron main process:

const { powerMonitor } = require("electron");

powerMonitor.on("suspend", () => {
  console.log("System is going to sleep");
});

powerMonitor.on("resume", () => {
  console.log("System woke up");
});

We can turn those events into database records:

function recordEvent(type) {
  db.insert({
    type,
    timestamp: new Date()
  });
}

powerMonitor.on("suspend", () => {
  recordEvent("system_sleep");
});

powerMonitor.on("resume", () => {
  recordEvent("system_wake");
});

Now our timeline knows when the machine slept.

Idle Detection

Electron also provides:

powerMonitor.getSystemIdleTime()

and:

powerMonitor.getSystemIdleState(60)

The latter can return states such as active, idle, or locked on supported systems.

For example:

const state = powerMonitor.getSystemIdleState(60);
const idleSeconds = powerMonitor.getSystemIdleTime();

console.log({
  state,
  idleSeconds
});

We can therefore distinguish:

Active
   ↓
Idle
   ↓
Locked
   ↓
Sleep

That's much more meaningful than simply measuring application runtime.

Putting Everything Together

Now imagine collecting events from four sources:

                 Electron App
                      │
       ┌──────────────┼──────────────┐
       │              │              │
  powerMonitor   systeminformation   active-window
       │              │              │
       │         CPU / RAM / Net     Active App
       │              │              │
       └──────────────┼──────────────┘
                      ▼
                Event Collector
                      │
                      ▼
                Local Database
                      │
                      ▼
              Activity Timeline

A simple event model could be:

{
  timestamp: Date.now(),
  type: "network",
  app: "docker",
  downloadBytes: 1824000,
  uploadBytes: 420000
}

Or:

{
  timestamp: Date.now(),
  type: "power",
  state: "sleep"
}

Or:

{
  timestamp: Date.now(),
  type: "application",
  app: "Code",
  window: "server.js - Visual Studio Code"
}

Now everything becomes queryable.

The Interesting Part: Correlation

This is where a normal system monitor becomes a desktop observability tool.

Suppose we see:

18:20  User idle
18:23  CPU 65%
18:24  Network 18 MB/s
18:26  Docker active
18:31  420 MB downloaded
18:42  Screen locked
18:45  Network still active
19:02  System sleep

We can generate an explanation:

Docker continued running after the user became idle and transferred approximately 420 MB before the system entered sleep.

That's an insight.

Not just a metric.

A Simple Collector

We could start with something surprisingly small:

const { powerMonitor } = require("electron");
const si = require("systeminformation");

async function collect() {
  const [cpu, memory, network] = await Promise.all([
    si.currentLoad(),
    si.mem(),
    si.networkStats()
  ]);

  return {
    timestamp: Date.now(),

    system: {
      idle: powerMonitor.getSystemIdleTime(),
      idleState: powerMonitor.getSystemIdleState(60),
      battery: powerMonitor.isOnBatteryPower()
    },

    cpu: cpu.currentLoad,

    memory: {
      active: memory.active,
      total: memory.total
    },

    network: network.map(n => ({
      interface: n.iface,
      rxPerSecond: n.rx_sec,
      txPerSecond: n.tx_sec
    }))
  };
}

setInterval(async () => {
  console.log(await collect());
}, 5000);

Five seconds later, we have the beginning of our own desktop telemetry system.

But Don't Collect Everything Every Second

This is an important design decision.

You don't necessarily want:

Every 1 second
    ↓
Everything
    ↓
Database

That can generate enormous amounts of useless data.

Instead, use different collection intervals.

Power events        Event-driven
App changes         Event-driven
CPU                 5-10 seconds
Memory              10 seconds
Network             1-5 seconds
Connections         10-30 seconds
Daily aggregates    1 minute+

And store state changes rather than repeated identical states.

For example, don't write:

10:00 VS Code
10:01 VS Code
10:02 VS Code
10:03 VS Code
10:04 VS Code

Store:

VS Code
10:00 → 10:04
Duration: 4 minutes

This makes the database dramatically smaller.

What About Electron's Network Logging?

If the thing you're monitoring is specifically your Electron application's own network traffic, Electron has another useful tool: netLog.

const { app, netLog } = require("electron");

app.whenReady().then(async () => {
  await netLog.startLogging("/tmp/electron-network-log");
});

Electron's netLog records network events for a session and supports different capture modes. Be careful with includeSensitive and everything, because those modes can capture sensitive information or transferred bytes.

This is different from systeminformation:

systeminformation
        ↓
Whole-system information

Electron netLog
        ↓
Your Electron application's network activity

That distinction is important.

The Product I Would Build

I wouldn't call this another "Task Manager."

I'd call it something like:

Desktop Observability

The dashboard could have four major views.

1. Activity

VS Code       4h 12m
Chrome        2h 31m
Slack         58m
Docker        1h 42m

2. Network

Download     8.4 GB
Upload       1.3 GB

Docker       3.1 GB
Chrome       2.7 GB
VS Code      1.2 GB

3. Power

Active       8h 21m
Idle         1h 08m
Sleep        7h 42m

Battery used 34%

4. Timeline

09:02  Wake
09:05  VS Code
10:32  Chrome
12:14  Idle
12:31  Docker network activity
13:02  Active
18:42  Lock
19:01  Sleep

And then the killer feature:

Ask the computer

Why was my laptop slow today?

What used the most bandwidth?

What happened while I was away?

Which applications are active in the background?

Did anything continue running after I locked the computer?

The Architecture

A practical first version could be:

Electron
│
├── Main Process
│   │
│   ├── powerMonitor
│   ├── systeminformation
│   ├── active-window
│   └── netLog
│
├── Collector
│   │
│   ├── power events
│   ├── app sessions
│   ├── CPU/memory
│   └── network stats
│
├── SQLite
│
└── React Renderer
    │
    ├── Dashboard
    ├── Timeline
    ├── Applications
    ├── Network
    └── Power

The key design principle is:

Collect locally. Correlate locally. Explain locally.

You don't need to send a user's entire computer activity to a cloud server just to tell them why their laptop was busy.

The Bigger Idea

The interesting thing here isn't systeminformation.

It isn't Electron's powerMonitor.

It isn't network monitoring.

It's the combination.

A computer is already generating thousands of signals:

Applications
Processes
CPU
Memory
Network
Power
Sleep
Wake
Idle
Lock
Unlock

Today, these signals are mostly presented independently.

A desktop observability layer could connect them.

And instead of asking:

"What's my CPU usage?"

we can finally ask:

"What happened on my computer while I wasn't looking?"

That's a much more interesting monitoring problem.

Smashing the "Blind Spot" Bug: How We Integrated Sentry to Catch Regressions in Real-Time

Nishikanta Ray — Sat, 08 Aug 2026 20:38:12 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The Challenge: Flying Blind in Production

Pull Request - https://github.com/NishikantaRay/InsightTrack/commit/a70ca0a00c8cd169a93b300cfcb450b5ecbde7f8

Before this summer, our analytics platform, InsightTrack, had a fundamental flaw in how it handled observability. We were tracking standard JavaScript errors via a basic window.onerror handler, but it was just noise. We had no stack traces, no grouped fingerprints, and absolutely no release context.

If a customer integrated 10 different sites into our platform, we couldn't accurately tell them if a specific spike in errors was a brand-new issue or a resurrected bug from three deployments ago. We were flying blind, and our users were feeling the pain of delayed bug resolutions.

The ultimate "bug" wasn't a single line of broken code; it was our entire error observability pipeline.

The Solution: A Deep-Dive Sentry Integration

We decided to smash this architectural bug by building a native, robust integration with Sentry. We didn't just want to add a widget; we wanted to bring Sentry's rich context (fingerprinted grouping, permalinks, regression status, and user-impact counts) directly into the InsightTrack dashboard so traffic and bugs could be watched side-by-side.

How We Built It

To make this work seamlessly at scale (where one customer might poll 10 independent Sentry projects simultaneously), we built a dual-path ingestion system:

The Polling Backstop: We set up a bounded worker pool (to prevent slow projects from stalling the fleet) that polls the Sentry API every 5 minutes. To respect rate limits, we built an adaptive cadence—active projects poll frequently, while quiet or erroring projects exponentially back off.
The Near-Real-Time Webhook: For instant visibility, we allowed users to point a Sentry Internal Integration webhook at our API. Using HMAC signatures verified in constant time against a stored secret, new or regressed issues hit our database in milliseconds.

The Golden Rule of our Architecture:
All writes (polls and webhooks) go to PostgreSQL. A background sync then pushes this data to DuckDB every 60 seconds. All dashboard reads and analytics queries come only from DuckDB for lightning-fast performance.

The "Smash" Moment 🔨

We deployed the Sentry integration to production on a Tuesday. On Wednesday morning, it paid off spectacularly.

A customer pushed a massive UI update to their e-commerce site. Within three minutes, our newly built Error Spike Alerts—which calculate a 2-sigma (2σ) z-score against a 7-day rolling mean of Sentry events—fired off an alarm in their dashboard.

Because we were pulling in Sentry's substatus API field, the issue surfaced with a glaring red Regressed badge right in our UI.

Instead of opening five different tabs to figure out what went wrong, the customer clicked the row in our dashboard. This triggered our Live Drill-Down feature, which makes an on-demand, real-time fetch directly to Sentry's /events/latest/ endpoint. The stack trace, breadcrumbs, and tags populated instantly.

The bug? A deprecated checkout component that had been accidentally uncommented during a merge conflict. It was identified, patched, and deployed before most of their users even woke up.

What I'm Proud Of

I am incredibly proud of how resilient we made this system. By decoupling the write path (PostgreSQL) from the read path (DuckDB), the dashboard remains incredibly fast even if the polling workers are churning through thousands of Sentry pagination cursors in the background.

I'm also proud of the AI Analyst (Pulse) tools we built on top of this. Because the data is so cleanly structured in DuckDB, we exposed read-only MCP tools (get_error_summary and get_error_issues). Now, users can literally chat with their analytics dashboard and ask, "Did yesterday's deploy cause any regressions?" and the AI will read the synced Sentry data to give them a perfect answer.

What I Learned

Handling external APIs requires defensive programming. Assuming an API will always return what you expect is a recipe for disaster. Building in graceful degradation and adaptive backoffs (like dropping to a 6-hour poll cadence on a 401 Auth Fail) kept our system from DDOSing ourselves with bad tokens.
Webhooks + Polling = The perfect marriage. Webhooks are great for speed, but polling is necessary for accurate counts and state reconciliation (e.g., soft-deleting issues that have aged out). Using both gives you the best of both worlds.

Thanks to Sentry and the DEV community for the motivation to finally smash this architectural bug!

The ultimate "bug" wasn't a single line of broken code; it was our entire error observability pipeline.
Github - https://github.com/nishikantaray/InsightTrack
Website - https://insightstrack.dev
Pull Request - https://github.com/NishikantaRay/InsightTrack/commit/a70ca0a00c8cd169a93b300cfcb450b5ecbde7f8
Portfolio - https://nishikanta.in/

Building an AI-native Second Brain with Multi-RAG, Knowledge Graphs, and MCP

Nishikanta Ray — Sat, 08 Aug 2026 02:38:21 +0000

Claude is incredibly good at reasoning.

But reasoning is only as useful as the context available to it.

Your architecture might be in GitHub. Your notes might be in Obsidian. Your decisions might be buried in Slack. Your research might be in PDFs. Your project history might exist across hundreds of conversations.

The information exists.

The problem is that your AI doesn't have a unified way to understand it.

So I started thinking:

What if Claude had a persistent Second Brain?

Not another chatbot. Not another vector database.

A knowledge layer that Claude can query, learn from, and write back to.

What Is a Second Brain?

A Second Brain is a personal knowledge management system that helps you capture, organize, connect, and retrieve information outside your biological memory.

Tools like Obsidian made this idea powerful.

You can write simple Markdown notes and connect them:

InsightTrack
    │
    ├── [[DuckDB]]
    ├── [[PostgreSQL]]
    └── [[MCP]]

Obsidian then creates a visual graph of those relationships.

But there is a limitation:

Obsidian knows the relationships you explicitly create.

AI can take this much further.

Imagine having:

architecture.md
      │
      └── mentions DuckDB

GitHub PR #481
      │
      └── changes DuckDB queries

Slack discussion
      │
      └── explains why DuckDB was selected

benchmark.md
      │
      └── contains performance results

A human may never manually connect all four.

An AI-native knowledge system can discover those relationships automatically.

From Second Brain to AI Brain

A traditional Second Brain mainly answers:

"Where did I save this?"

An AI-native Second Brain should answer:

"What do we know about this, where did it come from, and how is it connected to everything else?"

That requires more than vector search.

It requires multiple retrieval strategies working together.

Why Traditional RAG Isn't Enough

A typical RAG architecture looks like:

Documents
    ↓
Chunking
    ↓
Embeddings
    ↓
Vector Search
    ↓
LLM

This works extremely well for semantic similarity.

Ask:

"What was the reasoning behind our analytics architecture?"

Vector search can find relevant discussions even if the exact wording is different.

But now ask:

"Where is SYNC_BATCH_SIZE defined?"

That's an exact identifier.

Keyword search is better.

Or ask:

"What services depend on PostgreSQL?"

That's a relationship question.

A knowledge graph is better.

This is why I think a serious AI Second Brain needs Multi-RAG.

Multi-RAG: Different Questions Need Different Retrieval

Instead of relying on one retrieval technique:

Semantic Search
       +
Full-Text Search
       +
Knowledge Graph
       +
Metadata Filtering
       +
Reranking

Each method solves a different problem.

Semantic RAG

Good for understanding concepts.

"Why did we choose DuckDB?"

Keyword RAG

Good for exact information.

"Find references to SYNC_BATCH_SIZE."

Graph RAG

Good for relationships.

"Which services depend on PostgreSQL?"

Memory Retrieval

Good for decisions and persistent context.

"What did we decide about the analytics architecture?"

Together, they provide much richer context than vector search alone.

The Architecture

The important idea isn't any individual component.

It's how the components work together.

Obsidian / Documents / External Sources
                │
                ▼
           Event Queue
                │
                ▼
        Processing Workers
          │      │      │
          ▼      ▼      ▼
       Metadata Embedding Graph
          │      │      │
          └──────┼──────┘
                 ▼
          Knowledge Layer
                 │
      ┌──────────┼──────────┐
      ▼          ▼          ▼
   Vector      Search      Graph
   Search      Index       Store
      │          │          │
      └──────────┼──────────┘
                 ▼
          Retrieval Engine
                 │
                 ▼
              MCP / API
                 │
                 ▼
          Claude / GPT / AI

Obsidian Is the Human Interface

I don't see Obsidian as the entire Second Brain.

I see it as one of the best human-facing knowledge interfaces.

You can continue writing naturally:

# Analytics Architecture

InsightTrack uses [[DuckDB]] for analytical queries.

Data originates in [[PostgreSQL]].

See [[Pulse]] for the AI analytics layer.

Obsidian gives you human-readable notes and a native graph.

The AI system adds another layer on top.

Two Graphs, Two Purposes

This architecture can maintain two different graphs.

Obsidian Graph

Created from explicit links:

[[InsightTrack]]
       │
       ▼
   [[DuckDB]]
       │
       ▼
  [[Analytics]]

This is primarily useful for human navigation.

System Entity Graph

Automatically extracted from all available knowledge:

InsightTrack
      │
     USES
      │
   DuckDB
   /    \
READS   SYNCED_FROM
 /          \

Pulse PostgreSQL

The system graph can be built from:

Markdown
Source code
Git commits
Pull requests
Database schemas
Documentation
Conversations
Meeting notes

This graph isn't just for visualization.

It becomes part of retrieval.

Knowledge Engineering + Graph Engineering

This is where the architecture becomes more interesting than a normal RAG application.

Knowledge Engineering turns raw information into structured knowledge.

Meeting Transcript
        ↓
     Summary
        ↓
   Action Items
        ↓
Architecture Decision
        ↓
   Project Memory

Graph Engineering connects the resulting entities.

Project
   ↓
Repository
   ↓
  API
   ↓
Database
   ↓
Developer
   ↓
Decision

The result isn't just a collection of documents.

It's a connected knowledge model.

The Retrieval Engine

The retrieval engine is arguably the most important part.

Suppose you ask:

Why did we move analytics queries from PostgreSQL to DuckDB?

The system might retrieve:

GitHub PR
      +
Architecture Note
      +
Benchmark
      +
Slack Discussion
      +
Decision Memory

Then rerank and combine the results.

Instead of giving Claude thousands of chunks, the system builds a focused context:

Decision:
Use DuckDB for analytical workloads.

Evidence:
- Benchmarks showed better aggregation performance.
- PostgreSQL remains the transactional source of truth.
- The architecture PR implemented the separation.
- The Slack discussion explains the operational trade-offs.

Now Claude is reasoning over evidence + relationships + memory.

Long-Term Memory

A Second Brain shouldn't remember everything.

Otherwise, it becomes another giant database.

It should remember what matters.

For example:

Decision
────────────────────
PostgreSQL remains the source of truth.

Decision
────────────────────
DuckDB handles analytical reads.

Reason
────────────────────
Better analytical performance and isolation.

Status
────────────────────
Implemented.

This is much more valuable than storing thousands of raw conversations.

The system should distinguish between:

Raw Data
   ↓
Information
   ↓
Knowledge
   ↓
Memory

That's an important part of making AI memory useful.

MCP Connects the Second Brain to Claude

This is where the architecture becomes AI-native.

Instead of building a custom integration directly into Claude, expose the Second Brain through MCP.

For example:

search()
retrieve()
remember()
ingest()
graph()
timeline()

Claude can ask:

search(
  "Why did we choose DuckDB?"
)

The Second Brain handles the complexity.

Claude doesn't need to know whether the answer came from:

PostgreSQL
pgvector
Search Index
Knowledge Graph
Obsidian
GitHub
Slack

It simply receives useful context.

Claude Can Write Back

This is where it becomes more than traditional RAG.

The flow isn't only:

Knowledge → AI

It becomes:

Knowledge ↔ AI

For example:

"Remember that we decided to keep PostgreSQL as the source of truth."

Claude calls:

remember(...)

The system can then:

Extract Entities
       ↓
Extract Relationships
       ↓
   Create Memory
       ↓
Generate Embedding
       ↓
   Update Graph
       ↓
   Store Source

The knowledge base becomes progressively richer.

The Model Doesn't Own the Knowledge

This is one of the most important ideas.

Today:

Claude
   ↓
Claude Memory

Another AI:

GPT
   ↓
GPT Memory

Another:

Gemini
   ↓
Gemini Memory

Everything is isolated.

Instead:

             Claude
                │
               GPT
                │
             Gemini
                │
              Cursor
                │
                ▼
         ┌───────────────┐
         │  Knowledge OS │
         └───────────────┘

The model can change.

The knowledge stays.

The Bigger Idea

I don't think the future is simply:

"Give Claude more context."

I think it's:

Build a persistent knowledge layer that every AI can use.

Obsidian can remain your human knowledge interface.

GitHub can remain your source-code system.

Slack can remain your communication system.

Your databases can remain your data systems.

Your AI assistants can remain your reasoning systems.

The Knowledge OS connects them.

Beyond the Second Brain

A traditional Second Brain helps you remember.

An AI-native Second Brain helps AI understand.

It combines:

Knowledge Engineering — turning raw information into structured knowledge
Graph Engineering — discovering entities and relationships
Hybrid Retrieval — combining semantic, keyword, and graph search
Long-Term Memory — preserving decisions and important context
MCP — making the knowledge available to AI agents and assistants

The result is something larger than a note-taking application or a vector database.

An AI-native Knowledge Operating System that combines Knowledge Engineering, Graph Engineering, Hybrid Retrieval, and Long-Term Memory into a unified platform accessible through MCP and APIs.

The LLM provides the reasoning.

The Knowledge OS provides the memory.

And MCP becomes the bridge between them.

Mind Maps in MCP: A Powerful Feature or Just Eye Candy?

Nishikanta Ray — Sat, 01 Aug 2026 18:27:45 +0000

Mind maps are an excellent addition to an MCP ecosystem when they help users understand relationships between data, tools, and reasoning. They should complement—not replace—traditional dashboards, tables, and chat interfaces.

Introduction

Large Language Models have become incredibly good at answering questions. Ask about your analytics, codebase, documentation, or infrastructure, and they'll generate a detailed response in seconds.

But there's a problem.

Most AI conversations are linear, while the problems we're solving are non-linear.

Imagine asking:

Why did traffic drop yesterday?

A traditional AI responds with several paragraphs of text.

A mind map, however, immediately shows the relationships between traffic sources, landing pages, countries, devices, deployments, and JavaScript errors.

Instead of reading hundreds of words, you understand the entire situation in seconds.

This naturally raises an interesting question:

Should every MCP server support mind maps?

After exploring different architectures and use cases, my conclusion is simple:

Yes—but only when they help users understand relationships rather than simply displaying information.

Let's explore why.

What is a Mind Map in MCP?

An MCP (Model Context Protocol) server exposes tools that AI models can use.

Normally the workflow looks like this:

User
  │
  ▼
LLM
  │
  ▼
MCP Tool
  │
  ▼
Database
  │
  ▼
Response

The AI collects data, summarizes it, and returns text.

Now imagine adding a visualization layer.

             User
               │
               ▼
          AI Planner
               │
      ┌────────┴────────┐
      ▼                 ▼
 Analytics Tool    Documentation Tool
      ▼                 ▼
 Error Tool        Code Search Tool
      ▼                 ▼
      └────────┬────────┘
               ▼
     Relationship Builder
               ▼
      Interactive Mind Map

The AI still performs the same work.

The difference is that users now see how everything connects instead of reading a long explanation.

Why Traditional AI Isn't Always Enough

Humans naturally think in relationships.

When someone asks:

Why are conversions down?

They aren't thinking linearly.

They're exploring multiple possibilities simultaneously.

Conversions
│
├── Traffic
│
├── Checkout
│
├── Marketing
│
├── Performance
│
├── Returning Users
│
└── Errors

Each branch creates another investigation.

This is exactly how troubleshooting works.

A mind map simply visualizes that thought process.

Real Analytics Example

Suppose an AI receives this question:

Why did yesterday's traffic decrease?

Instead of returning paragraphs, it could generate something like:

Traffic Drop
│
├── Organic Search ↓
│
├── Homepage ↓
│
├── India ↓
│
├── JavaScript Errors ↑
│
├── Core Web Vitals ↓
│
└── New Deployment ✓

Within seconds the user understands:

Organic search decreased
Homepage was affected
Most impact came from India
JavaScript errors increased
A deployment happened recently

The graph immediately tells a story.

Where Mind Maps Work Best

1. Analytics Investigation

Traditional dashboards answer:

How many visitors?
Which country?
Which browser?

Mind maps answer:

Why did traffic change?
Which metrics influence each other?
What caused this anomaly?

Example:

Traffic
│
├── Organic
│   ├── Landing Pages
│   ├── Keywords
│   └── Countries
│
├── Paid
│
├── Social
│
└── Referral

This transforms disconnected metrics into connected insights.

2. Documentation Search

Imagine asking:

Explain authentication.

Instead of returning dozens of documents, the AI groups concepts together.

Authentication
│
├── OAuth
├── JWT
├── API Keys
├── Sessions
└── Permissions

Users instantly understand the overall architecture.

3. Codebase Exploration

Large repositories can be overwhelming.

Instead of thousands of files:

Backend
│
├── API
│
├── Database
│   ├── PostgreSQL
│   └── Prisma
│
├── Authentication
│
└── Workers

Developers can navigate the repository visually.

4. AI Agent Planning

Most AI agents perform many hidden steps.

A mind map exposes them.

Task
│
├── Research
│
├── Search
│
├── Compare
│
├── Generate
│
└── Validate

This makes debugging AI workflows dramatically easier.

The Biggest Advantages

Better Understanding

Humans recognize patterns much faster than paragraphs.

Instead of reading 800 words, users understand the entire structure in seconds.

Relationship Discovery

Traditional dashboards display numbers.

Mind maps display relationships.

For example:

Deployment
      │
      ▼
JavaScript Errors
      │
      ▼
Checkout Failure
      │
      ▼
Revenue Drop

That chain is much harder to discover from charts alone.

Easier Navigation

Users don't need every detail immediately.

They can expand only the sections they care about.

Traffic
  │
  ├── Organic
  │
  ├── Direct
  │
  └── Social

This keeps complex systems manageable.

Better AI Transparency

Instead of wondering why the AI reached a conclusion, users can inspect the reasoning path visually.

This increases trust in AI-generated answers.

Where Mind Maps Fail

Like every visualization, they have limitations.

Everything Becomes a Graph

One common mistake is trying to visualize every piece of information.

Imagine this:

Visitors
│
├── Countries
├── Browsers
├── Devices
├── Sessions
├── Events
├── Campaigns
├── Pages
├── Heatmaps
├── Errors
├── Funnels
├── Scroll Depth
├── UTM
├── Performance
└── ...

After a few hundred nodes, the graph becomes unreadable.

Tables Are Sometimes Better

Suppose someone asks:

Show today's visitors.

They don't need a graph.

They need this:

Hour	Visitors
9 AM	241
10 AM	318
11 AM	401

Not every question benefits from visualization.

Performance

Large graphs become expensive.

Some challenges include:

Thousands of nodes
Complex edge calculations
Layout algorithms
Browser memory usage
Slow rendering

Most graph libraries begin struggling once graphs become extremely large unless clustering or virtualization is implemented.

Additional Maintenance

Relationships don't build themselves.

Every new feature means:

New node types
New edge rules
Updated layouts
Better ranking algorithms

Graph maintenance quickly becomes its own subsystem.

Common Mistakes

Many developers build graphs that simply mirror database tables.

Example:

Country
   │
   ▼
India
   │
   ▼
Users
   │
   ▼
5000

That provides almost no value.

Instead, focus on relationships.

Example:

Traffic Drop
│
├── Homepage
├── Organic Search
├── India
└── JavaScript Errors

The graph explains why something happened.

Should Every MCP Include Mind Maps?

Not necessarily.

Here's a simple rule.

Great Fit	Poor Fit
Documentation	Raw SQL
AI reasoning	CSV exports
Research	Tables
Architecture	Dashboards
Agent workflows	Metric reports
Knowledge graphs	Simple searches

Mind maps are strongest when users need to understand connections, not just retrieve information.

A Better Architecture

Rather than replacing the existing interface, add mind maps as an exploration mode.

Dashboard
│
├── Visitors
├── Revenue
├── Funnels
├── Heatmaps
└── Errors

        │
        ▼

🧠 Explore Relationships

        │
        ▼

Traffic
│
├── Acquisition
├── Geography
├── Pages
├── Errors
└── Performance

The dashboard remains familiar.

The graph becomes a discovery tool.

This hybrid approach gives users the best of both worlds.

Recommended Libraries

If you're planning to build interactive graph visualizations, these libraries are worth considering.

React Flow

Perfect for:

Interactive editors
Expandable nodes
Workflow builders
AI reasoning graphs

Cytoscape.js

Excellent for:

Large relationship graphs
Knowledge visualization
Network analysis

AntV G6

Great for:

Enterprise dashboards
Knowledge graphs
Automatic layouts

Sigma.js

Designed for:

Very large graphs
High-performance rendering
Thousands of nodes

D3.js

Ideal when you need complete customization and don't mind writing more code.

My Recommendation

If I were designing an MCP platform today, I wouldn't make the mind map the default interface.

Instead, I'd treat it as an optional exploration layer.

Users should still have access to:

Traditional chat
Tables
Dashboards
Charts
SQL results

When they want to understand relationships, they can switch to the graph view.

This keeps the interface simple while providing powerful visualization when it's genuinely useful.

Final Thoughts

Mind maps aren't valuable because they look impressive.

They're valuable because they help users understand relationships that are difficult to see in text, tables, or dashboards.

For MCP ecosystems, they're especially effective for:

AI reasoning
Documentation discovery
Codebase exploration
Knowledge graphs
Multi-agent workflows
Analytics investigations

However, they should never replace dashboards or structured data.

The most effective MCP platforms will combine both approaches:

Chat for conversations
Dashboards for metrics
Mind maps for relationships

Use each interface where it excels.

In the end, the goal isn't to make AI prettier—it's to make complex information easier to understand.

Running Hermes Agent with Kokoro TTS: A Local-First AI Assistant Setup

Nishikanta Ray — Sun, 26 Jul 2026 07:19:14 +0000

Most AI agents today depend heavily on cloud APIs. They're fast, but every request costs money, depends on an internet connection, and sends your data to external providers.

Over the weekend, I experimented with Hermes Agent and Kokoro TTS, building a completely local voice-enabled AI assistant. The result was surprisingly good—responses were generated locally, converted into natural speech, and even delivered through Telegram.

It felt much closer to having a personal AI operating system than simply chatting with an LLM.

What is Hermes Agent?

Hermes Agent is an open-source AI agent framework from Nous Research designed for long-running AI workflows. Unlike a traditional chatbot, Hermes can:

Execute tools
Remember previous interactions
Learn reusable skills
Schedule recurring jobs
Connect with messaging platforms like Telegram
Work with both cloud and local models

It also supports OpenAI-compatible endpoints, making it easy to swap between cloud providers and local inference engines.

Why use Kokoro TTS?

Kokoro is a lightweight text-to-speech engine that runs locally and produces surprisingly natural voices.

Instead of sending text to cloud TTS providers, everything happens on your own machine.

Benefits

Completely offline
No API costs
Low latency
Privacy-friendly
Easy integration with local AI workflows

For demos, assistants, and automation, it feels almost instant.

Architecture

                User
                  │
          Telegram / CLI
                  │
          ┌───────────────┐
          │ Hermes Agent  │
          └──────┬────────┘
                 │
      ┌──────────┴──────────┐
      │                     │
 Local LLM             Tool Calls
(Qwen/Llama/etc.)      Browser/MCP/etc.
      │
      ▼
 Response Text
      │
      ▼
 Kokoro TTS
      │
      ▼
 Natural Voice
      │
      ▼
 Telegram Voice Message

The same response can be delivered as:

Text
Voice
Or both simultaneously

Why run locally?

A local-first setup offers several advantages:

Privacy

Sensitive conversations never leave your machine unless you explicitly enable external tools.

Zero inference cost

After downloading the models, you no longer pay per request.

Faster iteration

Developers can test prompts, agents, and automations without worrying about API limits.

Offline capability

Useful for demos, labs, or environments with limited internet access.

Typical Setup

A common local stack looks like this:

Hermes Agent
Ollama or another OpenAI-compatible local model server
Kokoro TTS
Telegram Gateway (optional)
MCP servers
Local memory and skills

Hermes can connect to local model servers such as Ollama, LM Studio, vLLM, llama.cpp, and other OpenAI-compatible endpoints.

Real Use Cases

Personal AI Assistant

Ask questions and receive spoken replies.

Coding Agent

Read repositories, execute tools, and explain code aloud.

Customer Support Bot

Generate text and voice responses without cloud inference costs.

Home Automation

Control local services with voice feedback.

Content Creation

Generate narration for tutorials, demos, podcasts, and videos.

Research Assistant

Summarise documents and listen instead of reading long reports.

Pros

✅ Local inference

✅ Better privacy

✅ No recurring API costs

✅ Natural voice responses

✅ Works with existing OpenAI-compatible APIs

✅ Flexible model switching

✅ Extensible through tools and skills

Cons

❌ Requires decent CPU/GPU resources

❌ Initial model downloads can be large

❌ Local models may be slower than premium cloud models

❌ Voice quality depends on the selected TTS model

❌ Some external tools (web search, browser automation, cloud speech) still require internet access if enabled.

Who should use this?

This setup is ideal for:

AI developers
Open-source enthusiasts
Privacy-conscious users
Local LLM experimentation
Voice assistant projects
Internal enterprise tooling

If you're already using MCP servers or self-hosted AI infrastructure, Hermes integrates naturally into that workflow.

Final Thoughts

Hermes Agent combined with Kokoro TTS creates a surprisingly capable local AI assistant. Instead of just generating text, the system can reason, use tools, remember context, and respond with natural speech—all while keeping inference largely on your own machine.

For developers building AI assistants, coding agents, research tools, or voice-enabled applications, this local-first stack offers an excellent balance between privacy, flexibility, and cost.

After experimenting with it over the weekend, it's easy to see why local AI workflows are becoming increasingly practical. Cloud models still have their place for the hardest reasoning tasks, but for many day-to-day workflows, a local setup is already more than capable.