DEV Community: puffball1567

KoutenDB v0.10.0: 72 Hours, Three Persistent Nodes, and Zero Client Errors

puffball1567 — Mon, 03 Aug 2026 12:23:51 +0000

I released KoutenDB v0.10.0.

Release:

https://github.com/puffball1567/koutendb/releases/tag/v0.10.0

KoutenDB is a ring-oriented document and vector database written in Nim. Its core idea is to make an application's locality boundary part of the read path:
place related data together, select that boundary before retrieval, and avoid making unrelated records candidates in the first place.

That idea needs more than a clean benchmark. A database also has to keep running while it writes persistent state, serves reads, tracks topology state, and eventually shuts down for verification.

For v0.10.0, I ran a local persistent three-node cluster continuously for 72 hours. The result was simple and useful: the run completed with zero client errors, and all three data directories passed offline verification after the servers stopped.

What the endurance run did

The runner started three local TCP nodes with persistence enabled and exercised
a mixed workload every 250 ms for 259,200 seconds.

The workload included:

TCP writes;
point reads using IDs returned by earlier writes;
JSON projection queries;
bounded ring reads with sort and limit;
ring-scoped exact vector retrieval;
metrics collection;
a final snapshot and offline verification after shutdown.

The runner builds its own copy of the server, CLI, and workload binaries under the run directory before it starts. This matters because rebuilding the source tree during a long run cannot silently change the executable being tested.

The exact run used commit 87c755c9130ec0bbf70a3903c75fd2bdae8b084b.

Final result

Operation	Completed
PUT	969,281
returned-ID GET	969,281
projection query	969,281
bounded ring read	969,281
ring-scoped retrieve	96,928
metrics read	48,464
client errors	0

That is 4,022,516 completed logical client operations across the 72-hour period.

After the workload finished, the runner took its final snapshot, stopped the three nodes, and ran offline kouten verify against every persistent data directory. All three verifications succeeded.

The final operational counters also showed:

zero pending handoffs;
zero handoff queue depth;
zero handoff failures, stale acknowledgements, or queue-full events;
zero remaining migrations;
zero universe-sync errors;
zero global retrieval requests.

The last point is important for KoutenDB's design. The workload used ring-scoped retrieval, and the metrics confirmed that it did not silently fall back to a cluster-wide retrieval path.

Why this is more useful than a startup smoke test

A startup test can show that a server accepts a request. It does not show what happens after hundreds of thousands of persistent writes, repeated read paths, periodic metrics calls, topology bookkeeping, and a full stop-and-reopen boundary.

This run specifically exercised the persistent cluster path over time:

records were written to disk-backed nodes;
later requests read data that had been accumulated by the same run;
cluster handoff and migration counters were observed throughout the run;
the servers were stopped cleanly;
the on-disk state was opened and verified offline.

It does establish a concrete baseline: KoutenDB can sustain a mixed, persistent local cluster workload for three days without a client-visible error or an offline integrity failure.

What changed in v0.10.0

The endurance runner is one part of a larger operational hardening release.

Operational configuration can now be loaded and verified before a server is used.
Write guardrails, audit events, capacity thresholds, and backup verification provide explicit operational checks.
Explicit scale-in migration and rolling topology activation add controlled drain, handoff, and progress behavior.
Cluster retrieval now remains in the requested ring rather than falling back to a cluster-wide scan.
Physical placement is decoupled from the logical orbit schedule, so a logical orbit boundary does not stall local request processing.
Handoff I/O is kept off the request loop, and stale handoff replay cannot resurrect an older mutation.
The optional FAISS backend was removed. Vector retrieval now follows the core KoutenDB model directly: select a ring first, then perform exact cosine ranking over that bounded candidate set.

These are not unrelated features. They move the project from a collection of locality experiments toward an operable persistent service: know what the server is configured to do, constrain unsafe operation, observe background state, and verify the data after a long workload.

Reproducing the run

The runner is included in the repository:

KOUTEN_SOAK_SECONDS=259200 \
KOUTEN_SOAK_WORKDIR=/tmp/koutendb-soak-72h \
examples/soak_72h.sh

It writes JSON Lines progress, node logs, final metrics, a final snapshot, offline verification output, the workload configuration, and a completed.ok marker only after all final verification steps succeed.

The full configuration and scope are documented here:

https://github.com/puffball1567/koutendb/blob/main/docs/soak-testing.md

The run is intentionally not a CI job. A three-day endurance test is useful because it is long enough to observe persistent operational behavior; it would make normal pull-request feedback unusably slow.

The next reliability question

This result validates the current local, disk-backed, buffered-durability cluster mode. The next endurance variants are clear: strong-durability writes, TLS and authenticated transport, Docker/container deployment, and eventually multiple machines.

Those tests extend this baseline rather than replace it. The v0.10.0 result is now a documented, reproducible reference point for future changes to storage, handoff, topology, and retrieval behavior.

Source:

https://github.com/puffball1567/koutendb

Clay Board Style System update: Link Navigation

puffball1567 — Mon, 03 Aug 2026 11:52:50 +0000

Clay Board Style System now provides typed native Link Navigation.

For readability, I will refer to Clay Board Style System as CBSS below.
CBSS is only an abbreviation used in this article; the project's official name
is Clay Board Style System.

The feature lets a native application move between screens with a flow similar
to a Single-Page Application: the window remains open, the navigation menu
remains in place, and only the active screen changes.

In this tutorial, we will build a small navigation menu with three destinations:

Home
Projects
Settings

Clicking a menu item switches the visible screen inside the same native SDL3
window. No DOM, browser history, WebView, or URL-string router is required.

Run the finished demo

The complete interactive demo is already included in the repository:

git clone https://github.com/puffball1567/clay-board-style-system.git
cd clay-board-style-system
nimble setupBundled
nimble navigationDemo

The full source is available in
examples/navigation_demo.nim.

Step 1: Define the screens

Start by describing the destinations as a normal Nim enum:

import clay_board_style_system

type Screen = enum
  homeScreen,
  projectsScreen,
  settingsScreen

let navigator = initStackNavigator(homeScreen)

navigator now owns the screen history. The initial destination is
homeScreen.

Because destinations are Nim values, invalid destinations are caught by the
compiler instead of becoming malformed route strings at runtime.

Step 2: Define a small set of styles

The navigation behavior does not impose a visual design. These styles create a
simple sidebar and content area:

proc appStyle(): UiStyle =
  uiStyle([
    decl("width", px(900)),
    decl("height", px(520)),
    decl("flex-direction", keyword("row")),
    decl("background-color", colorValue(rgb(0.04, 0.05, 0.08)))
  ])

proc menuStyle(): UiStyle =
  uiStyle([
    decl("width", px(220)),
    decl("height", px(520)),
    decl("padding", px(20)),
    decl("gap", px(10)),
    decl("flex-direction", keyword("column")),
    decl("background-color", colorValue(rgb(0.07, 0.09, 0.13)))
  ])

proc contentStyle(): UiStyle =
  uiStyle([
    decl("width", px(680)),
    decl("height", px(520)),
    decl("position", keyword("relative")),
    decl("overflow", keyword("hidden"))
  ])

proc screenStyle(): UiStyle =
  uiStyle([
    decl("position", keyword("absolute")),
    decl("left", px(0)),
    decl("top", px(0)),
    decl("width", px(680)),
    decl("height", px(520)),
    decl("padding", px(28)),
    decl("gap", px(12)),
    decl("flex-direction", keyword("column")),
    decl("background-color", colorValue(rgb(0.04, 0.05, 0.08)))
  ])

proc linkStyle(): UiStyle =
  uiStyle([
    decl("width", px(180)),
    decl("height", px(40)),
    decl("padding", px(10)),
    decl("align-items", keyword("center")),
    decl("background-color", colorValue(rgb(0.11, 0.15, 0.21))),
    decl("border-radius", px(6)),
    decl("cursor", keyword("pointer"))
  ])

proc linkTextStyle(): UiStyle =
  uiStyle([
    decl("font-size", px(14)),
    decl("line-height", px(20)),
    decl("color", colorValue(rgb(0.91, 0.94, 0.98)))
  ])

These are ordinary CBSS styles. A component library can replace all of them
without changing the navigation code.

Step 3: Build the menu and screens

Keep the three screen roots so they can be registered with the navigator. We
also keep the Back and Forward buttons for the next step.

type AppView = object
  home: NodeHandle
  projects: NodeHandle
  settings: NodeHandle
  backButton: ButtonHandle
  forwardButton: ButtonHandle
  refreshProjects: ButtonHandle
  notifications: CheckboxHandle

proc buildView(ui: UiRoot; navigator: Navigator[Screen]): AppView =
  ui.box(appStyle()):
    ui.box(menuStyle()):
      ui.text("My application")

      ui.link(
        navigator,
        homeScreen,
        "Home",
        style = linkStyle(),
        textStyle = linkTextStyle()
      )

      ui.link(
        navigator,
        projectsScreen,
        "Projects",
        style = linkStyle(),
        textStyle = linkTextStyle()
      )

      ui.link(
        navigator,
        settingsScreen,
        "Settings",
        style = linkStyle(),
        textStyle = linkTextStyle()
      )

      result.backButton = ui.button("Back")
      result.forwardButton = ui.button("Forward")

    ui.box(contentStyle()):
      ui.box(result.home, screenStyle()):
        ui.text("Home")
        ui.text("Welcome to the native application.")

      ui.box(result.projects, screenStyle()):
        ui.text("Projects")
        ui.text("Project data will be displayed here.")
        result.refreshProjects = ui.button("Refresh projects")

      ui.box(result.settings, screenStyle()):
        ui.text("Settings")
        result.notifications =
          ui.checkbox("Enable notifications", checked = true)

The hierarchy is visible directly in the code:

application
├── navigation menu
│   ├── Home Link
│   ├── Projects Link
│   ├── Settings Link
│   ├── Back button
│   └── Forward button
└── content area
    ├── Home screen
    ├── Projects screen
    └── Settings screen

ui.link() is a style-neutral native Link. It handles pointer activation,
Enter-key activation, keyboard focus, disabled state, and accessibility
semantics.

Step 4: Register the screens

Build the view once, then associate each typed destination with its screen root:

let ui = initUiRoot()
let navigator = initStackNavigator(homeScreen)
let view = buildView(ui, navigator)

let host = initNavigationScreenHost(ui, navigator)
host.registerScreen(homeScreen, view.home)
host.registerScreen(projectsScreen, view.projects)
host.registerScreen(settingsScreen, view.settings)

var interaction = initInteractionState()

if not host.sync(interaction):
  raise newException(ValueError, "initial screen could not be activated")

At this point, only Home is active.

When a Link receives a click or Enter-key event, it updates the navigator. After
the current platform-event batch, synchronize the screen host:

let screenChanged = host.sync(interaction)

In a real application this line belongs in the central event loop, after CBSS
has dispatched the current batch of SDL3 events. The complete demo shows that
integration without hiding it behind tutorial pseudocode.

Only the active screen participates in layout, painting, hit testing, keyboard
focus, and the visible accessibility tree. The other screens remain retained
but inert.

That is the SPA-like part: switching screens does not reconstruct the complete
application tree or open another native window.

Add Back and Forward

The same navigator provides history operations:

view.backButton.onClick = proc(event: DispatchResult): bool =
  if not navigator.back():
    echo "Already at the first screen"
  true

view.forwardButton.onClick = proc(event: DispatchResult): bool =
  if not navigator.forward():
    echo "Already at the latest screen"
  true

push, replace, back, and forward operate on typed history entries. Two
visits to Projects remain two separate entries, which allows CBSS to restore
focus for the correct visit when navigating through history.

Request data with Joubako

Navigation belongs to CBSS, while HTTP communication belongs to the
application's data layer.

I am also developing
Joubako, a separate async transport
client for Nim. It supports HTTP(S), typed JSON, standard Nim await,
result-aware callback composition, WebSockets, local IPC, and optional NIF/BIF
data exchange.

It can be started from a normal CBSS event handler without coupling networking
to the navigation system:

import std/asyncdispatch
import joubako

type Project = object
  id: int
  name: string

let api = newClient(newHttpTransport(), "https://api.example.com/")

proc loadProjects() {.async.} =
  let outcome = await api.getJson("projects", seq[Project])
  if outcome.isErr:
    echo "Request failed: ", outcome.error.msg
    return

  echo "Loaded projects: ", outcome.value.len
  # Update application state and mark the affected view dirty here.

view.refreshProjects.onClick = proc(event: DispatchResult): bool =
  asyncCheck loadProjects()
  true

Joubako is not bundled with CBSS and is not required for Link Navigation. An
application can use Joubako, another HTTP client, IPC, an FFI service, or a
fully local data source behind the same event-handler boundary.

Optional features

The basic menu above is enough for normal in-process screen navigation. CBSS
also provides optional APIs for:

focus restoration for each history entry;
animated screen transitions;
validated application deep links;
replacing one retained screen without rebuilding the application; and
injecting a custom navigation driver.

These features are documented in the
Native Navigation guide.

Why use typed destinations?

A browser URL is useful when the URL is the application's public address. A
native in-process screen does not always need that string boundary.

With CBSS:

navigator.push(projectsScreen)

is checked against Screen by Nim. A larger application can replace the enum
with a variant object that carries typed parameters, such as a project ID.

External deep links can still be decoded into the same destination type at the
edge of the application. Internal navigation remains typed.

Current status

Clay Board Style System v0.3 is a developer preview. Linux x86_64 with SDL3 is
currently the Tier 1 runtime. Windows and macOS portable builds run in CI, while
complete runtime validation on those platforms remains contributor-driven.

Link Navigation is covered by unit tests, retained-screen tests, transition and
focus tests, performance checks, and a real SDL3/Wayland E2E scenario.

Links

Feedback on the Link API and the navigation-menu authoring experience is very
welcome.

RAG Retrieval Optimization: Reduce Vector Search Before Ranking

puffball1567 — Mon, 03 Aug 2026 04:21:03 +0000

Most RAG performance advice begins at the ranking stage: choose a faster embedding model, tune an ANN index, reduce the result count, add a reranker, or cache common queries.

Those are useful techniques. But RAG retrieval optimization should start one step earlier:

Why is this query considering these vectors at all?

A request often already contains a reliable boundary: tenant, repository, product, language, document type, version, date range, permission scope, or the object currently open in the application. Applying that knowledge before vector ranking can reduce RAG latency, vector-search memory use, and unrelated context.

This is the difference between searching a whole corpus for similar documents and searching the authorized, relevant part of that corpus. It is also the specific problem that a locality-aware RAG database such as
KoutenDB explores.

Vector search metadata filtering is not one operation

Vector search metadata filtering commonly restricts a query by fields such as:

tenant_id = "acme"
AND product = "billing"
AND language = "en"
AND published = true
AND version = "2026.2"

Those constraints are necessary for both relevance and security. Their placement in the read path, however, changes the cost:

Strategy	What happens
Filter after broad retrieval	Rank widely, then discard ineligible candidates.
Filter-aware vector index	Use indexed metadata while selecting vector candidates.
Namespace or partition selection	Select an eligible subset, then search it.
Application-local routing	Route directly to a known tenant, source, or related-data neighborhood.

The first approach can still enforce correct access rules, but it may score vectors that should never have been candidates. The latter two are valuable when the application already knows a stable boundary.

For a multi-tenant support assistant, the tenant boundary is not merely a relevance hint. It is an authorization rule and a natural first search scope.
For a code assistant, the active repository and branch often serve the same purpose. For a product assistant, version and language can rule out most of the corpus before similarity becomes useful.

RAG optimization starts with the working set

The working set is everything a request touches: candidate vectors, metadata,payload bytes, reranker inputs, and finally model context. Reducing it changes several costs at once:

fewer vectors to score;
less vector memory to read;
fewer payloads to filter and project;
fewer candidates for a reranker;
less irrelevant context competing for the prompt budget.

This is not an argument for creating a partition for every tag. A pre-ranking boundary should be stable, known at request time, meaningful to the application,and strong enough to exclude large amounts of unrelated data. Tenant, source,product, and document-version boundaries often qualify; free-form tags usually do not.

There is also an essential quality check: a smaller scope is only an optimization if it retains the documents the user needs. A wrong partition can be fast because it is wrong.

KoutenDB: locality first, exact vector ranking second

KoutenDB is an open-source, embedded-capable document and vector database written in Nim. It is not a general replacement for PostgreSQL, a mature ANN vector database, or a global secondary-index engine.

Its vector path makes a focused trade-off:

the application chooses a ring, a semantic locality boundary;
KoutenDB retrieves candidates from that ring;
it performs dependency-free exact cosine ranking over that bounded set.

For example, a RAG application that already knows the tenant and product can place and query documentation in a corresponding ring:

import koutendb

var db = koutendb.open(dataDir = "data")
let hits = db.retrieve(
  @[1.0'f32, 0.0'f32],
  ring = "tenant/acme/product/billing",
  budget = 8
)

The ring is chosen by application logic, policy, or an import rule; KoutenDB does not claim to infer the right security or business boundary from embedding similarity. JSONL ingestion can derive rings from a field, for example with a tenant field and a tenant prefix.

Filters and projections still narrow results, but only after the local ring has been selected. This boundary is important: KoutenDB is a good fit when an application can name a meaningful pre-ranking scope. It is not the right primary tool when every query must perform global, cross-corpus discovery.

Measure vectors scanned, not only results returned

Returning fewer hits does not prove that a system did less search work. A useful RAG database should expose whether unrelated candidates were skipped before ranking.

KoutenDB reports total vectors, scanned candidates, skipped vectors, rings touched, candidate reduction, payload bytes, and estimated tokens. Its included working-set benchmark uses 10,000 vectors across 100 rings. In one local run, global retrieval scanned 10,000 vectors per query, while routed retrieval scanned 100: a 99% reduction. Measured latency in that run was 1,954.9 microseconds for the global path and 31.4 microseconds for the routed path.

An included RAG-style test kept recall at 1.000 for the correctly routed ring while reducing scanned candidates from 400 to 40 and estimated tokens from 615.2 to 231.6. It also demonstrates the failure case: selecting the wrong ring keeps the small scan but produces zero recall.

These are local synthetic measurements, not a universal latency claim or a claim that KoutenDB beats every vector database. They demonstrate a narrower invariant: candidates in unrelated rings are skipped before exact vector scoring, rather than discarded after a broad search.

When locality-first retrieval is useful

This approach is particularly relevant for:

multi-tenant RAG and tenant-based data isolation;
support systems scoped to a customer, product, or deployment;
code assistants scoped to a repository or service;
documentation assistants with known language and version;
local AI or desktop applications with a bounded local corpus;
systems that need nearby application history as well as similar chunks.

It is less suitable when global discovery is the core job, or when the application cannot identify a trustworthy boundary before retrieval. In that case, a broad ANN-oriented vector database with filter-aware indexing is often the better primary retrieval layer.

A checklist for RAG retrieval optimization

Before tuning a model or index, ask:

Which tenant, permission, source, version, or task boundary is known before retrieval?
Is that boundary enforced before candidates are ranked?
How many vectors are scanned, rather than merely returned?
Does scoped retrieval preserve recall on representative queries?
Can the system explain why each document was eligible?
Are payload projection and context budgets applied before the LLM call?
Can the same routing rules be validated offline?

The answer may lead to namespaces, metadata indexes, database partitions, or a locality-aware database. The common requirement is to make the pre-ranking boundary explicit and measurable.

Conclusion

The best way to reduce RAG cost or RAG latency is not always a more complicated ranking algorithm. Often it is avoiding a ranking problem that the application already knows is irrelevant.

Use metadata filtering for correctness. Use tenant, source, version, and task locality to reduce the eligible search space when those boundaries are trustworthy. Then measure scanned candidates, recall, latency, and context size together.

That is the design space KoutenDB explores: a database for RAG where application-level locality becomes part of the retrieval path before exact vector ranking begins.

KoutenDB v0.9.0: From a Locality Experiment to a Testable Database

puffball1567 — Sun, 02 Aug 2026 11:09:04 +0000

I released KoutenDB v0.9.0.

Release:

https://github.com/puffball1567/koutendb/releases/tag/v0.9.0

KoutenDB is a ring-oriented NoSQL document and vector database written in Nim.
Its central idea is deliberately narrow: when an application already knows a useful locality boundary, that knowledge should reduce the data a request has to inspect.

Previous releases established the data model, public API, native C ABI, and language-driver foundation. v0.9.0 concentrates on a harder question: can that locality model be tested against a realistic, persistent related-data read shape rather than only described as a design idea?

This is still a technical preview. It is not a claim that KoutenDB replaces PostgreSQL, Redis, or every document store. The goal of this release is to make the narrower claim measurable and reproducible.

The Request Shape: One Entity, Several Bounded Collections

Many web application detail pages need more than one record. A user page, for example, may need a profile, a few addresses, recent employment entries, preferences, recent orders, and notifications. Each collection has its own limit and sort order.

In a relational database, this can be expressed with several indexed queries or with a JSON aggregate query built from limited subqueries. Both are valid approaches. The question KoutenDB explores is different: if these collections are already known to belong to one user, can that relationship be represented as bounded nearby data from the start?

v0.9.0 models the example as subrings below a user ring:

kouten get --ring=users/<id> \
  --subring=profile,addresses,career,preferences,orders,notifications \
  --subring-limit=profile:1,addresses:3,career:2,preferences:1,orders:10,notifications:5 \
  --subring-rsort=orders:time,notifications:time

This says exactly what the endpoint needs: retrieve six nearby collections, with independent bounds. It is not a general query language and it is not intended to be one. The useful property is that the caller supplies the locality boundary before the read begins.

What Changed in v0.9.0

The release adds a benchmark for that heterogeneous related-data bundle and strengthens the corresponding read path.

readStellar now prepares projection state once and reuses it across subrings.
Stellar reads validate per-subring limits and descending time sorts.
Simple embedded reads with an empty filter, a positive limit, and id or time ordering can use a bounded ring-window read path.
Disk-backed segment reads reuse cached streams instead of repeatedly opening the same segment files.
import-jsonl supports chunked commits through --batch-size=N for larger data imports.

The point of these changes is not an isolated micro-optimization. A locality model only helps if its normal persistent read path remains bounded when the dataset grows.

A Reproducible Related-Data Comparison

The repository now includes a helper that builds fresh temporary KoutenDB and PostgreSQL data directories, loads the same logical user dataset, and measures the related-data bundle:

N=10000 READS=1000 examples/subring_bundle_postgres_bench.sh

One local run on 2026-07-21 used an AMD Ryzen 5 5600H, Linux 6.8, Nim 2.2.10, and PostgreSQL 14.23.

Users	Logical records	System and query shape	Returned records	Read latency
10,000	1,050,000	KoutenDB `users/<id>/*` stellar read with per-subring bounds	22 across 6 rings	196.859 us
10,000	1,050,000	PostgreSQL: six indexed `SELECT` statements	22	515 us
10,000	1,050,000	PostgreSQL: JSON aggregate over indexed limited subqueries	1 JSON bundle	236 us

These numbers describe one machine and one workload; they are not a universal database ranking. PostgreSQL's aggregate form is close to the measured KoutenDB result, and it expresses the result through a different but perfectly
reasonable abstraction. The value of the comparison is to keep the claim honest: KoutenDB's ring and subring model should be judged on workloads where the application can name a local working set.

Testing the Larger Locality Claim

v0.9.0 also records the completed disk-backed effect-validation run on generated data. The validation imports deterministic JSONL corpora and compares broad retrieval with ring-routed retrieval. It reports import time, records scanned, estimated token volume, and retrieval latency.

The standard scale-1000 matrix completed locally with a largest case of 13,500,000 documents.

Workload	Broad scan	Ring-routed scan	Estimated tokens: broad -> routed
`small-balanced`	168,000	24,000	692 -> 260
`near-distractors`	1,860,000	120,000	1,730 -> 433
`medium-noisy`	13,500,000	500,000	2,595 -> 692

The token column is an estimate for the generated retrieval payloads, not a benchmark of any model. More generally, this validation is not limited to AI workloads. Scanned records, transferred data, candidate memory, and downstream application work all increase when a request must inspect unrelated data.

The important result is that the test makes the expected trade-off visible:
locality can reduce the candidate set when the application supplies a valid ring boundary. It cannot invent a useful boundary when the application has none.

Offline Validation Before Production Traffic

The release also adds an offline JSONL path:

KOUTEN_REAL_JSONL=/path/to/export.jsonl \
QUERY_RING=docs/japan \
examples/offline_effect_validation.sh

This is intended for copied or exported data rather than a production service.
It gives a team a way to test whether a proposed ring layout narrows its own workload before asking the database to serve live traffic.

That boundary matters. A database experiment should be reproducible without requiring production credentials, traffic capture, or an unbounded benchmark environment.

Verification Included in the Release

The v0.9.0 release state was checked with Nim module checks, the public API test program, the smoke suite, package validation, and whitespace validation:

nim check src/kouten/store.nim
nim check src/koutendb.nim
nim check src/koutencli.nim
nim check tests/tapi.nim
nim c --nimcache:/tmp/nimcache_kouten_tapi -r tests/tapi.nim
scripts/test_all_smoke.sh
nimble check
git diff --check

The largest generated run remains an explicit manual validation rather than a default CI job. It is valuable precisely because it exercises a scale that a quick smoke test should not pretend to cover.

What Comes Next

v0.9.0 makes the locality hypothesis easier to test with persistent data and a concrete web-style read shape. The next work is about making that evaluation more operable: verification commands, controlled drain and snapshot workflows, backup checks, audit records, and safer topology transitions.

Those are v0.10 development goals, not claims included in v0.9.0.

KoutenDB remains an early project, but the direction is now more concrete:
data locality should be a testable part of a database read path, not only a diagram or a benchmark headline.

Source and release notes:

https://github.com/puffball1567/koutendb

Introducing Clay Board Style System: CSS-Inspired Native UI for Nim

puffball1567 — Sun, 02 Aug 2026 10:59:52 +0000

I am releasing the first developer preview of Clay Board Style System, a
CSS-inspired primitive engine for native GUI toolkits written in Nim.

For brevity, this article refers to Clay Board Style System as CBSS after
this point. CBSS is only an abbreviation used in writing; the project's
official name is Clay Board Style System.

Clay Board Style System starts from a simple observation: Web frontend work is
productive partly because CSS gives many frameworks a shared vocabulary for
layout and visual design. Native GUI development often has to rebuild that
foundation separately for every toolkit.

The project is an attempt to provide that primitive layer for native
applications without embedding a browser or recreating the Web platform.

Repository: https://github.com/puffball1567/clay-board-style-system

What Clay Board Style System is

Clay Board Style System provides a tree of primitive Box, Text, and Image
nodes, along with typed styles, layout, paint commands, hit testing, focus,
input dispatch, retained scrolling, and accessibility semantics.

The authoring model intentionally uses familiar CSS ideas:

spacing, borders, radii, colors, and typography;
row and column layout, alignment, sizing, clipping, and scrolling;
absolute and overlay positioning;
state styling for hover, active, focus, and disabled states.

The aim is that someone who can read a Web design article or a CSS example can
bring much of its layout and visual reasoning to a native application.

ui.box(uiStyle([
  decl("padding", px(24)),
  decl("gap", px(12)),
  decl("background-color", colorValue(rgb(0.10, 0.12, 0.16))),
  decl("border-radius", px(16))
])):
  ui.text("A native UI, styled with familiar ideas")

What it is not

Clay Board Style System is not a browser, a WebView, an HTML renderer, or a
full CSS implementation. It does not aim for DOM compatibility, JavaScript,
browser quirks, or bug-for-bug CSS behavior.

It is also not a finished component library. Buttons, dialogs, menus, inputs,
and editors should be built as compositions on top of the primitive layer.

This boundary matters. The goal is to reuse the useful language of Web UI
design while keeping a native rendering, input, and application model.

Why Nim

The first implementation target is Nim because it can produce small native
applications while remaining expressive enough for UI composition. Clay Board
Style System also exposes a versioned C ABI with opaque handles and fixed-size
value structs. That lets an existing C, C++, Rust, Zig, or Swift core be paired
with a Nim frontend, rather than requiring a rewrite before the UI can be
explored.

My hope is that this creates a practical entry point to Nim: keep the existing
engine or product core, build the frontend in Nim, and adopt Nim more broadly
only where it proves useful.

Current developer preview

Version 0.2 targets Linux x86_64 with SDL3. It includes the primitive
style/layout pipeline, text shaping through cosmic-text, event routing, focus,
text input and IME integration, retained scrolling, navigation state, a
semantic accessibility model, and the C ABI.

The project is still early. Windows and macOS require contributor validation;
platform accessibility transports, animation timelines, full rich-text layout,
and some windowing behavior are not complete. Public APIs may change before
1.0.

This is therefore a release for people who want to evaluate the approach,
build experiments, and help shape the primitive layer—not a claim that every
desktop application should use it today.

Try it

Clay Board Style System requires Nim 2.2 or newer, Rust/Cargo for the native
text and image bridges, and SDL3 development files on the current Linux target.

git clone https://github.com/puffball1567/clay-board-style-system.git
cd clay-board-style-system
nimble setupBundled
nimble test
nimble sdl3Demo

The next work is to make the project's visual promise easier to judge: richer
design demos, practical composite controls, and broader platform validation.
Issues, experiments, and feedback are welcome in the repository.

RAG Database Design: SQL, Full-Text Search, Vector Search, and Context Retrieval

puffball1567 — Thu, 23 Jul 2026 14:38:14 +0000

Retrieval-augmented generation is often introduced with a compact diagram:

documents -> embeddings -> vector database -> LLM

That is a useful starting point, but it can make one part of the architecture
look like the whole system.

Good RAG database design has to cover more than vector storage.

A production RAG application may also need to store original documents,
enforce tenant boundaries, find exact product names, filter by date, remember a conversation, invalidate stale content, combine several data sources, and keep the final prompt within a token budget.

No database becomes a "RAG database" merely by storing embeddings. Different database models solve different parts of retrieval.

This article looks at the roles of relational databases, document databases, caches, full-text search engines, and vector databases in RAG. It then examines a retrieval problem that is harder to express with similarity alone: finding a useful neighborhood of context when the application knows the center, but not the exact boundary of the answer.

Among practical AI use cases, this pattern appears in customer support, business-efficiency AI tools for sales, internal knowledge assistants, AI programming automation, and AI agent development. The same retrieval questions also apply to AI assistant development with internal data, where access control and source freshness matter as much as the generated answer.

How RAG works—and how it differs from fine-tuning

RAG retrieves external information when a request is made, then places selected evidence into the model's context. Fine-tuning changes model weights. The practical RAG vs fine-tuning differences are therefore about when and where knowledge is introduced.

RAG is often easier to update, cite, filter, or remove because the knowledge remains outside the model. Fine-tuning can change behavior, style, or learned patterns, but it is not a convenient replacement for a frequently changing document store.

The main RAG pros and cons follow from that separation. RAG can use current and private sources without retraining the model, but it adds ingestion, retrieval, permissions, evaluation, latency, and operations work. Understanding how AI generation works is not enough: a fluent model can still answer from weak, stale, or unauthorized evidence.

Database types and how to build RAG

A more complete RAG pipeline looks like this:

source data
  -> ingestion and normalization
  -> document and metadata storage
  -> candidate selection
  -> lexical or vector ranking
  -> reranking and context assembly
  -> LLM
  -> answer and citations

Databases can participate at several stages.

Responsibility	Typical data
Source of truth	Documents, records, revisions, users, permissions
Ingestion state	Jobs, checksums, parser versions, failures
Exact filtering	Tenant, category, status, language, date
Lexical retrieval	Keywords, names, identifiers, error codes
Semantic retrieval	Embeddings and similarity scores
Session state	Recent messages, selected sources, cached results
Context assembly	Related records, projections, token-bounded payloads

One database can perform several of these jobs. A small application does not need six databases just because six rows appear in the table. The point is to identify the retrieval job before choosing the storage engine. A useful database comparison begins with those jobs, not with a feature-count table.

When teams research how to build RAG, they often begin with an embedding model or a RAG vector database. A safer order is to identify the source of truth, access rules, update process, exact filters, candidate retrieval method, and evaluation plan first. The vector index is one component of that design.

To make the differences concrete, imagine that we are building a travel assistant for Kyoto. It stores information about landmarks, restaurants, cultural facilities, events, transit, opening hours, reservations, and user itineraries.

PostgreSQL and MySQL: structured truth and exact relationships

Relational databases are often the least surprising place to keep the authoritative application state around a RAG system.

For the travel assistant, a relational database could store:

users and accounts;
bookings and reservations;
saved places and itineraries;
tenant or organization boundaries;
document versions and ingestion status;
permissions and visibility rules;
structured opening hours and accessibility fields.

SQL is especially useful when correctness depends on exact relationships. If a private itinerary belongs to one user, or a document must not cross a tenant boundary, that rule should not depend on an embedding score.

A relational query can select the allowed document IDs first. The RAG pipeline can then rank only those documents.

SELECT document_id
FROM travel_documents
WHERE city = 'Kyoto'
  AND language = 'en'
  AND published = true
  AND valid_from <= CURRENT_TIMESTAMP
  AND valid_until > CURRENT_TIMESTAMP;

PostgreSQL also has JSON, full-text search, and extension support. The pgvector extension can keep vectors beside relational data, which is often enough for an early or moderately sized RAG application.

The important distinction is that adding a vector column does not remove the need to design permissions, document lifecycle, metadata, candidate selection, and prompt construction.

MongoDB and document databases: flexible content records

A document database is a natural fit when an ingested item already looks like a JSON object and different item types have different fields.

A landmark record might contain history, images, coordinates, and opening hours. A restaurant may have cuisine, price range, reservation rules, and dietary options. An event may have a start time, venue, ticket URL, and weather policy.

{
  "kind": "restaurant",
  "name": "Example Cafe",
  "area": "Higashiyama",
  "cuisine": ["Japanese", "Cafe"],
  "features": {
    "indoor": true,
    "vegetarianOptions": true
  }
}

MongoDB documents can
represent that shape without forcing every item type into the same set of columns. Metadata and source text can live together, and nested fields can be indexed when their access patterns are known.

Flexibility does not eliminate retrieval design, however. The application still has to decide which collection, fields, filters, and records should become candidates for the model.

The familiar RDB vs NoSQL differences still matter here. Relational systems center exact relationships and constraints; document databases center flexible record shapes and aggregate-oriented access. For business systems, NoSQL database selection criteria should include consistency, authorization, operational skills, backup, and migration—not only whether incoming data is JSON.

Redis: short-lived state and repeated work

Redis is useful around RAG even when it is not the primary document store.

Common uses include:

conversation session state;
cached retrieval results;
rate limits;
short-lived agent state;
job queues;
deduplication keys;
frequently reused context fragments.

If many users ask similar questions about the same attraction, caching a validated retrieval result may be cheaper than running the complete pipeline again.

Redis also supports persistence and several data structures, but a cache still needs an invalidation policy. Travel information makes this obvious: an answer about an event, temporary closure, or opening time can become wrong even when the cached text is internally consistent.

Redis documentation
describes it as more than a simple key/value cache, but its role in a RAG architecture should still be chosen explicitly.

OpenSearch: words, fields, and filters still matter

Embedding search is not a replacement for lexical search.

Users search for exact place names, train lines, product codes, legal terms, error messages, and quoted phrases. A model may produce similar embeddings for related text, but an exact identifier often deserves a lexical match.

OpenSearch can combine:

full-text queries;
BM25-style lexical relevance;
field filters;
aggregations;
geographic queries;
vector search;
search-oriented dashboards and operational tooling.

For the travel assistant, it could answer questions such as:

Find pages containing "Kiyomizu-dera" in English,
published in the Kyoto corpus, with an opening-hours field updated this month.

That query contains exact text, metadata, and freshness requirements. Those signals should not be reduced to one vector distance.

The OpenSearch documentation
covers both lexical and vector-oriented search, making it a possible single-engine choice when those retrieval modes need to work together.

Qdrant and vector databases: semantic similarity

Vector databases are designed around a different question:

Which stored items are closest to this query in embedding space?

That is valuable when the user and document do not share the same words.

For example, a visitor might ask:

Where can I spend a quiet indoor afternoon learning about local history?

A useful museum description may not contain that exact sentence. Semantic retrieval can still bring it into the candidate set.

Qdrant stores vectors with payload metadata and supports filtered similarity search. A typical request first restricts the eligible points, then ranks the remaining points by vector distance.

metadata filter
  -> approximate nearest-neighbor search
  -> top candidates
  -> optional reranker
  -> LLM context

This is an effective pattern, but it leaves an architectural question: how does the application decide the right filter and search namespace before similarity ranking begins?

Vector database similarity search algorithms, index type, filtering behavior, memory use, and update patterns all affect the result. A RAG vector DB comparison among Pinecone, Qdrant, pgvector, or OpenSearch should use the application's own corpus and filters. Lists of recommended vector databases cannot replace workload-specific evaluation.

RAG hybrid search: BM25, vectors, and reranking

Keyword and semantic retrieval are complementary. A RAG hybrid search pipeline can combine BM25 and vector scores so that exact names and identifiers remain visible while semantically related documents can still be discovered.

metadata and permission filters
  -> BM25 lexical candidates
  + vector similarity candidates
  -> merge and deduplicate
  -> reranker
  -> token-bounded context

For RAG accuracy improvement, rerank integration is often more useful than simply increasing the first-stage result count. A reranker can inspect a small candidate set more carefully, while the first stage remains optimized for recall.

RAG chunk size optimization techniques matter too. Small chunks can improve matching precision but lose surrounding meaning; large chunks preserve context but consume more tokens and may dilute relevance. The right chunk size depends on document structure, retrieval method, and the unit that must be cited.

RAG frameworks such as LangChain and LlamaIndex can connect loaders, embeddings, retrievers, and models. A LangChain RAG setup sample can demonstrate the wiring, but a framework cannot decide the correct authorization boundary, freshness rule, or candidate scope for an application.

An open-source RAG setup for internal search can combine PostgreSQL, OpenSearch or Qdrant, a local model, and an orchestration framework. Self-hosting changes the operations and privacy boundary, but it does not remove the need for the same retrieval and evaluation decisions.

RAG accuracy improvement and evaluation metrics

Retrieval quality and answer quality should be measured separately. Useful RAG evaluation metrics include:

recall and precision at k;
mean reciprocal rank or nDCG;
context relevance and context precision;
groundedness or faithfulness;
citation correctness;
answer completeness;
retrieved tokens, latency, and cost per request.

Tools such as Ragas can help automate parts of evaluation, but no RAG evaluation tool or tutorial can define the correct expected answer for every business workflow. A small, reviewed test set from real questions is still valuable.

RAG hallucination prevention measures should include more than adding extra documents. Stronger measures include current sources, explicit citations, permission checks, conflict detection, an option to abstain, and tests for questions whose answers are not present in the knowledge base.

RAG knowledge base automated updates also need versioning and invalidation. A pipeline should know which source revision produced a chunk, when an embedding must be rebuilt, and how deleted or expired information leaves caches and indexes.

RAG security measures and implementation cost

RAG security measures begin before retrieval. Tenant isolation, source authorization, secret removal, encryption, audit trails, and defenses against prompt injection in retrieved documents all belong in the design. General database security measures still apply even when the database is described as an AI component.

Organizations may also need to align collection and use of data with privacy rules, sector obligations, AI legal regulations, and their own AI ethics guidelines. Retrieval makes data available to a model; it does not create permission to use that data.

A complete RAG cost comparison includes ingestion, embedding generation, database storage, cache, network transfer, reranking, LLM input tokens, monitoring, backup, and engineering time. For a small business, the average AI implementation cost cannot be inferred from the model API price alone.

RAG cost reduction and token savings are closely related to candidate scope.

Passing fewer irrelevant records through retrieval and reranking can reduce database work and LLM input together. For local deployments, local LLM setup and PC requirements are a separate capacity decision, but smaller retrieved contexts can still reduce downstream memory and compute pressure.

Many AI implementation failures happen because a demo is evaluated only on a few successful questions. The practical AI implementation benefits appear when the system also handles stale data, missing answers, access boundaries, cost limits, and failure recovery.

The difficult part is often candidate scope

Consider this question:

I am at Kiyomizu-dera. It is raining, and I have two hours. What should I do
next?

Several retrieval methods can help:

lexical search can find pages that mention Kiyomizu-dera;
geographic search can find coordinates within a radius;
vector search can find semantically similar travel advice;
SQL can enforce opening hours, permissions, and reservation constraints;
a reranker can reorder the resulting candidates.

The challenge is not that any one of these methods is bad. The challenge is deciding what should enter the candidate set at all.

A useful answer may need:

nearby landmarks;
an indoor museum or gallery;
a restaurant that is currently open;
a temporary event happening today;
transit information;
accessibility information;
places already present in the user's itinerary.

The application knows the center: the visitor, Kiyomizu-dera, the current day, and the current situation. It does not yet know the exact boundary of the useful answer.

Semantic similarity is not operational relevance

The most semantically similar document is not necessarily the most useful document for the current request.

A beautifully written article about another Japanese temple may be close in embedding space but unusable within two hours. A short transit notice may be semantically distant from a tourism question but essential to the answer. A restaurant record may contain almost none of the words in the user's request, yet be relevant because it is nearby, open, and compatible with the itinerary.

RAG systems commonly address this with metadata filters:

{
  "city": "Kyoto",
  "area": "Higashiyama",
  "openNow": true,
  "weather": "rain",
  "categories": ["restaurant", "museum", "landmark"]
}

This is reasonable. But as the product grows, the filter becomes an
application-level retrieval plan. Someone must maintain the rules, namespaces, joins, and fan-out reads that reconstruct the context for every request.

Geographic distance is not the complete boundary either

If the question is simply "what is within one kilometer?", a geographic index such as PostGIS or OpenSearch geo search is the direct solution.

Useful travel context is not always a circle, however.

A river, hill, or railway can make two close points inconvenient to reach.
Rain changes which facilities are useful.
A family and a solo traveler may need different neighborhoods.
Opening hours and event dates change the available context.
Editorial groupings such as "rainy-day Kyoto" are not geographic regions.
One place can belong to an area guide, a station guide, and a personal itinerary at the same time.

Distance remains an important signal. It is simply not the only relationship that defines the field of view.

The application keeps rebuilding context

A conventional stack can solve this problem. It may use SQL joins, geographic indexes, search filters, a graph, application-maintained ID lists, and vector ranking.

The repeated work is reconstructing the same locality:

start from the current place
  -> identify the area
  -> find allowed categories
  -> join today's events
  -> add relevant transit
  -> add itinerary items
  -> build candidates
  -> rank candidates

The missing operation here is not another scoring algorithm. It is a reusable way to answer this earlier question:

What belongs in the neighborhood of this known center for this kind of
request?

Treat the reusable neighborhood as data

One design option is to store that neighborhood instead of reconstructing it from scratch for every request.

This does not automatically require a new database. An application can model the relationship with SQL junction tables, graph edges, materialized views, search documents, or maintained ID lists. The important change is conceptual:
candidate scope becomes data with its own lifecycle, rather than temporary query logic assembled immediately before ranking.

For the tourism example, a reusable neighborhood needs a few properties:

each place should keep one canonical record;
the same place should be able to appear in several contexts;
adding a place to a guide should not copy its payload;
reads should be bounded by category, depth, filters, limits, and token cost;
lexical or vector ranking should remain available after the neighborhood is selected.

If this relationship is occasional, application code may be enough. If the same pattern appears across users, places, projects, products, or time windows, it can be useful to make locality a first-class retrieval primitive.

How KoutenDB represents this retrieval pattern

KoutenDB is one database designed
around that pattern. It uses coordinate-like ring placement for canonical records and stellar visibility lenses for reusable views across those coordinates.

An importer or application can keep tourism data in canonical rings:

landmarks/kyoto/kiyomizudera
landmarks/kyoto/yasaka-pagoda
restaurants/kyoto/gion
culture/kyoto/national-museum
events/kyoto/2026-07-23
transit/kyoto/higashiyama

It can then attach useful coordinates to an area-centered view:

kouten stellar attach \
  --stellar=travel/kyoto/higashiyama \
  --ring=landmarks/kyoto/kiyomizudera

kouten stellar attach \
  --stellar=travel/kyoto/higashiyama \
  --ring=restaurants/kyoto/gion

kouten stellar attach \
  --stellar=travel/kyoto/higashiyama \
  --ring=culture/kyoto/national-museum

kouten stellar attach \
  --stellar=travel/kyoto/higashiyama \
  --ring=events/kyoto/2026-07-23

The application can read the surrounding context from the known center:

kouten get --stellar=travel/kyoto/higashiyama

Or narrow the same field of view to one category:

kouten get --stellar=travel/kyoto/higashiyama --subring=restaurants
kouten get --stellar=travel/kyoto/higashiyama --subring=culture

The same restaurant can also be visible from a rainy-day guide, a
station-centered guide, or a user's itinerary without copying its canonical payload.

KoutenDB does not infer geographic or editorial relevance from a path name.

The application, importer, or curation process still decides which coordinates belong in each neighborhood. The database preserves that decision so later reads do not have to reconstruct it from scratch.

This is more than a path prefix

A ring name deliberately looks like a readable path. Hierarchy is useful when parent and child data naturally belong together.

A path prefix alone, however, represents one tree. Real records participate in several views at once:

canonical catalog: restaurants/kyoto/gion/example-cafe
area guide:        travel/kyoto/higashiyama
weather guide:     travel/kyoto/rainy-day
personal plan:     travel/users/123/today

Copying the restaurant under every path creates synchronization work. Moving it to one path weakens the other views. A stellar lens instead stores a visibility relationship across existing coordinates.

The caller supplies a center and a cost boundary. Depth, branch budget, subrings, filters, projections, sorting, and limits control how much context is returned. The caller does not have to enumerate every individual record in the answer.

Locality can precede vector ranking

Locality-aware retrieval and vector search are not mutually exclusive.

A combined pipeline can look like this:

known center: user + place + time
  -> retrieve the configured local neighborhood
  -> apply current constraints
  -> vector-rank the smaller candidate set if needed
  -> rerank and enforce a token budget
  -> LLM

The vector database asks which candidates are semantically close. The locality layer asks which part of the data should be considered first.

This can reduce the number of payloads loaded, vectors compared, records reranked, bytes transferred, and tokens considered downstream. It does not guarantee that every query becomes faster. The model is useful only when the application can express meaningful locality.

Match the database to the retrieval question

The database decision becomes clearer when it starts from the question rather than the current RAG trend.

Retrieval question	Natural starting point
Which exact record or relationship is valid?	Relational query and indexes
Which JSON documents match known fields?	Document database or SQL/JSON
Which result can be reused briefly?	Cache or session store
Which documents contain these words?	Full-text search engine
Which documents are semantically similar?	Vector search
Which places are inside a literal radius?	Geographic index
Which context is useful around this known user, place, project, or time?	Locality-aware neighborhood retrieval

These are not exclusive choices. PostgreSQL may remain the source of truth while OpenSearch handles lexical retrieval. A vector database may rank semantically similar chunks. KoutenDB may be used as a locality-aware document and retrieval store, or as an earlier candidate-selection stage.

A smaller system may choose one database that covers enough of these roles. A larger system may separate them. Additional infrastructure is justified only when it removes a measured retrieval, correctness, or operations problem.

Practical database design steps for RAG are therefore:

identify the authoritative source and update owner;
define tenant, user, and document access boundaries;
list exact, lexical, semantic, geographic, and locality-aware queries;
choose the smallest set of database types that covers those queries;
measure retrieval quality, tokens, latency, and failure cases;
document database backup methods and database migration steps for the data that cannot be rebuilt;
use database performance tuning only after the candidate and access model are correct.

The final question is what the application already knows

RAG architecture often begins by asking which embedding model or vector database to use.

Another useful question comes first:

What does the application already know before retrieval begins?

It may already know the authenticated user, tenant, order, project, document group, current place, time window, or active task. That information can define a much smaller and more relevant starting region than the complete corpus.

If the request begins with no meaningful center, global lexical or vector search may be exactly right. If the request begins with a known center but an uncertain context boundary, preserving locality can keep the retrieval problem smaller before ranking and prompt construction begin.

The important choice is not SQL versus vectors, or one database product versus another. It is deciding whether the application needs exact truth, lexical matching, semantic similarity, geographic proximity, or a reusable neighborhood of context—and then giving each retrieval stage the data model it actually needs.

That is the foundation of a RAG database design that can grow beyond a demo.

Why KoutenDB Exists: You Know the Center, Not the Boundary

puffball1567 — Wed, 22 Jul 2026 08:04:06 +0000

Most database queries begin by describing the answer.

You provide a primary key, write a SQL predicate, select a collection, enter
search terms, or send an embedding to a vector database. The database then
finds the records that match.

That model works extremely well, but it is not the shape of every application
request.

Sometimes the application knows where the answer should begin, while the
complete boundary of useful information is still unclear.

For example:

a support request starts with a customer, but may need their profile, recent orders, the product involved, previous tickets, and a refund policy.
an incident investigation starts with a service and a point in time, but may need a nearby deployment, error logs, metrics, and related service events.
a RAG request starts with a project or topic, but the useful context may span documents, summaries, source metadata, and attached knowledge.
a game request starts with a player or place, but may need nearby entities, inventory, quests, and regional state.

In each case, the center is known. The exact context boundary is not.

That is the problem KoutenDB was created to explore.

Exact lookup is not the same as context retrieval

Suppose an API receives an order ID.

Finding the order itself is easy. Almost any database can do that efficiently.
The harder question is what else should travel with it.

Should the response include the customer? The shop? Recent payment events? The specific product documentation? An unresolved support ticket? The answer may depend on the request, product, current state, and amount of context the caller can afford to process.

A conventional application can assemble that context with:

SQL joins;
several document lookups;
application-maintained ID lists;
a graph traversal;
metadata filters;
full-text or vector search;
a cache built for one response shape.

These are all valid solutions. But they usually require the application to reconstruct the neighborhood every time it reads.

KoutenDB asks a different question:

What if the database preserved useful locality when the data was written, so
a later read could begin from that locality directly?

A ring is a coordinate, not just a container

KoutenDB stores a record with a coordinate-like ring.

users/123
users/123/orders
users/123/support
shops/1123
orders/A-001
products/sku-9/docs

The names are readable, but readability is not their main purpose. A ring is part of data placement and part of the read path.

If the application starts from users/123, KoutenDB can inspect that local hierarchy without beginning with an unrelated global collection. If it starts from orders/A-001, it can use an order-centered view instead.

This turns locality known by the application into retrieval information the database can retain.

The same data can be visible from another center

Related context does not always fit one permanent hierarchy. An order belongs to a customer, a shop, and a product at the same time.

KoutenDB therefore has a stellar lens. It can attach existing rings to a useful center without copying their payloads.

kouten put --ring=users/123 \
  --payload='{"kind":"user","name":"Alice"}' --codec=json

kouten put --ring=shops/1123 \
  --payload='{"kind":"shop","name":"Orbit Store"}' --codec=json

kouten put --ring=orders/A-001 \
  --payload='{"kind":"order","orderNo":"A-001","total":42}' --codec=json

kouten stellar attach \
  --stellar=commerce/order/A-001 --ring=users/123

kouten stellar attach \
  --stellar=commerce/order/A-001 --ring=shops/1123

kouten stellar attach \
  --stellar=commerce/order/A-001 --ring=orders/A-001

Now the application can read from the order as the center:

kouten get --stellar=commerce/order/A-001

The original records remain in their original rings. Attaching or detaching a ring changes what the lens can see; it does not duplicate or delete the payload.

This is useful when the same information needs to participate in several context views without turning each view into another copied document.

Example: retrieving useful information around a tourist destination

Travel information makes the center-versus-boundary problem easier to see.

Imagine that a visitor is looking at Kiyomizu-dera in Kyoto. The application knows the starting point, but a useful response probably should not contain only the temple record.

The visitor may also need:

other nearby landmarks;
restaurants and cafes;
museums, galleries, and cultural facilities;
events taking place that day;
transit information;
accessibility or opening-hour information.

There is no single permanent boundary for that context. A family, a solo traveler, and someone planning a rainy afternoon may need different parts of the same surrounding information.

An importer or application can keep each place in a canonical ring:

landmarks/kyoto/kiyomizudera
landmarks/kyoto/yasaka-pagoda
restaurants/kyoto/gion
culture/kyoto/national-museum
events/kyoto/2026-07-22
transit/kyoto/higashiyama

It can then attach the useful coordinates to an area-centered stellar lens:

kouten stellar attach \
  --stellar=travel/kyoto/higashiyama \
  --ring=landmarks/kyoto/kiyomizudera

kouten stellar attach \
  --stellar=travel/kyoto/higashiyama \
  --ring=landmarks/kyoto/yasaka-pagoda

kouten stellar attach \
  --stellar=travel/kyoto/higashiyama \
  --ring=restaurants/kyoto/gion

kouten stellar attach \
  --stellar=travel/kyoto/higashiyama \
  --ring=culture/kyoto/national-museum

kouten stellar attach \
  --stellar=travel/kyoto/higashiyama \
  --ring=events/kyoto/2026-07-22

The application can now begin with the Higashiyama area and retrieve the attached tourism context in one read:

kouten get --stellar=travel/kyoto/higashiyama

Or it can narrow the same view to restaurants or cultural facilities:

kouten get --stellar=travel/kyoto/higashiyama --subring=restaurants
kouten get --stellar=travel/kyoto/higashiyama --subring=culture

The same restaurant can also be attached to a travel/kyoto/rainy-day lens, a station-centered lens, or a day-plan lens without moving or copying its canonical payload.

KoutenDB does not automatically infer geographic relevance from these names.
The application, importer, or curation process still decides which coordinates belong in the neighborhood. The database's role is to preserve that decision and make it directly readable later.

This is the intended meaning of surrounding information: the caller starts from a useful place, receives several categories of nearby context together, and can narrow the field of view without having to list every individual place in the request.

Is this just a path?

A ring such as landmarks/kyoto/kiyomizudera certainly looks like an ordinary path. KoutenDB deliberately uses readable hierarchical names because paths are a useful way to express parent and child locality.

But a path by itself provides only hierarchy.

If all tourism data lived under one tree, an application could read a prefix
such as travel/kyoto/higashiyama/*. That works when every useful relationship fits the same permanent hierarchy and the caller already knows which branches to request.

The difficulty is that real data belongs to several contexts at once:

a restaurant belongs to its canonical restaurant catalog;
geographically, it belongs to an area;
operationally, it may belong to a currently open list;
editorially, it may belong to a rainy-day guide;
personally, it may belong to one visitor's itinerary.

Duplicating the same record under every path creates synchronization and consistency work. Moving it into one path makes the other views harder to express. Storing only path strings leaves the application responsible for maintaining and joining all of those relationships.

KoutenDB adds two things beyond the path syntax:

the ring hierarchy is understood by the retrieval engine and can be traversed with explicit depth, branch, filter, limit, sort, and projection controls.
stellar metadata can connect rings from different hierarchies into another coordinate-centered view without changing their canonical paths or copying their payloads.

The stellar lens is therefore not another directory. It is a stored visibility relationship across existing coordinates.

The same ring coordinate can also act as a boundary for authorization, dump/import, placement, and synchronization. A plain path string has none of those database semantics unless the application builds them separately.

So the path is the readable address. The KoutenDB-specific part is how that address participates in placement, neighborhood retrieval, and reusable views across otherwise separate branches.

Not knowing the boundary does not mean an unbounded query

"The caller does not know the exact boundary" should not mean "read
everything."

The caller still controls the field of view. KoutenDB can narrow a read with:

a subring;
hierarchy depth;
branch budget;
per-ring limits and sorting;
filters;
field selection;
an overall result limit.

For example, the same order-centered view can request only shop data or only a small projection:

kouten get --stellar=commerce/order/A-001 --subring=shops

kouten get --stellar=commerce/order/A-001 \
  --selection='{ kind name orderNo total }'

The distinction is important:

The application supplies a meaningful center and a cost boundary. It does
not have to enumerate every record relationship that may form the answer.

This is why KoutenDB is intended for surrounding information rather than only exactly bounded queries.

Why not use a collection or partition key?

Collections, namespaces, partitions, and shards can already reduce the amount of data a database examines.

The difference is what they represent.

A partition key commonly answers questions such as:

Where is this record stored?
Which node owns it?
Which partition can be pruned?

KoutenDB also wants locality to answer:

What information is likely to be useful around this record?
Which nearby rings should enter the current field of view?
Can the same existing ring become visible from another useful center?

That makes placement an application-visible retrieval concept, not only an internal scaling decision.

Why not begin with vector search?

Vector search answers a valuable question:

Which items are semantically similar to this query?

But similarity is not the same as operational relevance.

A customer's latest order, a permission record, and the applicable refund policy may be necessary context even when their embeddings are not the closest matches in a global corpus. Systems often compensate by adding namespaces and metadata filters before vector ranking.

KoutenDB makes that first narrowing step the main data model:

begin from a known user, order, tenant, topic, place, or time coordinate;
open the useful local neighborhood;
apply filters, vector ranking, reranking, or an LLM only to that smaller working set when needed.

Vectors remain useful. They are optional rather than the only way to discover related context.

For local AI and RAG systems, that distinction can matter. The expensive work is not limited to the final LLM call. Loading candidates, comparing vectors, reranking chunks, transferring payloads, and filling a context window all have a cost. A smaller first-stage working set can reduce work throughout the pipeline.

Why not keep this entirely in application code?

An application can maintain arrays of related IDs, write join tables, perform fan-out reads, and build cached response documents. Many applications already do.

The tradeoff is that locality becomes scattered across code and supporting infrastructure. One rule determines reads, another determines authorization, another controls exports, and another decides how data should be copied or synchronized.

KoutenDB's larger design goal is to let the same coordinate participate in:

retrieval scope;
authorization scope;
dump and import boundaries;
placement and compaction;
delayed synchronization between topologies.

The value is not just fewer lines in one query. It is keeping the application's concept of locality visible across the data lifecycle.

The database cost should follow the working set

Let the complete corpus contain N records, while the useful neighborhood for one request contains k records.

Many systems eventually introduce a first-stage index or filter so later work does not operate over all N. KoutenDB's central bet is that meaningful placement can provide part of that first stage directly.

The goal is to make more of the request cost follow k, the useful local working set, instead of repeatedly rediscovering locality from the full corpus.

This is not an automatic performance guarantee. It depends on whether the application can express useful locality. If all queries are arbitrary and global, there may be no meaningful center to exploit.

But many applications already begin with one:

an authenticated user;
a tenant;
an order;
a product;
a project;
a document group;
a region;
a service and time window;
an entity in a simulation.

KoutenDB makes that starting knowledge part of the database query model.

When this model is useful

KoutenDB is worth examining when all three of these statements are true:

requests naturally begin from a known coordinate;
the application frequently needs surrounding context, not only one exact record;
the useful context can be placed or attached meaningfully before it is requested.

It is less relevant when the main workload is arbitrary full-corpus analytics, global full-text search, independent lookup on every field, or strict relational constraints. Those problems already have databases built around them.

The reason for another database is not that existing databases forgot how to look up records.

It is that a growing class of application and AI requests does not begin with a complete description of the answer. It begins with a center.

KoutenDB exists to make that center useful.

The implementation, data-model demo, and reproducible benchmarks are available in the KoutenDB repository.

9 Databases Compared: PostgreSQL, MongoDB, Redis, and More

puffball1567 — Tue, 21 Jul 2026 16:22:06 +0000

Choosing a database can feel harder than it needs to be.

This beginner-friendly database comparison covers PostgreSQL, MySQL, SQLite, MongoDB, Redis, ClickHouse, OpenSearch, Qdrant, and KoutenDB. The goal is to help you decide which database fits your application—not to declare one winner.

Should you use PostgreSQL or MySQL? Is MongoDB easier for JSON? Is Redis only a cache? Do you need a vector database for RAG? And where do newer database ideas fit?

There is no single "best database." A better question is:

What kind of work does this database make simple?

This guide introduces nine databases without trying to rank them. Each one has a different center of gravity, and many real systems use more than one.

Quick Comparison

Database	Model	Strongest fit
PostgreSQL	Relational SQL	General-purpose applications and complex data relationships
MySQL	Relational SQL	Mainstream web applications and widely supported hosting stacks
SQLite	Embedded relational SQL	Local applications and simple single-file storage
MongoDB	Document database	JSON-like documents and evolving data shapes
Redis	Data structure store	Caching, sessions, counters, streams, and low-latency state
ClickHouse	Column-oriented SQL / OLAP	Analytics over large event and log datasets
OpenSearch	Search and analytics engine	Full-text search, log exploration, and observability
Qdrant	Vector database	Semantic similarity search, recommendations, and RAG
KoutenDB	Coordinate-local document/vector database	Related context around a useful starting point

1. PostgreSQL: A Strong General-Purpose Default

Best at: relational data, transactions, data integrity, and complex SQL.

PostgreSQL is a mature object-relational database with ACID transactions, foreign keys, many index types, JSON support, full-text search, replication, and a large extension ecosystem. Its ability to model relationships and enforce rules inside the database makes it a strong starting point for many
applications.

Good fit for:

SaaS products;
business systems;
financial and inventory data;
applications with joins and reporting;
teams that want one capable general-purpose database.

Keep in mind: A specialized cache, search engine, vector database, or OLAP system may still be useful beside PostgreSQL when one workload becomes large or specialized.

PostgreSQL overview

2. MySQL: A Familiar Choice for Web Applications

Best at: conventional relational workloads with broad hosting, tooling, and framework support.

MySQL is a relational SQL database with a long history in web development. It is widely supported by hosting providers, CMS platforms, application frameworks, and language libraries. That familiarity can reduce adoption and operations work for a small team.

Good fit for:

web applications;
ecommerce and CMS systems;
APIs built on an existing MySQL stack;
teams with MySQL operational experience.

Keep in mind: The PostgreSQL-versus-MySQL decision is rarely solved by a single feature checklist. Existing team knowledge, hosting, extensions, and application requirements often matter more.

MySQL overview

3. SQLite: A Database Without a Server

Best at: embedded, local, and single-file SQL storage.

SQLite runs inside the application process. It has no separate database server to deploy, and a complete database can live in one portable file. Despite its small footprint, it supports transactions, tables, indexes, triggers, and views.

Good fit for:

mobile and desktop applications;
command-line tools;
local-first software;
tests and prototypes;
small services with a simple deployment model.

Keep in mind: If many machines or remote clients need to write concurrently, a client/server database may be a better operational fit.

SQLite overview

4. MongoDB: Documents That Look Like Application Data

Best at: storing JSON-like documents with nested objects and arrays.

MongoDB stores records as BSON documents. This often maps naturally to objects already used by web applications and APIs. Related fields can be embedded in one document, while collections and indexes provide larger-scale organization and access paths.

Good fit for:

content and catalog data;
applications with nested documents;
rapidly evolving data shapes;
systems where a document is the natural unit of access.

Keep in mind: A flexible document model does not remove the need for data modeling. Relationships, document growth, indexes, and consistency boundaries still need deliberate design.

MongoDB document model

5. Redis: Fast Access to Useful Data Structures

Best at: low-latency access to keys, counters, sets, lists, streams, and other data structures.

Redis is often introduced as a cache, but it can also be used as a database, streaming engine, and message broker. Its commands make common operations such as incrementing a counter, expiring a session, maintaining a ranking, or publishing an event straightforward.

Good fit for:

caching;
sessions and rate limits;
counters and leaderboards;
queues, streams, and pub/sub;
temporary or frequently accessed state.

Keep in mind: Decide explicitly how persistence, recovery, memory use, and eviction should work before treating Redis as the only source of truth.

Redis overview

6. ClickHouse: Analytics Across Large Datasets

Best at: fast analytical queries over large numbers of rows.

ClickHouse is a column-oriented SQL database designed for online analytical processing (OLAP). Column-oriented storage is well suited to queries that scan a few fields across many records and calculate aggregates such as counts, totals, percentiles, or time-series summaries.

Good fit for:

product and business analytics;
logs, events, and traces;
large reporting datasets;
real-time analytical dashboards.

Keep in mind: ClickHouse is centered on analytical workloads. A traditional transactional database is usually easier for ordinary row-by-row application updates and relationship-heavy OLTP flows.

ClickHouse overview

7. OpenSearch: Search, Logs, and Observability

Best at: full-text search and exploration across indexed data.

OpenSearch provides a search and analytics engine together with APIs and dashboards. It supports text search, Query DSL, index management, aggregations, vector search, and observability-oriented workflows.

Good fit for:

site and product search;
log search;
application and infrastructure observability;
relevance-based text retrieval;
dashboards over indexed events.

Keep in mind: OpenSearch commonly works beside a primary application database. The indexing pipeline, cluster sizing, mappings, and lifecycle of duplicated search data become part of the system design.

OpenSearch documentation

8. Qdrant: Similarity Search for Vectors

Best at: finding items that are similar in embedding space.

Qdrant organizes data into collections of points. Each point can contain one or more vectors plus payload metadata. Its vector indexes and filtering support make it suitable for semantic retrieval where exact keywords are not enough.

Good fit for:

semantic search;
recommendations;
image, audio, or document similarity;
retrieval-augmented generation (RAG);
vector search combined with metadata filters.

Keep in mind: A vector database solves a specific retrieval problem. You still need to choose embedding models, evaluate relevance, design metadata, and plan memory and storage as collections grow.

Qdrant overview

9. KoutenDB: Related Data Around a Starting Point

Best at: retrieving a useful neighborhood of related data from a starting coordinate, even when the caller does not know the exact result boundary in advance.

KoutenDB is a ring-oriented NoSQL document/vector database written in Nim. A ring is not only a label: it is part of data placement and the read path.
Applications can place related data into nearby rings or attach existing rings to a stellar lens. A read can start from a user, order, product, topic, or other coordinate and bring nearby parent, child, sibling, or attached rings into the same field of view.

Good fit for:

surrounding context for a user, order, product, tenant, or topic;
related-data bundles for application APIs without enumerating every record;
knowledge retrieval where the useful neighborhood is known more clearly than its exact boundary;
reducing candidates before vector ranking or an LLM step;
experiments with locality-aware storage and retrieval.

What makes it different: KoutenDB does not require the caller to know the final boundary of the query. It needs a useful center. A stellar read follows the configured coordinate neighborhood, while depth, branch budget, subring, limit, and sort options control the field of view. Its central question is:
"What useful information lives around this point?"

KoutenDB repository

Which Database Should You Use?

If you are still unsure, start with the main job your database needs to do:

General relational application: PostgreSQL or MySQL
Embedded or local SQL: SQLite
JSON-like document storage: MongoDB
Cache and low-latency shared state: Redis
Large-scale analytics: ClickHouse
Full-text and log search: OpenSearch
Vector similarity search: Qdrant
Related context around a useful starting coordinate: KoutenDB

These choices are not mutually exclusive. A product might use PostgreSQL as its primary database, Redis for caching, OpenSearch for text search, and Qdrant for semantic retrieval. Another application may need only SQLite. A system whose data forms meaningful neighborhoods around users, orders, products, topics,
regions, or points in time may want to explore KoutenDB's locality model.

Frequently Asked Questions

What is the best database for a small SaaS application?

PostgreSQL and MySQL are both strong starting points for a conventional SaaS backend. They provide transactions, constraints, indexes, and mature operations without requiring several specialized databases on day one. Add another system only after a real cache, search, analytics, or retrieval bottleneck appears.

Which database should I use for a small or local project?

SQLite is often the simplest choice when the application runs locally and does not need a separate database server. For a server application used by multiple people or services, PostgreSQL or MySQL may provide an easier growth path.

Which database should I use for JSON data?

MongoDB is a natural candidate when the document is the main unit of storage and retrieval. PostgreSQL is also worth considering when JSON data must live beside relational data, transactions, and SQL queries. Choose from the access pattern, not only from the format of the incoming API payload.

Which database is best for caching and user sessions?

Redis is a common fit for caches, expiring sessions, counters, rate limits, and other short-lived shared state. A relational or document database can remain the durable source of truth while Redis serves the frequently accessed path.

Should I use ClickHouse or OpenSearch for logs?

Choose ClickHouse when SQL analytics and aggregation across large event sets are central. Choose OpenSearch when text search, relevance, interactive log exploration, and search-oriented dashboards are central. Some observability systems use both, but that additional complexity should solve a real need.

Which database should I use for RAG and vector search?

Qdrant is designed around vector similarity, payload filtering, and semantic retrieval. OpenSearch can combine text and vector search, while PostgreSQL can gain vector capabilities through extensions. KoutenDB addresses a different part of the problem: it can start from a useful coordinate, retrieve the nearby context, and reduce the candidate set before vector ranking or an LLM step. The caller does not need to know the exact final context boundary in advance.

Which database fits user-centered or multi-tenant related data?

PostgreSQL and MySQL commonly model tenants and users with keys, constraints, and indexes. KoutenDB is designed for reads that start from a coordinate such as tenant/acme or users/123, then bring nearby profile, orders, billing, or other attached information into view. The application supplies the center; it does not have to enumerate the complete result boundary first.

Does an application need more than one database?

Not necessarily. Starting with one general-purpose database keeps operations simple. Add Redis, OpenSearch, Qdrant, ClickHouse, KoutenDB, or another specialized system only when a clear workload justifies the additional moving parts.

Final Thought

Do not choose a database only because it wins a benchmark or appears in a popular architecture diagram. Ask a few practical questions first:

What does the application know before it starts a query?
Are relationships, documents, vectors, text, or aggregates the main shape?
Which consistency and durability guarantees are required?
How much operational complexity can the team support?
How will the data be exported or migrated later?

The best database is usually the one that makes your important access patterns simple, predictable, and maintainable.

KoutenDB v0.8.0: A Full Public-Surface Migration

puffball1567 — Mon, 20 Jul 2026 09:04:18 +0000

I released KoutenDB v0.8.0.

Release:

https://github.com/puffball1567/koutendb/releases/tag/v0.8.0

This release is different from v0.7.0. The previous release focused on storage,
transport, recovery, authorization, and operational hardening. v0.8.0 carries
that work forward under the project's new active name: KoutenDB.

This is primarily a naming and migration release, not a new storage-engine
feature release. However, the migration reaches across the complete public
surface, so it is more than changing the title in the README.

The v0.7.0 to v0.8.0 Diff

The Git diff between the two release tags reports:

188 changed files
3,970 insertions
4,203 deletions

Most of that volume is rename churn. Source modules, tests, examples, scripts,
documentation, CI, headers, and driver foundations all had to move together.

The high-level difference is:

Area	v0.7.0	v0.8.0
Project	RocheDB	KoutenDB
Nim package/module	`rochedb`	`koutendb`
CLI	`roche`	`kouten`
Daemon	`roched`	`koutend`
C ABI library	`librochedb.so`	`libkoutendb.so`
C header	`include/rochedb.h`	`include/koutendb.h`
C symbol prefix	`roche_*`	`kouten_*`
Repository	`puffball1567/rochedb`	`puffball1567/koutendb`

Configuration names and build variables were migrated to the KOUTEN_* and
KOUTENDB_* families as well.

Why KoutenDB?

Kouten comes from the Japanese word kouten (公転), meaning orbital
revolution: one body moving around another.

That meaning matches the database model. KoutenDB treats rings,
orbit-inspired placement, and locality as part of retrieval. When an
application has a meaningful locality boundary, the database can use it to
reduce unrelated reads, transferred bytes, candidate memory, and downstream
AI/RAG or application work.

The name changed, but this technical direction did not.

The Migration Reaches the Whole Public Surface

For Nim users, new code should import koutendb:

import koutendb

var db = koutendb.open(dataDir = "data")
let id = db.put("""{"title":"Hello"}""", ring = "docs/japan")
echo db.get(id)

Command-line examples now use kouten:

kouten put --ring=docs/japan --payload='{"title":"Hello"}'
kouten get --ring=docs/japan

Server processes now use koutend:

koutend --id=0 --peers=h1:7301,h2:7301,h3:7301 --data=/var/lib/kouten

The C ABI migration is intentionally explicit. Consumers should include
include/koutendb.h, link against libkoutendb.so, and call the kouten_*
symbols. This keeps the public native boundary internally consistent, but it
also means existing C ABI consumers need to update and rebuild.

Drivers and Package Names

The driver-facing names were migrated too. Published packages now use the
KoutenDB name for Rust, JavaScript/TypeScript, PHP, Python, and C++ distribution
paths, while the in-repository C#, Go, Kotlin, Node, PHP, Swift, and C++
foundations were updated to the same naming scheme.

One structural cleanup is also part of the v0.7.0 to v0.8.0 diff: the old
in-repository Python driver was removed from the core repository. Python support
continues as a separately maintained and published native TCP driver:

https://github.com/puffball1567/koutendb-python

This avoids keeping two copies of the Python implementation in different
release flows.

What Did Not Change?

KoutenDB remains a technical preview and research OSS project. Its core thesis
is still:

Use meaningful placement to reduce the amount of data that must be read,
transferred, held in memory, and passed to downstream systems.

The ring-oriented NoSQL document/vector model, locality-aware retrieval goal,
and smaller-working-set focus continue from v0.7.0. v0.8.0 does not claim a new
benchmark result or a new storage-engine capability merely because the code
diff is large.

Migration Quickstart

For a new Nim installation:

nimble install koutendb
kouten --help

To build from source:

git clone https://github.com/puffball1567/koutendb.git
cd koutendb
nimble check
scripts/test_all_smoke.sh

Existing integrations should check these areas:

Nim imports and package dependencies
CLI and daemon command names
environment and configuration variable names
C headers, shared-library names, and symbol prefixes
driver package and repository URLs
documentation and CI scripts that contain the previous names

Verification

The v0.8.0 rename was checked across the core Nim modules, CLI, daemon, C ABI,
core tests, CLI CRUD smoke tests, TLS-enabled C ABI tests, driver compatibility,
and the complete smoke suite. The rename PR also passed the Linux and macOS C
ABI jobs in GitHub Actions.

KoutenDB v0.8.0 is available here:

https://github.com/puffball1567/koutendb

RocheDB Is Now KoutenDB

puffball1567 — Mon, 20 Jul 2026 08:51:32 +0000

RocheDB has been renamed to KoutenDB.

This is the main change in KoutenDB v0.8.0, a naming and migration release.
The technical direction is unchanged: KoutenDB remains a ring-oriented NoSQL
document and vector store focused on locality-aware retrieval and smaller
working sets.

Release:

https://github.com/puffball1567/koutendb/releases/tag/v0.8.0

Why KoutenDB?

Kouten comes from the Japanese word kouten (公転), meaning orbital
revolution: one body moving around another.

The name fits the database model more directly. KoutenDB uses rings and
orbit-inspired placement as part of the retrieval path. Its goal is to use
meaningful locality to reduce unrelated reads, transferred bytes, candidate
memory, and downstream AI/RAG or application work.

What Changed?

Project name: KoutenDB
Nim package: koutendb
CLI: kouten
Daemon: koutend
C ABI library: libkoutendb.so
C header: include/koutendb.h
C ABI symbol prefix: kouten_*
Repository: https://github.com/puffball1567/koutendb

The documentation and examples have also been updated to use the KoutenDB
naming scheme.

Migrating

For new Nim installations:

nimble install koutendb
kouten --help

To work from source:

git clone https://github.com/puffball1567/koutendb.git
cd koutendb
nimble check
scripts/test_all_smoke.sh

C ABI users should now include include/koutendb.h, link against
libkoutendb.so, and use the kouten_* symbol family.

Older names may remain visible temporarily in historical posts and archived
package entries, but KoutenDB is now the active project, package, CLI, and
repository name.

Thank you to everyone who has followed the project so far. Development will
continue at:

https://github.com/puffball1567/koutendb

KoutenDB: Data Locality as a First-Stage Retrieval Index

puffball1567 — Sat, 18 Jul 2026 04:20:32 +0000

One of the ideas behind KoutenDB is easy to miss if we describe it only as a
NoSQL database, a document store, or a vector-aware storage engine for AI and
search workloads.

The stronger idea is this:

Data locality and data placement can become the first stage of retrieval.

In many databases and search systems, storage and indexing are treated as
separate layers. Data is stored somewhere, then database indexes, inverted
indexes, or vector indexes are built on top of it so that queries can find it
later.

KoutenDB tries to push part of that work into placement.

If related data is written into nearby rings, stellar neighborhoods, or
application-defined locality groups, then the first retrieval question changes
from:

Which global index should I search?

to:

Which local region of the data should I open first?

That difference matters for search infrastructure, recommendation systems, RAG
systems, LLM context stores, vector search pipelines, and large
retrieval-heavy applications.

For teams operating large-scale retrieval systems, this is close to several
daily engineering problems: candidate generation, candidate pruning, I/O
amplification, memory pressure, cache locality, and tail latency before the
ranking pipeline even starts.

Ranking is not the first problem

Search systems often discuss ranking algorithms and retrieval techniques:
PageRank, BM25, vector similarity, freshness, authority signals,
personalization, semantic search, reranking, and many other scoring methods.

Those are important.

But before ranking, a search or RAG pipeline has to decide what data even
deserves to be examined.

A simplified search, recommendation, or RAG retrieval pipeline looks like this:

Decide which data region to inspect.
Generate and prune candidate records.
Score or rank those candidates.
Rerank, summarize, or pass selected context downstream.

KoutenDB focuses on the first two stages.

It does not try to replace ranking algorithms. Instead, it tries to reduce how
much unrelated data those algorithms, vector search systems, or LLM rerankers
need to see.

If the candidate set is already narrow and relevant, later ranking work becomes
lighter. In a multi-stage ranking system, reducing weak candidates before the
first ranking stage can also reduce downstream reranking cost and tail latency.

Data placement as an index-like structure

Traditional database and search indexes are usually separate structures:

a B-tree;
a secondary index;
an inverted index for full-text search;
an ANN index for vector search;
a materialized lookup table.

Those structures are useful, and KoutenDB is not arguing that they should
disappear.

But KoutenDB adds another retrieval layer:

The physical and logical location of the data is also retrieval information.

A ring is not just a collection label. A ring says where the data belongs and
where future search, RAG, or application reads should probably begin.

For example:

web/github/nim/koutendb/page
web/github/nim/koutendb/keywords
web/github/nim/koutendb/summary
web/github/nim/koutendb/links
web/github/nim/koutendb/embedding

These records are not merely stored under similar names. They are intentionally
near each other because a search or RAG request about KoutenDB, Nim, or the
project's documentation may need them together.

That means the storage layout is already doing part of the candidate-selection,
candidate-pruning, and working-set reduction work.

Why this helps search engines and retrieval systems

Imagine a search engine, recommendation system, or AI search system that has
already collected pages, links, summaries, keywords, embeddings, entity hints,
language metadata, region metadata, and freshness information.

One approach is to store those pieces separately and repeatedly join or look up
the metadata needed for ranking, semantic search, or retrieval augmentation.

Another approach is to keep related pieces close enough that the system can
open a local region first, generate a smaller candidate set, and then rank what
it finds there.

KoutenDB is designed for the second approach.

The goal is not:

Replace PageRank.

The goal is:

Give PageRank, BM25, vector search, semantic ranking, and rerankers less
unrelated data to inspect.

If unrelated candidates are not placed near the query's natural locality, they
do not have to be opened first.

That can reduce:

records scanned;
payloads loaded into memory;
bytes transferred between services;
I/O amplification;
cache misses around unrelated data;
ranking and reranking work;
multi-stage ranking pressure;
tail latency from wide candidate reads;
downstream LLM context size;
cache pressure;
repeated lookup or join-style work around related metadata.

Why this helps RAG optimization and LLM context management

RAG systems and LLM applications are especially sensitive to candidate volume.

Even if the final answer only needs a few passages, the system may spend work
on many documents before that point:

retrieving candidates;
loading document chunks;
holding payloads in memory;
computing or comparing embeddings;
reranking;
trimming context;
sending text into an LLM.

If placement can reduce the candidate set before those steps, the savings are
not limited to database read latency. They can propagate through the rest of
the retrieval pipeline, reranking pipeline, and LLM serving path.

This is why KoutenDB's locality model is not only a storage concern. It is also
an AI efficiency concern.

In a RAG workload, reading less unrelated data can mean fewer tokens, less
reranking work, smaller LLM context windows, less memory pressure, and lower
LLM serving cost.

This is different from ordinary partitioning

At first glance, rings may look like database partitioning or sharding.

There is overlap, but KoutenDB is trying to use the idea more broadly.

Partitioning or sharding usually answers:

Which shard or partition owns this data?

KoutenDB rings and stellar neighborhoods also answer:

Which nearby data is likely to be useful with this data?

That second question is the important one.

For search, recommendation, RAG, and vector retrieval, the expensive part is
often not finding one exact row. It is candidate generation and candidate
pruning: finding a useful local neighborhood without opening a large amount of
weakly related data.

Not a replacement for analytical warehouses

This does not mean KoutenDB should replace BigQuery, Snowflake, ClickHouse, or
other analytical systems.

Large cross-dataset aggregation is a different problem. If the task is to scan
massive historical data and compute arbitrary analytics, a columnar warehouse
or analytics engine is the right tool.

KoutenDB is aimed at operational databases, search backends, recommendation
systems, RAG infrastructure, and retrieval-heavy paths where the application
already knows meaningful locality:

user;
tenant;
topic;
region;
source;
document group;
project;
time window;
search context;
RAG corpus segment;
vector-search namespace;
LLM memory or context boundary.

In those paths, many "aggregate", "related data", or "context retrieval" reads
can be reduced to:

Read this ring or nearby neighborhood, then summarize or rank it.

That is a different shape from global ad-hoc analytics.

The practical design rule

The practical rule is simple:

Do not put unrelated data near each other unless the application expects to
retrieve it together.

That sounds obvious, but many systems lose this information after ingestion.
The application knows that data belongs to a user, tenant, topic, source,
document group, vector-search namespace, or RAG corpus segment. Then later,
retrieval has to rediscover that relationship through indexes, joins, filters,
ranking features, or model-based reranking.

KoutenDB tries to keep that locality visible to the database.

When writing data, the application or import rule chooses a ring:

docs/japan/search
users/123/context
web/github/nim/koutendb
products/42/reviews/2026

When reading data, that same ring becomes the first retrieval scope for search,
recommendation, RAG, vector retrieval, or application reads.

That is the core idea.

What KoutenDB is trying to prove

The claim is not that every workload becomes faster automatically.

The claim is narrower:

If the storage layout captures useful data locality, then retrieval can avoid
opening much of the unrelated dataset before ranking, vector search,
reranking, or LLM context construction begins.

That is why KoutenDB benchmarks focus on working-set reduction, scanned records,
candidate memory, RAG token reduction, and tokens/query, not only raw key-value
latency.

In other words, KoutenDB is trying to make storage layout part of retrieval
architecture. The target is not only faster reads, but less unnecessary work in
candidate generation, pre-filtering, ranking, reranking, and context
construction.

Raw latency still matters. A locality-aware database cannot win if every local
read is slow.

But the larger bet is that reducing the amount of data touched can matter more
than making a huge scan slightly faster.

Why this could matter at large scale

At small scale, reading extra data is often tolerable.

At search-engine, vector-search, or AI-platform scale, "extra data" becomes
expensive:

more memory;
more I/O;
more I/O amplification;
more network traffic;
more ranking compute;
more reranking compute;
more LLM tokens;
more cache churn;
worse cache locality;
wider tail latency;
more infrastructure pressure.

If a system can avoid opening unrelated candidate regions in the first place,
the benefit can be much larger than a small percentage optimization in one
component.

The useful mental model is:

Data locality becomes the first-stage retrieval index.

Ranking still matters. Search indexes still matter. Vector indexes still
matter. Traditional database indexes still matter.

But if the first stage can avoid reading most unrelated data, every later stage
starts with a smaller problem.

That is one of the main things KoutenDB is trying to explore.

Project repository:

https://github.com/puffball1567/koutendb

KoutenDB v0.6.0: Locality Validation, Topology Remapping, and Safer Query Boundaries

puffball1567 — Fri, 17 Jul 2026 17:27:11 +0000

I released KoutenDB v0.6.0.

Release:

https://github.com/puffball1567/koutendb/releases/tag/v0.6.0

KoutenDB is a ring-oriented NoSQL document and vector database written in Nim.
The project is still a technical preview, but v0.6.0 is an important release
because it moves more of the project from concept and happy-path demos toward
measurable locality behavior.

The main theme of this release is:

If data locality is part of the database model, it should be tested as an
invariant, not only described as an idea.

What Changed

v0.6.0 focuses on five areas:

safer read filters;
topology remapping foundations;
locality validation workloads;
easier operational configuration;
practical use-case recipes.

The release adds typed KoutenFilterBuilder helpers so applications can build
read filters without string-concatenating JSON.

It also adds topology remapping primitives:

explicit arc tables;
weighted arcs;
deterministic virtual arcs;
topology validation;
remapFraction.

These do not mean KoutenDB has full online dynamic membership or live rebalance
yet. They are lower-level primitives for modeling ownership and remapping
behavior before exposing a larger operational protocol.

Locality As An Invariant

The most interesting part of v0.6.0 is the locality validation work.

KoutenDB's thesis is that meaningful placement can reduce unnecessary reads,
transfers, memory pressure, and downstream AI/RAG work. But that claim needs to
survive less friendly workloads than a clean first benchmark.

So v0.6.0 adds workloads for:

random writes;
delete-heavy patterns;
backfill-heavy patterns;
hot/cold data;
interleaved writes;
compaction before/after checks.

The invariant is simple:

The same logical ring query should return the same ID/payload set before and
after compaction, while KoutenDB reports locality metrics such as candidate
size and disk-span behavior.

That matters because data-locality systems can look good when data is inserted
cleanly once. Real systems mutate, delete, backfill, and query from odd angles.
This release starts testing that pressure directly.

Why This Matters For AI And RAG

KoutenDB is not only for AI workloads, but AI/RAG is one of the clearest places
where locality can matter.

In many retrieval-heavy systems, the expensive part is not always finding one
record. The expensive part is opening too much unrelated data, transferring it,
holding it in memory, reranking it, summarizing it, or passing it downstream as
LLM context.

KoutenDB tries to make the application's natural locality part of the retrieval
model.

For example:

docs/japan/support
tenant/acme/orders/2026
users/123/profile

These are not just labels after retrieval. In KoutenDB, rings are placement and
read-scope units. A good ring can reduce the candidate set before more
expensive ranking or application logic begins.

v0.6.0 does not claim that KoutenDB is universally faster than Redis,
PostgreSQL, MongoDB, Apache Arrow, or a dedicated vector database. The more
careful claim is narrower:

KoutenDB is building a database model where locality can be measured,
preserved, and used to reduce unnecessary retrieval work.

Safer Query Boundaries

This release also adds typed filter helpers.

Instead of building filter JSON by concatenating strings, applications can use
structured helper APIs. That is not a flashy database feature, but it matters
for a database that wants to be usable from application code and from multiple
drivers.

The same direction applies to the CLI and C ABI work in recent releases:
KoutenDB is trying to keep the public surface small, explicit, and testable.

Operational Configuration

v0.6.0 adds CLI connection config loading through:

kouten --config=kouten.json health

or:

KOUTEN_CONFIG=kouten.json kouten health

Example:

{
  "peers": ["127.0.0.1:17301"],
  "galaxy": "docs",
  "user": "alice",
  "password": "secret",
  "secretKey": "shared-secret",
  "tls": true,
  "tlsCaFile": "certs/ca.pem",
  "tlsServerName": "koutendb.internal"
}

This makes local demos and small deployments easier to repeat without copying a
long list of flags into every command.

Use-Case Recipes

I also added docs/use-case-recipes.md.

It covers examples such as:

list/detail screens;
membership records;
inventory-style locks;
webhook idempotency;
SaaS tenant isolation;
stellar neighborhood reads;
RAG corpus layout.

The point is to show where KoutenDB's model is useful outside benchmark scripts.
KoutenDB is not trying to replace every database shape. It is trying to be strong
when data has meaningful locality and when reducing the candidate working set
matters.

Try The Locality Demo

The locality demo can be run with:

examples/locality_layout_demo.sh

It exercises different write patterns, compaction, and logical result checks.

The important output is not just a speed number. It is whether KoutenDB can keep
the logical query result stable while reporting how the physical layout changes.

Current Boundaries

KoutenDB remains a technical preview.

Some important things are still not finished:

online dynamic membership;
live rebalance;
cluster transaction coordinator redundancy;
full production-grade operational hardening;
larger real-corpus benchmarks.

Universe sync remains an eventual-convergence primitive, not a consensus or
quorum system.

That boundary is intentional. I would rather keep the claims narrow and make
the measurements stronger than present KoutenDB as a finished replacement for
existing databases too early.

Links

Repository:

https://github.com/puffball1567/koutendb

Release:

https://github.com/puffball1567/koutendb/releases/tag/v0.6.0

Documentation:

https://puffball1567.github.io/koutendb/

The next work after v0.6.0 is hardening: C ABI safety, TLS/C ABI build
consistency, WAL integrity, data-directory locking, sync acknowledgement
safety, and clearer release gates.