DEV Community: Agave Information Soultions, LLC

We Tested 300 Phoenix and Scottsdale Small-Business Websites. Most Fail on a Phone.

Agave Information Soultions, LLC — Fri, 02 Oct 2026 16:49:43 +0000

Your website is usually the first thing a customer sees, and they see it on a phone. So we measured how local business websites hold up there.

On October 2, 2026 we took 300 independent small businesses from six chamber of commerce member directories, 150 in Scottsdale and 150 across Phoenix, Mesa, Tempe, Gilbert and Chandler, and ran each homepage through Google's PageSpeed Insights mobile test. We also read each homepage for the basics a local customer and Google both look for. No chains or franchises, and no business is named here.

The Headline: 14 of 297 Scored Good

Google's test scores a page from 0 to 100 on a phone. 90 and up is good, 50 to 89 needs improvement, and under 50 is poor. Of the 297 sites the test could score:

14 scored good (5%).
177 needed improvement (60%).
106 scored poor (36%).

The median score was 56.

Scottsdale and the Phoenix side came out the same. Both had a median of 56, and both had exactly 53 sites in the poor band. A Scottsdale address does not buy a faster website.

The Wait Is the Problem

The number that matters most to a customer is how long until the main content appears. Google calls it Largest Contentful Paint and wants it under 2.5 seconds.

287 of 297 sites (97%) took longer than 2.5 seconds in Google's phone test.
261 (88%) took longer than 4 seconds, the point Google calls poor.
The median was 9.7 seconds. 141 sites (47%) took longer than 10.

Google's test simulates a mid-range phone on a slower mobile connection, so it is harder than a new phone on good signal. That is on purpose: it is what a customer sees in a parking lot with two bars.

Real visitors tell a similar story where Google has data. Google collects real-visitor speed only for sites with enough traffic. Only 92 of the 300 (31%) had any. Of the 35 homepages with complete real-visitor data of their own, 21 passed Google's Core Web Vitals and 14 did not.

The Basics Are Missing, Too

Speed is not the only gap. On the homepage alone:

179 of 300 (60%) had no LocalBusiness structured data, the code that tells Google and AI assistants the business name, address, phone and hours in a form they read directly.
117 (39%) had no tap-to-call link. A customer on a phone has to copy the number by hand.
70 (23%) had no meta description, so Google writes the search snippet itself.
22 (7%) had no mobile viewport tag, so the page renders as a shrunken desktop page on a phone.
15 (5%) did not use HTTPS, and browsers mark them "Not secure".
29 of the 194 sites that show a copyright year show 2024 or earlier.

Which Platforms We Found

The page code names the platform on most sites. WordPress ran 143 (48%), Wix 21, Squarespace 19, GoDaddy's builder 12, Shopify 8 and Duda 5; 81 could not be identified. Median phone scores by platform: Duda 64 (5 sites), Wix 63, WordPress 56, GoDaddy 45.5, Squarespace 42, Shopify 39.5. The smaller groups are small, so read those as a direction, not a ranking. What decides the score is how a site is built and what it loads, more than which builder it sits on.

What to Fix First

If you own one of these sites, in order:

Run the test yourself. Google's PageSpeed Insights is free. Use the Mobile tab, and run it twice, because single runs vary.
Shrink what loads first. On most slow sites the biggest cost is a large hero image or video, plus scripts for sliders, chat widgets and trackers. Compress the image, size it for a phone and load the rest later.
Add tap-to-call. A phone link is one line of code and the most useful button a local site has.
Add LocalBusiness structured data with your name, address, phone, hours and service area, matching your Google Business Profile exactly.
Write a meta description for the homepage and each service page, in the words a customer would search.

If you would rather have someone look, our free website check reviews your site's speed and local search basics and tells you what to fix first. We build fast local websites for businesses in Phoenix and Scottsdale.

How We Did It

Sample: 300 independent small businesses with their own website, taken from the member directories of the Scottsdale, Mesa, Greater Phoenix, Tempe, Chandler and Gilbert chambers of commerce: 150 in Scottsdale and 150 on the Phoenix side (Mesa 58, Phoenix 47, Tempe 32, Gilbert 7, Chandler 6). Trades as each directory lists them, led by health care, restaurants, retail, real estate, personal services, legal, contractors and professional services. Chains and franchises, businesses with no website or only a social or directory page, and sites that failed to load twice were left out.
Speed: Google PageSpeed Insights API, mobile strategy, performance category, one run per homepage, on 2026-10-02 between 12:29 and 13:20 UTC. Three sites returned no score and are left out of the speed figures.
Real-visitor data: Google's Chrome UX Report as returned by the same API, counted only at the page's own level (not site-wide), and never filled in from the lab test.
Basics: one plain fetch of each homepage, reading the HTML as served. A feature added later by scripts, or present only on an inner page, counts as absent.
Platform: the generator tag or the platform's own asset paths only.

Agave Information Solutions builds custom software, local-first websites, SEO, and private AI search solutions for organizations that want to be found. Based in Scottsdale, serving all of Arizona.

When to Rewrite .NET in Rust (and When Not To)

Agave Information Soultions, LLC — Sun, 27 Sep 2026 07:00:00 +0000

Rewriting a working application in a new language is one of the most expensive decisions an engineering team can make, and one of the most frequently made for the wrong reasons. We modernize .NET systems for a living, so when the question came up for a production LTO tape archive application written in .NET 9, we did the unglamorous thing and actually ran the experiment. We ported the core of it to Rust: a four-crate workspace covering the catalog, the tape read and write engine, a command-line tool, and a desktop shell.

The interesting finding was not "Rust is faster." It is that the rewrite question is almost never about which language is better. Here is the honest scorecard from doing it for real: where Rust earned its keep, where it hurt, and the four questions that actually decide whether you should follow us.

Where Rust Earned Its Keep

Several of the wins were immediate and structural, not matters of taste.

The pipeline that was hardest in C# became the easiest in Rust. Our archive writer streams a file into a TAR container while computing a SHA-256 checksum in the same pass. In .NET that means nesting a TarOutputStream around a CryptoStream around a FileStream, and the disposal order matters: get it wrong and you get truncated writes on the error path. In Rust the same thing is a small generic wrapper, a "hashing writer" built on the standard Write trait, and the language's cleanup guarantees mean the flush happens correctly even when something fails halfway through. The Rust version was shorter than the C# version and harder to misuse. That is the good kind of win.

Reading our legacy data took zero glue code. The old catalog ships as JSON with PascalCase keys and a scattering of missing and nullable fields. One attribute on the Rust type handled the casing for every field at once, and a one-line default handled the missing keys. The C# equivalent is a per-field annotation marathon. We imported years of legacy archives without writing a single line of mapping code.

Dependency management is simply better. A Cargo workspace pins every crate version in one place, and each member project opts in with a single line. After living with NuGet's per-project package references plus a central props file to keep them aligned, this felt like an obvious upgrade. Adding a whole new crate to the workspace was four lines.

Whole categories of bug became visible at compile time. An unknown status code coming out of the database is a case the compiler forces you to handle, instead of silently mapping to zero the way a C# enum does. Error handling that would have been try/catch noise collapsed into clean, linear code. None of this is exotic Rust. It is the boring, everyday safety that adds up over a codebase.

And it is less code. The Rust port reproduces the same functionality in roughly a third to a half of the lines, partly because Rust collapses the plain-object, data-transfer-object, and validator triad that a .NET codebase spreads across three files into one struct with a couple of derived traits.

The Costs Nobody Puts in the Estimate

None of that makes the rewrite free. The costs that sink these projects are rarely the ones in the proposal.

There is no "just install the SDK." .NET's single-installer setup is an underrated superpower. Standing up the Rust toolchain on a locked-down Windows machine cost us 40 minutes fighting antivirus file-locks, a manual toolchain extraction, and hunting down a C linker the standalone download did not bundle. On one developer's laptop that is a one-time annoyance. Across a team and a continuous-integration pipeline, it is real recurring friction you have to own with a documented bootstrap script.

Some constraints are structural, not stylistic. Our SQLite handle is safe to move between threads but not to share across them, so the moment it sat behind a UI boundary we had to wrap it in a lock. That is the correct design for a single-writer catalog, but it is the kind of thing you only learn by hitting it, and a team new to the language will hit a dozen such walls before the patterns become second nature.

The biggest cost is not the language at all. The engine port went quickly. What does not port quickly is the years of productized ergonomics wrapped around it: the desktop chrome, the settings and credential storage, the installer, the code signing, the auto-update channel, the offline runtime bootstrapping for customers who are not online. Our desktop shell compiled and opened a real window in 25 minutes. Getting from "a window opens" to "a non-technical customer can install, trust, and update this" is the multi-week mountain. The language was never the hard part. The product around the language was.

So we sized it honestly: roughly 9 to 14 weeks for one competent-but-not-expert engineer to reach feature parity with the .NET app. A Rust veteran who has shipped a desktop application before would do it in 6 to 8. Either way, that is real time you are spending instead of shipping features your customers asked for.

The Four Questions That Actually Decide It

Notice that nothing above is "which language is better." A rewrite is a business decision wearing an engineering costume. Before you start one, answer these in order.

What specific problem does the current codebase have? "I want to learn Rust" is a fine reason to start a side project and a terrible reason to rewrite a product. If you cannot name the concrete pain (a class of bug, a performance floor you keep hitting, a deployment story you cannot stand behind), stop here. You are about to spend three months solving a problem you have not defined.
Is the rewrite cheaper than the friction it removes? Our number was 9 to 14 weeks. If the friction you are escaping is structural and would otherwise persist for a decade, that trade can pencil out. If it is accidental complexity that one focused refactor in C# would fix, it does not. Most of the time, it does not.
How strong is your team in the new language? One expert paired with someone learning produces good Rust. Three beginners produce code that is worse than the equivalent .NET team would have written, because the rate of subtle mistakes per week is higher while everyone is still climbing the curve. Be honest about which team you actually have.
What does the customer notice? A rewrite done under a feature freeze looks, from the outside, like the product was broken for six months and then magically became the same. That is the worst possible experience for someone paying you. If you rewrite, ship the new version before you retire the old one. Never freeze your customers in place while you rebuild.

So When Should You Actually Do It?

Rust earned the bet for us on this one product because the answers lined up. The friction was structural: a single self-contained binary and compile-time memory safety genuinely matter when you are writing irreplaceable archives to an 18-terabyte tape that no one will read back for a decade. The domain let the type system delete whole classes of bug. And we had the Rust depth to carry it. When those things are true, Rust is not hype. It is leverage that compounds for years.

But for most of the .NET systems we are brought in to assess, the answers do not line up. The pain is accidental complexity a refactor away. The team is all-.NET. The application is interface-heavy, where the chrome is the bulk of the value and the language underneath it is almost incidental. In those cases the right move is not a rewrite at all. It is modernizing the .NET you already have: the current runtime, ahead-of-time compilation, trimming, and a cleanup of the actual hotspots. That gets you most of the benefit for a fraction of the cost and none of the customer-facing freeze.

The language comparison is the easy, fun part of this conversation, which is exactly why it gets all the attention. The decision underneath it is harder and more boring: what does your codebase actually cost you today, and is twelve weeks the cheapest way to fix it? Answer that first. The language almost always sorts itself out once you have.

Originally published at agaveis.com.

Your Company Does Not Need Another AI Chatbot. It Needs a Knowledge Layer.

Agave Information Soultions, LLC — Sat, 26 Sep 2026 14:46:00 +0000

Every company adopting AI eventually runs into the same problem.

An employee opens ChatGPT, Claude, or another AI assistant and asks a question about the business:

How do we deploy this application?
What did we decide about the payment provider?
Where is the current policy?
Has this production error happened before?
Which database table owns this field?
What process should the support team follow?
Why was this architecture chosen?

The model usually does not know.

The employee then copies documents, source code, email threads, meeting notes, or internal wiki pages into the conversation. Another employee does the same thing the next day. A developer starts every coding session by explaining the application architecture again. Important lessons remain buried in old chat sessions that nobody else can access.

The company may be paying for powerful AI models, but every session starts with organizational amnesia.

That is not primarily a chatbot problem.

It is a knowledge architecture problem.

The Real Problem Is Not Missing Information

Most established companies already have plenty of information.

They have too much of it.

Knowledge is scattered across SharePoint, Google Drive, Git repositories, Confluence, Teams or Slack, ticketing systems, email, network drives, databases, employee laptops, Markdown files, old PDF manuals, meeting notes, chat histories, and the memory of the one employee who has been there for 17 years.

The difficult question is not:

Can an AI model read documents?

It can.

The difficult questions are:

Which document is current?
Which document is approved?
Which version was superseded?
Which employees are allowed to see it?
What information should never leave the company?
How can the answer cite its sources?
How do we know the retrieved information is correct?
What happens when a document is deleted or its permissions change?
Can the company change AI providers without rebuilding everything?

Uploading folders into a chatbot does not solve those questions.

It usually hides them.

Why We Are Building This at Agave Information Solutions

We began with a smaller version of the problem inside a busy development workflow.

Over time, we built an Obsidian knowledge vault containing project documentation, deployment procedures, architecture decisions, production incidents, database notes, infrastructure details, troubleshooting history, runbooks, lessons learned, and agent instructions.

The vault follows a pattern similar to Andrej Karpathy's idea of an LLM-maintained wiki: durable knowledge should live between raw source material and individual AI conversations.

That model works.

Instead of explaining an application from scratch every time we open Codex or Claude Code, an agent can start with the current project state, accepted architecture decisions, and known deployment traps.

But the vault grew.

It passed 1,000 Markdown files. Historical logs, archived plans, active runbooks, and project notes began competing for attention.

The problem was no longer whether the information existed. The problem was whether an agent could reliably answer:

Which five documents should I trust for this task?

That question led us beyond a personal Obsidian vault and toward a broader architecture that could work for other companies.

We are now building a governed organizational knowledge platform using PostgreSQL, pgvector, full-text search, an ASP.NET Core API, and the Model Context Protocol.

Our vault is tenant zero.

The long-term goal is a reusable system that connects approved company knowledge to the AI tools employees already use.

A "Shared Brain" Is the Wrong Mental Model

People often describe this kind of system as a company-wide shared brain.

The phrase is catchy, but technically dangerous.

A company does not need a machine pretending to be its brain. It needs a controlled knowledge service that can answer three questions:

What information is relevant?
Is the employee allowed to access it?
What source supports the answer?

The AI model should not become the source of truth.

The model is a consumer.

The source systems remain authoritative: the current runbook, the accepted architecture decision, the approved policy, the live application code, the production database, and the document with the correct access controls.

The knowledge platform sits between those sources and the AI assistant.

Its job is to retrieve the right evidence.

The Architecture

The system we are building follows this flow:


Company knowledge sources
        ↓
Ingestion and normalization
        ↓
Metadata, ownership, permissions, and versioning
        ↓
Structural document chunking
        ↓
Embeddings and full-text indexing
        ↓
PostgreSQL + pgvector
        ↓
Permission-aware hybrid retrieval
        ↓
REST API and MCP tools
        ↓
Codex, Claude, ChatGPT, internal portals, and agents

PostgreSQL becomes the retrieval and governance layer.

It does not replace the original documents.

If the source is an Obsidian vault, employees continue editing Markdown. If the source is SharePoint, SharePoint remains authoritative. If the source is GitHub, the repository remains authoritative.

The database stores a normalized and searchable representation that can be rebuilt.

That distinction matters.

A generated search index can be deleted and recreated. A company knowledge system that silently becomes the only copy of the truth is another future migration project wearing an AI hat.

Why PostgreSQL and pgvector

There are many specialized vector databases available.

We chose PostgreSQL because company knowledge is not just a pile of vectors.

A production system also needs tenants, users, groups, document ownership, access-control lists, source identifiers, version history, lifecycle status, effective dates, retention rules, retrieval logs, user feedback, and evaluation results.

That information is relational.

PostgreSQL can store the organizational metadata, perform native full-text search, and use pgvector for vector similarity in the same system. The pgvector project specifically supports combining vector retrieval with PostgreSQL full-text search, including fusion methods such as Reciprocal Rank Fusion.

For our first implementation, local PostgreSQL is intentional. The entire retrieval stack runs in a single Docker container on a developer machine: PostgreSQL 17 with pgvector, costing nothing and touching no production infrastructure. That is not a compromise. It is the fastest honest way to answer the only question that matters at this stage: does retrieval return the right evidence?

The same local-first rule applies to embeddings. The first slice generates them with a small open-source model running in-process, which means indexing sends nothing to any external API. Providers sit behind an interface, and the store records which model produced each vector, so upgrading to a commercial embedding model later is a deliberate re-index, not a silent drift.

Starting local does not trap the system on a laptop. Because the store is rebuildable from the sources by design, promoting it to managed infrastructure later is a re-ingest, not a migration. Amazon RDS for PostgreSQL ships pgvector on every currently offered PostgreSQL major version (as of mid-2026, all of them carry pgvector 0.8 or newer), and enabling it is a single CREATE EXTENSION vector;. The schema that proves retrieval quality on one machine is the same schema that will eventually serve multiple employees, multiple applications, and multiple AI providers from managed AWS infrastructure.

Do Not Use Vector Search Alone

The typical RAG tutorial works like this:

Split documents into chunks.
Convert each chunk into an embedding.
Store the embeddings.
Convert the user's question into an embedding.
Return the nearest chunks.
Ask an LLM to write an answer.

That is a useful demonstration.

It is not enough for company knowledge.

Semantic similarity can retrieve a document that sounds right while being completely wrong operationally.

Imagine that the database contains the current production deployment runbook, a deployment plan abandoned two years ago, a postmortem describing a temporary emergency fix, a draft migration strategy, and an archived version of the old application.

All five documents may be semantically similar to the question:

How do we deploy the application?

The nearest vector is not necessarily the correct answer.

This is why the retrieval process must begin with metadata and authority.

Before semantic search runs, the system should filter by tenant, user permissions, project, document type, lifecycle status, authority, classification, effective date, and archive status.

Only then should it compare embeddings.

Authority Must Be Explicit

Every company already has an authority hierarchy, whether it documents one or not.

For example:


Verified production state
    ↓
Approved policy or current runbook
    ↓
Accepted architecture decision
    ↓
Active project documentation
    ↓
Reference material
    ↓
Historical logs
    ↓
Archived or superseded documents

The retrieval system must understand those distinctions.

We model lifecycle metadata such as canonical, approved, active, reference, draft, historical, superseded, archived, and expired. The first slice enforces the core buckets today, and the finer authority tiers arrive with the hardening phase.

A canonical runbook receives a strong ranking advantage.

A superseded document is excluded.

An archived incident may be available when the user explicitly asks for history, but it should not appear as current guidance.

Age alone is not enough.

A five-year-old architecture decision may still be valid. A note written yesterday may still be an unreviewed guess.

The system must model authority, not just freshness.

Permissions Must Be Applied Before Retrieval

This is the point where a personal RAG system becomes an enterprise system.

Suppose an employee asks:

What did leadership decide about the acquisition?

The database may contain relevant documents, but the employee may not be authorized to see them.

The wrong architecture performs vector search across everything and removes unauthorized results afterward.

That is dangerous.

Titles, snippets, similarity scores, or generated summaries may already reveal sensitive information.

The correct sequence is:


Authenticate the user
    ↓
Resolve company and group membership
    ↓
Determine the permitted document set
    ↓
Apply classification and lifecycle rules
    ↓
Run lexical and vector retrieval
    ↓
Return permitted evidence

PostgreSQL provides Row-Level Security policies that can restrict which rows are visible to particular database roles or session contexts. Once enabled, policies can apply expressions that determine whether a row is visible or writable.

Application-level permission checks are still necessary, but database-level enforcement provides defense in depth.

For a multi-company platform, every relevant table should carry a tenant identifier from the beginning.

Adding tenancy after the data model is complete is how architecture teams accidentally invent a six-month rewrite.

Hybrid Search Works Better Than Semantic Search Alone

Company questions often contain exact technical language: error codes, database table names, API routes, customer identifiers, product names, class names, server names, and policy numbers.

Embeddings are good at meaning.

They are not always good at exact identifiers.

PostgreSQL full-text search can retrieve exact and lexical matches. pgvector can retrieve semantically similar passages. A fusion algorithm can combine both result sets.

Our pipeline uses:


Metadata and ACL filters
        ↓
PostgreSQL full-text retrieval
        +
pgvector semantic retrieval
        ↓
Reciprocal Rank Fusion
        ↓
Authority adjustment
        ↓
Optional reranking
        ↓
Cited evidence

We are starting with exact vector search rather than immediately adding approximate indexes.

Our first corpus is small enough that query quality matters more than shaving milliseconds off a benchmark.

Optimization without a measured problem is just architecture cosplay.

Documents Should Be Chunked by Structure

Another common RAG mistake is chopping documents every fixed number of characters.

That can produce chunks such as:


...restart the production server.

## Do Not Restart the Production Server

Under no circumstances...

The first chunk now contains advice that appears to say the exact opposite of the document.

Markdown already contains useful structure: document titles, headings, subheadings, lists, tables, code blocks, and wikilinks.

We parse the heading structure and create chunks around meaningful sections. The parser is code-fence aware, so a # comment inside a code block is never mistaken for a heading.

Each stored chunk retains context such as:


Document: Hiberden Deployment Runbook
Project: Hiberden
Type: Runbook
Authority: Canonical
Heading: Backups > Cloud Destinations > Resume

That improves retrieval and makes citations understandable.

It also allows the system to re-process only the documents that changed instead of paying to re-embed the entire corpus.

The Database Is an Index, Not the Author

The source document remains the editable artifact.

For our first implementation:


Obsidian Markdown + Git = source of truth
PostgreSQL = searchable representation
Embeddings = ranking signal
Full-text search = ranking signal
Metadata = authority and security boundary
MCP/API = controlled agent access

The ingestion process will:

Scan approved folders.
Exclude archives, credentials, caches, and unsupported files.
Read YAML frontmatter.
Parse Markdown structure.
Calculate content hashes.
Skip unchanged documents.
Create stable chunks.
Reuse unchanged embeddings.
Insert new versions transactionally.
Reconcile deleted and moved documents.

A filesystem watcher may provide fast updates, but periodic full reconciliation is still necessary. Watchers miss events. Production systems should assume that eventually something will fail while nobody is watching the watcher.

Connecting Codex and Claude Through MCP

We do not want every AI client to implement its own database access.

The platform exposes a controlled retrieval API and an MCP server.

MCP allows compatible AI clients to access defined tools and contextual data. Codex supports MCP servers through both its CLI and IDE integration.

The first tool is already live: search_knowledge, a read-only retrieval tool that Claude Code picks up from a one-file repository configuration. The next tools on the roadmap are narrow in the same way:


search_knowledge
get_document_section
get_current_project_context
submit_retrieval_feedback

The server will not expose:


run_arbitrary_sql
browse_every_customer
update_company_policy
execute_deployment

The first version is intentionally read-only.

The agent can retrieve evidence. It cannot silently rewrite the company's operating procedures because it had a particularly confident Tuesday.

The Smallest Useful Implementation, Built

We did not start with every company source and a polished chat interface.

The first complete vertical slice is:


One approved Markdown folder
    ↓
Structural parsing
    ↓
Chunking
    ↓
Embeddings
    ↓
PostgreSQL storage
    ↓
Hybrid search
    ↓
Cited API response
    ↓
One MCP tool
    ↓
Ten evaluation questions

We built that slice against a working vault of real project notes before publishing this post.

The first full run ingested 30 approved documents into 179 heading-scoped chunks and passed all ten golden evaluation questions, with mean retrieval latency around 50 milliseconds on a local container. The set includes a trap case where an archived near-duplicate of a current page must lose to the current page, and a case where the correct behavior is admitting the corpus does not contain the answer.

That proves the important parts:

The ingestion pipeline works.
The database model works.
Retrieval works.
Citations work.
Agents can use the service.
Quality can be measured.

Only after that works should the system expand into multiple tenants, source permissions, SharePoint, Google Drive, GitHub, Confluence, administrative portals, usage analytics, and customer onboarding.

Building six connectors before proving retrieval quality is a reliable way to create six pipelines that all return the wrong document.

How We Know Whether It Works

A RAG system cannot be judged by whether the answer sounds intelligent.

Language models are exceptionally good at making weak retrieval look convincing.

We maintain an evaluation set containing real questions and expected evidence.

An evaluation case defines:


question: How do we deploy this application to production?

expected_documents:
  - current deployment runbook

forbidden_documents:
  - archived deployment plan

required_authority:
  - canonical

expected_facts:
  - publish location
  - deployment verification

We track:

Whether the correct document appears in the top results
Whether the canonical source ranks above historical notes
Whether archived documents appear when they should not
Whether citations point to the correct section
Whether permission rules leak restricted content (once the permission model lands)
Whether the system correctly returns no answer
Retrieval latency
Embedding costs
User feedback

The most important test is not:

Did the model produce a good paragraph?

It is:

Did the platform give the model the correct evidence?

What This Means for Companies Adopting AI

Companies do not need to standardize every employee on one AI model.

Models will continue to change.

Some teams will use ChatGPT. Developers may use Codex or Claude Code. Other employees may use Microsoft Copilot, internal applications, or models running through Amazon Bedrock.

The durable investment is the knowledge layer behind them.

A well-designed platform allows the company to control sources, permissions, authority, retrieval, citations, logging, and evaluation.

The LLM becomes replaceable.

That is a healthier architecture than uploading the same employee handbook into six AI products and hoping each one remembers which version is current.

Where We Are Starting

The first tenant is a working Obsidian knowledge vault of real project notes.

That gives us a realistic test corpus containing years of application architecture, AWS infrastructure, deployment procedures, database design, production incidents, operational lessons, project history, and both current and obsolete documentation.

It also gives us the exact problem many companies face: valuable knowledge mixed with historical clutter.

We will document the implementation as we build it, including why we chose PostgreSQL and pgvector, how we model authority, how we chunk Markdown, how hybrid retrieval performs, how we connect Codex and Claude, how we test permissions, what the AWS infrastructure costs when we promote the store to RDS, what fails, and what turns out to be unnecessary.

The objective is not to produce another RAG tutorial that works against five clean PDFs.

The objective is to build a system that can survive real company data, real permissions, stale documents, changing AI tools, and employees who need the correct answer before the production deployment, not three days later during the postmortem.

The Takeaway

Most organizations do not need another standalone AI chatbot.

They need an organizational knowledge layer that can answer:

What information is current?
Who is allowed to see it?
What source supports the answer?
How do we know retrieval is working?
Can the same system serve multiple AI tools?

Our implementation uses Obsidian as the initial source, PostgreSQL and pgvector as the governed retrieval store, hybrid search for relevance, authority metadata for trust, and MCP for controlled agent access. All of it starts local: one container, one folder of approved documents, zero cloud spend, and nothing leaving the building until the company decides it should.

The first milestone is simple to describe:

An authenticated employee or AI agent can retrieve the correct, current, permitted company evidence with citations, and the company can prove why those results were returned.

The retrieval half of that milestone, correct, current, cited, and measured, is running today against that vault. The authenticated, permission-aware half is the next phase.

That is not a shared brain.

It is something more useful: company knowledge that employees and AI systems can finally use without losing control of it.

Agave Information Solutions builds custom software, database systems, AWS infrastructure, and private AI search solutions for organizations with complicated data. This series will document the platform from the first Markdown scan through a production-ready, permission-aware knowledge service.

Originally published at agaveis.com.

pgvector Without Embeddings: When a Feature Vector Beats Semantic Search

Agave Information Soultions, LLC — Sat, 26 Sep 2026 12:45:00 +0000

Almost every pgvector tutorial starts the same way. Take some text, run it through an embedding model, store the resulting vector, and search it with natural language. That is a real and useful pattern, and it is also why most engineers walk away thinking pgvector is a tool for one job: semantic search over text.

It is not. pgvector is a tool for finding the nearest vectors to a target vector. Where those vectors come from is entirely up to you. An embedding model is one source. A column of numbers you computed yourself is another, and for a large class of problems it is the better one.

We learned this building a "find similar players" feature for a baseball side project, and the lesson generalizes to almost any similarity problem over structured data. Here is the distinction nobody draws clearly, and how to know which side of it you are on.

What pgvector actually does

Strip away the AI framing and pgvector adds three things to Postgres. A column type, vector(N), that stores an array of N floats. A set of distance operators between two of those columns: <-> for L2 (Euclidean) distance, <=> for cosine distance, and <#> for negative inner product. And approximate-nearest-neighbor indexes (HNSW and IVFFlat) so that finding the closest vectors to a target stays fast as the table grows.

That is the whole job. Given a target vector, return the rows whose vectors are closest to it, ranked, quickly. pgvector has no opinion about whether your 384 numbers came out of a transformer or out of a SELECT and some arithmetic. The math is identical.

This matters because the moment you stop thinking "embedding column" and start thinking "any vector I can construct," a different design space opens up.

Embeddings versus feature vectors

An embedding is a vector produced by a model. You hand a chunk of text or an image to something like a sentence transformer or CLIP, and it returns a few hundred floats that encode the meaning of the input in a way you did not design and cannot fully interpret. Embeddings shine when the input is unstructured and you do not know in advance which features matter. You could never write down by hand what makes two paragraphs "similar in meaning." The model learned that for you.

A feature vector is a vector you build deliberately. You decide what the dimensions are, you compute each one from your own data, and you normalize them so they live on a comparable scale. Every dimension means something you can name. Feature vectors shine when the input is structured and you already know what makes two things similar.

The trap is treating these as the same thing because they both end up in a vector column. They are not. One is a learned, opaque representation of unstructured data. The other is an explicit, interpretable description of structured data. Calling a hand-built feature vector an "embedding" because it sounds more sophisticated is the kind of overstatement that falls apart the moment someone reads your code and finds a SELECT AVG(...) where they expected a model call. Name it for what it is.

The worked example: comparable players

We wanted to answer a coach's question: "show me pitchers similar to this one." Similarity here is not one number. A pitcher is a bundle of tendencies. What does he throw and how often. Where he locates each pitch. How hard he throws it. How his behavior changes with the count. No single SQL WHERE clause captures "pitches like this guy," which is the signal that you are in vector territory rather than filter territory.

So we built a feature vector per pitcher straight from the pitch-by-pitch data:

Arsenal mix: the percentage of each pitch type (fastball, slider, changeup, and so on). One dimension per type.
Location: the mean and spread of the horizontal and vertical location for each pitch type, computed from normalized zone coordinates.
Velocity: average and range, where it exists.
Count tendencies: how the pitch mix shifts in hitter-friendly versus pitcher-friendly counts.

Each of those is a plain aggregate over the pitcher's rows. Concatenate them, normalize, and you have a vector that describes how a pitcher actually operates. Two pitchers who attack the zone the same way land near each other in that space, and pgvector's nearest-neighbor search finds them.

The schema is unremarkable:

CREATE EXTENSION IF NOT EXISTS vector;

ALTER TABLE pitcher_profiles
  ADD COLUMN feature_vec vector(32);

CREATE INDEX ON pitcher_profiles
  USING hnsw (feature_vec vector_cosine_ops);

And the query is the part people expect to be hard and is not:

SELECT id, name
FROM pitcher_profiles
WHERE id <> @target_id
ORDER BY feature_vec <=> @target_vec
LIMIT 10;

In a .NET app with EF Core and the Pgvector provider, building the target vector and running that order-by is a few lines. The intelligence is not in the query. It is in the feature design.

Two things that decide whether it works

Normalization. Your dimensions start on wildly different scales. A pitch-mix percentage lives between 0 and 1. A velocity average might be 88. If you drop both into a vector raw, velocity dominates the distance and the mix percentages become noise. Standardize every dimension (z-score across the population, or min-max into a fixed range) before you store it, or your "similarity" just measures whichever feature happens to be biggest.

Weighting, which is the feature vector's superpower. Because you built the dimensions, you can decide that arsenal mix matters twice as much as location by scaling those dimensions before the distance is computed. You cannot do that with a black-box embedding. You get whatever the model decided was important. When you know your domain, that control is worth more than any learned representation, and it is the single best argument for the hand-built approach.

One caveat from the data: velocity was frequently missing, because the entry UI had no radar input. A vector with a sparse dimension behaves badly, since "missing" is not the same as "zero." Design the vector so it degrades gracefully when a feature is absent, either by imputing the population mean or by dropping the dimension and renormalizing. Real structured data is messier than the tutorial version, and the feature approach at least lets you see and handle that, because you can read every dimension.

When embeddings still win, and when to use neither

Feature vectors are the right tool when the input is structured and you can name what matters. Flip either of those and you want something else.

If the input is unstructured, embeddings win. Free-text scouting notes, a coach's written report, an image: you cannot hand-engineer features for those, so let a model do it. The strongest systems often run both. Use a feature vector for the structured tendencies and an embedding for the prose notes, then combine the two scores. pgvector holds both columns happily.

And sometimes the answer is no vector at all. If "similar" really comes down to one or two numeric columns, you do not need pgvector, you need a WHERE clause and an ORDER BY. Reaching for vector search there is theater. It looks advanced and adds a dependency to do what an index on a column already does. The gut check we use: vectors earn their place only when similarity spans many dimensions at once and no single filter captures it. Below that bar, plain SQL is not just adequate, it is better.

The takeaway

pgvector is a nearest-neighbor engine, not a semantic-search feature. Once you internalize that, "do I need an embedding model here" becomes a real question with a real answer instead of a reflex. For unstructured data where the important features are unknowable, embed. For structured data where you know exactly what makes two things alike, build the feature vector yourself, normalize it, weight it, and let pgvector do the search. You will ship something simpler, faster, and fully explainable, and you will be able to answer the only question that matters in a code review: why are these two results considered similar.

The honest version of that answer is worth more than the impressive-sounding one.

Agave Information Solutions builds custom software, data architecture, and on-premises AI systems out of Scottsdale, Arizona. If you have a similarity, search, or retrieval problem and are not sure whether you need a model or just better-shaped data, get in touch.

Originally published at agaveis.com.

MCP Server Security: Match the Guardrails to the Blast Radius

Agave Information Soultions, LLC — Sat, 26 Sep 2026 03:31:00 +0000

We run two Model Context Protocol servers in production. One of them can create, move, and delete things. The other cannot change a single byte, and never returns a person's name.

Both are correct. That is the point of this post.

Most MCP security writing worries about the wrong thing. It focuses on the malicious server: somebody publishes a poisoned tool, an assistant calls it, secrets leak. That is a real risk and it deserves the attention it gets. But if you operate a server rather than install one, it is not your daily problem. Your daily problem is that the assistant on the other end is not malicious, it is confidently wrong, and it is about to call a tool you gave it.

The useful question is therefore not "is this server secure." It is "when this call is a mistake, who absorbs the damage." Answer that and the guardrails design themselves.

Three doors, three postures

Between two products we ended up with three distinct surfaces, and each one landed somewhere different.

Door one: read and write, on the user's own machine

Hiberden is desktop software for LTO tape archiving, and its MCP server ships inside the application. It exposes around thirty tools and plenty of them write: create a project, assign a policy, move an archive, retire a destination, delete a copy.

Letting an assistant delete something sounds reckless until you work out who is exposed. The person driving the assistant owns the machine, owns the catalog, and could delete the same copy through the application's own interface in two clicks. Routing that through an assistant does not widen the blast radius. It is the same person, the same computer, the same file.

So the guardrails here are not about restricting capability. They are about making consent explicit and keeping receipts:

The server runs locally, over stdio, as a subprocess on the same machine as the data. There is no listening port and no inbound network surface.
Permissions are a setting inside the application, not something the assistant can grant itself.
Read-only is the default. Destructive operations sit behind their own separate switch that a human has to turn on deliberately.
Every action is written to an audit trail, so "what did it actually do" is answerable after the fact.
Credentials for the customer's own cloud storage stay in the operating system keyring. They are never returned by a tool and never pass through the assistant.
The catalog is a local file. There is no hosted service in the path, no account, and no telemetry.

The shape of that list is worth noticing. When the user absorbs the damage, you guard the decision: default to safe, make the dangerous thing an explicit choice, and write down what happened.

Door two: read only, on a shared service

Easy2257 is a hosted service holding 18 U.S.C. 2257 records for producers in a federally regulated vertical. Its MCP server exposes sixteen tools, and not one of them can create, change, sign, delete, or pay for anything.

Run the same test and you get a different answer. A mistaken write here does not ruin the operator's afternoon. It puts a false state change into a legal record about a third party who is not in the room and never consented to an assistant touching their file. The person absorbing the damage is not the person driving.

When you cannot afford a mistaken write, do not try to make writes safe. Remove them. The write path stays on the website, performed by a human, on purpose. Paid actions and legal attestations are somebody's to make deliberately, not an assistant's to infer.

Having removed what it can do, the remaining question is what it can see. Read-only is not automatically safe: a read-only tool that returns a date of birth is still a data breach waiting for an audience. So the second half of the posture is to shrink the payload until there is nothing worth stealing in it:

No names, no dates of birth, no ID numbers, no images, no addresses, no document contents, no file paths. Not redacted on the way out. Never in the response shape to begin with.
What comes back is compliance state: record codes, statuses, counts, dates, and content hashes. Records are referred to by public codes rather than by the people in them.
Every credential is scoped to exactly one account, and lookups of records outside it are throttled rather than merely denied, so the surface cannot be walked.
Keys are hashed at rest and shown exactly once, capped per account, and revoked automatically on a password reset. The interactive route uses short-lived tokens that disconnect in one click.

That last cluster matters more than it looks. The realistic failure is not somebody breaking the cryptography. It is a key that got pasted into a chat window eight months ago and nobody remembers. Caps, one-time display, and automatic revocation on password reset are all designed for that failure, not for a determined attacker.

Door three: write, for somebody else's software

The same service also has a partner API, and it is the genuinely dangerous door: it can provision an account, open a scene, and deliver a certificate, called by platforms we do not control.

Here the damage lands on third parties and the caller is a machine, so the guardrails go back up and change character again:

Live keys only, and no sandbox. Every call touches real records, real people, and real email, so there is no pretend mode that lulls anyone into treating it as low stakes.
The human who creates a live key verifies their identity first. Authentication attaches to a person, not just to a string.
Keys are shown once and stored only as a hash. Lose it and you rotate it, because we cannot show it to you again.
Every webhook is signed with a per-endpoint secret that rotates on its own. Verify the signature before you trust the body. A shared signing secret means one compromised integration compromises all of them.
Certificates are signed tokens verifiable offline against public keys. A platform never has to take our word for anything, and verification does not require calling us.
Records are consent-gated. A platform never receives a person's records without that person's consent, regardless of what the API key is entitled to.
Idempotency keys and rate limits, so a retry can never double-write. Assume every call happens more than once, because eventually it will.

The threat model people skip

Everything above assumes the assistant means well. That is the case worth designing for, because it is the common one.

A well-meaning assistant misreads intent constantly. It sees "clean up the old stuff," decides that means the archive from 2019, and reaches for a delete tool because you gave it one. It sees a job that failed and helpfully retries it four times. It reads a tool result containing text somebody else wrote and treats that text as an instruction, which is the part most teams have not internalised yet: a tool result is untrusted input, exactly like a form field, even when your own server produced it, because your server may be relaying content a customer typed.

Three habits follow from that:

Name tools for what they do, not for what they enable. A tool called delete_copy gets called deliberately. A tool called cleanup gets called speculatively.
Make destructive operations narrow and specific. Deleting one identified copy is recoverable reasoning. A tool that takes a filter and deletes everything matching it is a loaded weapon with a natural-language trigger.
Make retries free. Idempotency is a security control, not just a reliability one, because the assistant is going to retry and will not tell you it did.

The principle, in one line

Match the security to the blast radius. The door that can act authenticates people and signs what it sends. The door that can only read carries nothing worth stealing. And the door that runs on somebody's own machine, against their own files, can afford to do more, provided the dangerous parts are a deliberate choice and everything is written down.

What you should not do is pick one posture and apply it everywhere. A read-only server with an over-generous payload is more dangerous than a write-capable server with an explicit permission switch. Capability and exposure are separate axes, and it is worth deciding each one on purpose.

The desktop connector is documented at hiberden.app/mcp and the partner API at docs.easy2257.com, both including exactly what each surface can and cannot do. If you are wiring an assistant into something that matters, local AI integration and API design are both work we take on.

Originally published at agaveis.com.

Shipping an MCP Server: Desktop App vs Hosted Web App

Agave Information Soultions, LLC — Sat, 26 Sep 2026 02:29:00 +0000

We ship two Model Context Protocol servers. One runs inside a desktop application, on the machine where the data already lives. The other runs as a hosted service behind an account, in front of a multi-tenant database. They speak the same protocol and share almost none of the same decisions.

Most writing about MCP stops at the hello-world tutorial: register a tool, return a string, done. The interesting part starts after that, when you have to decide where the server runs, how it reaches users, how you update it, what it is allowed to touch, and what happens when the assistant calling it is confidently wrong. Those answers are not a matter of taste. They fall out of the deployment model almost mechanically.

Here is the comparison, decision by decision.

The two products

Hiberden is desktop software for LTO tape archiving. It keeps a local catalog of everything it has archived: every project, every copy on tape, disk, NAS and cloud, every file path, every checksum, and every overnight job. The MCP server ships inside the application.

Easy2257 is a hosted service that keeps 18 U.S.C. 2257 records for producers in a federally regulated vertical: who appeared in a scene, that identity was verified, which paperwork is signed, and a certificate proving the record existed at a point in time. Its MCP server runs as part of the service. Nothing is installed.

One catalog sits on your desk. The other sits in a database shared by every customer. Nearly everything below follows from that one difference.

1. Where the server runs decides everything else

The desktop server is a local process. The transport is stdio: the assistant launches the server as a subprocess on the same machine as the data. There is no network hop, no listening port, and no inbound surface to attack. The catalog it reads is an ordinary local file the user already owns.

The hosted server is a network service. The assistant connects across the internet to something holding many companies' records at once. Every single request has to answer "who is asking, and what are they entitled to see" before it can answer anything else.

The consequence is not subtle. On the desktop, authentication is mostly solved before you write a line of server code: the person is sitting at the machine, and the operating system already decided they can open that file. On a hosted server, authentication is the product surface. It is the first thing you build and the thing you keep rebuilding.

2. Distribution: an installer problem or a documentation problem

A desktop server has to physically arrive on the machine. A well-built one ships inside the installer, and it is also published to the official MCP registry as a versioned, checksummed bundle running over stdio, plus third-party directories like Glama, which is increasingly how assistants and their users discover servers at all.

A hosted server has nothing to install. The endpoint exists the moment the account exists. Distribution stops being an engineering problem and becomes a documentation problem: the work is making the connection instructions short enough that somebody finishes them.

Each model quietly inherits a different environment. Ship on the desktop and you inherit the user's machine: operating system versions, assistant clients that each store their config somewhere different, antivirus software with opinions about unsigned subprocesses. Ship hosted and you inherit the network: timeouts, retries, and clients that will absolutely call the same tool four times in a row.

3. Versioning: pinned forever, or changed for everyone at once

This is where the two models diverge hardest, and it is the one people underestimate.

On the desktop, the server version is pinned to the application version. A customer running last spring's build is running last spring's tools, and they may run them for years. You cannot fix a tool for everybody at once, and you cannot remove one either, because somebody who never updates will keep calling it forever. Deprecation is a release-notes problem measured in years, not sprints.

On the hosted service there is exactly one version and everyone is on it. A fix reaches every customer the moment it deploys. The flip side is that a breaking change breaks every customer simultaneously, and the software calling your tools is not your software. It is somebody's assistant, wired up months ago, that you cannot test against.

What that works out to in practice: on the desktop, make changes additive and never alter a tool signature that already shipped. On hosted, you can iterate faster, but treat the tool list as a published API contract, because that is what it is.

4. Credentials: keep them out of the conversation

The desktop application already holds credentials it needs for its own work, such as the keys for the customer's own cloud storage account. Those live in the operating system keyring. The rule we settled on is simple and worth stating plainly: credentials never pass through the assistant. The assistant asks the catalog questions and the catalog answers them. Keys are not part of the conversation, are never returned by a tool, and are never needed to call one.

The hosted server has the opposite problem, because the connection itself needs a credential. There are two routes in. A customer can generate an API key, which is hashed at rest and displayed exactly once, capped at ten per account, with every key revoked automatically if the account password is reset. Or they can sign in interactively and connect with short-lived tokens, which can be disconnected in one click. Either way, a credential is scoped to exactly one account, and attempts to look up records belonging to somebody else are throttled rather than merely refused.

5. What the tool surface can afford to do

Now the interesting part. The two servers made opposite choices, and both are right.

The desktop server exposes about thirty tools, and a good number of them write. They can create a project, assign a policy, move an archive between destinations, retire a destination, or delete a copy. That is defensible because the person driving the assistant already owns the data and the machine. Deleting a copy through an assistant is not more dangerous than deleting it through the application's own interface: same person, same computer, same file.

What it does require is that consent is explicit and the work leaves a record. Permissions are a setting inside the application rather than something an assistant can grant itself. Read-only is the default. Destructive operations sit behind their own separate switch. Every action is written to an audit trail.

The hosted compliance server exposes sixteen tools and none of them can create, change, sign, delete, or pay for anything. That is not timidity. On a shared compliance service, a write tool means an assistant can hallucinate a state change into somebody's federal records. The blast radius is not a ruined afternoon, it is a legal record about a real person. So the write path stays where it belongs: on the website, performed by a human being, on purpose.

The rule that falls out of the comparison: how much a tool is allowed to do should track who absorbs the damage when it is wrong.

6. What the tool surface is allowed to see

The second divergence is quieter and, we think, more important.

The desktop server returns file paths, file names, checksums, and tape barcodes. Every bit of that is already sitting on the user's own disk. Handing it to an assistant running as a subprocess on the same machine leaks nothing that was not already there to read.

The hosted compliance server deliberately returns none of the identity data it holds. No names, no dates of birth, no ID numbers, no images, no document contents, no file paths. What comes back is compliance state: record codes, statuses, counts, dates, and content hashes. Records are referred to by public codes rather than by the people in them.

That is not a gap in the feature set. It is the reason the surface is safe to point at a shared assistant at all. There is nothing in the payload worth stealing.

7. Discovery works differently too

A desktop server is discovered the way software is discovered: registries, directories, and the installer the user already ran. A hosted server is discovered the way an API is discovered: documentation, a setup page, and a connect button inside the product. If you are building one, budget real time for whichever of those applies. A server nobody can find is indistinguishable from a server that does not exist.

What we would tell someone starting

Decide where the server runs before you write a single tool. Transport, authentication, versioning strategy, and tool design all fall out of that one choice, and reversing it later means rewriting all four.
Know which set of hard problems you just signed up for. On the desktop they are distribution, versioning, and permission design. On hosted they are identity, tenancy, and blast radius.
Keep credentials out of the conversation. No tool should return one, and no tool should require the assistant to hold one.
Treat the tool list as a public API contract either way. On the desktop because you can never take a tool back. On hosted because when you take it back, you take it back from everyone at once.
Let the blast radius decide the capability. Not the other way around, and not whatever the framework made easy.

Both servers are live. The desktop connector is documented at hiberden.app/mcp, including exactly what it can and cannot do. If you are building something similar and want a second pair of eyes on the architecture, that is the kind of work we do in custom development and local AI integration.

Originally published at agaveis.com.

LTFS and LTO Across Windows, macOS, and Two Flavors of Linux

Agave Information Soultions, LLC — Sat, 26 Sep 2026 00:29:14 +0000

LTFS sounds like a solved cross-platform problem. There is one specification, one reference implementation on GitHub, and the same three letters appear in every vendor's datasheet. Write to the standard once and ship everywhere.

That is not how it goes. We build archiving software that targets Windows, macOS, and Linux from one Rust workspace, and the tape layer is the single place where the three operating systems refuse to converge. Not because of small API differences, but because the word LTFS names two genuinely different architectures depending on which machine you are standing at.

This post is what actually differs, written from the platform backends rather than from the specification. It includes a macOS build defect that is not documented anywhere we could find, and which will cost you a day the first time you meet it.

One name, two architectures

On Linux and macOS, LTFS is a userspace filesystem. You install a binary called ltfs, it links against FUSE, and you mount a cartridge at a path the same way you would mount anything else. The tape becomes a directory. Everything downstream is ordinary file I/O.

On Windows, LTFS is a command suite plus a filesystem driver. IBM ships a family of executables, and you drive the tape by calling them in sequence:

LtfsCmdDrives.exe      enumerate attached drives
LtfsCmdAssign.exe      bind a drive to a letter
LtfsCmdLoad.exe        load the cartridge
LtfsCmdFormat.exe      write a fresh LTFS volume
LtfsCmdEject.exe       take the cartridge offline
LtfsCmdUnassign.exe    release the letter

The cartridge does not appear at a path you chose. It appears at a drive letter, and the thing you check to confirm you are looking at tape rather than at somebody's USB stick is the volume's reported filesystem name.

That difference propagates through everything. A mount on Linux is a path that either has a marker in it or does not. A mount on Windows is a letter that may or may not currently be bound to what you think it is bound to. Any abstraction over both has to be honest that it is hiding two different failure modes, not one.

The tool inventory, side by side

Here is what each backend actually shells out to. This is the clearest single picture of the problem:

Windows   LtfsCmdDrives / Assign / Unassign / Load / Eject / Format
Linux     ltfs, mkltfs, umount, fusermount, mt, df
macOS     ltfs, mkltfs, umount, mount, df

Read the third line twice. macOS is missing two tools that Linux has, and one of them is load bearing.

macOS has no way to eject a tape

On Linux, taking a cartridge offline is mt -f /dev/nst0 offline. mt comes from mt-st, it is packaged everywhere, and it has been the answer for decades.

macOS ships no tape control tool at all. There is no mt, and no system equivalent. So the eject verb, which is trivial on the other two platforms, has no implementation available to it.

What you do about that is a design decision rather than a coding one. Our answer was to make the failure honest and typed rather than silent: unmount first, which flushes the index to tape and leaves the cartridge complete and readable on another machine, then return a specific "eject unsupported" result telling the operator the cartridge is safe to remove with the drive's own button. The caller that matters here is a spanning writer whose operator is already standing at the drive with the next cartridge in hand, so it needs to tell "I could not eject" apart from "something went wrong" without matching on error strings.

There is a second route. The FUSE layer can be asked at mount time to eject on unmount, using the IOKit backend and no external tool. But it is a mount-time option, which means committing at mount to whether the session ends in an eject. That turns eject from a verb into a mode, and it is a change to the whole spanning flow rather than a one-line swap. Worth knowing it exists before you design around its absence.

FUSE cleanup differs too

When an ltfs mount hangs and you kill it, you are left with a stale FUSE endpoint that will refuse the next mount attempt. Clearing it is not the same command on both unix platforms:

Linux    fusermount -u <mountpoint>
macOS    umount <mountpoint>     (or diskutil unmount force)

macOS has no fusermount. If your recovery path assumes it, your retry works in development and wedges on a Mac, and the symptom is a mount failure that looks like a hardware problem.

Timeouts are the other thing everyone gets wrong

Tape operations do not run on human timescales, and a single global timeout will either kill legitimate work or hide a wedged controller for hours.

Formatting a fresh LTO-9 cartridge is the extreme case. The first load of a new cartridge can take anywhere from forty minutes to two and a half hours before the drive will even talk to you, and the format itself runs long after that. A mount-class timeout of a few minutes will reliably kill a perfectly healthy format.

So the timeout has to be a property of the operation class, not of the session. The implementation here settles at three floors, applied identically on all three platforms:

probe             30 seconds     is the drive there
index flush        1 hour        unmount, writing the index to tape
long operations    4 hours       format, and first-load waits

The floor matters more than the number. A caller can configure a longer timeout, but it cannot configure one shorter than the class floor, because a caller that shortens a format timeout is not expressing a preference, it is expressing a misunderstanding.

Two flavors of Linux, and they are not the two you expect

Ask which Linux distributions a desktop application supports and you expect an answer like "Ubuntu and Fedora". For a tape tool the meaningful split is not by distribution brand. It is packaged versus portable, and each carries a different set of problems.

The .deb: integrated, and honest about it

A Debian package declares its dependencies, and the package manager enforces them. A well-built one names the GUI toolkit and the dialog helper explicitly, and CI fails the build if either declaration goes missing from the produced package. That is the useful property: on a Debian or Ubuntu machine, a missing dependency is a clear install-time error naming the package, not a mystery crash on first launch.

The cost is a hard glibc floor. Linux binaries get built on Ubuntu 22.04 rather than on whatever "latest" currently means, because a binary linked against a newer glibc simply will not start on an older machine, and archive operators run old machines on purpose. Building on latest is the single easiest way to ship something that works on every developer's laptop and on none of the customer's servers.

The AppImage: portable, and it moves the problem

An AppImage carries its own userland and runs on distributions you never tested. That is genuinely useful for a tool whose users are on RHEL derivatives, Arch, or something a facility built itself in 2019 and will not touch.

What it does not carry is the kernel side. FUSE is a kernel feature, the tape device nodes are the host's, and the permissions on them are the host's. So an AppImage moves the dependency problem from install time to run time, and converts a clear package-manager error into a runtime failure inside the sandbox. It broadens reach and narrows diagnosability at the same time. Ship both, and know which one a given support conversation is about before you start debugging.

Building LTFS on macOS, which is where the day goes

This section is the reason we wrote the post. Building the reference LTFS implementation on Apple Silicon is not hard in the sense of being intellectually difficult. It is hard in the sense that six separate things go wrong in sequence, each with an error message pointing somewhere other than the cause.

In order, on macOS 15 on an M2:

macFUSE needs Reduced Security. It is a third-party kernel extension, and Apple Silicon ships with Full Security, under which the kext cannot load at all. Changing that requires booting into Recovery. It cannot be done over SSH, it cannot be scripted, and it cannot be done for a customer. Budget a person at the keyboard.
autogen.sh skips libtool silently. No error, no warning, and configure then fails later for a reason that has nothing to do with libtool.
icu4c is keg-only and versioned. Plain icu4c now returns a 404 from the Homebrew API. You need the versioned formula, and because it is keg-only it never lands on PATH, so configure dies claiming a tool called genrb was not found. Nothing in that message mentions ICU.
configure demands a uuid pkg-config file. It asks for uuid >= 1.6. macOS does have UUID, in libSystem, with no .pc file anywhere, so the check fails on a platform that has the functionality. Installing ossp-uuid satisfies the check.
SNMP is on by default and cannot build. configure calls net-snmp-config --cflags, which is not there, and the resulting error is truncated in a way that makes it look like a FUSE problem. It is not. Pass --disable-snmp.
macFUSE's framework is not where the compiler looks. It lives under /Library/Filesystems/macfuse.fs/Contents/Frameworks/, which needs an explicit -F.

Work through all six and it compiles. Roughly thirty seconds on an M2. And then you run the one command that matters and it dies.

The link defect nobody documents

Enumerating drives is ltfs -o device_list. On a freshly built macOS LTFS with no drive attached, that segfaults. Exit 139, with the crash inside libsystem_malloc on what looks like recursive allocation.

Every instinct says hardware, or macFUSE, or the missing drive. None of those is it.

It is a link defect. Upstream's IOKit plugin leaves its IOKit and CoreFoundation symbols to dynamic lookup, and nothing in the build actually links the frameworks that define them. The binary links clean, loads, and then dies the moment it reaches for a symbol that was never bound. Adding the frameworks fixes it:

export LDFLAGS="-L$ICU/lib -framework IOKit -framework CoreFoundation"

With that in place, ltfs -o device_list behaves correctly: it exits 1 and prints an empty device list, which is the right answer when no drive is attached. A clean negative result instead of a crash.

The trap underneath the trap is what a caller does with the crash. Any code that treats "device_list failed" as "no drives present" will render an empty drive picker and tell the user they have no tape hardware, when the truth is the enumerator died. A segfault and an empty list are not the same fact and should never collapse into the same UI state.

What this costs you, and what to do about it

Five things we would tell anyone starting a cross-platform tape project:

Do not abstract over Windows and unix too early. They are a command suite against a drive letter and a FUSE mount against a path. An interface that pretends otherwise hides the failure modes you most need to see.
Make timeouts a property of the operation class, with floors a caller cannot lower. Formatting is a four-hour operation on hardware that is behaving perfectly.
Let a platform say "I cannot do this" in a typed way. macOS cannot eject. That is better expressed as a specific result the caller can branch on than as a generic failure it has to parse a string out of.
Never let a crashed probe read as an empty result. Enumeration failing and enumeration returning nothing are different, and only one of them should be shown to a user as "no hardware found".
Fix a build recipe everywhere at once. A typical recipe lives in a runbook, a knowledge base article, and a CI workflow. A fix that lands in one of three is a fix that will be un-learned by whoever reads the other two.

The honest scope of this post

Everything above is about the software layer: build systems, tool inventories, mount semantics, timeout policy, and packaging. It is drawn from platform backends we wrote and a macOS build we did on a real machine.

It is not a hardware compatibility report, and we are not going to imply otherwise. Our current Rust implementation has not yet run against a physical tape drive on any operating system. Coverage today comes from a virtual drive and a simulated backend, and the first run on real hardware is a scheduled validation window covering all three platforms. The cross-platform differences described here are real and cost real time, but the claim stops precisely at the software boundary, because a tape tool that overstates what it has tested is the exact thing this industry does not need more of.

If you are building on tape across operating systems and want to compare notes, or you want the archiving software this came out of, Hiberden is where that work lives. Adjacent problems we take on: NAS infrastructure, data archiving, and custom development when the tool you need does not exist yet.

Originally published at agaveis.com.

Permission Filters and pgvector: Why Your Most Restricted Users Get the Fewest Answers

Agave Information Soultions, LLC — Sat, 26 Sep 2026 00:15:23 +0000

If you build AI search over documents that not everyone may read, you filter by permission. In PostgreSQL with pgvector, that usually means a WHERE clause on the caller's groups and an ORDER BY on vector distance. With no vector index, that query is exact and always complete. Add an HNSW index for speed, and something quiet happens: the people allowed to read the least start getting the fewest answers. Nothing errors. Nothing leaks, because the filter still holds. The list just comes back short.

Here is the mechanism, measured, and the two fixes. Everything below ran on PostgreSQL 17.10 with pgvector 0.8.4, and the script is at the end.

The query

Chunks of documents carry embeddings. Documents carry the groups allowed to read them. A search for one caller looks like this:

SELECT c.id
FROM chunk c
JOIN document d ON d.id = c.document_id
WHERE d.allowed_principals && ARRAY['group:hr']
ORDER BY c.embedding <=> $1
LIMIT 20;

The && operator is array overlap: a document qualifies if the caller holds any one of its groups. With no vector index, pgvector does exact nearest neighbor search, which its documentation describes as providing perfect recall. This query returns 20 rows whenever 20 readable chunks exist.

What an HNSW index changes

pgvector's README says it plainly: with approximate indexes, filtering is applied after the index is scanned. The scan produces a fixed list of candidates, sized by hnsw.ef_search, which is 40 by default, and the WHERE clause runs on that list. If the caller may read 10 percent of the rows, about 4 of the 40 survive.

We measured it. 100,000 chunks across 10,000 documents: one document in ten readable by group:hr, the other nine by group:staff. Random 64-dimension vectors, an HNSW index with default settings, the same 40 random queries for each caller, LIMIT 20:

Caller	Can read	Rows returned, of 20
group:staff	90 percent	20 on every query
group:hr	10 percent	median 4, as few as 1, never more than 8

Same query, same index, same data. The only difference is who asked.

Why permissions make it worse

A category filter is the same for everyone, so a short result shows up the first time anyone tests it. A permission filter depends on who is asking. Developers test with broad accounts, and a broad account gets full results. The short list only appears for the restricted user: the new hire, the contractor, the three-person team with its own folder.

And a short list does not look like an error. An AI assistant handed 4 passages instead of 20 answers from the 4. If the passage that mattered was number 5, it can tell a new hire there is no policy on something that has one, and the new hire has no reason to doubt it.

Fix one: iterative index scans

pgvector 0.8.0 added iterative index scans. When the filter leaves too few rows, the scan keeps going through the index instead of stopping at the candidate list:

BEGIN;
SET LOCAL hnsw.iterative_scan = strict_order;
-- the search query
COMMIT;

SET LOCAL ends with the transaction that runs the search, so a pooled connection cannot carry the setting anywhere else. Outside a transaction it does nothing. The other mode, relaxed_order, can return rows slightly out of distance order in exchange for better recall, and pgvector's README shows a materialized CTE that puts them back in order.

On the same 40 queries, the HR caller got 20 of 20 every time, in either mode. It costs time, because the scan now looks at more of the index. The median query went from 1.7 ms to 27.8 ms with strict_order, and to 18.1 ms with relaxed_order, on one desktop. For comparison, exact search, with the index turned off, took 47.5 ms.

Fix two: give narrow permissions an exact path

Iterative scans have limits. A scan stops after visiting about hnsw.max_scan_tuples entries, 20,000 by default, or when it reaches a memory cap of work_mem times hnsw.scan_mem_multiplier. A caller who may read almost nothing can hit those limits before 20 matches turn up.

So we added a third group, group:legal, readable on one document in a thousand: 0.1 percent of the chunks. Left alone, the planner did the right thing. It used the GIN index on allowed_principals to find the ten readable documents, joined their chunks, and sorted them by exact distance: 20 of 20 on every query, at a median of 33 ms.

Then we forced the query through the HNSW index, to see what happens when the planner picks it. With iterative scans on and the default limits, the median was 16 of 20, and the worst query returned 10. Raising hnsw.max_scan_tuples to 100,000 alone changed nothing. Raising hnsw.scan_mem_multiplier to 4 alone changed nothing. Raising both got 20 of 20, at a median of 732 ms: about 22 times slower than the exact plan the planner had chosen on its own.

So the second fix is not a setting. It is an ordinary index on the permission column. pgvector's README calls an index on the filter column "a good place to start" and notes that exact indexes work well for conditions that match a low percentage of rows. With one in place, the planner has a fast, complete path for your narrowest callers. Confirm it with EXPLAIN as your most restricted real user, not as yourself.

The test that catches it

Every number in this post came from asking as a specific group. That is the test that finds this problem, and a test run as an administrator never will. If your retrieval has a test set, give it cases that run as a restricted group and expect a specific document back. When an index change starves that group, the case fails before a user notices.

The retrieval engine we build for clients works this way. Its vector search is exact, with no approximate index, until a corpus is large enough to need one and the recall cost has been measured. Its test sets run questions as a named group, and a case fails when an expected document is missing from the top five results or a forbidden one appears in them.

Measure your own documents

Random vectors keep the arithmetic clean: what a caller loses tracks what fraction of the rows they can read. Real embeddings cluster, and permissions often follow topic, so an HR user asking an HR question may lose less, and the same user asking about something outside their documents may lose more. The timings come from one desktop and will differ on yours. Both are reasons to measure your own documents, as your own restricted users.

If you are putting AI in front of documents that not everyone may read, this is the kind of detail we work through with organizations that keep their data in-house: private AI.

Run it yourself

Needs PostgreSQL 17 with pgvector 0.8.0 or later; run it with psql. Each count comes from one random query, so yours will differ from the medians above. The index build may print a notice that it no longer fits in maintenance_work_mem; that only makes the build slower.

-- PostgreSQL 17 with pgvector 0.8.0 or later. Run with psql.
CREATE EXTENSION IF NOT EXISTS vector;

-- 10,000 documents: one in ten readable by group:hr, one in a thousand also by group:legal.
CREATE TABLE document (id int PRIMARY KEY, allowed_principals text[] NOT NULL);
INSERT INTO document
SELECT g, CASE WHEN g % 1000 = 0 THEN ARRAY['group:hr', 'group:legal']
               WHEN g % 10 = 0 THEN ARRAY['group:hr']
               ELSE ARRAY['group:staff'] END
FROM generate_series(1, 10000) g;
CREATE INDEX ON document USING GIN (allowed_principals);

-- 100,000 chunks, ten per document, random 64-dimension embeddings.
CREATE TABLE chunk (id int PRIMARY KEY, document_id int NOT NULL REFERENCES document(id), embedding vector(64) NOT NULL);
INSERT INTO chunk
SELECT g, (g % 10000) + 1, (SELECT array_agg(random() + 0 * g + 0 * i) FROM generate_series(1, 64) i)::vector(64)
FROM generate_series(1, 100000) g;
ANALYZE document;
ANALYZE chunk;

-- One random query vector.
SELECT (SELECT array_agg(random()) FROM generate_series(1, 64))::vector(64)::text AS qv \gset

\echo 'No vector index (exact search). Rows returned of 20, for group:hr:'
SELECT count(*) FROM (SELECT c.id FROM chunk c JOIN document d ON d.id = c.document_id
  WHERE d.allowed_principals && ARRAY['group:hr'] ORDER BY c.embedding <=> :'qv' LIMIT 20) t;

-- Single process build, so it fits a small container's shared memory.
SET max_parallel_maintenance_workers = 0;
CREATE INDEX ON chunk USING hnsw (embedding vector_cosine_ops);

\echo 'HNSW index, defaults. group:staff (reads 90 percent), then group:hr (reads 10 percent):'
SELECT count(*) FROM (SELECT c.id FROM chunk c JOIN document d ON d.id = c.document_id
  WHERE d.allowed_principals && ARRAY['group:staff'] ORDER BY c.embedding <=> :'qv' LIMIT 20) t;
SELECT count(*) FROM (SELECT c.id FROM chunk c JOIN document d ON d.id = c.document_id
  WHERE d.allowed_principals && ARRAY['group:hr'] ORDER BY c.embedding <=> :'qv' LIMIT 20) t;

\echo 'Iterative scans on. group:hr again:'
SET hnsw.iterative_scan = strict_order;
SELECT count(*) FROM (SELECT c.id FROM chunk c JOIN document d ON d.id = c.document_id
  WHERE d.allowed_principals && ARRAY['group:hr'] ORDER BY c.embedding <=> :'qv' LIMIT 20) t;

Agave Information Solutions builds on-premises AI systems, data architecture, and custom software out of Scottsdale, Arizona. If your AI search has to respect who may read what, get in touch.

Originally published at agaveis.com.

EF6 to EF Core: 5 query translations that quietly break in production

Agave Information Soultions, LLC — Wed, 03 Jun 2026 05:08:00 +0000

Most teams plan an EF6 to EF Core migration as "swap the package, fix some namespaces, ship it." For straightforward CRUD codebases that's mostly true. For anything with real query complexity, you will hit a handful of translation differences that the upgrade tooling does not warn you about. The queries compile fine, the unit tests pass against an in-memory provider, and then production explodes the first time someone hits the page that runs the report.

These are the five that bit us hardest on real migrations. None of them are obvious from reading the official "what's new" page, and a couple of them are silent regressions in behavior rather than visible errors.

1. Client-side evaluation does not silently rescue you anymore

EF Core 1.x and 2.x had a feature called automatic client evaluation. If part of your LINQ query could not be translated to SQL, EF Core would happily run that part in memory after fetching the rest of the data from the database. It logged a warning, but the query worked.

EF Core 3.0 changed the default to throw instead of silently falling back. Code that ran fine for two years suddenly throws InvalidOperationException: The LINQ expression could not be translated.

The worst version of this is a query like:

csharp
var customers = ctx.Customers
    .Where(c => c.DisplayName.NormalizeForSearch() == query.NormalizeForSearch())
    .ToList();

NormalizeForSearch() is a custom C# extension method. EF6 happily fetched every row, ran the method on each one in memory, and filtered. The query was a full table scan and stayed quietly slow for years. After the EF Core upgrade it throws at runtime.

The remediation is not "add .AsEnumerable() before the Where" even though that "works." That just preserves the original full-table-scan behavior with the original O(n) memory cost. The right fix is to push the normalization into the query itself using EF.Functions.Like or a normalized column maintained at write time. Either of those produces real SQL with real index usage.

GroupBy translates much less aggressively

EF6 translated almost any GroupBy you threw at it. EF Core is far more conservative — for years, it required your GroupBy to fit a narrow shape (group key + aggregate functions) and would throw if you tried to project the group elements directly.

This works in both:


var totals = ctx.Orders
    .GroupBy(o => o.CustomerId)
    .Select(g => new { CustomerId = g.Key, Total = g.Sum(o => o.Amount) })
    .ToList();

This worked in EF6 but did not in EF Core 3-5, and only partially works in EF Core 7+:


var groups = ctx.Orders
    .GroupBy(o => o.CustomerId)
    .Select(g => new { CustomerId = g.Key, Orders = g.ToList() })
    .ToList();

The second one is asking for "give me each group along with the list of items in that gr# LINQ but hard to express in standard SQL. EF Core 7 added partial support via SQL Server window functions; older versions throw.

For migration purposes, audit every GroupBy in the codebase before you flip the switch. The shape that breaks most often is grouping followed by projecting the actual group elements rather than aggregations.

Restructuring usually means doing the grouping client-side after a flat query, or rewrifor aggregates.

String comparisons changed collation behavior

EF6 ran string comparisons through SQL Server with the database's default collation, whie accent-insensitive on most installations. EF Core leaves the comparison semantics up tothe provider, and the defaults are different across providers.

Where(c => c.Email == "foo@bar.com")

against a Latin1_General_CI_AS database returns rows where Email equals FOO@BAR.COM because the SQL collation is case-insensitive. The same query against PostgreSQL
returns zero rows because PostgreSQL is case-sensitive by default. If your application wver provision a PostgreSQL instance, you will spend a confused afternoon.

EF Core 5+ exposes EF.Functions.Collate for explicit collation control, and EF.Functionsttern matching that works the same across providers. Use those instead of relying on thedatabase default if your code might run against more than one provider.

A subtler form: string.Equals(c.Name, query, StringComparison.OrdinalIgnoreCase) translated to ordinary = in EF6 (the comparison enum was just ignored), but in EF Core it can throw "could not be translated" depending on version. Same shape of bug as #1.

DateTime.Now lands client-side in confusing ways

DateTime.Now evaluated in the context of a LINQ query does not have a consistent transla and providers.

var recent = ctx.Orders
    .Where(o => o.CreatedAt > DateTime.Now.AddDays(-7))
    .ToList();

EF6 translated this to WHERE CreatedAt > DATEADD(day, -7, GETDATE()) on SQL Server, whicery time. EF Core sometimes evaluates DateTime.Now once on the client at query construction time, then sends a constant to SQL. If your app builds the query at 12:00 and executes it at 12:30, EF6 uses 12:30, EF Core often uses 12:00.

For most queries this 30-minute drift does not matter. For session expiry, rate limiting, scheduled job pickups, or audit windows, it matters a lot.

The deterministic pattern: capture the time explicitly in a local variable and pass it as a parameter. Or use EF.Functions.GetUtcDate() to force server-side evaluation. Or move the time calculation into a
database function and call it explicitly. Any of those produces predictable behavior.

Lazy loading is off by default and the fix is not what you think

EF6 enabled lazy loading by default if your navigation properties were marked virtual. Ty fetches it.

EF Core does not enable lazy loading by default. The migration path that the documentatirosoft.EntityFrameworkCore.Proxies and call UseLazyLoadingProxies() on the options. That technically restores lazy loading, but it makes every navigation access into a separate database round trip — exactly the N+1 problem the rest of the EF Core team is trying to push you away from.

The better migration path is to audit where your code was relying on lazy loading and convert those sites to either explicit Include calls or projection into DTOs. It is more work, but the resulting queries
are faster, the N+1 patterns become visible at code review time instead of in production table.

If you do install the lazy loading proxies as a "get the migration unblocked" bridge, to put a real plan in place to remove it. The longer you ship on lazy proxies, the more code grows to depend on the implicit fetch behavior and the harder removal gets.

The pattern across all five

Every one of these breakages comes from the same root: EF6 was very permissive about hiding the gap between LINQ semantics and SQL semantics, and EF Core is less so. EF Core fails loudly (or silently
differently) in places where EF6 would have happily run something inefficient.

That sounds like a regression. It is actually the right tradeoff. The EF6 queries that "l table scans, N+1 patterns, or subtly wrong results that nobody noticed because the database was small enough that performance didn't expose them. EF Core's stricter posture forces those decisions to surface during migration instead of in production a year later when the data has grown.

The practical migration playbook:

Turn on LogTo for Microsoft.EntityFrameworkCore.Query.QueryCompilationStarting events during integration tests so you can see what each query actually translates to.
Run a load test against representative data volume before declaring the migration don the dev database often fail on production volume after the EF Core change.
Audit every GroupBy, every Where with custom methods, every navigation property access, and every DateTime.Now reference.
Treat Microsoft.EntityFrameworkCore.Proxies as a bridge, not a destination.

The "swap the package and ship it" plan turns into a multi-week project for non-trivial modernization is one of the things we work on most often at Agave Information Solutions(https://agaveis.com/dotnet-modernization). If you are heading into one of these migrations and want a sanity check on your pattern audit, the .NET Modernization (https://agaveis.com/dotnet-modernization)
page has more on how we approach it.

Agave Information Solutions

Agave Information Soultions, LLC — Wed, 03 Jun 2026 04:58:25 +0000

30 years on the Microsoft stack. Started writing .NET when the framework was still in beta in 2001.

Founded Tempest Telecom in 1996. We built the first Internet roaming service to provide access on all 7 continents, and a unified AAA platform that combined voice, dialup, WiFi, and satellite into one experience. The 2+ petabyte video platform that came out of that work is where I learned that query plans behave non-linearly at scale, and why backup strategy is architecture.

Founded Agave Information Solutions in Scottsdale in 2008. Today we modernize legacy .NET Framework applications onto modern .NET 10/9/8 with AI-ready architecture, design databases that need to handle real volume, and ship the kind of senior-engineering work most boutique shops outsource.

I write here about .NET migrations, EF Core gotchas, SQL Server query plans, AI integration into legacy stacks, and the lessons from running production systems at scale for a long time.

Find me elsewhere: