DEV Community: Harper

Build a Conversational AI Agent on Harper in 5 Minutes

Stephen Goldberg — Fri, 03 Apr 2026 17:21:07 +0000

Building AI agents usually means stitching together a database, a vector store, a caching layer, an API server, and a deployment pipeline. Five services, five sets of credentials, and a weekend gone.

We built an agent that does all of that on Harper — database, vector search, semantic cache, API, and deployment in one runtime. The full source is open. Here's how to get it running.

What You Get

A conversational chat agent powered by Claude with:

Semantic memory — every message is embedded and stored. Ask a question from three conversations ago and it remembers, powered by Harper's built-in HNSW vector index.
Semantic caching — ask the same question twice (or a rephrased version) and it answers instantly from Harper at $0.00 LLM cost. Over time, a popular agent builds a dense cache that handles most queries for free.
Web search — Anthropic's built-in server-side search, no extra API key.
Local embeddings — bge-small-en-v1.5 runs locally via llama.cpp inside Harper. No embedding API, no extra cost.
A chat UI — served directly from Harper at /Chat with per-response cost, latency, and token tracking.
A live savings counter — tracks how much money the cache has saved across all conversations.

Live demo →

Get It Running

Prerequisites: Node.js 22+ and an Anthropic API key.

git clone https://github.com/stephengoldberg/agent-example-harper.git
cd agent-example-harper
npm install

Add your key:

echo "ANTHROPIC_API_KEY=sk-ant-your-key-here" > .env

Start it:

npm run dev

Open http://localhost:9926/Chat. That's it.

On first run, the embedding model (~24 MB) downloads automatically. Every run after that starts in seconds.

See the Cache in Action

Ask a question — say, "Who is the president of the United States?" You'll see the metadata strip under the response:

LATENCY 5.2s · TOKENS 2,400 in / 150 out · COST $0.0098 · WEB 1 search

Now ask it again, or rephrase it — "who's the current US president?" The response comes back instantly:

LATENCY 0.03s · CACHE Served from Harper semantic cache — $0.00 · saved $0.0098

Same answer, zero LLM cost, sub-50ms. Harper's HNSW vector index found that the new question is semantically equivalent (cosine similarity ≥ 0.88) and served the cached response without calling Claude at all.

The savings panel in the sidebar tracks this across every conversation.

How It Works

Everything runs inside Harper — a unified runtime that combines a database, vector index, cache, and API server in one process:

Harper Agent — a JavaScript Resource class that handles POST /Agent. It's your agent logic, running in-process. No Express, no API framework.
Vector Store — @indexed(type: "HNSW", distance: "cosine") in the schema. One line gives you a full vector index. Harper searches it natively with conditions and comparator: 'lt' — no in-memory math, no full scans.
Semantic Cache — two layers. First: exact text match (free, instant). Second: Harper's HNSW index finds semantically similar past questions within a cosine distance threshold. Both run inside the database, not in your application code.
Local SLM — bge-small-en-v1.5 runs via llama.cpp in the same Node.js process. Embeddings cost nothing and never leave the machine.
Claude + Web Search — the only external call. Only happens on cache misses. Anthropic's built-in server-side search means no Google API key.

The entire schema is 24 lines of GraphQL. The agent logic is ~200 lines of JavaScript. There's no ORM, no database driver, no vector database client, no caching library, and no route definitions.

Deploy to Harper Fabric

npm run deploy

One command. Your agent is now running globally on Harper Fabric. No Docker, no Kubernetes, no CI/CD pipeline.

Why Harper

The traditional stack for an AI agent with memory and caching looks like: Postgres + Pinecone + Redis + Express + Docker + a deployment pipeline. That's six services to configure, connect, and maintain.

With Harper, it's one runtime:

What	Traditional	Harper
Database	Postgres/MongoDB	Built in
Vector search	Pinecone/Weaviate	Built in (`@indexed(type: "HNSW")`)
Semantic cache	Redis + custom logic	Built in (native HNSW conditions search)
API server	Express/Fastify	Auto-generated from schema
Embeddings	OpenAI/Voyage API ($)	Local SLM, $0
Deployment	Docker + K8s	`harperdb deploy .`

The cache is the real story at scale. Every question that hits the cache saves you the full cost of a Claude API call — tokens, web searches, and all. The savings compound: the more your agent is used, the less it costs per query.

Clone the repo, add your Anthropic key, and try it.

I Stopped Buying SaaS for RevOps. I Built What I Needed on Harper Instead.

Jacob Cohen — Mon, 02 Mar 2026 21:22:58 +0000

TL;DR: A non-engineer RevOps leader, replaced a manual, error-prone reporting process by using OpenAI Codex to build and deploy a fully automated, production-grade Growth Weekly Digest application on the Harper platform.

I run commercial operations at a startup. That means Salesforce administration, deal desk, forecasting, pipeline management, contract workflows, and a dozen other things that keep our revenue engine running. I am not a software engineer. My day job is operations.

Last month, I built and deployed a production internal application that pulls live data from Salesforce and Slack, generates AI-assisted commentary, routes it through a multi-role approval workflow, and publishes a weekly growth digest for our entire company. It has eight database tables, five user roles, three external integrations, a scheduler with leader-node election, retry logic, CI/CD with automated testing, and it runs on a scaled Harper Fabric deployment. I built it in a few days using OpenAI Codex as my coding partner. I didn't write a single line of code myself.

What does "built" mean when you don't write code? It means I spent a few days having a conversation. I described what I needed, Codex planned and executed, and we iterated together until it was right. I'd explain a workflow, review its approach, push back when something didn't fit, ask it what common RevOps patterns look like, and refine until I was satisfied with the plan. Then it wrote the code. I did the thinking. It did the typing. That's the entire method, and it's why this post isn't a tutorial. There's no code to walk through. There's a conversation that produced a production application.

This post is about how I did it, why I did it on Harper instead of buying another SaaS tool, and why I think this changes the math for every business operator who's currently stuck choosing between manual processes and overpriced software subscriptions.

The RevOps Problem Nobody Talks About

If you work in RevOps, you know this situation. You have a process that's valuable, but tedious. It requires pulling data from countless, ever-changing systems, assembling it into something digestible, adding your own analysis, and distributing it. It takes 30 minutes to an hour. It's not hard, it's just manual enough that it consistently loses priority against everything else you're doing.

For me, that process was our Growth Weekly Digest.

Every Friday for the better part of a year, I produced a report that tracked our weekly sales activity: closed won and closed lost deals, new pipeline created, pipeline by rep and channel, BDR metrics, and narrative commentary about what happened and what to watch. The audience started as our executive team, expanded to team leads, and eventually the whole company. Our product management team relies on it to keep a pulse on sales activity, which is a critical input for product prioritization and roadmap decisions. People across the company found it valuable because it gave them visibility into the growth engine without having to dig through Salesforce or chase down updates.

The process to create it was entirely manual.

I'd start by pulling up Salesforce reports and dashboards. I had some dashboards on Salesforce's subscribe feature to email me at 9AM as a trigger to get started. Then I'd extract the numbers and copy them into a Google Doc: deal counts, revenue, pipeline breakdowns; you know the drill. That copy-and-paste step alone took ten minutes minimum and was highly error-prone. One wrong cell and the whole week's narrative would be based on bad data. This happened more often than I'd like to admit. The whole process violated one of my favorite rules: never introduce human error when you don't have to. I knew that copying and pasting numbers between systems was a terrible workflow, but I did it anyway because there wasn't a better option.

Then came the commentary. The numbers alone don't tell the story. I needed to explain which deals advanced, which stalled, what happened in customer conversations, and what risks were emerging. To write that, I'd piece together what I remembered from the week, scan Slack conversations, check meeting notes, and sometimes use Slack AI to help me summarize activity. The whole thing took 30 to 45 minutes, depending on how much digging was required.

It doesn't sound like much. Here's the real cost: I stopped doing it.

When the year turned over, and we were finalizing 2026 quotas, I paused the digest. We were well into Q1 before I confronted the fact that I didn't want to go back to the manual grind, even though people found the report valuable. At a startup, everyone wears multiple hats. I do far more than RevOps in a given week. The digest was always the first thing to get deprioritized. (Whatever you do, don't tell my boss, even though she's probably already forwarded this post to everyone she knows.)

So for weeks, the report just didn't get written.

Why I Didn't Buy Another SaaS Tool

The obvious move would have been to buy software for this. Here's why I didn't.

Salesforce dashboards aren't robust enough for this kind of consolidated weekly reporting, and more importantly, they require paid Salesforce licenses for every viewer. If you've ever priced Salesforce seats, you know the pain. The per-user licensing costs add up fast, and we already carry more licenses than we'd like. We are a cost-conscious startup and we absolutely do not get our money's worth out of every seat we hold. There was no world in which we were going to buy CRM licenses for the entire company just so people could read a weekly summary.

Databox was something we used for other executive-level dashboards that provided similar, though not identical, value. It worked for a while, but we ran into issues with their Salesforce connector that broke our data pipeline, and even when it was working, it didn't do everything we needed it to. That was the end of Databox for us.

Tableau was so comically expensive for our use cases that it wasn't even worth a serious conversation, even though we could have used it for other reporting needs too.

This is the pattern every RevOps person knows. The SaaS options are either too limited, too expensive, too rigid, or some combination of all three. And every tool you add is another vendor, another login, another integration to maintain, another line item for Finance to question. The thing you actually need is always slightly different from what the tool provides, so you end up building workarounds on top of the tool, at which point you're doing custom work anyway, just on someone else's platform.

I realized I could spend a few days building exactly what I needed, with real integrations, real auth, real deployment, on Harper. And it would cost less ongoing effort than re-establishing the manual process I'd abandoned. No point solution. No per-seat reporting license. No compromises on what the report should contain or who can see it. Harper is a platform, not a tool. The weekly digest is one application on it. The next one I build runs on the same infrastructure, the same deployment model, the same data layer. That's a fundamentally different cost equation than buying a new SaaS product for every internal need.

What I Built

The Growth Weekly Digest is now a Harper application. It runs on Harper Fabric with its own database, API, web interface, and scheduled jobs, all in one deployable unit.

Note: All screenshots in this post use mock data. Our actual digest contains proprietary sales information.

Here's what it does each week:

Data collection. On a scheduled trigger (Friday mornings), the application queries Salesforce via JWT-authenticated API calls. It pulls Opportunity data: deal names, stages, forecast categories, amounts, ARR fields, close dates, lead sources, and rep assignments. It also queries OpportunityPartner objects to classify deals by channel (partner-sourced vs. direct). Separately, it reads recent activity from configured Slack channels to capture qualitative signals about deal movement and team conversations.

Metrics computation. The raw data is processed into structured, frozen metrics: closed won and closed lost counts and revenue broken out by channel, new pipeline created, pipeline by rep, and week-over-week comparisons. Each digest is a snapshot of that week's state.

Commentary generation. Metrics and Slack signals are sent to OpenAI's Responses API with a system prompt that enforces structured JSON output. The LLM produces commentary organized into wins, risks, and action items, each with citation links back to the original Slack messages so we can jump directly into a conversation if something needs follow-up. The AI commentary is never published automatically. It's a draft starting point.

Human review and approval. This is the part that matters. The generated digest enters a review workflow. I review the data and commentary for accuracy. Kelli, our VP of Sales, does the same. Either of us can edit the AI-generated commentary before approving. The system tracks each approver independently. Both must approve before the digest can be published. This is a real workflow with real gates, not a rubber stamp.

Publication and notification. Once both approvals are in, the digest is published to a company Slack channel. Published digests are also accessible through the web UI, authenticated via Google OAuth restricted to Harper email addresses.

Access controls. Five roles: company_reader, growth_editor, revops_approver, growth_manager, and admin. Readers only see published digests. Unpublished runs return a 404 for readers. They don't exist until they're ready. Editors and approvers see drafts and can act on them. Role assignments are stored in a Harper table and enforced server-side on every request.

The Architecture: One Runtime, Not a Rube Goldberg Machine

This is where Harper's value as a platform becomes concrete.

The entire application is a single Harper application. The database, the API, the web UI, the scheduler: they all ship together and deploy together. Here's what's inside:

Eight tables define the data model: DigestRun, DigestMetrics, DigestCommentary, DigestNote, DigestApproval, DigestPublication, DigestRoleMapping, and DigestCommentaryRevision. These are declared in schema files. When the application deploys, the tables exist. No external database. No migration scripts. No connection strings. No ORM. Related tables are linked via indexed fields like runId, so queries across the data model are fast and consistent. The application code interacts with these tables through Harper's Resource class, a unified API where your database tables, business logic, and HTTP endpoints are all part of the same runtime. There's no separate database driver, no connection pool, no serialization layer between your code and your data. This matters for AI-generated code because the agent doesn't have to wire together disconnected services. It writes application logic against a single integrated platform, and that code is production-ready and performant immediately.

A TypeScript service layer handles the digest lifecycle: generation, metrics computation, commentary requests, approval state derivation, publish-blocker validation, and data quality checks. Business rules live here. For example, you cannot generate a new digest for a week that already has a published one.

Three external connectors: Salesforce (JWT OAuth with RS256 signing, SOQL queries against Opportunity and OpportunityPartner objects), Slack (Web API with channel discovery, bounded concurrency, and thread fetching), and OpenAI (Responses API with structured JSON schema enforcement, citation sanitization, and configurable retry budgets).

A scheduler with two jobs: a weekly cron trigger for Friday morning digest generation, and a periodic retry loop for any LLM commentary requests that failed due to timeouts or rate limits. The scheduler includes leader-node gating. In a replicated Fabric deployment across multiple nodes, only one instance runs the scheduler. No duplicate digests.

Server-rendered HTML for the web interface. A dashboard listing all digest runs, detail views for each, and action endpoints for generate, approve, publish, and edit operations. No frontend framework. No separate build step.

A CI/CD pipeline built entirely by Codex using GitHub Actions. It runs strict typechecking, tests, a regex-based secrets scan, and a guard against raw environment variable usage in code. Every time we merge to main, it automatically deploys to our production Harper Fabric cluster and runs a post-deploy health check against the live API. I didn't configure any of this. Codex built the whole pipeline, and now I don't have to think about DevOps either.

Here's how the data flows:

Now consider what this would look like if I'd let an AI coding tool pick a typical stack:

A Node.js or Python backend framework. A managed PostgreSQL instance. An ORM. A frontend framework with its own build pipeline. A reverse proxy. A hosting provider for the API. A different hosting solution for the frontend. A managed cron service or sidecar process for scheduled jobs. A secrets manager. Probably Redis for session management. Maybe a container orchestration layer to tie it all together. Each one of those is a configuration surface, a potential point of failure, and something I'd need to understand and maintain.

With Harper, the application is the whole thing. Database, server, scheduler, auth foundation, deployment target. It's one runtime. That's not a marketing claim. That's what made it possible for me to build this in a few days.

How I Built It: Vibe Coding for Business Operators

I used OpenAI Codex as my implementation partner. Let me be precise about what that means.

I did: Requirements definition. System design. Architecture decisions. Data model design. Integration specification: which Salesforce objects to query, which Slack channels to read, what the approval workflow should be, what the role model should look like. Acceptance testing. Best-practice review using Harper Agent.

Codex did: All code. Every line of TypeScript, every schema, every route handler, every connector, every CI workflow, every test.

I did not: Write code. Debug code. Configure deployment. Set up CI/CD.

The working pattern was iterative. I'd switch between Codex's planning mode and coding mode constantly. In planning mode, I'd describe what I wanted, sometimes high-level ("We need a Salesforce connector that uses JWT OAuth and pulls Opportunity and Partner data"), sometimes very specific ("Add a guard that prevents generating a new digest for a week that already has a published run. No new routes. No schema changes. Redirect with structured flash data for the dashboard."). Codex would implement it.

After each implementation pass, I'd feed the output to Harper Agent for best-practice review against Harper's application conventions. Harper Agent would flag issues: patterns that didn't align with how Harper resources should be structured, configuration that could be cleaner. I'd take that feedback and send Codex back in with specific corrections. Harper is currently building out Harper Agent to handle more of this end-to-end, but even today this feedback loop worked well.

This is the part of building software I've always done well: designing systems, defining requirements, making architecture decisions, evaluating tradeoffs. The part I don't want to do, and historically couldn't do without engineering support, is the implementation labor: the typing, the integration wiring, the debugging. Codex handled that. And because it was building against Harper's unified runtime, the output was deployable from day one. No glue code. No service orchestration. No separate infrastructure to configure.

Harper's npm create harper@latest scaffold was the starting point. That gives you a project structure with schema files, resource definitions, configuration, and importantly, a skills/ directory that grounds the AI agent in Harper's architecture patterns. Codex consumed those skills files as context, which meant it wasn't guessing at how to structure a Harper application. It had the conventions built into its working memory.

To give you a sense of what that first conversation looked like, here's a prompt similar to the one I used to kick off the project. I uploaded my existing Google Doc digest as a reference so Codex could see exactly what the output should look like, then described what I wanted to build:

I want to build a Harper application that replaces this manual Google Doc "Weekly Digest" report (attached). The app should pull data from Salesforce and Slack automatically, compute the same metrics I've been assembling by hand, and generate a weekly digest. Build it as a Harper Application (https://docs.harperdb.io/docs/developers/applications), ensuring you use the Harper Resource class (https://docs.harperdb.io/docs/reference/resources). Start with the create-harper template (https://www.npmjs.com/package/create-harper). The app needs role-based auth, where company-wide readers can only see published digests. An approval workflow requiring two approvers before publish. OpenAI integration for drafting commentary from the Slack and Salesforce data. Scheduled generation on Fridays plus a manual trigger. A dashboard, digest detail view, and ops page. Strong environment validation, secure cookies, structured logs. CI pipeline with secrets scanning, strict typechecking, and tests. Once we get a version of the app working locally, we will be deploying to Harper Fabric (https://docs.harperdb.io/fabric). Let's work together and build a plan to implement this solution. We will plan to work iteratively and add features as we go.

That's it. That prompt, the attached Google Doc as a reference, and the skills files Harper provides for AI context were enough for Codex to scaffold the entire project structure and start building. From there, every feature was a conversation: "Add the Salesforce connector," "Wire up the approval gates," "Build the CI pipeline." Each one a sentence or two of intent, followed by Codex executing and me reviewing.

For context: this was my second Harper application built with Codex. The first was a personal project I created just to learn the workflow. The moment I saw how it worked, I realized I should be building our actual internal tooling this way, everything running on a single Harper backend that scales through Fabric without me thinking about infrastructure.

What Harper Made Possible

Let me be specific about what Harper provided versus what I would have had to solve myself on any other platform.

No database to provision. Schemas declared in files. Deploy the application, tables exist. Eight tables with indexes. No DBA, no managed instance, no connection pooling.

No application server to configure. Harper serves HTTP routes natively, both the web UI and the JSON API. No Express, no nginx, no port management.

No deployment pipeline to architect. I pointed Codex at the Fabric documentation and told it to handle deployment. It built a GitHub Actions pipeline that deploys to our production Harper Fabric cluster with rolling restart across replicated nodes, runs a post-deploy health check, and confirms the API is responding. Every merge to main triggers it automatically. I have never manually deployed this application.

No scaling to think about. Fabric handles replication. The only concession to multi-node was the leader-node scheduler gating to prevent duplicate scheduled jobs, and that was a straightforward pattern. I don't manage infrastructure. I don't think about infrastructure.

Auth handled by the platform. Harper has an OAuth plugin that handles OAuth 2.0 and OpenID Connect authentication out of the box, with support for Google, GitHub, Azure AD, and other providers. The plugin manages the OAuth flow, token refresh, session integration, and CSRF protection. On top of that, the application layer enforces role-based authorization checked server-side on every request. Having the database and the application in the same runtime made this simple. Role mappings are just another table, checked in the same process that serves the request.

Mobile-friendly because I asked for it. I told Codex I wanted the UI to work on phones. It built a responsive layout. That's it. No separate mobile project, no responsive framework to configure. Our executives do a lot of work on their phones, especially when traveling, and the old Google Doc was painful to read on a small screen. With Codex building on Harper, "make it mobile-friendly" was a sentence in a conversation, not a sprint.

The critical point: what I built locally is what runs in production. There is no gap. I run npm run dev locally, iterate with Codex, and when it's ready, the same application deploys to Fabric. One platform. One set of concerns. One thing to understand.

This is what makes it viable for a business operator to ship production software. Not because the coding is easier (Codex handles the coding), but because the operational surface area is small enough that a non-engineer can reason about the whole system. I don't need to understand Kubernetes, or Terraform, or how to wire a PostgreSQL connection pool. I need to understand Harper, and Harper is one thing.

What's Running Today

The application is live in production on Harper Fabric. It went live in mid-February 2026. The system now automatically produces a new digest every Friday. As of this writing, it has produced eight digests. That includes backfilled digests for the weeks I missed at the start of the year, which the system generated retroactively once it was live.

The weekly process now works like this: the app generates a digest on Friday morning. Kelli and I get a Slack notification. We review the metrics and commentary, make edits, and approve. When both approvals clear, the digest publishes to a company Slack channel and is available in the web UI for anyone to read. The review step takes a few minutes.

The time savings are real, at least 30 minutes of manual work eliminated every week, but that understates the actual impact. The real win is that the digest gets produced now. Every week. Automatically. Before this, the manual effort meant it was always the first thing I'd cut when the week got busy. That's not a time-savings story. That's a "the process actually works now" story.

I also plan to keep iterating. Our CEO has already discussed combining this with other internal tools the team is building on Harper into a single unified internal operations platform. In fact, the moment he saw this tool running, he asked me to write this blog post. (So here we are.) That convergence is natural because all these applications share the same data layer and deployment model. No integration work required. They're already on the same backend.

This is the part that I think gets missed in the "build vs. buy" conversation. When you buy a SaaS tool, you get one solution for one problem, and the next problem requires a new vendor. When you build on a platform like Harper, the first application is the hardest. The second one is easier because the platform is already there, the deployment model is already running, and your team already knows how it works. Harper is the only platform where coding agents can build and deploy enterprise-grade applications end to end, and once you have that foundation, every internal tool you need is just another application on the same infrastructure. The weekly digest was my first internal tool. It won't be my last.

The Math That Changed

Let me frame this for every RevOps operator, sales ops lead, or business systems person reading this.

You probably have a process right now that works like my old digest. Valuable output, tedious assembly, always at risk of being deprioritized. You've probably looked at SaaS tools to automate it and found them too expensive, too rigid, or too limited. You've probably thought about asking engineering to build something and decided it wasn't worth the political capital or the wait.

The math has changed. Here's why:

AI coding tools can now write production-quality code when given clear requirements and a well-structured platform to build against. The bottleneck was never the coding. It was the operational complexity of deploying and maintaining what the code produces. If your AI tool generates a great application but it requires you to manage a database, a web server, a cron service, a CDN, and a container orchestrator, you haven't saved yourself anything. You've just traded one kind of complexity for another.

Harper collapses that complexity. One runtime. One deployment target. One thing to maintain. Your database, your API, your frontend, your scheduled jobs, your auth: they're all the same application. When you deploy to Fabric, it scales. When you iterate locally, you're working against the same system that runs in production.

That's what makes it possible for someone like me, an operations person who understands systems but doesn't write code, to build and maintain production internal software. Not a prototype. Not a demo. A real application with enterprise integrations, approval workflows, role-based access control, CI/CD, and automated deployment.

The alternative is buying more SaaS. Another subscription. Another vendor. Another tool that does 70% of what you need and requires workarounds for the rest. Another integration to maintain. Another thing that breaks when the vendor pushes an update you didn't ask for.

Or you can build exactly what you need, on a platform you control, and ship it to production in days.

I know which one I'd pick. I just did.

Jake Cohen is Senior Director of Commercial Operations at Harper, where he has spent over eight years across Solutions Architecture, Product Management, Engineering Leadership, and Commercial Operations. He holds a B.S. in Computer Engineering from George Mason University. You can find him on LinkedIn.

To start building on Harper, visit harper.fast/vibe or run npm create harper@latest.

Build Your Own AI Ops Assistant — Part 6: Ship It

Stephen Goldberg — Thu, 26 Feb 2026 21:25:13 +0000

This is Part 6 of a 6-part series. Part 5 covers the knowledge loop.

Deploy to Production & What to Build Next

You've built an AI assistant that searches your internal tools in parallel, synthesizes answers with Claude, learns from feedback, and lives in Slack. Now let's deploy it to production, look at real costs, and talk about the extensions that take Harper Eye from good to indispensable.

Step 1: Deploy to Harper Fabric

If you've been developing locally, deploying to production is a single command. Make sure your .env has your Fabric credentials:

CLI_TARGET=your-cluster.your-org.harperfabric.com:9925
CLI_USERNAME=your-username
CLI_PASSWORD=your-password

And your CONFIG.env has all your API keys and tokens.

Deploy:

npm run deploy

This runs:

npx -y dotenv-cli -- harperdb deploy . restart=rolling replicated=true

What happens behind the scenes:

Your application code gets pushed to the Fabric cluster
Harper reads schema.graphql and creates/updates all tables and indexes
Environment variables from CONFIG.env are loaded
The cluster does a rolling restart with zero downtime
Your Resource Classes become live HTTPS endpoints

The whole process takes about 5-8 seconds. Your application is now live at:

https://your-cluster.your-org.harperfabric.com/

Update your Slack app's event URLs to point at this endpoint, and you're done.

Step 2: Verify Everything Works

Run through the checklist:

# Health check
curl https://your-cluster.your-org.harperfabric.com/HealthCheck

# API query
curl -X POST https://your-cluster.your-org.harperfabric.com/Api \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'user:pass' | base64)" \
  -d '{"query": "How does replication work?", "mode": "ask"}'

Then in Slack:

/harper-ask how does sharding work?

If you get a structured response with sources from your actual internal systems, you're live. You've just shipped a tool that would cost thousands per month from a vendor.

The Real Cost Breakdown

I've been running Harper Eye in production for our team. Here are the actual numbers:

Monthly Costs

Resource	Cost	Notes
Harper Fabric	~$25	Single instance handles everything: HTTP, database, vector search
Claude API (Anthropic)	~$30-50	~200-400 queries/month at ~$0.10-0.15/query (Sonnet)
Gemini Embeddings	$0	Free tier: 1,500 requests/min. We use maybe 1,000/month
Confluence API	$0	Included in existing Atlassian subscription
Zendesk API	$0	Included in existing subscription
Datadog API	$0	Included in existing subscription
GitHub API	$0	Included (authenticated requests: 5,000/hr)
Slack API	$0	Free for workspace apps
Total	~$55-75/mo	For the entire organization

Cost Per Query

With the knowledge base doing its job, many queries hit the cache and never touch Claude:

Query Type	Cost	Percentage of Queries
KB exact hit	~$0.001 (just the embedding call)	~30% after 2 months
Full orchestration	~$0.10-0.15 (embedding + Claude)	~70% initially, dropping over time
Blended average	~$0.07/query	Getting cheaper every month

The knowledge loop is doing real work. Every verified answer that gets cached means one fewer Claude API call in the future. After two months, about 30% of our queries return instant cached results. That percentage keeps climbing.

What This Replaces

SaaS Alternative	Monthly Cost
Glean or Dashworks (AI search)	$800-2,000
PagerDuty AIOps add-on	$500-1,000
Incident.io or similar	$400-800
Pinecone (vector DB)	$70-200
Total replaced	$1,770-4,000/mo

Savings: $1,700-3,900/month. That's $20,000-47,000/year.

And your custom system is better — it knows your terminology, your architecture, your people. It learns from your team's feedback. It doesn't forget when you cancel a subscription.

Extensions Worth Building

Once you have the core running, these extensions are each a day or less of work:

PagerDuty Webhooks

Create resources/PagerDutyWebhook.js that receives PagerDuty incident webhooks. When a new incident fires, Harper Eye automatically runs the full orchestration and posts the analysis to your Slack incident channel, before any human even looks at it.

export class PagerDutyWebhook extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    const event = data?.event;
    if (event?.event_type === 'incident.triggered') {
      const incident = event.data;

      // Auto-analyze the incident
      const result = await orchestrate(
        `PagerDuty incident: ${incident.title}. Service: ${incident.service?.summary}`,
        { mode: 'incident' }
      );

      // Post to your incident channel
      const slack = getSlackClient();
      await slack.chat.postMessage({
        channel: config.slack.incidentChannel(),
        blocks: [
          ...formatPagerDutyHeader(incident),
          ...formatIncidentResponse(result),
        ],
      });
    }
    return { ok: true };
  }
}

The result: PagerDuty fires at 2 am, and by the time your on-call engineer opens Slack, there's already an AI analysis with root cause candidates, relevant runbooks, customer impact assessment, and which colleague to escalate to.

Web Dashboard

The site/ directory serves static HTML at /app/. Build a simple dashboard for:

Knowledge base management: view, edit, and delete KB entries
Query analytics: most asked questions, resolution times, source utilization
Feedback trends: which topics get the most negative feedback (signals for documentation gaps)
Expert map: who on your team is the go-to person for which topics

No React. No build step. Vanilla HTML + CSS + JS that calls your /Api endpoint. Harper serves it as static files.

Knowledge Capture from Slack

Add intent classification to @-mentions so engineers can say @harper-eye save this in a Slack thread, and Harper Eye automatically extracts the key Q&A from the thread discussion and saves it to the knowledge base. The tribal knowledge from a debugging session is preserved forever without anyone having to write a wiki page.

Source Relevance Learning

Track which data sources actually contribute to useful answers. If Zendesk results never get cited for architecture questions, stop searching Zendesk for those queries. This saves API calls and reduces noise in Claude's context.

The Project Structure (Final)

harper-eye/
├── config.yaml              # Harper app config
├── schema.graphql           # 7 tables, 5 vector indexes
├── CONFIG.env               # Secrets (gitignored)
├── .env                     # Deploy creds (gitignored)
├── package.json
│
├── resources/               # HTTP endpoints (Resource Classes)
│   ├── SlackEvents.js       # Slack commands + @mentions
│   ├── SlackInteractivity.js # Feedback button handlers
│   ├── PagerDutyWebhook.js  # PagerDuty auto-analysis
│   ├── Api.js               # REST API for web dashboard
│   ├── HealthCheck.js       # GET /HealthCheck
│   └── Debug.js             # Debug endpoint
│
├── lib/                     # Core business logic
│   ├── orchestrator.js      # AI orchestration (the brain)
│   ├── knowledge-base.js    # KB CRUD + vector search + feedback
│   ├── embeddings.js        # Gemini embedding generation
│   ├── config.js            # Config loader
│   ├── slack-formatter.js   # Slack Block Kit formatting
│   └── slack-mentions.js    # @-mention expert suggestions
│
├── mcp/                     # Data source wrappers
│   ├── confluence.js        # Confluence REST API
│   ├── zendesk.js           # Zendesk REST API
│   ├── datadog.js           # Datadog REST API
│   ├── github.js            # GitHub REST API
│   └── harper-docs.js       # Documentation site search
│
└── site/                    # Static web UI
    ├── index.html           # Knowledge base dashboard
    ├── dashboard.html       # Analytics
    └── help.html            # Help page

Total lines of core logic: about 700. Total external infrastructure: zero (beyond Harper itself). Total time to build: 3 days.

What You've Built

Let's take a final inventory. Look at these two images side by side:

That's not a mockup. That's a production system that 35 engineers use every day, running for under $100/month. Here's what you now have:

An AI assistant that:

Searches 6+ internal data sources in parallel (2-4 seconds)
Synthesizes results with Claude into structured, cited responses
Lives in Slack with slash commands, @mentions, and threaded follow-ups
Returns verified cached answers instantly (sub-second)
Learns from team feedback without manual curation
Automatically degrades and purges bad answers
Knows which engineers are experts on which topics
Costs under $100/month for your entire organization

Running on a stack of:

Harper: one platform for HTTP, database, and vector search
Claude: AI synthesis with structured JSON output
Gemini: embedding generation (free tier)
Vanilla JS: Built into Harper, no framework, no build step

The compounding effects over time:

Month 1: ~0% KB cache hits. Every query goes through full orchestration.
Month 2: ~15-20% cache hits. Common questions return instantly.
Month 3: ~25-35% cache hits. Negative feedback has pruned bad answers.
Month 6: ~40-50% cache hits. Your AI costs are dropping while quality improves.
Month 12+: The knowledge base becomes your most valuable internal asset. It's the institutional memory that survives employee turnover.

The Bigger Picture

Here's what I've learned from building and running Harper Eye:

Custom beats generic, always. No vendor product will ever understand your architecture, your terminology, your people, or your incident history the way a custom tool does. The gap isn't about AI capability; Claude is the same Claude whether you use it through a vendor or directly. The gap is about context. Your context.

The feedback loop is everything. An AI assistant without a learning mechanism is a party trick. One that gets smarter every time someone uses it is an institution. The knowledge loop, verified answers, negative feedback, automatic degradation, is what makes Harper Eye more valuable the longer you run it.

Infrastructure should disappear. I didn't want to manage Postgres, Pinecone, Redis, Express, and a deployment pipeline. Harper let me put all of that in a single schema.graphql and config.yaml. The less time you spend on infrastructure, the more time you spend on the logic that actually makes your tool useful.

Build it yourself, but build it with the right tools. I built Harper Eye in 3 days, not because I'm fast, but because Claude wrote most of the code and Harper eliminated most of the infrastructure. The combination of AI-assisted development and a unified platform is what makes this feasible for a single engineer.

If you've followed along this far, you have everything you need. The code is real. The architecture is battle-tested. The costs are provably lower than the alternatives.

Build it. Run it. Let your team use it. Watch it get smarter.

And stop paying thousands a month for SaaS that doesn't know your name.

*If you build your own version, I'd love to hear about it. Reach out on the Harper Discord.

Build Your Own AI Ops Assistant — Part 5: The Knowledge Loop

Stephen Goldberg — Thu, 26 Feb 2026 18:32:10 +0000

This is Part 5 of a 6-part series. Part 4 covers the Slack integration.

Vector Search, Feedback & Self-Improvement

This is the part that separates a demo from a production system. Most AI assistants are stateless; they forget everything between conversations. Harper Eye gets smarter every time your team uses it. Good answers get cached and returned instantly. Bad answers get flagged, degraded, and eventually purged.

Here's what the knowledge loop looks like after a few weeks of real usage:

29 knowledge base articles, all created automatically when engineers clicked "Helpful." 24 feedback items — teaching the system what to avoid. Zero degraded entries, meaning the self-healing pipeline is working. That's institutional knowledge being built by your team without anyone having to write a wiki page. The knowledge loop is the compound interest of institutional knowledge.

How the Knowledge Loop Works

The beauty of this system: no one has to manually curate the knowledge base. Engineers just click thumbs up or thumbs down. The embeddings and similarity scores handle the rest.

Step 1: The Knowledge Base — Storage & Search

This is the core of the knowledge loop. Create lib/knowledge-base.js:

import { tables } from 'harperdb';
import crypto from 'crypto';
import { generateEmbedding } from './embeddings.js';

const SIMILARITY_HIGH = 0.85;   // Exact match — return cached
const SIMILARITY_LOW = 0.70;    // Partial match — inject as context
const NEGATIVE_FEEDBACK_THRESHOLD = 0.80;
const NEGATIVE_RATIO_DOWNGRADE = 0.3;  // 30%+ negative → downgrade
const NEGATIVE_RATIO_DELETE = 0.5;     // 50%+ negative → delete
const MIN_NEGATIVES_TO_DELETE = 2;     // Need at least 2 negatives

/**
 * Search the knowledge base for verified answers similar to the query.
 * Returns: { match: 'exact' | 'partial' | 'none', entry?, score?, degraded? }
 */
export async function searchKnowledgeBase(queryText, precomputedEmbedding = null) {
  let embedding;
  try {
    embedding = precomputedEmbedding ?? await generateEmbedding(queryText);
  } catch (err) {
    console.error('Embedding failed, skipping KB search:', err.message);
    return { match: 'none' };
  }

  try {
    // Vector similarity search using Harper's built-in HNSW index
    const results = tables.KnowledgeEntry.search({
      sort: { attribute: 'queryEmbedding', target: embedding },
      limit: 1,
    });

    const entries = [];
    for await (const entry of results) {
      entries.push(entry);
    }

    if (entries.length === 0) return { match: 'none' };

    const topEntry = entries[0];
    const storedEmbedding = topEntry.queryEmbedding;
    if (!storedEmbedding?.length) return { match: 'none' };

    // Compute cosine similarity
    const score = cosineSimilarity(embedding, storedEmbedding);

    if (score >= SIMILARITY_HIGH) {
      // Check if the entry has been degraded by negative feedback
      const negativeCount = topEntry.negativeCount ?? 0;
      const total = (topEntry.useCount ?? 0) + negativeCount;

      if (negativeCount > 0 && total > 0) {
        const ratio = negativeCount / total;
        if (ratio >= NEGATIVE_RATIO_DOWNGRADE) {
          // Downgrade: don't return as exact, let Claude regenerate
          return {
            match: 'partial',
            entry: serializeEntry(topEntry),
            score,
            degraded: true,
          };
        }
      }

      // Genuine exact match — increment use count and return
      await updateUseCount(topEntry);
      return { match: 'exact', entry: serializeEntry(topEntry), score };
    }

    if (score >= SIMILARITY_LOW) {
      return { match: 'partial', entry: serializeEntry(topEntry), score };
    }

    return { match: 'none' };
  } catch (err) {
    console.error('KB search failed:', err.message);
    return { match: 'none' };
  }
}

Let me highlight what's happening with Harper's vector search:

tables.KnowledgeEntry.search({
  sort: { attribute: 'queryEmbedding', target: embedding },
  limit: 1,
});

That's it. One line. Harper's built-in HNSW index handles approximate nearest-neighbor search across all stored embeddings. No Pinecone client, no pgvector extension, no separate vector database. You defined @indexed(type: "HNSW", distance: "cosine") in your schema and Harper handles the rest.

Step 2: Storing Verified Knowledge

When an engineer clicks "Helpful," we store the question-answer pair with a vector embedding. Future similar questions will match against it.

/**
 * Store a verified answer in the knowledge base.
 * Called when user clicks thumbs-up.
 */
export async function storeKnowledgeEntry({
  query,
  answer,
  sources,
  originalIncidentId,
  approvedByUserId,
  channelId,
}) {
  const embedding = await generateEmbedding(query);

  const entry = {
    id: crypto.randomUUID(),
    query,
    queryEmbedding: embedding,
    answer: typeof answer === 'string' ? answer : JSON.stringify(answer),
    sources: Array.isArray(sources) ? sources : [],
    originalIncidentId,
    approvedByUserId,
    approvedAt: new Date().toISOString(),
    channelId,
    useCount: 0,
    lastUsedAt: null,
    negativeCount: 0,
  };

  await tables.KnowledgeEntry.put(entry);
  console.log(`KB entry stored: ${entry.id} for "${query.slice(0, 80)}"`);
  return entry;
}

Once stored, the embedding is automatically added to the HNSW index. The next time someone asks a semantically similar question, searchKnowledgeBase() will find it.

Step 3: The Negative Feedback Loop

This is where the system self-heals. Negative feedback does two things:

Warns Claude about past failures so it doesn't repeat the same mistake
Degrades bad KB entries so they stop being returned as cached answers

/**
 * Search for past negative feedback on similar queries.
 * Used by the orchestrator to warn Claude about past failures.
 */
export async function searchNegativeFeedback(queryText, precomputedEmbedding = null) {
  let embedding;
  try {
    embedding = precomputedEmbedding ?? await generateEmbedding(queryText);
  } catch (err) {
    return [];
  }

  try {
    const results = tables.NegativeFeedback.search({
      sort: { attribute: 'queryEmbedding', target: embedding },
      limit: 3,
    });

    const matches = [];
    for await (const entry of results) {
      const stored = entry.queryEmbedding;
      if (!stored?.length) continue;

      const score = cosineSimilarity(embedding, stored);
      if (score >= NEGATIVE_FEEDBACK_THRESHOLD) {
        matches.push({
          category: entry.category,
          details: entry.details,
          originalQuery: entry.originalQuery,
          score,
        });
      }
    }
    return matches;
  } catch (err) {
    console.error('Negative feedback search failed:', err.message);
    return [];
  }
}

/**
 * Store negative feedback with embedding.
 * If linked to a KB entry, degrades that entry.
 */
export async function storeNegativeFeedback({
  queryId,
  originalQuery,
  category,
  details,
  userId,
  channelId,
  knowledgeEntryId,
}) {
  const embedding = await generateEmbedding(originalQuery);

  const record = {
    id: crypto.randomUUID(),
    queryId,
    originalQuery,
    queryEmbedding: embedding,
    category,
    details: details || null,
    userId,
    channelId,
    knowledgeEntryId: knowledgeEntryId || null,
    createdAt: new Date().toISOString(),
  };

  await tables.NegativeFeedback.put(record);

  // Degrade the linked KB entry if one exists
  if (knowledgeEntryId) {
    await degradeKnowledgeEntry(knowledgeEntryId);
  }

  return record;
}

The degradation logic is where it gets interesting:

async function degradeKnowledgeEntry(entryId) {
  try {
    const entry = await tables.KnowledgeEntry.get(entryId);
    if (!entry) return;

    const newNegativeCount = (entry.negativeCount ?? 0) + 1;
    const total = (entry.useCount ?? 0) + newNegativeCount;
    const ratio = newNegativeCount / total;

    if (newNegativeCount >= MIN_NEGATIVES_TO_DELETE && ratio >= NEGATIVE_RATIO_DELETE) {
      // This answer is doing more harm than good — remove it
      await tables.KnowledgeEntry.delete(entryId);
      console.log(`KB entry ${entryId} DELETED (ratio: ${ratio.toFixed(2)})`);
    } else {
      // Just increment the counter
      await tables.KnowledgeEntry.put({
        ...entry,
        negativeCount: newNegativeCount,
      });
      console.log(`KB entry ${entryId} negative count → ${newNegativeCount} (ratio: ${ratio.toFixed(2)})`);
    }
  } catch (err) {
    console.error('Failed to degrade KB entry:', err.message);
  }
}

The thresholds are tuned for safety:

30% negative ratio → the "exact" match gets downgraded to "partial," so Claude regenerates a fresh answer using the old one as reference
50% negative ratio AND at least 2 negatives → the entry is deleted entirely
This means a single bad vote can't kill an entry that's been helpful 10 times

The result: bad knowledge expires naturally. Good knowledge compounds. No human curation required.

Step 4: Cosine Similarity

The math behind the matching:

function cosineSimilarity(a, b) {
  let dot = 0;
  let magA = 0;
  let magB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

We compute this manually after Harper's HNSW search because Harper returns the nearest vectors but doesn't expose the similarity score directly. The HNSW index does the heavy lifting (finding the right candidates from potentially thousands of entries); we just compute the final score on the top result.

Step 5: The Use Count Tracker

Every time a cached answer is returned, we increment its use count. This serves two purposes: analytics (which answers are most valuable?) and feedback ratio calculation (a bad answer that's been used 100 times needs more than 2 complaints to be deleted).

async function updateUseCount(entry) {
  try {
    const existing = await tables.KnowledgeEntry.get(entry.id);
    if (!existing) return;
    await tables.KnowledgeEntry.put({
      ...existing,
      useCount: (existing.useCount ?? 0) + 1,
      lastUsedAt: new Date().toISOString(),
    });
  } catch (err) {
    console.error('Failed to update KB use count:', err.message);
  }
}

function serializeEntry(entry) {
  return {
    id: entry.id,
    query: entry.query,
    answer: entry.answer,
    sources: entry.sources,
    approvedByUserId: entry.approvedByUserId,
    approvedAt: entry.approvedAt,
    useCount: entry.useCount,
    negativeCount: entry.negativeCount ?? 0,
  };
}

Notice serializeEntry strips the embedding vector before returning. Those are 768-element float arrays; there is no need to send them downstream or serialize them in responses.

The Full Picture

Here's what your Harper Eye knowledge pipeline looks like now:

Scenario	What Happens	Speed
New question, no KB match	Full orchestration: embed → parallel search → Claude → respond	5-15s
Similar question asked before (verified)	KB exact match → return cached answer	<1s
Similar question but answer was bad	KB match downgraded → Claude regenerates with old answer as context	5-15s
Same mistake pattern detected	Negative feedback injected into Claude prompt → avoids repeating error	5-15s
Terrible answer accumulates complaints	KB entry auto-deleted → system reverts to fresh orchestration	Automatic

Every interaction makes the system better. Every thumbs-up builds institutional knowledge. Every thumbs-down teaches the system what to avoid. This is the compound interest of an AI assistant that works for your company, not a vendor's.

What's Next

In Part 6, we deploy to production, walk through the actual cost breakdown with real numbers, and explore extensions — PagerDuty webhooks for automatic incident analysis, a web dashboard for knowledge base management, and expert routing that knows who on your team to page.

Build Your Own AI Ops Assistant — Part 4: Slack Integration

Stephen Goldberg — Thu, 26 Feb 2026 16:41:41 +0000

This is Part 4 of a 6-part series. Part 3 covers the AI orchestrator.

Slash Commands, Events & Rich Responses

Your AI brain is built. Now let's put it where your team actually works: Slack. By the end of this part, your engineers will be able to type /incident replication lag on us-east or @harper-eye how does sharding work? and get a response that looks like this:

That's a real response to a real question from one of our engineers. Every @-mention, every source citation, every step — generated by the orchestrator from Part 3, formatted with Block Kit, and delivered right where the conversation is already happening.

Why Slack-First Matters

You could build the world's best web dashboard and your team wouldn't use it. Not because the dashboard is bad, but because context switching kills adoption. Your engineers live in Slack. When something breaks at 2 am, they're already in a channel. Making them open a separate tool, log in, type a query, wait for a response, then copy-paste the answer back into Slack, you've already lost them.

The Slack integration isn't a nice-to-have. It is the product for most of your team.

Step 1: Create a Slack App

Go to api.slack.com/apps and create a new app. Choose "From an app manifest" and use this JSON:

{
  "display_information": {
    "name": "Harper Eye",
    "description": "AI-powered ops assistant",
    "background_color": "#1a1a2e"
  },
  "features": {
    "bot_user": {
      "display_name": "Harper Eye",
      "always_online": true
    },
    "slash_commands": [
      {
        "command": "/incident",
        "url": "https://your-cluster.your-org.harperfabric.com/SlackEvents",
        "description": "Analyze an incident with AI",
        "usage_hint": "describe the issue",
        "should_escape": false
      },
      {
        "command": "/harper-ask",
        "url": "https://your-cluster.your-org.harperfabric.com/SlackEvents",
        "description": "Ask Harper a question",
        "usage_hint": "your question about the platform",
        "should_escape": false
      }
    ]
  },
  "oauth_config": {
    "scopes": {
      "bot": [
        "app_mentions:read",
        "chat:write",
        "commands",
        "channels:history",
        "groups:history",
        "users:read"
      ]
    }
  },
  "settings": {
    "event_subscriptions": {
      "request_url": "https://your-cluster.your-org.harperfabric.com/SlackEvents",
      "bot_events": ["app_mention"]
    },
    "interactivity": {
      "is_enabled": true,
      "request_url": "https://your-cluster.your-org.harperfabric.com/SlackInteractivity"
    }
  }
}

Replace your-cluster.your-org.harperfabric.com with your actual Fabric URL. Install the app to your workspace and grab the tokens from the "OAuth & Permissions" and "Basic Information" pages. Add them to your CONFIG.env:

SLACK_BOT_TOKEN=xoxb-...
SLACK_VERIFICATION_TOKEN=...
SLACK_SIGNING_SECRET=...

Step 2: Build the Slack Event Handler

This is the main Slack endpoint. It handles URL verification, slash commands, and @-mention events — all in a single Resource Class.

Create resources/SlackEvents.js:

import { Resource, tables } from 'harperdb';
import crypto from 'crypto';
import { config } from '../lib/config.js';
import { orchestrate } from '../lib/orchestrator.js';
import { generateEmbedding } from '../lib/embeddings.js';
import { searchKnowledgeBase } from '../lib/knowledge-base.js';
import {
  formatIncidentResponse,
  formatPlainText,
  formatAskResponse,
  formatAskPlainText,
} from '../lib/slack-formatter.js';
import { WebClient } from '@slack/web-api';

let slackClient;
function getSlackClient() {
  if (!slackClient) slackClient = new WebClient(config.slack.botToken());
  return slackClient;
}

export class SlackEvents extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    const body = data;

    // Slack URL verification — MUST return before any auth check
    if (body?.type === 'url_verification') {
      return { challenge: body.challenge };
    }

    // Verify request came from Slack
    if (!verifySlackRequest(body)) {
      const err = new Error('Invalid request');
      err.statusCode = 401;
      throw err;
    }

    // Slash commands: /incident and /harper-ask
    if (body?.command === '/incident') return handleSlashCommand(body, 'incident');
    if (body?.command === '/harper-ask') return handleSlashCommand(body, 'ask');

    // Event callbacks (@mentions)
    if (body?.type === 'event_callback') return handleEventCallback(body);

    return { ok: true };
  }
}

There are two critical things to notice here:

1. URL verification returns BEFORE auth. When you first configure your Slack app's event URL, Slack sends a challenge request. If you verify the signature first and the verification fails (which it can during initial setup), Slack never gets its challenge response and rejects your URL. Always handle url_verification first.

2. The post() method signature. Remember from Part 2: data is the parsed request body (second argument), not target (first argument). This is the most common mistake when building Harper Resource Classes.

Step 3: Handle Slash Commands

Slash commands have a critical constraint: Slack requires a response within 3 seconds. Your orchestrator takes 5-15 seconds. So the pattern is: acknowledge immediately, process asynchronously, then post the results via Slack's response_url.

function handleSlashCommand(body, mode = 'incident') {
  const { text, user_id, channel_id, response_url } = body;
  const isAsk = mode === 'ask';

  if (!text?.trim()) {
    return {
      response_type: 'ephemeral',
      text: isAsk
        ? 'Usage: `/harper-ask <your question>`'
        : 'Usage: `/incident <describe the issue>`',
    };
  }

  // Acknowledge immediately (returns within 3s)
  // Process asynchronously — fire and forget
  processQueryAsync({
    query: text.trim(),
    userId: user_id,
    channelId: channel_id,
    responseUrl: response_url,
    mode,
  });

  return {
    response_type: 'in_channel',
    text: ':mag: Searching across knowledge base...',
  };
}

The processQueryAsync function runs the orchestrator, formats the response, and posts it back:

async function processQueryAsync({
  query, userId, channelId, responseUrl, threadTs, updateTs, mode = 'incident',
}) {
  try {
    // Fast path: check KB before hitting Claude
    let result;
    let kbHit = false;

    try {
      const embedding = await generateEmbedding(query);
      if (embedding) {
        const kbResult = await searchKnowledgeBase(query, embedding);
        if (kbResult.match === 'exact' && kbResult.entry?.answer) {
          const cached = JSON.parse(kbResult.entry.answer);
          result = {
            summary: cached.summary ?? cached.answer ?? null,
            sources: cached.sources ?? [],
            steps: cached.steps ?? [],
            customerImpact: cached.customerImpact ?? null,
            escalation: cached.escalation ?? null,
            fromKnowledgeBase: true,
            knowledgeBaseEntryId: kbResult.entry.id,
          };
          kbHit = true;
        }
      }
    } catch (kbErr) {
      console.error('KB fast path failed:', kbErr.message);
    }

    // Full path: orchestrate if no KB hit
    if (!kbHit) {
      result = await orchestrate(query, { channelId, mode });
    }

    // Save to audit trail
    const queryId = crypto.randomUUID();
    await tables.IncidentQuery.put({
      id: queryId,
      slackUserId: userId,
      slackChannelId: channelId,
      query,
      response: JSON.stringify(result),
      sourcesUsed: result.sources?.map((s) => s.url ?? s.title) ?? [],
      createdAt: new Date().toISOString(),
    });
    result.queryId = queryId;

    // Format for Slack
    const isAsk = mode === 'ask';
    const blocks = isAsk ? formatAskResponse(result) : formatIncidentResponse(result);
    const fallbackText = isAsk ? formatAskPlainText(result) : formatPlainText(result);

    if (responseUrl) {
      // Slash command → post via response_url
      await fetch(responseUrl, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          response_type: 'in_channel',
          replace_original: true,
          blocks,
          text: fallbackText,
        }),
      });
    } else if (updateTs) {
      // @mention → update the ack message
      const slack = getSlackClient();
      await slack.chat.update({
        channel: channelId,
        ts: updateTs,
        blocks,
        text: fallbackText,
      });
    }
  } catch (err) {
    console.error('Error processing query:', err);
    const errorText = `:warning: Something went wrong: ${err.message}`;
    if (responseUrl) {
      await fetch(responseUrl, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ response_type: 'ephemeral', replace_original: true, text: errorText }),
      });
    }
  }
}

Notice the fast path: before ever calling the orchestrator (and spending API tokens), we check if the knowledge base already has a verified answer. If someone asked this question last week and another engineer confirmed the answer was good, we return it instantly. This is the knowledge loop at work; we'll build it fully in Part 5.

Step 4: Handle @-Mentions with Threaded Conversations

@-mentions are more powerful than slash commands because they support threaded follow-up conversations. Your engineer can ask a question, read the response, then reply in the thread with a follow-up, and Harper Eye has the full conversation context.

async function handleEventCallback(payload) {
  const event = payload.event;

  // Ignore bot messages to prevent loops
  if (event.bot_id || event.subtype === 'bot_message') {
    return { ok: true };
  }

  if (event.type === 'app_mention') {
    // Strip the @mention from the query text
    const query = event.text.replace(/<@[A-Z0-9]+>/g, '').trim();

    if (!query) {
      const slack = getSlackClient();
      await slack.chat.postMessage({
        channel: event.channel,
        thread_ts: event.ts,
        text: 'Ask me anything! Example: `@harper-eye how does replication work?`',
      });
      return { ok: true };
    }

    // Check for thread context (follow-up questions)
    const threadTs = event.thread_ts ?? event.ts;
    let conversationHistory = [];

    if (event.thread_ts) {
      // This is a reply in an existing thread — fetch conversation context
      conversationHistory = await getThreadConversationContext(
        event.channel,
        event.thread_ts
      );
    }

    // Post acknowledgment in the thread
    const slack = getSlackClient();
    const ack = await slack.chat.postMessage({
      channel: event.channel,
      thread_ts: threadTs,
      text: ':mag: Searching across knowledge base...',
    });

    // Process async, update the ack message when done
    processQueryAsync({
      query,
      userId: event.user,
      channelId: event.channel,
      threadTs,
      updateTs: ack.ts,
      conversationHistory,
    });
  }

  return { ok: true };
}

The thread context fetcher pairs user messages with bot responses to build conversation history:

async function getThreadConversationContext(channelId, threadTs) {
  try {
    const slack = getSlackClient();
    const result = await slack.conversations.replies({
      channel: channelId,
      ts: threadTs,
      limit: 20,
    });

    const history = [];
    let lastUserQuery = null;

    for (const msg of result.messages ?? []) {
      if (!msg.bot_id && !msg.subtype && msg.text) {
        lastUserQuery = msg.text.replace(/<@[A-Z0-9]+>/g, '').trim();
      } else if (msg.bot_id && lastUserQuery) {
        const summary = extractSummaryFromBlocks(msg.blocks) || msg.text;
        if (summary?.length > 20) {
          history.push({ query: lastUserQuery, summary });
        }
        lastUserQuery = null;
      }
    }
    return history;
  } catch (err) {
    console.error('Failed to fetch thread context:', err.message);
    return [];
  }
}

Step 5: Format Responses with Block Kit

Slack's Block Kit gives you rich, structured messages with headers, dividers, formatted text, and interactive buttons. Here's the formatter that turns orchestrator results into Slack blocks.

Create lib/slack-formatter.js:

/**
 * Format an incident analysis result into Slack Block Kit blocks.
 */
export function formatIncidentResponse(result) {
  const blocks = [];

  blocks.push({
    type: 'header',
    text: { type: 'plain_text', text: 'Incident Analysis', emoji: true },
  });

  // Knowledge base indicator
  if (result.fromKnowledgeBase) {
    blocks.push({
      type: 'context',
      elements: [{
        type: 'mrkdwn',
        text: ':brain: _Retrieved from knowledge base (verified past answer)._',
      }],
    });
  }

  // Summary
  if (result.summary) {
    blocks.push(
      { type: 'divider' },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `*Situation Summary*\n${result.summary}` },
      }
    );
  }

  // Customer Impact
  if (result.customerImpact) {
    blocks.push(
      { type: 'divider' },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `*Customer Impact*\n${result.customerImpact}` },
      }
    );
  }

  // Steps
  if (result.steps?.length) {
    const stepsText = result.steps.map((s, i) => `${i + 1}. ${s}`).join('\n');
    blocks.push(
      { type: 'divider' },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `*Recommended Steps*\n${stepsText}` },
      }
    );
  }

  // Sources with clickable links
  if (result.sources?.length) {
    const sourcesText = result.sources.map((s) => {
      if (typeof s === 'string') return `- ${s}`;
      if (s.url && s.title) return `- <${s.url}|${s.title}>`;
      return `- ${s.title ?? s.url ?? JSON.stringify(s)}`;
    }).join('\n');
    blocks.push(
      { type: 'divider' },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `*Sources*\n${sourcesText}` },
      }
    );
  }

  // Escalation
  if (result.escalation) {
    blocks.push(
      { type: 'divider' },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `:rotating_light: *Escalation*\n${result.escalation}` },
      }
    );
  }

  // Feedback buttons
  const buttonValue = JSON.stringify({
    queryId: result.queryId ?? null,
    knowledgeEntryId: result.knowledgeBaseEntryId ?? null,
  });

  blocks.push(
    { type: 'divider' },
    {
      type: 'actions',
      block_id: 'feedback_actions',
      elements: [
        {
          type: 'button',
          text: { type: 'plain_text', text: ':thumbsup: Helpful', emoji: true },
          action_id: 'feedback_positive',
          value: buttonValue,
          style: 'primary',
        },
        {
          type: 'button',
          text: { type: 'plain_text', text: ':thumbsdown: Not Helpful', emoji: true },
          action_id: 'feedback_negative',
          value: buttonValue,
        },
      ],
    }
  );

  return blocks;
}

Step 6: Handle Button Interactions

When someone clicks "Helpful" or "Not Helpful," Slack sends a payload to your interactivity endpoint. Create resources/SlackInteractivity.js:

import { Resource, tables } from 'harperdb';
import { config } from '../lib/config.js';
import { storeKnowledgeEntry, storeNegativeFeedback } from '../lib/knowledge-base.js';
import { WebClient } from '@slack/web-api';

let slackClient;
function getSlackClient() {
  if (!slackClient) slackClient = new WebClient(config.slack.botToken());
  return slackClient;
}

export class SlackInteractivity extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    // Slack sends interactivity payloads as form-encoded with a "payload" field
    const payloadStr = data?.payload ?? data;
    const payload = typeof payloadStr === 'string' ? JSON.parse(payloadStr) : payloadStr;

    if (payload.type === 'block_actions') {
      for (const action of payload.actions ?? []) {
        const meta = JSON.parse(action.value ?? '{}');

        if (action.action_id === 'feedback_positive') {
          await handlePositiveFeedback(meta, payload);
        }

        if (action.action_id === 'feedback_negative') {
          await handleNegativeFeedback(meta, payload);
        }
      }
    }

    return { ok: true };
  }
}

async function handlePositiveFeedback({ queryId }, payload) {
  if (!queryId) return;

  try {
    const entry = await tables.IncidentQuery.get(queryId);
    if (!entry) return;

    // Store as verified knowledge for future queries
    await storeKnowledgeEntry({
      query: entry.query,
      answer: entry.response,
      sources: entry.sourcesUsed ?? [],
      originalIncidentId: queryId,
      approvedByUserId: payload.user?.id,
      channelId: entry.slackChannelId,
    });

    // Acknowledge in Slack
    const slack = getSlackClient();
    await slack.chat.postMessage({
      channel: payload.channel?.id,
      thread_ts: payload.message?.ts,
      text: ':white_check_mark: Thanks! This answer has been saved to the knowledge base.',
    });
  } catch (err) {
    console.error('Failed to store positive feedback:', err.message);
  }
}

async function handleNegativeFeedback({ queryId, knowledgeEntryId }, payload) {
  // Open a modal asking what was wrong
  const slack = getSlackClient();
  await slack.views.open({
    trigger_id: payload.trigger_id,
    view: {
      type: 'modal',
      callback_id: 'negative_feedback_modal',
      private_metadata: JSON.stringify({ queryId, knowledgeEntryId }),
      title: { type: 'plain_text', text: 'What went wrong?' },
      blocks: [
        {
          type: 'input',
          block_id: 'category',
          element: {
            type: 'static_select',
            action_id: 'category_select',
            options: [
              { text: { type: 'plain_text', text: 'Inaccurate' }, value: 'inaccurate' },
              { text: { type: 'plain_text', text: 'Outdated' }, value: 'outdated' },
              { text: { type: 'plain_text', text: 'Too Generic' }, value: 'too_generic' },
              { text: { type: 'plain_text', text: 'Missing Info' }, value: 'missing_info' },
            ],
          },
          label: { type: 'plain_text', text: 'Category' },
        },
        {
          type: 'input',
          block_id: 'details',
          optional: true,
          element: {
            type: 'plain_text_input',
            action_id: 'details_input',
            multiline: true,
            placeholder: { type: 'plain_text', text: 'Any specific feedback?' },
          },
          label: { type: 'plain_text', text: 'Details' },
        },
      ],
      submit: { type: 'plain_text', text: 'Submit' },
    },
  });
}

When the user submits the modal, it comes back as a view_submission event. You'd handle that in the same post() method to call storeNegativeFeedback() — which we'll build fully in Part 5.

Deploy & Test

Deploy your updated code:

npm run deploy

Once deployed, verify your Slack app's event URL is pointing at your Fabric URL. Then try it:

/harper-ask how does replication work?

You should see the "🔍 Searching..." acknowledgment, followed by a rich Block Kit response with your answer, sources, and feedback buttons.

Try the @mention in a channel:

@harper-eye what happens when a node goes down?

Then reply in the thread:

@harper-eye what about if it's the primary node?

Harper Eye will include the previous Q&A as context for the follow-up, with no repetition and full continuity.

What You Have Now

Your team can now interact with Harper Eye entirely from Slack:

/incident <description> — incident analysis with remediation steps
/harper-ask <question> — general questions about your platform
@harper-eye <question> — @mention anywhere, supports threaded follow-ups
Feedback buttons — thumbs up saves to knowledge base, thumbs down collects structured feedback
Rich formatting — headers, sections, clickable source links, escalation callouts

All of it running on a single Harper instance. No Lambda functions, no SQS queues, no separate API gateway. One deploy target.

What's Next

In Part 5, we build the knowledge loop, the system that makes Harper Eye smarter every time your team uses it. Vector-indexed knowledge base, semantic similarity search, feedback-driven degradation, and the self-healing answer pipeline.

Build Your Own AI Ops Assistant — Part 3: The AI Brain

Stephen Goldberg — Wed, 25 Feb 2026 23:47:13 +0000

This is Part 3 of a 6-part series. Part 2 covers the foundation setup.

Claude Orchestrator & Multi-Source Search

This is where Harper Eye becomes genuinely powerful. By the end of this part, you'll have an AI assistant that searches all your internal data sources in parallel, synthesizes the results with Claude, and returns structured, cited responses like this:

About 400 lines of code, and every one of them earns its keep.

The Architecture of a Single Query

When someone asks Harper Eye a question, here's exactly what happens:

The key insight: everything that can run in parallel, does. The embedding generation happens once at the start, then gets reused for KB search, negative feedback search, expertise lookup, and code knowledge search. The six data sources all fire simultaneously. The total wall-clock time is dominated by the slowest source (usually 2-4 seconds) plus Claude's synthesis time, not the sum of all sources.

Here's the actual pipeline timing from a production query, you can see the parallelism in action:

Step 1: Build Your First Data Source Wrapper

Every external data source follows the same pattern: define the tools Claude could use, then implement simple fetch() calls to the source's REST API. No MCP subprocess servers, no complex client libraries, just HTTP.

Let's build the Confluence wrapper as our template. Create mcp/confluence.js:

import { config } from '../lib/config.js';

/**
 * Confluence search tool — direct HTTP calls to the REST API.
 * Provides tool definitions (Claude format) and execution functions.
 */

const tools = [
  {
    name: 'search_confluence',
    description: 'Full-text search across Confluence spaces for runbooks, postmortems, and architecture docs.',
    input_schema: {
      type: 'object',
      properties: {
        query: { type: 'string', description: 'Search query (CQL-compatible)' },
        space: { type: 'string', description: 'Optional space key to restrict search' },
        limit: { type: 'number', description: 'Max results (default 5)' },
      },
      required: ['query'],
    },
  },
  {
    name: 'get_confluence_page',
    description: 'Retrieve the body content of a specific Confluence page by ID.',
    input_schema: {
      type: 'object',
      properties: {
        pageId: { type: 'string', description: 'Confluence page ID' },
      },
      required: ['pageId'],
    },
  },
];

export function getConfluenceTools() {
  return tools;
}

export async function executeConfluenceTool(name, input) {
  const baseUrl = config.confluence.baseUrl();
  const auth = Buffer.from(
    `${config.confluence.email()}:${config.confluence.apiToken()}`
  ).toString('base64');
  const headers = {
    Authorization: `Basic ${auth}`,
    Accept: 'application/json',
  };

  switch (name) {
    case 'search_confluence':
      return searchConfluence(baseUrl, headers, input);
    case 'get_confluence_page':
      return getPageContent(baseUrl, headers, input);
    default:
      throw new Error(`Unknown Confluence tool: ${name}`);
  }
}

async function searchConfluence(baseUrl, headers, { query, space, limit = 5 }) {
  // Extract meaningful keywords for broader matching
  const keywords = query
    .toLowerCase()
    .replace(/[^a-z0-9\s\-_]/g, ' ')
    .split(/\s+/)
    .filter((w) => w.length >= 3 && !stopWords.has(w));

  let cql = keywords.length > 0
    ? keywords.map((kw) => `text ~ "${kw}"`).join(' AND ')
    : `text ~ "${query}"`;

  if (space) cql += ` AND space = "${space}"`;

  const url = `${baseUrl}/wiki/rest/api/search?cql=${encodeURIComponent(cql)}&limit=${limit}`;
  const res = await fetch(url, { headers });

  if (!res.ok) throw new Error(`Confluence search failed: ${res.status}`);

  const data = await res.json();
  return data.results.map((r) => ({
    title: r.content?.title ?? r.title,
    id: r.content?.id ?? r.id,
    url: `${baseUrl}/wiki${r.url ?? r.content?._links?.webui ?? ''}`,
    excerpt: stripHtml(r.excerpt ?? ''),
    lastModified: r.lastModified,
  }));
}

async function getPageContent(baseUrl, headers, { pageId }) {
  const url = `${baseUrl}/wiki/rest/api/content/${pageId}?expand=body.storage,version,space`;
  const res = await fetch(url, { headers });

  if (!res.ok) throw new Error(`Confluence get page failed: ${res.status}`);

  const data = await res.json();
  const body = stripHtml(data.body?.storage?.value ?? '');

  // Truncate to ~2000 tokens worth of text (roughly 8000 chars)
  const truncated = body.length > 8000
    ? body.slice(0, 8000) + '\n\n[Content truncated]'
    : body;

  return {
    title: data.title,
    id: data.id,
    space: data.space?.key,
    url: `${baseUrl}/wiki${data._links?.webui ?? ''}`,
    content: truncated,
  };
}

const stopWords = new Set([
  'the', 'what', 'how', 'when', 'where', 'does', 'this', 'that',
  'with', 'for', 'and', 'but', 'not', 'are', 'was', 'can', 'has',
  'have', 'from',
]);

function stripHtml(html) {
  return html
    .replace(/<[^>]*>/g, '')
    .replace(/&nbsp;/g, ' ')
    .replace(/&amp;/g, '&')
    .replace(/&lt;/g, '<')
    .replace(/&gt;/g, '>')
    .replace(/&quot;/g, '"')
    .replace(/\s+/g, ' ')
    .trim();
}

The pattern is always the same for every data source:

Define the tool schema (what Claude sees)
Export getXTools() for tool definitions
Export executeXTool(name, input) as the dispatcher
Implement each tool as a simple fetch() call
Return clean, structured data

Here are the other wrappers you'll build using this same pattern, I'll show the Zendesk one as another example since each API is slightly different:

Create mcp/zendesk.js:

import { config } from '../lib/config.js';

export async function executeZendeskTool(name, input) {
  const subdomain = config.zendesk.subdomain();
  const auth = Buffer.from(
    `${config.zendesk.email()}/token:${config.zendesk.apiToken()}`
  ).toString('base64');
  const headers = {
    Authorization: `Basic ${auth}`,
    Accept: 'application/json',
  };

  if (name === 'search_tickets') {
    return searchTickets(subdomain, headers, input);
  }
  throw new Error(`Unknown Zendesk tool: ${name}`);
}

async function searchTickets(subdomain, headers, { query, limit = 5 }) {
  const url = `https://${subdomain}.zendesk.com/api/v2/search.json?query=${encodeURIComponent(query)}&per_page=${limit}`;
  const res = await fetch(url, { headers });

  if (!res.ok) throw new Error(`Zendesk search failed: ${res.status}`);

  const data = await res.json();
  return (data.results ?? []).map((t) => ({
    id: t.id,
    subject: t.subject,
    status: t.status,
    priority: t.priority,
    description: (t.description ?? '').slice(0, 500),
    url: `https://${subdomain}.zendesk.com/agent/tickets/${t.id}`,
    createdAt: t.created_at,
    updatedAt: t.updated_at,
  }));
}

You'll follow the same pattern for Datadog (mcp/datadog.js), GitHub (mcp/github.js), and your documentation site (mcp/harper-docs.js). Each one is just fetch() calls to their respective REST APIs, returning clean JSON.

Step 2: Build the Embedding Pipeline

Embeddings are how Harper Eye understands meaning, not just keywords. When someone asks "why is my cluster slow?" and a knowledge base entry is titled "investigating replication latency," vector similarity catches the semantic match that keyword search would miss.

Create lib/embeddings.js:

import { config } from './config.js';

/**
 * Generate an embedding vector using Google Gemini.
 * Returns an array of 768 floats (cosine-similarity compatible).
 */
export async function generateEmbedding(text) {
  const apiKey = config.gemini.apiKey();
  const model = config.gemini.embeddingModel();

  const url = `https://generativelanguage.googleapis.com/v1beta/models/${model}:embedContent?key=${apiKey}`;

  const res = await fetch(url, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: `models/${model}`,
      content: { parts: [{ text }] },
      outputDimensionality: 768,
    }),
  });

  if (!res.ok) {
    const body = await res.text().catch(() => '');
    throw new Error(`Gemini embeddings failed: ${res.status} — ${body}`);
  }

  const data = await res.json();
  return data.embedding.values;
}

That's 25 lines. A single API call. And it gives you 768-dimensional vectors that work beautifully with Harper's HNSW indexes.

We use Gemini for embeddings because it's fast, the free tier is generous (1,500 requests/min), and the text-embedding-004 model produces high-quality vectors. You could swap this for OpenAI's embeddings or any other provider; the orchestrator doesn't care.

Step 3: Build the Orchestrator

This is the heart of Harper Eye. The orchestrator coordinates everything: checks the knowledge base, fetches all data sources in parallel, constructs the Claude prompt, parses the response, and records source relevance for future optimization.

Create lib/orchestrator.js:

import Anthropic from '@anthropic-ai/sdk';
import { config } from './config.js';
import { executeConfluenceTool } from '../mcp/confluence.js';
import { executeZendeskTool } from '../mcp/zendesk.js';
import { executeDatadogTool } from '../mcp/datadog.js';
import { executeGitHubTool } from '../mcp/github.js';
import { executeHarperDocsTool } from '../mcp/harper-docs.js';
import { searchKnowledgeBase, searchNegativeFeedback } from './knowledge-base.js';
import { generateEmbedding } from './embeddings.js';

Now for the system prompt. This is the most important prompt you'll ever write. It defines the personality, the constraints, and the output format of every response your team sees:

const SYSTEM_PROMPT = `You are Harper Eye, the internal assistant for [YOUR COMPANY].
You help engineers resolve incidents and answer questions about our platform.

CONTEXT RULES — CRITICAL:
- ALL answers must be specific to [YOUR COMPANY] and our platform.
  Never give generic advice.
- Use our internal terminology (define your key terms here).
- If you cannot find specific information in the provided data,
  say so — do not fill in with generic knowledge.
- Do NOT pad with generic steps like "check logs" or "restart the service"
  unless you can specify EXACTLY which logs, which service.

You will receive data pre-fetched from these sources:
- DOCUMENTATION: Official product docs (most authoritative)
- CONFLUENCE: Internal runbooks, postmortems, architecture docs
- DATADOG MONITORS: Currently alerting monitors and recent events
- DATADOG LOGS: Recent log entries matching the query
- ZENDESK: Customer-reported tickets
- GITHUB: Code search results
- SLACK HISTORY: Recent relevant messages from team channels
- TEAM EXPERTISE MAP: Who to contact for help on these topics

Analyze ALL provided data and respond with a JSON object:
{
  "summary": "What is happening, grounded in the data provided",
  "customerImpact": "Impact from Zendesk data, or null",
  "steps": ["Step 1 as a string", "Step 2", "Step 3"],
  "sources": [{"title": "...", "url": "...", "sourceType": "confluence|zendesk|datadog|github|docs"}],
  "escalation": "Who to loop in based on expertise data, or null"
}

CRITICAL FORMAT RULES:
- "steps" MUST be an array of plain strings. NOT objects.
- Every step should be specific and actionable, citing which source.
- Respond ONLY with the JSON object.`;

You'll want to customize this heavily for your company. The more specific your system prompt is about your architecture, your terminology, and your deprecated vs. current systems, the better Claude's responses will be.

Now the main orchestrate() function:

/**
 * Main orchestration loop.
 * @param {string} query - The user's question
 * @param {Object} options
 * @param {string} [options.channelId] - Slack channel for history search
 * @param {'incident'|'ask'} [options.mode] - Query mode
 * @param {Array} [options.conversationHistory] - Previous Q&A turns
 */
export async function orchestrate(query, {
  channelId,
  mode = 'incident',
  conversationHistory = [],
} = {}) {
  // Step 0: Generate embedding once, reuse everywhere
  let embedding;
  try {
    embedding = await generateEmbedding(query);
  } catch (err) {
    console.error('Embedding generation failed:', err.message);
    embedding = null;
  }

  // Step 1: Check knowledge base + negative feedback in parallel
  const [kbResult, negativeFeedback] = await Promise.all([
    embedding
      ? searchKnowledgeBase(query, embedding)
      : Promise.resolve({ match: 'none' }),
    embedding
      ? searchNegativeFeedback(query, embedding)
      : Promise.resolve([]),
  ]);

  // Step 1a: If KB has a high-confidence cached answer, return immediately
  if (kbResult.match === 'exact' && conversationHistory.length === 0) {
    try {
      const cachedAnswer = JSON.parse(kbResult.entry.answer);
      return {
        ...cachedAnswer,
        fromKnowledgeBase: true,
        knowledgeBaseScore: kbResult.score,
        knowledgeBaseEntryId: kbResult.entry.id,
      };
    } catch {
      // Parse failed — fall through to normal flow
    }
  }

  // Step 2: Build system prompt with KB context + negative feedback warnings
  let systemPrompt = SYSTEM_PROMPT;

  if (kbResult.match === 'partial') {
    // Inject partial match as reference context for Claude
    try {
      const pastAnswer = JSON.parse(kbResult.entry.answer);
      systemPrompt += `\n\nADDITIONAL CONTEXT — A similar past question was answered (similarity: ${kbResult.score.toFixed(2)}). Use as reference but verify:\nPast query: "${kbResult.entry.query}"\nPast answer: ${JSON.stringify(pastAnswer)}`;
    } catch { /* ignore parse errors */ }
  }

  if (negativeFeedback.length > 0) {
    // Warn Claude about past failures so it doesn't repeat them
    const warnings = negativeFeedback
      .map((fb) => `- "${fb.originalQuery}" was marked "${fb.category}"${fb.details ? `: "${fb.details}"` : ''}`)
      .join('\n');
    systemPrompt += `\n\nIMPORTANT — PAST NEGATIVE FEEDBACK:\n${warnings}\nAvoid repeating these mistakes.`;
  }

  // Step 3: Fetch ALL data sources in parallel
  const searchResults = await fetchAllSources(query, embedding, { channelId });

  // Step 4: Build context block from all results
  const contextParts = [];

  if (searchResults.harperDocs?.length)
    contextParts.push(`DOCUMENTATION:\n${JSON.stringify(searchResults.harperDocs)}`);
  if (searchResults.confluence?.length)
    contextParts.push(`CONFLUENCE:\n${JSON.stringify(searchResults.confluence)}`);
  if (searchResults.datadog?.monitors?.length)
    contextParts.push(`DATADOG MONITORS:\n${JSON.stringify(searchResults.datadog.monitors)}`);
  if (searchResults.datadog?.logs?.length)
    contextParts.push(`DATADOG LOGS:\n${JSON.stringify(searchResults.datadog.logs)}`);
  if (searchResults.zendesk?.length)
    contextParts.push(`ZENDESK TICKETS:\n${JSON.stringify(searchResults.zendesk)}`);
  if (searchResults.githubCode?.length)
    contextParts.push(`GITHUB CODE:\n${JSON.stringify(searchResults.githubCode)}`);
  if (searchResults.slackHistory?.length)
    contextParts.push(`SLACK HISTORY:\n${JSON.stringify(searchResults.slackHistory)}`);
  if (searchResults.expertise?.experts?.length)
    contextParts.push(`TEAM EXPERTISE MAP:\n${JSON.stringify(searchResults.expertise)}`);

  const contextBlock = contextParts.length > 0
    ? `\n\nData from all sources:\n\n${contextParts.join('\n\n')}`
    : '\n\nNo relevant data found in any source.';

  const userMessage = `Incident report: ${query}${contextBlock}\n\nAnalyze and respond with JSON.`;

  // Step 5: Call Claude
  const client = new Anthropic({ apiKey: config.claude.apiKey() });

  let response;
  for (let attempt = 1; attempt <= 3; attempt++) {
    try {
      response = await client.messages.create({
        model: config.claude.model(),
        max_tokens: 4096,
        system: systemPrompt,
        messages: [{ role: 'user', content: userMessage }],
      });
      break;
    } catch (apiErr) {
      if (apiErr.status === 429 && attempt < 3) {
        const retryAfter = Number(apiErr.headers?.['retry-after']) || (attempt * 15);
        await new Promise((r) => setTimeout(r, retryAfter * 1000));
        continue;
      }
      throw apiErr;
    }
  }

  // Step 6: Parse response
  const textBlock = response.content.find((b) => b.type === 'text');
  const rawResponse = textBlock?.text ?? '{}';
  return parseResponse(rawResponse);
}

Step 4: The Parallel Source Fetcher

This is where the "fast" in "12-second response" comes from. Every data source fires at the same time. If Confluence takes 3 seconds and Datadog takes 1 second, the total is 3 seconds, not 4.

/**
 * Fetch data from all configured sources in parallel.
 * Each source is wrapped in try/catch so failures don't block others.
 */
async function fetchAllSources(query, embedding, { channelId } = {}) {
  const results = {
    harperDocs: [],
    confluence: [],
    datadog: { monitors: [], events: [], hosts: [], logs: [] },
    zendesk: [],
    githubCode: [],
    slackHistory: [],
    expertise: { experts: [], channels: [] },
  };

  const fetches = [];

  // Documentation
  fetches.push(
    executeHarperDocsTool('search_harper_docs', { query, limit: 5 })
      .then((r) => { results.harperDocs = r; })
      .catch((err) => console.error('Docs search failed:', err.message))
  );

  // Confluence — wrapped in config check so it's skipped if not configured
  try {
    config.confluence.baseUrl(); // throws if not configured
    fetches.push(
      executeConfluenceTool('search_confluence', { query, limit: 5 })
        .then((r) => { results.confluence = r; })
        .catch((err) => console.error('Confluence failed:', err.message))
    );
  } catch { /* not configured — skip */ }

  // Datadog
  try {
    config.datadog.apiKey();
    fetches.push(
      executeDatadogTool('get_alerting_monitors', {})
        .then((r) => { results.datadog.monitors = r; })
        .catch((err) => console.error('Datadog monitors failed:', err.message))
    );
    fetches.push(
      executeDatadogTool('search_datadog_logs', {
        query: `status:(error OR warn) ${query}`,
        hours: 2,
        limit: 15,
      })
        .then((r) => { results.datadog.logs = r; })
        .catch((err) => console.error('Datadog logs failed:', err.message))
    );
  } catch { /* not configured */ }

  // Zendesk
  try {
    config.zendesk.subdomain();
    fetches.push(
      executeZendeskTool('search_tickets', { query, limit: 5 })
        .then((r) => { results.zendesk = r; })
        .catch((err) => console.error('Zendesk failed:', err.message))
    );
  } catch { /* not configured */ }

  // GitHub
  try {
    config.github.token();
    fetches.push(
      executeGitHubTool('search_github_code', { query, limit: 3 })
        .then((r) => { results.githubCode = r; })
        .catch((err) => console.error('GitHub failed:', err.message))
    );
  } catch { /* not configured */ }

  await Promise.all(fetches);
  return results;
}

The try { config.x.y() } catch {} pattern is intentional. If Datadog isn't configured, the config getter throws, and we skip that source entirely. Zero if statements, zero feature flags. Just configure an env var and the source lights up.

Step 5: The Response Parser

Claude outputs JSON, but LLMs are messy. Sometimes they wrap it in markdown code blocks. Sometimes the JSON has weird types. You need a parser that handles all the edge cases gracefully:

function parseResponse(text) {
  const jsonStr = extractJson(text);

  try {
    const parsed = JSON.parse(jsonStr);
    return normalizeResult(parsed);
  } catch (err) {
    // Last resort: regex out the summary field
    const match = text.match(/"(?:answer|summary)"\s*:\s*"((?:[^"\\]|\\.)*)"/);
    if (match) {
      return {
        summary: match[1].replace(/\\n/g, '\n').replace(/\\"/g, '"'),
        customerImpact: null,
        steps: [],
        sources: [],
        escalation: null,
      };
    }
    // If all parsing fails, return the raw text
    return { summary: text, steps: [], sources: [], escalation: null };
  }
}

function extractJson(text) {
  // Strategy 1: markdown code block
  const codeBlock = text.match(/```
{% endraw %}
(?:json)?\s*([\s\S]*?)
{% raw %}
```/);
  if (codeBlock) return codeBlock[1].trim();

  // Strategy 2: outermost braces
  const first = text.indexOf('{');
  const last = text.lastIndexOf('}');
  if (first !== -1 && last > first) return text.slice(first, last + 1);

  // Strategy 3: raw text
  return text.trim();
}

function normalizeResult(parsed) {
  // Normalize steps to plain strings (Claude sometimes returns objects)
  let steps = [];
  if (Array.isArray(parsed.steps)) {
    steps = parsed.steps.map((s) => {
      if (typeof s === 'string') return s;
      if (typeof s === 'object' && s !== null) {
        return s.step ?? s.action ?? s.description ?? JSON.stringify(s);
      }
      return String(s);
    });
  }

  // Normalize escalation to string
  let escalation = parsed.escalation ?? null;
  if (escalation && typeof escalation === 'object') {
    escalation = escalation.message ?? escalation.text ?? JSON.stringify(escalation);
  }

  return {
    summary: parsed.summary ?? parsed.answer ?? null,
    customerImpact: parsed.customerImpact ?? null,
    steps,
    sources: Array.isArray(parsed.sources) ? parsed.sources : [],
    escalation,
    relatedTopics: Array.isArray(parsed.relatedTopics) ? parsed.relatedTopics : [],
  };
}

The normalization code may look defensive, but it's the difference between a demo that works sometimes and a production system that works always. Claude will occasionally return steps as objects instead of strings, or wrap escalation in a nested object. This parser handles all of it.

Step 6: Wire It Into the API

Now update resources/Api.js to use the real orchestrator:

import { Resource, tables } from 'harperdb';
import crypto from 'crypto';
import { config } from '../lib/config.js';
import { orchestrate } from '../lib/orchestrator.js';

export class Api extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    const context = this.getContext();
    const authHeader = context.headers.get('authorization');
    if (!authHeader) {
      const err = new Error('Authorization required');
      err.statusCode = 401;
      throw err;
    }

    const { query, mode = 'ask' } = data ?? {};
    if (!query?.trim()) {
      const err = new Error('Missing "query" in request body');
      err.statusCode = 400;
      throw err;
    }

    // Run the full orchestration pipeline
    const result = await orchestrate(query.trim(), { mode });

    // Save to audit trail
    const queryId = crypto.randomUUID();
    await tables.IncidentQuery.put({
      id: queryId,
      query: query.trim(),
      response: JSON.stringify(result),
      sourcesUsed: result.sources?.map((s) => s.url ?? s.title) ?? [],
      createdAt: new Date().toISOString(),
    });
    result.queryId = queryId;

    return result;
  }
}

Test it:

curl -X POST http://localhost:9926/Api \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)" \
  -d '{"query": "How does replication work in our platform?", "mode": "ask"}'

You should get back a structured JSON response with a summary, sources from your actual data systems, and actionable steps — all grounded in your real documentation and internal knowledge.

What Just Happened

Let's appreciate what you built. In about 400 lines of code:

Six data sources searched in parallel in 2-4 seconds
Claude synthesizes all results into a structured, cited response
Knowledge base is checked first for instant cache hits
Negative feedback is checked to avoid repeating past mistakes
Unconfigured sources are skipped gracefully, no code changes needed
Rate limiting is handled with exponential backoff
Response parsing handles every edge case Claude might throw at you

The orchestrator is the most valuable part of the entire system. Everything else, Slack, the web UI, the feedback loop, is just plumbing to get queries in and results out. This is the brain.

What's Next

In Part 4, we wire this into Slack, slash commands, @-mention handling, threaded conversations, and rich Block Kit responses with feedback buttons. Your team will be able to ask Harper Eye questions without leaving the tool they already live in.

Build Your Own AI Ops Assistant — Part 2: Foundation

Stephen Goldberg — Wed, 25 Feb 2026 23:16:35 +0000

This is Part 2 of a 6-part series on building your own AI-powered internal assistant. Part 1 covers the why and architecture overview.

Harper Setup, Schema & Your First Endpoint

In Part 1, you saw what Harper Eye can do, structured incident analysis in Slack, expert routing, and a self-improving knowledge base. Now we build the foundation. In this part, you'll go from zero to a running application with a database, vector-indexed tables, and a deployed, live REST API. Takes about 30 minutes.

By the end of this article, you'll have the infrastructure for everything you see in this dashboard:

Step 1: Install Harper & Create Your Project

If you want to develop locally first (recommended), install Harper:

npm install -g harperdb

Now scaffold your project:

mkdir harper-eye && cd harper-eye
npm init -y

Update your package.json with the dependencies we'll need:

{
  "name": "harper-eye",
  "version": "1.0.0",
  "description": "AI-powered ops assistant — searches internal tools, synthesizes with Claude, learns from feedback",
  "type": "module",
  "engines": {
    "harperdb": "^4.4"
  },
  "dependencies": {
    "@anthropic-ai/sdk": "^0.39.0",
    "@slack/web-api": "^7.8.0"
  },
  "scripts": {
    "start": "harperdb run .",
    "dev": "harperdb dev .",
    "deploy": "npx -y dotenv-cli -- harperdb deploy . restart=rolling replicated=true"
  }
}

npm install

Notice what's not there: no Express, no database driver, no vector DB client, no Redis. Harper handles HTTP routing, data storage, and vector search natively. Your node_modules stays small.

Step 2: Define Your Database Schema

This is where Harper shines. Instead of writing migrations or setting up a separate database, you define your schema in a single GraphQL file and Harper creates everything automatically.

Create schema.graphql in your project root:

type IncidentQuery @table @export {
  id: ID @primaryKey
  slackUserId: String
  slackChannelId: String
  conversationId: String
  query: String
  response: String
  sourcesUsed: [String]
  createdAt: String
  resolvedAt: String
}

type KnowledgeEntry @table @export {
  id: ID @primaryKey
  query: String
  queryEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  answer: String
  sources: [String]
  originalIncidentId: String
  approvedByUserId: String
  approvedAt: String
  channelId: String
  useCount: Int
  lastUsedAt: String
  negativeCount: Int
}

type NegativeFeedback @table @export {
  id: ID @primaryKey
  queryId: String
  originalQuery: String
  queryEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  category: String
  details: String
  userId: String
  channelId: String
  knowledgeEntryId: String
  createdAt: String
}

type AuditLog @table @export {
  id: ID @primaryKey
  action: String
  userEmail: String
  query: String
  details: String
  source: String
  queryId: String
  entryId: String
  createdAt: String
}

type SlackExpertise @table @export {
  id: ID @primaryKey
  userId: String
  userName: String
  topic: String
  topicEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  channelId: String
  channelName: String
  answerCount: Int
  questionCount: Int
  lastActiveAt: String
  indexedAt: String
}

type SourceRelevance @table @export {
  id: ID @primaryKey
  query: String
  queryEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  mode: String
  sourceSignals: String
  createdAt: String
}

type CodeKnowledge @table @export {
  id: ID @primaryKey
  repo: String
  filePath: String
  fileName: String
  fileType: String
  content: String
  contentEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  description: String
  tier: Int
  lastIndexedAt: String
  commitSha: String
  url: String
}

Let me break down what's happening here, because there's a lot of power in these few lines:

@table tells Harper to create a database table. @export exposes it as a REST endpoint automatically. You get full CRUD (PUT, GET, DELETE) for free.

@primaryKey designates the unique identifier.

@indexed(type: "HNSW", distance: "cosine"): this is the magic line. It tells Harper to build an HNSW (Hierarchical Navigable Small World) vector index on that field, using cosine distance for similarity. This is the same algorithm that powers vector search in dedicated vector databases like Pinecone, but it's built into Harper natively. No separate service, no extra cost.

We have four vector-indexed fields across our tables:

Table	Vector Field	Purpose
`KnowledgeEntry`	`queryEmbedding`	Find similar past questions to return cached answers
`NegativeFeedback`	`queryEmbedding`	Find similar past complaints to avoid repeating mistakes
`SlackExpertise`	`topicEmbedding`	Find engineers who are experts on similar topics
`CodeKnowledge`	`contentEmbedding`	Find relevant source code by semantic meaning
`SourceRelevance`	`queryEmbedding`	Learn which data sources matter for which query types

That's five HNSW indexes running natively inside Harper, with zero additional infrastructure.

Step 3: Configure Your Application

Create config.yaml in your project root. This is Harper's application configuration, it tells Harper what to load and how to serve it:

# Load environment variables from CONFIG.env
loadEnv:
  files: 'CONFIG.env'

# Enable the REST API for all exported resources
rest: true

# Read GraphQL schemas to define database tables
graphqlSchema:
  files: 'schema.graphql'

# Load JavaScript modules as HTTP endpoints
jsResource:
  files: 'resources/*.js'

# Serve static web UI files at /app/
static:
  files: 'site/**'
  urlPath: 'app'
  index: true
  extensions: ['html']

Five lines of YAML and you have: environment variable loading, REST API generation, schema-driven database creation, custom HTTP endpoints, and static file serving. No boilerplate. No middleware chain. No app.use() rituals.

Now create your CONFIG.env file for secrets (this file is gitignored, never commit it):

# Claude (Anthropic)
CLAUDE_API_KEY=sk-ant-your-key-here
CLAUDE_MODEL=claude-sonnet-4-20250514

# Gemini (Embeddings)
GEMINI_API_KEY=your-gemini-key-here
GEMINI_EMBEDDING_MODEL=text-embedding-004

# Slack
SLACK_BOT_TOKEN=xoxb-your-bot-token
SLACK_VERIFICATION_TOKEN=your-verification-token
SLACK_SIGNING_SECRET=your-signing-secret

# Confluence (optional — add your data sources as you have them)
CONFLUENCE_BASE_URL=https://yourcompany.atlassian.net
CONFLUENCE_EMAIL=your-email@company.com
CONFLUENCE_API_TOKEN=your-confluence-token

# Zendesk (optional)
ZENDESK_SUBDOMAIN=yourcompany
ZENDESK_EMAIL=your-email@company.com
ZENDESK_API_TOKEN=your-zendesk-token

# Datadog (optional)
DATADOG_API_KEY=your-datadog-api-key
DATADOG_APP_KEY=your-datadog-app-key

# GitHub (optional)
GITHUB_TOKEN=ghp_your-github-token
GITHUB_ORG=your-org-name

Don't worry if you don't have all of these yet. Harper Eye is designed to gracefully skip any source that isn't configured, you can start with just Claude and one data source, then add more over time.

Create a config loader at lib/config.js:

/**
 * Config loader — reads values from environment variables (loaded from CONFIG.env).
 * Each getter throws if the value is missing, so unconfigured sources
 * are caught gracefully by the orchestrator's try/catch blocks.
 */
function env(key) {
  return () => {
    const val = process.env[key];
    if (!val) throw new Error(`Missing env: ${key}`);
    return val;
  };
}

export const config = {
  claude: {
    apiKey: env('CLAUDE_API_KEY'),
    model: () => process.env.CLAUDE_MODEL || 'claude-sonnet-4-20250514',
  },
  gemini: {
    apiKey: env('GEMINI_API_KEY'),
    embeddingModel: () => process.env.GEMINI_EMBEDDING_MODEL || 'text-embedding-004',
  },
  slack: {
    botToken: env('SLACK_BOT_TOKEN'),
    verificationToken: env('SLACK_VERIFICATION_TOKEN'),
    signingSecret: env('SLACK_SIGNING_SECRET'),
  },
  confluence: {
    baseUrl: env('CONFLUENCE_BASE_URL'),
    email: env('CONFLUENCE_EMAIL'),
    apiToken: env('CONFLUENCE_API_TOKEN'),
  },
  zendesk: {
    subdomain: env('ZENDESK_SUBDOMAIN'),
    email: env('ZENDESK_EMAIL'),
    apiToken: env('ZENDESK_API_TOKEN'),
  },
  datadog: {
    apiKey: env('DATADOG_API_KEY'),
    appKey: env('DATADOG_APP_KEY'),
  },
  github: {
    token: env('GITHUB_TOKEN'),
    org: () => process.env.GITHUB_ORG || '',
  },
};

This pattern is intentional. Each config getter is a function that throws if the value is missing. That means the orchestrator can wrap source calls in try/catch and gracefully skip any source that isn't configured. You never have to comment out code or manage feature flags; just add or remove environment variables.

Step 4: Build Your First Resource Class Endpoint

Resource Classes are how you define HTTP endpoints in Harper. If you've used Express, think of them as route handlers, but with a few important differences.

Create the resources/ directory and add a health check endpoint:

mkdir -p resources lib mcp site

Create resources/HealthCheck.js:

import { Resource } from 'harperdb';

export class HealthCheck extends Resource {
  static loadAsInstance = false;

  async get() {
    return {
      status: 'ok',
      service: 'harper-eye',
      timestamp: new Date().toISOString(),
    };
  }
}

That's it. That's a complete HTTP endpoint. The class name HealthCheck becomes the URL path /HealthCheck. The get() method handles GET requests. The return value is automatically serialized to JSON.

Critical rules for Resource Classes (these will save you hours of debugging):

Always use static loadAsInstance = false: this tells Harper to create instances per-request. Without this, you'll get confusing behavior.
Instance methods, not static methods: write async get(), not static async get(). The static methods are Harper's internal entry points.
The first argument to post() is NOT the request body — it's the "target" (resource context). The body is the second argument:

// ✅ CORRECT
async post(target, data) {
  const body = data; // This is your POST body
}

// ❌ WRONG — target is not the body
async post(request) {
  const body = request.body; // This won't work
}

Headers come from this.getContext(), not from a request parameter:

async post(target, data) {
  const context = this.getContext();
  const authHeader = context.headers.get('authorization');
}

To return errors, throw with a statusCode property, don't construct a Response object:

// ✅ CORRECT
const err = new Error('Unauthorized');
err.statusCode = 401;
throw err;

// ❌ WRONG
return new Response('Unauthorized', { status: 401 });

Step 5: Build the REST API Endpoint

Now let's build the endpoint that the web dashboard will use. This accepts questions via HTTP POST and returns AI-generated answers.

Create resources/Api.js:

import { Resource, tables } from 'harperdb';
import crypto from 'crypto';
import { config } from '../lib/config.js';

export class Api extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    const context = this.getContext();

    // Basic auth check (we'll build proper auth later)
    const authHeader = context.headers.get('authorization');
    if (!authHeader) {
      const err = new Error('Authorization required');
      err.statusCode = 401;
      throw err;
    }

    const { query, mode = 'ask' } = data ?? {};

    if (!query?.trim()) {
      const err = new Error('Missing "query" in request body');
      err.statusCode = 400;
      throw err;
    }

    // For now, return a placeholder — we'll wire in the orchestrator in Part 3
    const queryId = crypto.randomUUID();
    const result = {
      queryId,
      summary: `Received your question: "${query}" — orchestrator coming in Part 3!`,
      sources: [],
      steps: [],
    };

    // Save the query to the audit trail
    await tables.IncidentQuery.put({
      id: queryId,
      query: query.trim(),
      response: JSON.stringify(result),
      sourcesUsed: [],
      createdAt: new Date().toISOString(),
    });

    return result;
  }

  async get() {
    // Return recent queries for the dashboard
    const queries = [];
    for await (const entry of tables.IncidentQuery.search({
      sort: { attribute: 'createdAt', descending: true },
      limit: 20,
    })) {
      queries.push({
        id: entry.id,
        query: entry.query,
        createdAt: entry.createdAt,
      });
    }
    return { queries };
  }
}

Notice how we're using tables.IncidentQuery directly, no connection strings, no query builders, no ORM. Harper tables are imported directly from the harperdb module and are available anywhere in your application code. You put() to write, get() to read by ID, and search() to query.

Step 6: Run It Locally

Start the Harper dev server:

npm run dev

Harper will read your config.yaml, create the tables defined in schema.graphql, and start serving your Resource Classes. You should see output like:

Harper is running
Application loaded from /path/to/harper-eye

Test your health check:

curl http://localhost:9926/HealthCheck

{
  "status": "ok",
  "service": "harper-eye",
  "timestamp": "2025-02-24T10:30:00.000Z"
}

Test the API endpoint:

curl -X POST http://localhost:9926/Api \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)" \
  -d '{"query": "How does replication work?", "mode": "ask"}'

{
  "queryId": "a1b2c3d4-...",
  "summary": "Received your question: \"How does replication work?\" — orchestrator coming in Part 3!",
  "sources": [],
  "steps": []
}

Your data is already being stored. Check it:

curl http://localhost:9926/IncidentQuery \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)"

You'll see the query you just sent, stored in the database with a UUID, timestamp, and the response. This is the audit trail — every question your team ever asks Harper Eye gets recorded here.

Step 7: Deploy to Harper Fabric

Local development is great, but let's get this live. Sign up for Harper Fabric at fabric.harper.fast if you haven't already, then create a cluster.

Create a .env file (also gitignored) with your deploy credentials:

CLI_TARGET=your-cluster.your-org.harperfabric.com:9925
CLI_USERNAME=your-username
CLI_PASSWORD=your-password

Deploy:

npm run deploy

That's it. Harper deploys your application code, creates the tables from your schema, loads your environment variables, and starts serving. The deploy takes about 5-8 seconds for a restart.

Your application is now live at:

https://your-cluster.your-org.harperfabric.com/

Test it:

curl https://your-cluster.your-org.harperfabric.com/HealthCheck

What You Have So Far

Let's take stock. In about 30 minutes, you've built and deployed:

A database with 7 tables, including 5 HNSW vector indexes for semantic search
A REST API with automatic CRUD for all tables, plus two custom endpoints
An audit trail that records every query
A production deployment on Harper Fabric with a public HTTPS endpoint

Your project structure looks like this:

harper-eye/
├── config.yaml          # Harper app config
├── schema.graphql       # Database tables + vector indexes
├── CONFIG.env           # Secrets (gitignored)
├── .env                 # Deploy credentials (gitignored)
├── package.json
├── lib/
│   └── config.js        # Config loader
├── resources/
│   ├── HealthCheck.js   # GET /HealthCheck
│   └── Api.js           # POST /Api (query endpoint)
├── mcp/                 # (empty — data source wrappers in Part 3)
└── site/                # (empty — web UI in Part 6)

Total lines of code: about 120. Total external services: zero (Harper handles everything). Total monthly cost so far: whatever your Harper Fabric plan costs — which is as low as free.

What's Next

In Part 3, we build the AI brain, the orchestrator that searches all your data sources in parallel, feeds results to Claude, and returns structured, cited responses. That's where Harper Eye goes from "a database with an API" to "the most useful tool your engineering team has ever had."

We'll build:

The MCP tool wrappers for Confluence, Zendesk, Datadog, and GitHub
The embedding generation pipeline using Gemini
The parallel source fetcher
The Claude orchestration loop with structured JSON output
The response parser with fallback strategies

It's about 400 lines of code, and it's the most satisfying code you'll write all year.

I Built an AI Ops Assistant That Replaced $2,400/month in SaaS — In 3 Days, for Under $100/month

Stephen Goldberg — Wed, 25 Feb 2026 18:09:26 +0000

Last month, one of our engineers typed this into Slack:

/incident what would cause basic authentication to fail even with a properly formed authentication header when the user is present and the roles are correct

Nine seconds later — before anyone else had even read the message — Harper Eye posted a full incident analysis in the thread. It had identified the root cause (a misconfigured authorizeLocal setting in harperdb-config.yaml), cited three internal sources (our documentation, codebase, and Slack history), listed specific remediation steps, and @-mentioned the five engineers with the most expertise in authentication — by name, with clickable Slack handles.

No one had to context-switch. No one had to dig through four different dashboards. No one had to remember which runbook lived where. The answer was right there in the channel, with clickable links to every source.

That moment crystallized something I'd been feeling for a while: the most valuable internal tool your company will ever have is one that connects your existing knowledge and puts it in the hands of every engineer, instantly.

And you don't need to buy it from a vendor. You can build it yourself.

The Hidden Cost of Scattered Engineering Knowledge

If you run an engineering team, this scenario is your daily reality:
Your knowledge is scattered across six or more systems, documentation wikis, support tickets, monitoring dashboards, chat history, code repos, and incident postmortems. When something breaks at 2 am, your on-call engineer has to manually search each one, piece together context from fragments, and hope they find the right runbook before customers start complaining.

The real cost isn't the monthly SaaS bills (though we'll get to those). It's the compounding tax on every engineer, every day:

Time to resolution goes up. A junior engineer who joined last quarter doesn't know that "replication lag on us-east" was caused by a misconfigured sharding key three months ago. That tribal knowledge lives in a Slack thread they'll never find.

Expertise walks out the door. When your senior engineer leaves, their mental model of the system — the one that connects a Datadog alert to a Confluence runbook to a specific config file — goes with them.

Duplicate work compounds silently. The same question gets answered differently by different people in different channels. There's no feedback loop. No institutional memory.

Most companies try to solve this by buying SaaS tools. An AI chatbot here, an incident management platform there, a knowledge base product, a Slack integration. Before you know it, you're paying $2,000-5,000/month across multiple vendors, and none of them actually know your systems, your architecture, your people, or your internal terminology.

What Harper Eye Actually Does (And Why It's Different)

Harper Eye is not a chatbot with a Confluence integration. It's an AI orchestrator that treats your entire internal knowledge surface as a single searchable brain.

Here's what happens when someone asks Harper Eye a question, and I mean exactly what happens, because I'll show you every line of code in this series:

1. Instant Knowledge Base Check

Before anything else, Harper Eye checks if this question (or something semantically similar) has been asked and answered before. If a verified answer exists with high confidence, it returns it instantly, no API calls, no waiting. Sub-second response.

2. Parallel Multi-Source Search

If there's no cached answer, Harper Eye searches all your data sources simultaneously, not sequentially, not one-at-a-time. In parallel:

Internal documentation (Confluence, wikis, runbooks)
Support tickets (Zendesk — what are customers actually reporting?)
Monitoring data (Datadog — what's alerting right now? What do host metrics show?)
Source code (GitHub — what does the code actually do?)
Chat history (Slack — what have engineers said about this topic recently?)
Team expertise (Who on your team has answered similar questions before?)

This parallel search typically completes in 2-4 seconds across all sources. Here's what the pipeline timing actually looks like for a real query:

That's a real query, not a benchmark. 135 milliseconds for the embedding, 7 milliseconds for the knowledge base search, 3.5 seconds for all six data sources in parallel, and 5.5 seconds for Claude to synthesize. Nine seconds from question to fully cited answer.

3. AI Synthesis with Claude

All that raw data gets fed to Claude with a system prompt that is deeply specific to your company. Not generic advice — your architecture, your terminology, your config files, your API endpoints. Claude synthesizes it into a structured response:

Look at that response. It's not generic. It names the specific replication engine (Plexus), references the exact config file (harperdb-config.yaml), cites Slack conversations about a real customer deployment, and names five specific engineers to contact — all grounded in actual internal data, not LLM hallucination.

4. The Knowledge Loop (This Is the Magic)

After every response, engineers can give feedback — thumbs up or thumbs down. When someone marks an answer as helpful, it gets stored in a vector-indexed knowledge base. The next time a similar question comes in, Harper Eye returns that verified answer instantly.

When someone marks an answer as not helpful, that negative signal gets stored too. If an answer accumulates enough negative feedback, Harper Eye automatically degrades it — forcing a fresh search instead of returning stale information. Eventually, bad answers get purged entirely.

Your AI assistant gets smarter every time your team uses it. No retraining, no manual curation. Just a feedback loop that compounds institutional knowledge over time.

5. Expert Routing — It Knows Your People

This is the feature that no vendor tool offers. Harper Eye indexes your Slack conversations to build a map of which engineers answer questions about which topics. Not who's assigned to a team — who actually demonstrates expertise by helping others.

That's a live skills matrix with 35 engineers across 17 topics, built entirely by mining Slack history. When an incident requires escalation, Harper Eye doesn't guess; it tells you exactly who to loop in, based on who's actually answered questions about that topic before.

The Architecture: Surprisingly Simple

Here's the part that surprised me most: the entire system is about 700 lines of core logic running on a single platform.

The entire application runs on Harper; it's simultaneously the application server, the database, and the vector search engine. There's no separate Postgres for storage, no Pinecone for embeddings, no Redis for caching, no Express for HTTP. One platform, one deploy target, one bill.

Here's what the stack looks like:

Layer	Technology	Why
Runtime + Database + Vector Search	Harper (Fabric)	One platform handles HTTP endpoints, data storage, and HNSW vector indexing. No infrastructure to manage.
AI Synthesis	Claude (Anthropic API)	Structured JSON output, handles massive context windows for multi-source analysis.
Embeddings	Google Gemini	768-dimension vectors for semantic search. Free tier covers most usage.
Data Sources	Direct HTTP APIs	Simple `fetch()` calls to Confluence, Zendesk, Datadog, GitHub, and Slack APIs.
Frontend	Vanilla HTML/CSS/JS	Static files served by Harper. No build step, no React, no webpack.

Proof This Works: Real Numbers from Production

This isn't a prototype. Harper Eye has been running in production at Harper, serving a team of 35 engineers. Here are the real numbers:

222 queries answered. 29 knowledge base articles built automatically from positive feedback. 24 feedback items shaping future responses. 12+ engineers actively using it across the team. All of that from a system that costs us less than $100/month to run.

The Cost Breakdown That Changes the Math

Here's what a typical "AI-powered internal knowledge" stack costs from vendors:

Tool	Monthly Cost	What You Get
Guru / Glean / Dashworks	$800-2,000	AI search across your tools
PagerDuty AIOps add-on	$500-1,000	AI incident triage
Incident.io or similar	$400-800	Incident management + Slack bot
Vector DB (Pinecone, etc.)	$70-200	Embedding storage
Total	$1,770-4,000/mo	Separate tools that don't talk to each other

Here's what Harper Eye costs to run:

Resource	Monthly Cost
Harper Fabric (1 instance)	~$25
Claude API (Anthropic)	~$30-50 (depends on query volume)
Gemini embeddings	~$0 (free tier)
Confluence/Zendesk/Datadog APIs	$0 (included in existing subscriptions)
Total	~$55-75/mo

That's not a typo. Under $100/month to replace thousands in SaaS, with a system that's more integrated, more specific to your company, and gets smarter over time.

The gap is even wider when you factor in what vendors can't do: search your actual Slack history, know which engineers are experts on which topics, learn from your team's feedback, or speak in your company's specific terminology.

Why I Built This Instead of Buying It

I want to be direct about the motivation. I'm a technical co-founder at Harper. I could have justified buying a tool; there are plenty of good ones out there, and the budget would have been approved.

But here's the thing: every SaaS tool you buy is someone else's opinion about how your problem should be solved. It's built for the average customer, not for your specific architecture, your specific team, your specific knowledge topology.

Harper Eye knows that our replication engine is called Plexus (not NATS, that's deprecated). It knows that "Custom Functions" is a legacy term, and the current pattern is Resource Classes. It knows that when someone asks about "Fabric," they mean our managed cloud platform, not a generic concept. And when it escalates an incident, it doesn't tell you to "contact the on-call team" — it @-mentions @Kris and @David by name, because it knows from Slack data that they've answered 28 and 32 questions about clustering, respectively.

No vendor product will ever know that. And more importantly, you know things about your company that no vendor ever will. The question isn't whether AI can help your team, it's whether you want that AI to work for a vendor, or for you.

Let's build it.

What We're Building in This Series

Over the next five parts, you'll build your own version of Harper Eye from scratch. Every line of code is real, tested, and production-ready. By the end, you'll have a system that does everything you've seen in the screenshots above, running on your infrastructure, for your team.

Part 2: Foundation — Harper Setup, Schema & Your First Endpoint

Set up Harper, define your database schema with vector-indexed tables, and build your first Resource Class endpoint. By the end, you'll have a running application with a REST API.

Part 3: The AI Brain — Claude Orchestrator & Multi-Source Search

The core of the system. Build the orchestrator that searches all your data sources in parallel, feeds results to Claude, and returns structured, cited responses. This is where the real magic happens.

Part 4: Slack Integration — Slash Commands, Events & Rich Responses

Wire it into Slack so your team can use slash commands, @-mention the bot, and get beautifully formatted responses with feedback buttons — all inside the tool they already live in.

Part 5: The Knowledge Loop — Vector Search, Feedback & Self-Improvement

Build the system that makes Harper Eye smarter over time. Embedding generation, semantic similarity search, feedback collection, knowledge degradation, and expert routing.

Part 6: Ship It — Deploy to Production & What to Build Next

Deploy to Harper Fabric, walk through the real cost breakdown, and explore extensions like PagerDuty webhooks, a web dashboard, and automated knowledge capture from Slack threads.

Each part stands on its own — you'll have a working, deployed system after every article. But together, they build something that fundamentally changes how your team operates.

Prerequisites

To follow along, you'll need:

Node.js (v20+ recommended)
Harper Fabric account (free to sign up at fabric.harper.fast)
Anthropic API key (for Claude — console.anthropic.com)
Slack workspace where you can create an app
At least one data source to connect (Confluence, Zendesk, Datadog, or GitHub — even one is enough to start)

No prior experience with Harper is required. If you can write JavaScript and have used a REST API, you have everything you need.

8 Ways to Improve Application Performance

Margo McCabe — Thu, 10 Aug 2023 18:09:20 +0000

Intro

Application performance is critical for delivering a fast and responsive user experience. Slow performance, or high latency, can lead to frustrated users and lost revenue for the organization.

From a high level, application latency refers to the delay between the user's request and the application's response. Latency can not only impact the overall user experience and decrease engagement, but it can be a costly and complicated problem as well. App performance also impacts overall efficiency. Slow applications can slow down business processes, leading to decreased productivity and increased costs. All of these factors can reduce a company’s ability to compete in the market. In today’s world, customers expect fast and responsive applications. Businesses that are unable to deliver on this expectation risk losing customers to competitors who offer a better user experience.
‍
By implementing strategies and technologies that enable fast response times, organizations can streamline processes, improve productivity, and reduce spend.

What causes application latency or performance issues?

Often we take advantage of just how much is happening behind the scenes to deliver lightning fast responses, but we do notice immediately when there is a lag of more than just a few milliseconds. So what's causing this latency? Slow app performance can be caused by a variety of factors, including:
‍

Network issues: Slow network connectivity or high latency can cause app latency. This is particularly true for mobile apps, which often rely on cellular data connections that can be slower and less reliable than Wi-Fi connections.
Server issues: If an app relies on a server or cloud-based infrastructure, slow server response times or server downtime can cause app latency.
Inefficient code: Inefficient code can lead to slow app performance. This can be caused by factors such as poor algorithms, excessive use of CPU or memory resources, or inefficient use of network resources.
Too much data: If an app is processing or transmitting too much data, it can cause latency. This can be caused by factors such as large files, excessive requests, or inefficient data processing.
Device issues: Older or less powerful devices may struggle to run some apps, leading to app latency.
External factors: External factors such as interference or environmental factors can also contribute to app latency. For example, if an app relies on GPS data, poor GPS signal or interference can cause latency. ‍

In his article on Geo Distributed Databases, Jacob Cohen talks about some of the latency-sensitive applications where high latency quickly leads to poor user experience. Examples include Gaming, IoT, Robotics, and Autonomous Vehicles. Other examples of companies that clearly need to prioritize fast performance to keep users happy are apps like Netflix, Google, Uber, and Instagram.

Here are 8 ways to reduce application latency:

As with any solution, the specific techniques and strategies that will work best will depend on the specific needs and requirements of your application or organization. Below are some of the most common ways to improve your app's performance and keep your users happy:
‍

Use a distributed database or caching system to store and retrieve data more efficiently.
Optimize the network infrastructure to improve data transfer speeds and reduce bottlenecks.
Use a content delivery network (CDN) to distribute static and dynamic content to users more quickly.
Implement data compression techniques to reduce the amount of data that needs to be transferred.
Use load balancing and other techniques to distribute workloads across multiple servers and reduce the amount of time that each request takes to complete.
Optimize the application code to improve the efficiency and performance of the application.
Use in-memory databases and other types of high-speed storage to improve data access times.
Monitor and analyze application performance metrics to identify and address bottlenecks and other performance issues.

‍
Here at HarperDB, we talk a lot about latency, as it's one of the major challenges that we solve for organizations across the globe. It’s easy to see that application performance is important for user experience, business efficiency, customer satisfaction, and competitiveness. It’s highly beneficial to take the time to address the root cause of the problem as soon as possible, because by prioritizing fast application performance, organizations can improve their bottom line and build a strong brand image.

HarperDB: Staking a Claim as the Only Globally Distributed Edge Data Platform

Margo McCabe — Thu, 02 Feb 2023 16:41:21 +0000

Like most companies in general and especially tech startups, HarperDB has overcome many challenges to get to where we are today– which is the most exciting place we’ve been as a company thus far. We’ve learned that we were a couple years too early to market, and that being too early can surprisingly be more difficult than being too late. Over the years, we have explored different solutions and industries to find the right product-market fit, which has been a major roadblock. We learned from and built upon these experiences, for example if we hadn’t ventured into IIoT, HarperDB would not be what it is today in terms of scale and distributed capability. Due to perseverance, we have now positioned HarperDB as the only Globally Distributed Edge Data Platform, providing unprecedented value to our customers through avenues such as Edge Computing, AI/ML, APIs, and third-party integrations.

Considered a lean startup, HarperDB has achieved much more than competitors with much less funding under our belt. With 100K total HarperDB deployments and 10K community members/subscribers, our team continues to expand the product to meet the dynamic needs of our innovative customers. 2022 was HarperDB’s highest year in sales yet, and we expanded our enterprise partnerships and customer base, now working with companies like Lumen, Akamai, Google, Verizon, and Equinix to bring an end-to-end solution to the market.

More Reason to Celebrate

HarperDB is thrilled to be announced as one of Built in Colorado’s Best Places to Work, ranked at number two on the Best 2023 Startups list. Staying true to our core values as a company, our leadership has ensured that this continues to be a safe, fair, and enjoyable place to work, even through stages of rapid growth.
HarperDB’s CEO, Stephen Goldberg, was just named the 2023 Bill Daniels Ethical Leader of the Year. Now in its eighth year, the award recognizes local leaders that demonstrate remarkable integrity and ethics in business, exemplifying the ethical principles practiced by cable television pioneer Bill Daniels.
We just released HarperDB 4.0, which integrates with open-source connectivity platform NATS to enable limitless data distribution. Learn more in this SiliconAngle article.

As we wrap up 2022 and gain momentum for 2023, I sat down with HarperDB’s CEO & Co-Founder, Stephen Goldberg, for a brief Q&A:

What are HarperDB’s current strategic initiatives?

We started the company with the idea of trying to make the easiest database in the world that is also massively scalable. As we evolve, we still want to solve that, but we can also provide a lot of value driving low latency use cases because HarperDB is uniquely positioned as a fully distributed platform that allows for quickly distributing data globally anywhere on earth. So for now we are focused on distributed capabilities as well as performance.

Looking ahead, we will be focused on improving HarperDB as a platform. The product is extremely performant, easy to use, and highly distributable as is, but it could still use work as a platform in making developers’ lives easier so that they can have a global application platform in one place. This ties into edge computing because right now you can have HarperDB anywhere in the world, but in the future, we want that to be an entirely seamless process so that you can deploy globally with the click of a button.

Why is HarperDB focused on Edge Computing and how is the edge helping organizations that we work with?

Originally we were focused on IIoT and IoT, which is actually different from edge computing. IoT is really about the things (devices, etc.) in the field. While HarperDB has the ability to continue playing a role in that, where we can add the most value is at the edge– a step back from IoT. We are taking what is also the cloud and making it more local, closer to the end user, and much more distributed.

We accidentally built the most distributed database in the world that is uniquely horizontally scalable, so we found that our technology is really meaningful in these use cases where it can be deployed anywhere.

Telecom companies for example have noticed that HarperDB can be deployed anywhere in their network and solve a lot of problems around edge computing. Interestingly, people think that 5G makes edge computing easier, but in a lot of ways, it can make it more difficult. A lot of companies are dealing with major congestion issues due to different pipes of data going in and out from the cloud and now to and from the edge. This complex congestion in the data pipeline causes high latency and high cost. Because HarperDB can make your data available at the edge, edge computing can become real. You can build your entire application on HarperDB, so now you really have everything you need, along with the ability to use 5G and other globally distributed infrastructure now available.

What projects are you most excited about right now?

We’ve talked a lot about one of our customers named Edison Interactive. They were struggling with high latency and lack of real-time decision making, and we were able to simplify the transition to a distributed architecture, ultimately reducing latency from 5 seconds down to 20 milliseconds or less. (You can learn more about this case study on our website or on AWS’s blog)

We’re currently exploring similar use cases in industries like gaming, media, manufacturing, oil and gas, etc. Some of this we can’t quite talk about yet, but big things are happening. Overall, I’m not as excited about one specific area as much as being excited about the convergence of all these things and all these ideas that we’ve been talking about for a while finally becoming real.

What are some of the adoption challenges that enterprises face with Edge Computing?

There are some challenges around organizations taking the patterns that they’re comfortable with in the cloud and applying that to the edge. Security is obviously one that can also become interesting. It’s really important to understand that the edge, while similar to cloud, can be quite different.

The biggest challenge with implementing an edge strategy is the interoperability of the different partnerships that you need in place.

AWS made cloud so easy because they have everything you need consolidated into one package. At the edge, there is no dominant player enabling you to put your whole stack in one place, so organizations must put different solutions and partnerships together to achieve their end goal. With HarperDB, one large benefit is that we can run anywhere, on anything, and it doesn’t matter who’s above or below us– you get the same experience anywhere.

Is there space for AI and Machine Learning in Edge Computing? How does it all tie together?

We’re really excited about AI and machine learning (ML) on the edge. We’re already working on some projects in this space, Kevin from our team built a machine learning use case with HarperDB using TensorFlow for a customer. Machine learning is awesome for understanding your data after the fact in the cloud, but at the edge, you can do real-time ML and personalized content experiences and personalized interactions. ML and AI can require huge amounts of compute and huge amounts of data, but if you distribute that, you can do really exciting things such as having human interaction-like levels of data at the edge. By distributing to the edge you reduce the barrier to entry, reduce latency and cost, etc.

How is HarperDB partnering with other organizations to make Edge Computing real?

We’re partnered with Verizon, we worked with them on the Edison use case. Companies like Akamai are helping us distribute our technology globally. HarperDB is also partnering with a lot of cloud providers to run on their edge offerings with Wavelength and MEC (Multi-Access Edge Computing). These partnerships are exciting because to deliver a globally distributed platform, it takes more than just one piece of the puzzle, so these companies allow us to deliver a robust solution to our customers.

Predictions related to Edge, Cloud, and IoT moving forward in the next 5-10 years?

All of these siloed areas are moving towards a more powerful convergence. Machine learning and AI is one discipline, you also have data science, 5G, and different edge offerings. Then there’s a different sector of edge computing without 5G, which is much more distributed using existing technologies like containerization, Kubernetes, and other fabric-level technologies, to create a highly distributed edge. You also have blockchain, and so on. So we have this fairly fragmented set of disciplines which are beginning to converge and overlap. It’s too siloed right now but there is a gravitational pull that’s moving them towards each other.

Once these disciplines become more overlapped, we’ll begin to see more incredible use cases coming to fruition and the future becomes real. You have robotics, real-world human interaction with true AI (not just ML), and personalized interaction. For example, imagine walking into a store or hotel room and the entire experience is tailored to you, autonomous vehicles, etc… All of these different moving parts coming together will make these ambitions real, and it’s exciting, to say the least.

How to Build a Machine Learning Recommendation Engine w/ TensorFlow & HarperDB

thelogicdudes — Wed, 24 Aug 2022 18:49:00 +0000

This article explains how to create a recommendation system using HarperDB and Custom Functions.

With HarperDB we are able to deploy a machine learning model to servers located on the edge of the Internet to provide users with content recommendations. We have two examples to demonstrate how this can be done: a book recommender and a song recommender.

An Intro to Recommendation Systems

Recommendation engines are the most universally used type of machine learning. We interact with them everywhere, and it has become the driving force for how we explore new content. And while some are less than perfect, even the least of them are miles ahead of the previous methods we had available to find new things.

Take shows and movies for example. Do you remember the preview channel… staring at it for hours, trying to guess which titles would be worth further exploration? The same is true for audio content, where we were limited to radio stations to find new music. And books were even more difficult. I'd walk along the bookstore aisles picking random books to read their summary before finally finding one to take a chance on.

All of these methods did work, but they limited us. We could only truly explore the content which was made available by the system at hand - such as the preview channel, radio station, and bookstore shelves. But there is so much more content out there, especially today. So instead of having to choose from the limited options presented by the Best Sellers section at the bookstore, how does one find new content that’s based on their individual interests?

Recommendation Systems!

What is a Recommendation System? How do they work?

Recommendation systems are exactly what they sound to be; a system that provides recommendations. You give it an idea about what you like, and it points you in the direction of other things you might like.

There are many ways to set up such a system. For example, if we're talking about products, a system could be built that finds all of the users who bought the same product as you, and then displays the most common products among that group. This could be expanded by adding more items to the input. Such as moving from a single product to three products that you bought, grouping together all of the users who also bought those three items, and then displaying the most common other products among that group.

What if we're including ratings with the items of interest? For example, if many people are watching one particular movie, does that automatically mean it’s a good movie to recommend? And if not, then how would we prevent it from appearing in the recommendations? When including ratings and reviews, we could set up a filter to remove any content with a bad review to ensure the group of items we're aggregating always contains positive content. This is a great starting point. It works for many cases where you have a dataset of items and user interactions with those items.

The challenging part arises from the evolving nature of content. How would a song, book, or product that's new get into the output of the recommendation system?

Using Machine Learning

While we could keep adding manual filters to the content by sorting the ratings and weighting interactions with new items, there would be areas where we fall short. Especially when creating binary gates such as a positive/negative rating threshold to determine which items are included in the output.

This is where machine learning takes over. Using libraries such as TensorFlow Recommenders with Keras models, it's easy to shape the data in ways that will allow the items and users to be viewed and compared in a multidimensional perspective. Qualitative features such as item categories and user profile attributes can be mapped into mathematical concepts that can be quantitatively compared with one another, ultimately providing new insights and better recommendations.

Using a machine learning pipeline like this also allows the models to be continuously trained by taking the results of the previous model, the successfulness of those recommendations to drive user interactions, and using that data to create better models. I recently wrote an article about using TensorFlowJS and HarperDB for Machine Learning if you’d like to learn more.

HarperDB Recommenders

One of the awesome features of living on the edge is the ability to connect users to new insights with low latency and minimal traffic. With HarperDB Custom Functions, we can deploy a model to multiple nodes in a cluster, allowing the closest and most available to return the result to the user.

Let’s look at two example projects to demonstrate how this would look when providing recommendations to a user based on provided content to create a song recommender and a book recommender.

To ensure the examples are reusable and straightforward, these projects are the frontside of the process, where we're serving an already trained model. In production we would connect a second piece to this puzzle where we track user interactions in a central location, train new models, and deploy them back out to the edge with HaperDB.

HarperDB Song Recommender

Github repo: HarperDB/song-recommender

In our Song Recommender example, a user can find three of their favorite songs in the UI, and then the recommendation system returns other songs they're likely to enjoy.

There's a dataset called The Million Song Dataset that contains very detailed information on over one million songs, ranging from audio analysis to the location of the artist.

There's a nice subset of that data called The Echo Nest Taste Profile Subset which is a list of users, songs, and play counts. Each row in the data contains a userid, songid, and playcount. This is what we used to build the model for this project.

Training the Model

We created a Two-Tower model, which is a design where two latent vectors are created, compared during training to shape the embedding layers, and compared during inference to find the best match.

Tower One is the Query Tower. This is essentially the input to the equation, which in this case is three songs that the user selects.

Tower Two is the Candidate Tower. In this case it's the users in the dataset.

We find the user in the dataset most similar to the user using the UI, and then provide their most played songs as recommendations.

To look at the training specifics, here's the training notebook.

The basic steps that are taken are as follows:

Find the most played songs for each user
Create a query/candidate pair of three of those songs and the user ([songs], user)
Build two models and setup TFRS metrics for training
Feed the query/candidate pair into the model
When training is complete, create a new model that wraps the above two
Apply the BruteForce method to extract the best candidate match provided an input

Generating the Data

# create a dictionary of inputs and outputs
dataset = {'songs': [], 'user': []}
for user, songs in users_songs.items():
    for _ in range(len(songs) * 5):
        # randomly select n_songsns_in from the user's isbns
        selected_songs = np.random.choice(songs, n_songs_in)
        # add them to the inputs
        dataset['songs'].append(selected_songs)
        # add the user to the output
        dataset['user'].append(user)

Building the Models

# create the query and candidate models
n_embedding_dimensions = 24

## QUERY
songs_in_in = tf.keras.Input(shape=(n_songs_in))
songs_in_emb = tf.keras.layers.Embedding(n_songs+1, n_embedding_dimensions)(songs_in_in)
songs_in_emb_avg = tf.keras.layers.AveragePooling1D(pool_size=3)(songs_in_emb)
query = tf.keras.layers.Flatten()(songs_in_emb_avg)
query_model = tf.keras.Model(inputs=songs_in_in, outputs=query)

## CANDIDATE
user_in = tf.keras.Input(shape=(1))
user_emb = tf.keras.layers.Embedding(n_users+1, n_embedding_dimensions)(user_in)
candidate = tf.keras.layers.Flatten()(user_emb)
candidate_model = tf.keras.Model(inputs=user_in, outputs=candidate)

Creating the TensorFlow Recommenders Task

# TFRS TASK SETUP
candidates = dataset.batch(128).map(lambda x: candidate_model(x['user']))
metrics = tfrs.metrics.FactorizedTopK(candidates=candidates)
task = tfrs.tasks.Retrieval(metrics=metrics)


## TFRS MODEL CLASS
class Model(tfrs.Model):
    def __init__(self, query_model, candidate_model):
        super().__init__()
        self._query_model = query_model
        self._candidate_model = candidate_model
        self._task = task

    def compute_loss(self, features, training=False):
        query_embedding = self._query_model(features['songs'])
        candidate_embedding = self._candidate_model(features['user'])
        return self._task(query_embedding, candidate_embedding)

## COMPILE AND TRAIN MODEL
model = Model(query_model, candidate_model)
# load model weights - this is to resume training
# model._query_model.load_weights(weights_dir.format('query'))
# model._candidate_model.load_weights(weights_dir.format('candidate'))

model.compile(optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.1))
model.fit(dataset.repeat().shuffle(300_000).batch(4096), steps_per_epoch=50, epochs=30, verbose=1)

Deploying the Model

To deploy this model, the final step in the above notebook is converting it to TensorFlowJS.

From there, we add it to the Custom Function's directory along with the logic in recommend.js which converts the user's three favorite songs into tensors which are provided to the model.

The output of the model is a reference to the most similar user to the one using the UI.

The top songs for that user are then returned and displayed in the UI as our recommendations.

Creating an input tensor and getting results

 if (!this.model) {
   const modelPath = path.join(__dirname, '../', 'tfjs-model', 'model.json');
   this.model = await tf.loadGraphModel(`file://${modelPath}`);
 }

 const inputTensor = tf.tensor([songIdxs], [1, 3], 'int32')
 const results = this.model.execute(inputTensor)
 const r0 = await results[0].data()

HarperDB Book Recommender

Github repo: HarperDB/book-recommender

In our Book Recommender example, a user can find three of their favorite books in the UI, and then the recommendation system returns other books they're likely to enjoy.

There's a dataset on Kaggle that includes a list of users, books, and ratings for about 250,000 different titles

Training the Model

We again created a Two-Tower model, which is a design where two latent vectors are created, compared during training to shape the embedding layers, and compared during inference to find the best match.

Tower One is the Query Tower. This is essentially the input to the equation, which in this case is three books that the user rated highly (a 5 and above out of 10).

Tower Two is the Candidate Tower. In this case it's the users in the dataset.

We find the user in the dataset most similar to the user using the UI, and then provide their highest rated books as recommendations.

To look at the training specifics, here's the training notebook.

The basic steps that are taken are as follows:

Find the top rated books for each user
Create a query/candidate pair of three of those books and the user ([books], user)
Build two models and setup TFRS metrics for training
Feed the query/candidate pair into the model
When training is complete, create a new model that wraps the above two
Apply the BruteForce method to extract the best candidate match provided an input

Generating the Data

# create a dictionary of inputs and outputs
dataset = {'isbns': [], 'user': []}
for user_id, isbns in user_isbns.items():
    # use 5x the number of isbns gathered for the user
    # this ensures a larger amount of training data
    for _ in range(len(isbns) * 5):
        # randomly select n_isbns_in from the user's isbns
        selected_isbns = np.random.choice(isbns, n_isbns_in)
        # add them to the inputs
        dataset['isbns'].append(selected_isbns)
        # add the user to the output
        dataset['user'].append(user_idxs[user_id])

Building the Models

# create the query and candidate models
n_embedding_dimensions = 24

## QUERY
isbns_in_in = tf.keras.Input(shape=(n_isbns_in))
isbns_in_emb = tf.keras.layers.Embedding(n_isbns+1, n_embedding_dimensions)(isbns_in_in)
isbns_in_emb_avg = tf.keras.layers.AveragePooling1D(pool_size=3)(isbns_in_emb)
query = tf.keras.layers.Flatten()(isbns_in_emb_avg)
query_model = tf.keras.Model(inputs=isbns_in_in, outputs=query)

## CANDIDATE
isbns_out_in = tf.keras.Input(shape=(1))
isbns_out_emb = tf.keras.layers.Embedding(n_users+1, n_embedding_dimensions)(isbns_out_in)
candidate = tf.keras.layers.Flatten()(isbns_out_emb)
candidate_model = tf.keras.Model(inputs=isbns_out_in, outputs=candidate)

Creating the TensorFlow Recommenders Task

# TFRS TASK SETUP
candidates = dataset.batch(128).map(lambda x: candidate_model(x['user']))
metrics = tfrs.metrics.FactorizedTopK(candidates=candidates)
task = tfrs.tasks.Retrieval(metrics=metrics)


## TFRS MODEL CLASS
class Model(tfrs.Model):
    def __init__(self, query_model, candidate_model):
        super().__init__()
        self._query_model = query_model
        self._candidate_model = candidate_model
        self._task = task

    def compute_loss(self, features, training=False):
        query_embedding = self._query_model(features['isbns'])
        candidate_embedding = self._candidate_model(features['user'])
        return self._task(query_embedding, candidate_embedding)

## COMPILE AND TRAIN MODEL
model = Model(query_model, candidate_model)
# load model weights - this is to resume training
# model._query_model.load_weights(weights_dir.format('query'))
# model._candidate_model.load_weights(weights_dir.format('candidate'))

model.compile(optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.1))
model.fit(dataset.repeat().shuffle(300_000).batch(4096), steps_per_epoch=50, epochs=30, verbose=1)

Create the BruteForce Comparison

# create the index model to lookup the best candidate match for a query
index = tfrs.layers.factorized_top_k.BruteForce(model._query_model)
index.index_from_dataset(
    tf.data.Dataset.zip((
      dataset.map(lambda x: x['user']).batch(100),
      dataset.batch(100).map(lambda x: model._candidate_model(x['user']))
    ))
)
for features in dataset.shuffle(2000).batch(1).take(1):
    print('isbns', features['isbns'])
    scores, users = index(features['isbns'])
    print('recommended users', users)

Deploying the Model

To deploy this model, the final step in the above notebook is converting it to TensorFlowJS.

From there, we add it to the Custom Function's directory along with the logic in recommend.js which converts the user's three favorite books into tensors which are provided to the model.

The output of the model is a reference to the most similar user to the one using the UI.

The top rated books for that user are then returned and displayed in the UI as our recommendations.

Recap

And there you have it, that’s how you can create a recommendation system with HarperDB Custom Functions.

These machine learning models were pre-trained for these examples, as they do take 12+ hours to reach their most accurate state. In a production environment, there would most likely be a single instance responsible for continuously training the model and distributing it out to the other nodes on the edge.

Go ahead and launch a HarperDB instance with Custom Functions where you can pull in one of these repos and get recommendations for new songs and books to check out. If you get an interesting result that you enjoy, please let us know!

Thanks for reading,

–Kevin

MEAN Stack Alternative: More Power with Less Stack

Jaxon Repp — Fri, 12 Aug 2022 20:47:00 +0000

Technology stacks like LAMP, MEAN, and MERN have been a boon to developer productivity. With just four components you have everything you need to develop web applications quickly and easily. And while these stacks do provide a robust framework out of the box, they are not without complexity, and aren’t always designed to scale. As a result, organizations often find themselves transitioning away from their V1 application built in a four part stack to a simpler configuration. Reducing complexity is a great way to improve stability, which, in turn, is a great way to reduce costs.

Here at HarperDB, we refer to this process of simplification as Collapsing The Stack. Fewer moving parts means less can go wrong, after all. In this article, we’ll outline challenges with the four-part stack and propose some alternatives that can save money, reduce complexity, and help solve some of the issues of performance at scale.

A Primer on Four-Part Stacks

Below are some popular four-part stacks and their (4) components:

MERN

MongoDB
Express
React
NodeJS

MEAN

MongoDB
Express
Angular
NodeJS

LAMP (edited, because it’s been a while, and Javier Lopez is right in the comments)

Linux
Apache
MySQL
PHP

With each of these stacks, you have a database (Mongo or MySQL), a client-side framework (Angular or React), a server-side framework (Express or Laravel), and a core language runtime (NodeJS or PHP). With these four parts, you can rapidly generate the UIs, APIs, and persistence layers of your application.

But Wait, There’s More To Consider! Like, A LOT!

As I mentioned in the introduction, these are easy solutions to understand, and easy to deploy. But there are some drawbacks that one should consider when planning for the future.

Performant applications, in the simplest terms, have to do two things: respond quickly enough to keep users happy, and be able to handle all the users they just made so happy. Four-part stacks involve two links that can hinder your application’s ability to meet these two objectives: the API call to your database, and a database that can only support a limited number of users with the provisioned resources.

Performant APIs, Happy Users

First, let’s take a look at the API call to your database, and see how we can simplify that process and make it more performant.

In the old days, we used things like stored procedures and user-defined functions to put business logic into the database server. This was considered by most people (other than DBAs) to be a Very Bad Idea, indeed. The way we accomplished it at HarperDB was to integrate the API server into the database itself, a feature we call Custom Functions. By using server-side JavaScript and core methods that eliminate the second hop from the API to the database, you reduce latency and make users happier, without relying on a standalone server-side framework like Express or Laravel. Basically, we integrated a pre-configured Fastify server into HarperDB and let you define your routes and handlers- no more dependencies, configuration, or command line configuration. Plus, happy users experiencing lower latency is always a win.

Keeping Users Happy At Scale

Next, let’s look at supporting all your happy users. Scale is a challenge for applications in a lot of ways, but the biggest hurdle is the cost of all that success. Distributing load is a challenging prospect, but lots of tooling has arisen to solve it like AWS lambdas and other “serverless functions”. But when it comes to traditional four-part stacks, you now have a lot of moving parts to duplicate in order to handle additional users. That doesn’t seem like much fun, and it’s not.

The main obstacle is the fact that distributing data is WAY harder than distributing applications. And unless you plan for it upfront, you’re most likely going to find that you’re sending all your users to a cluster of databases for which you keep having to increase resources, a concept we call vertical scaling. Vertical scaling follows an exponential cost curve- a fancy way of saying that 2GB of RAM costs more than twice what 1GB of RAM costs… not much more, but once you start to need multiple VCPUs, multiple cores, and 128GB of RAM to support your happy users, your CFO is going to be… less happy. I think we can all agree that unhappy CFOs are no fun at parties, or anywhere else.

Truly distributed applications scale horizontally. That is to say, if one application node costs $10, and supports 1,000 happy users, then you just keep adding nodes for every 1,000 users you need to keep happy. The challenge of the four part stack is that none of the usual databases are designed to scale like that. We built HarperDB to solve exactly that problem. Happy users don’t stay happy for very long if you make them wait, and it stinks when that problem only reveals itself once you’re actually starting to see lots of users.

The HAN Stack: As Happy As A Wookie At Home For Christmas

Application development isn’t JUST about keeping users happy. It’s also about keeping developers, DevOps, and CFOs happy. And staying performant as you grow without having to swap out your database or API framework is one of the easiest design decisions you can make now to stay successful down the road.

To that end, we call our efforts to make this whole process less stressful the HAN Stack.

You may have already guessed that the HAN stack stands for HarperDB, Angular, and Node.js. Angular is an obvious choice for building enterprise-based applications because of its extensive built-in functionalities and community support. Node.js is an incredibly versatile and user-friendly JavaScript runtime. And HarperDB consolidates the API and database, while being orders of magnitude more performant.

With applications and data deployed as one, you can focus more on your application logic and less on your architecture. We’re still big fans of old-school HarperDB stacks like the HERN stack. But with the introduction of Custom Functions, you no longer need to be weighed down by a four-part stack as you grow.

How is This Possible?

With Custom Functions, you can now define your own API endpoints within HarperDB. Your custom endpoints will interact directly with core operations, such as “insert”, “search_by_hash”, “update”, and more. This eliminates the second connection from your API to the database in favor of a direct connection to the data layer, which usually reduces round-trip API latency by around 50%. Imagine how many happy users you’ll have! Plus, you can host your compiled Angular UI as a static resource right inside Custom Functions, so you can actually build your entire application in one place. It’s an indisputable win-win.

With more traditional last-generation tech stacks, you would need to deploy and host your backend API code on additional servers, then have them call out to the database layer for database needs. As Jake Cohen, Director of IT & Operations at HarperDB, mentioned in his article, “HarperDB is collapsing the stack onto a single server, which eliminates virtually all network latency. This frees up headroom for achieving higher throughput from a single server. Capitalizing on HarperDB’s already powerful horizontal scalability, this means you can now distribute both your APIs and your database to the edge.”

Simple World, Simple Life

There are many benefits to a simplified tech stack, both for the developer and the organization as a whole. A stack like the HAN stack makes developers’ lives easier, creating the opportunity to utilize modern technologies while reducing the number of programming languages or frameworks that they have to work with. This also enhances collaboration, allowing more developers from diverse backgrounds to work together on the same project.

For the organization, the shorter the stack, the easier it is to adapt to crisis and evolve when needed. Most modern infrastructures are moving towards a “collapse the stack” model, and this allows the organization to be agile while reducing costs and latency at the same time. The leaner your mission-critical stack, the lower your risk exposure. With the right technologies in place, you'll benefit from both a cost and functionality perspective. Simplicity is key, and that’s why HarperDB’s tagline has been “Simplicity without Sacrifice” from the beginning.

Click here to spin up an instance for free.