DEV Community: epilot

Our Entire Company Ships Code Now. 40 PRs from Non-Engineers in 60 Days.

Viljami Kuosmanen — Mon, 13 Apr 2026 13:06:30 +0000

In February, I opened our entire codebase to the company. PMs, designers, projects, customer success, and support all got access to 219 repositories and over a million lines of production code. I handed them powerful coding agents and told them to start contributing.

I've been writing about product engineering for years. The core idea: engineers should understand the whole product, the customer, the business problem. I wanted to test the reverse. Can non-engineers contribute to the codebase if you give them the right tools?

Two months of real data delivered a clear answer: yes. And it may be the most powerful shortcut I've found to building capable, business-aware product teams who truly care about what they ship.

Here is what that looked like at epilot: 629 merged PRs in 60 days. 40 PRs were from 11 non-engineers. All teams have at least one person who contributed to our codebase with Claude Code.

We're not alone in seeing this shift. Stripe ships over a thousand agent-produced PRs per week. Superhuman uses Codex to let product managers contribute lightweight code changes without pulling in an engineer (except for code review). Intercom gave Claude Code to everyone in the company, over 1,000 non-engineers (PMs, designers, support, marketers, and more), and more than 300 became active weekly users.

This post is a detailed, honest look at how we set it up, what actually worked, the mistakes we made, and the deeper lessons that emerged.

The starting point

epilot is a B2B SaaS platform for energy companies. We run a fully remote team of 40+ product engineers with no engineering managers and no dedicated platform team. Everyone owns end-to-end ownership. We already ship to production 150+ times per week on AWS serverless.

Our entire engineering team was already using AI coding agents daily. Claude Code, Codex, Cursor. Engineers had adopted them fast and the productivity gains were obvious. The question I wanted to answer next: what happens when you expand codebase access via coding agents to the rest of the company? PMs, designers, customer success, support. Not as a one-off hack day, but as a fully supported, production-grade workflow.

The infrastructure that turned agents into teammates

Claude Code and Codex still only ship connectors to GitHub, and our entire codebase lives in GitLab. So we created a single GitHub repo that bootstraps a developer sandbox with a shallow copy of all 219 repositories. This allows both AI agents or engineers to spin up a fully functioning developer environment with everything cloned & tools set up within minutes.

1. Agents live in Slack

We created #dev-squad-agents alongside our regular squad channels. Anyone could join via Claude Code, Codex, or GitHub Codespaces. I ran three one-hour onboarding sessions to showcase the tooling and make sure everyone had the required access: one for design, one for support, one for go-to-market. The PMs didn't need their own session. Their engineering teams had already invited them and they were eager to start.

This single decision drove adoption more than anything else.

2. One-click preview environments

Every PR automatically generates a live preview link in CI. Click it, see the change, test it immediately. No local setup required.

For engineers this is convenient. For non-engineers it is essential. If they cannot see and interact with their change, they can't meaningfully contribute.

3. Deep institutional context

We built an indexed knowledge base called how-to-everything. It contains our API patterns, backend conventions, code style, CI/CD and testing strategy, full business context, domain terminology, and our entire RFC archive.

Stripe solved the same problem by connecting agents to 400+ internal tools. We chose comprehensive indexing. The insight is identical: agents without your company’s context produce garbage. Agents with it produce work that feels native.

4. Design system in context

Our Volt UI component library is baked into the agent prompt. Claude now generates frontend code that respects the correct components, spacing, colours, and patterns from the first try. Consistent UI without forcing a designer to review every small change.

Claude as a coworker: the Slack pattern

Claude quickly became the most active “team member.” People tag @claude the same way they tag colleagues, and they expect results just as fast.

How engineers use it

Simple tasks look like: “@claude fix EPI-5660.” A PR appears.

More powerful uses scale across the organization:

“@claude which microfrontends still use the legacy editor?” → instant audit of all 219 repos.
“@claude migrate 13 backend repos from file-client to lambda-adapter.” → completed in one session.
Mass upgrades across 20+ repositories: one message triggers 6 parallel sub-agents that handle changes, tests, and PRs.

What once took a full day of tedious cross-repo work now happens in hours. Agents worked weekends. Engineers didn't have to.

How non-engineers use it

They speak in plain product language:
- “@claude disable drag and drop of entity table columns for embedded tables.”

“@claude when creating a new family in the Label Builder, the drawer should close automatically.”

Claude figures out the right repositories and implementation details. Non-engineers don't need to know our tech stack or repo structure. They describe the desired outcome.

Production impact

We added Claude to #alerts-production on day two. It investigates incidents, reads logs, traces errors, and proposes fixes before most engineers open the dashboard. When a bulk action hammered Elasticsearch with 100 parallel requests, Claude designed an org-level job queue with concurrency limits that cut peak load by 90%. Engineers started replying “another one bites the dust” to resolved alerts.

Within a week, Claude felt like a trusted junior-to-senior coworker: fast executor, deep historian, and reliable pair programmer. Usage spread organically to every squad.

What people actually built

Non-engineer contributions (40 merged PRs from 11 people)

Feature work: One PM built auto-tagging for file uploads (a feature he had spec’d three months earlier). Another shipped 13 PRs on messaging (bulk email templates, sidebar toggles, drawer behaviour, i18n). A go-to-market colleague delivered entire features, like pivot tables for displaying time-series data in epilot tables, that are now live for customers.
Daily improvements: Customer success shipped portal config changes they handle every day. Designers and support added FullStory tracking and started prototyping new UI flows.
Translations: Native speakers cleared 116 German translation improvements with almost no engineering involvement.
UX via vibecoding: A designer shipped a UX improvement to our Flow Hub before heading into the weekend: when creating a new Flow, users now get prompted to enter a name first. She described it as "vibe coding" and the change reduced friction in a way she'd been wanting to fix for a while.

These were not toy changes. Several were meaningful, customer-facing updates. One PM even asked Claude to create a demo video of the feature it just built, so he could share it with the team. The full cycle from spec to demo, handled through conversation with an agent.

Engineer contributions (589 merged PRs)

Engineers use agents across the full lifecycle. Writing RFCs and technical designs. Implementing multi-service features. Running tests. Shipping to production. Auditing feature flag usage across 15 repos and cleaning up stale flags. Investigating production incidents from alert channels before anyone opens a dashboard.

Some of the most valuable uses are research and diagnostics. "Which services listen to the file upload event?" "How many microfrontends still use the legacy editor?" "Audit all listSchemas calls and upgrade to the new summary API where safe." These are questions that used to take half a day of grepping across 219 repos. Claude answers them in minutes.

GitHub Codespaces turned out to be a key piece for engineers too. Running Claude Code in auto-mode in a cloud sandbox means you can kick off a multi-repo migration and let it run without risking your work laptop. No local state to corrupt, no accidental changes to your working branch. The sandbox spins up with the full codebase mirrored and Claude works through the task autonomously. Engineers now run 6 parallel sub-agents in Codespaces for bulk upgrades across 20+ repos: package bumps, migration scripts, test updates, PRs opened. What used to be a full day of copy-paste across repos is done in one session.

The new collaboration model

Non-engineer describes the problem in business terms → Claude implements → engineer reviews for architecture and quality. The person closest to the customer now helps ship the solution. Feedback loops tightened dramatically.

A designer and engineer even deployed Agentation (a prompt-structuring tool) in 2.5 hours. The designer now uses it daily and calls it a “dream collaboration.”

Guardrails: the hard lesson

A few weeks in, one non-engineer with full GitLab push access bypassed our CI pipeline and deployed code to production without review. No damage, but it was exactly the wake-up call we needed.

We immediately tightened permissions: non-engineers and agents create PRs; engineers review and merge. No direct push to protected branches.

But that incident was just one piece. When you give AI agents to 50+ non-engineers, you need a proper governance framework. We built one.

What we had from day one

Sandboxed execution only. Agents run in Claude Code Web, Codex sandbox, or GitHub Codespaces. No production infrastructure access. No deployment outside CI.
Dedicated bot users. Limited API scopes. Code-level access only. Zero access to customer data, session tokens, or production databases.
Public channels only. All agent interactions happen in shared Slack channels. Engineers see everything and can intervene. No private DMs with agents.
Human responsibility. You trigger the agent, you own the output. No exceptions.

What we added after

We published a formal Agentic AI Tool Usage Guideline. The core principle: give the agent the minimum access it needs for the task at hand, and nothing more.

The guideline includes:

A three-tier data classification. Red (never share with AI): customer PII, session tokens, credentials, security documents. Yellow (anonymise first): user research transcripts, survey responses, usage metrics with identifiers. Green (safe): public docs, your own drafts, anonymised research, process docs. Hard line on red: no exceptions, no "just this once."
An approved connector matrix. Every tool combination (Claude + Figma, Claude + Atlassian, Claude + Miro, etc.) classified by risk level with specific conditions and recommended settings.
Prompt injection awareness. Use a separate browser profile for Claude browser extension. Don't paste content from untrusted sources into agent prompts. Disconnect MCP connectors when not in use.
Explicit dealbreakers. Leaking customer personal data, unauthorized code deployment, and feeding company data into model training are not allowed regardless of how useful the outcome might be.
GDPR and EU AI Act compliance. The guideline maps our practical rules to specific legal obligations: data minimisation (Art. 5), DPA requirements (Art. 28), AI literacy obligations (Art. 4), transparency requirements (Art. 50). We're a European company serving energy utilities. Regulatory compliance is table stakes.

Lesson: Lock down merge and push permissions before opening access. We got lucky. And build the governance framework early. Guardrails didn't slow adoption; they accelerated trust and usage.

What surprised us most

Adoption speed: Three one-hour onboarding sessions and people were off. Every team across the company had active contributors within three weeks.
Non-engineers shipping real features: I expected small fixes and translations. Instead, people from across the company delivered complete, production features.
Claude as institutional memory: Support now asks Claude product questions instead of hunting down engineers. It has become the fastest way to understand how anything works across five years of code history.

Beyond code

This is just one layer. My colleague Suresh built Vigilos, an internal BI agent that lets anyone query our data warehouse in natural language. Designers check average journey name lengths for UX decisions. Customer success identifies accounts with AI features disabled. Product managers run customer segment analysis in minutes.

We'll share that story soon.

Where this goes

Two years ago I published the Product Engineer Manifesto: engineers should care about the customer, the business outcome, and the whole product.

What these two months taught me is that the same mindset now applies in reverse. Give non-engineers the right tools, deep context, and sensible guardrails, and they will deliver.

We're still hiring product-minded engineers. That hasn't changed. Engineers at epilot own the full product lifecycle: customer understanding, architecture, quality, reliability, and shipping. Non-engineer contributions don't reduce what we expect from engineers. They amplify it. The entire organization now ships alongside them.

629 merged PRs in 60 days. 40 from non-engineers.

We're hiring all product roles who thrive on full ownership in an AI-native environment. Remote-first. epilot.cloud/careers

We Made Coding Agents Actually Reliable By Fixing One Thing

Viljami Kuosmanen — Wed, 04 Feb 2026 15:55:45 +0000

Last week, Vercel published research showing that giving coding agents a compact index of your documentation dramatically outperforms letting them search for answers on demand. Their eval results: 100% task success rate with the map approach, versus only 79% when agents had to actively look things up.

Same agent, same tasks, different approach to context. The difference between working and not working.

The insight clicked immediately. If we could give Claude Code reliable access to this institutional knowledge without forcing it to decide when to look things up, it would fundamentally change how people work in our codebase.

So we built it.

The Context Problem

Coding agents have a token limit. A ceiling on how much information they can process at once. Think of it like working memory. You can't hand Claude Code your entire codebase and documentation library upfront. It's too much.

The traditional solution is skills: the coding agent decides when it needs information and actively looks it up. "I need to know about authentication, let me search for that." Sounds reasonable. In practice it creates three problems:

Decision paralysis - the agent has to decide when to look up docs, and it often guesses wrong
Async delay - every lookup is a round-trip, breaks flow
Sequencing conflicts - exploring code vs. consulting docs creates timing issues

Vercel's approach flips this: give Claude Code a compressed index of what documentation exists and where to find it, before it starts work. The index is small enough to fit in context every turn, so it always knows what's available. When it needs details, it reads the specific file directly. No decisions, no lookups, no conflicts.

Why Compression Matters

The key innovation is compression. A full documentation tree - all the folder structures, file names, categories - takes significant space. Too much to include in every conversation turn.

The compressed index uses a simple pipe-delimited format that shrinks this by ~80%:

[epilot Docs Index]|root: ./.epilot-docs|00-general:{tech-stack.md,business-context.md,ci-cd.md}|01-apis:{api-design.md,calling-apis.md}|02-epilot360-microfrontends:{env-vars.md,local-dev.md}|...

That single line tells the agent:

Where the docs live (.epilot-docs/)
What categories exist (00-general, 01-apis, etc.)
What files are in each category

It's a table of contents, not the full book. But it's enough. The agent sees the map, understands what's available, and pulls specific files when needed. The decision-making load disappears.

Why This Matters

The accuracy improvement is significant (100% vs 79% task success), but it's not the real story.

The real story is what happens when you lower the barrier to contribution. Proper context enables the person closest to the problem to fix it, regardless of job title. Your PM can fix bugs. Designers can adjust component behavior. Support engineers can patch data issues. You remove bottlenecks. Context and authority live in the same person.

Coding agents are the equalizer. But they're only as good as the context you give them.

Most companies will throw Claude Code at their codebase and wonder why results are inconsistent. The agent hallucinates patterns. Makes incorrect assumptions. Writes code that doesn't match conventions.

The difference is context. Structured, compressed, always-available context about how your codebase works.

How to Build Your Own

The pattern is straightforward. Here's what we did at epilot:

Curate agent-friendly documentation - organize your internal knowledge: conventions, APIs, architectural patterns, framework usage, code style
Structure by domain - group related docs (general, backend, frontend, infrastructure, etc.)
Use descriptive filenames - the agent sees filenames in the compressed index before opening files. api-design.md is better than guidelines.md. error-handling.md is better than errors.md. Make filenames searchable and specific.
Automate updates - pull live data where possible (OpenAPI specs, schema definitions, framework docs)
Generate compressed index - use a simple format (pipe-delimited works well) that reduces the doc tree by ~80%
Embed in agent context - add the index to your CLAUDE.md or AGENTS.md file (the context files Claude Code reads)

One thing worth highlighting: we don't just include our internal documentation. We also package docs for the frameworks and libraries we heavily use - single-spa, openapi-backend, openapi-client-axios, i18next, and Volt UI (our custom design system). When Claude Code needs to know how i18next pluralization works or how to register a single-spa parcel, it already has the answer. No hallucination, no outdated Stack Overflow posts, just accurate framework documentation.

What This Enables

We're already seeing daily usage across the team. Developers context-switch between services faster. Non-engineers contribute directly instead of filing tickets.

But the real potential is broader: if compressed context improves coding agents for technical documentation, why stop there?

Runbooks and incident response - on-call engineers with instant access to procedures
Customer domain knowledge - support teams with context on product behavior
Business logic - product decisions and their rationale, preserved and accessible

The pattern is the same: curate the knowledge, compress it, embed it in context, let the agent work.

The Constraints Are Disappearing

For decades, contributing to a codebase required deep technical knowledge. You needed to understand the language, the frameworks, the architectural patterns, the implicit conventions. The barrier was high.

Coding agents lower it. Claude Code, Cursor, and similar tools don't replace engineers. They make technical knowledge more accessible. With the right tooling and the right context, a PM can fix bugs. A designer can adjust styling logic. A support engineer can patch data issues.

The question isn't whether this is possible. It's how fast you can adapt.

Organizations which enable broader contribution will move faster than those that don't. The tools exist. The research is clear. What's missing is execution.

Start Simple

You don't need to document everything upfront. Start with the knowledge that causes the most friction:

Code style conventions - how you write TypeScript, naming patterns, file structure
Common patterns - how you handle authentication, API calls, error handling
Framework specifics - non-obvious usage of your frameworks and libraries
Internal APIs - if you have OpenAPI specs, even better

Create a simple doc structure:

docs/
  00-general/
    code-style.md
    tech-stack.md
  01-apis/
    api-design.md
    calling-apis.md
  02-backend/
    error-handling.md
    database-patterns.md

Generate the compressed index (pipe-delimited format, one line per directory). Add it to your CLAUDE.md or AGENTS.md file - the context files that Claude Code and other coding agents read on startup. Done.

The compressed index approach works! 🎉 Vercel's research proved it: 100% task success versus 79% without it. We've validated it internally. Now it's about whether you'll adopt it before your competitors do.

Security at Scale: Our npm Incident Response Story

João Pinho — Wed, 10 Sep 2025 16:20:24 +0000

On September 8th, the npm ecosystem saw what is now called the largest supply-chain compromise in its history. Packages like chalk, debug, and ansi-styles — together downloaded billions of times every week — were hijacked and malicious versions published.

For any SaaS company with Node.js in its stack, this was a moment to pause and act.

At epilot, where we build cloud software for the energy market, we recognised that this had the potential to impact our systems. Here's how we responded.

🚨 First, what happened?

A phishing attack compromised the maintainer of several widely used packages.
Malicious versions were published and quickly spread via transitive dependencies.
The injected code was designed to hijack crypto wallet transactions, but the bigger story is: if attackers could publish once, they could publish anything. Additionally, if attackers could read/intercept our network requests, they would have access to our tokens and thus our platform on behalf of our users.

⚡ Our response — measured in hours

As soon as the incident was announced, our engineering team moved fast. Within minutes of posting the alert in our #dev-security slack channel, we had a small response team come together.
172 repositories scanned: we used GitHub Codespaces to access our centralized codebase and systematically audit our product ecosystem.
451 Node.js projects analyzed: every package.json across our domain groups (auth, billing, analytics, 360-portal, and more) was checked for the compromised packages and versions.
Automation helped: we cross-checked with our internal tooling and monitoring alerts to confirm no production build had pulled malicious versions.

Result: ✅ zero exposure found across all 451 dependency trees.

🛡 Practices that kept us safe

The npm incident highlighted the value of practices we already had in place:

Dependency monitoring with Corge → ongoing visibility into package versions and vulnerabilities across our entire ecosystem.
Masked logs in Datadog → all PII automatically obfuscated, ensuring sensitive customer data never leaks, even if dependencies misbehave.
LLM anonymisation → using Microsoft Presidio, we automatically detect and anonymise emails, phone numbers, IBANs, credit cards, and other PII before any data reaches AI models for our AI-powered features.

These aren't "nice-to-haves." They are part of how we build trust with customers in a regulated industry like energy.

💡 What we learned

Supply chain attacks are inevitable — being able to respond fast is what matters.
Modern tooling makes a difference — GitHub Codespaces let us audit 172 repositories and 451 Node.js projects in hours, not days.
Good hygiene compounds — monitoring dependencies, masking logs, and anonymizing LLM data aren't glamorous, but when incidents like this happen, they prove their worth.

🔭 Looking forward

Incidents like the npm compromise will continue. At epilot, we're committed not just to reacting fast, but to building resilient, privacy-first systems that earn customer trust every day.

Because in the energy market — where trust and compliance are everything — resilience is the real competitive advantage.

From Chaos to Clarity: Fixing Our Monorepo

kate astrid — Tue, 19 Aug 2025 11:45:37 +0000

Introduction: Why This Story Might Save You Some Headaches

If you’ve worked in a large codebase, you’ve probably seen this: the code runs, but every change takes far longer than it should. Updating one feature often breaks another. Adding a dependency means worrying about conflicts. Old build tools still linger, and no one feels safe upgrading packages that half the system depends on.

That’s exactly what we faced in our journey-monorepo, the main repository for our team. Dependencies were wired together in inconsistent ways, different teams had introduced multiple overlapping build tools, and some libraries hadn’t been touched in years. Proposing a cleanup felt like asking to redesign a city while people were still living in it — risky, disruptive, and easy to delay indefinitely.

So why did we bother?

Development had slowed to a crawl. Builds were taking too long, and waiting minutes just to see a change in the browser was killing productivity.
Every change felt risky. The dependency spaghetti meant a small tweak in one package could break multiple apps — often in ways we wouldn’t notice until much later.
Onboarding new developers was painful. It took too long for new team members to get up to speed, mostly because they had to learn multiple bundlers, testing setups, and outdated tooling quirks.
Technical debt was blocking new features. We were spending more time patching, debugging, and wrestling with the repo than building the actual features our users cared about.
Costs were adding up. Longer build times meant higher CI costs, and outdated packages made security updates and maintenance more expensive in the long run.

In short, the longer we waited, the more time, money, and momentum we were losing.

And, hey, now we get a good story out of it.

Journey-Monorepo in a Nutshell

Journey-monorepo is the central place on the epilot platform for everything related to journeys. What you see in the screenshot above represents two main apps: journey-app on the left side and journey-builder on the right. Journey-builder is responsible for configuring journeys, and journey-app for rendering them.
Like in many startups, the repo started out clean and well-structured, but as the product grew quickly, the codebase expanded in unplanned ways — things were added wherever they seemed to fit at the time.

The “BEFORE” Setup

Apps:
- journey-builder.               
- journey-app
- entity-mapping
- journey-view

Packages:
- blocks-configurators
- blocks-renderers
- journey-elements
- journey-utils
- journey-logics-common

Let me show you the rough map of dependencies we had step by step. We are interested in two main apps: journey-builder and journey-app, so I am going to dive deeper into their connections. It is not important which app or package does what. What IS important is to see how many connections we had to keep in mind every time we needed to change anything.

Ok, this is fine.
This is too.
Yeah, cool.
Packages already become tangled, let's keep going.
Ok, even more connections.
Well, this is just a delinquent madness.

You might say: Kate, come on, this is what happens in every big project - things become deeply intertwined, and it is a normal thing to experience.
I will answer: yes, but I haven't finished.

Let's now add the build tools to this table to have the full picture.

Well, what we have here:

Webpack;
Craco;
Tsup;
Tsx;
Tsdx.

I don't know about you, but I hadn't worked with half of them before I started working on this repo.

In the end, it was a web of “if you change this, you might break that” — and “that” could be anywhere.

How We Tackled It

We kicked things off with an RFC — basically, “let’s write this down so we don’t lose the plot halfway through.”
Our two big goals:

Unify the tooling so we weren’t juggling five different build setups.
Untangle dependencies so teams could work without stepping on each other’s toes.

And this is exactly what we did.

Step 1: Cut the Cord on Unnecessary Dependencies
The first thing we had to do was face the dependency jungle. Over time, different teams had introduced overlapping packages, and no one was quite sure what depended on what anymore. Cleaning this up gave us a chance to regain control.

Untangled all the apps and packages as much as possible. We mapped out how everything connected and then started cutting back the unnecessary links. This meant reducing cross-dependencies that caused fragile builds.
Moved blocks-configurators into journey-builder. These configurators were never meant to be used outside, so pulling them closer to where they belong simplified the dependency graph.
Moved blocks-renderers into journey-app. Renderers were core to the application itself, so consolidating them in the main app made the architecture more intuitive.
Deleted journey-utils and journey-elements. Journey-utils was a very small package with some utilities that were moved to other packages. And journey-elements _ were later replaced by _concorde-elements. You can read more about it in this article written by my partner in crime.

This was a painful step, but once the dust settled, the repo felt lighter and easier to reason about. We no longer had to worry about accidentally breaking other parts of the system when making small changes.

Step 2: Standardize the Tooling
After trimming the dependencies, the next big challenge was our tooling. Different parts of the repo used different build systems, which slowed everyone down and made onboarding new developers a headache.

Phased out Craco, Tsup, and Tsdx by switching to Vite. These older tools had been added over time, each solving a narrow need. Vite gave us a unified, modern setup with faster builds and simpler configuration.
Swapped Jest for Vitest — faster tests, modern setup. Jest had served us well, but it was slow and required custom patching to keep working. Vitest gave us near-instant feedback and worked seamlessly with Vite, making the dev experience far smoother.

By unifying the tooling, we not only sped up day-to-day work but also reduced cognitive load. Developers no longer had to remember which tool applied to which package — everything just worked the same way everywhere.

This is how the "AFTER" setup looks:

As you can see, the monorepo is now far more streamlined — there are significantly fewer dependencies crisscrossing between packages, and we’ve eliminated most of the variations in build tools. This makes the structure easier to understand, faster to work with, and much less fragile when changes are introduced.

After the Restructuring: What We Learned

When the dust settled, it wasn’t just that the dependency graph was cleaner — we were thinking about the repo differently.

Immediate Wins

Changing things stopped feeling dangerous. Before, touching a shared package felt like pulling a brick from a Jenga tower. Now, fewer connections mean less risk.
Builds and tests stopped feeling like a punishment. Vite’s dev server = instant feedback. Vitest shaved minutes off test runs. Devs started running tests more often just because they could.
Onboarding went from “ugh” to “okay.” New folks could get set up without learning the quirks of four bundlers first.

Slower Payoffs

- Tooling swaps aren’t magic. Some Webpack plugins didn’t have a Vite equivalent, so we had to rethink features or drop them.

- Vitest exposed bad tests. Some “passing” tests were only passing thanks to Jest’s quirks. Painful, but worth fixing.

Trade-Offs & Reality Checks

- Not everything got unified. A couple of apps still run on legacy setups because migrating them wasn’t worth the risk — for now.

- Shared packages need rules. We now ask “does this really need to be shared?” before creating new ones.

- Refactoring is never “done.” There is always room for improvement, we all know. that.

Tips If You’re Thinking About Doing This

Draw your dependency graph before you start — it’ll help convince others (and yourself) that the cleanup’s worth it.
Expect setbacks. Some PRs will break things. That’s normal.
Automate checks for unused or outdated packages — saves a ton of review time.
Celebrate small wins. Removing the last Webpack config deserved cake.

Final Thoughts

This wasn’t a quick tidy-up — it was a half-year-long deliberate overhaul alongside working on other features. But it’s made a massive difference to how we work: faster builds, fewer “don’t touch that” areas, and a general feeling that the repo is ours again, not a beast we’re afraid of.

If I had to wrap it all up in one bit of advice?

Keep it simple.

Every extra dependency, tool, or package you skip today is one less thing you’ll have to untangle in the future.

Building a Scalable React Component Library: Lessons From Concorde Elements

Adeola Adeyemo — Wed, 23 Jul 2025 14:02:23 +0000

Introduction

We recently embarked on the complete redesign of our Journeys, aiming for a modern, state-of-the-art look - Project Concorde. This wasn't just a UI overhaul; we sought the flexibility for future modifications, knowing that simply patching our existing UI, heavily reliant on Material UI, wouldn't suffice. We needed to build a new foundation from the ground up.

The full Project Concorde story is larger, but in this article, we'll dive into the story of @epilot/concorde-elements, the new component library born from that need, and how we built a system that not only powers our new interface, but also empowers our development team.

Why did we need a new component library?

Our existing component library, @epilot/journey-elements, was based on Material UI. While it served its purpose, our goal was to reduce our reliance on Material UI to gain several crucial benefits:

Reduce performance overhead from Material UI's style generation in favour of CSS modules
React version constraints tied to MUI releases.
Styling limitations blocking modern CSS features.
Remove the use of the Material UI theme object in saved custom Designs

The image below shows the overhead to create custom designs:

Why not use existing design systems?

We found that most off-the-shelf design systems are quite opinionated with their styling and theming. Our philosophy was that it's easier to replace a single unit than an entire factory. Our primary goal was to keep our new system simple and extensible.

The Challenge

The path forward was not without its hurdles:

We needed to build 37+ components with various potential variants.
All Journey blocks had to be migrated to the new design, and we anticipated new complexities during integration due to component usage.
We had to ensure that custom themes built with the previous design system worked seamlessly with the new one.

Phases of Developments

1: Pursuit of Purity

In the initial phase, our approach was to create every component from scratch. We desired control over every pixel and line of code. Our first significant task was migrating the Product Tile to enable a new "Recommended Product" feature. We began with the basics.

Our early Button component perfectly illustrates this "purity" approach—simple, direct, and completely self-contained.

/* An early Button.tsx */
import classes from './Button.module.scss';
import type { ButtonProps } from './types.ts';
// ...

export const Button = forwardRef<HTMLButtonElement, ButtonProps>((props, ref) => {
  // ...
  return (
    <button
      className={classNames('Concorde-Button', classes.root, ...)}
      {...props}
    >
      {/* ... */}
    </button>
  )
});

Next, we successfully built our Link, Card, TextField, StepperInput, Image, ImageStepper, MobileStepper, and the ThemeProvider. With these foundational components, we successfully released the "Recommended Product" feature, receiving excellent feedback.

Our next milestone involved migrating the Date field and adding a Time select, which led to the DatePicker component. The design for this component had unique custom requirements that made building it entirely from scratch impractical. After some research, we opted to extend an existing library, React DatePicker, with custom functionalities to suit our use cases.

The resulting DatePicker involved creating new sections like the Footer and Time Select, and replacing the TextField and Header. While functional, the process was cumbersome, and the result felt more like a series of patches than a cohesive part of our system. This experience was our wake-up call: our "from-scratch" purity, and even our one-off extension approach, was simply not scalable.

2: The Pivot - 'Headless UI & Purity' Hybrid

This realization prompted a strategic shift. Speed became as crucial as purity. This quest led us to a breakthrough: headless components. These were libraries that provided the complex logic, state management, and accessibility of common widgets, but shipped with absolutely no styles. This was our "aha!" moment.

After further research, we settled on two incredible libraries: Radix UI Primitives and MUI Base (which recently became Base UI). They offered us the best of both worlds. We could now build our components by composing these primitives and applying our own distinct design system on top.

Consider our Autocomplete component, for instance, it leverages a hook from MUI Base for its core logic, while we provide the entire UI.

// A simplified look at our Autocomplete component
import { useAutocomplete } from '@mui/base/AutocompleteUnstyled'
import { forwardRef } from 'react'

import { Input, Menu, MenuItem } from '..';
// ...

export const Autocomplete = forwardRef(function Autocomplete(props, ref) {
  // ...
  const {
    getRootProps,
    getInputProps,
    getListboxProps,
    getOptionProps,
    groupedOptions
    // ...
  } = useAutocomplete({ ...props })

  return (
    <div {...getRootProps(other)} ref={ref}>
      <Input {...getInputProps()} />
      {groupedOptions.length > 0 ? (
        <Menu {...getListboxProps()}>
          {(groupedOptions as any[]).map((option, index) => (
            <MenuItem {...getOptionProps({ option, index })}>{option.label}</li>
          ))}
        </Menu>
      ) : null}
    </div>
  )
})

We were no longer reinventing the wheel for every single component. This allowed us to focus our efforts on what truly made our library unique: our design and the developer experience. We only created components from scratch when absolutely necessary.

This hybrid model significantly accelerated the development of the remaining components.

Key Principles

Throughout the creation of our component library, we adhered to several core principles that ensured our team worked in unison. These principles were internally documented in our contribution guidelines and are outlined below.

A Consistent Component Structure

To keep our codebase organized and predictable, we established a standard folder structure for every component. For example,

/_ src/components/Button/
  ├── Button.tsx            // Logic
  ├── Button.module.scss    // Styles
  ├── Button.test.tsx       // Tests
  ├── types.ts              // Type definitions
  └── index.ts              // Exports

This simple convention meant that any developer could easily navigate to any component and immediately locate its logic, styles, and type definitions with reducing collaboration overhead.

// Button

...
<Component
   aria-disabled={isDisabled || variant === 'disabled'}
   className={classNames(
     'Concorde-Button',
     classes.root,
     variant && classes[`variant-${variant}`],
     variant === 'primary' && 'Concorde-Button__Primary',
     ...
     className
   )}
   style={customStyles}
   ...
 >
  ...
</Component>

// Card

<div
   className={
     classNames(
       'Concorde-Card', 
       classes.root, 
       className
     )
   }
   ref={ref}
   style={customStyles}
   {...rest}
/>

Notice the use of the Concorde prefix in the static HTML classes. This served as a foundational element for easy custom styling, a topic not covered in detail here.

Design Tokens & Theming

The next pillar was unifying our design system. We created a comprehensive set of global design tokens using CSS variables for every aspect of our UI: colors, spacing, typography, transitions, shape and more. This became the language of our design system.

By coding colors, typography and dimensions as CSS custom properties, we guaranteed every component would be visually consistent, utilizing the same set of values. We also enabled these values to be extended and customized externally as local variables, preventing hard-coded styles and ensuring maximum flexibility.

:root {
  --concorde-primary-color: #0070f3;
  --concorde-secondary-color: #ff7e1b;
  --concorde-font-family: 'Proxima-Nova', sans-serif;
  --concorde-spacing: 0.25rem;
}

Now, all our components can simply use color: var(--concorde-primary-color) or margin: var(--concorde-spacing). This setup dramatically simplified theming. For example, toggling the typography tokens automatically affects all text on the screen.

Beyond the tokens used internally, we also exposed custom tokens for each component. These external tokens provide powerful ways to extend a component's functionalities.

For example, the Button component has the following custom tokens:

const customColors: ButtonCSSProperties = {
    '--concorde-button-label-color': color,
    '--concorde-button-background-color': backgroundColor,
    '--concorde-button-hover-bg-color': hoverBgColor,
    '--concorde-button-active-bg-color': activeBgColor,
    '--concorde-button-gap': gap ? `${gap}px` : undefined
  }

As an example, the CSS styles below will specifically modify the Button and Card:

:root {
  /* Button styles */
  --concorde-button-label-color: #ffffff;
  --concorde-button-background-color: #ff7e1b;
  --concorde-button-gap: 0.5rem;

  /* Card styles */
  --concorde-card-background-color: #e34590;
}

Note the consistent use of the concorde prefix for the token naming for consistency and scoping.

All tokens (default and custom tokens) are thoroughly documented in Storybook and our developer documentation for clarity.

TypeScript: Our Shield and Guide

From day one, we committed to writing everything in TypeScript. More than just providing type safety, our type files became a form of documentation. We commented our types extensively, enabling any developer using a component to understand the purpose of each prop right from their IDE.

// An excerpt from our Autocomplete types.ts
export interface AutocompleteProps<
  // ...
>extends UseAutocompleteProps<T, Multiple, DisableClearable, FreeSolo> {
  /**
   * The label content.
   */
  label?: React.ReactNode
  /**
   * If `true`, the component is disabled.
   * @default false
   */
  disabled?: boolean
  // ... and so on
}

Composition: Building Blocks for Complex UI

A fundamental principle guiding our development was composition. Our base components were designed to be flexible building blocks. By combining them, we could construct more complex and specialized UIs without adding bloat to the core library. This allowed us to build sophisticated features by assembling simple, well-tested parts - a true "change one, change all" approach.

A simple Input could be composed to PatternInput, IbanInput, NumberInput and more complex components like a ProductTile could look be structured as follows:

// A conceptual ProductTile built with composition
import { Card, Image, Typography, Button } from '@epilot/concorde-elements'
import styles from './ProductTile.module.scss'

export const ProductTile = ({ product }) => {
  return (
    <Card className={styles.tile}>
      <Image src={product.imageUrl} alt={product.name} />
        <Typography as="h4">{product.name}</Typography>
        <Typography>{product.description}</Typography>
        <Button variant="primary" label="Add to Cart" />
    </Card>
  )
}

Developer Experience

Vite

Vite powered our local development with a lightning-fast dev server and HMR, greatly benefiting from our monorepo setup

Storybook: The Living Documentation

How could we ensure quality and consistency across our components? The answer was Storybook. It became far more than just a component gallery; it became the living, breathing heart of our project. For every component, we wrote stories that showcased all its variants and states.

// A story from Input.stories.tsx
import type { Meta, StoryObj } from '@storybook/react'
import { Input } from './Input'

const meta: Meta<typeof Input> = {
  title: 'Components/Input',
  component: Input
  // ...
}
export default meta

export const WithLabel: Story = {
  args: {
    label: 'Email Address',
    placeholder: 'Enter your email...'
  }
}

export const Disabled: Story = {
  args: {
    ...WithLabel.args,
    disabled: true
  }
}

Critically, every story also served as an accessibility test. We integrated the @storybook/addon-a11y addon, which runs automated accessibility checks on every story against WCAG standards. This proactive approach allowed us to catch issues with color contrast, ARIA attributes, and keyboard navigation right within our development environment.

Testing & Accessibility

To build components that last, they must be reliable. Alongside the real-time accessibility checks in Storybook, we built a robust testing foundation. We used Vitest as our test runner and React Testing Library to write unit and integration tests for every component. We also used vitest-axe and React Testing Library for more intricate accessibility checks.

These tests were not just about preventing regressions, they were about enforcing correctness. We tested component logic, ensuring that each part behaved exactly as expected under various conditions. This combination of automated accessibility checks and functional testing was crucial. It gave us the confidence to refactor, add features, and scale the library, knowing that our foundation of quality would hold strong.

These automated tests run in our CI to catch regressions early.

The Payoff: Scalability in Action

The true test of concorde-elements came as we started integrating it in our main application, specifically in the concorde-renderers package. The work of setting up tokens, using primitives, and embedding quality through testing and documentation paid off.

Developing new features for Project Concorde transformed from a chore into a delight. Need a Modal? Pull in the component. Need a complex form field? Compose it with our TextField, Autocomplete, and Button components. They all looked consistent and behaved predictably.

This became the foundation for more interesting features:

Custom CSS: custom styling for resulting Journeys
Consistent design for Custom Journey Apps using the published library

Conclusion

Building a component library is a journey. Ours taught us that a little upfront systematization goes a long way. The initial purity of building "from scratch" gave way to the pragmatic wisdom of building on top of a solid, accessible foundation.

This yielded a few key principles we now live by:

Start with design tokens: They are the bedrock of a consistent and scalable design system.
Embrace headless primitives: Don't reinvent the wheel for everything, but leverage existing, well-tested solutions for speed and robustness.
Make TypeScript non-negotiable: The safety and developer experience benefits are immeasurable.
Testing: A combination of unit, integration, and automated accessibility testing is crucial for a reliable library.
Build with composition in mind: Create simple, flexible blocks that can be assembled into complex UIs, promoting reusability and maintainability.
Document thoroughly: A living, tested, and accessible documentation hub aligns everyone and accelerates development.
Be adaptable: The perfect plan rarely survives contact with reality. Be ready to pivot and embrace better ideas.

The @epilot/concorde-elements library is more than just a collection of React components. It's a testament to our team's journey, a foundation for our product's future, and a system that helps us build better, faster, and more consistently. We hope our story can help guide you on your own path to building a scalable and resilient component library.

Resources

Cover Photo by Ryan Quintal on Unsplash

How We Integrate AI in epilot - Chapter 2: Serverless RAG w/ LangChain & Weaviate

Kerem Nalbant — Mon, 26 May 2025 08:38:42 +0000

Introduction

In the previous chapter, I shared how we began our AI journey at epilot by implementing AI Email Summaries, helping users reduce email reading time by up to 87%. Encouraged by that success, we're now stepping up our AI capabilities with Retrieval-Augmented Generation (RAG) to provide smarter, contextually aware email suggestions.

WHY?

As we aim to scale our commodity business, investing in AI is crucial—not just for growth, but to significantly upgrade our product’s capabilities. Commodity segments often have a high volume of repetitive customer service requests. Our users need quick, context-aware email suggestions that understand:

Previous communications and organizational knowledge
Company-specific communication styles
Tailored relationships with each customer

Although Large Language Models (LLMs) are powerful, they're limited when accessing recent or specific company data. Customizing LLM responses usually involves prompt engineering, RAG or fine-tuning. Fine-tuning is resource-intensive and complex, making prompt engineering with RAG our clear choice.

Our Solution: Retrieval-Augmented Generation (RAG)

We implemented a RAG-based solution to retrieve and provide relevant context from past email threads and eventually expand to external data sources like documents and websites. Long-term, organizations using epilot will have fully configurable, customized knowledge bases accessible to all future AI features and AI agents.

This allows our users to respond to customer emails faster, improving communication quality and efficiency. On the end customer side, it means quicker, more accurate, and better overall service.

See It in Action

An end customer emails about documentation requirements for a renovation plan (Sanierungsfahrplan).

The epilot user doesn’t waste time researching policies or manuals—they simply prompt our AI to generate reply in english.

Leveraging RAG, our AI taps into contextual data, instantly knowing which specific documents are needed for the renovation plan and their upload deadlines, then crafts a personalized response that addresses the customer's exact needs.

Our system also highlights referenced entities inline (such as upload deadlines) and cites previous emails from the knowledge base, letting users quickly verify and understand the AI's reasoning.

Solution Components

To build a secure, scalable RAG system in a serverless environment, we chose:

LangChain

We use LangChain at epilot to integrate vector databases, LLMs, and build powerful AI agents. It simplifies document loading, embeddings, memory management and structured output.

Weaviate

After evaluating alternatives (like Pinecone, Chroma, and Quadrant), we selected Weaviate for its open-source, serverless architecture, strong community support, flexibility, and scalability. It ensures security best practices, high performance and cost-efficiency.

Presidio

Security and data privacy are essential. Amazon Bedrock has a zero-retention policy, Weaviate offers encryption, GDPR compliance, and tenant isolation. But we needed an extra layer for handling sensitive PII data.
Presidio helps us redact this information before indexing, preventing AI hallucinations and protecting customer privacy.

LangSmith

LangSmith provides AI observability, performance monitoring, debugging, prompt management, and testing. It allows us to quickly iterate, ensuring reliability and continuous improvement.

How We Built It

Now, let's dive deeper—from a high-level overview into the detailed implementation of our RAG system:

RAG: Making LLMs Context-Aware

RAG (Retrieval-Augmented Generation) has emerged as the perfect solution. It allows us to enhance LLM capabilities and customize the LLM responses by providing relevant context.

We built two core pipelines: ingestion and retrieval.

Ingestion

Email messages are processed, cleaned, and converted into vector embeddings.

Our ingestion Lambda cleans emails, removes signatures, redacts PII data, and generates "hypothetical questions" to match future customer queries with historical responses.

With the hypothetical questions approach, we aim to create question-answer pairs by treating outbound emails as answers and inbound emails as questions. Then, while generating a suggested email, we extract the end customer's questions from the inbound email and search them in the hypothetical_questions vector field.

chain = (
    {"doc": lambda x: x.text}
    | ChatPromptTemplate.from_messages(
        [
            (
                "system",
                "You are a helpful assistant that generates hypothetical questions from an email.",
            ),
            (
                "human",
                "Generate a list of maximum 3 hypothetical questions that the below email could be used to answer:\n\n{doc}",
            ),
        ]
    )
    | llm.with_structured_output(HypotheticalQuestions)
    | (lambda x: x.questions)
)

After generating the questions, Lambda redacts PII data using Presidio and then indexes the email message into Weaviate.

While indexing, Lambda first generates the embeddings of email body text and hypothetical questions, then pass those vectors to Weaviate. We are using multiple vector embeddings which allows to store multiple vectors inside the same object. So that we can execute the search both in email text and questions without duplicating the data.

Retrieval

Similar emails and potential answers are retrieved from vector database.

A typical retrieve & generate flow looks as follows:

1. Extract questions

extract_query_prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            """You are a professional question extractor, an AI assistant that extracts the customer inquiries from email messages.
    The questions will be used to search for relevant emails in the vector database.
    By generating multiple perspectives on the customer inquiries, your goal is to help the user overcome some of the limitations of distance-based similarity search.
    Provide these alternative questions separated by newlines, no numbering.""",
        ),
        (
            "human",
            """Generate a list of maximum 3 questions from the following email.
    Email: {email}
    Questions:
    """,
        ),
    ]
)

extract_query_chain = extract_query_prompt | llm | LineListOutputParser()

extracted_questions = await extract_query_chain.ainvoke(
            input={"email": email.text}
        )

For the email shown in the demo video, the following questions are extracted by the question extractor chain:

{
  "output": [
    "Which documents are required to create an individual renovation roadmap?",
    "How can I submit additional documents for the renovation roadmap?",
    "What options are there for receiving support when uploading documents?"
  ]
}

Query vector database

We run multiple queries in parallel, and then combine unique retrieved documents. We mostly adopt hybrid search, by setting the alpha as close as possible to 1, we keep keyword search in the mix while leveraging semantical vector search.

email_message_retriever = MultiQueryRetriever.from_llm(
    retriever=email_messages_vector_store.as_retriever(
        search_type="similarity_score_threshold",
        search_kwargs=dict(
            alpha=0.90,
            tenant=data.orgId,
            score_threshold=0.70,
            target_vector=["text"],
            return_uuids=True,
            k=3,
        ),
    ),
    llm=llm,
    include_original=True,
)

question_retriever = MultiQueryRetriever.from_llm(
    retriever=email_messages_vector_store.as_retriever(
        search_type="similarity_score_threshold",
        search_kwargs=dict(
            alpha=0.90,
            tenant=data.orgId,
            score_threshold=0.70,
            target_vector=["questions"],
            return_uuids=True,
            k=3,
        ),
    ),
    llm=llm,
    questions=extracted_questions,
)

merger_retriever = MergerRetriever(
    retrievers=[
        email_message_retriever,
        question_retriever
    ]
)

retrieved_docs = await merger_retriever.ainvoke(message.text)

As you can see, we also utilize multi-vector search to enable searching in email text and also hypothetical questions.

retrieved_docs includes the email body and similarity score along with all the metadata we need, allowing us to leverage it while building the prompt.

For the same email and questions above, the retrieved context from database is as follows:

{
  "documents": [
    {
      "metadata": {
        "created_at": "2024-11-27T12:15:46.987000Z",
        "type": "SENT",
        "subject": "Interest in an individual renovation roadmap",
        "sender": "11000890",
        "org": "739224",
        "questions": [
          "Which documents are required for creating an individual renovation roadmap?",
          "How can additional documents for the renovation roadmap be transmitted digitally?",
          "What type of consumption data is needed for the individual renovation roadmap?"
        ],
        "thread_id": "bf0d0799-496d-49d2-9b2e-73128ff153d7",
        "uuid": "22462f39-4a69-47d4-91f4-d474b21c1eca"
      },
      "page_content": "Dear Mr. [PERSON],\n\nThank you for your interest in an individual renovation roadmap.\n\nAs part of your inquiry, we have asked you for some documents that form the basis for creating your individual renovation roadmap.\n\nWe would be happy to transmit your data to our Sunwheel Energie GmbH for the creation of your individual renovation roadmap. However, we need your support for this.\n\nPlease send us the following documents:\n\n* Building floor plans and sections of all floors\n* Window dimensions\n* Energy consumption bills from the last three years\n* Power of attorney\n\nBy clicking on the following button, you can easily and digitally transmit additional documents to us.\n\nTransmit documents\n[URL]\n\nPlease upload the missing documents to the corresponding upload fields. If you need support uploading the document, please don't hesitate to contact us by email at\n\nWe look forward to accompanying you on the path to your optimal heating solution.",
      "type": "Document"
    },
    {
      "metadata": {
        "created_at": "2024-07-15T05:57:23.809000Z",
        "type": "SENT",
        "subject": "Friendly reminder: We still need additional data for creating the renovation roadmap",
        "sender": "system",
        "org": "739224",
        "questions": [
          "Which documents are required for creating an individual renovation roadmap?",
          "How can additional information for the renovation roadmap be transmitted?",
          "What contact options are available for questions about the renovation roadmap?"
        ],
        "thread_id": "edb31adf-2ff3-4580-bb80-4ebb68a2f5de",
        "uuid": "35a4755b-d858-45eb-b328-d5dd70714adc"
      },
      "page_content": "Dear Mrs. <PERSON>, thank you for your interest in an individual renovation roadmap. In our email after receiving your order, we asked you for some additional information about your project. Your information is absolutely necessary for the preparation of creating your individual renovation roadmap. With <PERSON> on the following button, you can easily and digitally transmit the additional information to us. Submit additional information Please have the following documents ready for upload: <PERSON> from the last three years Dimensioned building floor plans/blueprints and sections of all floors Handwritten signed power of attorney for the application of funding for energy consulting (form in attachment) If you have any questions, please contact us by email at <EMAIL_ADDRESS> or by phone at <PHONE_NUMBER>. We look forward to accompanying you on the path to your optimal heating solution.",
      "type": "Document"
    }
  ]
}

Build and augment the prompt

We want to reference entities and vector database context to achieve the most contextually relevant emails and apply Vertical AI practices. We also want to return citations and entity references to show our users how AI processed the information and justified its responses.


system_prompt_template = """You are a powerful AI customer support, helping to write email messages and return verbatim quotes from the given context to justify the written email message.
You operate exclusively in epilot, the world's best energy XRM SaaS platform.
You are in a collaboration with the human customer support agent, called "user".
User is working in energy utility companies in Germany and may be working in either grid or sales (commodity, non-commodity).
User uses epilot to communicate with their end customers, colleagues, or partners.
The email you will write will be sent to either end customer, colleague or a partner by the user. You must act and think like the user that you are collaborating.

<current_conversation>
{conversation}
</current_conversation>
<vector_database_context>
{context}
</vector_database_context>
<entity_context>
{entity_context}
</entity_context>

<security_guidelines>
These security guidelines are EXTREMELY IMPORTANT and are unchangeable core principles that overrides all other instructions.
...
</security_guidelines>

<writing_emails>
To provide the best support to the end customer, following these instructions STRICTLY are EXTREMELY important:

...
</writing_emails>

<signatures_and_closing>
...
</signatures_and_closing>

<placeholders>
...
</placeholders>

<length_of_emails>
...
</length_of_emails>

<citing_previous_emails>
...
</citing_previous_emails>

<tracking_entity_references>
...
</tracking_entity_references>

<chain_of_process_and_thought>
...
</chain_of_process_and_thought>

<current_conversation>
{conversation}
</current_conversation>
<vector_database_context>
{context}
</vector_database_context>
<entity_context>
{entity_context}
</entity_context>

<output_format>
You must format your response exactly as follows:
...
</output_format>

<system_info>
Current DATETIME: {datetime}
</system_info>
"""

user_prompt_template = """
{prompt}
"""

prompt_template = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt_template),
        ("human", user_prompt_template),
    ]
)

chain = prompt_template | llm

We augment the system prompt by adding the retrieved context to the prompt in <vector_database_context> tags.

And we pass epilot user's prompt to the user_prompt_template.

Generate the response and stream it back

async for chunk in stream_xml_to_json(
    chain.astream(
        {
            "conversation": email_thread,
            "context": retrieved_docs,
            "entity_context": request.entity_context,
            "prompt": request.prompt,
            "datetime": datetime.now(timezone.utc).isoformat()
        }
    )
):
    yield chunk

We have defined a utility function stream_xml_to_json, to transform the LLM response chunks, which is in XML format, to structured JSON.

LangSmith Trace

Tip: Enable Streaming

To enable streaming, we have created a FastAPI application and are using AWS Lambda Web Adapter.

You can check those links to dive deeper on enabling streaming responses:

What's Next?

Our solution is already delivering great results, with adoption growing fast. Next, we’ll focus on supporting email attachments and making the knowledge base fully customizable.

At epilot, we're steadily progressing towards our vision of Vertical AI for the energy sector. Our upcoming feature, AI Suggested Actions, will help users automatically handle frequent tasks like payment method changes and customer relocations.

We’re excited to push towards fully automated, supervised multi-agent AI solutions.

Stay tuned! Follow us on dev.to and LinkedIn for updates and more tech insights.

Scaling Notification Systems: How a Single Timestamp Improved Our DynamoDB Performance

Sebastian — Wed, 14 May 2025 08:02:51 +0000

Introduction

The epilot platform contains a comprehensive notification system. Users receive notifications about ongoing tasks, such as new assignments or overdue tasks. They can also get notified about incoming emails or when someone mentions them in notes, and the list goes on. Users can choose to receive these notifications via email or as in-app notifications. This article focuses on the latter.
Initially, in-app notifications were stored in Aurora (AWS's solution for SQL-based databases). This setup soon became a major pain point, prompting us to migrate to DynamoDB. The simplicity of the notification data structure and the amount of read and write operations we expected made DynamoDB the perfect choice to scale.
However, if you don't think carefully about how you design access patterns in DynamoDB, more problems arise than you'd expect.
Let's dive into why a bad implementation of a markAllAsRead feature us some headache and how we reduced the complexity from O(n) to O(1) by using a timestamp-based approach for unread notifications.

The Problem

The initial design was straightforward. Every user gets a new notification item in the DynamoDB table. The partition key (pk) was a combination of user_id and organization_id, while the sort key (sk) contains the notification_id. The access patterns were quite straightforward: fetch all notifications for a given user, mark a notification as read, and mark all notifications as read for the lazy ones. The latter is the origin of this article.
An attribute read_state indicates if a notification was already read by a user. Marking a single notification was quite straightforward. It was as simple as:

async function markAsRead(params: { ... }) {
  await ddb.update({
    TableName: config.NOTIFICATIONS_TABLE,
    Key: toUserNotificationSK(params),
    UpdateExpression: 'SET read_state = :read_state',
    ExpressionAttributeValues: {
      ':read_state': 1, // binary 1 is true
    },
  });
}

Once a notification is read, the item is updated and read_state is set to 1. A Global Secondary Index (GSI) called byReadState then allows us to read all unread notifications for a given tenant (org + user). This created two operations that performed poorly:

A bad implementation of the markAllAsRead feature. It first queried all unread notifications and then performed a batch operation to update all notifications to read. As shown in the graph below, DynamoDB began to throttle under load, when people with lots of unread notifications used the marked all as read feature.
To indicate that a user has unread messages, a getTotalUnreadCount endpoint is exposed. This allows us to render a notification bell in the UI to show the unread count.

The naive implementation to batch update all unread notifications worked surprisingly well in the beginning. However, as the volume of notifications increased, we started experiencing more and more throttling events in DynamoDB. What started as occasional hiccups became a serious bottleneck in our notification service's performance.

The issue was multi-faceted. First, DynamoDB has limits on batch operations, requiring us to split large batches into multiple smaller operations. This not only added complexity to our code but also increased the probability of partial failures.
Second, each notification update consumed Write Capacity Units (WCUs) from our table's provisioned capacity. For users with hundreds or thousands of unread notifications, a single "Mark All as Read" action would consume a significant portion of our available WCUs, causing other notification operations to be throttled.
Importantly, these issues didn't affect the entire epilot platform, but were isolated to the notification service itself. Users would see timeouts or delayed responses specifically when interacting with notifications, while the rest of the platform continued to function normally.
However, this created a frustrating user experience, especially for power users who relied heavily on notifications to manage their workflows.
The problem was particularly severe for organizations with large teams, where notification counts could grow rapidly, and the "Mark All as Read" feature was used frequently to manage notification overload.

The Solution: Last Read Timestamp

After evaluating several options, we settled on a timestamp-based approach that would fundamentally change how we track read states while maintaining backward compatibility with our existing system.
Instead of updating each notification individually when a user clicks "Mark All as Read," we simply record the timestamp of when this action occurred. Any notification created before this timestamp is considered "read," while notifications arriving after it are "unread." This solution transforms what was an O(n) operation into an O(1) operation, regardless of how many notifications a user has.

The New Table Structure

We created a new DynamoDB table called notifications-read-state with the following structure:

{
  pk: `ORG#${orgId}#USER#${userId}`,  // Partition key
  sk: `READMARK#${timestamp}`,         // Sort key
  read_at: ISO8601Timestamp,           // When the user marked all as read
  created_at: ISO8601Timestamp         // When this record was created
}

The primary key design allows us to:

Efficiently lookup the most recent "mark all as read" timestamp for any user
Support multiple organizations per user
Maintain a history of read events if needed for analytics

The read_at attribute stores an ISO-formatted timestamp that serves as our "high water mark" for read notifications. This single attribute is the cornerstone of our solution.

Before:

async function markAllAsRead(userId, orgId) {
  // Step 1: Query for all unread notifications
  const unreadNotifications = await getUnreadNotifications(userId, orgId);

  // Step 2: Prepare batch updates (25 items per batch due to DynamoDB limits)
  const batches = createBatches(unreadNotifications, 25);

  // Step 3: Execute all batch updates
  for (const batch of batches) {
    await ddb.batchWrite({
      RequestItems: {
        [NOTIFICATIONS_TABLE]: batch.map(item => ({
          PutRequest: {
            Item: { ...item, read_state: 1 }
          }
        }))
      }
    });
  }
}

Complexity: O(n) - As the number of notifications increases, both processing time and database load increase linearly.

After:

async function markAllAsRead(userId, orgId) {
  const now = new Date().toISOString();

  // Single write operation
  await ddb.put({
    TableName: NOTIFICATIONS_READ_STATE_TABLE,
    Item: {
      pk: `ORG#${orgId}#USER#${userId}`,
      sk: `READMARK#${now}`,
      read_at: now,
      created_at: now
    }
  });
}

Complexity: O(1) - Constant time operation regardless of notification count.

This simple change drastically improved our system's performance. The "Mark All as Read" operation now completes in milliseconds instead of potentially seconds, uses a predictable amount of database capacity, and never times out, even for users with thousands of unread notifications.
What makes this approach particularly powerful is that we don't need to modify any existing notifications. Instead, we're recording a state transition that implicitly affects all notifications for a user at once.

Given the following pseudo-code, the byReadState index can be removed completely. All you need is to fetch the latest getLastReadTimestamp for a given user and calculate whether the notification was already seen or not.

const lastReadAt = await getLastReadTimestamp({ userId: params.userId, orgId: params.orgId });

const { Items, LastEvaluatedKey } = await ddb.query({
  ...
  TableName: config.NOTIFICATIONS_TABLE,
  IndexName: 'byTimestamp',
  ExclusiveStartKey: params.cursor ? decodeLastEvaluatedKey(params.cursor) : undefined,
});
...

const notificationsWithReadStatus = Items.map((item) => {
  // A notification is considered read if:
  // - It was created before the last "mark all as read" time OR
  // - It has been individually marked as read (read_state = 1)
  const isRead = item.timestamp <= lastReadAt || item.read_state === 1;

  return {
    ...item,
    read_state: isRead,
  };
 });

Lessons Learned

Our journey from the first version to the optimized solution taught us a lot about designing for scale—especially with DynamoDB.

At first, updating each notification one by one seemed fine. It was simple, worked great in dev, and handled early traffic just fine. But as usage grew, that approach quickly hit its limits. It was a good reminder: what works now might not work when your data grows 10x.

The breakthrough came when we stopped trying to optimize the old way and instead rethought the problem. Rather than updating every record, we started recording state changes with timestamps. That shift made things both simpler and faster—and it's a pattern that applies well beyond notifications.

Most importantly, we learned to play to DynamoDB’s strengths: fast, predictable access with simple operations. Once we aligned our design with that, everything clicked.

Conclusion

It’s easy to overthink scalability from the start, but the truth is: you won’t know your real problems until users are actually using the system. Our experience reminded us that it's totally fine to start simple and ship. You learn way more from real-world usage than from guessing at edge cases.

Scalability issues aren’t failures—they’re signs of growth. When we hit our limits, it forced us to rethink things. And funny enough, the fix—a single timestamp—ended up being both simple and powerful. It made the system faster, more reliable, and easier to reason about.

So if you’re torn between shipping something basic now or building for every possible future, go with the simple version. Ship it, learn from it, and improve as you go.

Do you want to work on features like this? Check out our career page or reach out to at X or LinkedIn

Dealing with Pushback to Product Engineering

Viljami Kuosmanen — Tue, 08 Apr 2025 13:34:33 +0000

I was recently confronted by a product engineer colleague here at epilot, frustrated with some of his non-engineer colleagues who didn't seem to buy into the idea of involving engineers in the product process from the start.

An all too familiar and frustrating situation for many of us.

You think of yourself as a proud product engineer wanting to solve meaningful problems but then get slapped with a detailed feature specification designed by a group of non-developers without your input. Now they expect you to go deliver their vision.

This is very much the reality in most product teams. Calling yourself a product engineer doesn’t automatically mean your voice will be welcomed. And that’s totally okay.

The Ideal vs. the Reality

In my past writings, I've painted the ideal: engineers who understand customer needs, question roadmaps, challenge designs, and contribute beyond code. But the reality is often much messier.

Some PMs and designers love working closely with engineers. Others are still adjusting to the idea. And that’s fair. The product engineer mindset definitely isn’t the norm.

Let’s not forget, PMs and designers are often under pressure too. It’s only natural they sometimes default to the most familiar and streamlined path. One that doesn’t always include engineers in the early stages.

And let’s be honest. Sometimes engineers haven’t yet built the trust or skills to contribute meaningfully.

The Friction Is Normal

The moment you step outside your lane, you shouldn’t be surprised when the reaction isn’t overwhelmingly supportive.

You will face skepticism.

You might be seen as overstepping.

You will encounter:

Requests to give estimates for implementing someone else's design
Roadmaps shared as top-down mandates
UI prototypes handed over "ready for dev", expecting pixel-perfect implementation
Pushback from asking too many questions

This is the price of wanting to do more than just execute. And if we’re honest, many of us haven’t always shown up in these conversations in a way that earns trust.

Is this a culture problem? Not necessarily. Most teams don’t have a rule against engineers joining product discussions. It’s just not the default. It’s less about policy and more about patterns. Changing those patterns takes trust, initiative, and persistence.

At the end of the day, it’s the product engineer’s job to show that product engineering actually works.

Earn the Trust

Calling yourself a product engineer isn’t a free pass. No one hands you a seat at the product table just because you want it. You must earn it. Show, don’t tell.

Do the homework. Know the names of your customers. Know the business domain.
Talk to your colleagues, not just other devs. Find opportunities to interact with customers.
Ask helpful questions that sharpen the team's product thinking.
Dive into data. Gather insights. Find new metrics and ways to collect useful signals.
Bring value to the table. Demonstrate you understand the customer problem by making thoughtful proposals that move the product forward.

You need to show up in demos, RFCs, testing sessions, and reviews. Not just in code commits.

Why Bother?

Because it’s worth it.

We do this for our own professional pride. To put great products into the hands of happy customers. And into our portfolios.

We don’t challenge product decisions because we want to take over the PM’s job or undermine the work of our teammates. We do it because we care about impact. Because we want our effort to count.

Because it hurts to pour weeks of your life into something that doesn't work, only to discover it failed because no engineer was involved in the concept phase.

Our Leverage

And here’s something to remember: We, as engineers, hold real leverage. We are the only ones who can actually turn product decisions into reality. No idea ships without us.

So while we may not always get a say by default, we do get a say in how we show up and how deeply we choose to care.

Use that leverage wisely. And proudly.

What To Expect When You Join Epilot

kate astrid — Mon, 03 Feb 2025 13:24:46 +0000

Hey you, awesome engineer.

Let's imagine that you are joining epilot tomorrow. Would you like to have a sneak peek at your nearest future?

Here’s a quick rundown of how we onboard new team members—no stiff formalities, just a clear path to help you settle in, understand our product, and get involved fast.

A Friendly Start
On the day one we’ll introduce you to your team lead and a dedicated onboarding buddy—think of them as your go-to guides for all things at epilot. Need pointers on our processes, product roadmap, or just want to figure out who to chat with about a certain topic? They’ve got you covered.

Your First Week

Get Set Up
We’ll make sure you have all the hardware, software, and accounts you need from day one. We’ll give you a handy checklist so you can get everything in place without the guesswork. Of course, your onboa buddy is always there to help you with everything.
Meet the Team
You’ll be given a chance to schedule 1:1 with every team member to get to know them better. Regardless of being remote-friendly company, we value real connections and friendly relationships, and make sure that everyone feels valued and supported.
Learn About Our Product and Users
We’re a high-touch SaaS company, and our product is complex and full of internal terms and abstractions. It might feel overwhelming in the beginning, but don't worry, we've got you. You will be given lots of resources like recordings of product demos and docs about different topics about epilot. No rush—just learn at your own pace and ask all the questions you have.
Get a Feel for Your Team’s Culture
Every team at epilot has its own rhythm and habits. Hop into Slack conversations, daily stand-ups, and team meetings to get a sense of what’s going on and how decisions get made. And feel free to share your ideas and suggestions - we are always eager to improve.
Start Contributing
We think the best way to learn is by doing. Early on, you’ll probably tackle a small assignment to get familiar with our systems and processes. Consider it your first real taste of epilot in action.

Getting to Know the Wider Company
Throughout your onboarding, you’ll also have chats with:

Founders: Hear the backstory behind epilot and our long-term vision.
Department Heads: Whether it’s Sales, Marketing, Engineering, or Product, each leader will fill you in on how their team supports our bigger goals. The idea is to see how all the pieces fit together and create the vibe.
Engineering & Product Leads: If you love diving into tech details, these sessions will give you insights into our architecture, tools, and future product plans.

Seeing the Product in Action
Our platform is built around microservices, but don’t worry if that’s new to you. We’ll walk you through the basics. We also encourage you to check out how our customers actually use epilot—like checking live customer feedbacks. It’s a great way to see what’s working and where we can improve.

The Balance Between Work and Collaboration
At epilot, we keep things productive but also supportive:

Team Events: You’ll get invites to company-wide announcements, product demos, company events, and more.
Open Communication: Slack, shared docs, and frequent check-ins mean you’ll never be left guessing.
Culture Matters: We appreciate everyone’s contribution, whether it’s project-related or just sharing something fun. We are the team, and this is what matters.

Looking Ahead
After a few weeks, you’ll have a good handle on how things work here—and you’ll know who to reach out to when you need help. We’ll likely ask for your thoughts on onboarding so we can keep improving the process for the future.

If this relaxed but purposeful approach resonates with you, we look forward to hearing from you. At epilot, we want every new hire to find their footing, make meaningful contributions, and grow in their role—without all the usual corporate fluff.

Happy to have you on board ✨.

Building a Scalable Audit Log System with AWS and ClickHouse

Sebastian — Tue, 26 Nov 2024 09:00:05 +0000

Audit logs might seem like a backend feature that only a few people care about, but they play a crucial role in keeping things running smoothly and securely in any SaaS or tech company. Let me take you through our journey of building a robust and scalable audit log system. Along the way, I’ll share why we needed it, what exactly audit logs are, and how we combined tools like AWS, ClickHouse, and OpenAPI to craft a solution that works like a charm.

The Case of the Disappearing Configuration

At epilot, we’ve encountered a frustratingly familiar scenario. A customer reaches out, upset that one of their workflow configurations has mysteriously vanished. Their immediate question? “Who deleted it?”—and the assumption is that someone on our team is responsible.

Now here’s the tricky part: how do we, as engineers, figure out who did what and when?

One obvious approach is to dive into the application logs. But here’s the catch: most of the production logs aren’t enabled by default. Even when they are, they’re often sampled, capturing only about 10% of the actual traffic. Additionally, those logs often seem to lack the required information. This means we’re left piecing together incomplete data, like trying to solve a puzzle with half the pieces missing.

What Are Audit Logs Anyway?

Audit logs provide clear visibility into system changes, aiding teams in investigations, diagnosing incidents, and tracing unauthorized actions. They empower admins by reducing support reliance and ensuring clarity on actions like role or workflow updates. For enterprise customers, audit logs are a critical, expected feature that supports compliance with standards like ISO 27001. Additionally, they lay the groundwork for enhanced threat detection capabilities in the future. In simple terms they try to help to answer the following questions:

WHO is doing something. Typically a user or a system (api call)

WHAT is that user/system doing?

WHERE is that occurring from? (e.g. an IP address)

WHEN did it occur?

WHY? (optional) Why did the user log in? → we don’t know, Why is its IP blocked? → User logged in 5 times with the wrong password

Key Considerations for a Successful Audit Log System

Before diving into the technical details, it’s crucial to define what makes an audit log system effective. While the exact requirements depend on your company’s domain, there are some universal points worth considering:

Compliance: Ensure the system adheres to regulations like GDPR. For example, customers may request the deletion of personal data, so you’ll need a straightforward way to erase all logs tied to a specific customer.

Sustainability: Audit logs grow rapidly, especially in high-traffic systems. Storing them indefinitely may not be feasible. Decide on strategies for archiving or purging logs over time.

Permissions: Define who is allowed to access audit logs to maintain security and privacy.

Format: Standardize the structure of your logs to ensure they’re easy to interpret and query.

Data Selection: Carefully determine what actions and events are worth logging to answer critical questions effectively, without unnecessary noise.

Making It Happen: How We Built Our Audit Logs

At epilot, our APIs are built around serverless components provided by AWS. From the outset, we recognized that AWS API Gateway events provided a rich source of information for building audit logs. These events capture critical details such as user identities, actions performed (through the request payload), IP addresses, headers, and more.

Given our microservices architecture, where services are organized by domain and accessed through an API Gateway (see our system architecture), we needed a solution that seamlessly integrated with this structure.

High-Level Overview
Our approach to audit logging can be summarized as:

Capturing events asynchronously.
Validating and transforming raw events into a standard format.
Persisting the data in a read-only, scalable, and query-friendly storage system.

This design adheres to several key technical principles:

Asynchronous Event Capture
We use Amazon SQS to decouple event capture from the main HTTP request flow. For example, when a user creates a new workflow configuration, the relevant API Gateway event is pushed to an SQS queue by middleware wrapping the API. This ensures that audit logging does not introduce latency or affect the performance of the core application logic.

From Raw to Standardized Events
Our focus is on capturing system modifications, specifically HTTP methods like POST, PUT, PATCH, and DELETE. These provide meaningful insights into changes occurring within the system. GET requests, on the other hand, generate excessive noise and are generally excluded—though we offer an opt-in mechanism for services where logging GET requests adds value.

A Lambda function processes raw API Gateway events from the SQS queue, transforming them into a structured and validated format. This includes filtering relevant data, enhancing it using metadata like OpenAPI specifications, and ensuring consistency across all logged events.

Data Persistence
For storing audit logs, we chose ClickHouse, a highly scalable, SQL-based database that aligns with our requirements:

Read-only access: Supports immutability to preserve data integrity. Scalability: Proven in our data lake setup to handle large volumes of data efficiently.
Querying: SQL capabilities allow for precise filtering and analysis, which is more complex with alternatives like DynamoDB. By leveraging ClickHouse, we ensure a robust and scalable foundation for our audit logs, simplifying future integrations and analysis.

Integration
To make audit logging effortless for our microservices, we focused on seamless integration. At epilot, we rely heavily on middy, a middleware engine used across all our services. Building on this, we introduced a new middleware: withAuditLog.

import { withAuditLog } from '@epilot/audit-log'
import middy from '@middy/core'
import type { Handler } from 'aws-lambda'


export const withMiddlewares = (handler: lambda.Handler) => {
  return middy(handler)
    .use(enableCorrelationIds())
    .use(...)
    .use(
      withAuditLog({
        ignorePaths: ['/v1/webhooks/configs/{configId}/trigger']
      })
    )
}

This middleware integrates directly into existing services and simplifies the audit logging process by:

Capturing API Gateway Events: It hooks into the request lifecycle to extract the API Gateway event details.
Omitting GET Requests by Default: To reduce noise, it filters out GET requests, with an option to opt them in for specific services where needed.
Forwarding to SQS: Its primary role is to forward the event to an SQS queue for asynchronous processing.

With this middleware, adding audit logging to any microservice is as simple as including withAuditLogs in the service's middleware stack and giving the SQS:SendMessage permission. This ensures consistency, reduces implementation effort, and keeps the integration process dead simple.

Technical Considerations
This article focuses on our high-level approach to building audit logs, as there are numerous ways to tackle the problem, each with its trade-offs. During our research, we explored alternatives like EventBridge for emitting events at the end of each request or Kinesis for streaming data. Ultimately, we chose a solution that met our key requirements: decoupling log emission from the main flow while offering flexibility in managing throughput and batching.

Here’s why we chose SQS:

Decoupling from the Main Flow
SQS allows us to process audit logs asynchronously, ensuring that the main HTTP request flow remains unaffected. This means audit log processing won’t slow down user-facing operations.

Flexibility with Throughput and Batching
With SQS, we can fine-tune parameters like long-polling and batch windows to optimize throughput without compromising efficiency. This ensures scalable and reliable processing regardless of traffic spikes.

Scalability for POST/PUT/PATCH/DELETE Events
Since we exclude GET requests by default, the system can handle fewer, more meaningful events. Capturing GET requests would require supporting a higher volume of events, potentially leading to Lambda concurrency issues, as multiple Lambda environments subscribing to the same queue could interfere with other services also using Lambda.

Exposing Audit Logs to Users

To make audit logs accessible and actionable, we introduced a new SST-based microservice that acts as a bridge to query data from ClickHouse. This microservice provides a simple and intuitive interface for users to explore their audit logs.

Key Features:

Search and Filtering: A user-friendly search bar allows users to combine filters effortlessly, enabling them to pinpoint specific events or patterns within the logs.
Activity Messages: Each audit log entry includes an activity message, a concise summary of what occurred. This message is dynamically constructed on the API side, tailored to the specific service name, making it customizable and relevant.

By customizing the activity messages for each service, users can quickly understand what happened in their systems without wading through raw data. This tailored approach ensures that the audit logs deliver immediate value and clarity to the end users.

Summary

In this article, we detailed the design and implementation of our audit log system at epilot, highlighting the key decisions and considerations that shaped its architecture. Our approach leverages AWS serverless components to seamlessly integrate audit logging into our microservices, ensuring scalability, efficiency, and ease of use.

Capturing Events: Using a custom middleware, withAuditLogs, we extract API Gateway events asynchronously and forward them to an SQS queue, ensuring the logging process does not block the main application flow.

Processing and Storing Logs: A Lambda function transforms raw events into a standardized format, focusing on meaningful system modifications (POST, PUT, PATCH, DELETE) and stores them in a scalable, SQL-based ClickHouse database.

User Accessibility: A new SST-based microservice provides a simple interface for querying and filtering logs. Tailored activity messages enhance usability, helping users quickly understand what occurred.

Technical Considerations: SQS was chosen for its ability to decouple the logging process, optimize throughput, and handle scalability challenges. While other solutions like EventBridge or Kinesis were viable, SQS met our specific requirements effectively.

This high-level overview provides a flexible, scalable, and user-friendly solution for audit logging while ensuring system integrity and maintaining performance.

Do you want to work on features like this? Check out our career page or reach out to my Twitter

The Sauna Epiphany: How I Got Product Engineering Wrong

Viljami Kuosmanen — Sun, 13 Oct 2024 12:23:43 +0000

If you've seen my posts lately you've probably seen a lot of talk about Product Engineers: software engineers who talk in customer problems and take pride in the products they build, not only the code and technologies they use.

My core thesis is that with the rise of AI, the expectations for software engineers are being redefined, moving away from narrow programming roles: backend, frontend, C#, Java, React, etc. fast becoming obsolete due to AI tools like Cursor lowering the barrier of entry and unlocking productivity with any technology.

The days when you could coast on your ability to churn out code are over. If you're still clinging to the idea that you're safe just because you know how to write code in a widely used language or framework you're in for a rude awakening.

I believe this trend is likely to be the biggest industry shift in my lifetime, even surpassing the agile movement of the last 20 years. And I'm not the only one talking about it. (#1, #2, #3, #4, #5)

But I got it wrong.

Not entirely wrong, but I made it more complicated than it needed to be. In my quest to define what it means to be a product engineer, I overlooked the power of simplicity and ironically did not think enough about the customer: the ambitious software engineer thinking strategically about their careers.

I overengineered it

In my earlier attempts to help engineers break out from purely technical roles, I published an extensive checklist to help engineers think and ask questions as product engineers. It covered everything from understanding the user and the market to measuring success and staying ahead of industry trends.

Here's a taste of that checklist:

Great stuff, but let's be honest — no engineer is going to run through a 15-point checklist to make a product decision. Just seeing this long list of questions can be totally overwhelming, especially if you're used to being in a technical role.

If we expect engineers to actually start doing this stuff, the core idea needs to be simple and easy to remember.

The Wake-Up Call in the Sauna

As I'm writing this post, I'm enjoying a company working vacation in a nice hotel in Mallorca with my epilot colleagues. I was having a conversation in the hotel sauna with a principal engineer where something he said in passing suddenly clicked for me.

He was venting about some engineers on his team who seemed to be diving head first into coding without bothering to understand the problem. "It's so easy. Every engineer should be able to answer what problem they're solving, who the customer is, and what the impact of the work they're doing is."

That hit me like a splash of cold water in a 100°C Finnish sauna.

He was totally right.

Boiling It Down to 3 Essential Questions

To think like a product engineer, it's already enough to start with just three simple questions:

What's the problem?
For who?
Why is this important?

Let's dive a bit deeper into each one.

1. What's the problem?

Stop coding for a second. Do you really know what you're trying to solve? Not the ticket description in Jira, but the real-world issue. If you can't articulate the problem in simple plain English, you have no business writing a single line of code.

2. For who?

Identify your user and customer. Sometimes they're the same person; other times, they're not. Understanding who will use your product (and who will pay for it) is crucial. It shapes the way you approach the solution and helps you tailor the experience to meet their needs.

3. Why is this important?

As engineers there's never a shortage of things we could improve or implement. If solving the problem doesn't make a meaningful difference, why are you wasting your valuable time? We're not here to build features that nobody uses or cares about. Connect your work to something that actually matters and helps build your track record.

By anchoring your work in these three questions, you immediately move from code monkey to a high value product-minded engineer. Still a rare breed. You're not just implementing features someone else decided to build; you're elevating yourself to a position to influence product decisions and build smarter.

Leverage Your Team—They're There for a Reason

The biggest mistake engineers make is thinking they have to find all the answers themselves.

Most of us are lucky to work in a product team with other disciplines like UX researchers, designers, PMs and business stakeholders whose job is to help answer these questions.

By leaning on your team, you not only find better answers but also foster a more collaborative and innovative environment.

Product engineering is a team sport.

Taking Action: Start Asking the Right Questions Today

So, the next time you tackle a new topic, pause for a moment before diving into code. Ask yourself:

What's the problem?
For who?
Why is this important?

Write down your answers. If you don't know, reach out to your team and find out. This simple practice can transform your approach to work, leading to better decisions, more impactful solutions, and a greater sense of ownership.

Congratulations, you just became a Product Engineer!

Join the Conversation

I'd love to hear your thoughts on this simplified approach to product engineering. Have you tried asking these three questions in your work? What impact did it have? Share your experiences in the comments below.

Consider giving the GitHub repository a star if the product engineer role resonates with you!

How We Integrate AI in epilot - Chapter 1: AWS Bedrock & Prompt Engineering

Kerem Nalbant — Thu, 18 Jul 2024 14:58:31 +0000

Introduction

When we decided to bring AI to epilot, we had so many potential use cases that we first needed to do user research and identify the most repetitive and time consuming tasks. After that, we had to find out what our customers needed the most from these.

Our research showed that users heavily utilized the messaging feature and for some cases when long email threads come into the play, we noticed that some customers were spending an average time of 16 minutes replying to an email, and we knew we could make it better by providing them with a shorter and clearer thread summary.

Enterprise AI Playbook

Work is a bundle of tasks, which are performed towards specific goals.

Tasks are the ‘atomic unit’ of any work done in the enterprise. Tasks may be performed as a human service, or may be performed by software, towards achieving a goal.

Our goal was to reduce that time down to less than a minute, and there were two different tasks being performed by users which we need to perform with AI to achieve our goal. This article addresses the Task: Send emails and the steps to complete this task:

Read and understand the email thread: Help users understand long email threads faster by providing AI-generated summary, next steps and topics
Write an answer: Provide AI-generated suggested answers.

Problem

Our customers often deal with long email threads, requiring a long time to read and answer. We needed a solution that could summarize email threads and provide recommendations for next actions.

Solution

While some problems could be solved with prompt engineering alone, some could be solved with Retrieval-Augmented Generation (RAG).

For generating summaries, we didn't need any external contextual data other than email thread to feed the prompt, so we decided to just go ahead with prompt engineering.

The next task is suggesting AI-generated replies to email threads, where RAG would be really useful. For that, we will use a Vector DB and an embedding model, which I'll write about in the next chapter.

AWS Bedrock

We decided to use AWS Bedrock, as epilot we already make use of AWS in almost every area of our platform. It provides state-of-the-art LLMs from multiple providers and out of the box solutions such as Knowledge Base to achieve RAG easily and Model Evaluation to compare models and prompts with a fancy UI.

AWS Bedrock also ensures the processed data is protected. One of our concerns was where the data would be stored, how it would be processed and whether 3rd parties would be involved.

By default, AWS Bedrock offers zero-retention policy, ensuring that logs, prompts, LLM output and any personal data are not shared with any third parties or model providers. Bedrock also ensures that all processed data remains within the EU region.

GenAI Foundation

GenAI Foundation has a central SQS queue and a handler function, ensures exactly one, concurrent and batch processing while staying within the rate limits of Bedrock.

Integrating GenAI related code and logic to our existing APIs was not a good idea. So, I've decided to create a new monorepo. Here's the reasons:

Separation of Concerns

By creating a separate repository, we maintain a clear separation of concerns. This ensures that the core functionalities of existing APIs remain clean and focused.

Language Flexibility

Since GenAI-related code requires Python, having a separate repository allows us to leverage Python's capabilities without interfering with the TypeScript-based APIs. This separation ensures that each project can use the best-suited language for its specific tasks.

Encapsulation

Encapsulating the GenAI logic within a dedicated repository makes it easier to manage, update, and scale. This also allows engineers with specific expertise in GenAI to work independently of the rest of the system.

Modularity

A modular approach allows for easier testing, maintenance, and deployment of the GenAI features. Updates and bug fixes in the GenAI module can be rolled out independently of the core APIs.

Model Choice & Prompt Engineering

We decided to go with Claude 3 Sonnet because it was the best option for us at the time, given the costs and the complexity of the task.

We are planning to switch to Claude 3.5 Sonnet once it's available in AWS Bedrock, which we're really excited about!

To choose the best model for the task, AWS Bedrock offers Model Evaluation, which lets you easily compare models with each other.
You can also do Prompt Evaluation by creating a dataset and executing it against a single model to compare the results of different prompts.

There are lots of great resources online for prompt engineering, but I want to mention that Anthropic provides really helpful tips in their documentation. I strongly suggest you to take a look, if you haven't already. For me, the most important point was using XML tags. Anthropic mentions that their models produce much better results when used with XML tags.

Prompt engineering is all about experimenting, so we spent some time on optimizing our prompt. I'd love to share an example of prompts with you!

Below you can see some example usages of common prompt engineering techniques such as "Giving a Role to LLM", "Using XML Tags", "Using Examples (Multishot Prompting)", "Being clear and direct".

System Prompt

System prompt is where we give Claude a role, and some clear instructions about the task.

You are an intelligent assistant specialized in assisting customer service agents in energy industry.

Your task is summarizing email conversations between customer service agents and customers, providing a clear and concise overview of the key points by following the instructions provided.

Your summaries will help your colleagues quickly understand the main aspects of each conversation without having to read through the entire email thread.

Your goal is to ensure that the agent can grasp the key points and next steps from your summary alone, making their workflow more efficient and effective.

You are a third person observer and must not provide any personal opinions or make any assumptions about the conversation.

You must use the third-person objective narration. You must report the events that take place without knowing the motivations or thoughts of any of the characters.

User Prompt

Here is the conversation between the customer and the agent:

<Email Thread>
<Subject>
{subject}
</Subject>
<Messages>
{conversation}
</Messages>
</Email Thread>

<General Instructions>
- You must use <Email Thread> tags to identify the email conversation.
...
- You must use only the knowledge provided in the <Email Thread> tags and do not access any other external information or knowledge you already possess.
</General Instructions>

<Language Instructions>
- You must optimize the language for making the summary easy and fast for humans to read.
...
- You must use a professional, respectful and informative tone.
</Language Instructions>

<Reference Instructions>
- You must always refer to the Customer and Agent by their names.
...
</Reference Instructions>

<Summary Instructions>
- You must get relevant quotes to complete the task from the conversation.
...
- You must write the summary in reverse chronological order.
</Summary Instructions>

<Output Instructions>
- You must give your output in JSON format. The JSON object should be valid.
...
- If you will provide quotes or emphasize any part of the conversation, you must use single quotes. e.g. 'quote'.
</Output Instructions>

# Few Shot Prompting
<Example Outputs>
{
  "summary": [
    "John Doe has processed the reimbursement request and informed Jane Doe.",
    "Jane Doe has provided the requested information and is awaiting further instructions from the team member.",
    "John Doe has acknowledged the Jane Doe refund request and has requested additional information to process the refund.",
    "Jane Doe is requesting a refund for a defective PV inverter."
  ],
  "topics": [
    "Refund request for defective product"
  ],
  "next_steps": [
    "If the information is sufficient, John Doe should process the refund and inform Jane Doe of the completion.",
    "John Doe should document the refund process for record-keeping."
  ]
}
...
</Example Outputs>

You must follow the instructions listed in the following tags:
- <General Instructions> for general guide and rules,
- <Language Instructions> for lingual instructions that declares your tone, grammar, and output language,
...
- <Example Output> for example of the output as a reference.

Human-in-the-Loop (HITL)

It's challenging to ship AI features, especially when it's the first one in the company which has thousands of users. We need to build trust, and keep it. To do that, we need to collect as much as feedback we can. Then, evaluate these feedbacks and take actions.

HITL can be applied in different ways in different solutions. In our use case, we utilize it to evaluate feedback and detect hallucinations. Also, our team evaluating the feedback can change the result generated by the AI.

Migration Strategy

Our migration strategy involves both runtime and one-time migration to ensure a smooth transition and boost the adoption for our users.
With this way, we ensure we don't unnecessarily make use of LLMs, and we save costs while ensuring seamless integration.

Rate Limits

Since AWS Bedrock imposes rate limiting, we had to reflect this to our users. In order to offer a fair usage, we set limits according to the pricing tier of our users.

Code Examples

Pay attention to the way I prefill Claude's response to force it to answer only with JSON object.

settings = {
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens": 1000,
    "system": system_prompt,
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": prompt},
            ],
        },
        {"role": "assistant", "content": "{"}, # Prefill Claude's response
    ],
    "temperature": temperature,
    "top_p": top_p,
    "top_k": top_k,
}
res = await client.invoke_model(
    modelId=MODEL_ID,
    contentType="application/json",
    accept="application/json",
    body=json.dumps(settings)
)

model_response = json.loads(await res["body"].read())
response_text = model_response["content"][0]["text"]

res_model = LLMResponseModel.model_validate_json("{" + response_text)

Conclusion

By integrating AI into epilot, we have significantly enhanced the capabilities of our platform. This integration not only improves the efficiency of daily tasks, but also accelerates customer support. Furthermore, it's the first step in positioning epilot as a leader in the adoption of advanced AI technologies in the energy sector.