DEV Community: Viljami Kuosmanen

Our Entire Company Ships Code Now. 40 PRs from Non-Engineers in 60 Days.

Viljami Kuosmanen — Mon, 13 Apr 2026 13:06:30 +0000

In February, I opened our entire codebase to the company. PMs, designers, projects, customer success, and support all got access to 219 repositories and over a million lines of production code. I handed them powerful coding agents and told them to start contributing.

I've been writing about product engineering for years. The core idea: engineers should understand the whole product, the customer, the business problem. I wanted to test the reverse. Can non-engineers contribute to the codebase if you give them the right tools?

Two months of real data delivered a clear answer: yes. And it may be the most powerful shortcut I've found to building capable, business-aware product teams who truly care about what they ship.

Here is what that looked like at epilot: 629 merged PRs in 60 days. 40 PRs were from 11 non-engineers. All teams have at least one person who contributed to our codebase with Claude Code.

We're not alone in seeing this shift. Stripe ships over a thousand agent-produced PRs per week. Superhuman uses Codex to let product managers contribute lightweight code changes without pulling in an engineer (except for code review). Intercom gave Claude Code to everyone in the company, over 1,000 non-engineers (PMs, designers, support, marketers, and more), and more than 300 became active weekly users.

This post is a detailed, honest look at how we set it up, what actually worked, the mistakes we made, and the deeper lessons that emerged.

The starting point

epilot is a B2B SaaS platform for energy companies. We run a fully remote team of 40+ product engineers with no engineering managers and no dedicated platform team. Everyone owns end-to-end ownership. We already ship to production 150+ times per week on AWS serverless.

Our entire engineering team was already using AI coding agents daily. Claude Code, Codex, Cursor. Engineers had adopted them fast and the productivity gains were obvious. The question I wanted to answer next: what happens when you expand codebase access via coding agents to the rest of the company? PMs, designers, customer success, support. Not as a one-off hack day, but as a fully supported, production-grade workflow.

The infrastructure that turned agents into teammates

Claude Code and Codex still only ship connectors to GitHub, and our entire codebase lives in GitLab. So we created a single GitHub repo that bootstraps a developer sandbox with a shallow copy of all 219 repositories. This allows both AI agents or engineers to spin up a fully functioning developer environment with everything cloned & tools set up within minutes.

1. Agents live in Slack

We created #dev-squad-agents alongside our regular squad channels. Anyone could join via Claude Code, Codex, or GitHub Codespaces. I ran three one-hour onboarding sessions to showcase the tooling and make sure everyone had the required access: one for design, one for support, one for go-to-market. The PMs didn't need their own session. Their engineering teams had already invited them and they were eager to start.

This single decision drove adoption more than anything else.

2. One-click preview environments

Every PR automatically generates a live preview link in CI. Click it, see the change, test it immediately. No local setup required.

For engineers this is convenient. For non-engineers it is essential. If they cannot see and interact with their change, they can't meaningfully contribute.

3. Deep institutional context

We built an indexed knowledge base called how-to-everything. It contains our API patterns, backend conventions, code style, CI/CD and testing strategy, full business context, domain terminology, and our entire RFC archive.

Stripe solved the same problem by connecting agents to 400+ internal tools. We chose comprehensive indexing. The insight is identical: agents without your company’s context produce garbage. Agents with it produce work that feels native.

4. Design system in context

Our Volt UI component library is baked into the agent prompt. Claude now generates frontend code that respects the correct components, spacing, colours, and patterns from the first try. Consistent UI without forcing a designer to review every small change.

Claude as a coworker: the Slack pattern

Claude quickly became the most active “team member.” People tag @claude the same way they tag colleagues, and they expect results just as fast.

How engineers use it

Simple tasks look like: “@claude fix EPI-5660.” A PR appears.

More powerful uses scale across the organization:

“@claude which microfrontends still use the legacy editor?” → instant audit of all 219 repos.
“@claude migrate 13 backend repos from file-client to lambda-adapter.” → completed in one session.
Mass upgrades across 20+ repositories: one message triggers 6 parallel sub-agents that handle changes, tests, and PRs.

What once took a full day of tedious cross-repo work now happens in hours. Agents worked weekends. Engineers didn't have to.

How non-engineers use it

They speak in plain product language:
- “@claude disable drag and drop of entity table columns for embedded tables.”

“@claude when creating a new family in the Label Builder, the drawer should close automatically.”

Claude figures out the right repositories and implementation details. Non-engineers don't need to know our tech stack or repo structure. They describe the desired outcome.

Production impact

We added Claude to #alerts-production on day two. It investigates incidents, reads logs, traces errors, and proposes fixes before most engineers open the dashboard. When a bulk action hammered Elasticsearch with 100 parallel requests, Claude designed an org-level job queue with concurrency limits that cut peak load by 90%. Engineers started replying “another one bites the dust” to resolved alerts.

Within a week, Claude felt like a trusted junior-to-senior coworker: fast executor, deep historian, and reliable pair programmer. Usage spread organically to every squad.

What people actually built

Non-engineer contributions (40 merged PRs from 11 people)

Feature work: One PM built auto-tagging for file uploads (a feature he had spec’d three months earlier). Another shipped 13 PRs on messaging (bulk email templates, sidebar toggles, drawer behaviour, i18n). A go-to-market colleague delivered entire features, like pivot tables for displaying time-series data in epilot tables, that are now live for customers.
Daily improvements: Customer success shipped portal config changes they handle every day. Designers and support added FullStory tracking and started prototyping new UI flows.
Translations: Native speakers cleared 116 German translation improvements with almost no engineering involvement.
UX via vibecoding: A designer shipped a UX improvement to our Flow Hub before heading into the weekend: when creating a new Flow, users now get prompted to enter a name first. She described it as "vibe coding" and the change reduced friction in a way she'd been wanting to fix for a while.

These were not toy changes. Several were meaningful, customer-facing updates. One PM even asked Claude to create a demo video of the feature it just built, so he could share it with the team. The full cycle from spec to demo, handled through conversation with an agent.

Engineer contributions (589 merged PRs)

Engineers use agents across the full lifecycle. Writing RFCs and technical designs. Implementing multi-service features. Running tests. Shipping to production. Auditing feature flag usage across 15 repos and cleaning up stale flags. Investigating production incidents from alert channels before anyone opens a dashboard.

Some of the most valuable uses are research and diagnostics. "Which services listen to the file upload event?" "How many microfrontends still use the legacy editor?" "Audit all listSchemas calls and upgrade to the new summary API where safe." These are questions that used to take half a day of grepping across 219 repos. Claude answers them in minutes.

GitHub Codespaces turned out to be a key piece for engineers too. Running Claude Code in auto-mode in a cloud sandbox means you can kick off a multi-repo migration and let it run without risking your work laptop. No local state to corrupt, no accidental changes to your working branch. The sandbox spins up with the full codebase mirrored and Claude works through the task autonomously. Engineers now run 6 parallel sub-agents in Codespaces for bulk upgrades across 20+ repos: package bumps, migration scripts, test updates, PRs opened. What used to be a full day of copy-paste across repos is done in one session.

The new collaboration model

Non-engineer describes the problem in business terms → Claude implements → engineer reviews for architecture and quality. The person closest to the customer now helps ship the solution. Feedback loops tightened dramatically.

A designer and engineer even deployed Agentation (a prompt-structuring tool) in 2.5 hours. The designer now uses it daily and calls it a “dream collaboration.”

Guardrails: the hard lesson

A few weeks in, one non-engineer with full GitLab push access bypassed our CI pipeline and deployed code to production without review. No damage, but it was exactly the wake-up call we needed.

We immediately tightened permissions: non-engineers and agents create PRs; engineers review and merge. No direct push to protected branches.

But that incident was just one piece. When you give AI agents to 50+ non-engineers, you need a proper governance framework. We built one.

What we had from day one

Sandboxed execution only. Agents run in Claude Code Web, Codex sandbox, or GitHub Codespaces. No production infrastructure access. No deployment outside CI.
Dedicated bot users. Limited API scopes. Code-level access only. Zero access to customer data, session tokens, or production databases.
Public channels only. All agent interactions happen in shared Slack channels. Engineers see everything and can intervene. No private DMs with agents.
Human responsibility. You trigger the agent, you own the output. No exceptions.

What we added after

We published a formal Agentic AI Tool Usage Guideline. The core principle: give the agent the minimum access it needs for the task at hand, and nothing more.

The guideline includes:

A three-tier data classification. Red (never share with AI): customer PII, session tokens, credentials, security documents. Yellow (anonymise first): user research transcripts, survey responses, usage metrics with identifiers. Green (safe): public docs, your own drafts, anonymised research, process docs. Hard line on red: no exceptions, no "just this once."
An approved connector matrix. Every tool combination (Claude + Figma, Claude + Atlassian, Claude + Miro, etc.) classified by risk level with specific conditions and recommended settings.
Prompt injection awareness. Use a separate browser profile for Claude browser extension. Don't paste content from untrusted sources into agent prompts. Disconnect MCP connectors when not in use.
Explicit dealbreakers. Leaking customer personal data, unauthorized code deployment, and feeding company data into model training are not allowed regardless of how useful the outcome might be.
GDPR and EU AI Act compliance. The guideline maps our practical rules to specific legal obligations: data minimisation (Art. 5), DPA requirements (Art. 28), AI literacy obligations (Art. 4), transparency requirements (Art. 50). We're a European company serving energy utilities. Regulatory compliance is table stakes.

Lesson: Lock down merge and push permissions before opening access. We got lucky. And build the governance framework early. Guardrails didn't slow adoption; they accelerated trust and usage.

What surprised us most

Adoption speed: Three one-hour onboarding sessions and people were off. Every team across the company had active contributors within three weeks.
Non-engineers shipping real features: I expected small fixes and translations. Instead, people from across the company delivered complete, production features.
Claude as institutional memory: Support now asks Claude product questions instead of hunting down engineers. It has become the fastest way to understand how anything works across five years of code history.

Beyond code

This is just one layer. My colleague Suresh built Vigilos, an internal BI agent that lets anyone query our data warehouse in natural language. Designers check average journey name lengths for UX decisions. Customer success identifies accounts with AI features disabled. Product managers run customer segment analysis in minutes.

We'll share that story soon.

Where this goes

Two years ago I published the Product Engineer Manifesto: engineers should care about the customer, the business outcome, and the whole product.

What these two months taught me is that the same mindset now applies in reverse. Give non-engineers the right tools, deep context, and sensible guardrails, and they will deliver.

We're still hiring product-minded engineers. That hasn't changed. Engineers at epilot own the full product lifecycle: customer understanding, architecture, quality, reliability, and shipping. Non-engineer contributions don't reduce what we expect from engineers. They amplify it. The entire organization now ships alongside them.

629 merged PRs in 60 days. 40 from non-engineers.

We're hiring all product roles who thrive on full ownership in an AI-native environment. Remote-first. epilot.cloud/careers

We Made Coding Agents Actually Reliable By Fixing One Thing

Viljami Kuosmanen — Wed, 04 Feb 2026 15:55:45 +0000

Last week, Vercel published research showing that giving coding agents a compact index of your documentation dramatically outperforms letting them search for answers on demand. Their eval results: 100% task success rate with the map approach, versus only 79% when agents had to actively look things up.

Same agent, same tasks, different approach to context. The difference between working and not working.

The insight clicked immediately. If we could give Claude Code reliable access to this institutional knowledge without forcing it to decide when to look things up, it would fundamentally change how people work in our codebase.

So we built it.

The Context Problem

Coding agents have a token limit. A ceiling on how much information they can process at once. Think of it like working memory. You can't hand Claude Code your entire codebase and documentation library upfront. It's too much.

The traditional solution is skills: the coding agent decides when it needs information and actively looks it up. "I need to know about authentication, let me search for that." Sounds reasonable. In practice it creates three problems:

Decision paralysis - the agent has to decide when to look up docs, and it often guesses wrong
Async delay - every lookup is a round-trip, breaks flow
Sequencing conflicts - exploring code vs. consulting docs creates timing issues

Vercel's approach flips this: give Claude Code a compressed index of what documentation exists and where to find it, before it starts work. The index is small enough to fit in context every turn, so it always knows what's available. When it needs details, it reads the specific file directly. No decisions, no lookups, no conflicts.

Why Compression Matters

The key innovation is compression. A full documentation tree - all the folder structures, file names, categories - takes significant space. Too much to include in every conversation turn.

The compressed index uses a simple pipe-delimited format that shrinks this by ~80%:

[epilot Docs Index]|root: ./.epilot-docs|00-general:{tech-stack.md,business-context.md,ci-cd.md}|01-apis:{api-design.md,calling-apis.md}|02-epilot360-microfrontends:{env-vars.md,local-dev.md}|...

That single line tells the agent:

Where the docs live (.epilot-docs/)
What categories exist (00-general, 01-apis, etc.)
What files are in each category

It's a table of contents, not the full book. But it's enough. The agent sees the map, understands what's available, and pulls specific files when needed. The decision-making load disappears.

Why This Matters

The accuracy improvement is significant (100% vs 79% task success), but it's not the real story.

The real story is what happens when you lower the barrier to contribution. Proper context enables the person closest to the problem to fix it, regardless of job title. Your PM can fix bugs. Designers can adjust component behavior. Support engineers can patch data issues. You remove bottlenecks. Context and authority live in the same person.

Coding agents are the equalizer. But they're only as good as the context you give them.

Most companies will throw Claude Code at their codebase and wonder why results are inconsistent. The agent hallucinates patterns. Makes incorrect assumptions. Writes code that doesn't match conventions.

The difference is context. Structured, compressed, always-available context about how your codebase works.

How to Build Your Own

The pattern is straightforward. Here's what we did at epilot:

Curate agent-friendly documentation - organize your internal knowledge: conventions, APIs, architectural patterns, framework usage, code style
Structure by domain - group related docs (general, backend, frontend, infrastructure, etc.)
Use descriptive filenames - the agent sees filenames in the compressed index before opening files. api-design.md is better than guidelines.md. error-handling.md is better than errors.md. Make filenames searchable and specific.
Automate updates - pull live data where possible (OpenAPI specs, schema definitions, framework docs)
Generate compressed index - use a simple format (pipe-delimited works well) that reduces the doc tree by ~80%
Embed in agent context - add the index to your CLAUDE.md or AGENTS.md file (the context files Claude Code reads)

One thing worth highlighting: we don't just include our internal documentation. We also package docs for the frameworks and libraries we heavily use - single-spa, openapi-backend, openapi-client-axios, i18next, and Volt UI (our custom design system). When Claude Code needs to know how i18next pluralization works or how to register a single-spa parcel, it already has the answer. No hallucination, no outdated Stack Overflow posts, just accurate framework documentation.

What This Enables

We're already seeing daily usage across the team. Developers context-switch between services faster. Non-engineers contribute directly instead of filing tickets.

But the real potential is broader: if compressed context improves coding agents for technical documentation, why stop there?

Runbooks and incident response - on-call engineers with instant access to procedures
Customer domain knowledge - support teams with context on product behavior
Business logic - product decisions and their rationale, preserved and accessible

The pattern is the same: curate the knowledge, compress it, embed it in context, let the agent work.

The Constraints Are Disappearing

For decades, contributing to a codebase required deep technical knowledge. You needed to understand the language, the frameworks, the architectural patterns, the implicit conventions. The barrier was high.

Coding agents lower it. Claude Code, Cursor, and similar tools don't replace engineers. They make technical knowledge more accessible. With the right tooling and the right context, a PM can fix bugs. A designer can adjust styling logic. A support engineer can patch data issues.

The question isn't whether this is possible. It's how fast you can adapt.

Organizations which enable broader contribution will move faster than those that don't. The tools exist. The research is clear. What's missing is execution.

Start Simple

You don't need to document everything upfront. Start with the knowledge that causes the most friction:

Code style conventions - how you write TypeScript, naming patterns, file structure
Common patterns - how you handle authentication, API calls, error handling
Framework specifics - non-obvious usage of your frameworks and libraries
Internal APIs - if you have OpenAPI specs, even better

Create a simple doc structure:

docs/
  00-general/
    code-style.md
    tech-stack.md
  01-apis/
    api-design.md
    calling-apis.md
  02-backend/
    error-handling.md
    database-patterns.md

Generate the compressed index (pipe-delimited format, one line per directory). Add it to your CLAUDE.md or AGENTS.md file - the context files that Claude Code and other coding agents read on startup. Done.

The compressed index approach works! 🎉 Vercel's research proved it: 100% task success versus 79% without it. We've validated it internally. Now it's about whether you'll adopt it before your competitors do.

Dealing with Pushback to Product Engineering

Viljami Kuosmanen — Tue, 08 Apr 2025 13:34:33 +0000

I was recently confronted by a product engineer colleague here at epilot, frustrated with some of his non-engineer colleagues who didn't seem to buy into the idea of involving engineers in the product process from the start.

An all too familiar and frustrating situation for many of us.

You think of yourself as a proud product engineer wanting to solve meaningful problems but then get slapped with a detailed feature specification designed by a group of non-developers without your input. Now they expect you to go deliver their vision.

This is very much the reality in most product teams. Calling yourself a product engineer doesn’t automatically mean your voice will be welcomed. And that’s totally okay.

The Ideal vs. the Reality

In my past writings, I've painted the ideal: engineers who understand customer needs, question roadmaps, challenge designs, and contribute beyond code. But the reality is often much messier.

Some PMs and designers love working closely with engineers. Others are still adjusting to the idea. And that’s fair. The product engineer mindset definitely isn’t the norm.

Let’s not forget, PMs and designers are often under pressure too. It’s only natural they sometimes default to the most familiar and streamlined path. One that doesn’t always include engineers in the early stages.

And let’s be honest. Sometimes engineers haven’t yet built the trust or skills to contribute meaningfully.

The Friction Is Normal

The moment you step outside your lane, you shouldn’t be surprised when the reaction isn’t overwhelmingly supportive.

You will face skepticism.

You might be seen as overstepping.

You will encounter:

Requests to give estimates for implementing someone else's design
Roadmaps shared as top-down mandates
UI prototypes handed over "ready for dev", expecting pixel-perfect implementation
Pushback from asking too many questions

This is the price of wanting to do more than just execute. And if we’re honest, many of us haven’t always shown up in these conversations in a way that earns trust.

Is this a culture problem? Not necessarily. Most teams don’t have a rule against engineers joining product discussions. It’s just not the default. It’s less about policy and more about patterns. Changing those patterns takes trust, initiative, and persistence.

At the end of the day, it’s the product engineer’s job to show that product engineering actually works.

Earn the Trust

Calling yourself a product engineer isn’t a free pass. No one hands you a seat at the product table just because you want it. You must earn it. Show, don’t tell.

Do the homework. Know the names of your customers. Know the business domain.
Talk to your colleagues, not just other devs. Find opportunities to interact with customers.
Ask helpful questions that sharpen the team's product thinking.
Dive into data. Gather insights. Find new metrics and ways to collect useful signals.
Bring value to the table. Demonstrate you understand the customer problem by making thoughtful proposals that move the product forward.

You need to show up in demos, RFCs, testing sessions, and reviews. Not just in code commits.

Why Bother?

Because it’s worth it.

We do this for our own professional pride. To put great products into the hands of happy customers. And into our portfolios.

We don’t challenge product decisions because we want to take over the PM’s job or undermine the work of our teammates. We do it because we care about impact. Because we want our effort to count.

Because it hurts to pour weeks of your life into something that doesn't work, only to discover it failed because no engineer was involved in the concept phase.

Our Leverage

And here’s something to remember: We, as engineers, hold real leverage. We are the only ones who can actually turn product decisions into reality. No idea ships without us.

So while we may not always get a say by default, we do get a say in how we show up and how deeply we choose to care.

Use that leverage wisely. And proudly.

The Sauna Epiphany: How I Got Product Engineering Wrong

Viljami Kuosmanen — Sun, 13 Oct 2024 12:23:43 +0000

If you've seen my posts lately you've probably seen a lot of talk about Product Engineers: software engineers who talk in customer problems and take pride in the products they build, not only the code and technologies they use.

My core thesis is that with the rise of AI, the expectations for software engineers are being redefined, moving away from narrow programming roles: backend, frontend, C#, Java, React, etc. fast becoming obsolete due to AI tools like Cursor lowering the barrier of entry and unlocking productivity with any technology.

The days when you could coast on your ability to churn out code are over. If you're still clinging to the idea that you're safe just because you know how to write code in a widely used language or framework you're in for a rude awakening.

I believe this trend is likely to be the biggest industry shift in my lifetime, even surpassing the agile movement of the last 20 years. And I'm not the only one talking about it. (#1, #2, #3, #4, #5)

But I got it wrong.

Not entirely wrong, but I made it more complicated than it needed to be. In my quest to define what it means to be a product engineer, I overlooked the power of simplicity and ironically did not think enough about the customer: the ambitious software engineer thinking strategically about their careers.

I overengineered it

In my earlier attempts to help engineers break out from purely technical roles, I published an extensive checklist to help engineers think and ask questions as product engineers. It covered everything from understanding the user and the market to measuring success and staying ahead of industry trends.

Here's a taste of that checklist:

Great stuff, but let's be honest — no engineer is going to run through a 15-point checklist to make a product decision. Just seeing this long list of questions can be totally overwhelming, especially if you're used to being in a technical role.

If we expect engineers to actually start doing this stuff, the core idea needs to be simple and easy to remember.

The Wake-Up Call in the Sauna

As I'm writing this post, I'm enjoying a company working vacation in a nice hotel in Mallorca with my epilot colleagues. I was having a conversation in the hotel sauna with a principal engineer where something he said in passing suddenly clicked for me.

He was venting about some engineers on his team who seemed to be diving head first into coding without bothering to understand the problem. "It's so easy. Every engineer should be able to answer what problem they're solving, who the customer is, and what the impact of the work they're doing is."

That hit me like a splash of cold water in a 100°C Finnish sauna.

He was totally right.

Boiling It Down to 3 Essential Questions

To think like a product engineer, it's already enough to start with just three simple questions:

What's the problem?
For who?
Why is this important?

Let's dive a bit deeper into each one.

1. What's the problem?

Stop coding for a second. Do you really know what you're trying to solve? Not the ticket description in Jira, but the real-world issue. If you can't articulate the problem in simple plain English, you have no business writing a single line of code.

2. For who?

Identify your user and customer. Sometimes they're the same person; other times, they're not. Understanding who will use your product (and who will pay for it) is crucial. It shapes the way you approach the solution and helps you tailor the experience to meet their needs.

3. Why is this important?

As engineers there's never a shortage of things we could improve or implement. If solving the problem doesn't make a meaningful difference, why are you wasting your valuable time? We're not here to build features that nobody uses or cares about. Connect your work to something that actually matters and helps build your track record.

By anchoring your work in these three questions, you immediately move from code monkey to a high value product-minded engineer. Still a rare breed. You're not just implementing features someone else decided to build; you're elevating yourself to a position to influence product decisions and build smarter.

Leverage Your Team—They're There for a Reason

The biggest mistake engineers make is thinking they have to find all the answers themselves.

Most of us are lucky to work in a product team with other disciplines like UX researchers, designers, PMs and business stakeholders whose job is to help answer these questions.

By leaning on your team, you not only find better answers but also foster a more collaborative and innovative environment.

Product engineering is a team sport.

Taking Action: Start Asking the Right Questions Today

So, the next time you tackle a new topic, pause for a moment before diving into code. Ask yourself:

What's the problem?
For who?
Why is this important?

Write down your answers. If you don't know, reach out to your team and find out. This simple practice can transform your approach to work, leading to better decisions, more impactful solutions, and a greater sense of ownership.

Congratulations, you just became a Product Engineer!

Join the Conversation

I'd love to hear your thoughts on this simplified approach to product engineering. Have you tried asking these three questions in your work? What impact did it have? Share your experiences in the comments below.

Consider giving the GitHub repository a star if the product engineer role resonates with you!

Bullshit Tech Roles (satire)

Viljami Kuosmanen — Tue, 09 Jul 2024 23:50:55 +0000

In any sufficiently well-funded or otherwise successful tech company, a set of new roles will inevitably emerge—so crucial and revered by our industry that they've practically transcended the need to produce any tangible work. These roles specialize in the fine art of enabling others, in the hopes that, one day, the actual builders might be enabled enough to deliver products to customers.

I'm talking, of course, about the agile coaches and scrum masters, the architects and platform engineers; the growth hackers and data-driven researchers. These vanguards of innovation are vital to any team that wants to maintain a perpetual state of being maximally enabled, well-informed, and facilitated to perform their work.

Let's begin with the often underappreciated roles of agile coaches and scrum masters, the only people in a company actually certified™️ to set up the correct processes and tools to run an agile team. This, of course, is paramount for implementing agile—the popular software development philosophy that explicitly tells us to avoid focusing on processes and tools.

Agile coaches and scrum masters are professionals whose primary job is to ensure all work in teams is performed in sprints, with daily stand-up meetings to guarantee everyone is constantly enabled. This allows the velocity of a team to be meticulously measured in story points, which no one quite understands, but hey, at least we're definitely data-driven.

Surely, without their meticulous facilitation, teams would be left wandering aimlessly, unsure of how to break down their work into JIRA tickets. Their presence ensures that everyone stays aligned and that every team member is reminded of their blockers and sprint goals every single morning. Because nothing screams productivity like a daily stand-up where everyone takes turns saying, "no updates from my side."

Next, we have the architects. These exceptionally gifted, usually very senior engineers haven't touched any production code in the current decade. They are often found in their natural habitat, PowerPoint, where they sketch out their grand visions of future systems and envision platform rewrites for actual software engineers to implement.

Their role is to think so far ahead that their ideas become completely theoretical, creating software designs that are so advanced, they may never actually be implemented. But that's not the point. The point is to have a shared vision, to inspire, to draw boxes and lines that will one day guide the hands of actual engineers who might one day understand what they were trying to convey.

Then, there are the platform engineers. A demanding role with zero responsibility to deliver anything of value to customers. These unsung heroes dedicate their time to building and maintaining the internal tools and infrastructure that supposedly makes everyone else's job easier. Yet, their true skill lies in making things so complicated that only they can understand and manage them.

They manage infrastructure, design intricate CI/CD pipelines, common libraries and tools that add layers upon layers of necessary abstraction, which they insist must be adopted and standardized across all teams. The result? All teams having to learn and depend on custom in-house tools so complex that even the slightest changes require a detailed consultation with the platform team, ensuring their perpetual job security.

And then we have the growth hackers, the avant-garde marketers of the tech world. Their job is to come up with innovative ways to "hack" company growth, often resorting to questionable methods that straddle the line between clever marketing and outright deceit. They dive into data, running A/B tests, tweaking landing pages, and optimizing user funnels to squeeze out the tiniest incremental gains.

Of course, the real growth usually comes from the core product being genuinely useful, but why let that overshadow the need for an entire team devoted to marginal tweaks and vanity metrics?

Data-driven researchers are the prophets of the modern tech age. Armed with vast amounts of data and research, they uncover profound insights such as "users prefer faster load times" or "clearer buttons improve user engagement." Their work truly brings scientific rigor, managing to bring impressive-sounding numbers and great data visualizations to argue for any side of a decision, usually the one they already went with before doing any research.

These bullshit roles are the pillars upon which modern tech companies stand. They enable a culture where productivity is constantly measured, documented, and discussed, albeit often at the expense of actual productivity.

Without them, who would ensure that calendars get filled with back-to-back meetings so that everyone is too busy to notice no work is getting done? Who would write all the documents and Slack messages, ensuring no detail is left unshared or undiscussed? In doing so, these roles truly create a seamless, endless flow of communication, although one might argue that less talk and more doing might yield better results.

But where's the fun in that?

The Product Engineer Checklist

Viljami Kuosmanen — Tue, 04 Jun 2024 09:34:22 +0000

Download the PDF at productengineer.org

How to Think Like a Product Engineer

The following is a checklist of questions for product engineers:

1 Understand
- 1.1 Who's the user?
- 1.2 Who's the customer?
- 1.3 What's the market?
- 1.4 Ask Why
- 1.5 What do we already know?
2 Craft
- 2.1 Am I proud of what I'm building?
- 2.2 Does the product feel good?
- 2.3 How do I get there faster?
- 2.4 Teamwork
3 Growth
- 3.1 How do I measure the success of my work?
- 3.2 How do I maximise the impact of my work?
- 3.3 How do I stay ahead of the curve?
4 Product Vision
- 4.1 What’s our North Star?
- 4.2 How does my work impact the overall design of the product?
- 4.3 What's the ambition level?

1 Understand

1.1 Who's the user?

As a product engineer, your #1 goal is to create happy users. These are your fans! Always start with the user!

Your user is the person that primarily interacts with your product and whose experience your work will directly impact.

You may have multiple user groups. People who may have varying reasons to use your product or might interact with it in different ways from different angles, maybe on different kinds of devices.

You should understand how to help these users. What drives them to use your product? What delights them? What are their pains?

As a product engineer you should demand and support your team find answers to these questions before jumping into writing code.

1.2 Who's the customer?

Yes, this is a different question to "Who’s the user?". The customer is whoever pays for your product, not always who uses it.

You should know what makes your product valuable to your customers to make better decisions on what to invest your time.

Hint: If you're in B2B the answer always has to do with helping your customers save money or make more money. Understanding your customers' core business is key to understanding why they would pay for your product.

Most importantly, you want to make whoever is paying for your product look good. After all, they're the ones taking a risk by picking your product. You ALWAYS want to reward them for that.

1.3 What's the market?

Zooming out, let's take a look at the wider market landscape. Who are the potential customers we haven’t captured yet? Why would a customer pick a competitor’s product vs. mine? Are we leaving opportunities on the table?

What can I learn from similar products in the market? What are our USPs? How do I create a competitive advantage against competition?

Are there rules to the market? Are there regulations or industry standards I need to know about? Does my product have to look or feel a certain way to be taken seriously?

Knowing the market and regularly bechmarking yourself against other players helps become aware of your strengths and weaknesses and give ideas for where to invest strategically in your own product.

1.4 Ask Why

This is a bit of a product thinking cliche but still holds true: always pays to ask “why” a few times to uncover root causes of problems and underlying motivations.

Asking why can be helpful in almost any situation to build understanding in your team. Some examples:

Why should we invest into building this feature?
Why are customers asking for this feature?
Why is this a pain for our users?
Why do users give us that feedback?
Why now?

1.5 What do we already know?

It’s smart to build on what’s already known rather than always starting from scratch.

Always leverage the existing knowledge and experience of peers and leaders: founders, product leadership, designers, other product engineers, etc.

Examine the status quo to see how a problem is currently solved. Look at ideas, feedback, metrics, KPIs already collected in the past.

Do we have users already doing something like this? What are their existing workflows? How could we improve their experience?

Am I duplicating or potentially deprecating some functionality that already exists? Could this be achieved by extending or leveraging existing features? How are competitors solving this?

2 Craft

2.1 Am I proud of what I'm building?

Your track record as a product engineer is the products and features you've delivered. Your last feature represents your professional competence level. No excuses.

Ask yourself what quality standard do I want to set for the work I put out there?

Can I be proud of the product I worked on? Is my work well tested and polished? Did I cut corners where I shouldn't have?

As a highly paid professional engineer your craft is to produce high quality software which includes avoiding the creation of technical debt. Never ask for permission to improve quality!

What makes a great product engineer stand out from the average software engineer is an intense sense of professional pride in their work and product.

2.2 Does the product feel good?

This may be a slightly controversial take, but I believe a surprisingly large part of building great products is about developing a good taste for it.

Simply knowing the difference between great vs. average and not settling for just ”ok” helps tremendously in making good decisions as a product engineer.

Does the product feel smooth and consistent? Is it intuitive, simple, and familiar to the user? Or does it feel cheap and janky?

Note that this doesn’t only concern the visual aspects of your product. Just slapping a fancy UI design on a shaky foundation doesn’t create a great experience.

Simple. Elegant. Clean. This is what we're after as product engineers.

2.3 How do I get there faster?

The pace of innovation especially in software is so rapid that in order to be competitive you must deliver fast, early and often.

Too slow and your customers will lose trust in you while your competitors overtake you.

What many get wrong about agile and building products is optimizing for predictability with estimates and roadmaps. Rather, what you really should care about is visible and continuous progress towards product goals.

The goal of estimates should not be to try to be as accurate as possible but rather to set ambitious and yet achievable goals for yourself.

The question is not how long you think it will take to build, but how long should it take? What’s an acceptable amount of time and effort I should invest on this?

What’s the rollout strategy? How do I get this into customers' hands as soon as possible? What’s the MVP?

2.4 Teamwork

Building products is a team sport.

You are absolutely not expected to work alone and do everything yourself from writing code to doing user research.

Am I effectively communicating with my team? Am I leveraging my team members' strengths? Are we celebrating our successes?

Great teamwork results in great products. Invest in your team, and you will see the dividends in your product's success.

3 Growth

3.1 How do I measure the success of my work?

Our work as product engineers is not just about building and shipping feature after feature.

We make educated guesses about the most valuable thing to work on, so we should also be able to answer the question: What’s the impact of my work?

How many customers did we talk to to validate our progress? Are they coming back? How much money does it generate?

Use analytics tools and user feedback to help you understand what’s working and what’s not. Talk to real users to get qualitative insights about the product.

Most importantly, make sure to share your outcomes openly and transparently. What did I ship in the last few months? What were the results?

3.2 How do I maximise the impact of my work?

The most important question you should regularly ask yourself is: am I focused on the right thing?

Being a good engineer is finding the biggest bottlenecks that hold us back and figuring out how to solve them. You should prioritise your efforts ruthlessly.

Building products is a team sport. You should actively communicate what you’re working on and why, so that others can help you and keep you accountable. Demo progress frequently and actively seek feedback.

Don’t fear taking risks. Being bold and taking the lead on delivering new and innovative features you believe in can lead to big wins.

Ask yourself: How can I align my work with our company goals? How does my work directly benefit our users? Is there a way I can communicate my work better to others to get better feedback?

3.3 How do I stay ahead of the curve?

Stay updated on market trends. Attend conferences, read up, follow experts.

Make sure to research and benchmark competitor products, or other similar products whenever possible.

Hold frequent retrospectives and brainstorming sessions. Experiment with new ideas. Encourage creativity in your team.

What are the latest trends in my industry? How can I encourage innovation within my team? Are we taking enough time to learn from our successes & failures?

4 Product Vision

4.1 What’s our North Star?

To align your work in the context of the broader product vision, you should deeply care about what others in the company are doing and saying, making sure you fully understand and are committed to the overall product strategy. Asking “why” is crucial here.

What are our current product goals? What’s our growth strategy? Where do we want to be in the next 2-5 years?

When presenting your work it always helps to put it in the context of how it pushes us forward in the big picture. How does my work help us reach our North Star?

4.2 How does my work impact the overall design of the product?

Understanding the existing software’s design and architecture decisions helps make sure that your work fits well within the larger product design. Always consider how your work influences and is influenced by other components.

When adding new functionality, you should ask:

Could this be achieved by extending or leveraging existing core features?
Does my design follow a consistent style to the rest of the product?
Am I adding complexity or reducing it?

Simplicity is worth fighting for.

4.3 What's the ambition level?

It’s a good idea to define the ambition level when setting off to build something new. Are you aiming for an incremental improvement or a revolutionary new functionality?

Is this feature a USP or a basic expectation? What’s the ROI on building this feature? Does it make sense to spend the effort building this ourselves, or could we use an off the shelf library or service?

Your time is extremely valuable. Think like an owner: Would you invest your salary to build the feature, or rather on something else?

Product Engineer Mindset

If you enjoyed this checklist, you might also like the Product Engineer Manifesto on GitHub.

Consider giving the repository a star if the Product Engineering philosophy resonates with you!

What is a Product Engineer?

Viljami Kuosmanen — Thu, 29 Feb 2024 08:04:56 +0000

Read the full Product Engineer Manifesto at productengineer.org

Not Just Coders, but Builders

Picture this: engineers who don't just speak in code but in product design and customer problems.

Product Engineers are a special breed of engineers that don't just see themselves as coders or developers but as builders who deeply care about the products they build.

They're driven by a professional pride and desire to build great products, transcending the traditional developer role to become genuine drivers who go on to put their names behind great products they themselves played a key role in shaping.

The Counter to Hyperspecialisation

The tech world has seen for a long time a trend towards hyperspecialisation, with engineering roles becoming increasingly narrow.

We have frontend engineers, backend engineers, iOS engineers, DevOps engineers. We have developers defining their careers with a specific language: JavaScript engineers, Swift engineers, C# engineers, Python engineers. We even have developers that seemingly dedicate their entire careers to a single framework or tool: React Engineers, .NET engineers, Unity Engineers, Ruby on Rails engineers.

Feels like it's only a matter of time until we see job postings seeking for-loop engineers and variable naming engineers.

Jokes aside, specialisation isn't totally without its merits as it has allowed engineers to build deep expertise in specific technologies, a key component in building quality software. However, it also lead to silos where collaboration and broader product understanding became a nice-to-have, and often not even an expectation or focus for engineering roles.

Product Engineers stand as a counter-movement to this trend. They embody a holistic approach to engineering, where understanding the entire product and context around it is just as important as the technical skills required to build it. This broad perspective enables them to bridge gaps between different technical domains and ensure that the product serves its customers effectively.

AI: The New Playground

The rise of artificial intelligence (AI) tools in software development is rapidly setting new expectations for engineers. With AI tools becoming more sophisticated and capable, engineers are now expected to leverage these tools to enhance their work, not just in terms of speed and efficiency but also in terms of broadening their area of responsibilities.

Engineers in a post- ChatGPT and GitHub Copilot world are now expected to be able to work in more languages, incorporate more tools and libraries, and simply be able to deliver more, using a broader range of technologies and disciplines in their daily work.

It's probably not a great time to be the platform engineer whose job is to set up Jenkins pipelines for the product teams, when ChatGPT can easily help teams do it themselves.

For Product Engineers however this has unlocked a higher level of abstraction. With AI tools handling more of the routine and specialised tasks, engineers can allocate more time to creative problem-solving, ideation, and exploring new ways to meet customer needs.

This new level of abstraction allows Product Engineers to concentrate on product strategy, user experience, and overall system architecture without being bogged down by the intricacies of individual technologies.

The incorporation of AI into software development encourages a broader perspective, where the choice of technology becomes a means to an end, rather than an end in itself.

While AI didn't create the Product Engineer role (examples have been around for much longer); AI has made becoming a true Product Engineer much more accessible as a career path for software engineers.

Combining Product Thinking and Technical Execution

Product thinking involves understanding the user's needs, the market demands, and the business goals that drive a product's development. It's about seeing beyond the immediate task to grasp how each piece fits into the broader puzzle of the user experience.

Product Engineers ask the crucial questions: "How does this feature add value to the user? What impact will this improvement have on the product's overall vision? How can we measure the success of our work?"

They don't just build features based on specifications; they contribute to the design and roadmap of the product through a robust understanding of the customer's needs and business strategy.

This approach requires a balance of skills: the ability to dive deep into coding and system architecture, while also keeping an eye on the product roadmap and customer feedback.

By combining this mindset with quality technical execution, Product Engineers can uniquely ensure the product is headed to the right direction both technically and strategically, ensuring that their work directly contributes to the product's long term success.

The Manifesto: A call to Product Engineers

I've published a Product Engineer Manifesto on GitHub to more formally define the Product Engineer mindset.

Consider giving the repository a star if Product Engineering philosophy resonates with you!

After 3 years: Bets I made as Head of Engineering rewriting the epilot SaaS – worth it?

Viljami Kuosmanen — Thu, 15 Feb 2024 21:56:48 +0000

In this post, we take a look back at some of the most influential decisions I implemented early on in our SaaS rewrite journey at epilot and evaluate how well each one has played out after 3 years.

Skip ahead or read on until the conclusion for a full verdict of whether rewriting our app was a good idea to begin with.

Doubling down on AWS & serverless

Good bet 👍

To be fair, AWS was already the de-facto cloud platform used by the epilot team before I joined.

My contribution was to convince the team to go all in on a serverless architecture with an ambitious strategy to rewrite and replace our proprietary Java-based 3rd party monolith running on EC2, piece by piece with functionalities leveraging serverless AWS offering.

After 3 years, there are no regrets to choosing to be fully vendor-locked with AWS for the speed, scalability and minimal operations overhead it provides.

Microfrontends with single-spa

Good bet 👍

The decision early on to take a microservices approach in our frontend was made out of purely practical reasons.

Frankly our small team who lead the work on the new platform didn't have the time, the confidence, nor the desire to make long term decisions on the frontend tech stack, tooling and important design decisions on behalf of all our other teams, present and future.

I'm extremely happy we went down this path instead of imposing one framework and toolset for everyone. The single-spa approach lets us improve our frontend game each time a new microfrontend gets added to our application, which by now has happened more than 30 times! We've even ended up replacing entire MFEs (microfrontends) a few times, which isn't as bad as it sounds when the codebases are kept relatively small.

Notably the microfrontend approach also helped us support the transition from the legacy product to the new one, as we were able to treat the legacy app as just one of our many (micro)frontends, and slowly replace functionalities one by one by shipping new microfrontends.

Using React without a framework

Bad bet 💸

I know, "skill issue", right? But honestly I've found that even the really senior (read: expensive) React engineer hires tend to struggle with performance and end up wasting a lot of time on fixing major performance issues with React applications when they start to do anything meaningful.

Starting over, I would probably spend a bit more time choosing a React framework that works well in a microfrontend environment rather than encouraging all teams to start from scratch, and establishing no-framework as the implicit default.

Controversial: I might not even pick React for most MFEs anymore. We already use Svelte in parts of our app, and turns out Sveltekit works super well in a single-page-app environment like ours!

RFCs

Good bet 👍

The practice of writing RFCs to collaborate on software design and async decision making was one of the first and best things I introduced joining epilot.

To this day we use RFCs as the main way to share ideas and be transparent about the work that our engineers do. I'd consider this one a very good bet and would absolutely recommend a culture of writing RFCs for all product organizations.

API First

Good bet 👍

We give a ton of freedom to engineers to pick whatever languages, tools and techniques they see fit, but the one thing I've always demanded from our teams has been to design using machine-readable API contracts, such as OpenAPI or GraphQL.

I would say this is internally the most important way we collaborate on our software's design, using shared types and automated tests to ensure our implementations follow the common API contracts. This approach has radically reduced bugs and allowed our engineers to work together efficiently leveraging platform features built and maintained by other teams.

On top of that, being API First and publicly building and sharing our API contracts has allowed our customers and partners to effortlessly build on our product, and integrate it deeply with their own IT landscape using our API & SDK.

Continuous production deploys from `main`

Bad bet 💸

Turns out our customers simply did not tolerate the frequent changes and inevitable rollbacks of this approach.

We technically still ship continuously from main to our production environment, with an average of 150 production deployments every week, (with a lot of tests!), but now ship a monthly stable snapshot release to paying customers, and heavily utilise feature flags to roll out changes.

Not using feature flags from the beginning

Bad bet 💸

We did use GitLab feature flags on launch, but they were only used to turn features on/off between stages (dev, staging, production).

Turns out this wasn't enough. Our teams really needed better tools to give us fine-grained control over rolling out features to specific customers.

Going back, I wish we had started earlier with something like LaunchDarkly.

Hiring product-focused senior engineers remotely

Good bet 👍

This is a decision I'm very proud of. Keeping an almost unreasonably high bar to getting hired and seeking out individuals who care about the product, our customers and our business has allowed us to build and retain a motivated team with zero managers and architects.

Making engineer hiring remote-first allowed us to tap into an international pool of senior talent resulting in a team of more than 30 exceptional and motivated hands-on Product Engineers from more than 10 countries working for us, and growing... ❤️

In-house DevOps / platform team

Bad bet 💸

We tried to establish a platform team on two separate occasions but both times the team struggled to produce value for our teams. The DevOps engineer hired to lead the second platform team ended up leaving us for a bigger company after a few months.

Looking back, part of the reason for this was likely a combination of leveraging managed/serverless services that don't require much in-house automation work, and hiring senior engineers who can automate things for themselves when needed.

We've recently started experimenting working with a 3rd party DevOps consultancy to join us on a project-basis to help us with non-product related engineering tasks such as optimising our CI/CD pipelines.

Regardless, we don't anymore see any need to hire and establish an in-house DevOps team, or get into Platform Engineering of any sort in the foreseeable future.

Weekly Demos

Good bet 👍

The all-hands weekly demo is still easily my favourite recurring event at epilot. It's like having a company-wide celebration of shipping things, every week!

The weekly demo brings not only a positive buzz around all the cool stuff being built, but also the expectation that engineers personally showcase concrete customer value created in their teams every week. It keeps us accountable for consistent delivery and maintains a rapid feedback loop with the rest of the company.

Open-Source Engineering Principles

Good bet 👍

Transparently laying out our Engineering Principles in a public GitHub repo helped set clear expectations for our culture internally, and perhaps even more importantly for engineer candidates, many of whom have testified making the decision to apply at epilot because of our principles resonating with their own beliefs.

Hasura

Good bet 👍

One of the early key bets was the decision to use Hasura to interface with our legacy database to break up the monolith into microservices.

Event triggers listening on DB changes and pushing messages to SQS while using the GraphQL API to implement 2-way data sync with the legacy app turned out to be a great and reliable event-driven pattern for breaking up the monolith step by step.

Since shutting down the legacy product last year, Hasura was decommissioned together with the legacy Postgres database, but it truly served its purpose well during the migration to serverless. 🫡

DynamoDB as our go-to database

Good bet 👍

We haven’t had any fundamental issues with DynamoDB single table design in our backend microservices.

Of course it’s not suitable for every use case, and we have turned to Aurora serverless, Elasticsearch, and even InfluxDB in a few special cases, such as for search and analytical needs. But for 80% of the time, DynamoDB is fast, convenient and extremely worry-free as a default application database.

AWS OpenSearch service

Bad bet 💸

We started out on Amazon OpenSearch Service (still called Elasticsearch service back then) but were struck with a pretty bad incident where our domain became completely non-responsive in our development environment and the only thing we could do was to contact AWS support to resolve the issue. This was totally unacceptable for us as we heavily rely on Elasticsearch for our main functionalities.

Elastic Cloud ended up being far more reliable and manageable for us in the end. Having a self-service reboot button on the interface makes all the difference!

However with both providers rolling out their serverless solutions, which would be extremely attractive to us, we're keeping our eyes open and are open to experimenting again with AWS OpenSearch offering.

Datadog for observability & monitoring

Good bet 👍

Datadog is one of those tools that once you get used to it, you wonder how you ever lived without it.

The value we get from Datadog's UI for logs, traces, monitors and dashboards is insane, and absolutely worth every penny.

We use Datadog RUM for browser & UX monitoring. The RUM sessions and logs are extremely helpful for tracking down frontend crashes and bugs, as well as visually understanding user behaviours.

On top of that, we use pipeline observability to analyse CI pipelines, custom metrics to track business & application KPIs, and cloud security management for compliance reporting and threat detection.

Datadog for incident management & post-mortems

Good bet 👍

Datadog Incident Management has been absolutely vital in making sure we take the right steps to analyse and learn from our incidents. We use it during incidents for coordination and communication, as well as after the fact for post-mortem and analysis of past incidents and trends.

Datadog for browser tests

Bad bet 💸

Datadog synthetic browser tests were great to get started fast but quickly became slow, expensive, hard to expand and simply not good enough for our team.

We ended up switching over to Playwright.

Redshift Serverless

Bad bet 💸

Quite slow. Very expensive. At least for our use case of generating dashboards with Spectrum querying mostly parquet files from S3.

Our team is currently investigating replacing Redshift with Clickhouse.

Microservices

It's complicated 😎

At this point I've stopped referring to our product as having a microservices architecture. I don't find it a very helpful abstraction to understand a large, event-driven, interconnected product that's mostly FaaS.

Yes, we still develop and deploy our services as independent domain-specific modules with well-defined APIs.

In reality however our application is used by the customer as one big product with all the pieces needing to connect and work together to provide a service.

Thus, we must also test and ship the product as one. (Really, we have one large shared e2e test project that blocks all pipelines. When we're shipping our monthly stable release, we take a snapshot of all our bundles and ship those in one atomic import map file.)

I simply find it more useful to think of our product as one monolith, made up of a large number of independent infrastructure & FaaS modules with APIs.

Conclusion: Was the rewrite a good idea?

If you ask me now I would never recommend any SaaS company take the decision to fully rewrite their software. It's generally just a pretty dumb and painful thing to do. 😅

For epilot sadly we were left with no other choice.

The legacy version of our SaaS was built on top of a proprietary 3rd-party low-code platform Axon Ivy, which by no means was designed to run a multi-tenant SaaS, but that the external epilot dev team from Vietnam (affiliated with Axon) had managed to wrangle to work.

The fact that epilot was able to build a successful and fast-growing business on top of what's essentially a hacked low-code business process automation tool, is the ultimate proof to me that the tech stack really doesn't determine a company's success!

But we were seriously starting to hit the limits of our software, especially when it came to scalability and speed of development. We didn't even have access to the source code so that we could take over the development of the core application. All we could do was upgrade to new versions provided by the vendor and continue to hack the software to make it do more things it wasn't designed for.

Simply put, it wasn't sustainable for a fast growing software company. No developer wants to work on software like this. Especially not the talented and ambitious ones we wanted to work with.

It took us nearly 3 years, but in the end we managed to reach feature parity, replace, migrate and shut down the legacy app, with very little churn, growing year-over-year and last year doubling our MRR while migrating the last customers over to the new platform.

Feels good to say we're only getting started with our new 360 platform.

Looking back at the last 3 years, while I would never recommend anyone pursue a rewrite for a running SaaS business, while simultaneously trying to build a healthy, ambitious engineering team and culture, I'm convinced that for epilot this was the (only) way to go, and we're much better for it.

In the end we at epilot overwhelmingly consider this a…

Good bet 👍

What's next?

To keep things short in this post, I'll publish a separate Part 2 where I share our latest bets focused on building epilot into the next SaaS Unicorn. 🦄

Mandatory recruitment disclaimer: If this post made you at all curious, please check out our docs and check our open positions for more!

We’re hiring @ epilot

Comparing REST, GraphQL & tRPC

Viljami Kuosmanen — Tue, 10 Oct 2023 10:59:54 +0000

As an author and maintainer of open source libraries to work with REST APIs, and as someone making a living building an API First SaaS product with both REST and GraphQL APIs, I sometimes encounter strange attitudes towards different API styles.

"REST is outdated" "GraphQL is only good for mobile dev" "tRPC is too new for production" "No one likes to work with swagger"

It might be tempting to pit these technologies against each other, but the reality is, these paradigms are not direct competitors. Instead, each has its own unique strengths and use-cases, crafted for specific contexts and developer needs.

I was reminded of this fact recently when writing comparisons between my own open source REST tooling vs. GraphQL and tRPC:

// Detect dark theme var iframe = document.getElementById('tweet-1711285703840084201-562'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1711285703840084201&theme=dark" }

Let's take a deeper look at each technology and its strengths:

GraphQL

GraphQL is a query language for APIs developed by Facebook. It gives API clients full control over the data they query, making it extremely flexible and efficient for client-centric use cases.

GraphQL APIs define a strongly typed schema for the data and mutations they support which makes them discoverable and intuitive to develop against.

Teams building with GraphQL naturally take the API First (or Schema First) approach where the API contract is treated as a first class citizen in the software design instead of treating it as merely documentation.

When developing an app with a GraphQL API, it's common to use the GraphQL schema as the single source of truth for data models used within the application, and source types from it used end to end across the codebase.

GraphQL APIs are often used as gateways, merging multiple downstream APIs into a single, easy to use interface for a great application development experience. I would consider this one of GraphQL's killer apps.

Facebook's excellent Apollo GraphQL is definitely a best-in-class example of great API development experience.

REST

REST has been around for a while now. It's still probably the most widely adopted HTTP API design paradigm perhaps due to its resource-based implementation simplicity and the ability to leverage standard HTTP features directly.

With widely adopted open standards to define REST APIs, like OpenAPI specification (previously known as Swagger), REST can be used much like GraphQL with a single source of truth for the API contract.

Full disclosure: my own open source work in the last few years has revolved around the mission of bringing a GraphQL-like developer experience and type safety with OpenAPI for REST.

While REST APIs don't generally provide the same level of control to clients as GraphQL, sometimes this could be seen as a benefit especially in scenarios where strict control over data access and operations is crucial.

Widespread knowledge around REST contribute to its choice among organizations looking for a tried-and-tested approach to building APIs.

tRPC

While RPCs are by no means a new idea, tRPC is seen as the new kid on the API block. Much loved by developers who just want to build fast and do it with type safety.

Designed for full-stack Typescript applications, tRPC allows direct sharing of types between both the client and server, without relying on code generation.

Unlike GraphQL and REST, tRPC doesn't expose a standard machine-readable API schema to be consumed by clients, instead taking a more straightforward approach of exposing endpoints or procedures, essentially "just functions" invoked by the client to the server.

While the lightweight tRPC approach is optimal for teams just looking to build full stack applications, teams looking to build robust, re-usable APIs may better be served by the API design first approach of GraphQL or OpenAPI.

(I wonder if tRPC will eventually get its own YAML specification language)

Conclusion

In the landscape of software dev, it's easy to get caught up in the rhetoric of one technology being superior to another. But as we dive deeper into the nuances of GraphQL, REST, and tRPC, it becomes evident that each serves its unique purpose and has been created to solve specific problems in API development.

The right choice of an API paradigm is definitely not a one-size-fits-all solution.

It's essential to remember that the success of an API doesn't necessarily depend on the technology backing it, but more on its clarity, usability, reliability, and the developer experience it offers.

Technologies change, but the principles of good API design and the goal to write delightful, working software remains constant.

If you're interested in my Open Source work around API tooling, check out openapistack.co

How we do engineering @ epilot

Viljami Kuosmanen — Sun, 22 Jan 2023 15:35:57 +0000

Ever since open-sourcing our core Engineering Principles back in January 2021, I regularly get asked by candidates how and if the principles really work in practice at epilot.

In this post, I'll attempt to give a glimpse of concrete tools and practices we use in our product team to give an idea of how we apply our principles and what the engineering culture really looks like from the inside.

Team Structure

At time of writing, the epilot engineering team consists of 25 full-time remote software engineers organized into 6 squads, each with its own Product Manager and a shared design team of 4 talented product & UX designers.

Each squad is led by an engineer team lead, principally responsible for the team's delivery and engineer wellbeing.

The product leadership team is responsible for communicating our product strategy and maintains a Now-Next-Later roadmap, but ultimately it's the squads that make all delivery decisions.

Squads are formed around specific business domains within the epilot 360 platform, architected around Domain Driven Design and micro-ish services both in the backend and frontend.

Show, Don't Tell

We value real concrete working software, not roadmaps and fancy presentations.

Engineers showcase features in a weekly all-hands Product Demo session: a company-wide celebration and ruthless feedback session. 🎉

Features being demoed are deployed to production but are usually hidden behind a Feature Toggle, giving demo participants a chance to test and give feedback on features before release.

Most squads use a Kanban-style flow with a weekly planning cycle, but opt to showcase their progress every 2 weeks, with some squads hosting extra demos called "Open Houses" on off-weeks.

The engineering team also hosts an internal bi-weekly "Tech Exchange" to wind down with a beer, but also present and demo cool technical things.

API-first & serverless

The epilot backend is built as serverless microservices written mostly in Typescript, leveraging services such as AWS Lambda, Step Functions, EventBridge, API Gateway and AppSync.

DynamoDB is the most popular database solution enhanced by managed data stores such as Elasticsearch Cloud, Aurora Serverless and Redshift Serverless.

Our backend is pretty event-driven, with heavy use of EventBridge, SQS and Step Functions to implement asynchronous business logic.

Our frontend application consists of microfrontends written in React and Svelte orchestrated by the single-spa framework.

The epilot frontend is entirely built on the same APIs we expose to our customers and technical partners via our open-source SDK and public documentation.

We like open standards, so our APIs are defined using common machine-readable specs like GraphQL and OpenAPI.

We find the domain-driven architecture approach an efficient way to organize our teams to act independently while collaborating on software design with an API-first approach.

Organizing our codebase with microservices helps our fully remote engineering team focus on delivery and minimize bikeshedding over non-critical engineering decisions.

Freedom and Responsibility

Ok, I freely admit we stole this one from the Netflix culture deck. Shoulders of giants and so on. ♥

At epilot, every hire is trusted and treated as an adult from day one. We only hire smart people who we can trust and aren't afraid to terminate a relationship if that trust is broken.

We expect engineers to operate by principles and think strategically to benefit our customers and our business. (Max-the-MRR)

We intentionally make it very difficult to create company-wide policies to control which tools to use, what employees should be allowed to do and how they should organize their daily work.

It should be hard to introduce new processes but easy to get rid of existing bad ones.

The exception to this rule is security.

We encourage team members to challenge the status quo with RFCs proposing ways to improve the product and our ways of working.

RFCs - Solutions Over Problems

Among other things, RFCs written by engineers have so far led us to adopt continuous deployment, harden security, establish new teams, and even rewrite the entire product in a different tech stack.

These are often simple documents that start out by just describing a problem and expressing a wish or an idea to solve it.

RFCs at epilot have no official approval process. We expect the author to champion their proposal by garnering relevant feedback from peers and stakeholders, and then make an informed decision on how to proceed.

In most cases, the RFC author leads by example and implements the proposal themselves, usually just within her squad at first.

Good ideas spread and are adopted through all teams, not by mandate from above, but by actually improving the quality of our work.

Edit: A total of 192 RFCs have been published in our Confluence since they were introduced 2 years ago. 🔥

Every Week is Quality Week

The name of this principle came about after a group of engineers expressed concern about technical debt in their project due to pressure to deliver against deadlines.

At the time, it was already an established practice to fix bugs immediately and treat all types of bugs as critical, but many engineers still felt that they couldn't go against the wishes of product managers to follow the practice.

It was proposed to have a "quality week" at the end of every quarter to give engineers time to focus on quality topics.

As the quarter came to a close, we announced the quality week with instructions to all squads to focus solely on bugs, technical debt and other quality improvements they felt were important.

During the quality week, our engineers were extremely motivated to fix long-time issues that had affected our customers.

We made real, significant progress that week and everyone could see it.

On Monday following the first quality week, we opened this PR, making an announcement that every week from now on would be quality week:

It worked. We're still starting a new quality week every Monday.

No-Ops & Continuous Deployment

At epilot you will find a distinct lack of epilot-specific shared tooling. We actively avoid building our own tools and standardizing configurations and instead encourage using open-source tools like AWS CDK, lambda-powertools and single-spa to build our product.

We don't have dedicated DevOps engineers or a platform team to build custom tools and pipelines or run k8s clusters.

We expect squads to independently automate testing, deployment and monitoring of their features to minimize the need for manual operational tasks.

Engineers really get to pick any tools they like to do their job, as long as they weigh tech decisions against our principles.

I sometimes get engineers pushing me to leverage my position as head of department to standardize tech choices across the entire engineering team, but so far the approach of giving teams radical freedom has yielded organic adoption of best practices and tools where needed anyways.

Without extra push, all teams have converged to using the same GitLab CI templates, connecting up with Datadog to continuously deploy code with synthetic browser tests integrated into the pipeline to make sure we catch problems before they get to production.

The whole team also uses Datadog (another RFC) as the central observability solution to know what's happening in our platform at all times.

When stuff goes wrong, engineers owning the feature jump on a Slack huddle using Datadog's incident management feature to coordinate the response, cleanup and write a post-mortem.

Why principles matter

When I joined epilot, I made it my personal mission to build an engineering culture that I would personally want to join and be part of.

To me, the engineering principles are like a founding document; a source code or DNA of how to build a great engineering team.

I'm proud of how far we've come, but there's still lots to learn and improve as we grow and mature as a product team.

I fully expect that in 2 years, writing this blog post would look very different as the industry moves forward with new tools and practices. I hope that these principles will stick.

If the topics in this post resonated with you, make sure to check out our dev docs and see our open positions!

We're hiring @ epilot!

First year as Head of Engineering @epilot - Killing the Legacy Monolith

Viljami Kuosmanen — Fri, 17 Jun 2022 08:34:45 +0000

This post is a personal account of my experience at epilot.cloud and our journey rebuilding our legacy Java monolith SaaS from the ground up with AWS, serverless and microfrontends, mostly during 2021.

The text is written from the perspective of engineering and focuses on the challenges I faced taking on leadership of our growing team of 25+ engineers.

I fully acknowledge this is not the complete picture of epilot's growth during this time as we have many other amazing departments and individuals working to make epilot the next tech unicorn success story.

Full disclosure, I love this company and am heavily invested in the product's success after my first year and a half. 💙

(I'm the one dressed up as the doctor for our company's annual Cologne carnival celebration)

Beginnings

I joined the Cologne-based startup epilot in October 2020 as a cloud engineer to head a new platform team and aid the product team transition to serverless AWS.

The task felt right in my wheelhouse.

By that time I already had three real serverless products under my belt, had experience migrating monoliths to microservices, and had acquired more than 8 years of professional experience in total as a developer and team lead working mostly with AWS and other cloud platforms.

I knew startups. I had experience with the business domain. I knew the technology well.

This was my time to shine.

Build better software, faster.

As our CTO Szilard Toth put it in our first discussions, my job description was to help us "build better software, faster".

I took that mission statement very seriously.

Coming off a 4-year stint at Futurice, a lean service design focused consultancy, my first instinct was to make a current state analysis of my new company - epilot.

To get my bearings, I joined one of the engineering teams and got busy.

I learned about the challenges our teams were facing and got a deeper view into our legacy product, which had already been in active development for more than 3 years.

We were already doing a lot of things right: All three teams at the time were generally following agile practices, engineers were eager to discuss and learn their new tools, many had even taken the time to get AWS certified.

But there were certainly some big concerns.

At the end of my first full month, I was ready to present my learnings to the company.

First insights

I published an internal document titled "epilot Tech Review" detailing my assessment of the current state of epilot's engineering.

These were the main findings:

The decision made (before my time) to pick AWS, serverless and React as our foundation was a good one for what the team wanted to achieve: Low operating costs and high development speed. This was especially great for hiring due to many candidates being attracted to these modern technologies.
Our teams lacked experience working with these technologies. Progress was slow and unfortunate decisions were made with the very first small scale serverless projects.
Alarmingly, the engineering team had no viable plan or strategy to migrate the legacy product to the new serverless architecture. Everyone was excited about the new tech but we clearly had no clue how to actually build our product with it.

I emphasised in the document how important it was for epilot's success to move away from the legacy technology stack, a proprietary monolithic Java framework which had already proven to be a poor fit for our use case, difficult to run and extremely slow to develop new features on.

Our ability to deliver as an engineering team was looking pretty grim. The company had already invested a full year into going serverless with not much to show for it.

To make matters worse, some of our key engineers had just announced they were leaving the company.

In spite of our woes on the engineering side, epilot had a rare advantage:

A proven product-market fit – Even with our product leaving much to be desired, our paying customers were becoming fans!

Yes, they were sometimes frustrated with the quality of the software and our inability to deliver features they were asking for, but they were totally bought into the vision of what epilot could be and we were clearly delivering value for them.

I think this was thanks to our visionary leadership with deep industry knowledge knowing which problems to solve along with a great sales team and a MacGyver-like customer success team working around the limitations of our software.

We had the right idea, just not a great technical execution yet.

Luckily I knew I could make a difference when it comes to execution.

RFCs and API first

I went on to establish the platform team I was hired for. We started small with myself and an SRE who had been at epilot a year before me.

The first things the team tackled were improvements to developer workflows such as enabling an API Design First approach for our serverless APIs and building common CI/CD pipelines for the tools our teams were using.

I introduced RFCs for the teams as a tool to discuss designs and proposals to improve our product and work.

With key engineers leaving us, it was important to establish an open culture with teams making independent decisions while focusing on good software design and communication.

As one of the very first RFCs shared, I laid out my admittedly optimistic plan to move away from our legacy monolith app.

The radical 360 plan

I brought my plan directly to senior company leadership triggering a series of discussions in late 2020 to figure out our legacy migration strategy.

In sessions with our CEO Michel, CTO Szilard and head of sales Marlon, it became very clear this would not be just a migration from monolith to serverless. We needed to build the new version of epilot SaaS with a radically different design, starting from a new flexible data layer to power the diverse set of use cases our customers were asking for.

We called it epilot 360. And it would be my job to build it.

From the start of 2021, we established a new strike team to head the effort of building 360. I doubled as the leader of both the platform team and the new 360 team.

The 360 plan was radical. We would completely rebuild our main portal interface using frontend microservices and embed the old product into the new portal as a microfrontend to provide backwards compatibility while we rebuild and migrate features one by one in the new serverless world.

Getting the old monolith to work in the new portal was our first big technical hurdle. We developed a "headless" mode for the old app, where the UI was modified to fit into the new portal layout and implemented secure communication with the parent 360 app to pass information like the current user's session to the legacy app.

From the data side, we continued to split up the legacy SQL database into serverless microservices. We introduced Hasura to act as the gateway to synchronise data between the legacy database and our new serverless services.

By March 2021, we had migrated our first module to 360 using the new flexible data model and had shipped a new portal microfrontend for it.

We introduced Datadog for centralised logging and monitoring, as well as to provide crucial analytics and observability tools for our new serverless product.

Finally in July of 2021, we greeted all our users with a whole new 360 login and portal experience, wrapping the old familiar app in our brand new look with serverless features already replacing many legacy views.

This felt like the first big launch of 360. Although lots of work was still ahead to fully get rid of our legacy app, our users were happy to see us start to deliver on our promises with a new major release of the epilot product.

From here on the rate of new releases in our app would radically increase.

Head of Engineering

In May of 2021, I was officially promoted to head of engineering which meant I was no longer just in charge of the platform team and the 360 team. I was now in charge of the entire engineering team with our CTO shifting his focus away from engineering to product management.

In the meantime we had also ramped up our recruitment efforts. We published our in-house engineering principles and built a new recruitment landing page promise.epilot.cloud to showcase our culture. As a result, we managed to acquire some amazing new engineering talent during 2021. <3

So far it had been mostly just my small team of 4 developers working on 360. Now as the new head of department, I needed to shift the focus of all our by now 25+ engineers and 6 teams to work on the new 360 platform.

This came with some major challenges.

Scaling the team

One of the biggest challenges was the radically different underlying design of 360 with its flexible data model as opposed to our legacy product's traditional static data model.

Teams initially struggled to wrap their heads around the new flexible entities; not least because of my struggle to communicate it properly. Through some much needed visual design support and hands-on experience with the product, we slowly built up an understanding of our new 360 product together.

A major focus of the architectural design of 360 was on modularity by utilising frontend and backend microservices combined with our API First software design approach.

This was to allow teams to work independently giving them the freedom to choose their own tools and ways to solve problems.

We found this approach to be a big success as it kept our projects small with teams motivated to try out new technologies, share and learn from other teams, and quickly change approaches when needed.

After the first few months of teams working on 360 it was clear that we were able to move much faster than ever before with our new tech stack.

We adopted continuous deployment across all teams and quickly surpassed an average of >100 production releases each week. This was a huge achievement considering that back when I joined we were able to deploy only 1-2 times per week during a pre-defined maintenance window!

By the end of 2021, all our teams had shipped major features on the 360 product and we had reached enough feature parity for the first new users to be onboarded to epilot completely on 360 with no trace of the legacy product.

We had an MLP - Minimum Lovable Product!

Things would speed up even more throughout the first half of 2022 with teams getting comfortable with 360, shipping more serverless features to get us closer to the magical feature parity state with our rather large legacy product.

We opened up the 360 platform to external developers and partners with our public developer docs and open source SDK.

In the meantime our sales and customer success teams have been actively engaged with existing customers to help them migrate to 360. More and more users are now adopting the new features each week and we’re learning from them to make 360 even better.

Takeaways from the first year

As I'm writing this post in June 2022, after my first full year as head of engineering and after almost a year of running the main epilot application in production on AWS serverless and frontend microservices, I can't help but ask myself:

Was 360 a success?

While my ego would love to give a resounding "Yes!" as the answer, I can't be fully satisfied until we've fully completed the migration of all features from the legacy app and have successfully shut down the old monolith and moved on with happy users.

We're not quite there yet.

Are we building better software, faster? – Yes! The feedback from our customers is very positive and our product teams are delivering features 10x faster than we were back in 2020 when I joined. We're demoing new features every week now!

Are we on track getting rid of our legacy monolith? – Yes! At the time of writing, we're close to 90% feature parity with more users hopping over to 360 each week.

Was it way more difficult than initially thought? – Heck yes! The people side of things turned out to be much more complex than the technical side – as always.

It isn’t easy to align 5 remote product teams to work on a new product redesigned and rebuilt from the ground up.

It isn't easy to introduce big changes to our tech stack, product, and engineering culture.

It isn't easy to keep our users happy while introducing a new product with major design changes and new concepts to learn.

While none of these things are easy, they are surely worthwhile and we strive to get better every day.

Conclusion

Looking back, I've learned a lot in the past year about leadership, engineering and building products.

With our engineer teams now working well, I've been able to focus my daily work on helping our customers and technical partners hands-on with their adoption of 360. This has really helped me gain better perspective on where the 360 product stands today from our users' perspective.

I've received a ton of feedback; from customers, from colleagues, and from our leadership team.

I’m glad to say things are looking bright for epilot 360!

Still lots to learn and build.

We're hiring @ epilot!

Surprising Performance Lessons from React Microfrontends in Production

Viljami Kuosmanen — Sun, 22 May 2022 12:35:48 +0000

The epilot engineering team stands at 27 developers 1 year after the launch of our rewritten portal built on mostly* React microfrontends.

*Part of our app is written using other frontend frameworks, most notably the sidebar navigation written in Svelte.

Since the initial launch a year ago, our teams have gained a lot of experience running React microfrontends in production using single-spa.

While we expected to face challenges with our new frontend microservices architecture, after solving a few initial problems we haven't hit any major snags with single-spa in the first year.

To my surprise, most issues that have emerged in our codebase are general React pain points not specific to microfrontend architecture at all.

In an effort to share knowledge, I'll address the most common React performance issue we've seen reemerge in our teams in this post.

The state management problem

Here's a really common hook pattern I've seen emerge at one point in most of our React microfrontend projects:

// useFormState.jsx
import React from 'react'

const FormContext = React.createContext()

export const GlobalFormStateProvider = (props) => {
  const [formState, setFormState] = React.useState({})

  return (
    <FormContext.Provider value={{ formState, setFormState }}>
      {props.children}
    </FormContext.Provider>
  )
}

export const useFormState = () => React.useContext(FormContext)

// App.jsx
import { GlobalFormStateProvider } from './useFormState'
import { Form } from './Form' 

export const App = () => (
  <GlobalFormStateProvider>
    <Form />
  </GlobalFormStateProvider>
}

// Form.jsx
import React from 'react'
import { useFormState } from './useFormState'
import { api } from './api'

export const Form = () => (
  const { formState } = useFormState() 

  const handleSubmit = React.useCallback(
    () => api.post('/v1/submit', formState),
    [formState]
  )

  return (
    <form onSubmit={handleSubmit}>
      <FirstFormGroup />
      <SecondFormGroup />
    </form>
  )
)

const FirstFormGroup = () => (
  const { formState, setFormState } = useFormState()

  return (
    <div className="form-group">
      <input
        value={formState.field1}
        onChange={(e) => 
          setFormState({ ...formState, field1: e.target.value })}
      />
      <input
        value={formState.field2}
        onChange={(e) => 
          setFormState({ ...formState, field2: e.target.value })}
      />
    </div>
  )
)

const SecondFormGroup = () => (
  const { formState, setFormState } = useFormState()

   return (
    <div className="form-group">
      <input
        value={formState.field3}
        onChange={(e) => 
          setFormState({ ...formState, field3: e.target.value })}
      />
    </div>
  )
)

Many readers will immediately recognize antipatterns in the above example, but entertain the naïve perspective:

The useFormState() hook is very useful. No prop drilling. No fancy global state management libraries needed. Just native React.useState() shared in a global Context.

What's not to love here?

Perf issues

As nice as useFormState() seems, we'd quickly face performance issues due to components using it having to render on every setFormState() causing unnecessary, potentially expensive re-renders.

This is because we've subscribed all our Form components to re-render on all changes in FormContext by using React.useContext(FormContext) inside useFormState().

You might think React.memo to the rescue, but reading the React docs:

When the nearest above the component updates, this Hook will trigger a re-render with the latest context value passed to that MyContext provider. Even if an ancestor uses React.memo or shouldComponentUpdate, a re-render will still happen starting at the component itself using useContext.

Further, we're unnecessarily depending on the full formState object in all our form components.

Consider:

// formState is a dependency:
setFormState({ ...formState, field1: e.target.value })}
// formState not a dependency:
setFormState((formState) => ({ ...formState, field1: e.target.value }))

At this time, I would consider Context Providers using React.useState to store complex global app state a general React performance antipattern.

However, if React adds useContextSelector (RFC) I am positive the situation could change. 🤞

Lessons learned

Seeing antipatterns like these emerge in React projects even with fairly experienced frontend developers (think 5+ years of React) has lead me to consider performance as a topic that unfortunately requires pretty significant investment to produce quality output when working with React in general.

As always, there is No Silver Bullet. However, our frontend microservices architecture has enabled us to cheaply experiment with different approaches in different teams who have produced quite a few competing strategies to solve form performance:

Use of global state management libraries e.g. Redux, MobX and XState.
Use of dedicated form libraries e.g. react-hook-form
Use of this implementation of useContextSelector
Avoiding controlled form inputs (Leverage the web platform! 👐)

Additionally, thanks to the flexibility of single-spa we've been able to experiment outside of the React ecosystem with frameworks like Svelte and others which has been extremely promising and rewarding for our engineers.

We're hiring @ epilot!