Two weeks ago, I had a data pipeline platform with separate Import and Export systems.
Today, I have a workflow engine. And it's 37x faster.
Net result: ~106,000 lines deleted. 2.5 minute operations now take 4 seconds.
Here's what happened.
The Market Told Me Something
I kept seeing the same thing on X: people sharing screenshots of their n8n workflows. Visual canvases with nodes and connections. "Here's how I automated my entire marketing pipeline." "Here's my AI agent workflow."
And I realized: that's what people want. Not "data pipelines"—workflows. Visual, connectable, shareable.
My platform could import data from S3, transform it, export to BigQuery. Powerful stuff. But nobody's posting screenshots of an import configuration form.
I wanted Flywheel to be the thing people screenshot and share. That meant pivoting from "data pipeline tool" to "workflow automation platform."
The problem? My entire architecture was built around "Import" and "Export" as separate concepts. Not nodes on a canvas. Not a visual workflow you'd want to show off.
The Architecture I Had
Two complete systems, doing almost the same thing:
| Import System | Export System |
|---|---|
imports/handler.go |
exports/handler.go |
imports/service.go |
exports/service.go |
imports/dao.go |
exports/dao.go |
imports/model.go |
exports/model.go |
imports/subscriptions.go |
exports/subscriptions.go |
Plus, for each of 8 data providers (S3, BigQuery, Postgres, DynamoDB, Firestore, GCS, Pub/Sub, Domo):
| Import Code | Export Code |
|---|---|
{provider}/chunk_fetcher.go |
{provider}/export_writer.go |
{provider}/subscriptions.go |
{provider}/export_subscriptions.go |
That's 16 files per provider just for the import/export split.
And the kicker? They couldn't talk to each other. Want to move data from S3 directly to BigQuery? Can't. You had to import to my system first, then export out. Two operations. Two sets of overhead. Two points of failure.
The Pivot
What if Import and Export were just... different nodes in the same workflow?
Old mental model:
Import (External → Flywheel)
Export (Flywheel → External)
New mental model:
Source (read from anywhere) → Sink (write to anywhere)
A "Source" doesn't care if it's S3 or my internal storage. A "Sink" doesn't care if it's BigQuery or a file download.
Import = Source(S3) → Sink(Flywheel)
Export = Source(Flywheel) → Sink(BigQuery)
Direct ETL = Source(S3) → Sink(BigQuery) ← couldn't do this before
One unified workflow system. Not two separate systems pretending to be related.
Two Weeks, 1,310 Files
I had no users yet. No backward compatibility. No "gradual migration."
So I did the only sane thing: burn it down and rebuild.
Week 1: The Deletion
- Deleted entire
imports/package (~5,500 lines) - Deleted entire
exports/package (~5,900 lines) - Deleted entire
egress/package (~1,000 lines) - Deleted per-provider import/export code across 8 providers
- Deleted corresponding frontend:
ImportWizard,ExportWizard,ImportList,ExportList
By end of week 1, the app didn't compile. That's fine. Dead code can't hurt you.
Week 2: The Rebuild
- Created unified
pipelines/package with orchestrator - Created
conditions/package for filtering - Implemented Source/Sink capabilities for all 8 providers
- Built React Flow canvas editor for visual workflow building
- Wrote 700+ lines of architecture documentation
The new system has more functionality:
- Direct ETL (Source → Sink, no intermediate storage)
- Multi-source joins
- Conditional routing
- Enrichment nodes (database lookups mid-workflow)
Things that were architecturally impossible before fell out naturally from the unified model.
The 37x Speedup (This One Was Intentional)
The old architecture wasn't just duplicated—it was chaos.
Every provider had its own Pub/Sub consumer. Every provider had its own publisher. Messages were flying everywhere. Import triggers Export triggers chunk processing triggers completion handlers triggers cleanup.
I couldn't test it. I couldn't reason about it. I'd add a new provider and spend half my time debugging message flows that worked fine until they didn't.
The stress test told the story:
Old system: 2.5 minutes for 10K records
New system: 4 seconds for 10K records
The fix was simple: one orchestrator.
Instead of providers talking to each other through a spaghetti of Pub/Sub messages, the orchestrator controls everything:
Old: Provider A publishes → Queue → Provider B subscribes → publishes → Queue → ...
New: Orchestrator calls Provider A, gets data, calls Provider B, done.
The orchestrator knows the full workflow graph. It executes nodes in topological order. It handles errors in one place. It tracks progress in one place.
No more "which subscriber handles this message?" No more "why did this workflow stall?" No more "I changed one handler and broke three unrelated providers."
The 37x speedup came from:
- No message queue overhead - Direct function calls instead of Pub/Sub round-trips
- No intermediate storage - Stream directly from source to sink
- No duplicate work - One chunking strategy, not two
I didn't optimize. I centralized control. Performance followed.
How AI Made This Possible
1,310 files in 2 weeks is not human-speed refactoring.
Pattern Enforcement
Every provider needed the same structure:
// provider.go - capability registration
func (p *S3Provider) Capabilities() []string {
return []string{"source", "sink"}
}
// source.go - Reader implementation
func (p *S3Provider) CreateReader(ctx, config) (Reader, error)
// sink.go - Writer implementation
func (p *S3Provider) CreateWriter(ctx, config) (Writer, error)
Claude ensured all 8 providers followed this exactly. No "slightly different" implementations that would bite me later.
Blast Radius Mapping
Before deleting anything, Claude identified every file that referenced imports or exports:
- 47 files in the core packages
- 64 provider-specific files
- 89 test files
- 200+ frontend components
Nothing got orphaned. Nothing got forgotten.
Test Migration
Every deleted test had a corresponding new test. The test suite never went red during the refactor. When imports/service_test.go got deleted, equivalent coverage existed in pipelines/service_test.go.
What I Shipped
| Node Type | What It Does |
|---|---|
| Source | Read from S3, Postgres, BigQuery, Firestore, GCS, DynamoDB, Pub/Sub |
| Transform | Modify records with JSONata expressions |
| Enrichment | Look up additional data from databases |
| Join | Combine records from multiple sources |
| Sink | Write to any of the above + file downloads |
Users drag nodes onto a canvas, connect them, run the workflow. No "configure import then configure export" friction.
The Lesson
Deletion enables features.
The old architecture couldn't do direct ETL. It couldn't do joins. It couldn't do conditional routing. Not because I didn't want those features—because two separate systems couldn't coordinate them.
Unifying Import and Export into Source and Sink made all of that trivial.
Distributed systems are a last resort.
Every provider having its own consumer/publisher felt "scalable." In practice, it was untestable spaghetti. One orchestrator with direct control is simpler, faster, and actually works.
Don't distribute until you have to.
Early stage is the cheapest time to pivot.
This refactor would be impossible with real users. It was a complete model change. Database schema changes. API changes. Frontend changes.
If you're pre-launch and your architecture doesn't match your market positioning, now is when you fix it. The cost only goes up.
Try It Yourself Flywheel
If you're sitting on an architecture that doesn't match what users actually want:
Ask if your abstractions match user mental models. "Import/Export" was my mental model. "Workflow" was theirs.
Look for duplicate systems. If you built the same thing twice with different names, you can probably unify them.
Question your distributed architecture. If you can't easily test or reason about message flows, you probably over-distributed.
Early stage = cheap pivots. No users means no migration. Just delete and rebuild.
Sometimes the answer is mass deletion.
Have you ever pivoted your app's architecture to match market language? What triggered the change?
Top comments (0)