The One-Line Hook
QuantNest is a drag-and-drop workflow engine where non-programmers wire up trading strategies as directed graphs, and a poll-based executor evaluates triggers, routes through conditional branches, and executes broker/notification actions — all without writing a single line of code.
The hard problem wasn't the workflow execution, or the visual builder, or even the AI integration. It was making all three work together reliably under the constraint that trading actions have financial consequences and exactly-once execution matters.
Constraints & Requirements
I started with a few hard requirements:
No persistent connection to the database. The executor polls on an interval. This keeps the architecture simple — no WebSocket server, no long-lived cursor, no replica-set tailing. If the executor crashes, it just restarts and picks up on the next poll cycle.
Exactly-once semantics for trade actions. If the executor fires a buy order, crashes after the broker confirms but before persisting the result, and restarts — it should not buy again. This ruled out naive at-least-once polling.
Conditional branching must be user-comprehensible. Users need to see why a particular branch was taken. Every execution must be traceable — trigger evaluation, branch decision, node status, final outcome.
Token auth that survives concurrent requests. The frontend SPA makes multiple simultaneous API calls. If two 401s race and both try to refresh the token, only one should succeed, and the other should reuse the result.
Model-agnostic AI. Users bring their own API keys. The platform should route to Gemini, OpenAI, or Anthropic without caring which one is configured.
These constraints shaped every major design decision.
Architecture
The executor is the heart of the system. It runs an infinite loop:
wait 2000ms + backoff
→ fetch active workflows from MongoDB
→ for each, evaluate trigger condition
→ if triggered, lock workflow (Redis SET NX PX 5000)
→ execute graph recursively
→ persist execution trace
→ release lock
The backoff is intentionally linear (+250ms per empty poll, capped at 10s). Exponential felt wrong here — you want gradual backoff during quiet periods but fast recovery when conditions change. A trader's price trigger at 2s resolution needs to catch the breakout, not be sleeping in a 30-second backoff.
Key Design Decisions (and the tradeoffs I wrestled with)
1. Poll-based vs Event-driven
This was the first real decision. Event-driven (change streams, WebSocket, push) gives lower latency. But it adds a lot of complexity — reconnection logic, at-least-once delivery guarantees, ordering concerns.
Why I chose polling: Simplicity wins for an MVP. The poll loop is 63 lines of code. It handles crashes trivially (restart and re-poll). The linear backoff means quiet periods don't waste CPU, but active periods get fast response.
The tradeoff: Minimum trigger latency is 2 seconds. For a 1-minute candle strategy this is irrelevant. For a tick-level scalper it's unacceptable. At scale I'd add a push channel for time-sensitive workflows alongside the poll loop.
What I didn't expect: The empty-poll backoff interacts badly with workflows that trigger rarely. A workflow that fires once a day sees the poll interval drift to 10s after ~30 empty cycles. The fix was resetting the backoff counter after the first successful execution of the day, not on every poll cycle. I caught this when a user's market-open trigger fired 8 seconds late.
2. Token Rotation with Family Tracking
The auth design: 15-minute JWT access tokens + 7-day refresh tokens stored as SHA-256 hashes in MongoDB. Each refresh token has a familyId. When rotating, the old token is revoked and a new one is created in the same family. If a revoked token is presented, the entire family is revoked — this detects stolen token replay.
The frontend has a semaphore-based queue for concurrent refresh:
isRefreshing stops 401 from creating new refresh calls
→ if already refreshing, queue the pending request
→ on success, replay the queue
→ on failure, clear auth and redirect to login
Why I chose this: Because race conditions during token refresh are a real problem in SPAs. Without the queue, multiple failed requests cause cascading refreshes that lock out the user.
The tradeoff: The queue introduces memory pressure if many requests arrive while refresh is in-flight. In practice, this doesn't happen — the refresh window is ~200ms locally, and most SPAs don't fire 100 requests in that window.
What broke in production: The first version didn't have the queue. Two parallel API calls would both see 401, both try to refresh, the first would succeed, the second would fail because the first refresh rotated the token invalidating the second. The user would get logged out from a perfectly valid session. The queue + _retry flag fixed this, but it took two deployment cycles to get right. The lesson: always assume concurrent requests will race, even on a single-threaded browser event loop.
3. Conditional Branching as Source Handles
Nodes with conditions (conditional-trigger, if, recheck) expose two output handles: true and false. Edges connected to the true handle are followed when the condition evaluates to true, and vice versa.
The executor resolves branches at runtime:
function resolveConditionalEdges({ sourceNode, outgoingEdges, evaluatedCondition }) {
return outgoingEdges.filter(edge => {
if (edge.sourceHandle === "true") return evaluatedCondition === true;
if (edge.sourceHandle === "false") return evaluatedCondition === false;
return true; // non-conditional edges pass through
});
}
Why I chose this: It maps directly to the visual representation. Users see two labeled ports on the node. They drag edges from the port that matches their intent. No complex rule editor, no expression language to learn.
The tradeoff: This limits conditions to binary (true/false). Multi-way branching requires chaining if nodes. For trading strategies this is rarely an issue — most branching is "if condition met, buy; else wait." But for general-purpose automation it would be limiting.
What I'd do differently: Store the condition expression in the edge metadata instead of the source handle string. This would allow future extension to switch-case patterns without changing the node model.
4. Redis Locks for Execution Cooldown
Every workflow execution acquires a Redis lock with 5-second TTL:
SET lock:exec:<workflowId> <instanceId> NX PX 5000
If the lock can't be acquired, the execution is skipped. This prevents the same workflow from being evaluated by two poll cycles simultaneously (e.g., if a trigger evaluation takes longer than the poll interval).
Why I chose this over MongoDB atomic updates: Redis locks are cheaper (in-memory vs disk write) and have natural TTL expiry. If the executor crashes mid-execution, the lock self-releases in 5 seconds. MongoDB would require a manual cleanup job or a TTL index on a lock document.
The tradeoff: Distributed locks require all instances to agree on the Redis instance. If Redis goes down, all executions skip (fail-safe, which is correct for trading). Adding a second Redis for high availability would double the infra. For a single-instance deployment this is fine.
What bit me: The lock release used GETDEL with ownership verification — you can only release a lock you own. But a slow GC pause on the executor could cause the lock to expire, then get acquired by a second instance, then the first instance resumes and releases the second instance's lock. The fix: locks should be treated as advisory, not authoritative. The execution checks the lock state at the start but doesn't enforce it during the run. Double-execution is prevented by idempotency keys, not locks.
5. AI Provider Strategy Pattern
The AI builder accepts natural language and produces structured workflow graphs. Three providers implement the same interface:
interface StrategyPlannerProvider {
provider: string;
models: AiModelDescriptor[];
generatePlan(input, prompt): Promise<StrategyBuilderResponse>;
}
A resolver picks the right provider based on the user's model selection:
function resolvePlannerProvider(providers, requestedModel) {
// finds the provider that owns the requested model
// falls back to a default if no match
}
Why I chose this: Users bring their own API keys. Some prefer OpenAI, some Gemini, some Claude. The strategy pattern keeps the routing logic in one place. Adding a new provider is implementing one interface and registering it in the array.
The tradeoff: Each provider returns JSON in slightly different formats. The response parsing (Zod validation) has to be permissive enough to handle variance but strict enough to reject malformed graphs. A single provider with a well-defined output schema would be simpler but doesn't let users choose.
What surprised me: Gemini (gemini-2.5-flash) is consistently 2-3x faster than Claude Opus for the same prompt. The quality difference for simple strategy generation is negligible. The strategy pattern makes this optimization invisible to the user — they pick a model, and the routing happens transparently.
Error Handling by the Numbers
One of the most boring but important parts of the system: every API endpoint has explicit error paths. I counted them while writing this post:
-
121
res.status(4xx/5xx)calls across 6 route files - 11 Mongoose models with 9 indexes
- 16 non-retryable error keywords that prevent infinite retry loops (auth errors, validation errors, missing configuration)
- 3 error boundary layers: Zod input validation, service-level errors, catch-all 500s
The pattern is consistent:
try {
const parsed = SomeSchema.safeParse(req.body);
if (!parsed.success) { res.status(400).json(...); return; }
const result = await someOperation(parsed.data);
res.status(200).json(result);
} catch (error) {
if (isKnownError(error)) { res.status(error.statusCode).json(...); return; }
console.error(error);
res.status(500).json({ message: "Internal server error" });
}
Every route follows this. It's repetitive but it means no unhandled rejection crashes the server, and every error returns a structured response.
Benchmarks (Real Numbers, Not Marketing)
I ran these against the running system with Docker MongoDB + Redis on a Mac ARM laptop:
| Metric | p50 | p95 |
|---|---|---|
| API request (no DB) | 0.7ms | 1.8ms |
| Burst (50 concurrent) | 1.9ms | 17ms |
| Signin (bcrypt + JWT) | 89ms | 104ms |
| Concurrent signins (1000 users) | 8.4s | 14.6s |
| Redis SET | 3.7ms | 5.3ms |
| Redis GET | 3.5ms | 4.1ms |
The signin latency is dominated by bcrypt (cost factor 10). Under 1000 concurrent requests, bcrypt serializes on the CPU and latency climbs to 8 seconds. The bottleneck is intentional security, not the application. Switching to Argon2 or reducing the cost factor would improve throughput at the expense of password cracking resistance.
The API throughput (non-DB routes) is ~1000 req/s before rate limiting kicks in. The limiting factor is the event loop, not the database.
What Would I Do Differently at Scale?
1. CDC Instead of Polling
At 10,000+ active workflows, polling every 2 seconds becomes wasteful. I'd add MongoDB Change Streams for real-time trigger evaluation, falling back to polling for cold-start recovery. The poll loop would shift to a "dead letter" role — catch workflows that change streams missed.
2. Shard Workflows by Trigger Type
Price triggers need sub-second evaluation. Timer triggers are fine with 5-second resolution. Report-generation triggers can wait 30 seconds. Currently all workflows share one poll interval. Sharding by trigger type would let each poller run at its optimal frequency.
3. Replace bcrypt with Argon2id and Add a Session Cache
bcrypt at cost 10 gives 90ms per comparison. Under 1000 concurrent signins, this becomes 8 seconds of wall-clock time. Argon2id is hardware-resistant and faster on modern CPUs. Combined with a Redis session cache (O(1) lookup after first authentication), the majority of requests would bypass the hash entirely.
Currently there's no session cache — every authenticated request verifies the JWT signature (fast, ~1µs), but token refresh hits MongoDB with a findOne({tokenHash}) which is O(log n). Adding redisSet("session:<tokenHash>", userData, 3600) during token creation would make refreshes O(1) and reduce DB load.
4. Formalize the Circuit Breaker
Right now, consecutive failure thresholds (3 failures → warn, 5 failures → pause) are hardcoded in the execution service. I'd extract this into a formal circuit breaker with half-open state, configurable thresholds per action type, and exponential reset timeouts. Trading actions should have tighter thresholds than notification actions.
5. Webhook Gateway
The system currently has no webhook endpoint. For real-time broker callbacks (Zerodha order fills, Groww position updates), a webhook gateway with Redis-backed idempotency keys would replace the current polling-based state checking. The idempotency middleware already exists — it just needs a route to attach to.
What I Learned
Polling is underrated. Everyone reaches for WebSockets and event-driven architecture. A well-tuned poll loop with linear backoff handles 90% of use cases with 10% of the complexity.
Rate limiters are the first thing that breaks in benchmarks. I spent more time fighting
express-rate-limitduring load testing than writing the actual benchmark scripts. Plan your rate limit strategy before you need it.Bcrypt doesn't scale. 1000 concurrent signins on a single core brings the server to its knees. If you expect high auth throughput, cache sessions aggressively or use a faster hash.
Family-based token rotation catches real attacks. In testing, a revoked token was presented within minutes of rotation. The family revocation killed it. Without family tracking, a stolen refresh token would stay valid until the user signs out.
The 61µs AI pipeline overhead is a rounding error. The actual latency is 100% determined by the AI provider. Don't optimize your prompt building pipeline — optimize your model selection.
Check it out: QuantNest
Found a bug or have an idea? Open an issue.
Top comments (0)