DEV Community: Thibaut Flat-Patrick

nFolyo Post-MVP - Architecture and Tech Stack Review

Thibaut Flat-Patrick — Fri, 20 Jun 2025 06:52:15 +0000

Originally published on nfolyo blog

In nFolyo MVP Postmortem – What We Learned Building a Portfolio Tracker, we briefly introduced our tech stack and architecture. This post delves deeper into the technical decisions behind nFolyo, outlining our current architecture and its key limitations. We'll then present our approach for the upcoming v0.5.0 release, where we're restructuring our architecture with scalability and responsiveness in mind.

Architecture Overview

Before discussing any restructuring, it's crucial to understand how our services and web apps currently interact and their limitations.

The nFolyo Client App serves as our user-facing frontend. Built with NextJS/TypeScript, it uses a combination of RadixUI and custom-made components. This app performs minimal data calculations, delegating most data processing to our Python services.

nFolyo Core, built with Flask in Python, is responsible for updating user accounts. It provides importers for various brokers (currently only Interactive Brokers and Freetrade) to import activities such as trades, dividends, fees, cash transactions, and corporate actions. During import, all broker activity undergoes standardization to strip private information, eliminate differences between brokers, and ensure consistent data handling. We'll explore this process in a future post.

To compute metrics like market value, PnL, and TWRR, nFolyo Core fetches pricing and foreign exchange data from the nFolyo Finance service. nFolyo Finance serves as our price API that fetches, aggregates, and standardizes pricing data from external providers (such as Yahoo Finance), ensuring reliable data and fast retrieval.

You'll notice the nFolyo Admin web app in the diagram. We've recently moved all admin functionality from the nFolyo Client App to a separate VPN-protected web app. nFolyo Admin, built with Next.js and shadcn/ui, lets us manage beta test users, handle user accounts, and monitor performance through benchmarking data.

One detail omitted from our diagram for simplicity: both nFolyo Core and Finance services write to the nFolyo Benchmarker Database, which feeds data to the benchmarking tools in our admin web app. The benchmarker is a separate Python module we wrote to help us better understand our performance bottlenecks.

What Works, and What Doesn't

Let's be honest: nFolyo is slow. There are two main reasons: we're running on low-tier Azure services, and our architecture doesn't allow for proper task parallelization. Before scaling up our infrastructure, we need to address the fundamental architectural bottlenecks. Otherwise, we'll hit the same issues with larger user volumes and more features built on a shaky foundation, making future refactoring difficult.

Dissecting the `/update-account` Endpoint

Let's examine the /update-account endpoint that handles account updates. It's our worst-case scenario, implemented as a monolithic function for quick MVP development.

It performs these steps:

Collects all holding symbols and unique currency pairs from the user's holdings
Fetches latest price/FX data from nFolyo Finance
Updates holdings market value, PnL, and other metrics
Aggregates holding values into themes and account metrics (market value, PnL, etc)
For each holding:
- Fetches historical prices and FX data
- Calculates TWRR, market value over time, cash in and outflows
Aggregates these to themes and the overall account historical charts

As shown in the sequence diagram, there are many steps in a single call. During this time, nFolyo Core is blocked from serving other requests while the user's frontend displays the "updating your account" component.

While we've improved the component in v0.4.0 to provide more information about the current stage of the account update, it remains a long wait. We're not displaying information to the user as it becomes available (e.g., holding, theme account market value, PnL).

Benchmarking the Bottleneck

In the benchmarker screenshot below, you'll see /update-account taking 20 minutes to complete! While this example is particularly bad due to concurrent requests at peak CCU, it illustrates the worst-case scenario we need to address.

Notice that getHistoricalData takes about 1 minute 20 seconds — unacceptable, but not our biggest problem!

The performance.calculateHoldingPerformanceTWRR (111) is even worse, consuming 80% of the total time! This severely impacts user experience — nobody wants to wait minutes to view their account data. While some delay during initial import might be acceptable, it shouldn't be the norm.

Several factors contribute to this poor performance:

We process holdings sequentially, in this case, 111 holdings one after another.
The calculateHoldingPerformanceTWRR() function itself is inefficient. Pandas struggles with large datasets, especially when combining multiple tables (historical prices, historical FX data, corporate actions) for TWRR calculations.
Theme and account TWRR calculations must wait for all holdings to complete before they can start.

Solutions: Task Splitting, Parallelization, and New Price Provider

What are we doing to address these issues?

First, we'll tackle the 80% bottleneck by parallelizing the TWRR calculations. Instead of using Python threads, we're adopting a more scalable message queue pattern. We'll split or completely convert nFolyo Core into a Celery worker, queueing tasks to Redis or RabbitMQ.

This approach will allow us to queue all TWRR tasks and scale up workers for faster processing when needed. Even with just two Celery workers running in parallel, we could nearly halve processing time.

Secondly, we're switching price providers to reduce the 1-minute-20-second historical price retrieval to 10–20 seconds. Combined with scheduled background price and account updates, this should reduce friction between user actions and data visibility.

We'll optimize TWRR calculation performance by exploring Polars, a performance-focused alternative to Pandas, while improving our TWRR algorithm.

Finally, we'll split the /update-account endpoint into smaller tasks (separating quick operations like price updates from slower TWRR calculations). This lets users view current market values while TWRR charts process in the background.

Evolving the Architecture

Let's examine the suggested architecture diagram below that accommodates these solutions. This diagram omits some services and apps (like admin app, common and benchmarker DBs) to highlight the fundamental architectural changes.

The key improvement is offloading heavy tasks to Python Celery workers. These can scale up and run in the background while keeping the UI responsive and showing data as they become available.

This architecture coupled with Server-Sent Events should allow us to provide more frequent task status updates, eliminating our current expensive polling approach, the result of fast development for MVP and Azure function limitations (45sec timeout).

This is a suggested architecture, meaning that things are subject to change as we progress our development. At the moment we believe we'll split out a big chunk of the nFolyo Core business logic to nFolyo Core Celery Workers, starting with the heaviest tasks. nFolyo Core will remain to receive requests and orchestrate the heavier tasks.

The scheduled jobs will probably end up either queuing tasks directly to the message queue or do it via nFolyo Core endpoints. We're also exploring whether the nFolyo Client App itself should queue tasks directly. Many possibilities are open and we'll be revising our suggested architecture once they become more concrete.

Conclusion

In this post, we examined nFolyo's current architecture and tech stack, identified key bottlenecks, and highlighted the scalability challenges we're facing. We used /update-account as a case study to demonstrate where and why performance breaks down.

To address these issues, we're revising our architecture and refactoring critical services:

Introducing a message queue to enable parallel task execution.
Offloading heavy computations to Celery workers, with nFolyo Core orchestrating and dispatching tasks.
Breaking up monolithic functions into smaller, async-friendly tasks, with Server-Sent Events providing updates for a more responsive UI.
Scheduling background updates to execute heavy calculations ahead of user interaction.
Switching to a faster, more efficient price provider.

As we roll out v0.5.0, we'll share more learnings and code details. For now, we're focused on making nFolyo faster, smoother, and ready to scale.

nFolyo MVP Postmortem - What We Learned Building a Portfolio Tracker

Thibaut Flat-Patrick — Fri, 06 Jun 2025 20:39:09 +0000

Originally published at nfolyo blog

We do not learn from experience… we learn from reflecting on experience. — John Dewey

Introduction

nFolyo is a portfolio tracking and analysis tool built for long-term investors like us — with global portfolios invested across equities and ETFs, not just limited to US and EU markets.

It started as a collection of Python scripts and spreadsheets we used to manage our own portfolios. Over time, it evolved into a full-featured tool and became our learning ground for modern web development and microservice architectures.

We released our MVP in November 2024 after eight months of intense development. After iterating on early user feedback, we released version 0.4.0 in March 2025 with several improvements. Now, we're at a turning point that requires us to rethink our approach and make some difficult decisions.

To do that well, we need to pause and reflect.

This postmortem marks the start of the next chapter — a chance to look back on over a year of development, two releases, what worked, what didn't, and where we go from here.

History of Development

Our Tech Stack

nFolyo is made up of a frontend app (nfolyo-client) and two backend services:

nfolyo-core: handles account and portfolio updates.
nfolyo-finance: responsible for fetching pricing and historical data.

We'll dig into our architecture in a future post, but here's an overview of our current stack:

Next.js: For the frontend, we chose Next.js as it's a mature and powerful framework that supports full-stack development and server-side rendering.
Python and Flask: For the nfolyo-core and nfolyo-finance background services, we went with Python and Flask. Flask makes it easy to build APIs quickly, and Python is particularly strong for data processing.
MongoDB: We selected MongoDB as our database due to its scalability and our prior experience working with it.
Azure: We're deploying on Azure, using Docker containers and GitHub Actions for continuous deployment.

The Road to MVP

We set a minimum feature set before nFolyo was ready to be shared with other investors. Our original roadmap targeted a September 2024 release, but as you can see in our revised roadmap (pointing to October/November 2024), we underestimated the work involved!

MVP Launch, User Feedback & Iteration

We released the MVP in November 2024 to a closed group of beta users. It wasn't polished, and while we extensively tested our broker activity importers (Interactive Brokers and Freetrade), we couldn't guarantee it would parse every user's data.

We were fortunate to have extremely patient beta testers who not only helped us fix major bugs but also provided crucial feedback to improve our user experience. Their support was invaluable.

Over the next few months, we focused on the most critical bugs and usability issues:

Fixed price inaccuracies and broker importer errors.
Improved TWRR performance calculations.
Added clearer feedback during statement import and account updates.

v0.4.0 launched in March 2025, incorporating those changes. Check out the release notes nFolyo v0.4.0 – Improved Import and Account Status Updates for more details.

Where Are We Now? Pausing and Reflecting

Despite all the progress and great user feedback, we're now at a crossroads. Some of our early tech and architectural decisions, made to speed up MVP development, are starting to catch up with us. It's becoming clear that our current infrastructure won't scale up easily.

Without a more maintainable foundation, adding new features will only get harder and more expensive over time. We have a long list of ideas that could make nFolyo truly powerful and unique; however, without rethinking and refactoring our core architecture, those ideas risk remaining just ideas.

So what do we do?

We decided to pause the closed beta while we focus on the next major release. (You can still sign up to register your interest and we'll notify you as soon as the next version is out). This gives us the freedom to break things without the pressure of maintaining backward compatibility and continuous live operations.

Reflections

The Good

It's been a great learning experience — diving deep into Next.js, Python, Flask, and Azure with real hands-on challenges.

We've implemented many features in a short time, all while holding down full-time jobs.

We've had highly engaged users testing the app, providing valuable feedback, helping us uncover bugs we couldn't have found alone. Their time and support have been invaluable.

The Bad

1. Burnout

Balancing full-time jobs with late nights and weekends on a side project isn't sustainable. Passion keeps things going for a while, but long-term time management and maintaining a healthy work-life(-side-project) balance is essential.

2. Building in Isolation

We didn't engage publicly early enough. No community, no dev blog, no social presence. All these factors slowed down feedback and iteration loops significantly. Building nFolyo in public from day one could have helped us course-correct faster and save development time.

3. Poor UI Component Library Choices, Failure to Explore Alternatives

We started with Radix UI, but hit limitations when customizing components. We ended up creating many of our own, which introduced styling inconsistencies and duplicated effort (e.g., building an editable/sortable table).

In hindsight, switching to shadcn/ui early would've saved loads of time and given us more consistent styling out of the box. This decision alone added months of unnecessary frontend work.

4. No Broker API Auto-Sync

We assumed most users would be fine uploading broker statements manually. But feedback shows otherwise. Around 80% of users (ourselves included) want automatic sync via broker APIs to make their lives easier.

We avoided this to keep development simple, but going forward we need to support it to stay competitive. The challenge will be balancing automation with privacy. We're hesitant to rely on third-party aggregators like Yodlee or Plaid, but that's something we need to explore more.

5. Underinvesting in a Pricing API

We chose Yahoo Finance's public API because it was cheap and provided global coverage for equities and ETFs — perfect for proving out the MVP. But it's slow. Really slow. Historical pricing queries in particular introduce major latency, which affects calculations like TWRR performance charts. We've had to build a whole layer of caching and data cleanup just to make it performant enough, costing us weeks of effort. For serious use, we'll need a more robust solution.

6. No Async/Message Queue Architecture

To keep things simple early on, we built a monolithic background service that updates a user's account in one big API call. That was fine for launch, but on accounts with long trade histories and many holdings, it now takes up to 10 minutes. The processing isn't parallelized and quickly becomes a bottleneck, affecting the user experience.

What we really need is an asynchronous, distributed system using a message queue (e.g., RabbitMQ or Redis) and a task processor like Celery. That would let us split jobs into smaller tasks (units of work), process them in parallel, and give real-time progress updates using something like Server-Sent Events (SSE).

Not doing this is now our biggest architectural limitation. It made sense for the MVP, but it's blocking future progress.

7. Azure Deployment Pain Points

This was our first time deploying such a complex app on Azure. While getting things up and running wasn't too painful, we ran into real performance issues:

Our Flask services run as App Services; on low-tier plans, performance is poor, while high-tier plans are expensive. Additionally, the lack of async mechanisms means we're paying for idle services much of the time.

The Next.js frontend is deployed as a Static Web App using Azure Functions for APIs. Cold starts can cause 15 - 30 second delays, and Azure Functions 45-second timeout complicates long-polling and SSE.

We're rethinking our deployment strategy to better suit the performance demands of a scalable application.

Learnings

Here are the key takeaways from our journey so far.

1. Manage your Time Wisely

It's better to make consistent, small progress than to go all-in, burn out, and need a long break — or worse, lose all motivation and abandon the project. Sustainability > speed.

2. Build in Public Early

Share progress openly and engage users from the start. It might feel like you're moving slower, but early feedback helps you avoid building the wrong things — saving time in the long run.

3. Use Mature, Customizable UI Component Libraries

Don't reinvent the wheel. Component libraries like shadcn/ui are tried and tested and can save you weeks of work. Building our own sortable, editable table from scratch wasn't very wise and cost us precious development time, we should have just used TanStack Table. Don't make the same mistake.

4. Lay Solid Architectural Foundations (When it Matters)

If you're just validating an idea, quick-and-dirty is fine. But if, like us, you know this is something you want to build for the long term, invest early in scalable architecture. It'll save you months of technical debt later.

5. Choose Your APIs Carefully

Don't default to the cheapest solution. While it's fine to start lean, poor APIs can introduce major delays and reliability issues. At the very least, build abstraction layers (e.g., interfaces or base classes) so you can easily swap out the API later without a full rewrite.

Next Steps

To wrap things up, here's a quick overview of what's next for nFolyo. We'll go into more detail — including our roadmap to v0.5.0 — in future posts.

1. We're Intentionally Slowing Down Development

This is a marathon, not a sprint. To build something great — for ourselves and our users — we need to manage our time better and avoid burnout.

2. We're Building in Public and Sharing More

This blog is a big part of that. Expect more updates, learnings, and technical deep-dives as we go.

3. We're Refactoring Our Backend for Scalability

We're migrating to an architecture based on Celery with RabbitMQ (or Redis) for better performance and scalability. We plan to document this process, as most tutorials out there only scratch the surface and rarely cover real-world use cases.

4. We're Investing in Better Financial APIs and Exploring Broker API Integrations

We're currently evaluating APIs like Financial Modeling Prep and EODHD, and looking into direct broker integrations for trade syncing. Our goal is to offer automation and convenience — without compromising user privacy or relying on third-party aggregators like Yodlee and Plaid. That said, we're keeping our options open if they offer the best balance of utility and privacy.

In Conclusion

Thanks for reading. We hope you found something useful in our journey — whether you're building your own tool, considering a side project, or just curious about the inner workings of portfolio tracking apps.

Let us know what you think, we'd love to hear from you. You can find us on X and Reddit.