Striking a balance between delivering new features and investing in technical improvements is a challenge every SaaS company faces. In product-driven organizations, engineering leaders must champion initiatives that may not provide immediate customer value but are essential for long-term sustainability. At Hotjar, we embarked on a complex journey to modernize the architecture of our main front-end application. It was a three-year endeavor filled with twists and turns, but we emerged victorious—and with a wealth of knowledge.
Our North Star
Rewind to the latter half of 2020. The COVID-19 pandemic was in full swing, but as a remote-first company, Hotjar was uniquely positioned to weather the storm. Our focus remained fixed on the future and the challenges ahead.
Our main front-end project, the Insights Webapp, was a monolithic application pushing 200k+ lines of code. With a growing team of 20 engineers and an active hiring pipeline, the cracks were starting to show. Incidental coupling led to conflicts, Continuous Integration times ballooned, and Continuous Delivery became a bottleneck due to a shared deployment pipeline with the backend system. We needed a scalable, future-proof solution.
Inspired by our backend colleagues, we’ve set out to define our "North Star"—a set of guiding principles for our front-end platform. A small group of senior engineers and engineering leaders convened weekly to dissect the challenges and brainstorm solutions. After a few months of deliberation, our Frontend North Star emerged:
- Ownership
- Autonomy and Accountability
- Easy Deployment
- Performance and Metrics
While the "how" remained unclear, the "what" was crystal clear. These four pillars would underpin our efforts for the next three years.
Takeaway #1: In complex projects, meticulous upfront planning isn't always necessary. Aligning on core principles is paramount—the details can be ironed out along the way.
Transparency is a core value at Hotjar. We practiced it by sharing our thoughts and progress on an internal blogging platform. Our posts sparked discussions with backend colleagues, leading to the creation of a dedicated Slack channel for exchanging ideas and experiences. This open forum proved invaluable in refining our problem definition and exploring solutions.
Takeaway #2: Transparency works miracles. The sooner you share your idea, the more help you can get from people around you.
Embracing the Modular Monolith
With our guiding principles established, we turned our attention to identifying technical solutions that aligned with our vision. Simplicity and proven architectures were important – we wanted to avoid the pitfalls of unproven experimental technology. In early 2021, the modular monolith, built upon a monorepo foundation, emerged as a leading contender. We believed this approach, with its clearly defined NPM package boundaries, would deliver on our pillars of Ownership and Autonomy. The goal wasn't to completely refactor the codebase but to reorganize and simplify dependencies. It’s what the diagrams below present - they have a similar number of communication flows and the right one has even more components, but it's also evident that the new architecture’s flows (on the right) are much easier to understand.
Furthermore, our decision to decouple the front-end and back-end code into separate repositories and deployment pipelines would support our Easy Deployment objective. After evaluating various monorepo tools, we settled on NPM v7 workspaces and Lerna, a lightweight solution that seamlessly integrated with our existing infrastructure.
Upgrading our package manager was a necessary first step, a task that demanded considerable time and effort. Fortunately, a compelling motivator emerged: the opportunity to contribute to a high-stakes rebranding project. While we were defining our North Star, the product team had initiated a rebranding and application redesign, including the development of Hotjar's first full-fledged design system. This design system was architected as a separate repository with an NPM package published to our internal registry. While it facilitated rapid iterations, integrating the library into the web app proved cumbersome. Managing updates and bug fixes across two repositories was inefficient. The design system team needed a more streamlined workflow, and the monorepo approach promised to deliver. This situation allowed us to justify the investment in the monorepo infrastructure, highlighting its immediate benefits for a priority business objective, in addition to its long-term value.
Takeaway #3: Capitalize on opportunities to align your technical initiatives with business goals, even if they seem like "local optimizations." Every bit of support helps propel your plan forward.
Crowdsourcing
Alongside the infrastructure, we furnished teams with a basic template for new packages and architecture guidelines. The integration of the design system served as our first practical example. We contemplated the ideal structure for new packages and the patterns we should promote and discourage. Admittedly, we lacked concrete answers to many of our questions, given our limited experience with the new packaged architecture. We embraced a "learning by doing" approach, providing minimal guidance to the squads. We believed our talented engineers, immersed in product development, were best positioned to identify pain points and propose solutions.
This strategy sparked the emergence of new modules. Engineers from various squads enthusiastically experimented with the new tooling, creating new packages and extracting code from the existing codebase. Their hands-on experience yielded invaluable feedback and illuminated real-life challenges. This iterative process allowed us to refine our tooling and documentation, addressing the most pressing issues and empowering teams to build better products. This was a testament to Hotjar's Core Values in action: engineers challenged themselves, worked respectfully, and moved fast, releasing work in small increments.
Takeaway #4: Avoid overthinking and strive for minimal viable solutions. Observe, listen, and iterate based on real-world feedback.
Measuring Progress
Our approach wasn't purely reactive. We continuously evaluated the evolving technology landscape and its alignment with our North Star. As 2022 dawned, we bid farewell to Lerna and fully embraced NPM Workspaces, which had matured enough to meet our needs. We also introduced NX, a powerful task manager designed for monorepo architectures. Theoretically, NX promised faster Continuous Integration through intelligent task isolation and dependency resolution. However, NX also brought to light a lurking issue: circular dependencies.
We had anticipated this challenge during the transition period, as we extracted code from the legacy system into packages while retaining the old codebase as the application's entry point. Maintaining package purity during migration wasn't pragmatic, as new code often needed to reference legacy modules. This created the inevitable circular dependency. While starting the migration from core components ("leaves" of the dependency graph) would have been ideal, these lacked clear ownership, making it challenging to engage product engineers. We opted to accept the suboptimal architecture temporarily.
This marked our first foray into the fourth pillar of our North Star: Performance and Metrics. We introduced a metric to track the number of circular dependencies between packages, making it visible to the entire team. Initially, NX reported a staggering 1800 circular dependency chains! This underscored the importance of early monitoring. The presence of these dependencies negated the benefits of NX's sophisticated task isolation capabilities. While not entirely unexpected, this realization highlighted a communication gap. We had emphasized the improved developer experience that modularization would bring, but that reality was still distant. As we were progressing with the modularisation, oftentimes we had to make tactical compromises, and our metric wasn’t always continuously decreasing. That wasn’t the point - what mattered was that we knew how far we’re from the goal.
Takeaway #5: Define your metrics early and begin data collection immediately. Early insights illuminate potential problems and enable better decision-making.
Exploring Microfrontends
Concurrently, we initiated another stream of work focused on the Easy Deployment pillar. Our reasoning was that while our packages were currently tightly coupled, they wouldn't remain so indefinitely. This led us to explore the possibility of deploying them independently. NX offered built-in support for Webpack Module Federation, presenting an ideal opportunity to delve into this technology, even though we hadn't yet encountered the specific problem it addressed. With a peak development team of 30 engineers, and having decoupled our deployment pipeline from the backend, there wasn't a pressing need for isolated deployments. However, the concept of microfrontends had intrigued us from the outset of our North Star discussions.
While we had intentionally defined our North Star in terms of desired outcomes rather than specific solutions, we couldn't help but contemplate implementation details. This wasn't necessarily a mistake—the work was engaging and yielded valuable learning experiences. However, justifying the investment in this stream proved challenging. For several quarters, it remained a speculative endeavor. Convincing leadership to allocate resources based solely on the promise of future need was difficult. Engineers working on this stream grappled with motivation and questioned the significance of their efforts.
Fortunately, the product team presented another timely opportunity. Hotjar acquired PingPong, a small startup, and decided to integrate its offering into our application. After considering various microfrontend architectures, the newly formed integration team eagerly embraced one experimental microfrontend infrastructure to maintain autonomy while delivering a seamless customer experience. This provided the real-world testing ground we needed, allowing us to connect our prototypes to tangible product needs. The work ultimately proved valuable and successful, but the path to that success could have been smoother.
Takeaway #6: In a product-led organization, evaluate ideas through the lens of customer needs. Many excellent solutions may not be relevant to your current context. Recognize that timing is crucial to avoid frustration and maximize impact.
Evolving Iterations
Throughout the remainder of the year, we steadily progressed with our modularization efforts across various domain areas. We launched a more structured initiative to extract core modules from the legacy system and create well-defined packages for common code. Ironically, this increased the number of circular dependencies, seemingly pushing us further from our goal of autonomous, performant development experiences for product squads. Questions arose about the value and purpose of this work. With our first product successfully deployed as a microfrontend, uncertainty lingered about whether this was the intended approach for all products. The North Star helped us build and retain alignment on multiple levels which led to more precise visions and projects with specific goals that were key to our success.
These concerns signaled the need to revisit and refine our vision and strategy for modularization. While we had initially articulated our goals through the North Star, those principles now felt abstract and disconnected from our actions. We committed to addressing this disconnect and reaffirming the value of our investment. We retroactively analyzed team velocity across different areas of the codebase, finding strong evidence that well-defined modules accelerated value delivery. Armed with this data, we crafted a revised strategy document that clearly outlined the scope and goals, emphasizing our expectation for a fully packaged codebase. We explicitly excluded independent deployments via Module Federation, pending further evaluation of the trade-offs. This clarification restored focus and alignment within the team.
Takeaway #7: Maintaining alignment is an ongoing process, not a one-time achievement. Regularly reiterating goals and non-goals is essential throughout a project's lifecycle.
Scaling for the Future
After three years of hard work by the entire Hotjar frontend team, we reached a significant milestone in early 2024. Circular dependencies were eliminated, and all core modules and domain-related code were segregated into well-defined NPM packages. This effort unlocked the full potential of tooling like NX and paved the way for further improvements, such as TypeScript project references.
The modularization effort finished just in time for the integration of Hotjar products with modules of our parent company, Contentsquare. This wasn’t part of our initial 2020 plans when defining our North Star, but it aligned perfectly with the scalability goals we envisioned back then. It also emphasised how a decoupled, modular architecture can allow teams to adapt seamlessly to new requirements, including swapping or reusing specific parts of the system with minimal risk.
At the end of the day, this journey was more than a technical success - it was a demonstration of how thoughtful, forward-looking engineering can empower organizations to meet unforeseen challenges and deliver exceptional results. The real-world benefits of our investment demonstrated more than we could ever expect.
Takeaway #8: Plan for replaceability rather than specific, near future, scenarios. Flexible architectures open doors to opportunities you can’t yet foresee - the unknown unknowns.
Takeaways
Our three-year modularization journey has been a testament to the power of perseverance, adaptability, and a shared vision. We encountered unexpected hurdles, course-corrected along the way, and ultimately transformed our front-end architecture into a more scalable and maintainable system. Here's a recap of the key lessons we learned:
- Long-term visions don't require detailed upfront plans: Aligning on core principles is key.
- Transparency fosters collaboration and accelerates problem-solving: Share your ideas early and often.
- Capitalize on opportunities to align technical initiatives with business goals: Even small wins contribute to the larger vision.
- Avoid overthinking and strive for minimal viable solutions: Iterate based on real-world feedback.
- Define your metrics early and begin data collection immediately: Early insights enable better decision-making.
- In a product-led organization, evaluate ideas through the lens of customer needs: Timing is crucial.
- Maintaining alignment is an ongoing process: Regularly reiterate goals and non-goals.
- Plan for replaceability: You can never know what the future brings.
While our journey has reached a significant milestone, it's not over. We continue to refine our modularized architecture, explore new technologies, and optimize for developer experience and performance.
We're eager to hear from you! What challenges have you faced in your modularization efforts? What lessons have you learned? Share your experiences and insights in the comments below.
Top comments (0)