What is the problem?
After discovering renowned debugging tools such as Sentry and Open Replay, I made the decision to integrate them into one of our end-user applications with high traffic. However, the challenge arose in terms of performance and bundle size. With Sentry contributing more to the increase and Open Replay less, our bundle size surged by up to 30%. That's not ideal, is it?
To address this, we initiated a second production pipeline named Main Debug, directing only 5% of users to it. This proved sufficient for our debugging needs, and thus the journey begins from this point.
Solutions
We identified three solutions to tackle this challenge:
1. GitLab Feature Flags
2. Host Provider Weighted Load Balance
3. Load Balancing with Nginx
GitLab Feature Flags
As A/B testing gained popularity and captured everyone's attention, GitLab introduced a feature called "Feature Flags." This feature provides the capability to establish rules for user percentages, allowing for more controlled and strategic testing.
In summary, everything we desired was within reach, and we successfully navigated our way to it. However, a challenge emerged as we needed two separate branches for main and main-debug to introduce the configuration in main-debug while leaving main unchanged. Otherwise, although the performance was satisfactory, the bundle size would persist (unless utilizing Tree Shaking, a topic I'll delve into in another article). This solution may not be the perfect fit for our case, but feel free to explore its applicability in your scenario. (We'll explore feature flags in more detail in a future article.)
Host Provider Weighted Load Balance
We also considered another straightforward solution: utilizing Host Provider's DNS distribution via A records with the IPs of our production-ready pods.
However, we encountered challenges with this approach. There was a potential risk for the IP addresses to change for any reason, and this posed a significant risk. While we could ensure static IPs through coordination with the host provider, another concern arose. Placing the configuration outside our repository meant that certain individuals had access to it, prompting us to discard this solution. Nonetheless, for smaller-scale projects, I believe this is a swift and efficient solution.
Load Balancing with Nginx
To start, we need to configure our upstream and create only two upstreams—one for main and another for main-debug pods. Our upstream configuration would resemble the following:
upstream app {
server main-pod-address.svc:3004 weight=95;
server main-debug-pod-address.svc:3004 weight=5;
}
As you can observe, we utilized the weight keyword to distribute users between the two branches: 95% and 5%. Feel free to adjust this based on your specific requirements.
However, another issue arose. The default load balancing strategy, Round Robin, wasn't suitable for our front-end needs. Our domain received more than just HTML document requests – various chunks and CSS files needed to be retrieved. Using Round Robin could potentially crash the front-end since it might fetch chunks and CSS from the main-debug branch, which contains different code. To mitigate this, we opted for a Custom IP Hash strategy. (For additional strategies, consult the HTTP Load Balancing | NGINX Docs). This resulted in our upstream configuration appearing as follows:
upstream app {
hash $http_ar_real_ip consistent; # we used http_ar_real_ip header for our ips
server main-pod-address.svc:3004 weight=95;
server main-debug-pod-address.svc:3004 weight=5;
}
After configuring upstream we will step into config our server block in Nginx and our final server block is this:
server{
listen 80;
listen [::]:80; # For listening to ipv6
server_name behtarino.com;
access_log /dev/stdout;
location / {
proxy_pass http://app; # "app" name came from the upstream
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
And that’s it! now you have balanced users with your desired weight.
Bonus: What is Open Replay?
Explore User Interactions with OpenReplay, Your Open-Source Session Replay Solution
Discover the power of OpenReplay, an open-source session replay suite designed to empower developers to understand user interactions on their web applications. This invaluable tool allows developers to visually trace and analyze user actions, significantly expediting the troubleshooting process.
Beyond simple user action replay, OpenReplay provides comprehensive insights into the inner workings of your website or application. It captures crucial elements such as network activity, console logs, JavaScript errors, store actions/state, page speed metrics, CPU/memory usage, and more.
One standout feature of OpenReplay is its minimal footprint, utilizing a lightweight ~26KB tracker that asynchronously transmits essential data, ensuring minimal impact on overall performance.
For those who prioritize data control and security, OpenReplay is the ideal solution. As a self-hosted tool, all captured data remains within your cloud environment, affording you complete control and ownership. OpenReplay also boasts fine-grained security features for sanitizing user data.
Flexibility is at the core of OpenReplay's deployment strategy, allowing you to deploy it anywhere. It seamlessly integrates with major public cloud providers, including AWS, GCP, Azure, and Digital Ocean.
In addition to its session replay capabilities, OpenReplay offers a suite of features, including developer tools, product analytics, and co-browsing. Explore the possibilities with OpenReplay and elevate your understanding of user interactions on your web applications.
Results
It appears that our journey has reached its conclusion. Yet, I believe there are many more aspects to explore in this ever-evolving landscape. Please contribute and enrich this article further with your insights and experiences.
Thanks for reading. 🌼
Top comments (0)