Case Study: Orchestrating Two Frontend Systems on a Single Page with Optimal UX and CWV"

Introduction: I work as a Senior Software Engineer at Naukrigulf, primarily focusing on the frontend system. For those who aren't familiar, Naukrigulf is a job search site catering to the Gulf regions

Besides providing users with the best job search results, which is our product's top priority, we also place a strong emphasis on enhancing user experience.

Use Case: Recently, we embarked on a project to integrate personalised banners(birthday, festive greeting etc) into our search listing page, a high-traffic area, especially on mobile devices.

Our aim was to seamlessly integrate personalised banners amidst the job listings without disrupting the user's relevant job search experience.

Look at result:

Such scenarios are quite common in our domain, where displaying banners, ads, campaigns, or sponsored content within the results enhances the overall user experience and adds a personalised touch.

Solution: The Widgeting System, a separate system designed to handle frontend tasks like displaying banners, ads, and campaigns based on custom logic and rules. Managed by a dedicated team, it operates independently, ensuring seamless integration of UI components with its provided SDK.

Insights of our in house system

The Widgeting System is divided into two primary components:

1. Dashboard: Here, you input the page name, unique section ID, JavaScript path for the UI component, and rules like how many times or in which date range you want to render it.
2. Loading and initialising SDK: After specifying the page name, ensure a DOM node exists with the matching unique section ID. Once confirmed, the SDK loads and initializes accordingly. If the DOM node exists, the SDK downloads the provided JavaScript path and injects the UI component into it.

In our use case for the search result page, we need to display the banner after the third tuple of the search results as shown in the screenshot attached above.

We have two systems in place:

Web app: This system loads the search results from the search API.
Widgeting: This system loads the banner using the SDK.

Previously, the construction of the search results page followed this sequence:

1. Upon landing on the search results page, we initiated a call to the search API.
2. While the data loaded, we displayed skeleton/shimmer components to indicate loading progress.
3. Once the data was retrieved, we removed the shimmer components and constructed the data cards component.
4. In essence, we had to wait until the data was available before displaying it.

As for how Widgeting works:

1. First, you load the SDK JavaScript.
2. Next, you initialise the SDK with the required page name. This asynchronously loads the UI provided through the JavaScript path in the widget dashboard, assuming that the section ID which was mentioned in the dashboard, is present in the DOM with any element's id, it injects the UI into that element.

Considering these factors, our initial approach would involve:

First Approach:

Upon reaching the search result page:

1. Initiate a call to the search API.
2. Display skeleton components on the UI to indicate loading progress.
3. Simultaneously, load the SDK and initialise it with the corresponding page name. Additionally, add a div with the required section ID for loading the widget.
4. Both actions run in parallel to minimise synchronous waiting time.
5. Once both are loaded, organise the widget and its placement through DOM manipulation. However, this method could be resource-intensive and result in a suboptimal user experience due to visible UI movements.

This approach aims to optimise loading times but may result in a less than ideal user experience due to visible rearrangements on the page.

A little improvement for DOM manipulation

For organising the cards and banner within them, we utilise the CSS property "order," which operates effectively within a flex container (you can look up its functionality for more details).

Our current approach involves:

1. Displaying:
   • Skeleton of search results
   • DOM Node for the widget with a section ID and an order defined as 3
2. Removing the skeleton once the search page data becomes available. Meanwhile, the SDK manages the banner widget, ensuring it eventually loads on the screen according to the rules specified for that particular widget.

The issue arises from both systems independently constructing their UI without synchronising with each other, leading to two potential scenarios:

1. The banner is constructed while the search page displays the skeleton for search results. See this for example:

1. Search results are constructed while showing the skeleton of the banner after the third tuple.

In the second scenario, if the SDK decides not to show the banner based on certain rules (e.g., the festival promotion is no longer valid), it appears as though the skeleton provided for the banner should be removed.

This situation presents challenges in terms of Cumulative Layout Shift (CLS). If the banner skeleton doesn't have a defined height and is decided to be shown at runtime, it causes the fourth tuple to shift down to accommodate its height. Conversely, if the banner is no longer meant to be visible, it collapses at runtime, causing the fourth tuple to move up.

Both scenarios impact CLS, which we aim to avoid.

More ideas

After giving it much thought, we decided to take a hit, which meant that if the banner is ready to be displayed at the time we have our search results data, we will show the banner; otherwise, we will neglect it, since our main aim is to deliver the search results and we cannot compromise with that.

But this came as another challenge: how will we know if the banner is ready to be displayed by the SDK, as it is an asynchronous task that eventually attaches the banner DOM to the section ID we provided?

Another step forward

I came up with an idea: we can initially hide the banner DOM element by applying a "visibility: hidden" style. Once we're done retrieving the search results data and it's time to display the search results cards, we'll check if the banner has been injected into the provided DOM node. If it's already attached, we'll toggle the visibility of the banner DOM element. If not, we'll leave things as they are until the SDK finishes executing the rules for the banner.

see this:

But what happens if, after some time, the SDK finally decides to show the banner and injects it into the DOM using the provided DOM ID?

Here's the catch: since the SDK has attached the banner UI to the provided DOM, it will update the rules at the browser level, indicating that the banner was shown to the user, even if it's not actually visible due to the "visibility: hidden" property. This leads to inconsistent behavior.

For example, if we set a rule for a banner to be shown at most once to the user, even if it wasn't actually shown, it will still be counted as shown. Consequently, the user will never be able to see that banner.

*Next Idea : *

Ultimately, we concluded that we need some form of callback to know when the SDK will inject the banner into the provided DOM. If we receive this callback at the time of displaying search results, we'll toggle the visibility property of banner DOM without removing the DOM, as it will eventually be filled with the banner. However, if we haven't received the callback yet, we'll simply remove the provided banner DOM from the DOM tree and construct our search results page.

Furthermore, if the SDK later attempts to inject the banner into the DOM but the element with section-id is no longer present in the DOM, the banner won't be injected, and the rules won't be updated. This situation will be referred to as "banner-skipped," indicating that we were unable to load the SDK banner before the search results API.

Next step is to get a callback from SDK:

The Widgeting System serves as a general solution in our domain, meaning that we cannot request a feature based solely on our use case unless it's applicable to other scenarios as well.

In light of this, we decided to request a callback from the SDK indicating when it's ready to inject the banner into the provided DOM.
Fortunately, the Widget SDK supports a PubSub Model (If this functionality wasn't available, we would have implemented a callback mechanism similar to this to meet our requirements.), which emits an event with banner(widget) details after calculating the rules but before injecting any widget on the page.
We subscribed to this event and listened for it.

If the event wasn't available by the time we began constructing the search results page from the search API, we removed the provided DOM for the banner. However, if the event was available, the banner would be painted in the visibility hidden div, and we could simply toggle its visibility.

The decided approach and implementation were as follows:

1. We land on the search page.
2. We fire the get search results API.
3. We load the SDK:
   • Initialise the SDK with the page name.
4. We display the skeleton of the search results page, and the DOM node for banner injection at later stage.
5. We retrieve the search results from the API and start constructing the results page. During this process, we remove the DOM node for the banner or toggle its visibility based on whether we have received the event emitted by the SDK.

This approach seemed like a solution, but it resulted in more instances of banner-skipped hits, meaning the banner was less frequently visible to the user. This was because we were loading and initiating the banner after calling the search API or landing on the search results page.

To address this, we decided to take a step further and initialise the SDK with the page name before landing on the page itself. We were able to achieve this because we lazy load all our routes based on user navigation. Instead of loading all the JavaScript files at once, we load them as required.

During routing, we use a custom Wrapper component for the Route component, which adds some basic route-related properties required throughout the app. We managed to check the expected route to be loaded within that wrapper component and initiated the SDK page name call from there.

However, this approach led to another issue (It will eventually end, don't worry XD )

Since we were initiating the SDK init call even before our JavaScript for that particular route had been loaded, there were instances where the SDK completed its rules calculation and started looking for the provided DOM node to inject the banner before our JavaScript for that route had been downloaded. This occurred before our search page was mounted. Although this happened infrequently, we sought to improve it.

Another improvement:

To address this issue, we moved the SDK call initiation into the Search Results component initiator. This means that we initiated the SDK call just before rendering the Search component.

(For your information, we also attempted to initiate the SDK call after the component was mounted using the useEffect hook of the Search component. However, this resulted in the same problem of delaying the SDK init call and led to skipped-banner instances.)

By moving the SDK init call to just before rendering the Search component, we ensured that the JavaScript for the search component was loaded almost at the same time as the SDK initiated the page name call and calculated the rules for it. This allowed us to provide the DOM ID for the SDK to use for banner injection.

Look at the order in which we loaded search result js (srp.js)
and widget sdk javascript file.

*Whoahhh! Now is the time: *

After so many approaches, here's the final solution:

1. Load and fire the SDK init call just before the component is being mounted.
2. Load the JavaScript for the Search Component.
3. Fire the Search API and ensure that the DOM node IDs for the banner and the search results skeleton are in the DOM. The DOM node for the banner has visibility set to false by default.
4. As soon as the SDK emits the event for banner injection, we store it at our level to use it in the future.
5. Once the search API resolves, we check for the emitted and stored event at our end. We then construct the search results page, toggling the visibility style or removing the DOM node for the banner based on the stored value (if it is present).

Conclusion:

After extensive iterations and considerations, we developed a solution to efficiently display banners alongside search results on our platform. Initially, we encountered challenges with synchronising the banner display with the loading of search results. To address this, we implemented various strategies such as preloading the SDK, optimizing component mounting, and utilising a PubSub model for event handling. Ultimately, we refined our approach to ensure that the SDK initialisation occurs just before component mounting, allowing for seamless integration of banners with search results. Through these efforts, we achieved a streamlined process that optimises user experience while maintaining the integrity of our search functionality.