Screener.in Search API: A Performance Checkup! 🔎

Hey everyone! So, I've been diving deep into fundamental analysis lately, using a popular stock screener website to help me find promising companies. I'm a big believer in understanding the "why" behind every number, and that extends to the tools I use. You know how it is – you start with the big picture, but the developer in me always wants to peel back the layers and see how the sausage is made.

It all started innocently enough. I was using the screener's search functionality, finding companies that met my specific criteria. Everything seemed fine on the surface, clean and efficient. But my curiosity got the better of me. I began typing my search query, and, out of habit, I opened my browser's network tab to watch the API calls. That’s when I noticed something peculiar. For every single keystroke, a new API call was sent to the server! This means that if I typed "KPI Green Energy" the screener sent separate requests for "k," "kp," "kpi," "kpig," and finally "kpigreen". Each query is sent to the backend without any delay or optimization, leading to redundant network requests.

I did a little timing analysis to see what was going on, and here’s what I found:

• API Call Durations: The server itself seems remarkably fast. I saw response times of 164ms (ThrottlingURLLoader::OnReceiveResponse), 88ms (MojoURLLoaderClient::OnReceiveResponse), and 81ms (ResourceRequestSender::OnReceivedResponse). This suggests the backend is handling requests efficiently.

• XHR Events: The real culprit was on the client-side (my browser). The rapid-fire succession of XHRReadyStateChange and XHRLoad events confirmed my suspicion: no intelligent throttling or delay mechanism existed. Essentially, the browser was sending every single keystroke to the server immediately without any waiting period.

I found some serious problems with how the stock screener's search works. It's like a leaky faucet wasting water (bandwidth and server power). Here's what's wrong:

⚠️ The Problems:

• No Debouncing: Every time you type a letter, the search sends a message to the server. This is way too many messages! It's like sending a text message for every letter in a word instead of just sending the whole word at once. This makes the server work too hard and wastes internet speed.

• No Throttling: There's no speed limit on how fast it sends these messages. Type fast enough, and you'll flood the server with requests.

• Redundant Calls: Lots of the messages sent are for incomplete words (like "k," "kr," "kri") that don't matter once you finish typing. It's like writing down every unfinished sentence you start before finally finishing one.

• No Client-side Caching: If you type something wrong and backspace, it sends the same messages all over again instead of remembering what it already knows. It's like forgetting what you just said.

📊 The Effects:

1. Excessive Server Load: The server is overwhelmed by a high volume of requests, most of which are for incomplete queries.
2. Unnecessary Network Activity: The network is flooded with redundant data transfers, consuming bandwidth.
3. Slow Response Times: The constant stream of requests can lead to slower responses, as the server is busy handling unnecessary calls.

💡 Recommendations:

• Debouncing for API Calls

Debouncing is like having a "wait-and-see" approach. Instead of responding to every keystroke instantly, it waits for a specified period of inactivity before executing the search.
Introduce a debounce timer to delay API calls until the user has paused typing (e.g., 300ms).

let debounceTimer; 

const searchInput = document.getElementById("input");

searchInput.addEventListener("input", (event) => {
  const inputValue = event.target.value;
  const delay = 500; // Set delay to 500ms (0.5 seconds)

  // 1. Clear Any Existing Timer:
  //    If the user is still typing, we cancel any pending search.
  clearTimeout(debounceTimer);

  // 2. Start a New Timer:
  //    After the delay, execute the search only if the user stopped typing.
  debounceTimer = setTimeout(() => {
      // 3. Perform the Search Action:
      //    The searchTasks function is called only when no input has been received for 500ms
      console.log("Performing search for:", inputValue); 
      searchTasks(inputValue);  // Replace this line with actual search call
  }, delay);

});

// Dummy search function - Replace this with your actual search implementation
function searchTasks(query) {
  // Here you would perform the actual search logic
  // Example: Fetch data from API using 'query'
  console.log("Searching for: ", query);
}

How it Works in Practice

1. The user starts typing in the input.
2. Each character entered triggers the input event.
3. With each event, the previously scheduled timeout is cleared, and a new timer is set.
4. If the user pauses for 500 milliseconds, the setTimeout callback runs and calls the searchTasks function with the current input value.
5. If the user keeps typing, the timeout will continue to get cleared and reset, preventing searchTasks from being called until the typing stops for 500 milliseconds.

• Throttling to Rate-limit Calls

Throttling is like implementing a "gatekeeper". It limits the rate at which a function is allowed to be executed. Even if the user continues to type rapidly, a new search request is only sent after the specified delay, limiting the number of requests.
Limit the number of API calls over a specific time period (e.g., one call per second).

let lastSearchTime = 0; // Timestamp of the last search
const searchInput = document.getElementById("input");

searchInput.addEventListener("input", (event) => {
    const currentTime = Date.now();
    const throttleDelay = 700; // Throttle delay (700ms)

    const inputValue = event.target.value.trim();

  // 1. Check if enough time has passed since the last search
    if (currentTime - lastSearchTime >= throttleDelay) {
    // 2. Perform Search Action
        console.log("Throttled search for: ", inputValue);
        searchTasks(inputValue); // Execute the search logic
        lastSearchTime = currentTime; // Update last search time
    } else {
        console.log("Throttled, not searching for: ", inputValue);
    }
});

// Dummy search function - Replace this with your actual search implementation
function searchTasks(query) {
    // Here you would perform the actual search logic
    // Example: Fetch data from API using 'query'
    console.log("Searching for: ", query);
}

How it Works in Practice

1. The user starts typing in the input.
2. Each character entered triggers the input event.
3. With each event, the current time is compared with the time when the search was last triggered (lastSearchTime).
4. If the time difference is greater than the throttleDelay, the search logic (searchTasks function) is executed, and the lastSearchTime is updated.
5. If the time difference is less than the throttleDelay, the search is skipped, preventing the searchTasks function from being called multiple times. This makes sure the server is not overwhelmed by the search requests while the user is actively typing and reduces server load.

• Client-side Caching

Client-side caching stores results from previous searches directly in the user's browser. If the user searches for the same term (or a similar term that can use a cached result), the system can directly retrieve the result from the browser rather than making a new server call.
Example: If a user types krishna, cache the result for krish and kri for faster retrieval.

let searchCache = {}; // Initialize the cache
const searchInput = document.getElementById("input");

searchInput.addEventListener("input", async (event) => {
    const query = event.target.value.trim();

    if (!query) {
        // If empty query clear the results and return early.
        updateSearchResults([]);
        return;
    }

    const cachedResult = getCachedResult(query);
    if (cachedResult) {
       console.log("Using cache for query:", query);
       updateSearchResults(cachedResult);
       return;
    }

    console.log("Making API call for query:", query);
    // Simulate API call. In reality this would be an actual network call
    const results = await fetchSearchResults(query);

    if(results) {
        cacheSearchResults(query, results);
        updateSearchResults(results);
    }
});

function getCachedResult(query) {
    if(searchCache[query]) return searchCache[query];

    for (let i = query.length - 1; i > 0; i--) {
      const prefix = query.substring(0, i);
      if (searchCache[prefix]) {
          console.log("Using cached results for prefix: ", prefix);
          return searchCache[prefix];
      }
    }
    return null;
}

function cacheSearchResults(query, results) {
   // Store results for current query and all of its prefixes.
    for (let i = 1; i <= query.length; i++) {
        const prefix = query.substring(0, i);
        searchCache[prefix] = results;
    }
    console.log("Cache updated: ", searchCache);
}

// Simulate fetching results from an API (replace with actual API call).
async function fetchSearchResults(query) {
    return new Promise(resolve => {
        // Simulate API delay
        setTimeout(() => {
            const mockResults = generateMockResults(query);
            resolve(mockResults);
        }, 500);
    });
}

function updateSearchResults(results) {
    const resultsContainer = document.getElementById("results");
    resultsContainer.innerHTML = ""; // clear previous results

    if (results && results.length > 0) {
       results.forEach(result => {
          const resultItem = document.createElement("li");
          resultItem.textContent = result;
          resultsContainer.appendChild(resultItem);
      });
    } else {
        const noResults = document.createElement("li");
        noResults.textContent = "No results found";
        resultsContainer.appendChild(noResults);
    }
}

function generateMockResults(query) {
    const results = [];
    for (let i = 1; i <= 5; i++) {
        results.push(`${query} Result ${i}`);
    }
    return results;
}

How it Works

1. The user types in the search input.
2. The input event listener is triggered.
3. The getCachedResult function is called to check the cache.
4. If the result is in the cache, it's retrieved and displayed directly.
5. If the result is not in cache the fetchSearchResults function is called to fetch the search results from the API, which is simulated in our example code.
6. The results are then cached for the query and all of its prefixes using cacheSearchResults and are displayed to the user with the updateSearchResults function.

Key Points

1. Prefix Caching: The code caches results not just for the full query but also for each of its prefixes. This helps in making the search more responsive as the user types.
2. Mock API: For demonstration, I have simulated API calls using setTimeout. You should replace this with your actual API call function.
3. Simple Cache: This is a very basic implementation. For a real-world application, consider:
4. Cache Expiration: Results should expire after a certain period.
5. Max Size: Limit the size of the cache to prevent memory issues.
6. Cache Invalidation Strategies: Develop policies on how and when to remove items from the cache.
7. Empty Query Handling: Added an additional check to handle empty queries by clearing the results

To Use

1. Replace the mock fetchSearchResults function with your actual API calling logic.
2. Make sure there is an HTML element with id input for input field and an element with id results to show the search results.

• Send Big Messages If possible, send one big message with all the search information instead of lots of little ones.

Optimizing search functionalities is vital for building efficient applications. Debouncing, throttling, client-side caching, and batch requests are all powerful techniques that can dramatically improve your search system’s performance.
This synergy results in a more responsive, efficient, and overall better user experience. It decreases the load on the server and the network, and ultimately gives the users an experience that is smooth and performant.

I hope the Screener.in developers see this article. It looks like they missed some important website tricks that would make the search much faster and easier to use. They should check it out and improve things.

What are your thoughts on these insights? Let me know! 💭