Communication: Data Fetching Patterns

Big announcement!
I have started my daily learning journey of Frontend system design. And I'll be sharing insights from each module in the blogs. So, here's the start and there is much more to come!

In this blog, we’ll explore different data fetching mechanisms essential to front-end system design, including short polling, long polling, WebSockets, server-sent events (SSE), and webhooks. Each technique addresses specific needs for delivering data to and from the client and server, and understanding them is crucial for designing scalable, real-time web applications.

1. Short Polling

Short polling is a method where the client repeatedly sends requests to the server at regular intervals to check for updates.

setInterval(async () => {
  const response = await fetch('/api/get-posts');
  const data = await response.json();
  // Update UI with new data
}, 5000); // Poll every 5 seconds

• Short live communication
• No persistence of the data
• Less resource utility
• Server load due to the repeated requests
• Increased bandwidth usage

Eg - Stock market ticker, Social media feeds

2. Long Polling

Long polling is an enhancement over short polling, where the client sends a request, and the server holds the connection open until it has new data to return.

From Back-end, the response will be sent only when data is updated, till then it will hold the request. If there is no update for a long time then the timeout is processed.

Client side

async function subscribe() {
  let response = await fetch("/subscribe");

  if (response.status == 502) {
    // Status 502 is a connection timeout error, let's reconnect
    await subscribe();
  } else if (response.status != 200) {
    // An error - let's show it
    showMessage(response.statusText);
    // Reconnect in one second
    await new Promise(resolve => setTimeout(resolve, 1000));
    await subscribe();
  } else {
    // Get and show the message
    let message = await response.text();
    showMessage(message);
    // Call subscribe() again to get the next message
    await subscribe();
  }
}

subscribe();

• Single long-lived connection
• Provides real-time updates with fewer requests than short polling.
• Reduces unnecessary data transfer when there are no updates.
• The connection can be kept open for long durations, increasing server load.

Eg- Live customer support chats

3. Web Sockets

WebSockets allow for full-duplex communication between the client and server, making it the most efficient method for real-time data transfer.

The client opens a WebSocket connection with the server and both the client and server can send messages to each other over this single connection.

webSocket = new WebSocket(url, protocols);

// Send message
webSocket.send("Here's some text that the server is urgently awaiting!");

// Receive message
webSocket.onmessage = (event) => {
  console.log(event.data);
};

• Continuous bi-directional communication
• Multiple libraries to implement it - ws, socket.io etc.
• Efficient for high-frequency updates with low overhead
• Challenges - maintain communication with 1 server, handle failure and scaling, occupies the resources.

Eg- Live chat applications, Online multiplayer games

4. Server-Sent Events (SSE)

SSE provides a unidirectional stream of updates from the server to the client over an HTTP connection.

const evtSource = new EventSource("ssedemo.php");

evtSource.onmessage = (event) => {
  const newElement = document.createElement("li");
  const eventList = document.getElementById("list");

  newElement.textContent = `message: ${event.data}`;
  eventList.appendChild(newElement);
};

• Long live uni-directional communication
• Single HTTP connection
• Challenges - resource utilization, Browser compatibility and behaviour on inactive tabs

Eg - Feeds, Notifications

5. Webhooks

Webhooks are a server-to-server communication mechanism where the server sends data to a predefined URL when an event occurs. The client doesn’t need to keep checking the server for updates.

Popular for triggering actions between systems, such as payment notifications, GitHub events, or third-party service integrations.

Conclusion

Choosing the right communication method depends on your application's requirements. WebSockets and SSE are perfect for real-time and streaming data, while long polling offers a balance between performance and ease of use. Short polling is a simple solution for infrequent updates but can be resource-intensive, and webhooks are ideal for server-to-server notifications.

Each technique has its own advantages and limitations. Understanding these can help you make informed decisions to build efficient, responsive web applications.

This blog is open for suggestions and discussions!

Comments

[John Peters]

With the concept of sinalr, the older polling designs may become obsolete. Similar to pub/sub designs, push notification is a better concept as it follows the notify event pattern.