Real-time Updates

1. DNS Resolution: The client starts by resolving the domain name of the website to an IP address using DNS (Domain Name System)¹.
2. TCP Handshake: The client initiates a TCP connection with the server using a three-way handshake:

   • SYN: The client sends a SYN (synchronize) packet to the server to request a connection.
   • SYN-ACK: The server responds with a SYN-ACK (synchronize-acknowledge) packet to acknowledge the request.
   • ACK: The client sends an ACK (acknowledge) packet to establish the connection.
3. HTTP Request: Once the TCP connection is established, the client sends an HTTP GET request to the server to request the web page.
4. Server Processing: The server processes the request, retrieves the requested web page, and prepares an HTTP response.
5. HTTP Response: The server sends the HTTP response back to the client, which includes the requested web page content.
6. TCP Teardown: After the data transfer is complete, the client and server close the TCP connection using a four-way handshake:

   • FIN: The client sends a FIN (finish) packet to the server to terminate the connection.
   • ACK: The server acknowledges the FIN packet with an ACK.
   • FIN: The server sends a FIN packet to the client to terminate its side of the connection.
   • ACK: The client acknowledges the server's FIN packet with an ACK.

1. First, each round trip between client and server adds latency to our request, including those to establish connections before we send our application data.
2. Second, the TCP connection itself represents state that both the client and server must maintain. Unless we use features like HTTP keep-alive, we need to repeat this connection setup process for every request - a potentially significant overhead.

With Load Balancers

Layer 4 Load Balancers

• Maintain persistent TCP connections between client and server.
• Fast and efficient due to minimal packet inspection.
• Cannot make routing decisions based on application data.
• Typically used when raw performance is the priority.

Layer 7 Load Balancers

• Terminate incoming connections and create new ones to backend servers.
• Can route based on request content (URL, headers, cookies, etc.).
• More CPU-intensive due to packet inspection.
• Provide more flexibility and features.
• Better suited for HTTP-based traffic.

Approaches for Client Updates

Simple Polling: The Baseline

async function poll() {
  const response = await fetch('/api/updates');
  const data = await response.json();
  processData(data);
}

// Poll every 2 seconds
setInterval(poll, 2000);

Advantages

• Simple to implement.
• Stateless.
• No special infrastructure needed.
• Works with any standard networking infrastructure.
• Doesn't take much time to explain.

Disadvantages

• Higher latency than other methods. Updates might be delayed as long as the polling interval + the time it takes to process the request.
• Limited update frequency.
• More bandwidth usage.
• Can be resource-intensive with many clients, establishing new connections, etc.

When to use simple polling

Things to Discuss in Your Interview

Long Polling: The Easy Solution

1. Client makes HTTP request to server
2. Server holds request open until new data is available
3. Server responds with data
4. Client immediately makes new request
5. Process repeats

// Client-side of long polling
async function longPoll() {
  while (true) {
    try {
      const response = await fetch('/api/updates');
      const data = await response.json();
      
      // Handle data
      processData(data);
    } catch (error) {
      // Handle error
      console.error(error);
      
      // Add small delay before retrying on error
      await new Promise(resolve => setTimeout(resolve, 1000));
    }
  }
}

Assume the latency between the client and server is 100ms.

If we have 2 updates which occur within 10ms of each other, with long polling we'll receive the first update 100ms after it occurred (100ms of network latency) but the second update may be up to 290ms after it occurred (90ms for the first response to finish returning, 100ms for the second request to be made, and another 100ms to get the response).

Advantages

• Builds on standard HTTP and works everywhere HTTP works.
• Easy to implement.
• No special infrastructure needed.
• Stateless server-side.

Disadvantages

• Higher latency than alternatives.
• More HTTP overhead.
• Can be resource-intensive with many clients.
• Not suitable for frequent updates due to the issues mentioned above.
• Makes monitoring more painful since requests can hang around for a long time.
• Browsers limit the number of concurrent connections per domain, meaning you may only be able to have a few long-polling connections per domain.

When to Use Long Polling

Things to Discuss in Your Interview

Server-Sent Events (SSE): The Efficient One-Way Street

How SSE Works

1. Client establishes SSE connection
2. Server keeps connection open
3. Server sends messages when data changes or updates happen
4. Client receives updates in real-time

// Client-side
const eventSource = new EventSource('/api/updates');

eventSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  updateUI(data);
};

// Server-side (Node.js/Express example)
app.get('/api/updates', (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  const sendUpdate = (data) => {
    res.write(`data: ${JSON.stringify(data)}\n\n`);
  };

  // Send updates when data changes
  dataSource.on('update', sendUpdate);

  // Clean up on client disconnect
  req.on('close', () => {
    dataSource.off('update', sendUpdate);
  });
});

Advantages

• Built into browsers.
• Automatic reconnection.
• Works over HTTP.
• More efficient than long polling due to less connection initiation/teardown.
• Simple to implement.

Disadvantages

• One-way communication only.
• Limited browser support (not an issue for modern browsers).
• Some proxies and networking equipment don't support streaming. Nasty to debug!
• Browsers limit the number of concurrent connections per domain, meaning you may only be able to have a few SSE connections per domain.
• Makes monitoring more painful since requests can hang around for a long time.

When to Use SSE

Things to Discuss in Your Interview

WebSocket: The Full-Duplex Champion

How WebSocket Works

1. Client initiates WebSocket handshake over HTTP
2. Connection upgrades to WebSocket protocol
3. Both client and server can send messages
4. Connection stays open until explicitly closed

// Client-side
const ws = new WebSocket('ws://api.example.com/socket');

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  handleUpdate(data);
};

ws.onclose = () => {
  // Implement reconnection logic
  setTimeout(connectWebSocket, 1000);
};

// Server-side (Node.js/ws example)
const WebSocket = require('ws');
const wss = new WebSocket.Server({ port: 8080 });

wss.on('connection', (ws) => {
  ws.on('message', (message) => {
    // Handle incoming messages
    const data = JSON.parse(message);
    processMessage(data);
  });

  // Send updates to client
  dataSource.on('update', (data) => {
    ws.send(JSON.stringify(data));
  });
});

Extra Challenges

Advantages

• Full-duplex (read and write) communication.
• Lower latency than HTTP due to reduced overhead (e.g. no headers).
• Efficient for frequent messages.
• Wide browser support.

Disadvantages

• More complex to implement.
• Requires special infrastructure.
• Stateful connections, can make load balancing and scaling more complex.
• Need to handle reconnection.

When to Use WebSockets

Things to Discuss in Your Interview

WebRTC: The Peer-to-Peer Solution

• STUN: "Session Traversal Utilities for NAT" is a protocol and a set of techniques like "hole punching" which allows peers to establish publically routable addresses and ports. I won't go into details here, but as hacky as it sounds it's a standard way to deal with NAT traversal and it involves repeatedly creating open ports and sharing them via the signaling server with peers.
• TURN: "Traversal Using Relays around NAT" is effectively a relay service, a way to bounce requests through a central server which can then be routed to the appropriate peer.

How WebRTC Works

1. Peers discover each other through signaling server.
2. Exchange connection info (ICE candidates)
3. Establish direct peer connection, using STUN/TURN if needed
4. Stream audio/video or send data directly

// Simplified WebRTC setup
async function startCall() {
  const pc = new RTCPeerConnection();
  
  // Get local stream
  const stream = await navigator.mediaDevices.getUserMedia({
    video: true,
    audio: true
  });
  
  // Add tracks to peer connection
  stream.getTracks().forEach(track => {
    pc.addTrack(track, stream);
  });
  
  // Create and send offer
  const offer = await pc.createOffer();
  await pc.setLocalDescription(offer);
  
  // Send offer to signaling server
  signalingServer.send(offer);
}

When to Use WebRTC

Advantages

• Direct peer communication
• Lower latency
• Reduced server costs
• Native audio/video support

Disadvantages

• Complex setup (> WebSockets)
• Requires signaling server
• NAT/firewall issues
• Connection setup delay

Things to Discuss in Your Interview

Client Update Overview

• If you're not latency sensitive, simple polling is a great baseline. You should probably start here unless you have a specific need in your system.
• If you don't need bi-directional communication, SSE is a great choice. It's lightweight and works well for many use cases. There are some infrastructure considerations to keep in mind, but they're less invasive than with WebSocket and generally interviewers are less familiar with them or less critical if you don't address them.
• If you need frequent, bi-directional communication, WebSocket is the way to go. It's more complex, but the performance benefits are huge.
• Finally, if you need to do audio/video calls, WebRTC is the only way to go. In some instances peer-to-peer collaboration can be enhanced with WebRTC, but you're unlikely to see it in a system design interview.

Server-Side Push/pull

1. Pulling via Polling
2. Pushing via Consistent Hashes
3. Pushing via Pub/Sub

Pulling with Simple Polling

Advantages

• Dead simple to implement.
• State is constrained to our DB.
• No special infrastructure.
• Doesn't take much time to explain.

Disadvantages

• High latency.
• Excess DB load when updates are infrequent and polling is frequent.

When to Use Pull-Based Polling

Things to Discuss in Your Interview

Pushing via Consistent Hashes

1. Figure out which server User C is connected to.
2. Send the message to that server.
3. That server will look up which (websocket, SSE, long-polling) request is associated with User C.
4. The server will then write the message via the appropriate connection.

Simple Hashing

1. The client connects to a random server.
2. The server looks up the client's hash in Zookeeper to figure out which server is responsible for them.
3. The server redirects the client to the appropriate server.
4. The client connects to the correct server.
5. The server adds that client to a map of connections.

1. Our Update Server stays connected to Zookeeper and knows the addresses of all servers and the modulo N.
2. When the Update Server needs to send a message to User C, it can hash the user's id to figure out which server is responsible for them (Server 2) and sends the message there.
3. Server 2 receives the message, looks up which connection is associated with User C, and sends the message to that connection.

Consistent Hashing

Advantages

• Predictable server assignment
• Minimal connection disruption during scaling
• Works well with stateful connections
• Easy to add/remove servers

Disadvantages

• Complex to implement correctly
• Requires coordination service (like Zookeeper)
• All servers need to maintain routing information
• Connection state is lost if a server fails

When to Use Consistent Hashing

Things to Discuss in Your Interview

1. Signaling the beginning of a scaling event. Recording both the old and new server assignments.
2. Slowly disconnecting clients from the old server and having them reconnect to their newly assigned server.
3. Signaling the end of the scaling event and updating the coordination service with the new server assignments.
4. In the interim, having messages which are sent to both the old and new server until they're fully transitioned.

Pushing via Pub/Sub

1. The client establishes a connection with an endpoint server.
2. The endpoint server registers the client with the Pub/Sub service, often by creating a topic, subscribing to it, and keeping a mapping from topics to the connections associated with them.

1. Updates are pushed to the Pub/Sub service to the relevant topic.
2. The Pub/Sub service broadcasts the update to all clients subscribed to that topic.
3. The endpoint server receives the update, looks up which client is subscribed to that topic, and forwards the update to that client over the existing connection.

Advantages

• Managing load on endpoint servers is easy, we can use a simple load balancer with "least connections" strategy.
• We can broadcast updates to a large number of clients efficiently.
• We minimize the proliferation of state through our system.

Disadvantages

• We don't know whether subscribers are connected to the endpoint server, or when they disconnect.
• The Pub/Sub service becomes a single point of failure and bottleneck.
• We introduce an additional layer of indirection which can add to latency.
• There exist many-to-many connections between Pub/Sub service hosts and the endpoint servers.

When to Use Pub/Sub

Things to Discuss in Your Interview

Overview of Server-Side Push/Pull

1. Prefer polling if possible!
2. If you do need real-time updates, consider Pub/Sub. It applies to most use-cases and is easy to implement.
3. Finally, if your application requires a lot of state/resources for each connection, consistent hashing will provide a more optimal use of resources.

Wrapping Up

1. How do clients connect and receive updates from my system?
2. How do I trigger updates from the source and get them to the clients?