Skip to content

h3nr7

Data movement strategies in system design

The consumer sends a request and the source answers with its current state.

The idea in one line

Every strategy answers two questions: who starts the transfer, and how long the connection stays open.


Pull and long polling start with the consumer.

Push, webhooks, SSE and WebSockets start with the source.

A queue puts a broker between the two, so neither has to be online at once.

01 · Pull-based approach

Pull is simple, but the consumer sees a change only when it asks

The consumer sends a request and the source answers with its current state.

Your appconsumerServicesource1asks for the record2the data comes backrecord changes at the sourceNothing new arrives until the app asks again
Pull: request and response, then a window where changes go unseen.

The consumer asks the source for data whenever it needs it, and the source answers with the current state. The consumer decides when to ask, so the source stays simple and stateless. The cost is staleness: a change at the source is invisible until the next request, and asking more often wastes calls when nothing has changed.

Best for data that is read on demand or changes slowly.

Watch out for polling many clients at a short interval, which multiplies load for little new data.

Example

A dashboard page fetches an order when it opens and then refreshes it every 30 seconds.

setInterval(async () => {
  const res = await fetch('/api/orders/42');
  render(await res.json());
}, 30_000);

02 · Push-based approach

Push gets data there fast, but the source must cope with a consumer that is down

Each event is delivered as it happens; a delivery that gets no answer is retried.

Your appconsumerServicesourcepayment taken · delivered at oncerefund issued · delivered at onceparcel shipped · the app never answersRetried: the same event may arrive twice
Push: events delivered at once, one retried after a failed delivery.

The source sends data to the consumer the moment something happens, so the consumer never has to ask and latency stays low. In return the source must know where to deliver and what to do when the consumer is slow or down, which usually means retrying. Webhooks, SSE, WebSockets and queue delivery are all specific ways of pushing.

Best for changes that must reach consumers quickly.

Watch out for duplicates and gaps, so give each event an id the consumer can use to de-duplicate.

Example

A payments service pushes a refund.issued event to the accounting service as soon as a refund is made, and retries if accounting returns an error.

// source side: push on every state change
async function onRefundIssued(refund) {
  const subs = subscribers('refund.issued');
  for (const sub of subs) {
    await deliverWithRetry(sub.url, {
      id: refund.eventId,
      type: 'refund.issued',
      data: refund,
    });
  }
}

03 · Webhooks

A webhook is push over HTTP, so expect retries, duplicates and no ordering

Registration happens once; after that the provider calls your endpoint for each event and expects a 2xx reply.

Your appPOST /hooks/payPayment servicecharge succeeded1register the URL (once)2POST the event when it happens3200 OKRetriesif no 2xx answerDuplicatessame event can repeatNo orderingorder not guaranteed
Webhook: register, POST, 200 OK, and the three caveats to handle.

A webhook is push over plain HTTP. Your app registers a URL with the provider once; when an event happens, the provider sends an HTTP POST to that URL. Your endpoint must be reachable from the internet, verify the request signature, and answer with a 2xx status quickly. Providers retry anything that is not acknowledged, so duplicates and out-of-order events are normal and the handler has to cope with both.

Best for server-to-server notifications between systems you do not control.

Watch out for treating the endpoint as exactly-once; store the event id and, if order matters, re-fetch the object’s current state.

Example

Stripe sends charge.succeeded to /hooks/pay. The handler checks the signature, skips events it has already seen, queues the work and replies 200.

// rawBody = express.raw({ type: 'application/json' })
app.post('/hooks/pay', rawBody, async (req, res) => {
  const sig = req.headers['stripe-signature'];
  const event = stripe.webhooks.constructEvent(
    req.body, sig, SECRET);
  if (await seen(event.id)) {
    return res.sendStatus(200);  // duplicate
  }
  await enqueue(event);          // do the work later
  res.sendStatus(200);           // answer fast
});

04 · Long polling

Long polling holds the request open, so a pull behaves almost like a push

A request is answered the moment data exists; if the time limit passes first, the empty answer triggers the next request.

WorkerconsumerQueuesource1any work for me?the server keeps the request opena task arrives2answered the instant it exists3asks againnothing arrives before the time limit4empty answer, so ask again
Long polling: a held request, an answer, then an empty timeout.

Long polling is a pull that the server answers late. The client sends a request, the server holds it open until there is data or a timeout expires, and the client asks again straight away. You get close to push latency over ordinary HTTP, without the many empty round trips of short polling. The costs are one held connection per waiting client and a full set of HTTP headers on every round trip.

Best for clients that can only make plain HTTP requests, and queue workers.

Watch out for proxies or load balancers whose idle timeout is shorter than your wait, and many clients each holding a connection.

Example

An Amazon SQS worker asks for messages and waits up to 20 seconds on each call instead of hammering an empty queue.

while True:
    resp = sqs.receive_message(
        QueueUrl=URL,
        WaitTimeSeconds=20,
        MaxNumberOfMessages=10,
    )
    for m in resp.get('Messages', []):
        handle(m['Body'])
        sqs.delete_message(
            QueueUrl=URL,
            ReceiptHandle=m['ReceiptHandle'],
        )

05 · SSE vs WebSockets

SSE is a one-way stream; WebSockets are a two-way channel you manage yourself

Choose SSE when only the server talks; choose WebSockets when the client also sends often.

Server-sent eventsBrowserServerWebSocketsBrowserServerOnly the server sends eventsReconnects itself, resumes by idText events over plain HTTPEither side can push at any timeStarts as an HTTP upgrade (101)Reconnect logic is yours to write
Server-sent events against WebSockets: direction, reconnects and handshake.

Both keep one long-lived connection open so the server can push. Server-sent events stream text events down a single HTTP response in one direction; browsers ship the EventSource API, which reconnects by itself and sends the last event id it saw so the server can resume. A WebSocket starts as an HTTP upgrade (a 101 Switching Protocols reply) and then carries messages in both directions on one TCP connection. It gives you more freedom, but reconnects, heartbeats and backoff are your job or your library’s.

Best for SSE for notifications, live dashboards and streamed responses; WebSockets for chat, games and shared editing.

Watch out for both hold a connection per client, so budget for connection counts and idle timeouts on proxies and load balancers.

Example

A live order tracker only receives updates, so it uses SSE. A chat box sends and receives, so it uses a WebSocket.

// SSE: one-way, reconnects by itself
const es = new EventSource('/orders/42/events');
es.onmessage = e => render(JSON.parse(e.data));

// WebSocket: two-way, you handle reconnects
const ws = new WebSocket('wss://example.com');
ws.onmessage = e => show(e.data);
ws.send(JSON.stringify({ text: 'hi' }));

06 · Message queues

A queue lets the producer move on while the work waits for a worker

The producer only needs the queue to be up, and work survives a worker being offline.

Upload serviceproducerQueuework waits hereuntil someone takes itWorker Abroker pushes to itWorker Bfetches when readyWorker Coffline: work waitsproducepushpullAt-least-once deliverya message can be redelivered, so make workers idempotent
A message queue: one producer, one queue, three workers.

A queue puts a broker between the producer and the workers. The producer writes a message and moves on; the message waits until a worker takes it, and each message goes to one worker. If a worker is offline its share simply waits, and more workers can be added to drain a backlog. Some brokers push messages to consumers (RabbitMQ basic.consume), others make workers poll (Amazon SQS, RabbitMQ basic.get). Most deliver at least once, using acknowledgements or a visibility timeout to hand a message to someone else if a worker fails, so workers must be idempotent.

Best for background jobs, load levelling and decoupling services that run at different speeds.

Watch out for duplicates, best-effort ordering on standard queues (FIFO queues order messages within a message group), and a visibility timeout shorter than your slowest job.

Example

An upload service queues a thumbnail job per image; workers pick jobs up at their own pace. With RabbitMQ and Python:

# producer (upload service)
channel.basic_publish(
    exchange='',
    routing_key='thumbnails',
    body=json.dumps({'image_id': 123}),
)

# worker: the broker pushes jobs,
# prefetch caps how many are in flight
channel.basic_qos(prefetch_count=10)
channel.basic_consume(
    queue='thumbnails',
    on_message_callback=make_thumbnail,
)

Across all six

Four habits apply whichever strategy you pick

01

Give every event an id

A retry looks the same as a duplicate, so the consumer needs something to de-duplicate on.

02

Make handlers idempotent

Handling a message twice must leave the same result as handling it once.

03

Answer fast, work later

Acknowledge the webhook or message first and queue the slow work.

04

Plan for the consumer being away

Retries, resume ids and queues all exist because the other side is sometimes offline.

Side by side

Pick by who must start the transfer, and what a missed or duplicated message would cost

StrategyWho startsConnectionDirectionMain caveatPick it when
PullConsumerA short request each timeSource to consumerStale between requests; wasted callsData is read on demand or changes slowly
PushSourceDepends on the mechanismSource to consumerSource must handle slow or absent consumersChanges must arrive fast and you control both ends
WebhookSourceA new HTTP POST per eventSource to consumerRetries, duplicates, no ordering guaranteeA third party must notify your backend
Long pollingConsumerOne held HTTP requestSource to consumerHeld connections; proxy idle timeoutsOnly plain HTTP is available
SSEClient opens, server sendsOne long-lived HTTP responseServer to clientOne-way onlyLive feeds to browsers
WebSocketsClient opens, either side sendsOne long-lived TCP connectionBoth waysYou manage reconnectsInteractive two-way apps
Message queueProducer writes, worker takesThrough a brokerProducer to workerAt-least-once delivery; best-effort orderingBackground jobs and decoupled services

In short

Choose by who needs to know first, and how much loss you can live with.

01

Reading on demand

Pull, or long polling when you need it sooner.

02

A one-way feed to browsers

SSE, with reconnects built in.

03

Two-way and interactive

WebSockets, with your own reconnect logic.

04

Work that can wait

A queue, with idempotent workers.

Webhooks cover the case where a system you do not control has to tell yours.

References

Where the details were checked

The Stripe and MDN pages could not be opened while writing this. Their behaviour was checked through search-result excerpts and secondary write-ups, so confirm exact retry schedules and browser limits on the linked pages.