Data movement strategies in system design
The consumer sends a request and the source answers with its current state.
The idea in one line
Every strategy answers two questions: who starts the transfer, and how long the connection stays open.
Pull and long polling start with the consumer.
Push, webhooks, SSE and WebSockets start with the source.
A queue puts a broker between the two, so neither has to be online at once.
01 · Pull-based approach
Pull is simple, but the consumer sees a change only when it asks
The consumer sends a request and the source answers with its current state.
The consumer asks the source for data whenever it needs it, and the source answers with the current state. The consumer decides when to ask, so the source stays simple and stateless. The cost is staleness: a change at the source is invisible until the next request, and asking more often wastes calls when nothing has changed.
Best for data that is read on demand or changes slowly.
Watch out for polling many clients at a short interval, which multiplies load for little new data.
Example
A dashboard page fetches an order when it opens and then refreshes it every 30 seconds.
setInterval(async () => {
const res = await fetch('/api/orders/42');
render(await res.json());
}, 30_000);
02 · Push-based approach
Push gets data there fast, but the source must cope with a consumer that is down
Each event is delivered as it happens; a delivery that gets no answer is retried.
The source sends data to the consumer the moment something happens, so the consumer never has to ask and latency stays low. In return the source must know where to deliver and what to do when the consumer is slow or down, which usually means retrying. Webhooks, SSE, WebSockets and queue delivery are all specific ways of pushing.
Best for changes that must reach consumers quickly.
Watch out for duplicates and gaps, so give each event an id the consumer can use to de-duplicate.
Example
A payments service pushes a refund.issued event to the accounting service as soon as a refund is made, and retries if accounting returns an error.
// source side: push on every state change
async function onRefundIssued(refund) {
const subs = subscribers('refund.issued');
for (const sub of subs) {
await deliverWithRetry(sub.url, {
id: refund.eventId,
type: 'refund.issued',
data: refund,
});
}
}
03 · Webhooks
A webhook is push over HTTP, so expect retries, duplicates and no ordering
Registration happens once; after that the provider calls your endpoint for each event and expects a 2xx reply.
A webhook is push over plain HTTP. Your app registers a URL with the provider once; when an event happens, the provider sends an HTTP POST to that URL. Your endpoint must be reachable from the internet, verify the request signature, and answer with a 2xx status quickly. Providers retry anything that is not acknowledged, so duplicates and out-of-order events are normal and the handler has to cope with both.
Best for server-to-server notifications between systems you do not control.
Watch out for treating the endpoint as exactly-once; store the event id and, if order matters, re-fetch the object’s current state.
Example
Stripe sends charge.succeeded to /hooks/pay. The handler checks the signature, skips events it has already seen, queues the work and replies 200.
// rawBody = express.raw({ type: 'application/json' })
app.post('/hooks/pay', rawBody, async (req, res) => {
const sig = req.headers['stripe-signature'];
const event = stripe.webhooks.constructEvent(
req.body, sig, SECRET);
if (await seen(event.id)) {
return res.sendStatus(200); // duplicate
}
await enqueue(event); // do the work later
res.sendStatus(200); // answer fast
});
04 · Long polling
Long polling holds the request open, so a pull behaves almost like a push
A request is answered the moment data exists; if the time limit passes first, the empty answer triggers the next request.
Long polling is a pull that the server answers late. The client sends a request, the server holds it open until there is data or a timeout expires, and the client asks again straight away. You get close to push latency over ordinary HTTP, without the many empty round trips of short polling. The costs are one held connection per waiting client and a full set of HTTP headers on every round trip.
Best for clients that can only make plain HTTP requests, and queue workers.
Watch out for proxies or load balancers whose idle timeout is shorter than your wait, and many clients each holding a connection.
Example
An Amazon SQS worker asks for messages and waits up to 20 seconds on each call instead of hammering an empty queue.
while True:
resp = sqs.receive_message(
QueueUrl=URL,
WaitTimeSeconds=20,
MaxNumberOfMessages=10,
)
for m in resp.get('Messages', []):
handle(m['Body'])
sqs.delete_message(
QueueUrl=URL,
ReceiptHandle=m['ReceiptHandle'],
)
05 · SSE vs WebSockets
SSE is a one-way stream; WebSockets are a two-way channel you manage yourself
Choose SSE when only the server talks; choose WebSockets when the client also sends often.
Both keep one long-lived connection open so the server can push. Server-sent events stream text events down a single HTTP response in one direction; browsers ship the EventSource API, which reconnects by itself and sends the last event id it saw so the server can resume. A WebSocket starts as an HTTP upgrade (a 101 Switching Protocols reply) and then carries messages in both directions on one TCP connection. It gives you more freedom, but reconnects, heartbeats and backoff are your job or your library’s.
Best for SSE for notifications, live dashboards and streamed responses; WebSockets for chat, games and shared editing.
Watch out for both hold a connection per client, so budget for connection counts and idle timeouts on proxies and load balancers.
Example
A live order tracker only receives updates, so it uses SSE. A chat box sends and receives, so it uses a WebSocket.
// SSE: one-way, reconnects by itself
const es = new EventSource('/orders/42/events');
es.onmessage = e => render(JSON.parse(e.data));
// WebSocket: two-way, you handle reconnects
const ws = new WebSocket('wss://example.com');
ws.onmessage = e => show(e.data);
ws.send(JSON.stringify({ text: 'hi' }));
06 · Message queues
A queue lets the producer move on while the work waits for a worker
The producer only needs the queue to be up, and work survives a worker being offline.
A queue puts a broker between the producer and the workers. The producer writes a message and moves on; the message waits until a worker takes it, and each message goes to one worker. If a worker is offline its share simply waits, and more workers can be added to drain a backlog. Some brokers push messages to consumers (RabbitMQ basic.consume), others make workers poll (Amazon SQS, RabbitMQ basic.get). Most deliver at least once, using acknowledgements or a visibility timeout to hand a message to someone else if a worker fails, so workers must be idempotent.
Best for background jobs, load levelling and decoupling services that run at different speeds.
Watch out for duplicates, best-effort ordering on standard queues (FIFO queues order messages within a message group), and a visibility timeout shorter than your slowest job.
Example
An upload service queues a thumbnail job per image; workers pick jobs up at their own pace. With RabbitMQ and Python:
# producer (upload service)
channel.basic_publish(
exchange='',
routing_key='thumbnails',
body=json.dumps({'image_id': 123}),
)
# worker: the broker pushes jobs,
# prefetch caps how many are in flight
channel.basic_qos(prefetch_count=10)
channel.basic_consume(
queue='thumbnails',
on_message_callback=make_thumbnail,
)
Across all six
Four habits apply whichever strategy you pick
01
Give every event an id
A retry looks the same as a duplicate, so the consumer needs something to de-duplicate on.
02
Make handlers idempotent
Handling a message twice must leave the same result as handling it once.
03
Answer fast, work later
Acknowledge the webhook or message first and queue the slow work.
04
Plan for the consumer being away
Retries, resume ids and queues all exist because the other side is sometimes offline.
Side by side
Pick by who must start the transfer, and what a missed or duplicated message would cost
| Strategy | Who starts | Connection | Direction | Main caveat | Pick it when |
|---|---|---|---|---|---|
| Pull | Consumer | A short request each time | Source to consumer | Stale between requests; wasted calls | Data is read on demand or changes slowly |
| Push | Source | Depends on the mechanism | Source to consumer | Source must handle slow or absent consumers | Changes must arrive fast and you control both ends |
| Webhook | Source | A new HTTP POST per event | Source to consumer | Retries, duplicates, no ordering guarantee | A third party must notify your backend |
| Long polling | Consumer | One held HTTP request | Source to consumer | Held connections; proxy idle timeouts | Only plain HTTP is available |
| SSE | Client opens, server sends | One long-lived HTTP response | Server to client | One-way only | Live feeds to browsers |
| WebSockets | Client opens, either side sends | One long-lived TCP connection | Both ways | You manage reconnects | Interactive two-way apps |
| Message queue | Producer writes, worker takes | Through a broker | Producer to worker | At-least-once delivery; best-effort ordering | Background jobs and decoupled services |
In short
Choose by who needs to know first, and how much loss you can live with.
01
Reading on demand
Pull, or long polling when you need it sooner.
02
A one-way feed to browsers
SSE, with reconnects built in.
03
Two-way and interactive
WebSockets, with your own reconnect logic.
04
Work that can wait
A queue, with idempotent workers.
Webhooks cover the case where a system you do not control has to tell yours.
References
Where the details were checked
The Stripe and MDN pages could not be opened while writing this. Their behaviour was checked through search-result excerpts and secondary write-ups, so confirm exact retry schedules and browser limits on the linked pages.
- Source infographic: ByteByteGo, “Top Data Movement Strategies in System Design”
- RFC 6202: Known Issues and Best Practices for Long Polling and Streaming in Bidirectional HTTP
- RFC 6455: The WebSocket Protocol
- WHATWG WebSockets Living Standard
- MDN: Using server-sent events
- javascript.info: Server-Sent Events
- High Performance Browser Networking: Server-Sent Events
- Stripe: Webhooks
- Hookdeck: At-least-once vs exactly-once webhook delivery
- Amazon SQS: visibility timeout
- Amazon SQS FAQs: long polling and delivery semantics
- RabbitMQ: Consumers
- CloudAMQP: basic.consume vs basic.get