· SWE Editorial · System Design · 6 min read
Design a Rate Limiter: Architecture and Algorithms
Token bucket vs. leaky bucket vs. sliding window log — the algorithm choice is only half the design. This piece is about where the limiter physically sits in your request path, and why that placement decision matters as much as the algorithm itself.
Two separate decisions, one question
“Design a rate limiter” is really two questions wearing one costume: which algorithm decides whether a request is allowed, and where in the architecture that decision gets made. Most prep material focuses entirely on the first and treats the second as an afterthought. That’s backwards for the interview — the algorithm is a data structure choice; the placement is a systems design choice, and systems design is the thing being graded.
This article treats both as first-class, starting with a full algorithm comparison (including leaky bucket, which often gets skipped), then spending equal time on middleware placement.
The algorithm lineup
Token bucket
A bucket holds up to C tokens, refilling at R tokens/sec. Each request removes a token; empty bucket means reject (or queue). Bursts up to C are allowed as long as the bucket has accumulated tokens during idle periods.
Leaky bucket
Conceptually inverted from token bucket: requests enter a FIFO queue (the “bucket”) and are processed — “leak out” — at a fixed rate R, regardless of arrival burstiness. If the queue is full, new requests are dropped.
The distinction that trips people up: token bucket controls the rate of admission, letting bursts through immediately up to capacity. Leaky bucket controls the rate of processing, smoothing bursty input into a perfectly uniform output rate. If your downstream system genuinely cannot handle bursts (e.g., a fixed-capacity worker pool), leaky bucket is the better model because it enforces a constant output rate rather than just an average one.
Sliding window log
Every request’s timestamp is stored (typically in a sorted set keyed by client ID). To check a new request, prune timestamps outside the window and count what remains. Exact, but memory grows with request volume per key.
Fixed window counter
Increment a counter per fixed time bucket; reset at boundaries. Cheapest to implement, but allows up to 2x the intended rate across a window boundary (full burst at the end of window N, full burst at the start of window N+1).
Sliding window counter
A weighted blend of the current and previous fixed-window counts, approximating the sliding log’s accuracy at fixed-window memory cost. This is what most production systems actually run.
Full comparison table
| Algorithm | Admits bursts? | Smooths output? | Memory | Implementation complexity | Best fit |
|---|---|---|---|---|---|
| Token bucket | Yes, up to bucket size | No — bursts pass straight through | O(1) | Low-medium | User-facing APIs where short bursts are fine |
| Leaky bucket | No — queues and smooths | Yes — constant output rate | O(queue depth) | Medium | Protecting a fixed-capacity downstream (e.g., a worker pool, a legacy DB) |
| Sliding window log | No | N/A (exact accounting) | O(requests in window) | Medium | Low-volume, high-precision billing/quota enforcement |
| Fixed window counter | Yes (boundary artifact) | No | O(1) | Low | Coarse, non-critical internal limits |
| Sliding window counter | Slight | Approximate smoothing | O(1) | Medium | Default choice for most public APIs |
The interview-ready one-liner for each, if asked to pick fast: “Token bucket if bursts are fine and I want cheap state. Leaky bucket if my downstream truly needs a constant rate. Sliding window counter if I want near-exact enforcement without the memory cost of a full log.”
Middleware placement: the part everyone skips
Given an algorithm, where does the check actually run? Four real options, each with a genuinely different failure mode.
Option A: Client-side throttling
[Client SDK self-limits] -> Server (no protection if client is malicious/buggy)
Option B: Edge / API Gateway
[Client] -> [Gateway: rate limit check] -> [App Servers] -> [DB]
Option C: Service-level middleware
[Client] -> [Gateway] -> [App Server: rate limit check] -> [DB]
Option D: Sidecar / service mesh (e.g., Envoy + Redis)
[Client] -> [Gateway] -> [Sidecar: rate limit check] -> [App Server] -> [DB]Option A — client-side. Useful for cooperative clients (your own mobile app respecting a documented limit to avoid wasting battery/data on requests it can predict will fail) but provides zero protection against abuse, since a malicious or buggy client can simply not implement it. Never sufficient alone.
Option B — API gateway / edge. The most common real-world choice for public APIs. Centralizes limiting logic in one place (Kong, Envoy, Cloudflare, AWS API Gateway), protects every downstream service uniformly, and means individual services don’t need to reimplement limiting. Tradeoff: coarse-grained — hard to express business-logic-specific limits like “3 failed login attempts locks the account for 15 minutes,” which needs application context the gateway doesn’t have.
Option C — service-level middleware. Each service enforces its own limits with full access to business context (user tier, feature flags, request payload). More flexible, but logic and configuration get duplicated across services, and a service that forgets to add the middleware is unprotected.
Option D — sidecar in a service mesh. Splits the difference: per-service enforcement without duplicating logic in application code, since the sidecar proxy (Envoy, Linkerd) handles it uniformly via mesh-wide configuration. Adds operational complexity (running a mesh) that’s only worth it if you’re already using one for other reasons (mTLS, observability).
The strongest interview answer names a layered approach: coarse per-IP/per-API-key limits at the gateway to stop abuse cheaply and early, plus fine-grained per-user/per-action limits at the service level where business context is available. This two-tier structure is what most large-scale systems actually run, and stating it explicitly signals you’ve thought about placement as its own design axis, not just picked one box.
A worked example: rate-limiting a login endpoint
To make the placement decision concrete: a login endpoint typically needs both layers. At the gateway: a per-IP limit (say, 50 requests/minute) to blunt credential-stuffing bots hammering many accounts. At the service level: a per-account limit (5 failed attempts triggers a 15-minute lockout) that requires knowing which account is being targeted — information the gateway, operating purely on IP and route, doesn’t have. Token bucket fits the gateway layer (bursts of legitimate retries are fine); a fixed window or sliding window counter fits the account-lockout layer (you want strict, predictable enforcement, not burst tolerance, once a threshold is crossed).
Summary
Algorithm choice governs how a single decision is made; placement governs where and with what context that decision happens. A complete rate limiter design states both explicitly: pick token bucket or leaky bucket or sliding window counter based on whether bursts should pass through or be smoothed, then pick gateway, service, or a layered combination of both based on how much business context the decision needs.
Further reading
The 0-to-1 SWE Interview Playbook (Amazon: https://www.amazon.com/dp/B0H256Z1MF?tag=sirjohnnymai-20) covers this exact two-axis framework — algorithm plus placement — across rate limiters, URL shorteners, and messaging systems, with worked examples like the login-endpoint layering shown above.