· software-engineers Editorial · Career · 6 min read
Read Replica Scaling Database Architecture
How read replicas actually scale reads, where replication lag breaks correctness, and the routing patterns senior engineers use in production.
Why Read Replicas Exist
Every write-heavy production database eventually hits a wall: a single primary instance can only push so many reads before CPU, I/O, or connection limits cap throughput. In July 2026, with most SaaS backends running Postgres 17 or MySQL 8.4, the standard answer is read replica scaling — copying data asynchronously (or semi-synchronously) to one or more secondary nodes and routing read traffic away from the primary.
The mechanics are simple to state and hard to operate correctly. A primary streams a write-ahead log (WAL in Postgres, binlog in MySQL) to replicas. Replicas apply those changes and serve SELECT queries. Writes still go to the primary. This is horizontal read scaling, not horizontal write scaling — a critical distinction that trips up engineers who assume replicas solve write contention. They don’t. Sharding solves write contention. Replicas solve read contention.
Real production numbers: a single well-tuned Postgres primary on modern NVMe hardware handles roughly 15,000–40,000 simple read QPS before latency degrades. Add three replicas with a round-robin read router and you can push 60,000–120,000 read QPS without touching the primary’s write capacity. That’s the entire value proposition.
Replication Lag: The Problem Nobody Diagrams Correctly
The gap between “data committed on primary” and “data visible on replica” is replication lag, typically 10ms–2s under normal load, but capable of spiking to minutes during large writes, vacuum operations, or network partitions. This lag is the single most common source of production bugs in read-replica architectures.
The classic failure mode: a user submits a form (write to primary), the application redirects to a confirmation page (read from replica), and the replica hasn’t caught up yet — the user sees a blank or stale page. This is the “read-after-write” problem, and it is not a hypothetical. It is the number one replica-related bug reported in incident postmortems at companies running replica fleets.
Three mitigation patterns, ranked by how often they appear in production systems:
- Read-your-writes stickiness — route a user’s reads to the primary for N seconds after their own write, then fall back to replicas. Simple, effective, adds primary load proportional to write-active users only.
- Monotonic read tracking — pass a log sequence number (LSN) or GTID with each request; replicas reject reads that are behind that position, and the router retries against the primary or a caught-up replica.
- Causal consistency via session pinning — pin a session to one replica for its duration so at least reads are internally consistent, even if occasionally stale relative to the primary.
Most teams start with pattern 1 because it requires no schema or protocol changes — just an if (recentWrite) routeToPrimary() check in the data access layer.
Routing Patterns Compared
| Pattern | Write path | Read path | Failure behavior | Best for |
|---|---|---|---|---|
| Application-level routing | Primary connection pool | Round-robin replica pool | App must handle replica down (retry/fallback) | Small-to-mid teams, full control needed |
| Proxy-based (ProxySQL, PgBouncer + pgpool) | Proxy detects writes, sends to primary | Proxy load-balances reads | Proxy handles failover transparently | Mid-to-large teams, polyglot clients |
| Managed cloud router (RDS Proxy, Aurora reader endpoint) | Cluster endpoint | Reader endpoint auto-balances | Cloud provider handles replica health checks | Teams already on managed DB, want zero ops |
| Service-mesh-aware routing (read/write split at sidecar) | Sidecar intercepts SQL, routes by statement type | Sidecar load-balances | Mesh-level circuit breaking | Large orgs with existing service mesh |
The tradeoff axis is always the same: how much routing logic lives in your application versus infrastructure. Application-level routing gives you the most control over consistency guarantees (you can implement read-your-writes precisely) but couples every service to replica topology. Proxy-based and managed routers reduce operational burden but often only expose coarse “route to primary if statement contains INSERT/UPDATE/DELETE” logic, which breaks for read-modify-write patterns hidden inside stored procedures or ORMs that issue SELECT ... FOR UPDATE.
Failover and Promotion
When a primary dies, one replica must be promoted. This is where read-replica architectures are frequently under-designed. Key facts:
- Promotion is not instant. Postgres promotion (
pg_promote) typically takes 5–30 seconds depending on how much WAL is unapplied. During that window, writes fail entirely. - Replicas can diverge. If the old primary comes back online after a promotion, it now has writes the new primary never saw (split-brain risk). Production systems must fence the old primary (STONITH — Shoot The Other Node In The Head, or a managed equivalent) before it can rejoin.
- Replica lag affects RPO. If your most caught-up replica was 500ms behind at the moment of primary failure, you lose up to 500ms of committed writes on failover. This is your Recovery Point Objective, and it must be stated explicitly in any system design interview answer, not glossed over.
- Automated failover tools — Patroni for Postgres, Orchestrator for MySQL, or cloud-native equivalents (Aurora, Cloud SQL) — reduce promotion time to under 30 seconds but add operational complexity: they require a distributed consensus layer (etcd/Consul/ZooKeeper) to avoid electing two primaries simultaneously.
Interview Framing: What Interviewers Actually Test
When “design a system with read replicas” comes up in a system design interview, weak answers stop at “add replicas, round-robin reads.” Strong answers address, in order: (1) what consistency model the product actually needs — does a stale read for 200ms break the user experience, or is eventual consistency fine; (2) how read-after-write is handled for the specific write paths that matter (checkout confirmation, not analytics dashboards); (3) what happens during failover, including RPO/RTO numbers; (4) how replica lag is monitored (pg_stat_replication, replica lag in seconds as a first-class metric with alerting, not an afterthought).
This is exactly the kind of system design depth interviewers are probing for in mid-to-senior SWE loops in 2026, and it’s covered end-to-end with worked examples in The 0-to-1 SWE Interview Playbook — available on Amazon. The playbook walks through replica topology tradeoffs alongside sharding, caching layers, and the exact whiteboard sequencing interviewers expect.
FAQ
Q: Do read replicas help with write-heavy workloads? No. Replicas only offload reads. If your bottleneck is write throughput, you need sharding, a queue-based write buffer, or a different storage engine — replicas add zero write capacity and actually add slight overhead to the primary (streaming WAL to N replicas costs I/O and network).
Q: How many replicas is “too many”? Diminishing returns typically start at 5–7 replicas per primary because each replica adds WAL streaming overhead to the primary and increases the chance that at least one replica lags significantly, complicating routing logic. Past that point, most teams shard instead of adding more replicas.
Q: Can replicas be used for backups instead of read scaling?
Yes, and many teams do both — a dedicated “backup replica” that’s excluded from the read pool, runs pg_dump or snapshot jobs, and is allowed higher lag since it’s never read from directly by the application.