· software-engineers Editorial · Career · 5 min read
Cdc Change Data Capture Implementation Guide
Practical CDC implementation guide comparing Debezium, log-based vs trigger-based capture, and production pitfalls for 2026.
CDC Change Data Capture Implementation Guide
Change Data Capture (CDC) is the mechanism that turns a database’s internal write activity into a consumable stream of events. Instead of polling tables on a schedule, CDC taps the transaction log directly, giving downstream systems a near-real-time, ordered feed of inserts, updates, and deletes. As of July 2026, CDC underpins most event-driven architectures, cache invalidation systems, search index synchronization, and data lake ingestion pipelines. This guide covers the four implementation strategies engineering teams actually use in production, where each breaks down at scale, and what interviewers expect you to know when CDC comes up in a system design round.
Why CDC Exists: The Polling Problem
Before CDC became mainstream, teams synchronized data with one of two bad options: periodic full-table dumps or updated_at polling queries. Both have structural flaws.
Full-table dumps do not scale past a few million rows without saturating I/O and network bandwidth on every run. updated_at polling misses hard deletes entirely (a DELETE statement does not update a timestamp column) and misses updates made by triggers or bulk jobs that bypass the ORM layer setting that field. Both approaches also introduce latency measured in minutes, which breaks any use case requiring sub-second propagation, such as cache invalidation or fraud detection pipelines.
CDC solves this by reading the database’s write-ahead log (WAL in Postgres, binlog in MySQL, redo log in Oracle) — the same internal structure the database itself uses for replication and crash recovery. Every committed transaction is captured exactly once, in commit order, with zero additional load on the primary tables.
Four CDC Implementation Patterns
1. Log-Based CDC (Debezium, AWS DMS, Fivetran)
Log-based CDC reads the transaction log directly. Debezium is the dominant open-source implementation as of 2026, running as a Kafka Connect plugin that tails MySQL binlogs, Postgres logical replication slots, MongoDB oplogs, or SQL Server CDC tables.
Advantages: zero impact on source database performance, captures deletes natively, preserves transaction ordering, and can replay from any retained log position for recovery.
Tradeoffs: requires elevated database permissions (REPLICATION CLIENT in MySQL, superuser or rds_replication role in Postgres), and log retention windows limit how far back you can rewind before requiring a full re-snapshot.
2. Trigger-Based CDC
Database triggers fire on INSERT/UPDATE/DELETE and write the change to a shadow “outbox” table, which a separate poller reads and clears. This was the standard approach before log-based tools matured.
Advantages: works on any database without special permissions, simple to reason about, no dependency on vendor-specific log formats.
Tradeoffs: doubles write amplification on every transaction (the trigger write happens inside the same transaction as the original write), and triggers are notoriously hard to maintain across schema migrations. Most teams migrating off trigger-based CDC in 2026 cite maintenance burden as the primary driver.
3. Query-Based (Timestamp/Watermark Polling)
Still used for legacy systems or SaaS APIs with no log access. A scheduled job queries WHERE updated_at > :last_watermark, processes results, and advances the watermark.
Advantages: no special database access required, trivial to implement.
Tradeoffs: cannot capture hard deletes without a soft-delete convention, latency is bounded by poll interval, and clock skew across replicas can silently drop rows near the watermark boundary.
4. Dual-Write with Outbox Pattern
Rather than capturing changes after the fact, the application writes both the business record and an “event” row to an outbox table in the same transaction, then a CDC connector (usually log-based) tails only the outbox table. This avoids the classic dual-write consistency bug where a message broker publish succeeds but the database commit fails (or vice versa).
Comparison Table
| Pattern | Latency | DB Load | Captures Deletes | Setup Complexity | Best For |
|---|---|---|---|---|---|
| Log-based (Debezium) | Sub-second | None | Yes | Medium-High | Real-time pipelines, microservices sync |
| Trigger-based | Seconds | High (write amp) | Yes | Medium | Legacy DBs without log access |
| Query/watermark polling | Minutes | Medium | No (needs soft delete) | Low | Batch ETL, SaaS API sources |
| Outbox + log-based | Sub-second | Low | Yes | High | Transactional consistency across services |
Production Pitfalls Teams Hit in 2026
Schema evolution breaks consumers. Adding a column is usually safe; renaming or changing a column type without a compatibility layer (Avro schema registry, Protobuf field numbering) breaks every downstream consumer simultaneously. Always run CDC events through a schema registry with backward-compatibility enforcement.
Snapshot + stream handoff race conditions. Debezium and DMS both perform an initial full snapshot before switching to streaming mode. If the snapshot isn’t wrapped in a consistent read (using REPEATABLE READ or a replication slot’s LSN as the cutoff), rows written during the snapshot window can be duplicated or lost entirely.
Log retention exhaustion. If a Kafka Connect worker goes down for longer than the binlog/WAL retention period, the connector cannot resume and requires a full re-snapshot — an expensive, sometimes multi-hour operation on large tables. Teams should monitor connector lag against log retention as a first-class SLO.
Ordering guarantees only hold per-key. Kafka partitions by key (usually primary key), which guarantees ordering within a single row’s changes but not across rows. Downstream joins that assume global ordering will produce incorrect results.
FAQ
Q: Is Debezium the only production-grade log-based CDC tool in 2026? A: No — AWS DMS, Google Datastream, Fivetran, and Airbyte all offer managed log-based CDC. Debezium remains dominant for self-hosted Kafka-centric stacks because it’s open source and has the broadest connector ecosystem (Postgres, MySQL, MongoDB, SQL Server, Oracle, Cassandra).
Q: How do I handle CDC in a system design interview? A: Name the transaction log mechanism explicitly (WAL/binlog), explain why polling fails at scale (missed deletes, latency), and mention the outbox pattern if the question involves multi-service consistency. Interviewers are checking whether you understand log-based capture is fundamentally different from application-level polling.
Q: Can CDC replace a message queue entirely? A: Not directly — CDC produces raw row-level change events tied to database schema, while application events are typically higher-level business events (e.g., “OrderShipped”). Many teams use CDC to populate an outbox table that then emits clean business events, combining both patterns.
Engineers preparing for system design rounds where CDC, event-driven architecture, and data pipeline questions come up should also shore up core interview fundamentals. The 0-to-1 SWE Interview Playbook (https://www.amazon.com/dp/B0H256Z1MF?tag=sirjohnnymai-20) covers the system design frameworks, including data pipeline and event-driven design patterns, that interviewers at data-intensive companies test for in 2026 loops.