Voyager Labs

Rebuilding a Kafka pipeline without downtime

Consumer groups made the migration dull, which is exactly what we wanted: new consumers joined the new topic, lag was compared per partition until it matched, then traffic was cut over during a maintenance window that nobody outside the team noticed.

We kept the old topic writable for eleven days. The dual-write layer was a small producer wrapper, four lines of configuration, not a new service. The lesson: the migration plan matters more than the tooling; every step had a measurement that said whether to continue.

← All notes