Replication: The Replicated* Engines

The trade-off

A single ClickHouse node is a single point of failure. Its disk fails, or someone runs a destructive ALTER against it, or the box itself dies, and every table on it is unavailable — or, without a separate backup, gone. ReplicatedMergeTree trades that fragility for durability (the data exists in more than one place) and availability (a query can still be served if one replica is down) — in exchange for running more infrastructure: multiple ClickHouse nodes to hold the replicas, and a ClickHouse Keeper cluster to coordinate them. Replication works at the level of an individual table, not the entire server — you opt specific tables in by choosing a Replicated* engine, not by flipping a cluster-wide switch.

That infrastructure cost comes with two things worth being explicit about before going further, because both get glossed over by "just replicate it" advice:

Replication is not backup. A replica faithfully copies whatever happens to the table — including a bad DROP TABLE, an accidental TRUNCATE, or a mutation that corrupts data. If what you're actually worried about is operator or application error rather than hardware failure, more replicas don't help; you need backups, ideally ones with some retention or immutability, kept separately from every replica.

Replication is asynchronous by default, which means "we have replicas" does not mean "every replica is always in sync." There is a real, if usually small, window where a replica is behind. Whether that window is a non-issue or a correctness bug depends entirely on what the workload needs — the same "what staleness can this query tolerate" question from the ReplacingMergeTree lesson, one layer further down the stack.

The alternatives, steelmanned

Single-node ClickHouse with backups (BACKUP/RESTORE, or scheduled snapshots to object storage). The case for it: this is dramatically simpler to operate. No Keeper cluster to run, patch, and monitor; no replication lag to reason about on any query; roughly half (or less) the storage and compute cost of running live standby replicas nobody is querying most of the time. For workloads where the business impact of "unavailable for the time it takes to restore from last night's backup" rounds to zero — an internal reporting table refreshed hourly by batch ETL, a dev or staging environment, anything genuinely tolerant of an occasional multi-hour gap — this is not a lesser choice made under budget pressure. It's the correctly-sized one. Paying continuously for live replica infrastructure to protect against a risk a nightly backup already absorbs is real, ongoing cost for a problem that doesn't exist for that specific table.

Managed or cloud-provided replication. The case for it: you get the durability and availability outcome without personally operating the coordination system underneath it. ClickHouse Cloud is the concrete example — it doesn't run classic ReplicatedMergeTree under the hood at all. It uses SharedMergeTree, described as "a cloud-native replacement of the ReplicatedMergeTree engines that is optimized to work on top of shared storage," where, unlike ReplicatedMergeTree, replicas "[don't] require... to communicate with each other" directly — "all communication happens through shared storage and clickhouse-keeper" — and "for an end-user, nothing needs to be changed to start using the SharedMergeTree engine family." In other words: you still write CREATE TABLE statements that look like the ones in this lesson, but the vendor absorbs the job of running and monitoring the coordination layer. This is the right call for a team that wants the outcome (durable, available data) without the ops capacity — or the appetite — to run a Keeper cluster themselves. The cost is real too: vendor lock-in, less control over exact replica topology, and an ongoing subscription cost that, at large enough scale, can exceed the cost of infrastructure you'd run yourself.

Self-hosted ReplicatedMergeTree — the subject of this lesson. The case for it: full control over topology and infrastructure, works anywhere you can run ClickHouse (your own cloud VMs, on-prem, wherever), and it composes directly with sharding — the Distributed table engine covered in Module 6 — if you also need to scale writes and storage horizontally, not just tolerate a node failure. The cost you pay is in infrastructure and ongoing ops attention rather than vendor margin: running a Keeper cluster, monitoring replica health, and building the team's fluency in a new class of failure mode. It's the right choice when the durability/availability need is real, you're already running (or are willing to run and staff) multi-node infrastructure, and the team has — or is deliberately building — the operational capability a coordination system demands.

How replication actually works

You need enough of the mechanism to reason about the trade-off, not to operate a cluster blind.

A ReplicatedMergeTree table takes two extra parameters beyond a plain MergeTree: zoo_path, "the path to the table in ClickHouse Keeper," and replica_name, "the replica name in ClickHouse Keeper." In practice these are almost always templated with {shard} and {replica} macros defined per-server, so the identical CREATE TABLE statement runs unmodified on every node:

CREATE TABLE events
(
    event_time DateTime,
    event_type LowCardinality(String),
    user_id    UInt64
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/events', '{replica}')
ORDER BY (event_type, event_time);

Replication is asynchronous and multi-master: any replica can accept an INSERT. Data is written to the replica that received the insert first, then copied to the other replicas, with latency roughly equal to the time it takes to transfer that block of compressed data over the network. By default, an INSERT only waits for confirmation from one replica before returning success to the client; the insert_quorum setting trades insert latency for a stronger guarantee — waiting for confirmation from multiple replicas before acknowledging the write. That default is exactly why "we added replication" doesn't mean "every replica is always caught up" — there's a real window, usually small, where a replica hasn't yet received the latest inserts.

Replication also gives you automatic insert deduplication: identical data blocks written more than once are only stored once. That makes retrying a failed or ambiguous INSERT safe by default — a client that isn't sure whether its write succeeded can just retry it, without a separate idempotency layer. It's a narrower, automatic version of the "upstream deduplication" alternative from the ReplacingMergeTree lesson — it protects against network-level retries, not against your application inserting the same logical row twice for unrelated reasons.

One detail worth being precise about, because it's easy to assume the opposite: by default, merges are not performed once and shipped to other replicas. "Only the source data to insert is transferred over the network. Further data transformation (merging) is coordinated and performed on all the replicas in the same way." Each replica redoes the same merge independently, driven by the same coordinated instructions — so the CPU cost of merging is paid on every replica by default, not once, which is part of the real compute cost you're signing up for, not just the storage cost of extra copies. That default isn't absolute: ClickHouse also has settings — prefer_fetch_merged_part_size_threshold/prefer_fetch_merged_part_time_threshold, and execute_merges_on_single_replica_time_threshold, which lets "only a single replica start[] the merge immediately, and other replicas wait... to download the result instead of doing merges locally" — that let a replica fetch an already-merged part from another replica instead, specifically to speed up very large or slow merges. The default behavior is still the one to plan capacity around; the fetch-instead settings are an escape hatch for expensive merges, not the norm.

ClickHouse Keeper is what makes all of this coordinated rather than chaotic. It stores replica metadata and the shared log of "what should exist" — not a copy of your column data — and it's involved in inserts, ALTERs, and failure recovery, but "ZooKeeper is not used in SELECT queries." A struggling Keeper cluster can stall writes and coordination without necessarily stalling reads already in flight. Keeper itself is a small ClickHouse-native reimplementation of ZooKeeper's coordination protocol, built on RAFT — a consensus algorithm that lets a cluster of nodes agree on shared state even when some of them are unreachable, by requiring any change to be accepted by a majority before it counts. Keeper is compatible with the ZooKeeper client protocol, and like any RAFT-based system it needs its own majority — a quorum — to keep working: a 3-node Keeper cluster tolerates one node crashing. Run Keeper on a single node and you haven't removed your single point of failure — you've relocated it, from "the ClickHouse node" to "the Keeper node," and now a coordination-layer failure can stall every replicated table on the cluster at once, not just the data on one box.

Replica health itself is something you check, not something you assume — system.replicas reports whether a replica is in read-only mode, whether its Keeper session has expired, and how many parts have been permanently lost across the table's history. Reading that table fluently as a diagnostic tool is Module 7 territory; for now, the thing worth knowing is that "is our replication healthy" is a question with a concrete, queryable answer, not a guess.

Decision framework

The reasoning a senior engineer runs through, in order:

1. What does downtime or data loss on this specific table actually cost? If the honest answer rounds to zero — an internal dashboard refreshed hourly, a table nobody queries outside business hours — single-node plus backups is the correctly-sized answer, not a corner cut.
2. If durability or availability genuinely matters, can it be bought from a managed vendor for less than the cost of operating it yourself? A team without dedicated infrastructure or on-call capacity for a coordination system very often comes out ahead paying a vendor to run one, even at a real dollar premium.
3. If self-hosting, are you already running multiple ClickHouse nodes for another reason — sharding for write or storage scale? If so, the marginal cost of also replicating is much lower: you already carry the multi-node operational surface, and adding Keeper coordination on top of infrastructure you're already staffing is a smaller step than standing up a coordination system from a single node.
4. Do you need protection against a node dying, or protection against bad data? These are different failure classes with different fixes. Replication addresses the first. Only backups — kept separately, ideally with some retention or immutability — address the second, and you need them regardless of what you decide about replication.
5. If replicating, does this workload need insert_quorum's stronger guarantee, or is the default asynchronous, eventually-caught-up replica tolerable? The same staleness-tolerance question that drove the ReplacingMergeTree decision framework, applied one layer down: a dashboard that can be a few seconds behind on one replica is a very different case from a query that must never read stale data no matter which replica serves it.

Case studies

A plausible good fit: business-critical analytics under continuous query load. Companies like Cloudflare, Uber, and Yandex are widely known to run ClickHouse for analytics at real production scale — the general shape of "continuously queried, business-critical, already running enough infrastructure that adding replication is a marginal step, not a first step" is the textbook fit for self-hosted ReplicatedMergeTree: teams at that scale are typically already sharding for data volume, which means the multi-node operational surface Keeper needs already exists.

A composite bad fit, representative of a common mistake rather than a single named incident: a small team runs two shards times three replicas — six ClickHouse nodes — plus a separate three-node Keeper cluster, nine machines in total, for a reporting table refreshed weekly and queried by roughly five internal stakeholders. The actual availability requirement — tolerant of an hour of downtime, refreshed weekly regardless — never justified live standby infrastructure at all, let alone nine machines' worth. The team ends up spending real on-call attention on Keeper session issues for a table whose worst-case failure was always "the weekly refresh runs an hour late."

Anti-pattern gallery

Running Keeper on a single node. Keeper needs its own quorum to tolerate a failure — a 3-node cluster survives one crash, a 1-node "cluster" survives none. Putting Keeper on one box doesn't remove your single point of failure; it moves it from the ClickHouse layer to the coordination layer, and a coordination-layer failure now threatens every replicated table on the cluster simultaneously rather than just the data on one node.

Treating replication as a backup strategy. A replica applies whatever happens to the source table, mistakes included — a bad DROP TABLE or a destructive mutation propagates to every replica just as faithfully as legitimate data does. If the real risk is operator or application error rather than hardware failure, the fix is backups kept separately, not more live copies of the same mutable state.

Assuming synchronous consistency by default. Reading from one replica immediately after writing to a different one can return a stale result, because replication is asynchronous unless insert_quorum (and a matching read-side setting) says otherwise. A team that assumes replicas are always instantly in sync ships a quiet read-your-own-write bug, not a hardware-fault-tolerance win.

Starving Keeper of resources by co-locating it with an already-loaded ClickHouse node "to save a server." Because writes and coordination on every replicated table depend on Keeper being responsive, a Keeper process competing for CPU or memory with a busy ClickHouse process can degrade replication cluster-wide, not just on the shared box.