Known Failure Modes and Their Signatures

The mental model: a failure has a signature before it has a diagnosis

Every failure mode this course has covered eventually resolves to the same shape: a specific named exception, a specific timing pattern, and a specific system table that already knows something is wrong. mergetree-on-disk-format and codebase-tour-clickhouse-source both hit DB::Exception: Too many parts (300) — that's a named exception, TOO_MANY_PARTS, with a hard-threshold timing pattern (writes work fine, then abruptly fail past a count). debugging-with-system-tables covered MEMORY_LIMIT_EXCEEDED — a different named exception, a different timing pattern (fails immediately, on the query that caused it, not gradually). The skill this lesson adds isn't a ninth failure to memorize. It's the move that happens before any of those specific diagnoses: reading the signature first, so you know which lesson's playbook even applies.

system.errors is the fastest read that exists for this, and most people never open it. It aggregates, server-wide, every named exception ClickHouse has thrown since startup — one row per error code that has fired at least once, with a running count (value), the timestamp and text of the most recent occurrence (last_error_time, last_error_message), and whether it came from a local query or a remote one in a distributed query (remote). Before writing a single hypothesis-specific query against system.mutations, system.merges, or system.replicas, one query against system.errors — filtered to the last hour, sorted by count — tells you what kind of thing has actually been failing, by name, without guessing:

SELECT name, code, value, last_error_time, last_error_message
FROM system.errors
WHERE last_error_time > now() - INTERVAL 1 HOUR
ORDER BY value DESC;

That single query is doing the work three separate lessons' worth of hypothesis-forming used to require doing by hand: if TOO_MANY_PARTS shows up, you're in mergetree-on-disk-format and codebase-tour-clickhouse-source territory. If MEMORY_LIMIT_EXCEEDED shows up, you're in debugging-with-system-tables territory. If TABLE_IS_READ_ONLY or KEEPER_EXCEPTION shows up — names this course hasn't used yet — you're in new territory this lesson covers directly.

The error code number, not the error text, is the durable identifier. Error text can be reworded across ClickHouse versions; a blog post from three years ago describing "Table is in readonly mode" might not match this server's exact wording. The number doesn't move. system.errors' code column and every Code: N. DB::Exception: ... line ClickHouse ever prints both refer back to one canonical source of truth: a single registry file mapping every code to one name, checked directly against the current master branch for this lesson.

One category this course has only discussed conceptually is replication — and its signatures don't look like anything covered so far. replicated-engines, back in Module 5, built the trade-off case for ReplicatedMergeTree and named system.replicas as "Module 7 territory" without opening it. Replication failures don't throw one clean exception the way TOO_MANY_PARTS does — they show up as a state (a replica that's behind, or read-only, or has quietly lost data) that system.replicas reports directly, no exception required at all. That's the second half of this lesson's mental model: some failures announce themselves as an exception; others announce themselves only as a status column you have to know to check.

Codebase tour: where signatures actually come from

Debugging walkthrough: "one replica keeps failing writes and won't stay healthy"

Symptom. A three-replica ReplicatedMergeTree table backs a customer-facing dashboard. Analysts querying through a load balancer that round-robins across replicas notice the numbers occasionally look stale for a few seconds, then catch up — mildly annoying, but tolerable. Then a direct write attempt against one specific replica fails outright: Code: 242. DB::Exception: Table is in readonly mode. The other two replicas are accepting writes fine.

Step 1 — don't guess the category, check system.errors first. Before assuming this is a Keeper networking blip, a disk problem, or something else entirely, the fast global read this lesson opened with:

SELECT name, code, value, last_error_time, last_error_message
FROM system.errors
WHERE last_error_time > now() - INTERVAL 1 HOUR
ORDER BY value DESC
LIMIT 5;

┌─name───────────────┬─code─┬─value─┬────last_error_time─┬─last_error_message──────────────────┐
│ TABLE_IS_READ_ONLY  │  242 │    47 │  2026-07-13 09:41:02 │ Table is in readonly mode            │
│ KEEPER_EXCEPTION    │  999 │     6 │  2026-07-13 09:22:15 │ Coordination::Exception: Session ex… │
└─────────────────────┴──────┴───────┴─────────────────────┴───────────────────────────────────┘

Two things this rules in and rules out in one query: TABLE_IS_READ_ONLY (code 242) confirms the write failure genuinely is the replica's read-only state, not a one-off transient error, since it fired 47 times in the last hour. KEEPER_EXCEPTION (code 999), firing earlier and far less often, is a real clue about why — this replica had a Keeper coordination problem roughly twenty minutes before the read-only errors started piling up.

Step 2 — confirm the replica's own state directly, not just the error log. system.replicas, per ReplicatedMergeTreeRestartingThread's own responsibility for this state, is the direct source of truth:

SELECT database, table, replica_name, is_readonly, readonly_start_time,
       is_session_expired, absolute_delay, queue_size, active_replicas, total_replicas
FROM system.replicas
WHERE table = 'events';

┌─replica_name─┬─is_readonly─┬─readonly_start_time─┬─is_session_expired─┬─absolute_delay─┬─queue_size─┬─active_replicas─┬─total_replicas─┐
│ replica-1     │           0 │                 NULL │                  0 │              0 │          2 │               3 │              3 │
│ replica-2     │           0 │                 NULL │                  0 │              1 │          3 │               3 │              3 │
│ replica-3     │           1 │  2026-07-13 09:22:15 │                   1 │             62 │         14 │               2 │              3 │
└───────────────┴─────────────┴──────────────────────┴─────────────────────┴────────────────┴────────────┴─────────────────┴────────────────┘

replica-3 is confirmed is_readonly = 1, and readonly_start_time lines up almost exactly with the KEEPER_EXCEPTION timestamp from Step 1 — this is is_readonly's documented trigger condition in action: "turned on... during session reinitialization in ClickHouse Keeper." active_replicas = 2 on this row (versus 3 on the healthy replicas' rows) confirms the same story from the cluster's perspective: the other two replicas can also see that replica-3 has dropped out of the active set. absolute_delay = 62 seconds on replica-3 explains the analysts' "occasionally stale" complaint precisely — this is the replica intermittently serving reads through the load balancer, 62 seconds behind, while it's stuck.

Step 3 — is this self-healing, or genuinely stuck? ReplicatedMergeTreeRestartingThread's own doc comment already told you the design intent: it monitors the Keeper session and "if it expired, it will reinitialize it" — meaning a KEEPER_EXCEPTION followed by a brief is_readonly = 1 window that clears on its own, without intervention, is the expected, correctly-functioning behavior of a coordination-layer blip, not a bug. The way to tell "still recovering" from "genuinely stuck" is system.replication_queue, checked for this specific replica:

SELECT type, num_tries, last_exception, num_postponed, postpone_reason
FROM system.replication_queue
WHERE replica_name = 'replica-3'
ORDER BY num_tries DESC
LIMIT 3;

┌─type────────┬─num_tries─┬─last_exception──────────────────────────────────────────────┬─num_postponed─┬─postpone_reason─┐
│ GET_PART    │        36 │ DB::Exception: Directory .../detached/covered-by-broken_20260… │             0 │                 │
│ MERGE_PARTS │         2 │                                                                │             1 │ Not executing…  │
└─────────────┴───────────┴────────────────────────────────────────────────────────────────┴───────────────┴─────────────────┘

num_tries = 36 on a single GET_PART task, with a real, repeating last_exception, is the signature that distinguishes this from a simple session blip: this isn't "still reconnecting," it's failing the same specific operation over and over. The DIRECTORY_ALREADY_EXISTS-style conflict in that message ("directory already exists and is not empty") is a real, documented shape of this failure — a genuine, publicly filed ClickHouse issue shows an almost identical symptom: a replica stuck read-only after a restart, repeatedly logging attempts to detach the same part to a directory that already exists, cycling through _try1 through _try9 suffixes without ever resolving on its own, on a replica where the other replicas stayed healthy the whole time. That issue's own resolution path required manual intervention — detach, drop, re-attach, and restore — specifically because a genuinely conflicting on-disk directory isn't something ReplicatedMergeTreeRestartingThread's automatic session-reinitialization logic is designed to resolve; it resolves Keeper session state, not on-disk directory collisions.

Step 4 — apply the right remediation for which specific failure this is, not a generic restart. ClickHouse ships two different SYSTEM statements for two genuinely different replica problems, and picking the wrong one wastes an incident's worth of time: SYSTEM RESTART REPLICA "reinitialize[s the] Zookeeper session's state for ReplicatedMergeTree table, will compare current state with Zookeeper as source of truth and add tasks to Zookeeper queue if needed" — the right tool when Keeper is the source of truth and the replica's local queue just needs to resync against it. SYSTEM RESTORE REPLICA is a different, narrower tool: it "restores a replica if data is possibly present but Zookeeper metadata is lost" — the opposite direction, for when local data exists but Keeper's record of it doesn't, and it only operates on a replica already in read-only mode. Given a repeatedly-failing GET_PART with a real on-disk conflict — not a missing-metadata problem — RESTART REPLICA is the correct first attempt, since Keeper's queue is intact and the goal is getting replica-3 to re-attempt its pending tasks cleanly against that source of truth. But the cited issue above is a real caution against expecting that alone to resolve every case: its own reported resolution needed manual intervention (detach the conflicting part, drop it, re-attach, and restore) precisely because the conflict was a genuine collision already sitting on disk — the kind of state a session resync doesn't remove on its own. The working sequence this evidence supports: try RESTART REPLICA, then check system.replication_queue again a few cycles later; if the same task is still failing with the same last_exception, that repetition itself is the signal to stop retrying the same fix and move to manual removal of the conflicting on-disk state instead of escalating straight to RESTORE REPLICA, which addresses missing Keeper metadata, not an on-disk collision.

Failure mode gallery

Replication lag without read-only — a replica that's simply behind. Signature: is_readonly = 0, but absolute_delay is nonzero and, checked repeatedly, climbing rather than holding steady — "how big lag in seconds the current replica has." This is a fundamentally different failure from the walkthrough above: the replica is healthy and actively working, just not caught up. Root cause is almost always throughput, not correctness — and debugging-with-system-tables' central mental model applies here directly, not just by analogy: MergeTreeBackgroundExecutor, the same shared pool that lesson showed merges and mutations competing for, is documented as handling "merges, mutations, fetches and so on" — "fetches" is a replica pulling parts to catch up. A table under heavy insert or merge load on the same node can starve that same replica's fetch tasks of pool capacity, the identical shared-resource competition that lesson diagnosed for a stuck mutation, just showing up here as climbing absolute_delay instead. queue_size climbing alongside absolute_delay (rather than queue_size being small while absolute_delay is large) points specifically at "genuinely can't keep up," not "caught up but hasn't updated its delay figure yet."

A stuck read-only replica — this lesson's walkthrough. Signature: TABLE_IS_READ_ONLY (242) firing repeatedly in system.errors, is_readonly = 1 in system.replicas persisting well past a normal Keeper reconnection window, and — the detail that actually separates "stuck" from "recovering" — a system.replication_queue entry with a high num_tries and a real, repeating last_exception, rather than an empty queue quietly waiting for a session to reinitialize.

Permanently lost parts — a failure that never throws at query time at all. system.replicas' lost_part_count column is unlike almost everything else in this course's diagnostic toolkit: it isn't a live state that clears once things recover, it's "the number of data parts lost in the table by all replicas in total since table creation," a value "persisted in ClickHouse Keeper and can only increase." A nonzero, monotonically increasing lost_part_count is a genuine data-loss signal, not a transient symptom — and it's worth cross-referencing against mergetree-on-disk-format's CHECKSUM_DOESNT_MATCH failure mode, since a replica repeatedly losing parts to hardware-level corruption (that lesson's bit-flip cause) is exactly the kind of underlying condition that would show up here as a slowly climbing lost_part_count, independent of whether any single query ever surfaced a checksum error to a user.

Recognition index — every named failure this course has covered, by code. This isn't a fourth deep dive; it's the reference card this lesson exists to hand you, so a symptom's error code routes you to the right earlier lesson immediately instead of re-deriving the diagnosis from scratch:

A slow query with no memory-limit exception and a clean-looking EXPLAIN (plan) doesn't have an error-code signature at all — that's query-to-execution-plan's system.processors_profile_log territory, the reminder that not every failure this course covers throws a named exception; some only show up as a timing pattern in a table you have to know to check.

Hands-on forensics: the triage checklist

1. Query system.errors first, always, before forming any hypothesis. Filter to a recent time window, sort by count. This alone tells you which named exception, if any, has actually been firing.
2. If an error code appeared, look it up against the recognition index above (or, if it's not there, ErrorCodes.cpp directly) to find which earlier lesson's playbook applies.
3. If no error appeared but something still looks wrong (a stale dashboard, a slow-feeling query with no exception), the failure is a state, not an event — go directly to the system table that reports the relevant state: system.replicas for anything replication-shaped, system.parts for anything storage-shaped, system.processors_profile_log for anything execution-shaped.
4. For anything replication-shaped specifically, read system.replicas for three independent signals, not one: is_readonly (stuck?), absolute_delay (behind, but healthy?), lost_part_count (permanent loss?). These are three different failures with three different remediations — conflating them is the single easiest mistake to make at this depth.
5. Before restarting or restoring anything, confirm which specific remediation the evidence actually supports — RESTART REPLICA for a Keeper-state resync, RESTORE REPLICA only for missing Keeper metadata on an already-read-only replica — rather than reaching for whichever SYSTEM statement sounds most like "fix it."