Reading the ClickHouse Source: A Guided Tour

The mental model: the codebase mirrors the query's journey

Open github.com/ClickHouse/ClickHouse and the top level of src/ looks intimidating — dozens of directories, no obvious reading order. The mental model that makes it navigable: almost every one of those directories corresponds to one stage a SQL query passes through, in order, from the moment it's typed into a client to the moment rows come back. If you can hold the stages of a query's journey in your head, you already know which directory a given question belongs in, before you've read a single file inside it.

The stages, at the level this lesson cares about:

1. A query arrives at the server. Something has to receive bytes off a socket and recognize "here's a query." (src/Server, programs/server)
2. The query text becomes a structured tree. SQL is a string; nothing downstream wants to work with a string. (src/Parsers)
3. Something decides what kind of query this is, and dispatches to code that knows how to run that kind. A SELECT is handled completely differently from an INSERT or an ALTER. (src/Interpreters)
4. For SELECT specifically, the query tree gets analyzed and turned into a plan — a tree of logical steps: read this table, filter these rows, aggregate, sort, limit. (src/Analyzer, src/Planner, src/Processors/QueryPlan)
5. The logical plan becomes a physical pipeline — a graph of small, composable units that actually pull, transform, and push blocks of data. (src/Processors, src/QueryPipeline)
6. Something runs that pipeline, scheduling its units across threads until data comes out the other end. (src/Processors/Executors)
7. Underneath all of it, something actually stores and retrieves the data — for this course, almost always a MergeTree table. (src/Storages, src/Storages/MergeTree)

Every question you'll ever have about ClickHouse's behavior is, underneath, a question about one of these seven stages. "Why is my INSERT slow" is stage 7. "Why did my JOIN run out of memory" is stage 5 (specifically, one particular kind of processing unit). "Why does EXPLAIN show this plan and not that one" is stage 4. Before opening any file, name the stage — it cuts the codebase from "500,000 lines" to "the one directory that actually matters for this question."

Debugging walkthrough: "why isn't my table's part count going down?"

This is one of the single most common real ClickHouse symptoms, and it's a good exercise precisely because the honest answer requires source, not just system tables — the system tables tell you what is happening, and only the source tells you why.

The symptom. INSERTs into an events table have been running fine for weeks. Over the last few days, INSERT latency has crept up, and one just failed outright with an error mentioning "Too many parts."

Step 1 — confirm what the system tables already show. This course's Module 2 and Module 3 already taught system.parts; it's the first stop, not new territory:

SELECT partition_id, count() AS active_parts
FROM system.parts
WHERE table = 'events' AND active
GROUP BY partition_id
ORDER BY active_parts DESC
LIMIT 5;

┌─partition_id─┬─active_parts─┐
│ 202607       │         1842 │
│ 202606       │           41 │
│ 202605       │            9 │
└──────────────┴──────────────┘

One partition — the current month — has an order of magnitude more active parts than the others. That's the anomaly, not "parts exist at all."

Step 2 — check whether merges are even running.

SELECT count() FROM system.merges WHERE table = 'events';

┌─count()─┐
│       3 │
└─────────┘

Merges are running — this rules out "the background merge process is entirely stuck," a different failure mode with a different source-level cause (see the failure gallery below). The question sharpens: merges are happening, so why isn't the part count actually falling?

Step 3 — check the actual thresholds against the actual count. MergeTreeSettings.cpp declares parts_to_delay_insert (default 1000) and parts_to_throw_insert (default 3000) directly in source, with the throw threshold's own doc comment stating plainly that past it, INSERT is interrupted with the "Too many parts" exception. At 1,842 active parts in one partition, this table is already past the delay threshold (inserts are being artificially slowed down as a backpressure mechanism) and closing in on the throw threshold. The system tables have now told you what: merges are running, but not fast enough to keep this partition's part count under control.

Step 4 — read the source to understand why not enough, not just not enough. This is the step a symptom-only debugging approach skips, and it's where SimpleMergeSelector's own doc comment (quoted in the codebase tour above) becomes the actual answer, not background color: "Older data parts should be merged less frequently than newer data parts. Larger data parts should be merged less frequently than smaller data parts." That's not a bug — it's the selector deliberately trading some part-count growth for lower total write amplification, on the assumption that fewer, larger merges are cheaper overall than always aggressively flattening everything. If this table's actual ingestion pattern is many small, frequent INSERTs (a busy events pipeline, say), the selector's own designed-in caution about merging large parts too often can genuinely lose a race against a high enough small-part arrival rate — which is a capacity and ingestion-pattern problem, not a broken merge process.

The diagnosis this reasoning chain actually supports: not "merges are broken" (system.merges already ruled that out) and not "the selector has a bug" (its logic is doing exactly what its own comment says it will do) — the honest read is that this partition's sustained small-insert rate has outgrown what the selector's write-amplification-conscious heuristic can flatten in time, which points toward either batching inserts more aggressively upstream (fewer, larger INSERTs — a pattern this course's loading-data lesson already recommended for unrelated reasons) or, if the workload genuinely can't batch further, revisiting parts_to_delay_insert/max_parts_in_total with real understanding of the trade-off being adjusted, not as a number copied from a forum post.

Failure mode gallery

"Too many parts" (DB::Exception, mentions parts_to_throw_insert). Signature: INSERT throughput degrades over hours-to-days, then a specific insert fails outright with an exception naming the setting. This is the walkthrough above, and it's real enough to have its own long-running GitHub issue thread. Root cause is almost never "ClickHouse is broken" — it's a mismatch between insert batching, partition key cardinality, and the merge selector's deliberately conservative heuristics, exactly the chain the walkthrough traced.

Memory limit (for query) exceeded on a query containing a JOIN. Signature: the query worked fine at smaller data volumes and started failing as one side of the join grew. Source-level cause: HashJoin (the file this tour's last stop pointed to) builds an in-memory hash table from one side of the join before streaming the other side past it — this course's glossary already documents that "the build side must fit comfortably in memory" for the default hash join, which is exactly the constraint this error is reporting a violation of. The fix isn't always "raise max_memory_usage" — GraceHashJoin and FullSortingMergeJoin exist specifically as non-memory-bound alternatives that spill to disk, which is worth knowing before reflexively raising a memory ceiling on a query whose actual data shape will keep growing.

Merges stop making progress on a specific partition or table, even though system.merges shows attempts. Signature: system.merges is genuinely empty (unlike the walkthrough above, where merges were running, just not fast enough) or repeatedly starts and aborts the same merge. Source-level cause to check: MergeTreeDataMergerMutator's part-selection path returns a typed failure — SelectMergeFailure::Reason, either CANNOT_SELECT or NOTHING_TO_MERGE — a real, if granular, distinction between "there's genuinely nothing eligible to merge right now" and "something is actively preventing an otherwise-eligible merge from being selected" (a stuck mutation, a TTL merge selector waiting on its own schedule, or a replication-related lock, depending on the specific deployment). This is a case where the enum's mere existence is diagnostically useful: a "merges aren't happening" symptom is at least two different source-level conditions, not one, and treating them identically wastes debugging time.

Hands-on forensics: the mental checklist

Given any "ClickHouse is doing something I don't understand" symptom, before opening a source file:

1. Name the stage (the seven-stage model above). Is this about a query arriving, being parsed, being planned, running, or about data at rest?
2. Find the interface, not the file. Almost every stage in this tour is one small interface (IInterpreter, IProcessor, IStorage, IJoin, IMergeSelector) with many implementations. Finding the interface tells you the shape of the answer before you've read a single implementation.
3. Read the header's doc comment before the source. Every file this tour cited had a doc comment that, alone, answered most of the "what is this for" question — IProcessor.h, MergeTreeDataMergerMutator.h, and SimpleMergeSelector.h all did real explanatory work before a single line of implementation. ClickHouse's internals documentation isn't in a separate docs site for this layer — it's the comments, and they're frequently better than anything you'd find by searching.
4. Cross-reference against a system table before trusting a source-level theory. The walkthrough above didn't jump straight to SimpleMergeSelector — it confirmed the shape of the symptom with system.parts and system.merges first. A hypothesis about internals that contradicts what the system tables actually show is a hypothesis to revise, not force onto the data.