Chromium Waterfall

The project-visible grouping of LUCI builders and builder groups whose green, red, skipped, or failed states show Chromium’s continuous-integration health after and around landing.

The word waterfall is historical, but the signal is current. In Buildbot-era Chromium, contributors watched waterfall pages where builders were arranged in columns and builds flowed downward over time. LUCI replaced the old masters, and Milo became the web UI for reading builders, builds, groups, consoles, and logs. The name stayed because the work stayed: a contributor, sheriff, or downstream vendor still needs one surface that answers whether the shared tree is healthy, which builders are red, and what signal is pre-submit, post-submit, experimental, or performance-specific.

What It Is

The Chromium Waterfall is not one machine and not one test suite. It is the contributor-facing name for a set of LUCI builder groups whose build results represent the health of the project at several points in the landing and release pipeline.

A builder is a configured job definition: for example, a Linux builder that checks out a particular revision, compiles a target, runs a named test suite, and uploads results. A bot or worker is the machine or Swarming task that runs a builder’s work. Older Chromium prose often says bot where current infrastructure documentation says builder; the distinction matters when diagnosing a failure. A builder is the recipe and configuration. A bot is the execution slot.

A builder group is a named collection of related builders. LUCI Milo presents those groups in project pages and console views. A waterfall is a historical term for a grouping of builders or builder groups whose results are read together. Chromium’s infra glossary still names the Main Waterfall and Perf Waterfall because those groupings remain useful even though the underlying system is LUCI and Buildbucket rather than Buildbot.

Buildbucket is the scheduling and build-record service underneath LUCI. Milo is the UI that lets contributors search builders, inspect a specific build, read logs, and move from a red build to the failed step. Continuous Verifier (CV), often called the commit queue in contributor-facing workflow, drives the builders that gate a patch at landing time.

The signals these builders carry are related but not synonyms, and what separates them is timing. A try builder checks a proposed patch before it lands. A CQ builder checks that patch in the commit queue regime, the subset or mirror the commit queue’s submit path runs. A CI builder checks the project after changes have landed, or on a schedule. The waterfall is where those ongoing CI signals become visible to the humans and automation responsible for tree health.

Why It Matters

A green local test run, a green CQ pass, and a green waterfall answer different questions. Local tests answer whether the contributor’s checkout passed the tests they ran. CQ answers whether the selected pre-submit gate accepted a patch set at landing time. The waterfall answers whether the configured CI surface is still healthy as the tree continues to move.

Collapsing those states produces bad decisions. A contributor may say “CQ passed” after a post-submit Mac builder turns red; the statement is true and still irrelevant. CQ did not run that builder, or it did not see the failure under the same tree state. A downstream vendor may see a red builder and assume every product built from Chromium is broken; the red state may instead be an FYI builder, an experimental configuration, or a platform-specific suite not on the main tree-closing path. A sheriff may close the tree on a failure that a feature team treats as “only a test,” because the builder’s position in the waterfall gives it project-level blocking authority.

The waterfall is also where cost enters test promotion. The Chromium Chronicle guidance on adding tests to the waterfall describes a ladder: start in FYI CI to collect signal without closing the tree, promote to main CI when the test is stable enough to guard the project, and add CQ coverage only when pre-submit cost is justified. That ladder exists because every additional builder has infrastructure cost and contributor latency. The question is not “should this test run?” but “where should this signal live: FYI CI, main CI, CQ, or a specialized perf surface?”

For a CIO or Head of Engineering shipping a Chromium-based product, the waterfall is part of upstream risk assessment. A downstream team can ask whether a regression was caught before branch cut, whether a failed test belonged to main CI or optional coverage, and whether a vendor-specific patch would need new builder coverage to be safe upstream. Without the waterfall vocabulary, those questions become vague arguments about “CI.” With it, they become concrete questions about builder groups, promotion level, tree-closing authority, and pre-submit cost.

How to Recognize It

The visible surface is LUCI Milo at ci.chromium.org. A project page groups builders by project and builder group; a console view lays out builders across revisions; a build page opens one builder run with steps, logs, swarming tasks, test results, and links to related builds. The URL often encodes the group path, such as the Chromium project’s main console, and the UI lets the reader search by builder name when a Gerrit or sheriff message cites one.

The builder name carries operational clues. A name with try usually belongs to pre-submit or manual tryjob evaluation. A name under a main CI group is part of continuous tree health. A perf builder belongs to a performance measurement regime whose failure has to be read statistically, often alongside the chromeperf dashboard. An FYI builder is deliberately looser: it gathers signal while a new configuration, suite, or platform proves itself before it can close the tree.

The failure color is only the first clue. A red builder says some step failed; it doesn’t say whether the failure closes the tree, whether a sheriff will revert the culprit, or whether the suite is flaky. The next clues are the builder group, failed step, blame range, recent CLs, test history, and whether the same failure repeats across related builders. That is why Tree Sheriff and Perf Sheriff are roles, not dashboards. The dashboard reports the state; the role interprets it.

The old vocabulary still appears in live work. A Gerrit comment may say “the bot is red” even when the linked surface is a LUCI builder. A design doc may say “add this to the waterfall” when the work is really to define or modify Buildbucket configuration. A developer.chrome.com article may say “waterfall” to mean the CI promotion target for a test suite. In Chromium prose, the current system and historical name coexist; the reader has to translate without flattening them.

How It Plays Out

A contributor lands a patch after Commit Queue Gate accepts it. Two hours later, a main CI builder on a platform not covered by the selected CQ set turns red. The Tree Sheriff reads the Milo console, opens the failed build, checks the blame range, and identifies the landed patch as the likely culprit. The author is offline in Europe. The sheriff reverts, links the failed builder and test log, and the tree returns to green. CQ did its job, and the waterfall did a different job afterward.

A test owner wants to add a new browser test suite for a feature under active development. Adding it directly to CQ would slow every relevant CL and punish contributors while the suite is still flaky. The owner starts the suite on FYI CI, watches the waterfall for repeated failures, fixes flakes, and only then promotes it to main CI. CQ comes later, if the suite’s signal is important enough to justify pre-submit latency. The waterfall is the proving ground between “we can run this test” and “this test may block everyone.”

A downstream enterprise-browser team sees a red performance builder during a milestone they plan to consume. The red state is not a simple build break. It points to a perf builder whose dashboard alert needs bisection and owner routing. The team’s release lead checks whether Perf Sheriff bisected and filed the regression before branch cut, rather than assuming the red signal means the milestone is unusable. The waterfall is the entry point; the performance regression process is the interpretation layer.

Consequences

The waterfall makes Chromium’s shared health visible. Contributors can distinguish local, try, CQ, main CI, FYI, and perf signals instead of treating every green or red result as equivalent. Sheriffs can act quickly because they have a common surface for builder state, failed steps, blame ranges, and logs. Downstream consumers can audit whether upstream caught a class of failure before they absorb the code.

The vocabulary also prevents overclaiming. A green CQ pass doesn’t prove the main waterfall will stay green. A green main waterfall doesn’t prove every optional configuration is healthy. A red FYI builder doesn’t carry the same authority as a red main tree-closing builder. Each statement has a scope, and the scope is encoded in the builder group and promotion level.

The cost is cognitive and operational. Chromium’s builder surface is large, and names can reflect history rather than clean taxonomy. The same failure may appear as a LUCI builder, a Milo console tile, a Buildbucket record, a Swarming task, and a Gerrit-linked tryjob. New contributors can see all of those names before they know which one matters. The waterfall gives the project one shared view, but it doesn’t remove the judgment needed to decide whether a red tile is a culprit, a symptom, a flake, or an experimental signal.

Notes for Agent Context

When reading Chromium CI state, do not treat local tests, try builders, CQ builders, CI builders, FYI builders, and perf builders as interchangeable signals. Before calling a CL safe to land or safe to re-land, identify which builder group failed, whether the failure was pre-submit or post-submit, and whether the builder is main CI, CQ, FYI, optional try coverage, or performance-specific. Don’t recommend retrying or ignoring a red builder until the failed step, blame range, repeat history, and builder promotion level are known. When a test addition is proposed, start by asking where the signal belongs in the promotion ladder: FYI CI for signal gathering, main CI for tree health, or CQ only when the pre-submit cost is justified.

Related Articles

Sources

Chromium’s infra glossary defines the current vocabulary: builder, builder group, waterfall, Main Waterfall, Perf Waterfall, LUCI, Buildbucket, Milo, Swarming, and commit queue terminology. The Chromium new-builder guide explains the relationship among try builders, CI builders, CQ builders, optional try builders, and mirrored builder groups, including the tradeoff when try and CI coverage differ. The LUCI UI tour documents the contributor-facing project, builder, console, and build views that replaced Buildbot-era masters.

The contributor-facing article “Adding tests to the waterfall” in Chromium Chronicle explains FYI CI, main CI, CQ promotion, and why new tests should prove themselves before they block every CL. The historical Chromium Buildbot console tour preserves the older terms still used in contributor speech: bot, builder, build step, close-the-tree configuration, and waterfall. The generated cr-buildbucket.cfg file records the live configuration boundary where waterfall, CI, and tryserver ACLs and service accounts remain distinct.

Technical Drill-Down

• docs/infra/glossary.md (pinned a099554) — the vocabulary source for builder groups, waterfalls, LUCI, Milo, Buildbucket, CV, CQ, and Swarming.
• docs/infra/new_builder.md (pinned a099554) — the guide to adding or changing try, CI, and CQ builders; useful for understanding mirroring and optional try coverage.
• docs/tour_of_luci_ui.md (pinned a099554) — the UI map for project pages, builder pages, console views, build pages, and search.
• Chromium Chronicle #14, “Adding tests to the waterfall” — contributor-facing guidance for promoting tests from FYI CI to main CI and then, when justified, to CQ.
• Tour of the Chromium Buildbot — historical Buildbot-era vocabulary that still shapes how contributors talk about bots, builders, build steps, and waterfall pages.
• cr-buildbucket.cfg (pinned cb9c87f) — generated Buildbucket configuration showing distinct waterfall, CI, and tryserver permission and service-account regimes.