Weâre partway through a migration of our build system to Bazel. Bazel is a modern build system with better performance characteristics and correctness guarantees than we currently have with
go build. Today, you can perform almost all day-to-day CRDB dev tasks with Bazel rather than with
make is deprecated and we will remove this option for building Cockroach at some point in the next release cycle.
For the few tasks that you can't/don't know how to do with Bazel, please ask a question or contribute a solution. The Bazel migration is actively in progress and we always accept contributions even for minor/QOL improvements.
NOTE: for specific debugging tips on:
- Build failures, see âHow to ensure your code builds with Bazelâ
- Test failures, see âHow to ensure your tests can run in the Bazel sandboxâ
Follow the directions on "Getting and building CockroachDB from source" or "Building from source on macOS" to get your development environment set up.
Note that you do need a full installation of XCode to build on macOS. (No, a command-line tools instance does not suffice.) If youâve already installed a command-line tools instance as part of setting up Homebrew or other tools, switch to the full XCode instance and accept the XCode license agreement with:
sudo xcode-select -s /Applications/Xcode.app && sudo xcodebuild -license accept
dev is a light wrapper around Bazel that is well-suited to specific CRDB development tasks. Properly, when we say
dev, weâre referring to the Go binary whose source lives in
cockroach/pkg/cmd/dev, side-by-side with the Cockroach source. At the top level of the repo there is a wrapper script also called
dev whose only job is to build
pkg/cmd/dev and then invoke it with the passed-in arguments. This means that you can always type
./dev to run the version of
dev at the commit youâve checked out, which is typically what you want to do.
dev is meant to supplement
bazel rather than replace it.
dev makes certain tasks easier than
bazel, but simple builds and tests are easy enough to run via
Before trying to build anything, run the following:
dev notices any issues with your system that might prevent you from being able to build, it will let you know and tell you how to fix it. Make sure to run
dev doctor before you ask for help just in case it catches something you didnât know about.
dev will take a while to run the first time you execute it since it needs to build itself. Be patient.
Start by building
cockroach-short as follows:
bazel build pkg/cmd/cockroach-short
./dev build short also works as an alias. Bazel will pretty-print build output to the terminal.
You can also build the full
cockroach binary which includes
bazel build pkg/cmd/cockroach --config=with_ui
./dev build (or equivalently,
./dev build cockroach) is a synonym for this.
dev build is a light wrapper for
bazel build that supports aliases for common targets (for example,
./dev build crlfmt instead of the harder-to-remember
bazel build @com_github_cockroachdb_crlfmt//:crlfmt).
dev also copies binaries out of the Bazel output directories for you into your workspace; for example,
bazel build pkg/cmd/cockroach-short puts the binary in
_bazel/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short, but if you
dev build short the binary will be staged at
./dev help build for more information about what you can do with
dev build. Note that you can pass additional arguments directly to
bazel build by adding them after
# build "verbosely", outputting all commands as they are run ./dev build short -- -s
dev lets you cross-compile with the
--cross option, as in:
./dev build --cross
--cross takes an optional argument which is the platform to cross-compile to:
dev will copy the built binaries into the
artifacts directory in this case. Note that cross-building requires Docker. Cross-compiling should work on M1 Macs, but this support is experimental, so report issues if you should observe any.
For more debugging tips on building with Bazel, see âHow to ensure your code builds with Bazelâ.
You can run all tests in a given package (for example,
bazel test pkg/sql:all
bazel test pkg/sql/... will instead run all tests under the
pkg/sql directory recursively, including all the tests under
pkg/sql/rowinfra among others.
If the test passes, youâll see a brief message indicating which tests were run and their status:
INFO: Elapsed time: 7.842s, Critical Path: 7.26s INFO: 48 processes: 3 internal, 45 darwin-sandbox. INFO: Build completed successfully, 48 total actions //pkg/sql/types:types_test PASSED in 0.7s
If the test doesnât pass, Bazel will print the location of the testâs log file:
INFO: Elapsed time: 8.763s, Critical Path: 7.94s INFO: 46 processes: 1 internal, 45 darwin-sandbox. INFO: Build completed, 1 test FAILED, 46 total actions //pkg/sql/types:types_test FAILED in 1.5s /private/var/tmp/_bazel_ricky/be70b24e7357091e16c49d70921b7985/execroot/cockroach/bazel-out/darwin-fastbuild/testlogs/pkg/sql/types/types_test/test.log
You can examine that file to see the complete logs for the test.
./dev test is provided as a shorthand for
bazel test with some additional conveniences. Run
./dev test with the name of one or more packages to execute all tests from those packages:
./dev test pkg/sql
dev test has a
--stress flag for running tests under
--race for running tests with the race detector enabled.
Next to the
test.log file produced by your test, you can find a
test.xml file. This file contains specific information on each test run and its status, as well as timing information.
-v argument to
dev test will result in more verbose logging as well as more detailed information written to the
test.xml. You can make this the default behavior on your machine by adding
test --test_env=GO_TEST_WRAP_TESTV=1 to your
dev test allows you to pass additional arguments directly to Bazel by putting them after
--: for example,
dev test pkg/sql/types -- --verbose_failures --sandbox_debug.
To get test results printed as tests are being run add
-v -- --test_output streamed to the test command. Note that this reduces test parallelism.
For more tips on debugging test failures, see âHow to ensure your tests can run in the Bazel sandboxâ
# Run acceptance tests ./dev acceptance # Run compose tests ./dev compose # Run benchmarks for pkg/sql/parser ./dev bench pkg/sql/parser # Generate code (run this before submitting your PR). ./dev generate # Run lints ./dev lint # logic tests! ./dev testlogic --files=$FILES --subtests=$SUBTESTS --config=$CONFIG # Open a container running the "bazelbuilder" image. Requires Docker. ./dev builder
This is a (non-exhaustive) 1-to-1 mapping of
dev commands to their
make equivalents. Feel free to add to this
dev script uses Bazel to build
pkg/cmd/dev before running unless another
dev binary with the same increasing integer ID has already been built. Generally
dev will invoke the
dev binary âas ofâ that commit, which should usually be the correct behavior. However, if the default behavior does not work for some reason, you can find all the built versions of
TODO(a UI developer): fill in gaps for UI dev workflow
1. Switch to a new branch
2. If your workflow involves an IDE, generate your protos
./dev gen protobuf
Your IDE relies on generated files for many tasks (e.g. code navigation, IntelliSense, debugging), and will complain unless you have re-generated those files.
If you need to re-generate all generated go files, use the slower
./dev gen go
If the above fails, run the slowest
./dev gen to update all of your generated files.
You may recall that with
make , this step was not necessary. If youâre curious why, see this slack thread.
3. Write some code!
If you add new files or imports, run
./dev gen bazel before compiling or running a test.
compilepkg: missing strict dependencies: is usually the indicator that
./dev gen bazel needs to be re-run.
to skip this step, see tip below on
Build the binary:
./dev build short
4. Run a test
On an IDE: your normal workflow should work if your generated files are up to date (See step 2).
From the command line:
./dev test [path/to/pkg] --filter [test_name]
5. Before opening/updating a PR:
./dev lint --short (maybe additionally
make lintshort as
dev's linter doesnât have 100% coverage yet)
Assert your workspace is clean by running
./dev gen bazel . If you modified other generated files, run the appropriate
./dev gen [file_type] command.
Bazel has a configuration file called .bazelrc. You can put a global configuration file at
~/.bazelrc or a per-repository file at
.bazelrc.user in the root of your
Tired of running
./dev gen bazel? Set the
ALWAYS_RUN_GAZELLE env-var to automatically run
./dev gen bazel before every
dev test or
dev build incantation. Note this does add a tiny delay â noticeable when iterating on small tests through
echo 'export ALWAYS_RUN_GAZELLE=1' >> ~/.zshrc
If you have
bazel will fail with an error like
ccache: error: Failed to create temporary file for /home/alyshanjahani/.ccache/tmp/message_li.stdout: Read-only file system. To avoid this you should get the
ccache links out of your
PATH manually (i.e. uninstall
ccache), and then you might need to do
bazel clean --expunge.
Alternatively, if you would like to use Bazel with
ccache, you can enable support for writing outside the sandbox by adding the following to your
- For MacOS/Darwin:
- For Linux:
If youâre using a different ccache directory (
ccache --get-config cache_dir) point to that instead.
You can always use
bazel directly instead of going through
dev but there are some things you might want to keep in mind:
You should still ask
dev doctor if your machine is up-to-snuff before you try to
bazel build. The checks it performs arenât
dev doctor also sets up a local cache for you.
dev prints out the (relevant) calls to
bazel it makes before it does so. You can therefore run
dev once just to learn how to ask Bazel to perform your build/test and then just directly call into
bazel on subsequent iterations.
When running tests under stress, race, orÂ
devÂ does the legwork to invoke with the necessary flags with bazel. This involves running under another binary (
stress), running with certain gotags (
race), or allowing certain paths outside the bazel sandbox to be written to (
testdata). Feel free to see the actualÂ
bazelÂ command invoked and tweak as necessary.
If you want to build with the UI, you must include the
--config with_ui argument to
bazel build. (
dev takes care of this for you if you are using it.)
If you want to build a test without running it, you must include the the
--config test argument to
bazel build. (
dev takes care of this for you if you are using it.)