1 Overview
2 Introduction to TestServer and TestCluster in Go unit tests
3 High-level TestServer API
4 Automatic cluster virtualization
- 4.1 Background - quick intro to cluster virtualization
- 4.2 TestServer and automatic cluster virtualization

Overview

TestServer and TestCluster are two frameworks we built to create Go unit tests for CockroachDB, for tests that need a functional SQL layer and/or KV layer.

TestServer simulates a single CockroachDB node.
TestCluster simulates a multi-node CockroachDB cluster. Each node is simulated using one TestServer.

By default, TestServer uses in-RAM storage (not persisted) to make tests faster.

For integration tests, consider using roachtest instead. See: the pages https://cockroachlabs.atlassian.net/wiki/spaces/CRDB/pages/3156574380 and Roachtest vs TestServer for details.

For other unit tests that do not need a full SQL or KV layer, consider a more narrow unit test that only instantiates the components it cares about. This will make the test run faster.

Here are the main differences between a regular CockroachDB node (e.g. one started via cockroach start) and one simulated via TestServer:

Component	Behavior in `cockroach start`	Behavior in TestServer

Component

Behavior in cockroach start

Behavior in TestServer

SQL, HTTP and KV layers

identical

Coordination glue around SQL/HTTP/KV to create an overall running server (topLevelServer in code)

identical

Server configuration

Via CLI flags, defaults useful for production deployments.

Via TestServerArgs struct; defaults useful for testing.

For example, KV storage is in-RAM by default (no persistence) to make tests faster. There is also a predefined TLS configuration with a self-signed CA.

Of note, cockroach demo uses TestServer under the hood so all the properties and limitations of TestServer apply to cockroach demo as well.

Introduction to TestServer and TestCluster in Go unit tests

Tests mostly use the following programming pattern:

func TestSomething(t *testing.T) {
  defer leaktest.AfterTest()()  // verify no goroutine leaks
  defer log.Scope(t).Close(t)   // capture test logs intelligently

  ctx := context.Background()
  srv := serverutils.StartServerOnly(t, base.TestServerArgs{})  // initialize and start a TestServer
  defer srv.Stopper().Stop(ctx) // ensure the TestServer gets cleaned up at the end of the test
  
  ts := srv.ApplicationLayer()  // see below for an explanation
  
  // ... use ts in test code ...

Alternatively, the following is also possible:

srv, db, kvDB := serverutils.StartServer(t, base.TestServerArgs{})

as it is equivalent to:

    srv := serverutils.StartServerOnly(t, base.TestServerArgs{})
    db := srv.ApplicationLayer().SQLConn(t, "")  // access to SQL
    kvDB := srv.ApplicationLayer().DB()          // access to KV

When a test needs to exercise a cluster of 2 or more nodes connected together, it can use TestCluster:

In a nutshell, the result of StartCluster (TestClusterInterface) has a Server(nodeIdx) method which returns a different TestServer for each node in the simulated cluster.

Note: there is no benefit to using StartCluster with just 1 node. Prefer StartServer in that case.

High-level TestServer API

Let’s inspect the interface of Serverutils.StartServerOnly():

The result of StartServerOnly is TestServerInterface, which contains the following two main methods: ApplicationLayer() and StorageLayer(). They refer to the following architectural diagram:

In short, ApplicationLayer() gives the test code a handle to the application layer of CockroachDB, containing the SQL and HTTP components; while StorageLayer() gives the test a handle to the storage layer with KV (replication, transactions) and lower level storage (Pebble).

In these two interfaces we see the following methods, for example:

Finally, a few methods are implemented by TestServerInterface directly by inheriting from TestServerController; for example Stopper() is part of TestServerController. They correspond to “orchestration-level” control of the server.

Automatic cluster virtualization

Background - quick intro to cluster virtualization

CockroachDB has supported cluster virtualization since v20.x, as the underlying technology to CC Serverless. Cluster virtualization corresponds to the virtualization of the application layer (as per the diagram above). It’s possible for the SQL and HTTP components to be fully encapsulated into a “virtual cluster” such that multiple virtual clusters can exist side-by-side on top of the same storage layer.

This corresponds to the following diagram (theoretical):

In this diagram, the “System” box is a special SQL interface to the storage layer which exists outside of virtualization; it corresponds to the “control layer” of the virtualization system and can be used to set parameter across all virtual clusters.

TestServer and automatic cluster virtualization

By default, TestServer automatically randomizes the server architecture between one of the following choices:

That is, a virtual cluster is started probabilistically inside the TestServer and the TestServer’s .ApplicationLayer() accessor is configured to point to it.

Tests should be implemented to primarily access SQL/HTTP via the ApplicationLayerInterface returned by .ApplicationLayer() to automatically get coverage with and without cluster virtualization.

Meanwhile, regardless of the randomization, the following invariants are always true:

.StorageLayer() always points to the storage layer inside TestServer.
.SystemLayer() always points to the special system interface inside TestServer (previously known as “system tenant” in previous versions of CockroachDB).