cockroach start

On this page

This page explains the cockroach start command, which you use to start a new multi-node cluster or add nodes to an existing cluster.

Synopsis

Start a node to be part of a new multi-node cluster:

$ cockroach start <flags, including --join>

Initialize a new multi-node cluster:

$ cockroach init <flags>

Add a node to an existing cluster:

$ cockroach start <flags, including --join>

View help:

$ cockroach start --help

Flags

The cockroach start command supports the following general-use, networking, security, and logging flags.

Many flags have useful defaults that can be overridden by specifying the flags explicitly. If you specify flags explicitly, however, be sure to do so each time the node is restarted, as they will not be remembered. The one exception is the --join flag, which is stored in a node's data directory. We still recommend specifying the --join flag every time, as this will allow nodes to rejoin the cluster even if their data directory was destroyed.

General

Networking

Security

Locality

The --locality flag accepts arbitrary key-value pairs that describe the location of the node. Locality should include a region key-value if you are using CockroachDB's Multi-region SQL capabilities.

Depending on your deployment you can also specify country, availability zone, etc. The key-value pairs should be ordered into locality tiers from most inclusive to least inclusive (e.g., region before availability zone as in region=eu-west-1,zone=eu-west-1a), and the keys and order of key-value pairs must be the same on all nodes. It's typically better to include more pairs than fewer.

• Specifying a region with a region tier is required in order to enable CockroachDB's multi-region capabilities.

• CockroachDB spreads the replicas of each piece of data across as diverse a set of localities as possible, with the order determining the priority. Locality can also be used to influence the location of data replicas in various ways using high-level multi-region SQL capabilities or low-level replication zones.

• When there is high latency between nodes (e.g., cross-availability zone deployments), CockroachDB uses locality to move range leases closer to the current workload, reducing network round trips and improving read performance, also known as "follow-the-workload". In a deployment across more than 3 availability zones, however, to ensure that all data benefits from "follow-the-workload", you must increase your replication factor to match the total number of availability zones.

• Locality is also a prerequisite for using the Multi-region SQL abstractions, table partitioning, and Node Map Enterprise features.

Example

The following shell commands use the --locality flag to start 9 nodes to run across 3 regions: us-east-1, us-west-1, and europe-west-1. Each region's nodes are further spread across different availability zones within that region.

Nodes in us-east-1:

cockroach start --locality=region=us-east-1,zone=us-east-1a # ... other required flags go here

cockroach start --locality=region=us-east-1,zone=us-east-1b # ... other required flags go here

cockroach start --locality=region=us-east-1,zone=us-east-1c # ... other required flags go here

Nodes in us-west-1:

cockroach start --locality=region=us-west-1,zone=us-west-1a # ... other required flags go here

cockroach start --locality=region=us-west-1,zone=us-west-1b # ... other required flags go here

cockroach start --locality=region=us-west-1,zone=us-west-1c # ... other required flags go here

Nodes in europe-west-1:

cockroach start --locality=region=europe-west-1,zone=europe-west-1a # ... other required flags go here

cockroach start --locality=region=europe-west-1,zone=europe-west-1b # ... other required flags go here

cockroach start --locality=region=europe-west-1,zone=europe-west-1c # ... other required flags go here

For another multi-region example, see Start a multi-region cluster.

For more information about how to use CockroachDB's multi-region capabilities, see the Multi-Region Capabilities Overview.

Storage

• Storage engine
• Store
• Write Ahead Log (WAL) Failover

Storage engine

The --storage-engine flag is used to choose the storage engine used by the node. Note that this setting applies to all stores on the node, including the temp store.

As of v21.1 and later, CockroachDB always uses the Pebble storage engine. As such, pebble is the default and only option for the --storage-engine flag.

Store

The --store flag allows you to specify details about a node's storage.

To start a node with multiple disks or SSDs, you can use either of these approaches:

• Configure the disks or SSDs as a single RAID volume, then pass the RAID volume to the --store flag when starting the cockroach process on the node.
• Provide a separate --store flag for each disk when starting the cockroach process on the node. For more details about stores, see Start a Node.

The --store flag supports the following fields. Note that commas are used to separate fields, and so are forbidden in all field values.

Write Ahead Log (WAL) Failover

On a CockroachDB node with multiple stores, you can mitigate some effects of disk stalls by configuring the node to failover each store's write-ahead log (WAL) to another store's data directory using the --wal-failover flag.

Failing over the WAL may allow some operations against a store to continue to complete despite temporary unavailability of the underlying storage. For example, if the node's primary store is stalled, and the node can't read from or write to it, the node can still write to the WAL on another store. This can give the node a chance to eventually catch up once the disk stall has been resolved.

When WAL failover is enabled, CockroachDB will take the the following actions:

• At node startup, each store is assigned another store to be its failover destination.
• CockroachDB will begin monitoring the latency of all WAL writes. If latency to the WAL exceeds the value of the cluster setting storage.wal_failover.unhealthy_op_threshold, the node will attempt to write WAL entries to a secondary store's volume.
• CockroachDB will update the store status endpoint at /_status/stores so you can monitor the store's status.

Enable WAL failover

To enable WAL failover, you must take one of the following actions:

• Pass --wal-failover=among-stores to cockroach start, or
• Set the environment variable COCKROACH_WAL_FAILOVER=among-stores before starting the node.

Writing log files to local disk using the default configuration can lead to cluster instability in the event of a disk stall. It's not enough to failover your WAL writes to another disk: you must also write your log files in such a way that the forward progress of your cluster is not stalled due to disk unavailability.

Therefore, if you enable WAL failover and log to local disks, you must also update your logging configuration as follows:

1. Disable audit logging. File-based audit logging cannot coexist with the WAL failover feature. File-based audit logging provides guarantees that every log message makes it to disk, or CockroachDB must be shut down. For this reason, resuming operations in the face of disk unavailability is not compatible with audit logging.
2. Enable asynchronous buffering of file-groups log sinks using the buffering configuration option. The buffering configuration can be applied to file-defaults or individual file-groups as needed. Note that enabling asynchronous buffering of file-groups log sinks is in preview.
3. Set max-staleness: 1s and flush-trigger-size: 256KiB.
4. When buffering is enabled, buffered-writes must be explicitly disabled as shown in the following example. This is necessary because buffered-writes does not provide true asynchronous disk access, but rather a small buffer. If the small buffer fills up, it can cause internal routines performing logging operations to hang. This will in turn cause internal routines doing other important work to hang, potentially affecting cluster stability.

5. The recommended logging configuration for using file-based logging with WAL failover is as follows:

     file-defaults:
       buffered-writes: false
       buffering:
         max-staleness: 1s
         flush-trigger-size: 256KiB
         max-buffer-size: 50MiB
   

As an alternative to logging to local disks, you can configure remote log sinks that are not correlated with the availability of your cluster's local disks. However, this will make troubleshooting using cockroach debug zip more difficult, since the output of that command will not include the (remotely stored) log files.

Disable WAL failover

To disable WAL failover, you must restart the node and either:

• Pass the --wal-failover=disabled flag to cockroach start, or
• Set the environment variable COCKROACH_WAL_FAILOVER=disabled before restarting the node.

Monitor WAL failover

You can monitor if WAL failover occurs using the following metrics:

• storage.wal.failover.secondary.duration: Cumulative time spent (in nanoseconds) writing to the secondary WAL directory. Only populated when WAL failover is configured.
• storage.wal.failover.primary.duration: Cumulative time spent (in nanoseconds) writing to the primary WAL directory. Only populated when WAL failover is configured.
• storage.wal.failover.switch.count: Count of the number of times WAL writing has switched from primary to secondary store, and vice versa.

The storage.wal.failover.secondary.duration is the primary metric to monitor. You should expect this metric to be 0 unless a WAL failover occurs. If a WAL failover occurs, you probably care about how long it remains non-zero because it provides an indication of the health of the primary store.

You can access these metrics via the following methods:

• The Custom Chart Debug Page in DB Console.
• By monitoring CockroachDB with Prometheus.

Logging

By default, cockroach start writes all messages to log files, and prints nothing to stderr. This includes events with INFO severity and higher. However, you can customize the logging behavior of this command by using the --log flag:

Defaults

See the default logging configuration.

Standard output

When you run cockroach start, some helpful details are printed to the standard output:

CockroachDB node starting at 
build:               CCL v24.2.5 @ 2024-11-14 00:00:00 (go1.12.6)
webui:               http://localhost:8080
sql:                 postgresql://root@localhost:26257?sslmode=disable
sql (JDBC):          jdbc:postgresql://localhost:26257/defaultdb?sslmode=disable&user=root
RPC client flags:    cockroach <client cmd> --host=localhost:26257 --insecure
logs:                /Users/<username>/node1/logs
temp dir:            /Users/<username>/node1/cockroach-temp242232154
external I/O path:   /Users/<username>/node1/extern
store[0]:            path=/Users/<username>/node1
status:              initialized new cluster
clusterID:           8a681a16-9623-4fc1-a537-77e9255daafd
nodeID:              1

Examples

Start a multi-node cluster

To start a multi-node cluster, run the cockroach start command for each node, setting the --join flag to the addresses of the initial nodes.

For a cluster in a single region, set 3-5 --join addresses. Each starting node will attempt to contact one of the join hosts. In case a join host cannot be reached, the node will try another address on the list until it can join the gossip network.

$ cockroach start \
--certs-dir=certs \
--advertise-addr=<node1 address> \
--join=<node1 address>,<node2 address>,<node3 address> \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--certs-dir=certs \
--advertise-addr=<node2 address> \
--join=<node1 address>,<node2 address>,<node3 address> \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--certs-dir=certs \
--advertise-addr=<node3 address> \
--join=<node1 address>,<node2 address>,<node3 address> \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--insecure \
--advertise-addr=<node1 address> \
--join=<node1 address>,<node2 address>,<node3 address> \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--insecure \
--advertise-addr=<node2 address> \
--join=<node1 address>,<node2 address>,<node3 address> \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--insecure \
--advertise-addr=<node3 address> \
--join=<node1 address>,<node2 address>,<node3 address> \
--cache=.25 \
--max-sql-memory=.25

Then run the cockroach init command against any node to perform a one-time cluster initialization:

$ cockroach init \
--certs-dir=certs \
--host=<address of any node>

$ cockroach init \
--insecure \
--host=<address of any node>

Start a multi-region cluster

In this example we will start a multi-node local cluster with a multi-region setup that uses the same regions (passed to the --locality flag) as the multi-region MovR demo application.

1. Start a node in the us-east1 region:

   cockroach start --locality=region=us-east1,zone=us-east-1a \
                     --insecure --store=/tmp/node0 \
                     --listen-addr=localhost:26257 \
                     --http-port=8888 \
                     --join=localhost:26257,localhost:26258,localhost:26259
   

2. Start a node in the us-west1 region:

   cockroach start --locality=region=us-west1,zone=us-west-1a \
                     --insecure \
                     --store=/tmp/node2 \
                     --listen-addr=localhost:26259 \
                     --http-port=8890 \
                     --join=localhost:26257,localhost:26258,localhost:26259
   

3. Start a node in the europe-west1 region:

   cockroach start --locality=region=europe-west1,zone=europe-west-1a \
                     --insecure \
                     --store=/tmp/node1 \
                     --listen-addr=localhost:26258 \
                     --http-port=8889 \
                     --join=localhost:26257,localhost:26258,localhost:26259
   

4. Initialize the cluster:

   cockroach init --insecure --host=localhost --port=26257
   

5. Connect to the cluster using cockroach sql:

   cockroach sql --host=localhost --port=26257 --insecure
   

6. Issue the SHOW REGIONS statement to verify that the list of regions is expected:

   SHOW REGIONS;
   

       region    | zones | database_names | primary_region_of
   ---------------+-------+----------------+--------------------
     europe-west1 | {}    | {}             | {}
     us-east1     | {}    | {}             | {}
     us-west1     | {}    | {}             | {}
   (3 rows)
   

For more information about running CockroachDB multi-region, see the Multi-region Capabilities Overview.

For a more advanced example showing how to run a simulated workload on a multi-region CockroachDB cluster on your local machine, see Low Latency Reads and Writes in a Multi-Region Cluster.

Start a multi-node cluster across private networks

Scenario:

• You have a cluster that spans GCE and AWS.
• The nodes on each cloud can reach each other on public addresses, but the private addresses aren't reachable from the other cloud.

Approach:

1. Start each node on GCE with --locality set to describe its location, --locality-advertise-addr set to advertise its private address to other nodes in on GCE, --advertise-addr set to advertise its public address to nodes on AWS, and --join set to the public addresses of 3-5 of the initial nodes:

   $ cockroach start \
   --certs-dir=certs \
   --locality=cloud=gce \
   --locality-advertise-addr=cloud=gce@<private address of node> \
   --advertise-addr=<public address of node> \
   --join=<public address of node1>,<public address of node2>,<public address of node3> \
   --cache=.25 \
   --max-sql-memory=.25
   

2. Start each node on AWS with --locality set to describe its location, --locality-advertise-addr set to advertise its private address to other nodes on AWS, --advertise-addr set to advertise its public address to nodes on GCE, and --join set to the public addresses of 3-5 of the initial nodes:

   $ cockroach start \
   --certs-dir=certs \
   --locality=cloud=aws \
   --locality-advertise-addr=cloud=aws@<private address of node> \
   --advertise-addr=<public address of node> \
   --join=<public address of node1>,<public address of node2>,<public address of node3> \
   --cache=.25 \
   --max-sql-memory=.25
   

3. Run the cockroach init command against any node to perform a one-time cluster initialization:

   $ cockroach init \
   --certs-dir=certs \
   --host=<address of any node>
   

Add a node to a cluster

To add a node to an existing cluster, run the cockroach start command, setting the --join flag to the same addresses you used when starting the cluster:

$ cockroach start \
--certs-dir=certs \
--advertise-addr=<node4 address> \
--join=<node1 address>,<node2 address>,<node3 address> \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--insecure \
--advertise-addr=<node4 address> \
--join=<node1 address>,<node2 address>,<node3 address> \
--cache=.25 \
--max-sql-memory=.25

Create a table with node locality information

Start a three-node cluster with locality information specified in the cockroach start commands:

$ cockroach start --insecure --port=26257 --http-port=26258 --store=cockroach-data/1 --cache=256MiB --locality=region=eu-west-1,cloud=aws,zone=eu-west-1a

$ cockroach start --insecure --port=26259 --http-port=26260 --store=cockroach-data/2 --cache=256MiB --join=localhost:26257 --locality=region=eu-west-1,cloud=aws,zone=eu-west-1b

$ cockroach start --insecure --port=26261 --http-port=26262 --store=cockroach-data/3 --cache=256MiB --join=localhost:26257 --locality=region=eu-west-1,cloud=aws,zone=eu-west-1c

You can use the crdb_internal.locality_value built-in function to return the current node's locality information from inside a SQL shell. The example below uses the output of crdb_internal.locality_value('zone') as the DEFAULT value to use for the zone column of new rows. Other available locality keys for the running three-node cluster include region and cloud.

$ cockroach sql --insecure

> CREATE TABLE charges (
  zone STRING NOT NULL DEFAULT crdb_internal.locality_value('zone'),
  id INT PRIMARY KEY NOT NULL
);

> INSERT INTO charges (id) VALUES (1);

> SELECT * FROM charges WHERE id = 1;

     zone    | id
+------------+----+
  eu-west-1a |  1
(1 row)

The zone column has the zone of the node on which the row was created.

In a separate terminal window, open a SQL shell to a different node on the cluster:

$ cockroach sql --insecure --port 26259

> INSERT INTO charges (id) VALUES (2);

> SELECT * FROM charges WHERE id = 2;

     zone    | id
+------------+----+
  eu-west-1b |  2
(1 row)

In a separate terminal window, open a SQL shell to the third node:

$ cockroach sql --insecure --port 26261

> INSERT INTO charges (id) VALUES (3);

> SELECT * FROM charges WHERE id = 3;

     zone    | id
+------------+----+
  eu-west-1c |  3
(1 row)

Start a cluster with separate RPC and SQL networks

Separating the network addresses used for intra-cluster RPC traffic and application SQL connections can provide an additional level of protection against security issues as a form of defense in depth. This separation is accomplished with a combination of the --sql-addr flag and firewall rules or other network-level access control (which must be maintained outside of CockroachDB).

For example, suppose you want to use port 26257 for SQL connections and 26258 for intra-cluster traffic. Set up firewall rules so that the CockroachDB nodes can reach each other on port 26258, but other machines cannot. Start the CockroachDB processes as follows:

$ cockroach start --sql-addr=:26257 --listen-addr=:26258 --join=node1:26258,node2:26258,node3:26258 --certs-dir=~/cockroach-certs

Note the use of port 26258 (the value for listen-addr, not sql-addr) in the --join flag. Also, if your environment requires the use of the --advertise-addr flag, you should probably also use the --advertise-sql-addr flag when using a separate SQL address.

Clusters using this configuration with client certificate authentication may also wish to use split client CA certificates.

See also

• Initialize a Cluster
• Manual Deployment
• Kubernetes Overview
• Local Deployment
• cockroach Commands Overview

Was this helpful?