> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Replication and Rebalancing

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

This page guides you through a simple demonstration of how CockroachDB replicates, distributes, and rebalances data. Starting with a 3-node local cluster, you'll write some data and verify that it replicates in triplicate by default. You'll then add 2 more nodes and watch how CockroachDB automatically rebalances replicas to efficiently use all available capacity.

## Before you begin

Make sure you have already <InternalLink path="install-cockroachdb">installed CockroachDB</InternalLink>.

## Step 1. Generate certificates

1. Create two directories:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ mkdir certs my-safe-directory
   ```
2. Create the CA (Certificate Authority) certificate and key pair:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach cert create-ca \
   --certs-dir=certs \
   --ca-key=my-safe-directory/ca.key
   ```
3. Create the certificate and key pair for your nodes:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach cert create-node \
   localhost \
   $(hostname) \
   --certs-dir=certs \
   --ca-key=my-safe-directory/ca.key
   ```

   Because you're running a local cluster and all nodes use the same hostname (`localhost`), you only need a single node certificate. Note that this is different than running a production cluster, where you would need to generate a certificate and key for each node, issued to all common names and IP addresses you might use to refer to the node as well as to any load balancer instances.
4. Create a client certificate and key pair for the `root` user:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach cert create-client \
   root \
   --certs-dir=certs \
   --ca-key=my-safe-directory/ca.key
   ```

## Step 2. Start a 3-node cluster

1. Use the <InternalLink path="cockroach-start">`cockroach start`</InternalLink> command to start node 1:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach start \
   --certs-dir=certs \
   --store=rep-node1 \
   --listen-addr=localhost:26257 \
   --http-addr=localhost:8080 \
   --join=localhost:26257,localhost:26258,localhost:26259
   ```
2. In a new terminal, start node 2:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach start \
   --certs-dir=certs \
   --store=rep-node2 \
   --listen-addr=localhost:26258 \
   --http-addr=localhost:8081 \
   --join=localhost:26257,localhost:26258,localhost:26259
   ```
3. In a new terminal, start node 3:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach start \
   --certs-dir=certs \
   --store=rep-node3 \
   --listen-addr=localhost:26259 \
   --http-addr=localhost:8082 \
   --join=localhost:26257,localhost:26258,localhost:26259
   ```
4. In a new terminal, use the <InternalLink path="cockroach-init">`cockroach init`</InternalLink> command to perform a one-time initialization of the cluster, sending the request to any node on the `--join` list:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach init --certs-dir=certs --host=localhost:26257
   ```

   You'll see the following message:

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   Cluster successfully initialized
   ```

   At this point, each node also prints helpful <InternalLink path="cockroach-start#standard-output">startup details</InternalLink> to its log.

## Step 3. Create a SQL user

You'll use a non-`root` user for running a client workload and accessing the DB Console.

1. In the same terminal, as the `root` user, open the <InternalLink path="cockroach-sql">built-in SQL shell</InternalLink> against any node:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach sql \
   --certs-dir=certs \
   --host=localhost:26257
   ```
2. Create the `maxroach` user with a password:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > CREATE USER maxroach WITH PASSWORD '<your password';
   ```
3. Assign the `maxroach` user to the `admin` role:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > GRANT admin TO maxroach;
   ```

   This role assignment is for convenience for this tutorial; it gives the user access to all data without the need for additional privileges. For more details, see <InternalLink path="security-reference/authorization#admin-role">Authorization</InternalLink>.
4. Exit the SQL shell:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > \q
   ```

## Step 4. Write data

1. In the same terminal, run the <InternalLink path="cockroach-workload">`cockroach workload`</InternalLink> command to generate an example `intro` database. In the connection string, replace `<password` with the password you created earlier for `maxroach`:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach workload init intro \
   'postgres://maxroach:<password@localhost:26257?sslmode=verify-full&sslrootcert=certs/ca.crt'
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   I200925 17:34:51.380259 1 workload/workloadsql/dataload.go:140  imported mytable (0s, 42 rows)
   ```
2. Re-open the SQL shell, this time as the `maxroach` user:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach sql \
   --user=maxroach \
   --certs-dir=certs \
   --host=localhost:26257
   ```

   Enter the user's password when prompted.
3. Verify that the new `intro` database was added with one table, `mytable`:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > SHOW DATABASES;
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
     database_name
   +---------------+
     defaultdb
     intro
     postgres
     system
   (4 rows)
   ```

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > SHOW TABLES FROM intro;
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
     table_name
   --------------
     mytable
   (1 row)
   ```

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > SELECT * FROM intro.mytable WHERE (l % 2) = 0;
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
     l  |                          v
   +----+------------------------------------------------------+
      0 | !__aaawwmqmqmwwwaas,,_        .__aaawwwmqmqmwwaaa,,
      2 | !"VT?!"""^~~^"""??T$Wmqaa,_auqmWBT?!"""^~~^^""??YV^
      4 | !                    "?##mW##?"-
      6 | !  C O N G R A T S  _am#Z??A#ma,           Y
      8 | !                 _ummY"    "9#ma,       A
     10 | !                vm#Z(        )Xmms    Y
     12 | !              .j####mmm#####mm#m##6.
     14 | !   W O W !    jmm###mm######m#mmm##6
     16 | !             ]#me*Xm#m#mm##m#m##SX##c
     18 | !             dm#||+*$##m#mm#m#Svvn##m
     20 | !            :mmE=|+||S##m##m#1nvnnX##;     A
     22 | !            :m#h+|+++=Xmm#m#1nvnnvdmm;     M
     24 | ! Y           $#m>+|+|||##m#1nvnnnnmm#      A
     26 | !  O          ]##z+|+|+|3#mEnnnnvnd##f      Z
     28 | !   U  D       4##c|+|+|]m#kvnvnno##P       E
     30 | !       I       4#ma+|++]mmhvnnvq##P`       !
     32 | !        D I     ?$#q%+|dmmmvnnm##!
     34 | !           T     -4##wu#mm#pw##7'
     36 | !                   -?$##m####Y'
     38 | !             !!       "Y##Y"-
     40 | !
   (21 rows)
   ```
4. Exit the SQL shell:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > \q
   ```

## Step 5. Verify replication

1. To understand replication in CockroachDB, it's important to review a few concepts from the <InternalLink path="architecture/overview">architecture</InternalLink>:

| Concept     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Range**   | CockroachDB stores all user data (tables, indexes, etc.) and almost all system data in a giant sorted map of key-value pairs. This keyspace is divided into "ranges", contiguous chunks of the keyspace, so that every key can always be found in a single range.  From a SQL perspective, a table and its secondary indexes initially map to a single range, where each key-value pair in the range represents a single row in the table (also called the primary index because the table is sorted by the primary key) or a single row in a secondary index. As soon as that range reaches <InternalLink path="configure-replication-zones">the maximum range size</InternalLink>, it <InternalLink path="architecture/distribution-layer#range-splits">splits into two ranges</InternalLink>. This process continues for these new ranges as the table and its indexes continue growing. |
| **Replica** | CockroachDB replicates each range (3 times by default) and stores each replica on a different node.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |

2. With those concepts in mind, open the DB Console at [http://localhost:8080](http://localhost:8080/) and log in with the `maxroach` user.
3. On the **Overview** page, note that the **Replicas** count is the same on all three nodes. This indicates:
   * There are this many "ranges" of data in the cluster. These are mostly internal "system" ranges since you haven't added much table data.
   * Each range has been replicated 3 times (according to the CockroachDB default).
   * For each range, each replica is stored on different nodes.

## Step 6. Add two more nodes

1. Back in the terminal, add a fourth node:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach start \
   --certs-dir=certs \
   --store=rep-node4 \
   --listen-addr=localhost:26260 \
   --http-addr=localhost:8083 \
   --join=localhost:26257,localhost:26258,localhost:26259
   ```
2. In a new terminal, add a fifth node:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach start \
   --certs-dir=certs \
   --store=rep-node5 \
   --listen-addr=localhost:26261 \
   --http-addr=localhost:8084 \
   --join=localhost:26257,localhost:26258,localhost:26259
   ```

## Step 7. Watch data rebalance

Back in the DB Console, you'll see that there are now 5 nodes listed.

At first, the replica count will be lower for nodes 4 and 5. Very soon, however, you'll see those numbers even out across all nodes, indicating that data is being automatically rebalanced to utilize the additional capacity of the new nodes.

## Step 8. Stop the cluster

1. When you're done with your test cluster, press `ctrl-c` in each terminal where a node is running.

<Note>
  For the last 2 nodes, the shutdown process will take longer (about a minute each) and will eventually force the nodes to stop. This is because, with only 2 of 5 nodes left, a majority of replicas are not available, and so the cluster is no longer operational. To speed up this process, you can press `ctrl-c` a second time.
</Note>

2. To restart the cluster at a later time, run the same `cockroach start` commands as earlier from the directory containing the nodes' data stores.

   If you do not plan to restart the cluster, you may want to remove the cluster's certificates and data stores:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ rm -rf certs my-safe-directory rep-node1 rep-node2 rep-node3 rep-node4 rep-node5
   ```

## What's next?

Explore other CockroachDB benefits and features:

* <InternalLink path="demo-replication-and-rebalancing">Replication & Rebalancing</InternalLink>
* <InternalLink path="demo-cockroachdb-resilience">CockroachDB Resilience</InternalLink>
* <InternalLink path="demo-serializable">Serializable Transactions</InternalLink>