Upgrade to CockroachDB v23.1

​

Terminology

• A new major release is performed multiple times per year. The major version number indicates the year of release followed by the release number, starting with 1. For example, the latest major release is .
• Each major release is maintained across patch releases that contain improvements including performance or security enhancements and bug fixes. Each patch release increments the major version number with its corresponding patch number. For example, patch releases of use the format .x.
• All major and patch releases are suitable for production environments, and are therefore considered “production releases”. For example, the latest production release is .
• Prior to an upcoming major release, alpha, beta, and release candidate (RC) binaries are made available for users who need early access to a feature before it is available in a production release. These releases append the terms alpha, beta, or rc to the version number. These “testing releases” are not suitable for production environments and are not eligible for support or uptime SLA commitments. For more information, refer to the .

​

Step 1. Verify that you can upgrade

> SHOW CLUSTER SETTING version;

• Any earlier v23.1 release: to .
• A production release: .0 to .

​

Step 2. Prepare to upgrade

​

Ensure you have a valid license key

• The cluster will run without limitations for a specified grace period. During that time, alerts are displayed that the cluster needs a valid license key. For more information, refer to the .
• The cluster is at the end of the grace period if no before then.

SET CLUSTER SETTING diagnostics.reporting.enabled = true;

• The cluster will run without limitations for a 7-day grace period. During that time, alerts are displayed that the cluster needs to send telemetry.
• The cluster is if telemetry is not received before the end of the grace period.

​

Review breaking changes

​

Check load balancing

​

Check cluster health

• Under Node Status, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as SUSPECT or DEAD, identify why the nodes are offline and either restart them or decommission them before beginning your upgrade. If there are DEAD and non-decommissioned nodes in your cluster, it will not be possible to finalize the upgrade (either automatically or manually).
• Under Replication Status, make sure there are 0 under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it’s important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade.
• In the Node List:
  • Make sure all nodes are on the same version. If any nodes are behind, upgrade them to the cluster’s current version first, and then start this process over.
• In the Metrics dashboards:
  • Make sure , , and capacity are within acceptable values for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider to your cluster before beginning your upgrade.

​

Check decommissioned nodes

1. To check the status of decommissioned nodes, run the command:
   cockroach node status --decommission
   
   In the output, verify that the value of the membership field of each node is decommissioned. If any node’s membership value is decommissioning, that node is not fully decommissioned.
2. If any node is not fully decommissioned, try the following:
   1. First, reissue the decommission command. The second command typically succeeds within a few minutes.
   2. If the second decommission command does not succeed, recommission and then decommission it again. Before continuing the upgrade, the node must be marked as decommissioned.

​

Back up cluster

​

Pause changefeed jobs

PAUSE JOB {changefeed_job_ID};

PAUSE ALL CHANGEFEED JOBS;

​

Reset SQL statistics

SELECT crdb_internal.reset_sql_stats();

​

Step 3. Decide how the upgrade will be finalized

1. Upgrade to , if you haven’t already.
2. Start the shell against any node in the cluster.
3. Set the cluster.preserve_downgrade_option :
   SET CLUSTER SETTING cluster.preserve_downgrade_option = '';
   
   It is only possible to set this setting to the current cluster version.

​

Features that require upgrade finalization

• The statement allows you to create scheduled changefeeds.
• The .

​

Step 4. Perform the rolling upgrade

1. .
2. Visit and download the CockroachDB full binary for your architecture.
3. Extract the archive. In the following instructions, replace with the path to the extracted archive directory.
4. If you have a previous version of the cockroach binary in your $PATH, rename the outdated cockroach binary, and then move the new one into its place. If you get a permission error because the cockroach binary is located in a system directory, add sudo before each command. The binary will be owned by the effective user, which is root if you use sudo.
   i="$(which cockroach)"; mv "$i" "$i"_old
   
   cp -i {COCKROACHDB_DIR}/cockroach /usr/local/bin/cockroach
   
5. If a cluster has corrupt descriptors, a major-version upgrade cannot be finalized. In CockroachDB v23.1.6 and subsequent 23.1 versions, automatic descriptor repair is enabled and cannot be disabled. In CockroachDB v23.1.11 and above, automatic descriptor repair is available but disabled by default. To enable it in these versions, set the environment variable COCKROACH_RUN_FIRST_UPGRADE_PRECONDITION to true after installing the v23.1 binary but before restarting the cockroach process on the node. Monitor the for errors. If a descriptor cannot be repaired automatically, contact support for assistance completing the upgrade.
6. Start the node so that it can rejoin the cluster. Without a process manager like systemd, re-run the command that you used to start the node initially, for example:
   cockroach start \
       --certs-dir=certs \
       --advertise-addr={node address} \
       --join={node1 address},{node2 address},{node3 address}
   
   If you are using systemd as the process manager, run this command to start the node:
   systemctl start {systemd config filename}
   
   Re-run the command that you used to start the node initially, for example:
   cockroach start \
       --certs-dir=certs \
       --advertise-addr={node address} \
       --join={node1 address},{node2 address},{node3 address}
   
7. Verify the node has rejoined the cluster through its output to or through the .
8. If you use cockroach in your $PATH, you can remove the previous binary:
   rm /usr/local/bin/cockroach_old
   
   If you leave versioned binaries on your servers, you do not need to do anything.
9. After the node has rejoined the cluster, ensure that the node is ready to accept a SQL connection. Unless there are tens of thousands of ranges on the node, it’s usually sufficient to wait one minute. To be certain that the node is ready, run the following command:
   cockroach sql -e 'select 1'
   
   The command will automatically wait to complete until the node is ready.
10. Repeat these steps for the next node.

​

Step 5. Roll back the upgrade (optional)

1. Perform a rolling upgrade, as described in the previous section, but replace the upgraded cockroach binary on each node with the binary for the previous version.
2. Restart the cockroach process on the node and verify that it has rejoined the cluster before rolling back the upgrade on the next node.
3. After all nodes have been rolled back and rejoined the cluster, finalize the rollback in the same way as you would finalize an upgrade, as described in the next section.

​

Step 6. Finish the upgrade

1. If you disabled auto-finalization in step 3, monitor the stability and performance of your cluster for at least a day. If you decide to roll back the upgrade, repeat the rolling restart procedure with the previous binary. Otherwise, perform the following steps to re-enable upgrade finalization to complete the upgrade to v23.1. Cockroach Labs recommends that you either finalize or roll back a major-version upgrade within a relative short period of time; running in a partially-upgraded state is not recommended. A cluster that is not finalized on cannot be upgraded to v23.1 until the upgrade is finalized.
2. Once you are satisfied with the new version, run against any node in the cluster to open the SQL shell.
3. Re-enable auto-finalization:
   > RESET CLUSTER SETTING cluster.preserve_downgrade_option;
   
   A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command SHOW CLUSTER SETTING version will return “. You can monitor the process of the migration in the DB Console . Migration jobs have names in the format -{migration-id}. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs.

1. To confirm that finalization has completed, check the cluster version:
   > SHOW CLUSTER SETTING version;
   
   If the cluster continues to report that it is on the previous version, finalization has not completed. If auto-finalization is enabled but finalization has not completed, check for the existence of where decommission has stalled. In most cases, issuing the decommission command again resolves the issue. If you have trouble upgrading, contact Support.
2. If you paused jobs before starting the upgrade process, you can now the jobs:
   RESUME JOB {changefeed_job_ID};
   
   Or, resume all changefeed jobs:
   RESUME ALL CHANGEFEED JOBS;
   
   Check that changefeeds are running:
   SHOW CHANGEFEED JOBS;
   

​

Troubleshooting

​

See also