> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Work with virtual clusters

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

Enable <InternalLink path="cluster-virtualization-overview">cluster virtualization</InternalLink> in your CockroachDB cluster to set up a <InternalLink path="physical-cluster-replication-overview">**physical cluster replication (PCR)**</InternalLink> stream. This page is a guide to working with virtual clusters.

## Connect to a virtual cluster

This section shows how to use [SQL clients](#sql-clients) or the [DB Console](#db-console) to connect to a virtual cluster.

When <InternalLink path="physical-cluster-replication-overview">PCR</InternalLink> is enabled, the virtual cluster is named `main`.

### SQL clients

This section shows how to connect using <InternalLink path="cockroach-sql">`cockroach sql`</InternalLink> when cluster virtualization is enabled.

Unless you specify which virtual cluster to connect to, when you connect using a SQL client, you are logged into the default virtual cluster. When <InternalLink path="physical-cluster-replication-overview">PCR</InternalLink> is enabled, the default virtual cluster is named `main`.

To connect to a specific virtual cluster, add the `GET` URL parameter `options=-ccluster={virtual_cluster_name}` to the connection URL. Replace <code>{'{virtual_cluster_name}'}</code> with the name of the virtual cluster. You must use `--url` rather than `--host`.

For example:

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach sql --url \
"postgresql://root@{node IP or hostname}:26257?options=-ccluster={virtual_cluster_name}&sslmode=verify-full" \
--certs-dir "certs"
```

Replace:

* <code>{'{node IP or hostname}'}</code>: the IP or hostname of a cluster node.
* <code>{'{virtual_cluster_name}'}</code>: the name of a virtual cluster.
* <code>{'{certs_dir}'}</code>: The directory containing the cluster's certificates.

#### Connect to the system virtual cluster

<Note>
  You should only connect to the system virtual cluster for cluster administration and to manage PCR. To work with databases, tables, or workloads, connect to a virtual cluster.
</Note>

To connect to the system virtual cluster, pass the `options=-ccluster=system` parameter in the URL. You must have the `admin` role on the system virtual cluster.

For example, to connect to the system virtual cluster using the `cockroach sql` command:

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach sql --url \
"postgresql://root@{node IP or hostname}:26257?options=-ccluster=system&sslmode=verify-full" \
--certs-dir "certs"
```

### DB Console

This section shows how to connect using the <InternalLink path="ui-overview">DB Console</InternalLink> when cluster virtualization is enabled.

Unless you specify which virtual cluster to connect to, when you connect using the DB Console, you are logged into the default virtual cluster. When <InternalLink path="physical-cluster-replication-overview">PCR</InternalLink> is enabled, the default virtual cluster is named `main`. In order to view metrics on <InternalLink path="physical-cluster-replication-technical-overview">replication lag</InternalLink> for a PCR job, connect to the standby cluster's DB Console.

To connect to a specific virtual cluster, add the `GET` URL parameter `options=-ccluster={virtual_cluster_name}` to the DB Console URL. Replace <code>{'{virtual_cluster_name}'}</code> with the name of the virtual cluster.

If the same SQL user has the `admin` role on the system virtual cluster and also has roles on other virtual clusters, that user can switch among them from the top of the DB Console.

#### Connect to the system virtual cluster

You should only connect to the system virtual cluster for cluster administration. To work with databases, tables, or workloads, connect to a virtual cluster.

To connect to the system virtual cluster using the DB Console, add the `GET` URL parameter `options=-ccluster=system` to the DB Console URL.

## Grant access to the system virtual cluster

To <InternalLink path="grant">grant</InternalLink> access to the system virtual cluster, you must connect to the system virtual cluster as a user with the `admin` role, then grant either of the following to the SQL user:

* The `admin` <InternalLink path="security-reference/authorization#admin-role">role</InternalLink> grants the ability to read and modify system tables and cluster settings on any virtual cluster, including the system virtual cluster.
* The `VIEWSYSTEMDATA` <InternalLink path="security-reference/authorization#supported-privileges">system privilege</InternalLink> grants the ability to read system tables and cluster settings on any virtual cluster, including the system virtual cluster.

To prevent unauthorized access, you should limit the users with access to the system virtual cluster.

## Observability

When cluster virtualization is enabled, cluster log entries and metrics are scoped to a virtual cluster or to the system virtual cluster.

### View cluster logs with cluster virtualization enabled

When cluster virtualization is enabled, each cluster log is labeled with the name of the virtual cluster that generated it, including the system virtual cluster. For example, this log message relates to a virtual cluster named `demo`:

```none theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
I230815 19:31:07.290757 922 sql/temporary_schema.go:554 ⋮ [T4,demo,n1] 148  found 0 temporary schemas
```

### Work with metrics with cluster virtualization enabled

When cluster virtualization is enabled, metrics are also scoped to a virtual cluster or to the system virtual cluster, and are labeled accordingly. All metrics are visible from the system virtual cluster, but metrics scoped to the system virtual cluster are not visible from a virtual cluster. Metrics related to SQL activity and jobs are visible only from a virtual cluster.

For example, in the output of the <InternalLink path="prometheus-endpoint">Prometheus endpoint</InternalLink> on a cluster with a virtual cluster named `demo`, the metric `sql_txn_commit_count` is shown separately for the `demo` virtual cluster and the system virtual cluster:

```none theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
sql_txn_commit_count{tenant="system"} 0
sql_txn_commit_count{tenant="demo"} 0
```

When connected to a virtual cluster from the DB Console, metrics which measure SQL and related activity show data scoped to the virtual cluster. All other metrics are collected system-wide and display the same data on all virtual clusters including the system virtual cluster.

## Disaster recovery

When cluster virtualization is enabled, <InternalLink path="backup">backup</InternalLink> and <InternalLink path="restore">restore</InternalLink> commands are scoped to the virtual cluster by default.

### Back up a virtual cluster

To back up a virtual cluster:

1. [Connect to the virtual cluster](#connect-to-a-virtual-cluster) you want to back up as a user with the `admin` role on the virtual cluster.
2. <InternalLink path="backup">Back up the cluster</InternalLink>. Only the virtual cluster's data and settings are included in the backup, and data and settings for other virtual clusters or for the system virtual cluster is omitted.

For details about restoring a backup of a virtual cluster, refer to [Restore a virtual cluster](#restore-a-virtual-cluster).

### Back up the entire cluster

To back up the entire CockroachDB cluster, including all virtual clusters and the system virtual cluster:

1. [Connect to the system virtual cluster](#connect-to-the-system-virtual-cluster) as a user with the `admin` role on the system virtual cluster.
2. <InternalLink path="backup">Back up the cluster</InternalLink>, and include the `INCLUDE_ALL_SECONDARY_TENANTS` flag in the `BACKUP` command. All virtual clusters and the system virtual cluster are included in the backup.

### Restore a virtual cluster

You can restore a backup of a virtual cluster to:

* The original virtual cluster on the original CockroachDB cluster.
* A different virtual cluster on the original CockroachDB cluster.
* A different virtual cluster on a different CockroachDB cluster with cluster virtualization enabled.

To restore only a virtual cluster:

1. [Connect to the destination virtual cluster](#connect-to-a-virtual-cluster) as a user with the `admin` role on the virtual cluster.
2. <InternalLink path="restore">Restore the cluster</InternalLink>. Only the virtual cluster's data and settings are restored.

### Restore the entire cluster

To restore the entire CockroachDB cluster, including all virtual clusters and the system virtual cluster:

1. [Connect to the destination system virtual cluster](#connect-to-the-system-virtual-cluster) as a user with the `admin` role on the system virtual cluster.
2. <InternalLink path="restore">Restore the cluster</InternalLink> from a backup that included the the `INCLUDE_ALL_SECONDARY_VIRTUAL_CLUSTERS` flag. All virtual clusters and the system virtual cluster are restored.

## Configure cluster settings

When <InternalLink path="cluster-virtualization-overview">cluster virtualization</InternalLink> is enabled, each <InternalLink path="cluster-settings">cluster setting</InternalLink> is scoped to either a virtual cluster or the system virtual cluster.

* When a cluster setting is scoped to a virtual cluster, it affects only the virtual cluster and not the system virtual cluster. To configure a cluster setting that is scoped to a virtual cluster, you must have the `admin` role on the virtual cluster, and you must connect to the virtual cluster before configuring the setting. The majority of cluster settings are scoped to a virtual cluster and are visible only when connected to it.
* When a cluster setting is scoped to the system virtual cluster, it affects the entire CockroachDB cluster. To configure a cluster setting that is scoped to the system virtual cluster, you must have the `admin` role on the system virtual cluster, and you must connect to the system virtual cluster before configuring the setting.
* When a cluster setting is system-visible, it can be set only from the system virtual cluster but can be queried from any virtual cluster. For example, a virtual cluster can query a system-visible cluster setting's value to help adapt to the CockroachDB cluster's configuration.

For more details, including the scope of each cluster setting, refer to <InternalLink path="cluster-virtualization-setting-scopes">Cluster Setting Scopes with Cluster Virtualization enabled</InternalLink>.

## Upgrade a cluster

To upgrade to a new major version when cluster virtualization is enabled, you must:

1. Replace the binary on each node and restart the node.
2. <InternalLink path="upgrade-cockroach-version#finalize-a-major-version-upgrade-manually">Finalize</InternalLink> the upgrade on the system virtual cluster to upgrade it (if auto-finalization is disabled) or roll back the upgrade if you decide not to finalize it. Until it is finalized, the cluster still operates in compatibility with the previous major version, and virtual clusters cannot be upgraded.
3. Finalize the upgrade on a virtual cluster to upgrade it, or roll back the upgrade if you decide not to finalize it. Until it is finalized, a virtual cluster still operates in compatibility with the previous major version, and some features may not be available on the virtual cluster.

This allows you to roll back an upgrade of the system virtual cluster without impacting schemas or data in virtual clusters. The system virtual cluster can be at most one <InternalLink version="releases" path="index#release-schedule">Regular release</InternalLink> ahead of virtual clusters. For example, a system virtual cluster on CockroachDB v24.3 can have virtual clusters on CockroachDB v24.1 (a Regular release) or v24.2 (an Innovation release).

<Note>
  Both the `auto_upgrade.enabled` and `preserve_downgrade_option` <InternalLink path="cluster-settings">cluster settings</InternalLink> are scoped to a virtual cluster. To prevent automatic finalization of an upgrade, you must do one of the following:

  * Set `auto_upgrade.enabled` to `false` before beginning an upgrade. This setting persists after an upgrade. To finalize the upgrade manually, refer to <InternalLink path="upgrade-cockroach-version">Upgrade a cluster</InternalLink>. This is the preferred way to disable auto-upgrade.
  * Set `preserve_downgrade_option` to the cluster's current major version. This setting is reset during upgrade finalization and must be set again before the next major-version upgrade.

  Either setting, if enabled, disables auto-finalization.
</Note>

To apply a patch-version upgrade, you must replace the binary on each node and restart the node. Finalization is not required.

## See also

* <InternalLink path="cluster-virtualization-overview">Cluster Virtualization Overview</InternalLink>
* <InternalLink path="cluster-virtualization-setting-scopes">Cluster Setting Scopes</InternalLink>
* <InternalLink path="cluster-virtualization-metric-scopes">Cluster Metric Scopes</InternalLink>
* <InternalLink path="physical-cluster-replication-overview">Physical Cluster Replication</InternalLink>