> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Changefeed Best Practices

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

This page describes best practices to consider when starting changefeeds on a CockroachDB cluster. We recommend referring to this information while planning your cluster's changefeeds and following the links in each of the sections for more details on a topic.

<Note>
  To help in planning your cluster's changefeeds on CockroachDB Cloud clusters, refer to the <InternalLink version="cockroachcloud" path="costs">Understand CockroachDB Cloud Costs</InternalLink> page for detail on how CDC is billed monthly based on usage.
</Note>

## Plan the number of watched tables for a single changefeed

When creating a changefeed, it's important to consider the number of changefeeds versus the number of tables to include in a single changefeed:

* Changefeeds each have their own memory overhead, so every running changefeed will increase total memory usage.
* Creating a single changefeed that will watch hundreds of tables can affect the performance of a changefeed by introducing coupling, where the performance of a target table affects the performance of the changefeed watching it. For example, any <InternalLink path="changefeed-messages#schema-changes">schema change</InternalLink> on any of the tables will affect the entire changefeed's performance.

To watch multiple tables, we recommend creating a changefeed with a comma-separated list of tables. However, we do **not** recommend creating a single changefeed for watching hundreds of tables.

Cockroach Labs recommends monitoring your changefeeds to track <InternalLink path="monitor-and-debug-changefeeds">retryable errors</InternalLink> and <InternalLink path="architecture/storage-layer#protected-timestamps">protected timestamp</InternalLink> usage. Refer to the <InternalLink path="monitor-and-debug-changefeeds">Monitor and Debug Changefeeds</InternalLink> page for more information.

## Maintain system resources and running changefeeds

When you are running more than 10 changefeeds on a cluster, it is important to monitor the <InternalLink path="ui-overload-dashboard">CPU usage</InternalLink>. A larger cluster will be able to run more changefeeds concurrently compared to a smaller cluster with more limited resources.

We recommend limiting the number of changefeeds per cluster to 80.

To maintain a high number of changefeeds in your cluster:

* Connect to different nodes to <InternalLink path="create-changefeed">create</InternalLink> each changefeed. The node on which you start the changefeed will become the *coordinator* node for the changefeed job. The coordinator node acts as an administrator: keeping track of all other nodes during job execution and the changefeed work as it completes. As a result, this node will use more resources for the changefeed job. For more detail, refer to <InternalLink path="how-does-a-changefeed-work">How does a changefeed work?</InternalLink>.
* Consider logically grouping the target tables into one changefeed. When a changefeed <InternalLink path="pause-job">pauses</InternalLink>, it will stop emitting messages for the target tables. Grouping tables of related data into a single changefeed may make sense for your workload. However, we do not recommend watching hundreds of tables in a single changefeed. For more detail on protecting data from garbage collection when a changefeed is paused, refer to <InternalLink path="protect-changefeed-data">Garbage collection and changefeeds</InternalLink>.

## Monitor changefeeds

We recommend starting the changefeed with the <InternalLink path="monitor-and-debug-changefeeds#using-changefeed-metrics-labels">`metrics_label` option</InternalLink>, which allows you to measure metrics per changefeed. Metrics label information is sent with time-series metrics to the Prometheus endpoint.

The key areas to monitor when running changefeeds:

* <InternalLink path="monitor-and-debug-changefeeds#recommended-changefeed-metrics-to-track">Retryable errors</InternalLink>: `changefeed.error_retries`
* <InternalLink path="monitor-and-debug-changefeeds#recommended-changefeed-metrics-to-track">Failures</InternalLink>: `changefeed.failures`
* [CPU usage](#maintain-system-resources-and-running-changefeeds) for more than 10 changefeeds: <InternalLink path="ui-overload-dashboard">Overload Dashboard</InternalLink>
* <InternalLink path="protect-changefeed-data">Protected timestamp and garbage collection</InternalLink>:
  * `jobs.changefeed.protected_age_sec`
  * `jobs.changefeed.currently_paused`
  * `jobs.changefeed.expired_pts_records`
  * `jobs.changefeed.protected_record_count`

## Manage changefeeds and schema changes

When a schema change is issued that causes a column backfill, it can result in a changefeed emitting <InternalLink path="changefeed-messages#schema-changes-with-column-backfill">duplicate messages</InternalLink> for an event. We recommend issuing schema changes **outside of explicit transactions**. For more details on schema changes and column backfill generally, refer to the <InternalLink path="online-schema-changes">Online Schema Changes</InternalLink> page.

You can also use the <InternalLink path="create-changefeed">`schema_change_events`</InternalLink> and <InternalLink path="create-changefeed">`schema_change_policy`</InternalLink> options to define a schema change type and an associated policy that will modify how the changefeed behaves under the schema change.

## Lock the schema on changefeed watched tables

Use the `schema_locked` <InternalLink path="with-storage-parameter">storage parameter</InternalLink> to indicate that a <InternalLink path="online-schema-changes">schema change</InternalLink> is not currently ongoing on a watched table. This allows the changefeed to take a fast path that avoids checking if there are schema changes that could require synchronization between <InternalLink path="how-does-a-changefeed-work">changefeed aggregators</InternalLink>. This helps to decrease the latency between a write committing to a table and it emitting to the <InternalLink path="changefeed-sinks">changefeed's sink</InternalLink>.

Enable `schema_locked` on the watched table with the <InternalLink path="alter-table">`ALTER TABLE`</InternalLink> statement:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ALTER TABLE watched_table SET (schema_locked = true);
```

CockroachDB attempts to automatically unset `schema_locked` before performing a schema change and reapply it when done. However, certain schema changes (such as `ALTER TABLE ... SET LOCALITY`) cannot automatically unset it. For these cases, you must manually unlock the table with `schema_locked = false`, complete the schema change, and then lock the table again with `schema_locked = true`. The changefeed will run as normal while `schema_locked` is unset, but it will not benefit from the performance optimization.

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ALTER TABLE watched_table SET (schema_locked = false);
```

## See also

For details on tuning changefeeds for throughput, durability, and improving latency, refer to the <InternalLink path="advanced-changefeed-configuration">Advanced Changefeed Configuration</InternalLink> page.