Time interval selection
The time interval selector at the top of each tab allows you to filter the view for a predefined or custom time interval. Use the navigation buttons to move to the previous, next, or current time interval. When you select a time interval, the same interval is selected for all charts on the Metrics page.Changefeed Status
| Short Name | CockroachDB Metric Name | Description | Usage |
|---|---|---|---|
| Running | changefeed.running | Number of currently running changefeeds, including sinkless | This metric tracks the total number of all running changefeeds. |
| Paused | jobs.changefeed.currently\_paused | Number of changefeed jobs currently considered Paused | Monitor and alert on this metric to safeguard against an inadvertent operational error of leaving a changefeed job in a paused state for an extended period of time. Changefeed jobs should not be paused for a long time because the protected timestamp prevents garbage collection. |
| Failures | changefeed.failures | Total number of changefeed jobs which have failed | This metric tracks the permanent changefeed job failures that the jobs system will not try to restart. Any increase in this counter should be investigated. An alert on this metric is recommended. |
Retryable Errors
| Short Name | CockroachDB Metric Name | Description | Usage |
|---|---|---|---|
| Errors | changefeed.error\_retries | Total retryable errors encountered by all changefeeds | This metric tracks transient changefeed errors. Alert on “too many” errors, such as 50 retries in 15 minutes. For example, during a rolling upgrade this counter will increase because the changefeed jobs will restart following node restarts. There is an exponential backoff, up to 10 minutes. But if there is no rolling upgrade in process or other cluster maintenance, and the error rate is high, investigate the changefeed job. |
Emitted Messages
| Short Name | CockroachDB Metric Name | Description | Usage |
|---|---|---|---|
| Emitted messages | changefeed.emitted\_messages | Messages emitted by all feeds | This metric provides a useful context when assessing the state of changefeeds. This metric characterizes the rate of changes being streamed from the CockroachDB cluster. |
Emitted Bytes
| Short Name | CockroachDB Metric Name | Description | Usage |
|---|---|---|---|
| Emitted bytes | changefeed.emitted\_bytes | Bytes emitted by all feeds | This metric provides a useful context when assessing the state of changefeeds. This metric characterizes the throughput bytes being streamed from the CockroachDB cluster. |
Commit Latency
| Short Name | CockroachDB Metric Name | Description | Usage |
|---|---|---|---|
| P99, P90 | changefeed.commit\_latency | Event commit latency: a difference between event MVCC timestamp and the time it was acknowledged by the downstream sink. If the sink batches events, then the difference between the oldest event in the batch and acknowledgement is recorded; Excludes latency during backfill | This metric provides a useful context when assessing the state of changefeeds. This metric characterizes the end-to-end lag between a committed change and that change applied at the destination. |
Oldest Protected Timestamp
| Short Name | CockroachDB Metric Name | Description | Usage |
|---|---|---|---|
| Protected Timestamp Age | jobs.changefeed.protected\_age\_sec | The age of the oldest PTS record protected by changefeed jobs | Changefeeds use protected timestamps to protect the data from being garbage collected. Ensure the protected timestamp age does not significantly exceed the GC TTL zone configuration. Alert on this metric if the protected timestamp age is greater than 3 times the GC TTL. |