Skip to main content
CockroachDB includes metrics to monitor , , and jobs. You can use monitoring integrations to alert when there are anomalies, such as backups that have failed or restore jobs encountering a retryable error. We recommend setting up monitoring to alert when anomalies occur. Depending on the tier of your CockroachDB Cloud cluster, you can use the following tools to monitor backup and restore metrics:
CockroachDB StandardCockroachDB AdvancedCockroachDB Basic
✔✔✔
Prometheus✔✔
Datadog✔✔
You can then use the following SQL statements to inspect details relating to schedules, jobs, and backups:
For detail on that Cockroach Labs stores for your CockroachDB Cloud cluster, refer to the Backup and Restore page in the Cloud Console.
Metrics are reported per node. Therefore, it is necessary to retrieve metrics from every node in the cluster. For example, if you are monitoring whether a backup fails, it is necessary to track scheduled_backup_failed on each node.

​
Prometheus

This section outlines the available backup and restore job metrics with Prometheus. For instructions on accessing the metricexport endpoint for Prometheus, refer to:
  • .
  • .
We recommend the following guidelines:
  • Use the schedules.BACKUP.last-completed-time metric to monitor the specific backup job or jobs you would use to recover from a disaster.
  • Configure alerting on the schedules.BACKUP.last-completed-time metric to watch for cases where the timestamp has not moved forward as expected.
MetricDescription
schedules.BACKUP.failedThe number of scheduled backup jobs that have failed. Note: A stuck scheduled job will not increment this metric.
schedules.BACKUP.last-completed-timeThe Unix timestamp of the most recently completed scheduled backup specified as maintaining this metric. Note: This metric only updates if the schedule was created with the .
schedules.BACKUP.protected\_age\_secThe age of the oldest protected by backup schedules.
schedules.BACKUP.protected\_record\_countThe number of held by backup schedules.
schedules.BACKUP.startedThe number of scheduled backup jobs that have started.
schedules.BACKUP.succeededThe number of scheduled backup jobs that have succeeded.
schedules.round.reschedule\_skipThe number of schedules that were skipped due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the schedule option.
schedules.round.reschedule\_waitThe number of schedules that were rescheduled due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the schedule option.
jobs.backup.currently\_pausedThe number of backup jobs currently considered .
jobs.backup.currently\_runningThe number of backup jobs currently running in Resume or OnFailOrCancel state.
jobs.backup.fail\_or\_cancel\_retry\_errorThe number of backup jobs that failed with a retryable error on their failure or cancelation process.
jobs.backup.fail\_or\_cancel\_completedThe number of backup jobs that successfully completed their failure or cancelation process.
jobs.backup.fail\_or\_cancel\_failedThe number of backup jobs that failed with a non-retryable error on their failure or cancelation process.
jobs.backup.protected\_age\_secThe age of the oldest protected by backup jobs.
jobs.backup.protected\_record\_countThe number of held by backup jobs.
jobs.backup.resume\_failedThe number of backup jobs that failed with a non-retryable error.
jobs.backup.resume\_retry\_errorThe number of backup jobs that failed with a retryable error.
jobs.restore.currently\_pausedThe number of restore jobs currently considered .
jobs.restore.currently\_runningThe number of restore jobs currently running in Resume or OnFailOrCancel state.
jobs.restore.fail\_or\_cancel\_failedThe number of restore jobs that failed with a non-retriable error on their failure or cancelation process.
jobs.restore.fail\_or\_cancel\_retry\_errorThe number of restore jobs that failed with a retryable error on their failure or cancelation process.
jobs.restore.protected\_age\_secThe age of the oldest protected by restore jobs.
jobs.restore.protected\_record\_countThe number of held by restore jobs.
jobs.restore.resume\_completedThe number of restore jobs that successfully resumed to completion.
jobs.restore.resume\_failedThe number of restore jobs that failed with a non-retryable error.
jobs.restore.resume\_retry\_errorThe number of restore jobs that failed with a retryable error.

​
Datadog

You can export the following schedule backup metrics to Datadog using the , or access the Cloud Console Monitoring page to enable the integration. For instructions, refer to:
  • .
  • .

​
Available metrics in Datadog

MetricDescription
schedules.BACKUP.succeededThe number of scheduled backup jobs that have succeeded.
schedules.BACKUP.startedThe number of scheduled backup jobs that have started.
schedules.BACKUP.last\_completed\_timeThe Unix timestamp of the most recently completed backup by a schedule specified as maintaining this metric.
schedules.BACKUP.failedThe number of scheduled backup jobs that have failed.