> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Backup and Restore Monitoring

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

CockroachDB includes metrics to monitor <InternalLink path="backup">backup</InternalLink>, <InternalLink path="restore">restore</InternalLink>, and <InternalLink path="create-schedule-for-backup">scheduled backup</InternalLink> jobs. You can use monitoring integrations to alert when there are anomalies, such as backups that have failed or restore jobs encountering a retryable error.

You can access the [Prometheus Endpoint](#prometheus-endpoint) to track and alert on backup and restore metrics. We recommend setting up monitoring to <InternalLink path="monitoring-and-alerting#alerting-tools">alert</InternalLink> when anomalies occur. You can then use the following SQL statements to inspect details relating to schedules, jobs, and backups:

* <InternalLink path="show-schedules">`SHOW SCHEDULES`</InternalLink>
* <InternalLink path="show-jobs">`SHOW JOBS`</InternalLink>
* <InternalLink path="show-backup">`SHOW BACKUP`</InternalLink>

<Note>
  Metrics are reported per node. Therefore, it is necessary to retrieve metrics from every node in the cluster. For example, if you are monitoring whether a backup fails, it is necessary to track `scheduled_backup_failed` on each node.
</Note>

## Prometheus endpoint

You can access the <InternalLink path="prometheus-endpoint">Prometheus endpoint</InternalLink> for backup and restore metrics.

Refer to the <InternalLink path="monitor-cockroachdb-with-prometheus">Monitor CockroachDB with Prometheus</InternalLink> tutorial for guidance on installing and setting up Prometheus and Alertmanager to track metrics.

### Available metrics

We recommend the following guidelines:

* Use the `schedules.BACKUP.last-completed-time` metric to monitor the specific backup job or jobs you would use to recover from a disaster.
* Configure alerting on the `schedules.BACKUP.last-completed-time` metric to watch for cases where the timestamp has not moved forward as expected.

| Metric                                    | Description                                                                                                                                                                                                                                                                                                                                   |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `schedules.BACKUP.failed`                 | The number of scheduled backup jobs that have failed. **Note:** A stuck scheduled job will not increment this metric.                                                                                                                                                                                                                         |
| `schedules.BACKUP.last-completed-time`    | The Unix timestamp of the most recently completed scheduled backup specified as maintaining this metric. **Note:** This metric only updates if the schedule was created with the <InternalLink path="create-schedule-for-backup#schedule-options">`updates_cluster_last_backup_time_metric` option</InternalLink>.                            |
| `schedules.BACKUP.protected_age_sec`      | The age of the oldest <InternalLink path="create-schedule-for-backup#protected-timestamps-and-scheduled-backups">protected timestamp record</InternalLink> protected by backup schedules.                                                                                                                                                     |
| `schedules.BACKUP.protected_record_count` | The number of <InternalLink path="create-schedule-for-backup#protected-timestamps-and-scheduled-backups">protected timestamp records</InternalLink> held by backup schedules.                                                                                                                                                                 |
| `schedules.BACKUP.started`                | The number of scheduled backup jobs that have started.                                                                                                                                                                                                                                                                                        |
| `schedules.BACKUP.succeeded`              | The number of scheduled backup jobs that have succeeded.                                                                                                                                                                                                                                                                                      |
| `schedules.round.reschedule_skip`         | The number of schedules that were skipped due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the <InternalLink path="create-schedule-for-backup">`on_previous_running=skip`</InternalLink> schedule option.     |
| `schedules.round.reschedule_wait`         | The number of schedules that were rescheduled due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the <InternalLink path="create-schedule-for-backup">`on_previous_running=wait`</InternalLink> schedule option. |
| `jobs.backup.currently_paused`            | The number of backup jobs currently considered <InternalLink path="pause-job">paused</InternalLink>.                                                                                                                                                                                                                                          |
| `jobs.backup.currently_running`           | The number of backup jobs currently running in `Resume` or `OnFailOrCancel` state.                                                                                                                                                                                                                                                            |
| `jobs.backup.fail_or_cancel_retry_error`  | The number of backup jobs that failed with a retryable error on their failure or cancelation process.                                                                                                                                                                                                                                         |
| `jobs.backup.fail_or_cancel_completed`    | The number of backup jobs that successfully completed their failure or cancelation process.                                                                                                                                                                                                                                                   |
| `jobs.backup.fail_or_cancel_failed`       | The number of backup jobs that failed with a non-retryable error on their failure or cancelation process.                                                                                                                                                                                                                                     |
| `jobs.backup.protected_age_sec`           | The age of the oldest <InternalLink path="create-schedule-for-backup#protected-timestamps-and-scheduled-backups">protected timestamp record</InternalLink> protected by backup jobs.                                                                                                                                                          |
| `jobs.backup.protected_record_count`      | The number of <InternalLink path="create-schedule-for-backup#protected-timestamps-and-scheduled-backups">protected timestamp records</InternalLink> held by backup jobs.                                                                                                                                                                      |
| `jobs.backup.resume_failed`               | The number of backup jobs that failed with a non-retryable error.                                                                                                                                                                                                                                                                             |
| `jobs.backup.resume_retry_error`          | The number of backup jobs that failed with a retryable error.                                                                                                                                                                                                                                                                                 |
| `jobs.restore.currently_paused`           | The number of restore jobs currently considered <InternalLink path="pause-job">paused</InternalLink>.                                                                                                                                                                                                                                         |
| `jobs.restore.currently_running`          | The number of restore jobs currently running in `Resume` or `OnFailOrCancel` state.                                                                                                                                                                                                                                                           |
| `jobs.restore.fail_or_cancel_failed`      | The number of restore jobs that failed with a non-retriable error on their failure or cancelation process.                                                                                                                                                                                                                                    |
| `jobs.restore.fail_or_cancel_retry_error` | The number of restore jobs that failed with a retryable error on their failure or cancelation process.                                                                                                                                                                                                                                        |
| `jobs.restore.protected_age_sec`          | The age of the oldest <InternalLink path="architecture/storage-layer#protected-timestamps">protected timestamp record</InternalLink> protected by restore jobs.                                                                                                                                                                               |
| `jobs.restore.protected_record_count`     | The number of <InternalLink path="architecture/storage-layer#protected-timestamps">protected timestamp records</InternalLink> held by restore jobs.                                                                                                                                                                                           |
| `jobs.restore.resume_completed`           | The number of restore jobs that successfully resumed to completion.                                                                                                                                                                                                                                                                           |
| `jobs.restore.resume_failed`              | The number of restore jobs that failed with a non-retryable error.                                                                                                                                                                                                                                                                            |
| `jobs.restore.resume_retry_error`         | The number of restore jobs that failed with a retryable error.                                                                                                                                                                                                                                                                                |

## See also

* <InternalLink path="third-party-monitoring-tools">Third-Party Monitoring Integrations</InternalLink>
* <InternalLink path="manage-a-backup-schedule">Manage a Backup Schedule</InternalLink>
* <InternalLink path="backup-and-restore-overview">Backup and Restore Overview</InternalLink>
* <InternalLink path="backup-validation">Backup Validation</InternalLink>