> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Backup and Restore Monitoring

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

CockroachDB includes metrics to monitor <InternalLink path="backup">backup</InternalLink>, <InternalLink path="restore">restore</InternalLink>, and <InternalLink path="create-schedule-for-backup">scheduled backup</InternalLink> jobs. You can use monitoring integrations to alert when there are anomalies, such as backups that have failed or restore jobs encountering a retryable error.

You can access the [Prometheus Endpoint](#prometheus-endpoint) to track and alert on backup and restore metrics. We recommend setting up monitoring to <InternalLink path="monitoring-and-alerting#alerting-tools">alert</InternalLink> when anomalies occur. You can then use the following SQL statements to inspect details relating to schedules, jobs, and backups:

* <InternalLink path="show-schedules">`SHOW SCHEDULES`</InternalLink>
* <InternalLink path="show-jobs">`SHOW JOBS`</InternalLink>
* <InternalLink path="show-backup">`SHOW BACKUP`</InternalLink>

<Note>
  Metrics are reported per node. Therefore, it is necessary to retrieve metrics from every node in the cluster. For example, if you are monitoring whether a backup fails, it is necessary to track `scheduled_backup_failed` on each node.
</Note>

## Prometheus endpoint

You can access the <InternalLink path="prometheus-endpoint">Prometheus endpoint</InternalLink> for backup and restore metrics.

Refer to the <InternalLink path="monitor-cockroachdb-with-prometheus">Monitor CockroachDB with Prometheus</InternalLink> tutorial for guidance on installing and setting up Prometheus and Alertmanager to track metrics.

### Available metrics

We recommend the following guidelines:

* Use the `schedules.BACKUP.last-completed-time` metric to monitor the specific backup job or jobs you would use to recover from a disaster.
* Configure alerting on the `schedules.BACKUP.last-completed-time` metric to watch for cases where the timestamp has not moved forward as expected.

| Metric                                        | Description                                                                                                                                                                                                                                                                                                                                     |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `schedules.BACKUP.failed`                     | The number of scheduled backup jobs that have failed. **Note:** A stuck scheduled job will not increment this metric.                                                                                                                                                                                                                           |
| `schedules.BACKUP.last-completed-time`        | The Unix timestamp of the most recently completed scheduled backup specified as maintaining this metric. **Note:** This metric only updates if the schedule was created with the <InternalLink path="create-schedule-for-backup#schedule-options">`updates\_cluster\_last\_backup\_time\_metric` option</InternalLink>.                         |
| `schedules.BACKUP.protected\_age\_sec`        | The age of the oldest <InternalLink path="create-schedule-for-backup#protected-timestamps-and-scheduled-backups">protected timestamp record</InternalLink> protected by backup schedules.                                                                                                                                                       |
| `schedules.BACKUP.protected\_record\_count`   | The number of <InternalLink path="create-schedule-for-backup#protected-timestamps-and-scheduled-backups">protected timestamp records</InternalLink> held by backup schedules.                                                                                                                                                                   |
| `schedules.BACKUP.started`                    | The number of scheduled backup jobs that have started.                                                                                                                                                                                                                                                                                          |
| `schedules.BACKUP.succeeded`                  | The number of scheduled backup jobs that have succeeded.                                                                                                                                                                                                                                                                                        |
| `schedules.round.reschedule\_skip`            | The number of schedules that were skipped due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the <InternalLink path="create-schedule-for-backup">`on\_previous\_running=skip`</InternalLink> schedule option.     |
| `schedules.round.reschedule\_wait`            | The number of schedules that were rescheduled due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the <InternalLink path="create-schedule-for-backup">`on\_previous\_running=wait`</InternalLink> schedule option. |
| `jobs.backup.currently\_paused`               | The number of backup jobs currently considered <InternalLink path="pause-job">paused</InternalLink>.                                                                                                                                                                                                                                            |
| `jobs.backup.currently\_running`              | The number of backup jobs currently running in `Resume` or `OnFailOrCancel` state.                                                                                                                                                                                                                                                              |
| `jobs.backup.fail\_or\_cancel\_retry\_error`  | The number of backup jobs that failed with a retryable error on their failure or cancelation process.                                                                                                                                                                                                                                           |
| `jobs.backup.fail\_or\_cancel\_completed`     | The number of backup jobs that successfully completed their failure or cancelation process.                                                                                                                                                                                                                                                     |
| `jobs.backup.fail\_or\_cancel\_failed`        | The number of backup jobs that failed with a non-retryable error on their failure or cancelation process.                                                                                                                                                                                                                                       |
| `jobs.backup.protected\_age\_sec`             | The age of the oldest <InternalLink path="create-schedule-for-backup#protected-timestamps-and-scheduled-backups">protected timestamp record</InternalLink> protected by backup jobs.                                                                                                                                                            |
| `jobs.backup.protected\_record\_count`        | The number of <InternalLink path="create-schedule-for-backup#protected-timestamps-and-scheduled-backups">protected timestamp records</InternalLink> held by backup jobs.                                                                                                                                                                        |
| `jobs.backup.resume\_failed`                  | The number of backup jobs that failed with a non-retryable error.                                                                                                                                                                                                                                                                               |
| `jobs.backup.resume\_retry\_error`            | The number of backup jobs that failed with a retryable error.                                                                                                                                                                                                                                                                                   |
| `jobs.restore.currently\_paused`              | The number of restore jobs currently considered <InternalLink path="pause-job">paused</InternalLink>.                                                                                                                                                                                                                                           |
| `jobs.restore.currently\_running`             | The number of restore jobs currently running in `Resume` or `OnFailOrCancel` state.                                                                                                                                                                                                                                                             |
| `jobs.restore.fail\_or\_cancel\_failed`       | The number of restore jobs that failed with a non-retriable error on their failure or cancelation process.                                                                                                                                                                                                                                      |
| `jobs.restore.fail\_or\_cancel\_retry\_error` | The number of restore jobs that failed with a retryable error on their failure or cancelation process.                                                                                                                                                                                                                                          |
| `jobs.restore.protected\_age\_sec`            | The age of the oldest <InternalLink path="architecture/storage-layer#protected-timestamps">protected timestamp record</InternalLink> protected by restore jobs.                                                                                                                                                                                 |
| `jobs.restore.protected\_record\_count`       | The number of <InternalLink path="architecture/storage-layer#protected-timestamps">protected timestamp records</InternalLink> held by restore jobs.                                                                                                                                                                                             |
| `jobs.restore.resume\_completed`              | The number of restore jobs that successfully resumed to completion.                                                                                                                                                                                                                                                                             |
| `jobs.restore.resume\_failed`                 | The number of restore jobs that failed with a non-retryable error.                                                                                                                                                                                                                                                                              |
| `jobs.restore.resume\_retry\_error`           | The number of restore jobs that failed with a retryable error.                                                                                                                                                                                                                                                                                  |

## See also

* <InternalLink path="third-party-monitoring-tools">Third-Party Monitoring Integrations</InternalLink>
* <InternalLink path="manage-a-backup-schedule">Manage a Backup Schedule</InternalLink>
* <InternalLink path="backup-and-restore-overview">Backup and Restore Overview</InternalLink>
* <InternalLink path="backup-validation">Backup Validation</InternalLink>