Metrics are reported per node. Therefore, it is necessary to retrieve metrics from every node in the cluster. For example, if you are monitoring whether a backup fails, it is necessary to track
scheduled_backup_failed on each node.Prometheus endpoint
You can access the for backup and restore metrics. Refer to the tutorial for guidance on installing and setting up Prometheus and Alertmanager to track metrics.Available metrics
We recommend the following guidelines:- Use the
schedules.BACKUP.last-completed-timemetric to monitor the specific backup job or jobs you would use to recover from a disaster. - Configure alerting on the
schedules.BACKUP.last-completed-timemetric to watch for cases where the timestamp has not moved forward as expected.
| Metric | Description |
|---|---|
schedules.BACKUP.failed | The number of scheduled backup jobs that have failed. Note: A stuck scheduled job will not increment this metric. |
schedules.BACKUP.last-completed-time | The Unix timestamp of the most recently completed scheduled backup specified as maintaining this metric. Note: This metric only updates if the schedule was created with the . |
schedules.BACKUP.protected\_age\_sec | The age of the oldest protected by backup schedules. |
schedules.BACKUP.protected\_record\_count | The number of held by backup schedules. |
schedules.BACKUP.started | The number of scheduled backup jobs that have started. |
schedules.BACKUP.succeeded | The number of scheduled backup jobs that have succeeded. |
schedules.round.reschedule\_skip | The number of schedules that were skipped due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the schedule option. |
schedules.round.reschedule\_wait | The number of schedules that were rescheduled due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the schedule option. |
jobs.backup.currently\_paused | The number of backup jobs currently considered . |
jobs.backup.currently\_running | The number of backup jobs currently running in Resume or OnFailOrCancel state. |
jobs.backup.fail\_or\_cancel\_retry\_error | The number of backup jobs that failed with a retryable error on their failure or cancelation process. |
jobs.backup.fail\_or\_cancel\_completed | The number of backup jobs that successfully completed their failure or cancelation process. |
jobs.backup.fail\_or\_cancel\_failed | The number of backup jobs that failed with a non-retryable error on their failure or cancelation process. |
jobs.backup.protected\_age\_sec | The age of the oldest protected by backup jobs. |
jobs.backup.protected\_record\_count | The number of held by backup jobs. |
jobs.backup.resume\_failed | The number of backup jobs that failed with a non-retryable error. |
jobs.backup.resume\_retry\_error | The number of backup jobs that failed with a retryable error. |
jobs.restore.currently\_paused | The number of restore jobs currently considered . |
jobs.restore.currently\_running | The number of restore jobs currently running in Resume or OnFailOrCancel state. |
jobs.restore.fail\_or\_cancel\_failed | The number of restore jobs that failed with a non-retriable error on their failure or cancelation process. |
jobs.restore.fail\_or\_cancel\_retry\_error | The number of restore jobs that failed with a retryable error on their failure or cancelation process. |
jobs.restore.protected\_age\_sec | The age of the oldest protected by restore jobs. |
jobs.restore.protected\_record\_count | The number of held by restore jobs. |
jobs.restore.resume\_completed | The number of restore jobs that successfully resumed to completion. |
jobs.restore.resume\_failed | The number of restore jobs that failed with a non-retryable error. |
jobs.restore.resume\_retry\_error | The number of restore jobs that failed with a retryable error. |