> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Chart Debug Page

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

The **Custom Chart** debug page in the DB Console lets you create one or multiple custom charts showing any combination of [available metrics](#available-metrics).

The definition of the customized dashboard is encoded in the URL. To share the dashboard with someone, send them the URL. Like any other URL, it can be bookmarked, sit in a pinned tab in your browser, etc.

To view the Custom Chart page, <InternalLink path="ui-overview">access the DB Console</InternalLink>, click **Advanced Debug** In the left-hand navigation bar, and in the **Reports** section select **Custom Time Series Chart**.

## Use the Custom Chart page

<img src="https://mintcdn.com/cockroachlabs/OOen9WAZoOzmgNIZ/images/v26.3/ui-custom-chart-debug-00.png?fit=max&auto=format&n=OOen9WAZoOzmgNIZ&q=85&s=fd5e03bf99f309500be9c0b9ea2aa189" alt="DB Console" width="2500" height="1356" data-path="images/v26.3/ui-custom-chart-debug-00.png" />

On the **Custom Chart** page, you can set the time span for all charts, add new custom charts, and customize each chart:

* To set the time span for the page, use the dropdown menu above the charts and select the desired time span. In addition, once you have selected a metric to display, you can drag within the chart itself to a set a custom time range.

* To add a chart, click **Add Chart** and customize the new chart.

* To customize each chart, use the **Units** dropdown menu to set the units to display. Then use the table below the chart to select the metrics being queried, and how they'll be combined and displayed. Options include:

<table><thead><tr><th> Column </th><th> Description </th></tr></thead><tbody><tr><td> Metric Name </td><td> How the system refers to this metric, e.g., <code>sql.bytesin</code>. </td></tr><tr><td> Downsampler </td><td><p> The "Downsampler" operation is used to combine the individual datapoints over the longer period into a single datapoint. We store one data point every ten seconds, but for queries over long time spans the backend lowers the resolution of the returned data, perhaps only returning one data point for every minute, five minutes, or even an entire hour in the case of the 30 day view. </p><p> Options: <ul><li><strong>AVG</strong>: Returns the average value over the time period.</li><li><strong>MIN</strong>: Returns the lowest value seen.</li><li><strong>MAX</strong>: Returns the highest value seen.</li><li><strong>SUM</strong>: Returns the sum of all values seen.</li></ul></p></td></tr><tr><td> Aggregator </td><td><p> Used to combine data points from different nodes. It has the same operations available as the Downsampler. </p><p> Options: <ul><li><strong>AVG</strong>: Returns the average value over the time period.</li><li><strong>MIN</strong>: Returns the lowest value seen.</li><li><strong>MAX</strong>: Returns the highest value seen.</li><li><strong>SUM</strong>: Returns the sum of all values seen.</li></ul></p></td></tr><tr><td> Rate </td><td><p> Determines how to display the rate of change during the selected time period. </p><p> Options: <ul><li><strong>Normal</strong>: Returns the actual recorded value. </li><li><strong>Rate</strong>: Returns the rate of change of the value per second. </li><li><strong>Non-negative Rate</strong>: Returns the rate-of-change, but returns 0 instead of negative values. A large number of the stats we track are actually tracked as monotonically increasing counters so each sample is just the total value of that counter. The rate of change of that counter represents the rate of events being counted, which is usually what you want to graph. "Non-negative Rate" is needed because the counters are stored in memory, and thus if a node resets it goes back to zero (whereas normally they only increase). </li></ul></p></td></tr><tr><td> Source </td><td> The set of nodes being queried, which is either: <ul><li> The entire cluster. </li><li> A single, named node. </li></ul></td></tr><tr><td> Per Node </td><td> If checked, the chart will show a line for each node's value of this metric. </td></tr></tbody></table>

## Examples

### Query user and system CPU usage

<img src="https://mintcdn.com/cockroachlabs/OOen9WAZoOzmgNIZ/images/v26.3/ui-custom-chart-debug-01.png?fit=max&auto=format&n=OOen9WAZoOzmgNIZ&q=85&s=15fa7cf5f94ed1d0aa62f29b05144542" alt="DB Console" width="2268" height="1478" data-path="images/v26.3/ui-custom-chart-debug-01.png" />

To compare system vs. userspace CPU usage, select the following values under **Metric Name**:

* `sys.cpu.sys.percent`
* `sys.cpu.user.percent`

The Y-axis label is the **Count**. A count of 1 represents 100% utilization. The **Aggregator** of **Sum** can show the count to be above 1, which would mean CPU utilization is greater than 100%.

Checking **Per Node** displays statistics for each node, which could show whether an individual node's CPU usage was higher or lower than the average.

## Essential Metrics to Monitor

For important metrics to visualize in a custom dashboard, refer to:

* <InternalLink path="essential-metrics-self-hosted">Essential Metrics for CockroachDB self-hosted Deployments</InternalLink>
* <InternalLink path="essential-metrics-advanced">Essential Metrics for CockroachDB Advanced Deployments</InternalLink>

## Available metrics

<Note>
  Some of the metrics listed below are already visible in other areas of the <InternalLink path="ui-overview">DB Console</InternalLink>.
</Note>

<table className="crdb-fixed-table"><thead><tr><th style={{width: "26%"}}>CockroachDB Metric Name</th><th style={{width: "48%"}}>Description</th><th style={{width: "12%"}}>Type</th><th style={{width: "14%"}}>Unit</th></tr></thead><tbody> <tr><td><div id="addsstable.delay.total" class="anchored"><code>addsstable.delay.total</code></div></td><td>Amount by which evaluation of AddSSTable requests was delayed</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="addsstable.proposals" class="anchored"><code>addsstable.proposals</code></div></td><td>Number of SSTable ingestions proposed (i.e. sent to Raft by lease holders)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="admission.elastic_cpu.nanos_exhausted_duration" class="anchored"><code>admission.elastic\_cpu.nanos\_exhausted\_duration</code></div></td><td>Total duration when elastic CPU tokens (tokens measured in nanoseconds) were exhausted, as observed by the token granter (not waiters). This is reported in nanoseconds from 26.1 onwards, and was microseconds before that.</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="admission.granter.elastic_io_tokens_exhausted_duration.kv" class="anchored"><code>admission.granter.elastic\_io\_tokens\_exhausted\_duration.kv</code></div></td><td>Total duration when Elastic IO tokens were exhausted, as observed by the token granter (not waiters). This is reported in nanoseconds from 26.1 onwards, and was microseconds before that.</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="admission.granter.io_tokens_exhausted_duration.kv" class="anchored"><code>admission.granter.io\_tokens\_exhausted\_duration.kv</code></div></td><td>Total duration when IO tokens were exhausted, as observed by the token granter (not waiters). This is reported in nanoseconds from 26.1 onwards, and was microseconds before that.</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="admission.granter.slots_exhausted_duration.kv" class="anchored"><code>admission.granter.slots\_exhausted\_duration.kv</code></div></td><td>Total duration when KV slots were exhausted, as observed by the slot granter (not waiters). This is reported in nanoseconds from 26.1 onwards, and was microseconds before that.</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="admission.io.overload" class="anchored"><code>admission.io.overload</code></div></td><td>1-normalized float indicating whether IO admission control considers the store as overloaded with respect to compaction out of L0 (considers sub-level and file counts).</td><td>GAUGE</td><td>PERCENT</td></tr> <tr><td><div id="admission.wait_durations.cpu" class="anchored"><code>admission.wait\_durations.cpu</code></div></td><td>Wait time durations for requests that waited</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="admission.wait_durations.elastic-cpu" class="anchored"><code>admission.wait\_durations.elastic-cpu</code></div></td><td>Wait time durations for requests that waited</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="admission.wait_durations.elastic-stores" class="anchored"><code>admission.wait\_durations.elastic-stores</code></div></td><td>Wait time durations for requests that waited</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="admission.wait_durations.kv" class="anchored"><code>admission.wait\_durations.kv</code></div></td><td>Wait time durations for requests that waited</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="admission.wait_durations.kv-stores" class="anchored"><code>admission.wait\_durations.kv-stores</code></div></td><td>Wait time durations for requests that waited</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="admission.wait_durations.sql-kv-response" class="anchored"><code>admission.wait\_durations.sql-kv-response</code></div></td><td>Wait time durations for requests that waited</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="admission.wait_durations.sql-sql-response" class="anchored"><code>admission.wait\_durations.sql-sql-response</code></div></td><td>Wait time durations for requests that waited</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="auth.cert.conn.latency" class="anchored"><code>auth.cert.conn.latency</code></div></td><td>Latency to establish and authenticate a SQL connection using certificate</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="auth.cert.san.conn.success" class="anchored"><code>auth.cert.san.conn.success</code></div></td><td>Number of successful SQL connections using SAN-based certificate authentication</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="auth.cert.san.conn.total" class="anchored"><code>auth.cert.san.conn.total</code></div></td><td>Total number of SQL connection attempts using SAN-based certificate authentication</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="auth.gss.conn.latency" class="anchored"><code>auth.gss.conn.latency</code></div></td><td>Latency to establish and authenticate a SQL connection using GSS</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="auth.jwt.conn.latency" class="anchored"><code>auth.jwt.conn.latency</code></div></td><td>Latency to establish and authenticate a SQL connection using JWT Token</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="auth.ldap.conn.latency" class="anchored"><code>auth.ldap.conn.latency</code></div></td><td>Latency to establish and authenticate a SQL connection using LDAP</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="auth.ldap.conn.latency.internal" class="anchored"><code>auth.ldap.conn.latency.internal</code></div></td><td>Internal Auth Latency to establish and authenticate a SQL connection using LDAP(excludes external LDAP calls)</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="auth.password.conn.latency" class="anchored"><code>auth.password.conn.latency</code></div></td><td>Latency to establish and authenticate a SQL connection using password</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="auth.scram.conn.latency" class="anchored"><code>auth.scram.conn.latency</code></div></td><td>Latency to establish and authenticate a SQL connection using SCRAM</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="capacity" class="anchored"><code>capacity</code></div></td><td>Total storage capacity</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="capacity.available" class="anchored"><code>capacity.available</code></div></td><td>Available storage capacity</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="capacity.used" class="anchored"><code>capacity.used</code></div></td><td>Used storage capacity</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="changefeed.backfill_count" class="anchored"><code>changefeed.backfill\_count</code></div></td><td>Number of changefeeds currently executing backfill</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.commit_latency" class="anchored"><code>changefeed.commit\_latency</code></div></td><td>Event commit latency: a difference between event MVCC timestamp and the time it was acknowledged by the downstream sink. If the sink batches events, then the difference between the oldest event in the batch and acknowledgement is recorded. Excludes latency during backfill.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="changefeed.emitted_bytes" class="anchored"><code>changefeed.emitted\_bytes</code></div></td><td>Bytes emitted by all feeds</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="changefeed.emitted_messages" class="anchored"><code>changefeed.emitted\_messages</code></div></td><td>Messages emitted by all feeds</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="changefeed.error_retries" class="anchored"><code>changefeed.error\_retries</code></div></td><td>Total retryable errors encountered by all changefeeds</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="changefeed.failures" class="anchored"><code>changefeed.failures</code></div></td><td>Total number of changefeed jobs which have failed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="changefeed.max_behind_nanos" class="anchored"><code>changefeed.max\_behind\_nanos</code></div></td><td>The most any changefeed's persisted checkpoint is behind the present</td><td>GAUGE</td><td>NANOSECONDS</td></tr> <tr><td><div id="changefeed.running" class="anchored"><code>changefeed.running</code></div></td><td>Number of currently running changefeeds, including sinkless</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="clock-offset.meannanos" class="anchored"><code>clock-offset.meannanos</code></div></td><td>Mean clock offset with other nodes</td><td>GAUGE</td><td>NANOSECONDS</td></tr> <tr><td><div id="distsender.errors.notleaseholder" class="anchored"><code>distsender.errors.notleaseholder</code></div></td><td>Number of NotLeaseHolderErrors encountered from replica-addressed RPCs</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="distsender.rpc.sent.nextreplicaerror" class="anchored"><code>distsender.rpc.sent.nextreplicaerror</code></div></td><td>Number of replica-addressed RPCs sent due to per-replica errors</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="exec.latency" class="anchored"><code>exec.latency</code></div></td><td>Latency of batch KV requests (including errors) executed on this node. This measures requests already addressed to a single replica, from the moment at which they arrive at the internal gRPC endpoint to the moment at which the response (or an error) is returned. This latency includes in particular commit waits, conflict resolution and replication, and end-users can easily produce high measurements via long-running transactions that conflict with foreground traffic. This metric thus does not provide a good signal for understanding the health of the KV layer.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="go.scheduler_latency" class="anchored"><code>go.scheduler\_latency</code></div></td><td>Go scheduling latency</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="intentcount" class="anchored"><code>intentcount</code></div></td><td>Count of intent keys</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.auto_create_partial_stats.currently_paused" class="anchored"><code>jobs.auto\_create\_partial\_stats.currently\_paused</code></div></td><td>Number of auto\_create\_partial\_stats jobs currently considered Paused</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.auto_create_partial_stats.currently_running" class="anchored"><code>jobs.auto\_create\_partial\_stats.currently\_running</code></div></td><td>Number of auto\_create\_partial\_stats jobs currently running in Resume or OnFailOrCancel state</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.auto_create_partial_stats.resume_failed" class="anchored"><code>jobs.auto\_create\_partial\_stats.resume\_failed</code></div></td><td>Number of auto\_create\_partial\_stats jobs which failed with a non-retriable error</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="jobs.auto_create_stats.currently_paused" class="anchored"><code>jobs.auto\_create\_stats.currently\_paused</code></div></td><td>Number of auto\_create\_stats jobs currently considered Paused</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.auto_create_stats.currently_running" class="anchored"><code>jobs.auto\_create\_stats.currently\_running</code></div></td><td>Number of auto\_create\_stats jobs currently running in Resume or OnFailOrCancel state</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.auto_create_stats.resume_failed" class="anchored"><code>jobs.auto\_create\_stats.resume\_failed</code></div></td><td>Number of auto\_create\_stats jobs which failed with a non-retriable error</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="jobs.backup.currently_paused" class="anchored"><code>jobs.backup.currently\_paused</code></div></td><td>Number of backup jobs currently considered Paused</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.backup.currently_running" class="anchored"><code>jobs.backup.currently\_running</code></div></td><td>Number of backup jobs currently running in Resume or OnFailOrCancel state</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.changefeed.currently_paused" class="anchored"><code>jobs.changefeed.currently\_paused</code></div></td><td>Number of changefeed jobs currently considered Paused</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.changefeed.protected_age_sec" class="anchored"><code>jobs.changefeed.protected\_age\_sec</code></div></td><td>The age of the oldest PTS record protected by changefeed jobs</td><td>GAUGE</td><td>SECONDS</td></tr> <tr><td><div id="jobs.create_stats.currently_running" class="anchored"><code>jobs.create\_stats.currently\_running</code></div></td><td>Number of create\_stats jobs currently running in Resume or OnFailOrCancel state</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.currently_paused" class="anchored"><code>jobs.row\_level\_ttl.currently\_paused</code></div></td><td>Number of row\_level\_ttl jobs currently considered Paused</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.currently_running" class="anchored"><code>jobs.row\_level\_ttl.currently\_running</code></div></td><td>Number of row\_level\_ttl jobs currently running in Resume or OnFailOrCancel state</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.delete_duration" class="anchored"><code>jobs.row\_level\_ttl.delete\_duration</code></div></td><td>Duration for delete requests during row level TTL.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="jobs.row_level_ttl.num_active_spans" class="anchored"><code>jobs.row\_level\_ttl.num\_active\_spans</code></div></td><td>Number of active spans the TTL job is deleting from.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.resume_completed" class="anchored"><code>jobs.row\_level\_ttl.resume\_completed</code></div></td><td>Number of row\_level\_ttl jobs which successfully resumed to completion</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.resume_failed" class="anchored"><code>jobs.row\_level\_ttl.resume\_failed</code></div></td><td>Number of row\_level\_ttl jobs which failed with a non-retriable error</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.rows_deleted" class="anchored"><code>jobs.row\_level\_ttl.rows\_deleted</code></div></td><td>Number of rows deleted by the row level TTL job.</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.rows_selected" class="anchored"><code>jobs.row\_level\_ttl.rows\_selected</code></div></td><td>Number of rows selected for deletion by the row level TTL job.</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.select_duration" class="anchored"><code>jobs.row\_level\_ttl.select\_duration</code></div></td><td>Duration for select requests during row level TTL.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="jobs.row_level_ttl.span_total_duration" class="anchored"><code>jobs.row\_level\_ttl.span\_total\_duration</code></div></td><td>Duration for processing a span during row level TTL.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="jobs.row_level_ttl.total_expired_rows" class="anchored"><code>jobs.row\_level\_ttl.total\_expired\_rows</code></div></td><td>Approximate number of rows that have expired the TTL on the TTL table.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.total_rows" class="anchored"><code>jobs.row\_level\_ttl.total\_rows</code></div></td><td>Approximate number of rows on the TTL table.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="kv.concurrency.locks" class="anchored"><code>kv.concurrency.locks</code></div></td><td>Number of active locks held in lock tables. Does not include replicated locks (intents) that are not held in memory</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="kv.rangefeed.catchup_scan_nanos" class="anchored"><code>kv.rangefeed.catchup\_scan\_nanos</code></div></td><td>Time spent in RangeFeed catchup scan</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="kvflowcontrol.eval_wait.elastic.duration" class="anchored"><code>kvflowcontrol.eval\_wait.elastic.duration</code></div></td><td>Latency histogram for time elastic requests spent waiting for flow tokens to evaluate</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="kvflowcontrol.eval_wait.regular.duration" class="anchored"><code>kvflowcontrol.eval\_wait.regular.duration</code></div></td><td>Latency histogram for time regular requests spent waiting for flow tokens to evaluate</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="kvflowcontrol.send_queue.bytes" class="anchored"><code>kvflowcontrol.send\_queue.bytes</code></div></td><td>Byte size of all raft entries queued for sending to followers, waiting on available elastic send tokens</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="leases.epoch" class="anchored"><code>leases.epoch</code></div></td><td>Number of replica leaseholders using epoch-based leases</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="leases.expiration" class="anchored"><code>leases.expiration</code></div></td><td>Number of replica leaseholders using expiration-based leases</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="leases.leader" class="anchored"><code>leases.leader</code></div></td><td>Number of replica leaseholders using leader leases</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="leases.liveness" class="anchored"><code>leases.liveness</code></div></td><td>Number of replica leaseholders for the liveness range(s)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="leases.transfers.error" class="anchored"><code>leases.transfers.error</code></div></td><td>Number of failed lease transfers</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="leases.transfers.success" class="anchored"><code>leases.transfers.success</code></div></td><td>Number of successful lease transfers</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="livebytes" class="anchored"><code>livebytes</code></div></td><td>Number of bytes of live data (keys plus values)</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="liveness.heartbeatfailures" class="anchored"><code>liveness.heartbeatfailures</code></div></td><td>Number of failed node liveness heartbeats from this node</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="liveness.heartbeatlatency" class="anchored"><code>liveness.heartbeatlatency</code></div></td><td>Node liveness heartbeat latency</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="liveness.livenodes" class="anchored"><code>liveness.livenodes</code></div></td><td>Number of live nodes in the cluster (will be 0 if this node is not itself live)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="logical_replication.commit_latency" class="anchored"><code>logical\_replication.commit\_latency</code></div></td><td>Event commit latency: a difference between event MVCC timestamp and the time it was flushed into disk. If we batch events, then the difference between the oldest event in the batch and flush is recorded</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="logical_replication.events_dlqed" class="anchored"><code>logical\_replication.events\_dlqed</code></div></td><td>Row update events sent to DLQ</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="logical_replication.events_ingested" class="anchored"><code>logical\_replication.events\_ingested</code></div></td><td>Events ingested by all replication jobs</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="logical_replication.logical_bytes" class="anchored"><code>logical\_replication.logical\_bytes</code></div></td><td>Logical bytes (sum of keys + values) received by all replication jobs</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="logical_replication.replicated_time_seconds" class="anchored"><code>logical\_replication.replicated\_time\_seconds</code></div></td><td>The replicated time of the logical replication stream in seconds since the unix epoch.</td><td>GAUGE</td><td>SECONDS</td></tr> <tr><td><div id="physical_replication.logical_bytes" class="anchored"><code>physical\_replication.logical\_bytes</code></div></td><td>Logical bytes (sum of keys + values) ingested by all replication jobs</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="physical_replication.replicated_time_seconds" class="anchored"><code>physical\_replication.replicated\_time\_seconds</code></div></td><td>The replicated time of the physical replication stream in seconds since the unix epoch.</td><td>GAUGE</td><td>SECONDS</td></tr> <tr><td><div id="queue.gc.pending" class="anchored"><code>queue.gc.pending</code></div></td><td>Number of pending replicas in the MVCC GC queue</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="queue.gc.process.failure" class="anchored"><code>queue.gc.process.failure</code></div></td><td>Number of replicas which failed processing in the MVCC GC queue</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="queue.lease.pending" class="anchored"><code>queue.lease.pending</code></div></td><td>Number of pending replicas in the replica lease queue</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="queue.merge.pending" class="anchored"><code>queue.merge.pending</code></div></td><td>Number of pending replicas in the merge queue</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="queue.merge.process.failure" class="anchored"><code>queue.merge.process.failure</code></div></td><td>Number of replicas which failed processing in the merge queue</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="queue.merge.process.success" class="anchored"><code>queue.merge.process.success</code></div></td><td>Number of replicas successfully processed by the merge queue</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="queue.merge.processingnanos" class="anchored"><code>queue.merge.processingnanos</code></div></td><td>Nanoseconds spent processing replicas in the merge queue</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="queue.raftlog.pending" class="anchored"><code>queue.raftlog.pending</code></div></td><td>Number of pending replicas in the Raft log queue</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="queue.raftlog.process.failure" class="anchored"><code>queue.raftlog.process.failure</code></div></td><td>Number of replicas which failed processing in the Raft log queue</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="queue.raftlog.process.success" class="anchored"><code>queue.raftlog.process.success</code></div></td><td>Number of replicas successfully processed by the Raft log queue</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="queue.raftlog.processingnanos" class="anchored"><code>queue.raftlog.processingnanos</code></div></td><td>Nanoseconds spent processing replicas in the Raft log queue</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="queue.replicagc.pending" class="anchored"><code>queue.replicagc.pending</code></div></td><td>Number of pending replicas in the replica GC queue</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="queue.replicagc.process.failure" class="anchored"><code>queue.replicagc.process.failure</code></div></td><td>Number of replicas which failed processing in the replica GC queue</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="queue.replicagc.process.success" class="anchored"><code>queue.replicagc.process.success</code></div></td><td>Number of replicas successfully processed by the replica GC queue</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="queue.replicagc.purgatory" class="anchored"><code>queue.replicagc.purgatory</code></div></td><td>Number of replicas in the replica GC queue's purgatory, waiting for a prerequisite (e.g. left neighbor GC) before they can be GC'd</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="queue.replicate.pending" class="anchored"><code>queue.replicate.pending</code></div></td><td>Number of pending replicas in the replicate queue</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="queue.replicate.process.failure" class="anchored"><code>queue.replicate.process.failure</code></div></td><td>Number of replicas which failed processing in the replicate queue</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="queue.replicate.process.success" class="anchored"><code>queue.replicate.process.success</code></div></td><td>Number of replicas successfully processed by the replicate queue</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="queue.replicate.replacedecommissioningreplica.error" class="anchored"><code>queue.replicate.replacedecommissioningreplica.error</code></div></td><td>Number of failed decommissioning replica replacements processed by the replicate queue</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="raft.scheduler.latency" class="anchored"><code>raft.scheduler.latency</code></div></td><td>Queueing durations for ranges waiting to be processed by the Raft scheduler. This histogram measures the delay from when a range is registered with the scheduler for processing to when it is actually processed. This does not include the duration of processing.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="raftlog.behind" class="anchored"><code>raftlog.behind</code></div></td><td>Number of Raft log entries followers on other stores are behind. This gauge provides a view of the aggregate number of log entries the Raft leaders on this node think the followers are behind. Since a raft leader may not always have a good estimate for this information for all of its followers, and since followers are expected to be behind (when they are not required as part of a quorum) \*and\* the aggregate thus scales like the count of such followers, it is difficult to meaningfully interpret this metric.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="range.adds" class="anchored"><code>range.adds</code></div></td><td>Number of range additions</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="range.merges" class="anchored"><code>range.merges</code></div></td><td>Number of range merges</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="range.snapshots.send-queue" class="anchored"><code>range.snapshots.send-queue</code></div></td><td>Number of snapshots queued to send</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="range.splits" class="anchored"><code>range.splits</code></div></td><td>Number of range splits</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="ranges" class="anchored"><code>ranges</code></div></td><td>Number of ranges</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="ranges.decommissioning" class="anchored"><code>ranges.decommissioning</code></div></td><td>Number of ranges with at lease one replica on a decommissioning node</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="ranges.unavailable" class="anchored"><code>ranges.unavailable</code></div></td><td>Number of ranges with fewer live replicas than needed for quorum</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="ranges.underreplicated" class="anchored"><code>ranges.underreplicated</code></div></td><td>Number of ranges with fewer live replicas than the replication target</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="rebalancing.cpunanospersecond" class="anchored"><code>rebalancing.cpunanospersecond</code></div></td><td>Average CPU nanoseconds spent on processing replica operations in the last 30 minutes.</td><td>GAUGE</td><td>NANOSECONDS</td></tr> <tr><td><div id="rebalancing.lease.transfers" class="anchored"><code>rebalancing.lease.transfers</code></div></td><td>Number of lease transfers motivated by store-level load imbalances</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="rebalancing.queriespersecond" class="anchored"><code>rebalancing.queriespersecond</code></div></td><td>Number of kv-level requests received per second by the store, considering the last 30 minutes, as used in rebalancing decisions.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="rebalancing.range.rebalances" class="anchored"><code>rebalancing.range.rebalances</code></div></td><td>Number of range rebalance operations motivated by store-level load imbalances</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="rebalancing.replicas.cpunanospersecond" class="anchored"><code>rebalancing.replicas.cpunanospersecond</code></div></td><td>Histogram of average CPU nanoseconds spent on processing replica operations in the last 30 minutes.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="rebalancing.replicas.queriespersecond" class="anchored"><code>rebalancing.replicas.queriespersecond</code></div></td><td>Histogram of average kv-level requests received per second by replicas on the store in the last 30 minutes.</td><td>HISTOGRAM</td><td>COUNT</td></tr> <tr><td><div id="replicas" class="anchored"><code>replicas</code></div></td><td>Number of replicas</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="replicas.leaseholders" class="anchored"><code>replicas.leaseholders</code></div></td><td>Number of lease holders</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="requests.slow.latch" class="anchored"><code>requests.slow\.latch</code></div></td><td>Number of requests that have been stuck for a long time acquiring latches. Latches moderate access to the KV keyspace for the purpose of evaluating and replicating commands. A slow latch acquisition attempt is often caused by another request holding and not releasing its latches in a timely manner. This in turn can either be caused by a long delay in evaluation (for example, under severe system overload) or by delays at the replication layer. This gauge registering a nonzero value usually indicates a serious problem and should be investigated.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="requests.slow.lease" class="anchored"><code>requests.slow\.lease</code></div></td><td>Number of requests that have been stuck for a long time acquiring a lease. This gauge registering a nonzero value usually indicates range or replica unavailability, and should be investigated. In the common case, we also expect to see 'requests.slow\.raft' to register a nonzero value, indicating that the lease requests are not getting a timely response from the replication layer.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="requests.slow.raft" class="anchored"><code>requests.slow\.raft</code></div></td><td>Number of requests that have been stuck for a long time in the replication layer. An (evaluated) request has to pass through the replication layer, notably the quota pool and raft. If it fails to do so within a highly permissive duration, the gauge is incremented (and decremented again once the request is either applied or returns an error). A nonzero value indicates range or replica unavailability, and should be investigated.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="rocksdb.block.cache.hits" class="anchored"><code>rocksdb.block.cache.hits</code></div></td><td>Count of block cache hits</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="rocksdb.block.cache.misses" class="anchored"><code>rocksdb.block.cache.misses</code></div></td><td>Count of block cache misses</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="rocksdb.compactions" class="anchored"><code>rocksdb.compactions</code></div></td><td>Number of table compactions</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="rocksdb.read-amplification" class="anchored"><code>rocksdb.read-amplification</code></div></td><td>Number of disk reads per query</td><td>GAUGE</td><td>CONST</td></tr> <tr><td><div id="round-trip-latency" class="anchored"><code>round-trip-latency</code></div></td><td>Distribution of round-trip latencies with other nodes. This only reflects successful heartbeats and measures gRPC overhead as well as possible head-of-line blocking. Elevated values in this metric may hint at network issues and/or saturation, but they are no proof of them. CPU overload can similarly elevate this metric. The operator should look towards OS-level metrics such as packet loss, retransmits, etc, to conclusively diagnose network issues. Heartbeats are not very frequent (\~seconds), so they may not capture rare or short-lived degradations.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="rpc.connection.avg_round_trip_latency" class="anchored"><code>rpc.connection.avg\_round\_trip\_latency</code></div></td><td>Sum of exponentially weighted moving average of round-trip latencies, as measured through a gRPC RPC. Since this metric is based on gRPC RPCs, it is affected by application-level processing delays and CPU overload effects. See rpc.connection.tcp\_rtt for a metric that is obtained from the kernel's TCP stack. Dividing this Gauge by rpc.connection.healthy gives an approximation of average latency, but the top-level round-trip-latency histogram is more useful. Instead, users should consult the label families of this metric if they are available (which requires prometheus and the cluster setting 'server.child\_metrics.enabled'); these provide per-peer moving averages. This metric does not track failed connection. A failed connection's contribution is reset to zero.</td><td>GAUGE</td><td>NANOSECONDS</td></tr> <tr><td><div id="rpc.connection.failures" class="anchored"><code>rpc.connection.failures</code></div></td><td>Counter of failed connections. This includes both the event in which a healthy connection terminates as well as unsuccessful reconnection attempts. Connections that are terminated as part of local node shutdown are excluded. Decommissioned peers are excluded.</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="rpc.connection.healthy" class="anchored"><code>rpc.connection.healthy</code></div></td><td>Gauge of current connections in a healthy state (i.e. bidirectionally connected and heartbeating)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="rpc.connection.healthy_nanos" class="anchored"><code>rpc.connection.healthy\_nanos</code></div></td><td>Gauge of nanoseconds of healthy connection time On the prometheus endpoint scraped with the cluster setting 'server.child\_metrics.enabled' set, the constituent parts of this metric are available on a per-peer basis and one can read off for how long a given peer has been connected</td><td>GAUGE</td><td>NANOSECONDS</td></tr> <tr><td><div id="rpc.connection.heartbeats" class="anchored"><code>rpc.connection.heartbeats</code></div></td><td>Counter of successful heartbeats.</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="rpc.connection.tcp_rtt" class="anchored"><code>rpc.connection.tcp\_rtt</code></div></td><td>Kernel-level TCP round-trip time as measured by the Linux TCP stack. This metric reports the smoothed round-trip time (SRTT) as maintained by the kernel's TCP implementation. Unlike application-level RPC latency measurements, this reflects pure network latency and is less affected by CPU overload effects. This metric is only available on Linux.</td><td>GAUGE</td><td>NANOSECONDS</td></tr> <tr><td><div id="rpc.connection.tcp_rtt_var" class="anchored"><code>rpc.connection.tcp\_rtt\_var</code></div></td><td>Kernel-level TCP round-trip time variance as measured by the Linux TCP stack. This metric reports the smoothed round-trip time variance (RTTVAR) as maintained by the kernel's TCP implementation. This measures the stability of the connection latency. This metric is only available on Linux.</td><td>GAUGE</td><td>NANOSECONDS</td></tr> <tr><td><div id="rpc.connection.unhealthy" class="anchored"><code>rpc.connection.unhealthy</code></div></td><td>Gauge of current connections in an unhealthy state (not bidirectionally connected or heartbeating)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="rpc.connection.unhealthy_nanos" class="anchored"><code>rpc.connection.unhealthy\_nanos</code></div></td><td>Gauge of nanoseconds of unhealthy connection time. On the prometheus endpoint scraped with the cluster setting 'server.child\_metrics.enabled' set, the constituent parts of this metric are available on a per-peer basis and one can read off for how long a given peer has been unreachable</td><td>GAUGE</td><td>NANOSECONDS</td></tr> <tr><td><div id="schedules.BACKUP.failed" class="anchored"><code>schedules.BACKUP.failed</code></div></td><td>Number of BACKUP jobs failed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="schedules.BACKUP.last-completed-time" class="anchored"><code>schedules.BACKUP.last-completed-time</code></div></td><td>The unix timestamp of the most recently completed backup by a schedule specified as maintaining this metric</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="schedules.scheduled-row-level-ttl-executor.failed" class="anchored"><code>schedules.scheduled-row-level-ttl-executor.failed</code></div></td><td>Number of scheduled-row-level-ttl-executor jobs failed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="seconds.until.enterprise.license.expiry" class="anchored"><code>seconds.until.enterprise.license.expiry</code></div></td><td>Seconds until enterprise license expiry (0 if no license present or running without enterprise features)</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.expiration.ca" class="anchored"><code>security.certificate.expiration.ca</code></div></td><td>Expiration for the CA certificate. 0 means no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.expiration.ca-client-tenant" class="anchored"><code>security.certificate.expiration.ca-client-tenant</code></div></td><td>Expiration for the Tenant Client CA certificate. 0 means no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.expiration.client" class="anchored"><code>security.certificate.expiration.client</code></div></td><td>Minimum expiration for client certificates, labeled by SQL user. 0 means no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.expiration.client-ca" class="anchored"><code>security.certificate.expiration.client-ca</code></div></td><td>Expiration for the client CA certificate. 0 means no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.expiration.client-tenant" class="anchored"><code>security.certificate.expiration.client-tenant</code></div></td><td>Expiration for the Tenant Client certificate. 0 means no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.expiration.node" class="anchored"><code>security.certificate.expiration.node</code></div></td><td>Expiration for the node certificate. 0 means no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.expiration.node-client" class="anchored"><code>security.certificate.expiration.node-client</code></div></td><td>Expiration for the node's client certificate. 0 means no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.expiration.ui" class="anchored"><code>security.certificate.expiration.ui</code></div></td><td>Expiration for the UI certificate. 0 means no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.expiration.ui-ca" class="anchored"><code>security.certificate.expiration.ui-ca</code></div></td><td>Expiration for the UI CA certificate. 0 means no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.ttl.ca" class="anchored"><code>security.certificate.ttl.ca</code></div></td><td>Seconds till expiration for the CA certificate. 0 means expired, no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.ttl.ca-client-tenant" class="anchored"><code>security.certificate.ttl.ca-client-tenant</code></div></td><td>Seconds till expiration for the Tenant Client CA certificate. 0 means expired, no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.ttl.client" class="anchored"><code>security.certificate.ttl.client</code></div></td><td>Seconds till expiration for the client certificates, labeled by SQL user. 0 means expired, no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.ttl.client-ca" class="anchored"><code>security.certificate.ttl.client-ca</code></div></td><td>Seconds till expiration for the client CA certificate. 0 means expired, no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.ttl.client-tenant" class="anchored"><code>security.certificate.ttl.client-tenant</code></div></td><td>Seconds till expiration for the Tenant Client certificate. 0 means expired, no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.ttl.node" class="anchored"><code>security.certificate.ttl.node</code></div></td><td>Seconds till expiration for the node certificate. 0 means expired, no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.ttl.node-client" class="anchored"><code>security.certificate.ttl.node-client</code></div></td><td>Seconds till expiration for the node's client certificate. 0 means expired, no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.ttl.ui" class="anchored"><code>security.certificate.ttl.ui</code></div></td><td>Seconds till expiration for the UI certificate. 0 means expired, no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="security.certificate.ttl.ui-ca" class="anchored"><code>security.certificate.ttl.ui-ca</code></div></td><td>Seconds till expiration for the UI CA certificate. 0 means expired, no certificate or error.</td><td>GAUGE</td><td>TIMESTAMP\_SEC</td></tr> <tr><td><div id="sql.conn.failures" class="anchored"><code>sql.conn.failures</code></div></td><td>Number of SQL connection failures</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.conn.latency" class="anchored"><code>sql.conn.latency</code></div></td><td>Latency to establish and authenticate a SQL connection</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="sql.conns" class="anchored"><code>sql.conns</code></div></td><td>Number of open SQL connections</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sql.ddl.count" class="anchored"><code>sql.ddl.count</code></div></td><td>Number of SQL DDL statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.delete.count" class="anchored"><code>sql.delete.count</code></div></td><td>Number of SQL DELETE statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.distsql.contended_queries.count" class="anchored"><code>sql.distsql.contended\_queries.count</code></div></td><td>Number of SQL queries that experienced contention</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.exec.latency" class="anchored"><code>sql.exec.latency</code></div></td><td>Latency of SQL statement execution</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="sql.failure.count" class="anchored"><code>sql.failure.count</code></div></td><td>Number of statements resulting in a planning or runtime error</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.full.scan.count" class="anchored"><code>sql.full.scan.count</code></div></td><td>Number of full table or index scans</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.insert.count" class="anchored"><code>sql.insert.count</code></div></td><td>Number of SQL INSERT statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.mem.root.current" class="anchored"><code>sql.mem.root.current</code></div></td><td>Current sql statement memory usage for root</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="sql.new_conns" class="anchored"><code>sql.new\_conns</code></div></td><td>Number of SQL connections created</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.query.count" class="anchored"><code>sql.query.count</code></div></td><td>Number of SQL operations started including queries, and transaction control statements</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.routine.delete.count" class="anchored"><code>sql.routine.delete.count</code></div></td><td>Number of SQL DELETE statements successfully executed within routine invocation</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.routine.insert.count" class="anchored"><code>sql.routine.insert.count</code></div></td><td>Number of SQL INSERT statements successfully executed within routine invocation</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.routine.select.count" class="anchored"><code>sql.routine.select.count</code></div></td><td>Number of SQL SELECT statements successfully executed within routine invocation</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.routine.update.count" class="anchored"><code>sql.routine.update.count</code></div></td><td>Number of SQL UPDATE statements successfully executed within routine invocation</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.select.count" class="anchored"><code>sql.select.count</code></div></td><td>Number of SQL SELECT statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.service.latency" class="anchored"><code>sql.service.latency</code></div></td><td>Latency of SQL request execution</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="sql.statements.active" class="anchored"><code>sql.statements.active</code></div></td><td>Number of currently active user SQL statements</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sql.txn.abort.count" class="anchored"><code>sql.txn.abort.count</code></div></td><td>Number of SQL transaction abort errors</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.txn.begin.count" class="anchored"><code>sql.txn.begin.count</code></div></td><td>Number of SQL transaction BEGIN statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.txn.commit.count" class="anchored"><code>sql.txn.commit.count</code></div></td><td>Number of SQL transaction COMMIT statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.txn.latency" class="anchored"><code>sql.txn.latency</code></div></td><td>Latency of SQL transactions</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="sql.txn.rollback.count" class="anchored"><code>sql.txn.rollback.count</code></div></td><td>Number of SQL transaction ROLLBACK statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.txns.open" class="anchored"><code>sql.txns.open</code></div></td><td>Number of currently open user SQL transactions</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sql.update.count" class="anchored"><code>sql.update.count</code></div></td><td>Number of SQL UPDATE statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="storage.disk-slow" class="anchored"><code>storage.disk-slow</code></div></td><td>Number of instances of disk operations taking longer than 10s</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="storage.disk-stalled" class="anchored"><code>storage.disk-stalled</code></div></td><td>Number of instances of disk operations taking longer than 20s</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="storage.disk.iopsinprogress" class="anchored"><code>storage.disk.iopsinprogress</code></div></td><td>IO operations currently in progress on the store's disk (as reported by the OS)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="storage.disk.read-max.bytespersecond" class="anchored"><code>storage.disk.read-max.bytespersecond</code></div></td><td>Maximum rate at which bytes were read from disk (as reported by the OS)</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="storage.disk.read.bytes" class="anchored"><code>storage.disk.read.bytes</code></div></td><td>Bytes read from the store's disk since this process started (as reported by the OS)</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="storage.disk.read.count" class="anchored"><code>storage.disk.read.count</code></div></td><td>Disk read operations on the store's disk since this process started (as reported by the OS)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="storage.disk.write-max.bytespersecond" class="anchored"><code>storage.disk.write-max.bytespersecond</code></div></td><td>Maximum rate at which bytes were written to disk (as reported by the OS)</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="storage.disk.write.bytes" class="anchored"><code>storage.disk.write.bytes</code></div></td><td>Bytes written to the store's disk since this process started (as reported by the OS)</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="storage.disk.write.count" class="anchored"><code>storage.disk.write.count</code></div></td><td>Disk write operations on the store's disk since this process started (as reported by the OS)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="storage.keys.tombstone.count" class="anchored"><code>storage.keys.tombstone.count</code></div></td><td>Approximate count of DEL, SINGLEDEL and RANGEDEL internal keys across the storage engine.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="storage.l0-level-size" class="anchored"><code>storage.l0-level-size</code></div></td><td>Size of the SSTables in level 0</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="storage.wal.failover.switch.count" class="anchored"><code>storage.wal.failover.switch.count</code></div></td><td>Count of the number of times WAL writing has switched from primary to secondary and vice versa.</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="storage.wal.failover.write_and_sync.latency" class="anchored"><code>storage.wal.failover.write\_and\_sync.latency</code></div></td><td>The observed latency for writing and syncing to the logical Write-Ahead Log.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="storage.wal.fsync.latency" class="anchored"><code>storage.wal.fsync.latency</code></div></td><td>The fsync latency to the Write-Ahead Log device.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="storage.write-stall-nanos" class="anchored"><code>storage.write-stall-nanos</code></div></td><td>Total write stall duration in nanos</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="storage.write-stalls" class="anchored"><code>storage.write-stalls</code></div></td><td>Number of instances of intentional write stalls to backpressure incoming writes</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="storeliveness.heartbeat.failures" class="anchored"><code>storeliveness.heartbeat.failures</code></div></td><td>Number of Store Liveness heartbeats that failed to be sent out by the Store Liveness Support Manager</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sys.cgo.allocbytes" class="anchored"><code>sys.cgo.allocbytes</code></div></td><td>Current bytes of memory allocated by cgo</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="sys.cgo.totalbytes" class="anchored"><code>sys.cgo.totalbytes</code></div></td><td>Total bytes of memory allocated by cgo, but not released</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="sys.cpu.combined.percent-normalized" class="anchored"><code>sys.cpu.combined.percent-normalized</code></div></td><td>Current user+system cpu percentage consumed by the CRDB process, normalized 0-1 by number of cores</td><td>GAUGE</td><td>PERCENT</td></tr> <tr><td><div id="sys.cpu.host.combined.percent-normalized" class="anchored"><code>sys.cpu.host.combined.percent-normalized</code></div></td><td>Current user+system cpu percentage across the whole machine, normalized 0-1 by number of cores</td><td>GAUGE</td><td>PERCENT</td></tr> <tr><td><div id="sys.cpu.sys.ns" class="anchored"><code>sys.cpu.sys.ns</code></div></td><td>Total system cpu time consumed by the CRDB process</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="sys.cpu.sys.percent" class="anchored"><code>sys.cpu.sys.percent</code></div></td><td>Current system cpu percentage consumed by the CRDB process</td><td>GAUGE</td><td>PERCENT</td></tr> <tr><td><div id="sys.cpu.user.ns" class="anchored"><code>sys.cpu.user.ns</code></div></td><td>Total user cpu time consumed by the CRDB process</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="sys.cpu.user.percent" class="anchored"><code>sys.cpu.user.percent</code></div></td><td>Current user cpu percentage consumed by the CRDB process</td><td>GAUGE</td><td>PERCENT</td></tr> <tr><td><div id="sys.gc.count" class="anchored"><code>sys.gc.count</code></div></td><td>Total number of GC runs</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sys.gc.pause.ns" class="anchored"><code>sys.gc.pause.ns</code></div></td><td>Total GC pause</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="sys.gc.pause.percent" class="anchored"><code>sys.gc.pause.percent</code></div></td><td>Current GC pause percentage</td><td>GAUGE</td><td>PERCENT</td></tr> <tr><td><div id="sys.go.allocbytes" class="anchored"><code>sys.go.allocbytes</code></div></td><td>Current bytes of memory allocated by go</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="sys.go.heap.allocbytes" class="anchored"><code>sys.go.heap.allocbytes</code></div></td><td>Cumulative bytes allocated for heap objects.</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="sys.go.heap.heapfragmentbytes" class="anchored"><code>sys.go.heap.heapfragmentbytes</code></div></td><td>Total heap fragmentation bytes, derived from bytes in in-use spans minus bytes allocated</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="sys.go.totalbytes" class="anchored"><code>sys.go.totalbytes</code></div></td><td>Total bytes of memory allocated by go, but not released</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="sys.goroutines" class="anchored"><code>sys.goroutines</code></div></td><td>Current number of goroutines</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sys.host.disk.iopsinprogress" class="anchored"><code>sys.host.disk.iopsinprogress</code></div></td><td>IO operations currently in progress on this host (as reported by the OS)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sys.host.disk.read.bytes" class="anchored"><code>sys.host.disk.read.bytes</code></div></td><td>Bytes read from all disks since this process started (as reported by the OS)</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="sys.host.disk.read.count" class="anchored"><code>sys.host.disk.read.count</code></div></td><td>Disk read operations across all disks since this process started (as reported by the OS)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sys.host.disk.write.bytes" class="anchored"><code>sys.host.disk.write.bytes</code></div></td><td>Bytes written to all disks since this process started (as reported by the OS)</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="sys.host.disk.write.count" class="anchored"><code>sys.host.disk.write.count</code></div></td><td>Disk write operations across all disks since this process started (as reported by the OS)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sys.host.net.recv.bytes" class="anchored"><code>sys.host.net.recv.bytes</code></div></td><td>Bytes received on all network interfaces since this process started (as reported by the OS)</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="sys.host.net.send.bytes" class="anchored"><code>sys.host.net.send.bytes</code></div></td><td>Bytes sent on all network interfaces since this process started (as reported by the OS)</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="sys.host.net.send.tcp.retrans_segs" class="anchored"><code>sys.host.net.send.tcp.retrans\_segs</code></div></td><td>The number of TCP segments retransmitted across all network interfaces. This can indicate packet loss occurring in the network. However, it can also be caused by recipient nodes not consuming packets in a timely manner, or the local node overflowing its outgoing buffers, for example due to overload. Retransmissions also occur in the absence of problems, as modern TCP stacks err on the side of aggressively retransmitting segments. The linux tool 'ss -i' can show the Linux kernel's smoothed view of round-trip latency and variance on a per-connection basis.  Additionally, 'netstat -s' shows all TCP counters maintained by the kernel.</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sys.rss" class="anchored"><code>sys.rss</code></div></td><td>Current process RSS</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="sys.runnable.goroutines.per.cpu" class="anchored"><code>sys.runnable.goroutines.per.cpu</code></div></td><td>Average number of goroutines that are waiting to run, normalized by number of cores</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sys.totalmem" class="anchored"><code>sys.totalmem</code></div></td><td>Total memory (both free and used)</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="sys.uptime" class="anchored"><code>sys.uptime</code></div></td><td>Process uptime</td><td>COUNTER</td><td>SECONDS</td></tr> <tr><td><div id="txn.restarts" class="anchored"><code>txn.restarts</code></div></td><td>Number of restarted KV transactions</td><td>HISTOGRAM</td><td>COUNT</td></tr> <tr><td><div id="txn.restarts.serializable" class="anchored"><code>txn.restarts.serializable</code></div></td><td>Number of restarts due to a forwarded commit timestamp and isolation=SERIALIZABLE</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="txn.restarts.txnaborted" class="anchored"><code>txn.restarts.txnaborted</code></div></td><td>Number of restarts due to an abort by a concurrent transaction (usually due to deadlock)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="txn.restarts.txnpush" class="anchored"><code>txn.restarts.txnpush</code></div></td><td>Number of restarts due to a transaction push failure</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="txn.restarts.unknown" class="anchored"><code>txn.restarts.unknown</code></div></td><td>Number of restarts due to a unknown reasons</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="txn.restarts.writetooold" class="anchored"><code>txn.restarts.writetooold</code></div></td><td>Number of restarts due to a concurrent writer committing first</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="txnwaitqueue.deadlocks_total" class="anchored"><code>txnwaitqueue.deadlocks\_total</code></div></td><td>Number of deadlocks detected by the txn wait queue</td><td>COUNTER</td><td>COUNT</td></tr> </tbody></table>

**Total metrics (240)**

## See also

* <InternalLink path="troubleshooting-overview">Troubleshooting Overview</InternalLink>
* <InternalLink path="support-resources">Support Resources</InternalLink>
* <InternalLink path="monitoring-and-alerting#raw-status-endpoints">Raw Status Endpoints</InternalLink>