Skip to main content
Multi-dimensional metrics are additional time series with extra labels. This page will help you understand the potential size of the Prometheus scrape payload for your workload when multi-dimensional metrics are enabled. The number of multi-dimensional metrics can significantly increase based on their associated labels, which increases cardinality. The export of multi-dimensional metrics can be enabled by two :

​
Enable child metrics

Child metrics are specific, detailed metrics that are usually related to a higher-level (parent or aggregate) metric. They often provide more granular or specific information about a particular aspect of the parent metric. The parent metrics and their potential child metrics are determined by the specific feature the cluster is using. The is disabled by default. To enable it, use the statement:
New in v25.1.3: The (default: true) reports an aggregate time series for applicable multi-dimensional metrics. When set to false, it stops reporting the aggregate time series, preventing double counting when querying those metrics.

​
All clusters

An RPC (Remote Procedure Call) connection is a communication method used in distributed systems, like CockroachDB, to allow one program to request a service from a program located in another computer on a network without having to understand the network’s details. In the context of CockroachDB, RPC connections are used for inter-node communication. For instance, if Node 1 sends a request to Node 2, and Node 2 dials back (sends request back to Node 1), it ensures that communication is healthy in both directions. This is referred to as a “bidirectionally connected” and “heartbeating” RPC connection. When child metrics is enabled, for all clusters the rpc.connection.* metrics are exported per-peer with labels for remote_node_id, remote_addr, and class. The class label may have the following values: system, default, and raft. The cardinality increases with the number of nodes. An aggregated metric is also included. For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
rpc.connection.avg_round_trip_latency
rpc.connection.failures
rpc.connection.healthy
rpc.connection.healthy_nanos
rpc.connection.heartbeats
rpc.connection.inactive
rpc.connection.unhealthy
rpc.connection.unhealthy_nanos

​
Secure clusters

When child metrics is enabled, for the security.certificate.expiration.client is exported per SQL user with a label for sql_user. The sql_user label may have the values of the cluster’s users who are logged into a node using client security certificates. The cardinality increases with the number of SQL users. An aggregated metric is also included, however since this is a sum of the child metric values which represent timestamps, it is not usable. For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
security.certificate.expiration.client
Minimum expiration for client certificates, labeled by SQL user. 0 means no certificate or error.GAUGESECONDS

​
Virtual clusters

When child metrics is enabled, for the kv.tenant_rate_limit.* metrics and other kv-related metrics are exported per virtual cluster with a label for tenant_id. The tenant_id label may have the values: system or the id of the virtual cluster. The cardinality increases with the number of virtual clusters. An aggregated metric is also included.
With virtual clusters, while the tenant_id label on kv metrics is only exported when child metrics is enabled, is exported whether child metrics is enabled or disabled.
For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
abortspanbytes
gcbytesage
intentage
intentbytes
intentcount
keybytes
keycount
kv.tenant_rate_limit.current_blocked
kv.tenant_rate_limit.num_tenants
kv.tenant_rate_limit.read_batches_admitted
kv.tenant_rate_limit.read_bytes_admitted
kv.tenant_rate_limit.read_requests_admitted
kv.tenant_rate_limit.write_batches_admitted
kv.tenant_rate_limit.write_bytes_admitted
kv.tenant_rate_limit.write_requests_admitted
livebytes
livecount
lockbytes
lockcount
rangekeybytes
rangekeycount
rangevalbytes
rangevalcount
sysbytes
syscount
totalbytes
valbytes
valcount

​
Clusters with changefeeds

When child metrics is enabled and are created on the cluster, the changefeed.* metrics are exported per changefeed metric label with a label for scope. The scope label may have the values set using the metrics_label option. The cardinality increases with the number of changefeed metric labels. An aggregated metric is also included. For example, when you create two changefeeds with the metrics labels employees and office_dogs, the counter metric changefeed_error_retries exports child metrics with a scope for employees and office_dogs. In addition, the default scope will also be exported which includes changefeeds started without a metrics label.
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
changefeed.admit_latency
changefeed.aggregator_progress
changefeed.backfill_count
changefeed.backfill_pending_ranges
changefeed.batch_reduction_count
changefeed.checkpoint_progress
changefeed.cloudstorage_buffered_bytes
changefeed.commit_latency
changefeed.emitted_batch_sizes
changefeed.emitted_bytes
changefeed.emitted_messages
changefeed.error_retries
changefeed.filtered_messages
changefeed.flush_hist_nanos
changefeed.flushed_bytes
changefeed.flushes
changefeed.internal_retry_message_count
changefeed.kafka_throttling_hist_nanos
changefeed.lagging_ranges
changefeed.message_size_hist
changefeed.parallel_io_in_flight_keys
changefeed.parallel_io_pending_rows
changefeed.parallel_io_queue_nanos
changefeed.parallel_io_result_queue_nanos
changefeed.running
changefeed.schema_registry.registrations
changefeed.schema_registry.retry_count
changefeed.sink_batch_hist_nanos
changefeed.sink_io_inflight
changefeed.size_based_flushes

​
Clusters with logical data replication jobs

When child metrics is enabled and are created on the cluster, the logical_replication_*_by_label metrics are exported per LDR metric label. The label may have the values set using the label option. The cardinality increases with the number of LDR metric labels. For example, when you create two LDR jobs with the metrics labels ldr_job1 and ldr_job2, the metrics logical_replication_*_by_label export child metrics with a label for ldr_job1 and ldr_job2.
Note that the logical_replication_* metrics without the _by_label suffix may be inaccurate with multiple LDR jobs.
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescriptionTypeUnit
logical_replication.catchup_ranges_by_label
logical_replication.events_dlqed_by_label
logical_replication.events_ingested_by_label
logical_replication.replicated_time_by_label
logical_replication.scanning_ranges_by_label

​
Clusters with row-level TTL jobs

When child metrics is enabled and are created on the cluster with the , the jobs.row_level_ttl.* metrics are exported per TTL job with ttl_label_metrics enabled with a label for relation. The value of the relation label may have the format: {database}_{schema}_{table}_{primary key}. The cardinality increases with the number of TTL jobs with ttl_label_metrics enabled. An aggregated metric is also included. For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
jobs.row_level_ttl.delete_duration
jobs.row_level_ttl.num_active_spans
jobs.row_level_ttl.rows_deleted
jobs.row_level_ttl.rows_selected
jobs.row_level_ttl.select_duration
jobs.row_level_ttl.span_total_duration
jobs.row_level_ttl.total_expired_rows
jobs.row_level_ttl.total_rows

​
Metrics of type histogram

When child metrics is enabled with changefeeds or row-level TTL jobs, be aware that metrics of type HISTOGRAM will increase cardinality quickly. For example, when you create two changefeeds with the metrics labels employees and office_dogs, the histogram metric changefeed_flush_hist_nanos exports child metrics for each bucket for each metrics label. In addition, the default scope will also be exported which includes changefeeds started without a metrics label. Therefore, in this example, changefeed_flush_hist_nanos exports child metrics for each bucket for the scope values: default, employees and office_dogs:

​
Enable detailed latency metrics

New in v25.1: The sql.stats.detailed_latency_metrics.enabled labels the latency metric sql.exec.latency.detail with the . To estimate the cardinality of the set of all statement fingerprints, use the sql.query.unique.count metric. For most workloads, this metric ranges from dozens to hundreds. sql.stats.detailed_latency_metrics.enabled is disabled by default. To enable it, use the statement:
Be aware that the sql.exec.latency.detail metric is of type HISTOGRAM and will quickly increase the cardinality of exported metrics. For workloads with over a couple thousand statement fingerprints, exercise caution when enabling sql.stats.detailed_latency_metrics.enabled. For workloads with tens of thousands of distinct query fingerprints, leave this cluster setting set to false.
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescriptionTypeUnit
sql.exec.latency.detail
For example: