Common Issues to Monitor

• CPU
• Memory
• Storage and disk I/O

​

CPU

​

CPU planning

​

CPU monitoring

​

Service latency

• The and graphs on the SQL dashboard show the time in nanoseconds between when the cluster . This time does not include returning results to the client.

• High CPU usage.
• An I/O bottleneck.

​

CPU usage

• The graph on the Hardware and Overload dashboards shows the CPU consumption by the CockroachDB process, and excludes other processes on the node. Expected values for a healthy cluster: CPU utilized by CockroachDB should not persistently exceed 80%. Because this metric does not reflect system CPU usage, values above 80% suggest that actual CPU utilization is nearing 100%.

​

Workload concurrency

• The graph on the Overview and SQL dashboards shows the 10-second moving average of SELECT, UPDATE, INSERT, and DELETE statements being executed per second on the cluster or node, as well as Total queries. The latest QPS value for the cluster is also displayed with the Queries per second counter on the Metrics page. Expected values for a healthy cluster: At any time, the total number of connections actively executing SQL statements should not exceed 4 times the number of vCPUs in the cluster. You can find them in the Active Executions view in the or . You can find the number of open connections in the or . For more details on configuring connection pools, see .

• High CPU usage.
• Degradation in SQL response time.
• Over time, an unhealthy LSM and cluster instability.

​

LSM health

• The graph on the Overload dashboard shows the health of the , which are implemented as log-structured merge (LSM) trees. Level 0 is the highest level of the LSM tree and consists of files containing the latest data written to the . For more information about LSM levels and how LSMs work, see . Expected values for a healthy cluster: An IO Overload value greater than 1.0 generally indicates an overload in the Pebble LSM tree. High values indicate heavy write load that is causing accumulation of files in level 0. These files are not being compacted quickly enough to lower levels, resulting in a .

• The Read Amplification graph on the shows the average number of disk reads per logical SQL statement, also known as the factor. Expected values for a healthy cluster: Read amplification factor should be in the single digits. A value exceeding 50 for 1 hour strongly suggests that the LSM tree has an unhealthy shape.
• The STORAGE indicates an unhealthy LSM with the following:
  • Frequent compaction status messages.
  • High-read-amplification warnings, e.g., sstables (read amplification = 54).

​

Node health

• The on the Cluster Overview page indicates whether nodes are online (LIVE) or have crashed (SUSPECT or DEAD).
• The /health endpoint of the returns a 500 error when a node is unhealthy.
• A can notify when a node has been down for 15 minutes or more.

​

Memory

​

Memory planning

• Provision at least 4 GiB of RAM per vCPU.
• Disable Linux memory swapping. Over-allocating memory on production machines can lead to unexpected performance issues when pages have to be read back into memory.
• The default value for --cache is 128 MiB. For production deployments, set --cache to 25% or higher. To determine appropriate settings for --cache and --max-sql-memory, use the following formula:
  (2 * --max-sql-memory) + --cache <= 80% of system RAM
  

​

Memory monitoring

​

Node process restarts

• The on the Cluster Overview page indicates whether nodes are online (LIVE) or have crashed (SUSPECT or DEAD).
• When deploying on , the kubectl get pods output contains a RESTARTS column that tracks the number of restarts for each CockroachDB pod.
• The OPS will record a whenever a node rejoins the cluster after being offline.
• A can notify when a node has restarted more than once in the last 10 minutes.

Verify OOM errors

• Monitor dmesg to determine if a node crashed because it ran out of memory:
  $ sudo dmesg | grep -iC 3 "cockroach"
  
  The following output indicates the node crashed due to insufficient memory:
  $ host kernel: Out of Memory: Killed process <process_id> (cockroach).
  
• When deploying on , run kubectl logs {pod-name} and look for OOM errors in the log messages.

​

SQL memory usage

• The graph on the SQL dashboard shows the current amount of memory in KiB allocated to the SQL layer. Expected values for a healthy cluster: This value should not exceed the size. A healthy threshold is 75% of allocated --max-sql-memory.
• The “active query dump”, enabled by default with the diagnostics.active_query_dumps.enabled , is a record of anonymized active queries that is written to disk when a node is detected to be under memory pressure. You can use the active query dump to correlate specific queries to OOM crashes. Active query dumps have the filename activequeryprof.{date-and-time}.csv and are found in the heap_profiler directory in the configured . They are also included when running .
• A SHOW STATEMENTS statement can on the cluster that may be consuming excessive memory.
• A error in the logs indicates that --max-sql-memory, the memory allocated to the SQL layer, was exceeded by the operation referenced in the error. For guidance on resolving this issue, see .

CREATE TABLE name (first STRING(100), last STRING(100));

ALTER TABLE name ALTER first TYPE STRING(99);

​

Database memory usage

• The graph on the Runtime dashboard shows the total memory in use by CockroachDB processes. The RSS (resident set size) metric represents actual CockroachDB memory usage from the OS/Linux/pod point of view. The Go and CGo metrics represent memory allocation and total usage from a CockroachDB point of view. Expected values for a healthy cluster: Go Allocated will depend on workload but should not exceed by more than 100%. CGo Allocated should not exceed the size and CGo Total should not exceed the size by more than 15%.

​

Storage and disk I/O

​

Storage and disk planning

​

Storage and disk monitoring

​

Storage capacity

• The graph on the Overview and Storage dashboards shows the available and used disk capacity in the CockroachDB . Expected values for a healthy cluster: Used capacity should not persistently exceed 80% of the total capacity.
• A can notify when a node has less than 15% of free space remaining.

​

Disk IOPS

• The graph on the Hardware dashboard shows the number of disk reads and writes in queue. Expected values for a healthy cluster: This value should be 0 or single-digit values for short periods of time. If the values persist in double digits, you may have an I/O bottleneck.
• The Linux tool iostat (part of sysstat) can be used to monitor IOPS. In the device status output, avgqu-sz corresponds to the Disk Ops In Progress metric. If service times persist in double digits on any node, this means that your storage device is saturated and is likely under-provisioned or misconfigured.

• Degradation in SQL response time.
• An unhealthy LSM.

​

Node heartbeat latency

• The and graphs on the show the time elapsed between node heartbeats. Expected values for a healthy cluster: Less than 100ms in addition to the between nodes in the cluster.

• Whether a node is still a member of a cluster (this is used by ).
• Whether a node is dead (in which case ).
• How to avoid placing replicas on dead, decommissioning or unhealthy nodes, and to make decisions about lease transfers.

​

Command commit latency

• The Command Commit Latency: 50th percentile and Command Commit Latency: 99th percentile graphs on the show how quickly are being committed by nodes in the cluster. This is a good signal of I/O load. Expected values for a healthy cluster: On SSDs (), this should be between 1 and 100 milliseconds. On HDDs, this should be no more than 1 second.

​

See also

• (CockroachDB Advanced)