Changefeed Monitoring Guide

CockroachDB are long-running that work as a pipeline of change events emitted from a through to the . The change events from the of CockroachDB progress through the changefeed pipeline and are encoded into that the changefeed job delivers to a sink, such as Kafka. This guide provides recommendations for monitoring and alerting on changefeeds throughout the pipeline to ensure reliable operation and quick problem detection.

​

Overview

To monitor changefeed jobs effectively, it is necessary to have a view of the high-level metrics that track the health of the changefeed overall and metrics that track the different stages of the changefeed pipeline. The changefeed pipeline contains three main sections that start at the of CockroachDB and end at the with message delivery. The work completed in each of these pipeline sections feeds metrics that you can track and use to identify where issues could occur.

• Rangefeed: Connects changefeeds to a long-lived request called a , which pushes changes as they happen.
• Processing: Prepares the change events from the rangefeed into by encoding messages into the specified .
• Sink: Delivers changefeed messages to the .

Where noted in the following sections, you can use changefeed to measure metrics per changefeed. You can enable Datadog integration on your cluster to collect data and alert on selected CockroachDB metrics using the Datadog platform.

​

Overall pipeline metrics

​

High-level performance metrics

• Metrics:
  • changefeed.max_behind_nanos
    • Description: The maximum lag in nanoseconds between the timestamp of the most recent emitted by the changefeed and the current time. Indicates how far behind the changefeed is in processing changes.
    • Use with .
    • Investigation needed: If changefeed.max_behind_nanos is consistently increasing.
  • (now() - changefeed.checkpoint_progress)
    • Description: The progress of changefeed . Indicates how recently the changefeed state was persisted durably. Critical for monitoring changefeed .
    • Investigation needed: If checkpointing falls too far behind the current time.
• Impact:
  • Slow processing of changes and updates to downstream sinks.
  • Recovery time after failures, potential for increase in change backlog.
  • Ability to resume from last known good state.
  • Resource usage during catch-up after restarts or failures.

​

End-to-end performance metrics

​

End-to-end commit latency

• Metric: changefeed.commit_latency
• Description: The time between when a commits and when the changefeed emits the corresponding changes. This time includes rangefeed and buffer delays.
• Use with .

​

Batch latency

• Metric: changefeed.sink_batch_hist_nanos
• Description: The end-to-end latency from the change event being put in a batch to the changefeed emitting the event message to the sink.
• Investigation needed: If this metric has consistently high values, it indicates that there are bottlenecks in the pipeline.

​

Protected timestamp age

• Metric: jobs.changefeed.protected_age_sec
• Description: The age of the record for the changefeed.
• Investigation needed: If this metric has high values, it may indicate changefeed processing delays or resource constraints.
• Impact: High values can and lead to increased disk usage and degraded performance for some workloads.

​

Progress tracking

• Metrics:
  • high_water_timestamp from
  • (now() - changefeed.aggregator_progress)
    • Use with .
  • changefeed.max_behind_nanos
    • Use with .
  • (now() - changefeed.checkpoint_progress)
• Description: These metrics help to measure how far behind the changefeed is from the current time.
• Investigation: A growing delay indicates changefeed processing and messages emitting cannot keep up with the rate of change events.

​

Pipeline section metrics

​

Rangefeed

​

Buffer memory usage

• Metric: changefeed.buffer_entries.allocated_mem.rangefeed
• Description: The current quota pool memory allocation between the rangefeed and the KV feed.
• Impact: High memory usage may indicate backpressure.

​

Rangefeed buffer latency

• Metrics:
  • changefeed.stage.rangefeed_buffer_checkpoint.latency (count/sum/bucket)
  • changefeed.stage.rangefeed_buffer_value.latency (count/sum/bucket)
• Description: Latency within the rangefeed section of the changefeed pipeline.
• Impact: Indicates potential scanning or changefeed catch-up issues.
• Use with .

​

KV feed

• Metric: changefeed.stage.kv_feed_wait_for_table_event.latency (count/sum/bucket)
• Description: Latency within the processing section of the changefeed pipeline.
• Impact: Potential bottlenecks in encoding, batching, sending data, or at the . Use this metric in conjunction with the downstream delivery metrics.

​

Processing — aggregation and encoding

​

Aggregator buffer health

• Metrics:
  • changefeed.buffer_entries.allocated_mem.aggregator
  • changefeed.stage.kv_feed_buffer.latency (count/sum/bucket)
• Description: The current quota pool memory allocation between the KV feed and the sink. Latency within the processing section of the changefeed pipeline.
• Use with .

​

Encoding performance

• Metric: changefeed.stage.encode.latency (count/sum/bucket)
• Description: Latency encoding data within the processing section of the changefeed pipeline.
• Impact: High encoding latency can create a bottleneck for the entire changefeed pipeline.
• Use with .

​

Sink

​

Sink performance

• Metric: changefeed.parallel_io_queue_nanos
• Description: The time that outgoing requests to the sink spend waiting in a queue due to in-flight requests with conflicting keys.
• Use with .

​

Sink errors

• Metric: changefeed.sink_errors
• Description: The number of changefeed errors caused by the sink.
• Impact: Indicates connectivity or downstream processing issues.
• Use with .

​

Downstream delivery

• Metrics:
  • changefeed.stage.downstream_client_send.latency (count/sum/bucket)
  • changefeed.internal_retry_message_count
• Description: Latency when flushing messages from a sink’s client to the downstream sink. (This includes sends that failed for most, but not all sinks.) The number of messages for which an aggregator node attempted to retry.
• Impact: Indicates connectivity or downstream processing issues.
• Scoped by changefeed_job_id

​

See also