> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Transaction Retry Error Reference

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

When a <InternalLink path="transactions">transaction</InternalLink> is unable to complete due to <InternalLink path="performance-best-practices-overview">contention</InternalLink> with another concurrent or recent transaction attempting to write to the same data, CockroachDB will <InternalLink path="transactions#automatic-retries">automatically attempt to retry the failed transaction</InternalLink> without involving the client (i.e., silently). If the automatic retry is not possible or fails, a *transaction retry error* is emitted to the client.

Transaction retry errors fall into two categories:

* *Serialization errors* indicate that a transaction failed because it could not be placed into a serializable ordering among all of the currently-executing transactions.
  * <InternalLink path="demo-serializable">`SERIALIZABLE`</InternalLink> transactions must address these errors with client-side intervention, where the client [initiates a restart of the transaction](#client-side-retry-handling), and [adjusts application logic and tunes queries](#minimize-transaction-retry-errors) for greater performance.
  * <InternalLink path="read-committed">`READ COMMITTED`</InternalLink> transactions do **not** return <InternalLink path="transaction-retry-error-reference">serialization errors</InternalLink> that require client-side handling.
* *Internal state errors* indicate that the cluster itself is experiencing an issue, such as being <InternalLink path="ui-overload-dashboard">overloaded</InternalLink>, which prevents the transaction from completing. These errors generally require both cluster-side and client-side intervention, where an operator addresses an issue with the cluster before the client then [initiates a restart of the transaction](#client-side-retry-handling).

All transaction retry errors use the `SQLSTATE` error code `40001`, and emit error messages with the string <InternalLink path="common-errors#restart-transaction">`restart transaction`</InternalLink>. Further, each error includes a [specific error code](#transaction-retry-error-reference) to assist with targeted troubleshooting.

When experiencing transaction retry errors, you should follow the guidance under [Actions to take](#actions-to-take), and then consult the reference for your [specific transaction retry error](#transaction-retry-error-reference) for guidance specific to the error message encountered.

## Overview

At the default `SERIALIZABLE` isolation level, CockroachDB always attempts to find a <InternalLink path="demo-serializable">serializable ordering</InternalLink> among all of the currently-executing transactions.

Whenever possible, CockroachDB will <InternalLink path="transactions#automatic-retries">auto-retry a transaction internally</InternalLink> without notifying the client. CockroachDB will only send a serialization error to the client when it cannot resolve the error automatically without client-side intervention.

<InternalLink path="read-committed">`READ COMMITTED`</InternalLink> transactions can transparently resolve serialization errors by <InternalLink path="architecture/transaction-layer#read-snapshots">retrying individual statements</InternalLink> rather than entire transactions.

## Actions to take

In most cases, the correct actions to take when encountering transaction retry errors are:

1. Under `SERIALIZABLE` isolation, update your application to support <InternalLink path="transaction-retry-error-reference#client-side-retry-handling">client-side retry handling</InternalLink> when transaction retry errors are encountered. Follow the guidance for the <InternalLink path="transaction-retry-error-reference#transaction-retry-error-reference">specific error type</InternalLink>.
2. Take steps to <InternalLink path="transaction-retry-error-reference#minimize-transaction-retry-errors">minimize transaction retry errors</InternalLink> in the first place. This means reducing transaction contention overall, and increasing the likelihood that CockroachDB can <InternalLink path="transactions#automatic-retries">automatically retry</InternalLink> a failed transaction.

### Client-side retry handling

When running under <InternalLink path="demo-serializable">`SERIALIZABLE`</InternalLink> isolation, your application should include client-side retry handling when the statements are sent individually, such as:

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> BEGIN;

> UPDATE products SET inventory = 0 WHERE sku = '8675309';

> INSERT INTO orders (customer, status) VALUES (1, 'new');

> COMMIT;
```

To indicate that a transaction must be retried, CockroachDB signals a serialization error with the `SQLSTATE` error code `40001` and an error message that begins with the string `"restart transaction"`.

To handle these types of errors, you have the following options:

* If your database library or framework provides a method for retryable transactions (it will often be documented as a tool for handling deadlocks), use it.
* If you're building an application in the following languages, Cockroach Labs has created adapters that include automatic retry handling:
  * **Go** developers using [GORM](https://github.com/jinzhu/gorm) or [pgx](https://github.com/jackc/pgx) can use the [`github.com/cockroachdb/cockroach-go/crdb`](https://github.com/cockroachdb/cockroach-go/tree/master/crdb) package. For an example, see <InternalLink path="build-a-go-app-with-cockroachdb">Build a Go App with CockroachDB</InternalLink>.
  * **Python** developers using [SQLAlchemy](https://www.sqlalchemy.org/) can use the [`sqlalchemy-cockroachdb` adapter](https://github.com/cockroachdb/sqlalchemy-cockroachdb). For an example, see <InternalLink path="build-a-python-app-with-cockroachdb-sqlalchemy">Build a Python App with CockroachDB and SQLAlchemy</InternalLink>.
  * **Ruby (Active Record)** developers can use the [`activerecord-cockroachdb-adapter`](https://rubygems.org/gems/activerecord-cockroachdb-adapter). For an example, see <InternalLink path="build-a-ruby-app-with-cockroachdb-activerecord">Build a Ruby App with CockroachDB and Active Record</InternalLink>.
* If you're building an application with another driver or data access framework that is <InternalLink path="third-party-database-tools">supported by CockroachDB</InternalLink>, we recommend reusing the retry logic in our <InternalLink path="example-apps">"Simple CRUD" Example Apps</InternalLink>. For example, **Java** developers accessing the database with [JDBC](https://jdbc.postgresql.org/) can reuse the example code implementing retry logic shown in <InternalLink path="build-a-java-app-with-cockroachdb">Build a Java app with CockroachDB</InternalLink>.
* If you're building an application with a language and framework for which we do not provide example retry logic, you might need to write your own retry logic. For an example, see the <InternalLink path="transaction-retry-error-example">Client-side retry handling example</InternalLink>.
* **Advanced users, such as library authors**: See <InternalLink path="advanced-client-side-transaction-retries">Advanced Client-Side Transaction Retries</InternalLink>.

#### Client-side retry handling example

For a conceptual example of application-defined retry logic, and testing that logic against your application's needs, see the <InternalLink path="transaction-retry-error-example">client-side retry handling example</InternalLink>.

### Minimize transaction retry errors

In addition to the steps described in [Client-side retry handling](#client-side-retry-handling), which detail how to configure your application to restart a failed transaction, there are also a number of changes you can make to your application logic to reduce the number of transaction retry errors that reach the client application under `SERIALIZABLE` isolation.

Reduce failed transactions caused by <InternalLink path="architecture/transaction-layer#timestamp-cache">timestamp pushes</InternalLink> or <InternalLink path="architecture/transaction-layer#read-refreshing">read invalidation</InternalLink>:

* Limit the number of affected rows by following <InternalLink path="apply-statement-performance-rules">optimizing queries</InternalLink> (e.g., avoiding full scans, creating secondary indexes, etc.). Not only will transactions run faster, lock fewer rows, and hold locks for a shorter duration, but the chances of <InternalLink path="architecture/transaction-layer#read-refreshing">read invalidation</InternalLink> when the transaction's <InternalLink path="architecture/transaction-layer#timestamp-cache">timestamp is pushed</InternalLink>, due to a conflicting write, are decreased because of a smaller read set (i.e., a smaller number of rows read).
* Break down larger transactions (e.g., <InternalLink path="bulk-delete-data">bulk deletes</InternalLink>) into smaller ones to have transactions hold locks for a shorter duration. For example, use <InternalLink path="common-table-expressions">common table expressions</InternalLink> to group multiple clauses together in a single SQL statement. This will also decrease the likelihood of <InternalLink path="architecture/transaction-layer#timestamp-cache">pushed timestamps</InternalLink>. For instance, as the size of writes (number of rows written) decreases, the chances of the transaction's timestamp getting bumped by concurrent reads decreases.
* Use <InternalLink path="select-for-update">`SELECT FOR UPDATE`</InternalLink> to aggressively lock rows that will later be updated in the transaction. Updates must operate on the most recent version of a row, so a concurrent write to the row will cause a retry error (<InternalLink path="transaction-retry-error-reference#retry_write_too_old">`RETRY_WRITE_TOO_OLD`</InternalLink>). Locking early in the transaction forces concurrent writers to block until the transaction is finished, which prevents the retry error. Note that this locks the rows for the duration of the transaction; whether this is tenable will depend on your workload. For more information, see [When and why to use `SELECT FOR UPDATE` in CockroachDB](https://www.cockroachlabs.com/blog/when-and-why-to-use-select-for-update-in-cockroachdb).
* Use historical reads (<InternalLink path="as-of-system-time">`SELECT... AS OF SYSTEM TIME`</InternalLink>), preferably <InternalLink path="follower-reads#when-to-use-bounded-staleness-reads">bounded staleness reads</InternalLink> or <InternalLink path="follower-reads#run-queries-that-use-exact-staleness-follower-reads">exact staleness with follower reads</InternalLink> when possible to reduce conflicts with other writes. This reduces the likelihood of <InternalLink path="transaction-retry-error-reference#retry_serializable">`RETRY_SERIALIZABLE`</InternalLink> errors as fewer writes will happen at the historical timestamp. More specifically, writes' timestamps are less likely to be pushed by historical reads as they would <InternalLink path="architecture/transaction-layer#transaction-conflicts">when the read has a higher priority level</InternalLink>. Note that if the `AS OF SYSTEM TIME` value is below the closed timestamp, the read cannot be invalidated.
* When replacing values in a row, use <InternalLink path="upsert">`UPSERT`</InternalLink> and specify values for all columns in the inserted rows. This will usually have the best performance under contention, compared to combinations of <InternalLink path="select-clause">`SELECT`</InternalLink>, <InternalLink path="insert">`INSERT`</InternalLink>, and <InternalLink path="update">`UPDATE`</InternalLink>.
* If applicable to your workload, assign <InternalLink path="column-families#default-behavior">column families</InternalLink> and separate columns that are frequently read and written into separate columns. Transactions will operate on disjoint column families and reduce the likelihood of conflicts.
* For workloads where large <InternalLink path="update">`UPDATE`</InternalLink> or <InternalLink path="insert">`INSERT`</InternalLink> transactions run concurrently over similar key ranges, watch for <InternalLink path="architecture/transaction-layer#transaction-records">transaction record</InternalLink> anchor hotspots (for example, many concurrent transactions with <InternalLink path="architecture/transaction-layer#transaction-records">records</InternalLink> on the same <InternalLink path="architecture/glossary#range">range</InternalLink>). In these cases, consider enabling the <InternalLink path="cluster-settings">`transaction.randomized_anchor_key.enabled`</InternalLink> cluster setting to randomize the location of transaction anchor keys. This can spread transaction records across ranges and reduce hotspotting. Only use this setting after confirming anchor hotspots via contention and range-level observability.
* As a last resort, consider adjusting the <InternalLink path="architecture/transaction-layer#closed-timestamps">closed timestamp interval</InternalLink> using the `kv.closed_timestamp.target_duration` <InternalLink path="cluster-settings">cluster setting</InternalLink> to reduce the likelihood of long-running write transactions having their <InternalLink path="architecture/transaction-layer#timestamp-cache">timestamps pushed</InternalLink>. This setting should be carefully adjusted if **no other mitigations are available** because there can be downstream implications (e.g., historical reads, change data capture feeds, statistics collection, handling zone configurations, etc.). For example, a transaction *A* is forced to refresh (i.e., change its timestamp) due to hitting the maximum <InternalLink path="architecture/transaction-layer#closed-timestamps">*closed timestamp*</InternalLink> interval (closed timestamps enable <InternalLink path="follower-reads#how-stale-follower-reads-work">Follower Reads</InternalLink> and <InternalLink path="change-data-capture-overview">Change Data Capture (CDC)</InternalLink>). This can happen when transaction *A* is a long-running transaction, and there is a write by another transaction to data that *A* has already read.

<Note>
  If you increase the `kv.closed_timestamp.target_duration` setting, it means that you are increasing the amount of time
  by which the data available in <InternalLink path="follower-reads">Follower Reads</InternalLink> and <InternalLink path="change-data-capture-overview">CDC
  changefeeds</InternalLink> lags behind the current state of the cluster. In other words,
  there is a trade-off here: if you absolutely must execute long-running transactions that execute concurrently with
  other transactions that are writing to the same data, you may have to settle for longer delays on Follower Reads
  and/or CDC to avoid frequent serialization errors. The anomaly that would be exhibited if these transactions were not
  retried is called [write skew](https://www.cockroachlabs.com/blog/what-write-skew-looks-like).
  Increase the chance that CockroachDB can <InternalLink path="transactions#automatic-retries">automatically retry</InternalLink> a failed transaction:
</Note>

* <InternalLink path="transactions#batched-statements">Send statements in transactions as a single batch</InternalLink>. Batching allows CockroachDB to <InternalLink path="transactions#automatic-retries">automatically retry</InternalLink> a transaction when <InternalLink path="architecture/transaction-layer#read-refreshing">previous reads are invalidated</InternalLink> at a <InternalLink path="architecture/transaction-layer#timestamp-cache">pushed timestamp</InternalLink>. When a multi-statement transaction is not batched, and takes more than a single round trip, CockroachDB cannot automatically retry the transaction. For an example showing how to break up large transactions in an application, see <InternalLink path="build-a-python-app-with-cockroachdb-sqlalchemy#break-up-large-transactions-into-smaller-units-of-work">Break up large transactions into smaller units of work</InternalLink>.

<a id="result-buffer-size" />

* Limit the size of the result sets of your transactions to less than the value of the <InternalLink path="cluster-settings">`sql.defaults.results_buffer.size` cluster setting</InternalLink>, so that CockroachDB is more likely to <InternalLink path="transactions#automatic-retries">automatically retry</InternalLink> when <InternalLink path="architecture/transaction-layer#read-refreshing">previous reads are invalidated</InternalLink> at a <InternalLink path="architecture/transaction-layer#timestamp-cache">pushed timestamp</InternalLink>. When a transaction returns a result set larger than the configured buffer size, even if that transaction has been sent as a single batch, CockroachDB cannot automatically retry the transaction. You can change the results buffer size for all new sessions using the <InternalLink path="cluster-settings">`sql.defaults.results_buffer.size` cluster setting</InternalLink>, or for a specific session using the `results_buffer_size` <InternalLink path="connection-parameters#additional-connection-parameters">connection parameter</InternalLink>.

### Interpreting log messages

In CockroachDB v25.4, the `meta={... key=/Table/...}` field that appears in log output for serialization conflicts identifies the transaction's <InternalLink path="architecture/transaction-layer#transaction-records">transaction record key</InternalLink> (also known as the *anchor key*), not necessarily the key where the conflict occurred. This anchor key is the first key written by the transaction and is where its record is stored.

<InternalLink path="crdb-internal#view-all-contention-events">Contention events</InternalLink> that are recorded when <InternalLink path="cluster-settings">`sql.contention.record_serialization_conflicts.enabled`</InternalLink> is `true` use this anchor key when populating the recorded conflict.

Only the following error types may add a conflicting key to a contention event:

* `TransactionRetryError`
* `WriteTooOld`
* `ExclusionViolationError`

## Transaction retry error reference

Note that your application's retry logic does not need to distinguish between the different types of serialization errors. They are listed here for reference during <InternalLink path="performance-recipes#transaction-contention">advanced troubleshooting</InternalLink>.

* [`RETRY_WRITE_TOO_OLD`](#retry_write_too_old)
* [`RETRY_SERIALIZABLE`](#retry_serializable)
* [`RETRY_ASYNC_WRITE_FAILURE`](#retry_async_write_failure)
* [`ReadWithinUncertaintyIntervalError`](#readwithinuncertaintyintervalerror)
* [`RETRY_COMMIT_DEADLINE_EXCEEDED`](#retry_commit_deadline_exceeded)
* [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found)
* [`ABORT_REASON_CLIENT_REJECT`](#abort_reason_client_reject)
* [`ABORT_REASON_PUSHER_ABORTED`](#abort_reason_pusher_aborted)
* [`ABORT_REASON_ABORT_SPAN`](#abort_reason_abort_span)
* [`ABORT_REASON_NEW_LEASE_PREVENTS_TXN`](#abort_reason_new_lease_prevents_txn)
* [`ABORT_REASON_TIMESTAMP_CACHE_REJECTED`](#abort_reason_timestamp_cache_rejected)
* [injected by `inject_retry_errors_enabled` session variable](#injected-by-inject_retry_errors_enabled-session-variable)

Each transaction retry error listed includes an example error as it would appear from the context of the client, a description of the circumstances that cause that error, and specific guidance for addressing the error.

### RETRY\_WRITE\_TOO\_OLD

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError: ... RETRY_WRITE_TOO_OLD ...
```

**Error type:** Serialization error

**Description:**

The `RETRY_WRITE_TOO_OLD` error occurs when a transaction *A* tries to write to a row *R*, but another transaction *B* that was supposed to be serialized after *A* (i.e., had been assigned a higher timestamp), has already written to that row *R*, and has already committed. Under `SERIALIZABLE` isolation, this is a common error when you have too much contention in your workload.

**Action:**

Under <InternalLink path="demo-serializable">`SERIALIZABLE`</InternalLink> isolation:

1. Retry transaction *A* as described in [client-side retry handling](#client-side-retry-handling).
2. Adjust your application logic as described in [minimize transaction retry errors](#minimize-transaction-retry-errors). In particular, try to:
   1. Send all of the statements in your transaction in a <InternalLink path="transactions#batched-statements">single batch</InternalLink>.
   2. Use <InternalLink path="select-for-update">`SELECT FOR UPDATE`</InternalLink> to aggressively lock rows that will later be updated in the transaction.

Under <InternalLink path="read-committed">`READ COMMITTED`</InternalLink> isolation:

1. `RETRY_WRITE_TOO_OLD` errors are only returned in rare cases that can be avoided by adjusting the [result buffer size](#result-buffer-size).

### RETRY\_SERIALIZABLE

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError: ... RETRY_SERIALIZABLE ...
```

The error message for `RETRY_SERIALIZABLE` contains additional information about the transaction conflict which led to the error, as shown below. This error message can also be viewed in the <InternalLink path="ui-overview">DB Console</InternalLink> by navigating to <InternalLink path="ui-insights-page#failed-execution">**Insights Page** → **Workload Insights** → **Transaction Executions** and clicking on the transaction ID to see the **Failed Execution** insight</InternalLink>.

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE  - failed preemptive refresh due to conflicting locks on /Table/106/1/918951292305080321/0 [reason=wait_policy] - conflicting txn: meta={id=1b2bf263 key=/Table/106/1/918951292305080321/0 iso=Serializable pri=0.00065863 epo=0 ts=1700512205.521833000,2 min=1700512148.761403000,0 seq=1}): "sql txn" meta={id=07d42834 key=/Table/106/1/918951292305211393/0 iso=Serializable pri=0.01253025 epo=0 ts=1700512229.378453000,2 min=1700512130.342117000,0 seq=2} lock=true stat=PENDING rts=1700512130.342117000,0 wto=false gul=1700512130.842117000,0
SQLSTATE: 40001
HINT: See: https://www.cockroachlabs.com/docs/v25.4/transaction-retry-error-reference.html#retry_serializable
```

**Error type:** Serialization error

**Description:**

<Tip>
  <InternalLink path="read-committed">`READ COMMITTED`</InternalLink> transactions do **not** produce `RETRY_SERIALIZABLE` errors.
</Tip>

At a high level, the `RETRY_SERIALIZABLE` error occurs when a transaction's timestamp is moved forward, but the transaction performed reads at the old timestamp that are no longer valid at its new timestamp. More specifically, the `RETRY_SERIALIZABLE` error occurs in the following three cases under <InternalLink path="demo-serializable">`SERIALIZABLE`</InternalLink> isolation:

1. When a transaction *A* has its timestamp moved forward (also known as *A* being "pushed") as CockroachDB attempts to find a serializable transaction ordering. Specifically, transaction *A* tried to write a key that transaction *B* had already read, and *B* was supposed to be serialized after *A* (i.e., *B* had a higher timestamp than *A*). CockroachDB will try to serialize *A* after *B* by changing *A*'s timestamp, but it cannot do that when another transaction has subsequently written to some of the keys that *A* has read and returned to the client. When that happens, the `RETRY_SERIALIZATION` error is signalled. For more information about how timestamp pushes work in our transaction model, see the <InternalLink path="architecture/transaction-layer#timestamp-cache">architecture docs on the transaction layer's timestamp cache</InternalLink>.
2. When a <InternalLink path="transactions#transaction-priorities">high-priority transaction</InternalLink> *A* does a read that runs into a write intent from another lower-priority transaction *B*, and some other transaction *C* writes to a key that *B* has already read. Transaction *B* will get this error when it tries to commit, because *A* has already read some of the data touched by *B* and returned results to the client, and *C* has written data previously read by *B*.
3. When a transaction *A* is forced to refresh (i.e., change its timestamp) due to hitting the maximum <InternalLink path="architecture/transaction-layer#closed-timestamps">*closed timestamp*</InternalLink> interval (closed timestamps enable <InternalLink path="follower-reads#how-stale-follower-reads-work">Follower Reads</InternalLink> and <InternalLink path="change-data-capture-overview">Change Data Capture (CDC)</InternalLink>). This can happen when transaction *A* is a long-running transaction, and there is a write by another transaction to data that *A* has already read. Unfortunately, there is no indication from this error code that a too-low closed timestamp setting is the issue. Therefore, you may need to rule out cases 1 and 2.

*Failed preemptive refresh*

<a id="failed_preemptive_refresh" />

In the three preceding cases, CockroachDB will try to validate whether the read-set of the transaction that had its timestamp (`timestamp1`) pushed is still valid at the new timestamp (`timestamp3`) at commit time. This mechanism is called "performing a <InternalLink path="architecture/transaction-layer#read-refreshing">read refresh</InternalLink>". If the read-set is still valid, the transaction can commit. If it is not valid, the transaction will get a `RETRY_SERIALIZABLE - failed preemptive refresh` error. The refresh can fail for two reasons:

1. There is a committed value on a key that was read by the transaction at `timestamp2` (where `timestamp2` occurs between `timestamp1` and `timestamp3` ). The error message will contain `due to encountered recently written committed value`. CockroachDB does not have any information about which conflicting transaction wrote to this key.
2. There is an intent on a key that was read by the transaction at `timestamp2` (where `timestamp2` occurs between `timestamp1` and `timestamp3` ). The error message will contain `due to conflicting locks`. CockroachDB does have information about the conflicting transaction to which the intent belongs. The information about the <InternalLink path="ui-insights-page#serialization-conflict-due-to-transaction-contention">conflicting transaction</InternalLink> can be seen on the <InternalLink path="ui-insights-page">DB Console Insights page</InternalLink>.

**Action:**

Under <InternalLink path="demo-serializable">`SERIALIZABLE`</InternalLink> isolation:

1. Retry transaction *A* as described in [client-side retry handling](#client-side-retry-handling).
2. Adjust your application logic as described in [minimize transaction retry errors](#minimize-transaction-retry-errors). In particular, try to:
   1. Send all of the statements in your transaction in a <InternalLink path="transactions#batched-statements">single batch</InternalLink>.
   2. Use historical reads with <InternalLink path="as-of-system-time">`SELECT... AS OF SYSTEM TIME`</InternalLink>.
   3. Use <InternalLink path="select-for-update">`SELECT FOR UPDATE`</InternalLink> to aggressively lock rows for the keys that were read and could not be refreshed.

### RETRY\_ASYNC\_WRITE\_FAILURE

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError: ... RETRY_ASYNC_WRITE_FAILURE ...
```

**Error type:** Internal state error

**Description:**

The `RETRY_ASYNC_WRITE_FAILURE` error occurs when some kind of problem with your cluster's operation occurs at the moment of a previous write in the transaction, causing CockroachDB to fail to replicate one of the transaction's writes. This can happen if a <InternalLink path="architecture/replication-layer#leases">lease transfer</InternalLink> occurs while the transaction is executing, or less commonly if you have a <InternalLink path="cluster-setup-troubleshooting#network-partition">network partition</InternalLink> that cuts off access to some nodes in your cluster.

**Action:**

1. Retry the transaction as described in [client-side retry handling](#client-side-retry-handling). This is worth doing because the problem with the cluster is likely to be transient.
2. Investigate the problems with your cluster. For cluster troubleshooting information, see <InternalLink path="cluster-setup-troubleshooting">Troubleshoot Cluster Setup</InternalLink>.

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ReadWithinUncertaintyIntervalError

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError:
        read at time 1591009232.376925064,0 encountered previous write with future timestamp 1591009232.493830170,0 within uncertainty interval `t <= 1591009232.587671686,0`;
        observed timestamps: [{1 1591009232.587671686,0} {5 1591009232.376925064,0}] meta={key=/Table/9373/10/5293921467191001339/0 ...}
```

**Error type:** Serialization error

**Description:**

The `ReadWithinUncertaintyIntervalError` can occur when two transactions which start on different gateway nodes attempt to operate on the same data at close to the same time, and one of the operations is a write. The uncertainty comes from the fact that we cannot tell which one started first - the clocks on the two gateway nodes may not be perfectly in sync.

For example, if the clock on node *A* is ahead of the clock on node *B*, a transaction started on node *A* may be able to commit a write with a timestamp that is still in the "future" from the perspective of node *B*. A later transaction that starts on node *B* should be able to see the earlier write from node *A*, even if *B*'s clock has not caught up to *A*. The "read within uncertainty interval" occurs if we discover this situation in the middle of a transaction, when it is too late for the database to handle it automatically. When node *B*'s transaction retries, it will unambiguously occur after the transaction from node *A*.

<Note>
  This behavior is non-deterministic: it depends on which node is the
  <InternalLink path="architecture/life-of-a-distributed-transaction#leaseholder-node">leaseholder</InternalLink> of the underlying data
  range. It’s generally a sign of contention. Uncertainty errors are always possible with near-realtime reads under
  contention.
  **Action:**
</Note>

Under <InternalLink path="demo-serializable">`SERIALIZABLE`</InternalLink> isolation:

1. Be prepared to retry on uncertainty (and other) errors, as described in [client-side retry handling](#client-side-retry-handling).
2. Adjust your application logic as described in [minimize transaction retry errors](#minimize-transaction-retry-errors). In particular, try to:
   1. Send all of the statements in your transaction in a <InternalLink path="transactions#batched-statements">single batch</InternalLink>.
   2. Use historical reads with <InternalLink path="as-of-system-time">`SELECT... AS OF SYSTEM TIME`</InternalLink>.
3. If you <InternalLink path="operational-faqs">trust your clocks</InternalLink>, you can try lowering the <InternalLink path="cockroach-start#flags">`--max-offset` option to `cockroach start`</InternalLink>, which provides an upper limit on how long a transaction can continue to restart due to uncertainty.

Under <InternalLink path="read-committed">`READ COMMITTED`</InternalLink> isolation:

1. `ReadWithinUncertaintyIntervalError` errors are only returned in rare cases that can be avoided by adjusting the [result buffer size](#result-buffer-size).

<Note>
  Uncertainty errors are a sign of transaction conflict. For more information about transaction conflicts, see
  <InternalLink path="architecture/transaction-layer#transaction-conflicts">Transaction conflicts</InternalLink>.
</Note>

### RETRY\_COMMIT\_DEADLINE\_EXCEEDED

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError: TransactionRetryError: transaction deadline exceeded ...
```

**Error type:** Internal state error

**Description:**

The `RETRY_COMMIT_DEADLINE_EXCEEDED` error means that the transaction timed out due to being pushed by other concurrent transactions. This error is most likely to happen to long-running transactions. The conditions that trigger this error are very similar to the conditions that lead to a [`RETRY_SERIALIZABLE`](#retry_serializable) error, except that a transaction that hits this error got pushed for several minutes, but did not hit any of the conditions that trigger a `RETRY_SERIALIZABLE` error. In other words, the conditions that trigger this error are a subset of those that trigger `RETRY_SERIALIZABLE`, and that this transaction ran for too long (several minutes).

<Note>
  Read-only transactions do not get pushed, so they do not run into this error.
</Note>

This error occurs in the cases described below.

1. When a transaction *A* has its timestamp moved forward (also known as *A* being "pushed") as CockroachDB attempts to find a serializable transaction ordering. Specifically, transaction *A* tried to write a key that transaction *B* had already read. *B* was supposed to be serialized after *A* (i.e., *B* had a higher timestamp than *A*). CockroachDB will try to serialize *A* after *B* by changing *A*'s timestamp.
2. When a <InternalLink path="transactions#transaction-priorities">high-priority transaction</InternalLink> *A* does a read that runs into a write intent from another lower-priority transaction *B*. Transaction *B* may get this error when it tries to commit, because *A* has already read some of the data touched by *B* and returned results to the client.
3. When a transaction *A* is forced to refresh (change its timestamp) due to hitting the maximum <InternalLink path="architecture/transaction-layer#closed-timestamps">*closed timestamp*</InternalLink> interval (closed timestamps enable <InternalLink path="follower-reads#how-stale-follower-reads-work">Follower Reads</InternalLink> and <InternalLink path="change-data-capture-overview">Change Data Capture (CDC)</InternalLink>). This can happen when transaction *A* is a long-running transaction, and there is a write by another transaction to data that *A* has already read.

**Action:**

1. The `RETRY_COMMIT_DEADLINE_EXCEEDED` error is one case where the standard advice to add a retry loop to your application may not be advisable. A transaction that runs for long enough to get pushed beyond its deadline is quite likely to fail again on retry for the same reasons. Therefore, the best thing to do in this case is to shrink the running time of your transactions so they complete more quickly and do not hit the deadline.
2. If you encounter case 3 above, you can increase the `kv.closed_timestamp.target_duration` setting to a higher value. Unfortunately, there is no indication from this error code that a too-low closed timestamp setting is the issue. Therefore, you may need to rule out cases 1 and 2 (or experiment with increasing the closed timestamp interval, if that is possible for your application - see the note below).

If you increase the `kv.closed_timestamp.target_duration` setting, it means that you are increasing the amount of time
by which the data available in <InternalLink path="follower-reads">Follower Reads</InternalLink> and <InternalLink path="change-data-capture-overview">CDC
changefeeds</InternalLink> lags behind the current state of the cluster. In other words,
there is a trade-off here: if you absolutely must execute long-running transactions that execute concurrently with
other transactions that are writing to the same data, you may have to settle for longer delays on Follower Reads
and/or CDC to avoid frequent serialization errors. The anomaly that would be exhibited if these transactions were not
retried is called [write skew](https://www.cockroachlabs.com/blog/what-write-skew-looks-like).
See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ABORT\_REASON\_ABORTED\_RECORD\_FOUND

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_ABORTED_RECORD_FOUND) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_ABORTED_RECORD_FOUND` error means that the client application is trying to use a transaction that has been aborted. This happens in one of the following cases:

* Write-write conflict: Another <InternalLink path="transactions#transaction-priorities">high-priority transaction</InternalLink> *B* encountered a write intent by our transaction *A*, and tried to push *A* 's timestamp.
* Cluster overload: *B* thinks that *A* 's transaction coordinator node is dead, because the coordinator node hasn't heartbeated the transaction record for a few seconds.
* Deadlock: Some transaction *B* is trying to acquire conflicting locks in reverse order from transaction *A*.

**Action:**

If you are encountering deadlocks:

* Avoid producing deadlocks in your application by making sure that transactions acquire locks in the same order.

If you are using only default <InternalLink path="transactions#transaction-priorities">transaction priorities</InternalLink>:

* This error means your cluster has problems. You are likely overloading it. Investigate the source of the overload, and do something about it. The best place to start investigating is the <InternalLink path="ui-overload-dashboard">**Overload Dashboard**</InternalLink>.

If you are using <InternalLink path="transactions#transaction-priorities">high- or low-priority transactions</InternalLink>:

1. Retry the transaction as described in [client-side retry handling](#client-side-retry-handling)
2. Adjust your application logic as described in [minimize transaction retry errors](#minimize-transaction-retry-errors).

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ABORT\_REASON\_CLIENT\_REJECT

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_CLIENT_REJECT) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_CLIENT_REJECT` error is caused by the same conditions as the [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found), and requires the same actions. The errors are fundamentally the same, except that they are discovered at different points in the process.

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ABORT\_REASON\_PUSHER\_ABORTED

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_PUSHER_ABORTED` error is caused by the same conditions as the [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found), and requires the same actions. The errors are fundamentally the same, except that they are discovered at different points in the process.

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ABORT\_REASON\_ABORT\_SPAN

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_ABORT_SPAN) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_ABORT_SPAN` error is caused by the same conditions as the [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found), and requires the same actions. The errors are fundamentally the same, except that they are discovered at different points in the process.

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ABORT\_REASON\_NEW\_LEASE\_PREVENTS\_TXN

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_NEW_LEASE_PREVENTS_TXN) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_NEW_LEASE_PREVENTS_TXN` error occurs because the <InternalLink path="architecture/transaction-layer#timestamp-cache">timestamp cache</InternalLink> will not allow transaction *A* to create a <InternalLink path="architecture/transaction-layer#transaction-records">transaction record</InternalLink>. A new lease wipes the timestamp cache, so this could mean the <InternalLink path="architecture/life-of-a-distributed-transaction#leaseholder-node">leaseholder</InternalLink> was moved and the duration of transaction *A* was unlucky enough to happen across a lease acquisition. In other words, leaseholders got shuffled out from underneath transaction *A* (due to no fault of the client application or schema design), and now it has to be retried.

**Action:**

Retry transaction *A* as described in [client-side retry handling](#client-side-retry-handling).

### ABORT\_REASON\_TIMESTAMP\_CACHE\_REJECTED

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_TIMESTAMP_CACHE_REJECTED` error occurs when the <InternalLink path="architecture/transaction-layer#timestamp-cache">timestamp cache</InternalLink> will not allow transaction *A* to create a <InternalLink path="architecture/transaction-layer#transaction-records">transaction record</InternalLink>. This can happen due to a <InternalLink path="architecture/distribution-layer#range-merges">range merge</InternalLink> happening in the background, or because the <InternalLink path="architecture/transaction-layer#timestamp-cache">timestamp cache</InternalLink> is an in-memory cache, and has outgrown its memory limit (about 64 MB).

**Action:**

Retry transaction *A* as described in [client-side retry handling](#client-side-retry-handling).

### injected by `inject_retry_errors_enabled` session variable

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
 TransactionRetryWithProtoRefreshError: injected by `inject_retry_errors_enabled` session variable
```

**Error type:** Internal state error

**Description:**

When the `inject_retry_errors_enabled` <InternalLink path="set-vars">session variable</InternalLink> is set to `true`, any statement (with the exception of <InternalLink path="set-vars">`SET` statements</InternalLink>) executed in the session inside of an explicit transaction will return this error.

For more details, see <InternalLink path="transaction-retry-error-example#test-transaction-retry-logic">Test transaction retry logic</InternalLink>.

**Action:**

To turn off error injection, set the `inject_retry_errors_enabled` session variable to `false`.

## See also

* <InternalLink path="common-errors">Common Errors and Solutions</InternalLink>
* <InternalLink path="transactions">Transactions</InternalLink>
* <InternalLink path="performance-best-practices-overview#transaction-contention">Transaction Contention</InternalLink>
* <InternalLink path="transaction-retry-error-example">Transaction Retry Error Example</InternalLink>
* <InternalLink path="ui-transactions-page">DB Console Transactions Page</InternalLink>
* <InternalLink path="architecture/transaction-layer">Architecture - Transaction Layer</InternalLink>