- All source data is migrated to the target .
- This approach utilizes .
- is supported, though this example will not use it. See for an example of a migration that uses failback replication.
Example scenario
You have a small (300 GB) database that provides the data store for a web application. You want to migrate the entirety of this database to a new CockroachDB cluster. Business cannot accommodate a full maintenance window, but it can accommodate a brief (<60 second) halt in traffic. The application runs on a Kubernetes cluster. Estimated system downtime: 3-5 minutes.Before the migration
- Install the tools.
- Review the and documentation.
- and .
- Recommended: Perform a dry run of this full set of instructions in a development environment that closely resembles your production environment. This can help you get a realistic sense of the time and complexity it requires.
- Understand the prerequisites and limitations of the MOLT tools:
Limitations
MOLT Fetch limitations
- Only tables with types of , , or can be sharded with .
-
GEOMETRYandGEOGRAPHYtypes are not supported. -
OID LOBtypes in PostgreSQL are not supported, although similar types likeBYTEAare supported.
MOLT Replicator limitations
- Replication modes require connection to the primary instance (PostgreSQL primary, MySQL primary/master, or Oracle primary). MOLT cannot obtain replication checkpoints or transaction metadata from replicas.
- Running DDL on the source or target while replication is in progress can cause replication failures.
TRUNCATEoperations on the source are not captured. OnlyINSERT,UPDATE,UPSERT, andDELETEevents are replicated.- Changes to virtual columns are not replicated automatically. To migrate these columns, you must define them explicitly with .
MOLT Verify limitations
- MOLT Verify compares 20,000 rows at a time by default, and row values can change between batches, potentially resulting in temporary inconsistencies in data. To configure the row batch size, use the
--row_batch_size. - MOLT Verify checks for collation mismatches on columns. This may cause validation to fail when a is used as a primary key and the source and target databases are using different .
- MOLT Verify might give an error in case of schema changes on either the source or target database.
- cannot yet be compared.
Step 1: Prepare the source database
In this step, you will:- Create a dedicated migration user on your source database.
- Configure the source database for replication.
Create migration user on source database
Create a dedicated migration user (for example,MIGRATION_USER) on the source database. This user is responsible for reading data from source tables during the migration. You will pass this username in the source connection string.
SUPERUSER role to the user (recommended for replication configuration):
ALTER TABLE command for each table to replicate.
Configure source database for replication
Connect to the primary instance (PostgreSQL primary, MySQL primary/master, or Oracle primary), not a replica. Replicas cannot provide the necessary replication checkpoints and transaction metadata required for ongoing replication.
wal_level to logical in postgresql.conf or in the SQL shell. For example:
Step 2: Prepare the target database
Define the target tables
Convert the source table definitions into CockroachDB-compatible equivalents. CockroachDB supports the PostgreSQL wire protocol and is largely .-
The source and target table definitions must match. Review to understand which source types can be mapped to CockroachDB types.
For example, a PostgreSQL source table defined as
CREATE TABLE migration_schema.tbl (pk INT PRIMARY KEY);must have a corresponding schema and table in CockroachDB:- MOLT Fetch can automatically define matching CockroachDB tables using the option.
- If you define the target tables manually, review how MOLT Fetch handles . You can use the MOLT Schema Conversion Tool to create matching table definitions.
-
Every table must have an explicit primary key.
Avoid using sequential keys. To learn more about the performance issues that can result from their use, refer to the . If a sequential key is necessary in your CockroachDB table, you must create it manually, after using to load and replicate the data.
- Review to understand how computed columns and partitioned tables can be mapped to the target, and how target tables can be renamed.
-
By default on CockroachDB,
INTis an alias forINT8, which creates 64-bit signed integers. PostgreSQL and MySQL default to 32-bit integers. Depending on your source database or application requirements, you may need to change the integer size to4. For more information, refer to .
Schema Conversion Tool
The (SCT) converts source table definitions to CockroachDB-compatible syntax. It requires a free .-
Upload a source
.sqlfile to convert the syntax and identify in the table definitions. -
Import the converted table definitions to a CockroachDB cluster:
- When migrating to CockroachDB Cloud, the Schema Conversion Tool automatically .
- When migrating to a self-hosted CockroachDB cluster, and pipe the directly into .
Drop constraints and indexes
To optimize data load performance, drop all non-PRIMARY KEY and on the target CockroachDB database before migrating:
- (you do not need to drop this constraint when using
drop-on-target-and-recreatefor table handling)
Do not drop constraints.
Create the SQL user
Create a SQL user in the CockroachDB cluster that has the necessary privileges. To create a usercrdb_user in the default database (you will pass this username in the target connection string):
migration_schema schema and internal MOLT tables like _molt_fetch_exceptions in the public CockroachDB schema:
Ensure that you are connected to the target database.
drop-on-target-and-recreate will not be used), grant the following privileges on the schema:
public CockroachDB schema:
drop-on-target-and-recreate, and the target schema has pre-existing tables that were created by a different user, you may need to change table ownership to allow the migration user to drop those tables. Make the following alteration for each table:
IMPORT INTO or COPY FROM on the target tables:
IMPORT INTO privileges
Grant SELECT, INSERT, and DROP (required because the table is taken offline during the IMPORT INTO) privileges on all tables being migrated:
EXTERNALIOIMPLICITACCESS :
COPY FROM privileges
Grant privileges to the user:
Set session variable
Ensure that thestatement_timeout is set to 0s for the user:
Replication privileges
Grant permissions to create the staging schema for replication:Configure GC TTL
Before starting the initial data load, configure the on the source CockroachDB cluster to ensure that historical data remains available when replication begins. The GC TTL must be long enough to cover the full duration of the data load. Increase the GC TTL before starting the data load. For example, to set the GC TTL to 24 hours:The GC TTL duration must be higher than your expected time for the initial data load.
Step 3: Load data into CockroachDB
In this step, you will:- Configure MOLT Fetch with the flags needed for your migration.
- Run MOLT Fetch.
- Understand how to continue a load after an interruption.
Configure MOLT Fetch
The includes detailed information about how to , and how to . When you runmolt fetch, you can configure the following options for data load:
- : Specify URL‑encoded source and target connections.
- : Specify schema and table names to migrate.
- : Export data to cloud storage or a local file server.
- : Specifies whether data will only be loaded into/from intermediate storage.
- : Divide larger tables into multiple shards during data export.
- : Choose between
IMPORT INTOandCOPY FROM. - : Determine how existing target tables are initialized before load.
- : Define any row-level transformations to apply to the data before it reaches the target.
- : Configure metrics collection during initial data load.
molt fetch command and its flags. Follow , especially those related to security.
At minimum, the molt fetch command should include the source, target, data path, and flags:
Run MOLT Fetch
Perform the initial load of the source data.-
Issue the command to move the source data to CockroachDB. This example command passes the source and target connection strings as environment variables, writes intermediate files to S3 storage, and uses the
truncate-if-existstable handling mode to truncate the target tables before loading data. It also limits the migration to a single schema and filters three specific tables to migrate. The defaults toIMPORT INTO. You must include--pglogical-replication-slot-nameand--pglogical-publication-and-slot-drop-and-recreateto automatically create the publication and replication slot during the data load. -
Check the output to observe
fetchprogress. If you included the--pglogical-replication-slot-nameand--pglogical-publication-and-slot-drop-and-recreateflags, a publication namedmolt_fetchis automatically created:Astarting fetchmessage indicates that the task has started:data extractionmessages are written for each table that is exported to the location in--bucket-path:data importmessages are written for each table that is loaded into CockroachDB:Afetch completemessage is written when the fetch task succeeds:
Continue MOLT Fetch after an interruption
If MOLT Fetch fails while loading data into CockroachDB from intermediate files, it exits with an error message, fetch ID, and continuation token for each table that failed to load on the target database.- All data has been exported from the source database into intermediate files on or .
- The initial load of source data into the target CockroachDB database is incomplete.
- The load uses
IMPORT INTOrather thanCOPY FROM.
Only one fetch ID and set of continuation tokens, each token corresponding to a table, are active at any time. See .
Step 4: Verify the initial data load
Use to confirm that the source and target data is consistent. This ensures that the data load was successful.Run MOLT Verify
-
Run the command, specifying the source and target connection strings and the tables to validate.
-
Check the output to observe
verifyprogress. Averification in progressindicates that the task has started:starting verifymessages are written for each specified table:Afinished row verificationmessage is written after each table is compared. Ifnum_successequalsnum_truth_rowsand the error counters (num_missing,num_mismatch,num_extraneous, andnum_column_mismatch) are all0, the table verified successfully. Any non-zero values in the error counters indicate data discrepancies that need investigation. For details on each field, refer to the page.Averification completemessage is written when the verify task succeeds:
Step 5: Finalize the target schema
Add constraints and indexes
Add any constraints or indexes that you previously removed from the CockroachDB schema to facilitate data load.If you used the
--table-handling drop-on-target-and-recreate option for data load, only and constraints are preserved. You must manually recreate all other constraints and indexes.Step 6: Begin forward replication
In this step, you will:- Configure MOLT Replicator with the flags needed for your migration.
- Start MOLT Replicator.
- Understand how to continue replication after an interruption.
Configure MOLT Replicator
When you runreplicator, you can configure the following options for replication:
- Replication connection strings: Specify URL-encoded source and target database connections.
- Replicator flags: Specify required and optional flags to configure replicator behavior.
- Tuning parameters: Optimize replication performance and resource usage.
- Replicator metrics: Monitor replication progress and performance.
Replication connection strings
MOLT Replicator uses--sourceConn and --targetConn to specify the source and target database connections.
--sourceConn specifies the connection string of the source database:
--targetConn specifies the target CockroachDB connection string:
Replicator flags
Configure the following flags for continuous replication. For details on all available flags, refer to .| Flag | Description |
|---|---|
Required. PostgreSQL replication slot name. Must match the slot name specified with --pglogical-replication-slot-name in the MOLT Fetch command. | |
Required. PostgreSQL publication name. Must match the publication name created either manually or automatically with --pglogical-publication-and-slot-drop-and-recreate in the MOLT Fetch command. | |
Required. Target schema name on CockroachDB where tables will be replicated. Schema name must be fully qualified in the format database.schema. | |
Required. Staging schema name on CockroachDB for replication metadata and checkpoints. Schema name must be fully qualified in the format database.schema. | |
| Automatically create the staging schema if it does not exist. Include this flag when starting replication for the first time. | |
Enable Prometheus metrics at a specified {host}:{port}. Metrics are served at http://{host}:{port}/_/varz. | |
| Path to a that enables data filtering, routing, or transformations. For examples, refer to . |
Tuning parameters
Configure the following to optimize replication throughput and resource usage. Test different combinations in a pre-production environment to find the optimal balance of stability and performance for your workload.The following parameters apply to PostgreSQL, Oracle, and CockroachDB (failback) sources.
| Flag | Description |
|---|---|
| Control the maximum number of concurrent target transactions. Higher values increase throughput but require more target connections. Start with a conservative value and increase based on target database capacity. | |
| Balance throughput and latency. Controls how many mutations are batched into each query to the target. Increase for higher throughput at the cost of higher latency. | |
| Control memory usage during operation. Increase to allow higher throughput at the expense of memory; decrease to apply backpressure and limit memory consumption. | |
| Set larger than by a safety factor to avoid exhausting target pool connections. Replicator enforces setting parallelism to 80% of this value. | |
| Reduce the number of queries to the target by combining multiple mutations on the same primary key within each batch. Disable only if exact mutation order matters more than end state. | |
| Improve apply throughput for independent tables and table groups that share foreign key dependencies. Increases memory and target connection usage, so ensure you increase or reduce . | |
Set to the maximum allowable time between flushes (for example, 10s if data must be applied within 10 seconds). Works with to control when buffered mutations are committed to the target. | |
| Lower this value if constraint violations resolve quickly on your workload to make retries more frequent and reduce latency. Do not lower if constraint violations take time to resolve. | |
Applies to (replicator start) scenarios only. Balance memory usage and throughput. Increase to read more rows at once from the CockroachDB staging cluster for higher throughput, at the cost of memory pressure. Decrease to reduce memory pressure and increase stability. |
Replicator metrics
MOLT Replicator metrics are not enabled by default. Enable Replicator metrics by specifying the flag with a port (orhost:port) when you start Replicator. This exposes Replicator metrics at http://{host}:{port}/_/varz. For example, the following flag exposes metrics on port 30005:
Start MOLT Replicator
With initial load complete, start replication of ongoing changes on the source to CockroachDB using .MOLT Fetch captures a consistent point-in-time checkpoint at the start of the data load (shown as
cdc_cursor in the fetch output). Starting replication from this checkpoint ensures that all changes made during and after the data load are replicated to CockroachDB, preventing data loss or duplication. The following steps use the checkpoint values from the fetch output to start replication at the correct position.-
Run the
replicatorcommand, using the same slot name that you specified with--pglogical-replication-slot-nameand the publication name created by--pglogical-publication-and-slot-drop-and-recreatein the Fetch command. Use--stagingSchemato specify a unique name for the staging database, and include--stagingCreateSchemato have MOLT Replicator automatically create the staging database:
Check that replication is working
-
Verify that Replicator is processing changes successfully. To do so, check the MOLT Replicator logs. Since you enabled debug logging with
-v, you should see connection and row processing messages: You should see periodic primary keepalive messages:When rows are successfully replicated, you should see debug output like the following:These messages confirm successful replication. You can disable verbose logging after verifying the connection.
Continue MOLT Replicator after an interruption
Run thepglogical command using the same --stagingSchema value from your initial replication command.
Be sure to specify the same --slotName value that you used during your initial replication command. The replication slot on the source PostgreSQL database automatically tracks the LSN (Log Sequence Number) checkpoint, so replication will resume from where it left off.
http://localhost:30005/_/varz to track replication progress.
Step 7: Stop application traffic
Once the inital data load has been verified and the target schema has been finalized, it’s time to begin the cutover process. First, stop application traffic to the source. Scale down the Kubernetes cluster to zero pods.Application downtime begins now.It is strongly recommended that you perform a dry run of this migration in a test environment. This will allow you to practice using the MOLT tools in real time, and it will give you an accurate sense of how long application downtime might last.
Step 8: Stop forward replication
Before you can cut over traffic to the target, the changes to the source database need to finish being written to the target. Once the source is no longer receiving write traffic, MOLT Replicator will take some seconds to finish replicating the final changes. This is known as drainage.-
Wait for replication to drain, which means that all transactions that occurred on the source database have been fully processed and replicated. There are several ways to determine that replication has fully drained:
-
When replication is caught up, you will not see new
upserted rowslogs. -
If you set up the replication metrics endpoint with in the preceding steps, metrics are available at:
Use the following Prometheus alert expression to observe when the combined rate of upserts and deletes is
0for each schema: - You can also check Prometheus metrics associated with replication lag, including , , and .
-
When replication is caught up, you will not see new
-
Cancel replication by entering
ctrl-cto issue aSIGTERMsignal. This returns an exit code0.

