This page has details about each release of the following MOLT (Migrate Off Legacy Technology) tools:
Cockroach Labs recommends using the latest available version of each tool. See Installation.
Installation
To download the latest MOLT Fetch/Verify binary:
Operating System | AMD 64-bit | ARM 64-bit |
---|---|---|
Windows | Download | Download |
Linux | Download | Download |
Mac | Download | Download |
For previous binaries, refer to the MOLT version manifest.
MOLT Fetch is supported on Red Hat Enterprise Linux (RHEL) 9 and above.
To install the LMS, refer to MOLT Live Migration Service.
To update the LMS once installed, configure the following Helm chart values:
image:
tag: 0.2.6
image.tag
specifies the LMS version. This must match the installed molt-lms-cli
version, which can be queried with molt-lms-cli version
.
For more information, refer to Configuration.
July 10, 2024
MOLT Fetch/Verify 1.1.1 is available.
- Fixed a bug that led to incorrect list continuation file behavior if a trailing slash was provided in
--bucket-path
. - Fixed a bug with extracting the filename from a failed import URL. Previously, an older filename was being used, which could result in duplicated data. Now, the filename that is used in import matches what is stored in the exceptions log table.
- Added a
--use-implicit-auth
flag that determines whether implicit authentication is used for cloud storage import URIs.
July 8, 2024
MOLT Fetch/Verify 1.1.0 is available.
- Fixed a bug where integer values for
JSONB
/JSON
types were not correctly handled when migrating from a PostgreSQL or CockroachDB source. - Optimized the logic for clearing continuation tokens. Now, the clearing option is only presented in the relevant modes and when there are tokens to clear.
- CockroachDB sources now exclusively accept connections from
admin
orroot
users. - The statement timeout can now be adjusted during the export phase. This guards against hanging queries.
May 31, 2024
MOLT Fetch/Verify 1.0.0 is available.
- Renamed the
--table-splits
flag to--concurrency-per-table
, which is more descriptive. - Increased the default value of
--import-batch-size
to1000
. This leads to better performance on the target post-migration. Each individual import job will take longer, since more data is now imported in each batch, but the sum total of all jobs should take the same (or less) time.
May 29, 2024
MOLT Fetch/Verify 0.3.0 is available.
- Added an
--import-batch-size
flag, which configures the number of files to be imported in eachIMPORT
job. - In some cases on the previous version, binaries would not work due to how
molt
was being built. Updated the build method to use static linking, which creates binaries that should be more portable. VARBIT
<>BOOL
conversion is now allowed for Fetch and Verify. The bit array is first converted toUINT64
. A resulting1
or0
is converted totrue
orfalse
accordingly. If theUINT64
is another value, an error is emitted.
May 20, 2024
MOLT Fetch/Verify 0.2.1 is available.
- MOLT tools now enforce secure connections to databases as a default. The
--allow-tls-mode-disable
flag allows users to override that behavior if secure access is not possible. - When using MySQL as a source,
--table-concurrency
and--export-concurrency
are strictly set to1
. - Fixed a bug involving history retention for
DECIMAL
values.
May 3, 2024
MOLT Fetch/Verify 0.2.0 is available.
- Fetch now supports CockroachDB multi-region tables.
- Fetch now supports continuous replication for PostgreSQL and MySQL source databases via the
--ongoing-replication
flag. When Fetch finishes the initial data load phase, it will start the replicator process as a subprocess, which runs indefinitely until the user ends the process with aSIGTERM
(ctrl-c
). - Replicator flags for (PostgreSQL and MySQL) are now supported, allowing users to further configure the
--ongoing-replication
mode for their use case. - Added the
--type-map-file
flag, which enables custom type mapping for schema creation. - Fixed a bug where primary key positions could be missed when creating a schema with multiple primary keys.
- Added a default mode for MySQL sources that ensures consistency and does not leverage parallelism. New text is displayed that alerts the user and links to documentation in cases where fetching from MySQL might not be consistent.
- Logging for continuation tokens is now omitted when data export does not successfully complete.
April 10, 2024
MOLT Fetch/Verify 0.1.4 is available.
- Fixed an inconsistency where inserting
'null'
as a string intoJSON
/JSONB
columns did not align with PostgreSQL's case-sensitive null handling.
March 22, 2024
MOLT Fetch/Verify 0.1.2 is available.
- Fixed a bug where running Verify with a PostgreSQL or CockroachDB source would cause an error when there are dropped columns on the table.
March 13, 2024
MOLT Fetch/Verify 0.1.1 is available.
- An error is now displayed if the original column is a PostgreSQL nested array.
- Added sharded data export functionality for faster performance.
- Added support for the
--table-handling 'drop-on-target-and-recreate'
flag for MySQL sources, enabling automatic schema recreation. - Added support for
ENUM
data types in schema creation. - Deprecated the
--s3-bucket
and--gcp-bucket
flags in favor of a single--bucket-path
flag.
February 22, 2024
MOLT Fetch/Verify 0.1.0 is available.
- Added implicit authentication for GCP sources.
- Resolved a CSV write deadlock issue for large file exports.
- Added new dashboard metrics:
molt_verify_num_tables
(categorizes tables asverified
,missing
, orextraneous
) andmolt_verify_duration_ms
(how long a Verify run took to complete, in milliseconds). - Previously, if a comparison between heterogeneous column types occurred, Verify would panic and stop execution. Now, Verify no longer panics, but records the error and reports it to logging, and also marks the row as having mismatching columns. The error message is logged at most once per shard to avoid log spewing. Additionally, introduced the concept of conditional success, which means that all comparable row columns were compared and succeeded with the same value. Upon conditional success, the user is instructed to check all warnings and verify the data.
- Marked the
--live
and--continuous
flags as dependencies for other sub-configuration flags. If a flag is set without its dependency, MOLT will indicate which dependency flag is not correctly set. - Added the RFC for MOLT Fetch, which explains the architectural decisions and considerations of how Fetch was built. The RFC also outlines limitiations of the current system and further improvements around usability, performance, and integration with replication tools.
- Added logging annotations for data-level errors and information that surfaced during the Verify process, as well as summary status logs about the overall functioning of the job (e.g., 10000 truth rows seen, 5000 mismatches, etc.). These can be filtered so that exact tables and rows can be triaged in one place.
- Changed the output for the running summary statistics so that they are now more easily indexable by log processing tools. Previously, it was more human-readable, but fields were not easily indexable out of the box. Now, there is a balance between readability and downstream processing.
- Added tagging for summary and data logs so that users can filter the logs in a variety of log processing tools, or
grep
in the command line. This solves the pain point of all the logs going to the same location without any differentiation. Now, it is possible to view types of logs one at a time. Summary logs report aggregated results on either a task or step in a task (e.g., task took 10 minutes; 500 tables processed overall). Data logs report on individual or row-level operations and quantitative markers (e.g., 1000 rows completed so far; failed to verify item with PK 1000) - Marked the
--s3-bucket
,--gcp-bucket
, and--local-path
flags as mutually exclusive because they all represent different intermediary stores. - Adjusted settings for the number of rows and the size of data to be flushed so that only one rule is applied at any given time.
- Fixed a bug where outputs containing special characters were not formatted correctly.
- When running the import query for Fetch, the import statement is now printed if the logger is in debug mode.
- Updated the MOLT README with an OS + Docker-based edge case with Fetch.
- Logging for Fetch and Verify now defaults to structured JSON logging, which contains at least the message, level, and formatted time string. The
--use-console-writer
flag enables prettier console writing that has colored and human-readable log output. Structured logging is the default because it is easier to parse with log ingestion tools and is more efficient for the application to output. - Task/data logs can now be written to specified log files using
--log-file
. If the flag is left empty, the MOLT tool will only write tostdout
, which is more efficient.
May 31, 2024
LMS 0.2.6 is available.
May 30, 2024
LMS 0.2.5 is available.
January 26, 2024
LMS 0.2.4 is available.
- Added a restriction for shadowing mode according to the cutover type. Consistent cutover must be executed without any shadowing (i.e., must be performed with the non-sync mode).