MOLT Releases

To download the latest MOLT Fetch/Verify binary:

For previous binaries, refer to the MOLT version manifest.

To install the LMS, refer to MOLT Live Migration Service.

To update the LMS once installed, configure the following Helm chart values:

image:
  tag: 0.2.6  

image.tag specifies the LMS version. This must match the installed molt-lms-cli version, which can be queried with molt-lms-cli version.

For more information, refer to Configuration.

July 10, 2024

MOLT Fetch/Verify 1.1.1 is available.

• Fixed a bug that led to incorrect list continuation file behavior if a trailing slash was provided in --bucket-path.
• Fixed a bug with extracting the filename from a failed import URL. Previously, an older filename was being used, which could result in duplicated data. Now, the filename that is used in import matches what is stored in the exceptions log table.
• Added a --use-implicit-auth flag that determines whether implicit authentication is used for cloud storage import URIs.

July 8, 2024

MOLT Fetch/Verify 1.1.0 is available.

• Fixed a bug where integer values for JSONB/JSON types were not correctly handled when migrating from a PostgreSQL or CockroachDB source.
• Optimized the logic for clearing continuation tokens. Now, the clearing option is only presented in the relevant modes and when there are tokens to clear.
• CockroachDB sources now exclusively accept connections from admin or root users.
• The statement timeout can now be adjusted during the export phase. This guards against hanging queries.

May 31, 2024

MOLT Fetch/Verify 1.0.0 is available.

• Renamed the --table-splits flag to --concurrency-per-table, which is more descriptive.
• Increased the default value of --import-batch-size to 1000. This leads to better performance on the target post-migration. Each individual import job will take longer, since more data is now imported in each batch, but the sum total of all jobs should take the same (or less) time.

May 29, 2024

MOLT Fetch/Verify 0.3.0 is available.

• Added an --import-batch-size flag, which configures the number of files to be imported in each IMPORT job.
• In some cases on the previous version, binaries would not work due to how molt was being built. Updated the build method to use static linking, which creates binaries that should be more portable.
• VARBIT <> BOOL conversion is now allowed for Fetch and Verify. The bit array is first converted to UINT64. A resulting 1 or 0 is converted to true or false accordingly. If the UINT64 is another value, an error is emitted.

May 20, 2024

MOLT Fetch/Verify 0.2.1 is available.

• MOLT tools now enforce secure connections to databases as a default. The --allow-tls-mode-disable flag allows users to override that behavior if secure access is not possible.
• When using MySQL as a source, --table-concurrency and --export-concurrency are strictly set to 1.
• Fixed a bug involving history retention for DECIMAL values.

May 3, 2024

MOLT Fetch/Verify 0.2.0 is available.

• Fetch now supports CockroachDB multi-region tables.
• Fetch now supports continuous replication for PostgreSQL and MySQL source databases via the --ongoing-replication flag. When Fetch finishes the initial data load phase, it will start the replicator process as a subprocess, which runs indefinitely until the user ends the process with a SIGTERM (ctrl-c).
• Replicator flags for (PostgreSQL and MySQL) are now supported, allowing users to further configure the --ongoing-replication mode for their use case.
• Added the --type-map-file flag, which enables custom type mapping for schema creation.
• Fixed a bug where primary key positions could be missed when creating a schema with multiple primary keys.
• Added a default mode for MySQL sources that ensures consistency and does not leverage parallelism. New text is displayed that alerts the user and links to documentation in cases where fetching from MySQL might not be consistent.
• Logging for continuation tokens is now omitted when data export does not successfully complete.

April 10, 2024

MOLT Fetch/Verify 0.1.4 is available.

• Fixed an inconsistency where inserting 'null' as a string into JSON/JSONB columns did not align with PostgreSQL's case-sensitive null handling.

March 22, 2024

MOLT Fetch/Verify 0.1.2 is available.

• Fixed a bug where running Verify with a PostgreSQL or CockroachDB source would cause an error when there are dropped columns on the table.

March 13, 2024

MOLT Fetch/Verify 0.1.1 is available.

• An error is now displayed if the original column is a PostgreSQL nested array.
• Added sharded data export functionality for faster performance.
• Added support for the --table-handling 'drop-on-target-and-recreate' flag for MySQL sources, enabling automatic schema recreation.
• Added support for ENUM data types in schema creation.
• Deprecated the --s3-bucket and --gcp-bucket flags in favor of a single --bucket-path flag.

February 22, 2024

MOLT Fetch/Verify 0.1.0 is available.

• Added implicit authentication for GCP sources.
• Resolved a CSV write deadlock issue for large file exports.
• Added new dashboard metrics: molt_verify_num_tables (categorizes tables as verified, missing, or extraneous) and molt_verify_duration_ms (how long a Verify run took to complete, in milliseconds).
• Previously, if a comparison between heterogeneous column types occurred, Verify would panic and stop execution. Now, Verify no longer panics, but records the error and reports it to logging, and also marks the row as having mismatching columns. The error message is logged at most once per shard to avoid log spewing. Additionally, introduced the concept of conditional success, which means that all comparable row columns were compared and succeeded with the same value. Upon conditional success, the user is instructed to check all warnings and verify the data.
• Marked the --live and --continuous flags as dependencies for other sub-configuration flags. If a flag is set without its dependency, MOLT will indicate which dependency flag is not correctly set.
• Added the RFC for MOLT Fetch, which explains the architectural decisions and considerations of how Fetch was built. The RFC also outlines limitiations of the current system and further improvements around usability, performance, and integration with replication tools.
• Added logging annotations for data-level errors and information that surfaced during the Verify process, as well as summary status logs about the overall functioning of the job (e.g., 10000 truth rows seen, 5000 mismatches, etc.). These can be filtered so that exact tables and rows can be triaged in one place.
• Changed the output for the running summary statistics so that they are now more easily indexable by log processing tools. Previously, it was more human-readable, but fields were not easily indexable out of the box. Now, there is a balance between readability and downstream processing.
• Added tagging for summary and data logs so that users can filter the logs in a variety of log processing tools, or grep in the command line. This solves the pain point of all the logs going to the same location without any differentiation. Now, it is possible to view types of logs one at a time. Summary logs report aggregated results on either a task or step in a task (e.g., task took 10 minutes; 500 tables processed overall). Data logs report on individual or row-level operations and quantitative markers (e.g., 1000 rows completed so far; failed to verify item with PK 1000)
• Marked the --s3-bucket, --gcp-bucket, and --local-path flags as mutually exclusive because they all represent different intermediary stores.
• Adjusted settings for the number of rows and the size of data to be flushed so that only one rule is applied at any given time.
• Fixed a bug where outputs containing special characters were not formatted correctly.
• When running the import query for Fetch, the import statement is now printed if the logger is in debug mode.
• Updated the MOLT README with an OS + Docker-based edge case with Fetch.
• Logging for Fetch and Verify now defaults to structured JSON logging, which contains at least the message, level, and formatted time string. The --use-console-writer flag enables prettier console writing that has colored and human-readable log output. Structured logging is the default because it is easier to parse with log ingestion tools and is more efficient for the application to output.
• Task/data logs can now be written to specified log files using --log-file. If the flag is left empty, the MOLT tool will only write to stdout, which is more efficient.

May 31, 2024

LMS 0.2.6 is available.

May 30, 2024

LMS 0.2.5 is available.

January 26, 2024

LMS 0.2.4 is available.

• Added a restriction for shadowing mode according to the cutover type. Consistent cutover must be executed without any shadowing (i.e., must be performed with the non-sync mode).