Publication date: April 30, 2021 (Preliminary advisory issued on April 6, 2021)
Description
Cockroach Labs has discovered a bug, which under certain circumstances may result in data loss when restoring from an incremental backup.
If a backup coincided with an in-progress index creation (backfill), RESTORE, or IMPORT, it is possible that a subsequent incremental backup will not include all of the indexed, restored or imported data.
Full backups are unaffected by this issue.
Statement
This issue affects CockroachDB versions 19.1 and later. It has been resolved in CockroachDB v20.1.15, v20.2.8 and the forthcoming v21.0.0. Users are advised to upgrade.
These versions are available on our releases page.
To our knowledge, this issue has not been observed to affect any user. It was discovered via an internal investigation by Cockroach Labs engineers.
In addition to resolving the issue when creating incremental backups, these fixes can, in some cases, detect invalid indexes at restore time and correct them automatically. Users may see a performance impact in this case.
The following GitHub issues and pull requests describe the issues and fixes:
Questions about this technical alert can be directed to our support team, or to your customer success contact at Cockroach Labs.
Mitigation
The issue is resolved in CockroachDB v20.1.15, v20.2.8 and the forthcoming v21.0.0. They are available on our releases page. Users are advised to upgrade.
If you are unable to upgrade, and wish to operate with an abundance of caution in the meantime:
- If you create or change indexes, or do an
IMPORTorRESTORE, take a full backup after the operation completes; do not rely on incremental backups taken while doing this operation. - If you're unsure about the above, then only consider full backups to be reliable. Consider incremental backups to be unreliable, if those incremental backups were taken during in-progress index backfills or imports, or if you are unsure.
- If you suspect that you have restored an incremental backup into production, where secondary indexes were being backfilled while that incremental backup was running, you may wish to recreate those indexes (and drop the old ones).
Impact
Incremental backups are only available in CockroachDB's enterprise version. Full backups are unaffected.
- This issue may result in undetected data loss, if invalid incremental backups have been restored into production.
- This issue may result in incorrect query results, if invalid incremental backups have been restored into production.
- Incremental backups at rest may be invalid, if the backup occurred during certain incomplete bulk operations, as described above.
Was this helpful?
