Customer-Managed Encryption Keys (CMEK) Overview

Customer-Managed Encryption Keys (CMEK) allow you to protect data at rest in a CockroachDB Dedicated advanced private cluster using a cryptographic key that is entirely within your control, hosted in a supported cloud provider key-management system (KMS). This key is called the CMEK key.

You can manage your CMEK keys using one or more of the following services:

• Amazon Web Services (AWS) KMS
• Google Cloud Platform (GCP) KMS

To learn more, visit Managing Customer-Managed Encryption Keys (CMEK) for CockroachDB Dedicated.

CockroachDB Dedicated includes support for referring to CMEK keys in HashiCorp Vault Secrets Manager, which can distribute keys stored in multiple KMS systems, as long as the actual keys are stored in AWS KMS or GCP KMS.

CockroachDB Cloud communicates with the KMS platform using the KMS platform's API, and you manage CockroachDB Cloud's access to the CMEK key using the KMS platform's identity and access management (IAM) system. The CMEK key is never present in a cluster and CockroachDB Cloud never has direct access to the CMEK key material. When CMEK is enabled, the CMEK key must be available before the cluster can start and the cluster's newly-written data at rest can be accessed.

This article describes how CMEK works in CockroachDB Dedicated clusters. To configure CMEK, see Managing Customer-Managed Encryption Keys (CMEK) for CockroachDB Dedicated.

Benefits of CMEK

This section describes some of the ways that CMEK can help you protect your data and meet business and regulatory requirements, even when you don't own the underlying infrastructure that hosts your data.

• Enforcement of data retention policies: To ensure that a cluster's data is permanently inaccessible, you can destroy the CMEK key or keys that protect its data.

  Warning:

  Keep these points in mind before destroying a CMEK key:

  • If a CMEK key is destroyed, the cluster's data can't be recovered by you or by CockroachDB Cloud, even by restoring from a CockroachDB Cloud-managed backup. After enabling CMEK, do not disable, schedule for destruction, or destroy a CMEK that is in use by clusters. Instead, first rotate the cluster to use a new CMEK or decommission the cluster, and then use your KMS platform's audit logs to verify that the CMEK is no longer being used.

  • To protect against inadvertent data loss, your KMS platform may impose a waiting period before a key is permanently deleted. This waiting period may be configurable when you create the key. Check the documentation for your KMS platform for details about how long before a key deletion is permanent and irreversible.

• Enforcement of data domiciling and locality requirements: In a multi-region cluster, you can confine an individual database to a single region or multiple regions. For more information and limitations, see Data Domiciling with CockroachDB. When you enable CMEK on a multi-region cluster, you can optionally assign a separate CMEK key to each region, or use the same CMEK key for multiple related regions.

• Enforcement of encryption requirements: With CMEK, you have control the CMEK key's encryption strength. The CMEK key's size is determined by what your KMS provider supports.

  You can use your KMS platform's controls to configure the regions where the CMEK key is available, enable automatic rotation schedules for CMEK keys, and view audit logs that show each time the CMEK key is used by CockroachDB Cloud. CockroachDB Cloud does not need any visibility into these details.

• Separation of concerns: With CMEK, you give CockroachDB Cloud permission to encrypt and decrypt using the CMEK, but Cockroach Labs has no access to the CMEK's key material. The ability to create keys and manage IAM access to them can be delegated to a limited group of trusted individuals, who may be distinct from the organization's cluster admins.

• Infrastructure flexibility: If your CMEK keys are stored in multiple KMS systems or tenants, you can use HashiCorp Vault Key Management Secrets Engine to give your cluster access to your CMEK keys, as long as the cluster and keys are stored in the same deployment environment (GCP or AWS).

The following example shows some of the ways that CMEK can help you meet business and regulatory requirements.

Imagine that you have a business requirement to verify that a cluster's data is inaccessible when you delete the cluster. Software as a Service (SaaS) makes such requirements more difficult to verifiably meet, because you don't have visibility into what happens to the cluster's resources after the cluster disappears from your view. This sort of requirement might make it more challenging for your organization to move some workloads to SaaS.

CMEK helps you to enforce such business rules on CockroachDB Cloud clusters. With CMEK, you can actively and verifiably enforce this requirement without waiting for CockroachDB Cloud to destroy the cluster's resources. Instead, with a single operation you can revoke the cluster's ability to use the CMEK key. This will trigger a cluster restart, and the restart will fail because the CMEK key is unavailable. After verifying that the restart has failed, you can delete the cluster in CockroachDB Cloud.

How CMEK works

When you create a CockroachDB Dedicated cluster, its data at rest on cluster disks is not encrypted by default. However, the disks themselves are automatically encrypted by cryptographic keys owned and managed by the cloud providers themselves.

When you enable CMEK on a CockroachDB Dedicated advanced private cluster, CockroachDB Cloud creates two kinds of encryption keys and begins to use them to protect newly-written data at rest. CockroachDB Cloud manages these encryption keys and propagates them to cluster nodes.

1. The data key is a Data Encryption Key (DEK), and is used to encrypt and decrypt cluster data before it is read from or written to disks attached to a cluster's nodes. Each time the cluster is started or restarted, and each time a node and related disks are added to a cluster, CockroachDB Dedicated uses the store key to encrypt and decrypt data keys. Each cluster node maintains its own list of data keys. The data key is automatically rotated monthly and is always encrypted at rest by the store key.

2. The store key is a Key Encryption Key (KEK) that encrypts the data keys at rest. It is encrypted at rest by the CMEK key before being propagated to cluster nodes, and is decrypted in memory on each node when the cluster starts. The store key is not automatically rotated.

For more details about encryption in CockroachDB Cloud, see Encryption At Rest.

Enabling CMEK has no impact on a cluster's existing data.

At the time that you enable CMEK for a cluster, CockroachDB Cloud:

1. Creates the store key and encrypts it using the CMEK key.
2. Creates the data key and encrypts it using the encrypted store key.
3. Propagates the store key and data key to cluster nodes.
4. Starts the cluster.

Going forward:

1. The cluster can start only when the CMEK key is available to decrypt the store key. If the CMEK key becomes unavailable or you revoke the cluster's access to the key, the cluster can't start until access is restored.
2. Each time the store key is rotated, the new store key is also encrypted using the CMEK key.
3. Each time the data key is rotated, the new data key is encrypted using the encrypted store key.
4. Each time a node writes new data to disk, it is encrypted using the current data key. Data is read using the data key that was used to encrypt it.

Rotation of a CMEK key

You can rotate a CMEK key for a CockroachDB Dedicated cluster either by creating a new version of the existing CMEK key or by creating a new CMEK key. At a high level:

To begin using a new version of an existing CMEK key:

1. In your KMS platform, you can either configure automatic rotation for the CMEK key, or you can perform a manual rotation.
2. CockroachDB Cloud does not automatically re-encrypt the store key using the new CMEK key version. For each region you want to update, you must also perform a rotation using the CockroachDB Cloud API without modifying the CMEK key URI. CockroachDB Cloud re-encrypts the store key using the new CMEK key version.

To begin using an entirely new CMEK key:

1. Within your KMS platform, you create a new CMEK key.
2. Next, you perform a rotation using the CockroachDB Cloud API and provide the new CMEK key URI.

To learn more about rotating a CMEK key using the CockroachDB Cloud API, visit Rotate a CMEK key.

Backup and restore operations on a cluster with CMEK

This section describes how enabling CMEK changes backup and restore operations on a cluster.

Backups in CockroachDB Dedicated are triggered in two ways, only one of which is affected by CMEK.

• You can perform a manual backup by using the BACKUP SQL command to back up database objects in a cluster. To restore from a manual backup, you use the RESTORE SQL command. Manual backups are not automatically encrypted, but you can optionally encrypt a manual backup by specifying an encryption key when you run the BACKUP command. Enabling CMEK has no impact on manual backups. To learn about encrypting manual backups, see Take and Restore Encrypted Backups.

• CockroachDB Cloud automatically backs up clusters on a set schedule that is not configurable. You can view, manage, or restore from these backups using the CockroachDB Cloud Console. Managed backups operate on all databases, tables, views, and scheduled jobs in the cluster. Managed backups can be restored only to the cluster where they were taken. Managed backups are automatically encrypted using data keys that are distinct from those used to encrypt the cluster's data.

When CMEK is enabled for a cluster, managed backups change in the following ways:

• You can no longer restore from a managed backup that was taken before CMEK was enabled.
• The data keys used to encrypt managed backups in CockroachDB Cloud are encrypted using the CMEK key before being written to persistent storage. If the CMEK is not available, managed backups will fail to run.
• The CMEK that was used to encrypt a managed backup must be available to restore that backup.

FAQs

This section provides answers to frequently-asked questions (FAQs) about CMEK.

If we don’t enable CMEK for our CockroachDB Dedicated clusters, are those encrypted in some manner by default?

Yes, CockroachDB Dedicated cluster disks are encrypted by default using keys managed by each cloud provider.

What steps should I take before enabling CMEK for a cluster?

CMEK can be enabled only on a private cluster on CockroachDB Dedicated advanced. Refer to Create Private Clusters.

Can we enable CMEK for an existing cluster that wasn't created as a private cluster?

An existing cluster cannot be migrated to a private cluster. Contact your account team for advice about how to migrate or restore your existing cluster's data to a new private cluster on CockroachDB Dedicated advanced.

If we enable CMEK for a cluster that has been in use for some time, is the existing data encrypted at that time?

CockroachDB Dedicated does not force encryption of previously-written data but instead relies on normal storage engine churn for desired encryption. That means the new key is used to encrypt newly-written data, while previously-written data remains unencrypted unless it's rewritten.

Can we enable CMEK for a new region when it's added to a CMEK-enabled cluster?

Yes, when you add a new region to a CMEK-enabled cluster, you must enable CMEK for that region. Refer to Add a Region to a CMEK-enabled Cluster.

Is the data encryption key rotated at some set duration or periodically? If yes, is there a way to customize the duration?

Yes, the data encryption key is rotated automatically once every month. It’s not possible to customize that duration. The new key is used to encrypt new writes, while the old data is still encrypted with the old data keys unless it’s rewritten.

Can we rotate the CMEK for a cluster after a certain time or at some periodic interval?

You can rotate a CMEK key for a CockroachDB Dedicated cluster either by creating a new version of the existing CMEK key or by creating a new CMEK key. At a high level:

To begin using a new version of an existing CMEK key:

1. In your KMS platform, you can either configure automatic rotation for the CMEK key, or you can perform a manual rotation.
2. CockroachDB Cloud does not automatically re-encrypt the store key using the new CMEK key version. For each region you want to update, you must also perform a rotation using the CockroachDB Cloud API without modifying the CMEK key URI. CockroachDB Cloud re-encrypts the store key using the new CMEK key version.

To begin using an entirely new CMEK key:

1. Within your KMS platform, you create a new CMEK key.
2. Next, you perform a rotation using the CockroachDB Cloud API and provide the new CMEK key URI.

To learn more about rotating a CMEK key using the CockroachDB Cloud API, visit Rotate a CMEK key.

Are CockroachDB Dedicated managed backups also encrypted using the CMEK?

Yes, the managed backups stored in CockroachDB Cloud infrastructure are also encrypted using the CMEK, using CoackroachDB’s encrypted backup capability. Internally, a backup data key is wrapped by the CMEK, and then the backup data key is used for encrypting the backup.

As part of managed backup encryption, is the same backup data key used to encrypt all backups for a cluster?

A different backup data key is used for each full cluster backup, while the same backup data key is used for incremental backups on top of a full cluster backup. In all cases, the backup data key is encrypted with CMEK for a CMEK-enabled cluster.

How are the store key (Key Encryption Key) and the data key (Data Encryption Key) stored on the cluster?

The store key is only stored as encrypted by the CMEK, while it’s available as decrypted only in memory for the CockroachDB process to use. The data key is stored as encrypted by the store key, along with the data files on cluster disks.

Can we use the CockroachDB Cloud Console to enable or revoke a CMEK for a cluster?

Not yet. Currently, you must use the CockroachDB Cloud API or the CockroachDB Terraform provider.

Is it possible to self-serve restore a CMEK-enabled cluster in case of a cluster failure or disaster scenario?

Not yet. To restore a failed CMEK-enabled cluster, please create a support ticket for Cockroach Labs providing your cluster ID and organization ID.

Limitations

The CMEK feature has the following limitations:

• CMEK is not yet available for CockroachDB Dedicated on Azure. To express interest, contact your Cockroach Labs account team.
• To enable or revoke a CMEK on a cluster, you must use the Cloud API or the CockroachDB Terraform provider. It's not possible to enable a CMEK using the CockroachDB Cloud Console.
• If you add a new region to a cluster with CMEK enabled, you must configure a CMEK for the new region to protect its data.
• If the CMEK is not available due to a misconfiguration or a KMS outage, a cluster's managed backups will begin to fail, but no customer notification is sent from CockroachDB Cloud via email. However, Cockroach Labs support is notified if such a failure occurs.

See also

• Managing Customer-Managed Encryption Keys (CMEK) for CockroachDB Dedicated
• Create Private Clusters
• Encryption At Rest