Encryption

On this page

Data encryption and decryption is the process of transforming plaintext data to cipher-text and vice versa using a key or password.

Encryption in flight

CockroachDB uses TLS 1.2 for inter-node and client-node authentication as well as setting up a secure communication channel. Once the secure channel is set up, all inter-node and client-node network communication is encrypted using a shared encryption key as per the TLS 1.2 protocol. This feature is enabled by default for all secure clusters and needs no additional configuration.

Encryption at Rest (Experimental)(Enterprise)

Encryption at Rest provides transparent encryption of a node's data on the local disk.

Encryption at Rest allows encryption of all files on disk using AES in counter mode, with all key sizes allowed.

Encryption is performed in the storage layer and configured per store. All files used by the store, regardless of contents, are encrypted with the desired algorithm.

To allow arbitrary rotation schedules and ensure security of the keys, we use two layers of keys:

• Store keys are provided by the user in a file. They are used to encrypt the list of data keys (see below). This is known as a key encryption key: its only purpose is to encrypt other keys. Store keys are never persisted by CockroachDB. Since very little data is encrypted using this key, it can have a very long lifetime without risk of reuse.

• Data keys are automatically generated by CockroachDB. They are used to encrypt all files on disk. This is known as a data encryption key. Data keys are persisted in a key registry file, encrypted using the store key. The key has a short lifetime to avoid reuse.

Store keys are specified at node startup by passing a path to a locally readable file. The file must contain 32 bytes (the key ID) followed by the key (16, 24, or 32 bytes). The size of the key dictates the version of AES to use (AES-128, AES-192, or AES-256). For an example showing how to create a store key, see Generating key files below.

Also during node startup, CockroachDB uses a data key with the same length as the store key. If encryption has just been enabled, the key size has changed, or the data key is too old (default lifetime is one week), CockroachDB generates a new data key.

Any new file created by the store uses the currently-active data key. All data keys (both active and previous) are stored in a key registry file and encrypted with the active store key.

After startup, if the active data key is too old, CockroachDB generates a new data key and marks it as active, using it for all further encryption.

CockroachDB does not currently force re-encryption of older files but instead relies on normal RocksDB churn to slowly rewrite all data with the desired encryption.

Rotating keys

Key rotation is necessary for Encryption at Rest for multiple reasons:

• To prevent key reuse with the same encryption parameters (after encrypting many files).
• To reduce the risk of key exposure.

Store keys are specified by the user and must be rotated by specifying different keys. This is done by setting the key parameter of the --enterprise-encryption flag to the path to the new key, and old-key to the previously-used key.

Data keys will automatically be rotated at startup if any of the following conditions are met:

• The active store key has changed.
• The encryption type has changed (different key size, or plaintext to/from encryption).
• The current data key is rotation-period old or more.

Data keys will automatically be rotated at runtime if the current data key is rotation-period old or more.

Once rotated, an old store key cannot be made the active key again.

Upon store key rotation the data keys registry is decrypted using the old key and encrypted with the new key. The newly-generated data key is used to encrypt all new data from this point on.

Changing encryption type

The user can change the encryption type from plaintext to encryption, between different encryption algorithms (using various key sizes), or from encryption to plaintext.

When changing the encryption type to plaintext, the data key registry is no longer encrypted and all previous data keys are readable by anyone. All data on the store is effectively readable.

When changing from plaintext to encryption, it will take some time for all data to eventually be re-written and encrypted.

Recommendations

There are a number of considerations to keep in mind when running with encryption.

Deployment configuration

To prevent key leakage, production deployments should:

• Use encrypted swap, or disable swap entirely.
• Disable core files.

CockroachDB attempts to disable core files at startup when encryption is requested, but it may fail.

Key handling

Key management is the most dangerous aspect of encryption. The following rules should be kept in mind:

• Make sure that only the UNIX user running the cockroach process has access to the keys.
• Do not store the keys on the same partition/drive as the CockroachDB data. It is best to load keys at run time from a separate system (e.g., Keywhiz, Vault).
• Rotate store keys frequently (every few weeks to months).
• Keep the data key rotation period low (default is one week).

Other recommendations

A few other recommendations apply for best security practices:

• Do not switch from encrypted to plaintext, this leaks data keys. When plaintext is selected, all previously encrypted data must be considered reachable.
• Do not copy the encrypted files, as the data keys are not easily available.
• If encryption is desired, start a node with it enabled from the first run, without ever running in plaintext.

Examples

Generating key files

Cockroach determines which encryption algorithm to use based on the size of the key file. The key file must contain random data making up the key ID (32 bytes) and the actual key (16, 24, or 32 bytes depending on the encryption algorithm).

Generating a key file can be done using the cockroach CLI:

$ cockroach gen encryption-key -s 128 /path/to/my/aes-128.key

Or the equivalent openssl CLI command:

$ openssl rand -out /path/to/my/aes-128.key 48

Starting a node with encryption

Encryption is configured at node start time using the --enterprise-encryption command line flag. The flag specifies the encryption options for one of the stores on the node. If multiple stores exist, the flag must be specified for each store.

The flag takes the form: --enterprise-encryption=path=<store path>,key=<key file>,old-key=<old key file>,rotation-period=<period>.

The allowed components in the flag are:

The key and old-key components must always be specified. They allow for transitions between encryption algorithms, and between plaintext and encrypted.

Starting a node for the first time using AES-128 encryption can be done using:

$ cockroach start --store=cockroach-data --enterprise-encryption=path=cockroach-data,key=/path/to/my/aes-128.key,old-key=plain

Checking encryption status

Encryption status can be seen on the node's stores report, reachable through: http(s)://nodeaddress:8080/#/reports/stores/local (or replace local with the node ID). For example, if you are running a local cluster, you can see the node's stores report at https://localhost:8888/#/reports/stores/local.

The report shows encryption status for all stores on the selected node, including:

• Active store key information.
• Active data key information.
• The fraction of files/bytes encrypted using the active data key.

CockroachDB relies on RocksDB compactions to write new files using the latest encryption key. It may take several days for all files to be replaced. Some files are only rewritten at startup, and some keep older copies around, requiring multiple restarts. You can force RocksDB compaction with the cockroach debug compact command (the node must first be stopped).

A more detailed list of encryption keys in use for each file is available using the cockroach debug encryption-status command.

Information about keys is written to the logs, including:

• Active/old key information at startup.
• New key information after data key rotation.

Changing encryption algorithm or keys

Encryption type and keys can be changed at any time by restarting the node. To change keys or encryption type, the key component of the --enterprise-encryption flag is set to the new key, while the key previously used must be specified in the old-key component.

For example, we can switch from AES-128 to AES-256 using:

$ cockroach start --store=cockroach-data --enterprise-encryption=path=cockroach-data,key=/path/to/my/aes-256.key,old-key=/path/to/my/aes-128.key

Upon starting, the node will read the existing data keys using the old encryption key (aes-128.key), then rewrite the data keys using the new key (aes-256.key). A new data key will be generated to match the desired AES-256 algorithm.

To check that the new key is active, use the stores report page in the Admin UI to check the encryption status.

To disable encryption, specify key=plain. The data keys will be stored in plaintext and new data will not be encrypted.

To rotate keys, specify key=/path/to/my/new-aes-128.key and old-key=/path/to/my/old-aes-128.key. The data keys will be decrypted using the old key and then encrypted using the new key. A new data key will also be generated.

Encryption known issues

Unencrypted backups

Backups taken with the BACKUP statement are not encrypted even if Encryption at Rest is enabled. Encryption at Rest only applies to the CockroachDB node's data on the local disk. If you want encrypted backups, you will need to encrypt your backup files using your preferred encryption method.

A workaround for the issue is to use a cloud storage provider that is configured to transparently encrypt your data (e.g., AWS S3 default encryption).

Encryption for touchpoints with other services

• S3 backup encryption
• Encrypted comms with Kafka

See also

• Client Connection Parameters
• Manual Deployment
• Orchestrated Deployment
• Local Deployment
• Other Cockroach Commands

Was this helpful?