This article assumes you have already deployed CockroachDB securely on a single Kubernetes cluster using the Operator or Helm. However, it's possible to configure these settings before starting CockroachDB on Kubernetes.
By default, self-signed certificates are used when using the Operator or Helm to securely deploy CockroachDB on Kubernetes. However, the recommended approach is to use cert-manager
for certificate management. For details, refer to Deploy cert-manager for mTLS.
This page explains how to:
- Authenticate an Operator or Helm deployment using a custom CA
- Rotate security certificates
- Secure the webhooks (Operator)
If you are running a secure Helm deployment on Kubernetes 1.22 and later, you must migrate away from using the Kubernetes CA for cluster authentication. The recommended approach is to use cert-manager
for certificate management. For details, refer to Deploy cert-manager for mTLS.
All kubectl
steps should be performed in the namespace where you installed the Operator. By default, this is cockroach-operator-system
.
Use a custom CA
By default, the Operator will generate and sign 1 client and 1 node certificate to secure the cluster.
To use your own certificate authority instead, add the following to the Operator's custom resource before initializing the cluster:
spec:
nodeTLSSecret: {node_secret_name}
clientTLSSecret: {client_secret_name}
Replace:
{node_secret_name}
: the name of the Kubernetes secret that contains the generated client certificate and key.{client_secret_name}
: the name of the Kubernetes secret that contains the generated node certificate and key.
Currently, the Operator requires that the client and node secrets each contain the filenames tls.crt
and tls.key
.
Apply the new settings to the cluster:
$ kubectl apply -f example.yaml
By default on secure deployments, the Helm chart will generate and sign one client certificate and one node certificate to secure the cluster.
To use your own certificate authority instead, specify the following in the custom values file you created when deploying the cluster:
tls:
enabled: true
certs:
provided: true
nodeSecret: {node_secret_name}
clientRootSecret: {client_secret_name}
Replace:
{node_secret_name}
: the name of the Kubernetes secret that contains the generated client certificate and key.{client_secret_name}
: the name of the Kubernetes secret that contains the generated node certificate and key.
Apply the custom values to override the default Helm chart values:
$ helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb
Example: Authenticate with cockroach cert
This example uses cockroach cert
commands to generate and sign the CockroachDB node and client certificates. To learn more about the supported methods of signing certificates, refer to Authentication.
Complete the following steps before initializing the cluster.
Create two directories:
mkdir certs my-safe-directory
Directory Description certs
You'll generate your CA certificate and all node and client certificates and keys in this directory. my-safe-directory
You'll generate your CA key in this directory and then reference the key when generating node and client certificates. Create the CA certificate and key pair:
cockroach cert create-ca \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.key
Create a client certificate and key pair for the root user:
cockroach cert create-client root \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.key
Upload the client certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the Operator:
kubectl create secret generic cockroachdb.client.root \ --from-file=tls.key=certs/client.root.key \ --from-file=tls.crt=certs/client.root.crt \ --from-file=ca.crt=certs/ca.crt
secret/cockroachdb.client.root created
Create the certificate and key pair for your CockroachDB nodes, specifying the namespace you used when deploying the cluster. This example uses the Operator's default namespace (
cockroach-operator-system
):cockroach cert create-node localhost \ 127.0.0.1 \ cockroachdb-public \ cockroachdb-public.cockroach-operator-system \ cockroachdb-public.cockroach-operator-system.svc.cluster.local \ *.cockroachdb \ *.cockroachdb.cockroach-operator-system \ *.cockroachdb.cockroach-operator-system.svc.cluster.local \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.key
Upload the node certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the Operator:
kubectl create secret generic cockroachdb.node \ --from-file=tls.key=certs/node.key \ --from-file=tls.crt=certs/node.crt \ --from-file=ca.crt=certs/ca.crt
secret/cockroachdb.node created
Check that the secrets were created on the cluster:
kubectl get secrets
NAME TYPE DATA AGE cockroachdb.client.root Opaque 3 13s cockroachdb.node Opaque 3 3s default-token-6js7b kubernetes.io/service-account-token 3 9h
Add
nodeTLSSecret
andclientTLSSecret
to the Operator's custom resource, specifying the generated secret names:spec: clientTLSSecret: cockroachdb.client.root nodeTLSSecret: cockroachdb.node
Create two directories:
mkdir certs my-safe-directory
Directory Description certs
You'll generate your CA certificate and all node and client certificates and keys in this directory. my-safe-directory
You'll generate your CA key in this directory and then reference the key when generating node and client certificates. Create the CA certificate and key pair:
cockroach cert create-ca \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.key
Create a client certificate and key pair for the root user:
cockroach cert create-client root \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.key
Upload the client certificate and key to the Kubernetes cluster as a secret:
kubectl create secret generic cockroachdb.client.root --from-file=certs
secret/cockroachdb.client.root created
Create the certificate and key pair for your CockroachDB nodes:
cockroach cert create-node localhost \ 127.0.0.1 \ my-release-cockroachdb-public \ my-release-cockroachdb-public.default \ my-release-cockroachdb-public.default.svc.cluster.local \ *.my-release-cockroachdb \ *.my-release-cockroachdb.default \ *.my-release-cockroachdb.default.svc.cluster.local \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.key
Note:This example assumes that you followed our deployment example, which uses
my-release
as the release name. If you used a different value, be sure to adjust the release name in this command.Upload the node certificate and key to the Kubernetes cluster as a secret:
kubectl create secret generic cockroachdb.node --from-file=certs
secret/cockroachdb.node created
Check that the secrets were created on the cluster:
kubectl get secrets
NAME TYPE DATA AGE cockroachdb.client.root Opaque 3 41m cockroachdb.node Opaque 5 14s default-token-6qjdb kubernetes.io/service-account-token 3 4m
Specify the following in the custom values file you created when deploying the cluster, using the generated secret names:
tls: enabled: true certs: provided: true clientRootSecret: cockroachdb.client.root nodeSecret: cockroachdb.node
Apply the custom values to override the default Helm chart values:
helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb
Rotate security certificates
You may need to rotate the node, client, or CA certificates in the following scenarios:
- The node, client, or CA certificates are expiring soon.
- Your organization's compliance policy requires periodic certificate rotation.
- The key (for a node, client, or CA) is compromised.
- You need to modify the contents of a certificate, for example, to add another DNS name or the IP address of a load balancer through which a node can be reached. In this case, you would need to rotate only the node certificates.
Example: Rotate certificates signed with cockroach cert
If you previously authenticated with cockroach cert
, follow these steps to rotate the certificates using the same CA:
Create a new client certificate and key pair for the root user, overwriting the previous certificate and key:
cockroach cert create-client root \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.key \ --overwrite
Upload the new client certificate and key to the Kubernetes cluster as a new secret, renaming them to the filenames required by the Operator:
kubectl create secret generic cockroachdb.client.root.2 \ --from-file=tls.key=certs/client.root.key \ --from-file=tls.crt=certs/client.root.crt \ --from-file=ca.crt=certs/ca.crt
secret/cockroachdb.client.root.2 created
Create a new certificate and key pair for your CockroachDB nodes, overwriting the previous certificate and key. Specify the namespace you used when deploying the cluster. This example uses the Operator's default namespace (
cockroach-operator-system
):cockroach cert create-node localhost \ 127.0.0.1 \ cockroachdb-public \ cockroachdb-public.cockroach-operator-system \ cockroachdb-public.cockroach-operator-system.svc.cluster.local \ *.cockroachdb \ *.cockroachdb.cockroach-operator-system \ *.cockroachdb.cockroach-operator-system.svc.cluster.local \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.key \ --overwrite
Upload the new node certificate and key to the Kubernetes cluster as a new secret, renaming them to the filenames required by the Operator:
kubectl create secret generic cockroachdb.node.2 \ --from-file=tls.key=certs/node.key \ --from-file=tls.crt=certs/node.crt \ --from-file=ca.crt=certs/ca.crt
secret/cockroachdb.node.2 created
Add
nodeTLSSecret
andclientTLSSecret
to the Operator's custom resource, specifying the new secret names:spec: clientTLSSecret: cockroachdb.client.root.2 nodeTLSSecret: cockroachdb.node.2
Check that the secrets were created on the cluster:
kubectl get secrets
NAME TYPE DATA AGE cockroachdb.client.root.2 Opaque 3 4s cockroachdb.node.2 Opaque 3 1s default-token-6js7b kubernetes.io/service-account-token 3 9h
Note:Remember that
nodeTLSSecret
andclientTLSSecret
in the Operator's custom resource must specify these secret names. For details, see Use a custom CA.Apply the new settings to the cluster:
kubectl apply -f example.yaml
The pods will terminate and restart one at a time, using the new certificates.
You can observe this process:
kubectl get pods
NAME READY STATUS RESTARTS AGE cockroach-operator-655fbf7847-lvz6x 1/1 Running 0 4h29m cockroachdb-0 1/1 Running 0 4h16m cockroachdb-1 1/1 Terminating 0 4h16m cockroachdb-2 1/1 Running 0 43s
Delete the existing client secret that is no longer in use:
kubectl delete secret cockroachdb.client.root
secret "cockroachdb.client.root" deleted
Delete the existing node secret that is no longer in use:
kubectl delete secret cockroachdb.node
secret "cockroachdb.node" deleted
The Helm chart includes values to configure a Kubernetes cron job that regularly rotates certificates before they expire.
If you previously authenticated with cockroach cert
, follow these steps to ensure the certificates are rotated:
Upload the CA certificate that you previously created to the Kubernetes cluster as a secret:
kubectl create secret generic cockroachdb.ca \ --from-file=certs/ca.crt
secret/cockroachdb.ca created
Specify the following in the custom values file you created when deploying the cluster, using the generated secret name:
selfSigner: enabled: true caProvided: true caSecret: cockroachdb.ca rotateCerts: true
Note:selfSigner.enabled
andselfSigner.rotateCerts
aretrue
by default in the Helm chart values.Customize the following
selfSigner
fields to set the frequency of certificate rotation. These should correspond to the durations of the CA, client, and node certificates.selfSigner: minimumCertDuration: 624h caCertDuration: 43800h caCertExpiryWindow: 648h clientCertDuration: 672h clientCertExpiryWindow: 48h nodeCertDuration: 8760h nodeCertExpiryWindow: 168h
caCertDuration
,clientCertDuration
, andnodeCertDuration
specify the duration in hours of the CA, client, and node certificates, respectively.caCertExpiryWindow
,clientCertExpiryWindow
, andnodeCertExpiryWindow
specify the timeframe in hours during which the CA, client, and node certificates, respectively, should be rotated before they expire.minimumCertDuration
specifies the minimum duration in hours for all certificates. This is to ensure that the client and node certificates are rotated within the duration of the CA certificate. This value must be less than:cacertExpiryWindow
- The difference of
clientCertDuration
andclientExpiryWindow
- The difference of
nodeCertDuration
andnodeCertExpiryWindow
Certificate duration is configured when running
cockroach cert
. You can check the expiration dates of thecockroach cert
certificates by running:cockroach cert list --certs-dir=certs
For each certificate, the output includes its certificate file and expiration, the key file for node and client certificates, a Notes column with additional details, and an Errors column that is empty unless there is an error.
Apply the custom values to override the default Helm chart values:
helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb
The certificates will be rotated during the specified expiry windows.
Deploy cert-manager
for mTLS
Cockroach Labs recommends using cert-manager
to sign certificates for cluster authentication. cert-manager
manages certificates and certificate issuers as resource types in Kubernetes clusters, to simplify the process of obtaining, renewing and using those certificates.
Previously, the Helm chart used a self-signer for cluster authentication. This approach is no longer recommended.
Install a supported version of
cert-manger
. For a new cluster, Cockroach Labs recommends using the latest supported version. Refer to installed you will find itcert-manager
Installation in thecert-manager
project's documentation.Create a file named
issuer.yaml
that configures anIssuer
, which represents a certificate authority that can sign certificates. This example creates an issuer that can sign self-signed CA certificates. To customize your issuer, refer to Issuer Configuration in thecert-manager
project's documentation.apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: cockroachdb spec: selfSigned: {}
Use
kubectl apply
to create the issuer from the YAML file:kubectl apply -f issuer.yaml
Enable and configure
cert-manager
in the Helm chart'svalues.yaml
file. The following options are required. For more options, refer tocert-manager
in the CockroachDB Helm chart documentation.tls.certs.selfSigner.enabled: false tls.certs.certManager: true tls.certs.certManagerIssuer.kind: Issuer tls.certs.certManagerIssuer.name: cockroachdb
- To disable signing self-signed certificates, set
tls.certs.selfSigner.enabled
tofalse
. - Set
tls.certs.certManagerIssuer.kind
to eitherIssuer
orClusterIssuer
. To get started,Issuer
is recommended.ClusterIssuer
is cluster-scoped; when referencing a secret via thesecretName
field, only secrets in thecluster-resource
namespace (cert-manager
by default) are searched. To learn more, refer to Cluster Resource Namespace in thecert-manager
project's documentation. - Set
certManagerIssuer.name
to the name of the issuer you created in the previous step.
- To disable signing self-signed certificates, set
Apply the updated Helm chart:
helm install my-release --values values.yaml cockroachdb/cockroachdb
Replace
values.yaml
with the name of your Helm chart's values file.
Secure the webhooks
The Operator ships with both mutating and validating webhooks. Communication between the Kubernetes API server and the webhook service must be secured with TLS.
By default, the Operator searches for the TLS secret cockroach-operator-webhook-ca
, which contains a CA certificate. If the secret is not found, the Operator auto-generates cockroach-operator-webhook-ca
with a CA certificate for future runs.
The Operator then generates a one-time server certificate for the webhook server that is signed with cockroach-operator-webhook-ca
. Finally, the CA bundle for both mutating and validating webhook configurations is patched with the CA certificate.
You can also use your own certificate authority rather than cockroach-operator-webhook-ca
. Both the certificate and key files you generate must be PEM-encoded. See the following example.
Example: Using OpenSSL to secure the webhooks
These steps demonstrate how to use the openssl genrsa
and openssl req
subcommands to secure the webhooks on a running Kubernetes cluster:
Generate a 4096-bit RSA private key:
openssl genrsa -out tls.key 4096
Generate an X.509 certificate, valid for 10 years. You will be prompted for the certificate field values.
openssl req -x509 -new -nodes -key tls.key -sha256 -days 3650 -out tls.crt
Create the secret, making sure that you are in the correct namespace:
kubectl create secret tls cockroach-operator-webhook-ca --cert=tls.crt --key=tls.key
secret/cockroach-operator-webhook-ca created
Remove the certificate and key from your local environment:
rm tls.crt tls.key
Roll the Operator deployment to ensure a new server certificate is generated:
kubectl rollout restart deploy/cockroach-operator-manager
deployment.apps/cockroach-operator-manager restarted