Certificate Management with the CockroachDB Operator

This page describes how to manage TLS certificates in a deployment.

The CockroachDB operator is in .

TLS configuration options

You can either allow the operator to generate self-signed certificates, provide a custom CA certificate and generate all other certificates, or use your own certificates.

All self-signed certificates

By default, the certificates are created automatically by a self-signer utility, which requires no configuration beyond setting a custom certificate duration if desired. This utility uses cockroach cert to automatically generate self-signed certificates for the nodes and root client which are stored in a secret. You can see these certificates by running kubectl get secrets:

kubectl get secrets

crdb-cockroachdb-ca-secret                 Opaque                                2      23s
crdb-cockroachdb-client-secret             kubernetes.io/tls                     3      22s
crdb-cockroachdb-node-secret               kubernetes.io/tls                     3      23s

If you are deploying on OpenShift you must also set cockroachdb.tls.selfSigner.securityContext.enabled to false to mitigate stricter security policies.

Custom CA certificate

If you wish to supply your own CA certificates to the deployed nodes but allow automatic generation of client certificates, create a Kubernetes secret with the custom CA certificate. To perform these steps using the cockroach cert command:

mkdir certs

mkdir my-safe-directory

cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key

Set cockroachdb.tls.selfSigner.caProvided to true and specify the secret where the certificate is stored:

cockroachdb:
  tls:
    enabled: true
    selfSigner:
      enabled: true
      caProvided: true
      caSecret: {ca-secret-name}

If you are deploying on OpenShift you must also set cockroachdb.tls.selfSigner.securityContext.enabled to false to mitigate stricter security policies.

All custom certificates

Set up your certificates and load them into your Kubernetes cluster as secrets using the following commands:

mkdir certs

mkdir my-safe-directory

cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key

cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key

kubectl create secret generic cockroachdb-root --from-file=certs

secret/cockroachdb-root created

cockroach cert create-node --certs-dir=certs --ca-key=my-safe-directory/ca.key localhost 127.0.0.1 my-release-cockroachdb-public my-release-cockroachdb-public.cockroach-ns my-release-cockroachdb-public.cockroach-ns.svc.cluster.local *.my-release-cockroachdb *.my-release-cockroachdb.cockroach-ns *.my-release-cockroachdb.cockroach-ns.svc.cluster.local
kubectl create secret generic cockroachdb-node --from-file=certs

secret/cockroachdb-node created

The subject alternative names are based on a release called my-release in the cockroach-ns namespace. Make sure they match the services created with the release during Helm install.

To optionally supply certificates with cert-manager, set cockroachdb.tls.certManager.enabled to true, and cockroachdb.tls.certManager.issuer to an IssuerRef (as they appear in certificate resources) pointing to a clusterIssuer or issuer that you have set up in the cluster:

cockroachdb:
  tls:
    enabled: true
    certManager:
      enabled: true
      caConfigMap: cockroachdb-ca
      nodeSecret: cockroachdb-node
      clientRootSecret: cockroachdb-root
      issuer:
        group: cert-manager.io
        kind: Issuer
        name: cockroachdb-cert-issuer
        clientCertDuration: 672h
        clientCertExpiryWindow: 48h
        nodeCertDuration: 8760h
        nodeCertExpiryWindow: 168h

The following Kubernetes application describes an example issuer, including the trust manager to automatically copy the CA certificate from a secret to a ConfigMap:

apiVersion: v1
kind: Secret
metadata:
  name: cockroachdb-ca
  namespace: cockroach-ns
data:
  tls.crt: [BASE64 Encoded ca.crt]
  tls.key: [BASE64 Encoded ca.key]
type: kubernetes.io/tls
---
apiVersion: cert-manager.io/v1alpha3
kind: Issuer
metadata:
  name: cockroachdb-cert-issuer
  namespace: cockroach-ns
spec:
  ca:
    secretName: cockroachdb-ca
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: cockroachdb-ca
spec:
  sources:

    - secret:
        name: cockroachdb-ca
        key: tls.crt
  target:
    configMap:
      key: ca.crt
    namespaceSelector:
      matchLabels:
       kubernetes.io/metadata.name: cockroachdb

If your certificates are stored in TLS secrets, such as secrets generated by cert-manager, the secret will contain files named: ca.crt, tls.crt, and tls.key. For CockroachDB, rename these files as applicable to match the following naming scheme: ca.crt, node.crt, node.key, client.root.crt, and client.root.key. Add the following to the values file:

cockroachdb:
  tls:
    enabled: true
    externalCertificates:
      enabled: true
      certificates:
        nodeSecretName: {node_secret_name}
        nodeClientSecretName: {client_secret_name}

Replace the following placeholder values:

: The name of the Kubernetes secret that contains the generated node certificate and key.
: The name of the Kubernetes secret that contains the generated client certificate and key.

Example: Generate and sign custom certificates using `cockroach cert`

The following example uses to generate and sign the CockroachDB node and client certificates. To learn more about the supported methods of signing certificates, refer to .

Create two directories:
mkdir certs my-safe-directory

Create the CA certificate and key pair:

cockroach cert create-ca \
  --certs-dir=certs \
  --ca-key=my-safe-directory/ca.key

Create a client certificate and key pair for the root user:

cockroach cert create-client root \
  --certs-dir=certs \
  --ca-key=my-safe-directory/ca.key

Upload the client certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the :

kubectl create secret generic cockroachdb.client.root \
  --from-file=tls.key=certs/client.root.key \
  --from-file=tls.crt=certs/client.root.crt \
  --from-file=ca.crt=certs/ca.crt

secret/cockroachdb.client.root created

Create the certificate and key pair for your CockroachDB nodes, specifying the namespace you used when deploying the cluster. This example uses the cockroach-ns namespace:

cockroach cert create-node localhost \
  127.0.0.1 \
  cockroachdb-public \
  cockroachdb-public.cockroach-ns \
  cockroachdb-public.cockroach-ns.svc.cluster.local \
  *.cockroachdb \
  *.cockroachdb.cockroach-ns \
  *.cockroachdb.cockroach-ns.svc.cluster.local \
  --certs-dir=certs \
  --ca-key=my-safe-directory/ca.key

Upload the node certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the :

kubectl create secret generic cockroachdb.node \
  --from-file=tls.key=certs/node.key \
  --from-file=tls.crt=certs/node.crt \
  --from-file=ca.crt=certs/ca.crt

secret/cockroachdb.node created

Check that the secrets were created on the cluster:

kubectl get secrets

NAME                      TYPE                                   DATA   AGE
cockroachdb.client.root   Opaque                                   3    13s
cockroachdb.node          Opaque                                   3     3s
default-token-6js7b       kubernetes.io/service-account-token      3     9h

Add cockroachdb.tls.externalCertificates.certificates.nodeSecretName and cockroachdb.tls.externalCertificates.certificates.nodeClientSecretName to the values file used to deploy the cluster:

cockroachdb:
tls:
  enabled: true
  externalCertificates:
    enabled: true
    certificates:
      nodeSecretName: cockroachdb.node
      nodeClientSecretName: cockroachdb.client.root

Rotate security certificates

You may need to rotate the node, client, or CA certificates in the following scenarios:

The node, client, or CA certificates are expiring soon.
Your organization’s compliance policy requires periodic certificate rotation.
The key (for a node, client, or CA) is compromised.
You need to modify the contents of a certificate, for example, to add another DNS name or the IP address of a load balancer through which a node can be reached. In this case, you would need to rotate only the node certificates.

Certificates generated by the self-signer utility are re-generated by the utility when the expiry time is near. Certificates managed with cert-manager are also automatically renewed. Whether the new certificates are generated automatically or manually, the CockroachDB pods must be restarted in order to pick up the new certificates.

Example: Manually rotate certificates signed with `cockroach cert`

If you previously created and signed certificates with cockroach cert, follow these steps to manually rotate the certificates using the same CA:

Create a new client certificate and key pair for the root user, overwriting the previous certificate and key:

cockroach cert create-client root \
  --certs-dir=certs \
  --ca-key=my-safe-directory/ca.key \
  --overwrite

Upload the new client certificate and key to the Kubernetes cluster as a new secret, renaming them to the filenames required by the :

kubectl create secret generic cockroachdb.client.root.2 \
  --from-file=tls.key=certs/client.root.key \
  --from-file=tls.crt=certs/client.root.crt \
  --from-file=ca.crt=certs/ca.crt

secret/cockroachdb.client.root.2 created

Create a new certificate and key pair for your CockroachDB nodes, overwriting the previous certificate and key. Specify the namespace you used when . This example uses the cockroach-ns namespace:

cockroach cert create-node localhost \
  127.0.0.1 \
  cockroachdb-public \
  cockroachdb-public.cockroach-ns \
  cockroachdb-public.cockroach-ns.svc.cluster.local \
  *.cockroachdb \
  *.cockroachdb.cockroach-ns \
  *.cockroachdb.cockroach-ns.svc.cluster.local \
  --certs-dir=certs \
  --ca-key=my-safe-directory/ca.key \
  --overwrite

Upload the new node certificate and key to the Kubernetes cluster as a new secret, renaming them to the filenames required by the :

kubectl create secret generic cockroachdb.node.2 \
  --from-file=tls.key=certs/node.key \
  --from-file=tls.crt=certs/node.crt \
  --from-file=ca.crt=certs/ca.crt

secret/cockroachdb.node.2 created

Add cockroachdb.tls.externalCertificates.certificates.nodeClientSecretName and cockroachdb.tls.externalCertificates.certificates.nodeSecretName to the values file used to :

cockroachdb:
  tls:
    externalCertificates:
      enabled: true
      certificates:
        nodeClientSecretName: "cockroachdb.client.root.2"
        nodeSecretName: "cockroachdb.node.2"

Check that the secrets were created on the cluster:

kubectl get secrets

NAME                        TYPE                              DATA   AGE
cockroachdb.client.root.2   Opaque                               3    4s
cockroachdb.node.2          Opaque                               3    1s
default-token-6js7b         kubernetes.io/service-account-token  3    9h

Remember that nodeSecretName and nodeClientSecretName in the operator configuration must specify these secret names. For details, see the .

Apply the new settings to the cluster:

helm upgrade --reuse-values $CRDBCLUSTER ./cockroachdb-parent/charts/cockroachdb --values ./cockroachdb-parent/charts/cockroachdb/values.yaml -n $NAMESPACE

The pods will terminate and restart one at a time, using the new certificates. You can observe this process:

kubectl get pods

NAME                                  READY   STATUS        RESTARTS   AGE
cockroach-operator-655fbf7847-lvz6x   1/1     Running         0      4h29m
cockroachdb-0                         1/1     Running         0      4h16m
cockroachdb-1                         1/1     Terminating     0      4h16m
cockroachdb-2                         1/1     Running         0        43s

Delete the existing client secret that is no longer in use:

kubectl delete secret cockroachdb.client.root

secret "cockroachdb.client.root" deleted

Delete the existing node secret that is no longer in use:

kubectl delete secret cockroachdb.node

secret "cockroachdb.node" deleted

Secure the webhooks

The operator ships with both mutating and validating webhooks. Communication between the Kubernetes API server and the webhook service must be secured with TLS. By default, the searches for the TLS secret cockroach-operator-certs, which contains a CA certificate. If the secret is not found, the operator auto-generates cockroach-operator-certs with a CA certificate for future runs. The operator then generates a one-time server certificate for the webhook server that is signed with cockroach-operator-certs. Finally, the CA bundle for both mutating and validating webhook configurations is patched with the CA certificate. You can also use your own certificate authority rather than cockroach-operator-certs. Both the certificate and key files you generate must be PEM-encoded. See the following example.

Example: Using OpenSSL to secure the webhooks

These steps demonstrate how to use the openssl genrsa and openssl req subcommands to secure the webhooks on a running Kubernetes cluster:

Generate a 4096-bit RSA private key:
openssl genrsa -out tls.key 4096
Generate an X.509 certificate, valid for 10 years. You will be prompted for the certificate field values.
openssl req -x509 -new -nodes -key tls.key -sha256 -days 3650 -out tls.crt

Create the secret, making sure that you are in the correct namespace:

kubectl create secret tls cockroach-operator-certs --cert=tls.crt --key=tls.key

secret/cockroach-operator-certs created

Remove the certificate and key from your local environment:
rm tls.crt tls.key

Roll the operator deployment to ensure a new server certificate is generated:

kubectl rollout restart deploy/cockroach-operator-manager

deployment.apps/cockroach-operator-manager restarted

​TLS configuration options

​All self-signed certificates

​Custom CA certificate

​All custom certificates

​Example: Generate and sign custom certificates using cockroach cert

​Rotate security certificates

​Example: Manually rotate certificates signed with cockroach cert

​Secure the webhooks

​Example: Using OpenSSL to secure the webhooks

TLS configuration options

All self-signed certificates

Custom CA certificate

All custom certificates

Example: Generate and sign custom certificates using `cockroach cert`

Rotate security certificates

Example: Manually rotate certificates signed with `cockroach cert`

Secure the webhooks

Example: Using OpenSSL to secure the webhooks