Export Logs From a CockroachDB Dedicated Cluster

Perform the following steps to enable log export from your CockroachDB Dedicated cluster to AWS CloudWatch.

1. Create the desired target AWS CloudWatch log group by following the Create a log group in CloudWatch logs instructions. If you already have a log group created, you may skip this step. To send logs to more than one target log group, see the custom configuration option in step 9 below.

2. Find your CockroachDB Dedicated cluster ID:

   1. Visit the CockroachDB Cloud console cluster page.
   2. Click on the name of your cluster.
   3. Find your cluster ID in the URL of the single cluster overview page: https://cockroachlabs.cloud/cluster/{your_cluster_id}/overview.

3. Determine your CockroachDB Dedicated cluster's associated AWS Account ID. This command uses the third-party JSON parsing tool jq to isolate just the needed account_id field from the results:

   curl --request GET \
     --url https://cockroachlabs.cloud/api/v1/clusters/{your_cluster_id} \
     --header 'Authorization: Bearer {secret_key}' | jq .account_id
   

   See API Access for instructions on generating the {secret_key}.

4. Create a cross-account IAM role in your AWS account:

   1. In the AWS console, visit the IAM page.
   2. Select Roles and click Create role.
   3. For Trusted entity type, select AWS account.
   4. Choose Another AWS account.
   5. For Account ID, provide the CockroachDB Dedicated AWS Account ID from step 3.
   6. Finish creating the IAM role with a suitable name. These instructions will use the role name CockroachCloudLogExportRole. You do not need to add any permissions.
   Note:

   You will need the Amazon Resource Name (ARN) for your cross-account IAM role later in this procedure.

5. Select the new role, and create a new policy for this role. These instructions will use the policy name CockroachCloudLogExportPolicy.

6. Select the new policy, and paste the following into the Permissions tab, with the {} JSON option selected:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Action": [
                   "logs:CreateLogGroup",
                   "logs:CreateLogStream",
                   "logs:DescribeLogGroups",
                   "logs:DescribeLogStreams",
                   "logs:PutRetentionPolicy",
                   "logs:PutLogEvents"
               ],
               "Effect": "Allow",
               "Resource": [
                   "arn:aws:logs:*:{your_aws_acct_id}:log-group:{log_group_name}:*"
               ]
           }
       ]
   }
   

   Where:

   • {your_aws_acct_id} is the AWS Account ID of the AWS account where you created the CockroachCloudLogExportRole role, not the AWS Account ID of your CockroachDB Dedicated cluster. You can find your AWS Account ID on the AWS IAM page.
   • {log_group_name} is the target AWS CloudWatch log group you created in step 1.

   This defines the set of permissions that the CockroachDB Dedicated log export feature requires to be able to write logs to CloudWatch.

   If desired, you may also limit log export from your CockroachDB Dedicated cluster to a specific single AWS region, by providing the name of the desired region as the fourth value to the Resource entry. For example:

   "Resource": [
       "arn:aws:logs:us-east-1:{your_aws_acct_id}:log-group:{log_group_name}:*"
   ]
   

   Specifying an AWS region that you do not have a cluster in, or a region that only partially covers your cluster's nodes will result in missing logs.

7. Copy the Amazon Resource Name (ARN) of the CockroachCloudLogExportRole role found under Summary, which is needed for the next step.

8. Use one of the following Cloud API commands to enable log export for your CockroachDB Dedicated cluster. The first presents a basic configuration, where all logs are sent to AWS CloudWatch using the default settings. The second allows for more detailed customization of the logging configuration, such as the ability to send certain log channels to specific target log groups, or the ability to redact sensitive log entries.

   1. To enable log export for your CockroachDB Dedicated cluster with default logging configuration, issue the following Cloud API command:

      curl --request POST \
        --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/logexport \
        --header "Authorization: Bearer {secret_key}" \
        --data '{"type": "AWS_CLOUDWATCH", "log_name": "{log_group_name}", "auth_principal": "{role_arn}"}'
      

      Where:

      • {cluster_id} is your CockroachDB Dedicated cluster ID as determined in step 3.
      • {secret_key} is your CockroachDB Dedicated API key. See API Access for instructions on generating this key.
      • {log_group_name} is the target AWS CloudWatch log group you created in step 1.
      • {role_arn} is the ARN for the CockroachCloudLogExportRole role you copied in step 8.

   2. To enable log export for your CockroachDB Dedicated cluster with custom logging configuration:

      1. Consult the log export entry on the CockroachDB Cloud API Reference and select the Schema tab to view the supported log configuration options, and determine the customized logging configuration you would like to use.

         For example, consider the following configuration:

         {
          "type": "AWS_CLOUDWATCH",
          "log_name": "default",
          "auth_principal": "{role_arn}",
          "redact": true,
          "region": "",
          "omitted_channels": [ "SESSIONS", "SQL_PERF"],
          "groups": [
                  {
                      "log_name": "sql",
                      "channels": ["SQL_SCHEMA", "SQL_EXEC"],
                      "redact": false
                  },
                  {
                      "log_name": "devops",
                      "channels": ["OPS", "HEALTH", "STORAGE"],
                      "min_level": "WARNING"
                  }
          ]
         }
         

         This configuration:

         • Enables redaction globally for all log entries emitted to AWS CloudWatch.
         • Does not send log entries in the SESSIONS and SQL_PERF logging channels.
         • Sends log entries in the SQL_SCHEMA and SQL_EXEC logging channels to a AWS CloudWatch log group named sql, and overrides (disables) the global redaction configuration for just these two log channels only.
         • Sends log entries in the OPS, HEALTH, and STORAGE logging channels to an AWS CloudWatch log group named devops, but only for those entries that are of log severity level WARNING or higher.
         • Sends log entries in all other logging channels to the default AWS CloudWatch log group.

      2. Once you have determined the configuration you'd like to use, edit the configuration to be a single line, the required form for passing to the configuration command in the next step. To accomplish this easily, use a third party minifier, such as json minifier. The above configuration becomes the following single line, suitable for the next step's POST command:

         {"type":"AWS_CLOUDWATCH","log_name":"default","auth_principal":"{role_arn}","redact":true,"region":"","omitted_channels":["SESSIONS","SQL_PERF"],"groups":[{"log_name":"sql","channels":["SQL_SCHEMA","SQL_EXEC"],"redact":false},{"log_name":"devops","channels":["OPS","HEALTH","STORAGE"],"min_level":"WARNING"}]}
         

      3. Then, to enable log export for your CockroachDB Dedicated cluster with the above example custom logging configuration, issue the following Cloud API command:

         curl --request POST \
           --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/logexport \
           --header "Authorization: Bearer {secret_key}" \
           --data '{"type":"AWS_CLOUDWATCH","log_name":"default","auth_principal":"{role_arn}","redact":true,"region":"","omitted_channels":["SESSIONS","SQL_PERF"],"groups":[{"log_name":"sql","channels":["SQL_SCHEMA","SQL_EXEC"],"redact":false},{"log_name":"devops","channels":["OPS","HEALTH","STORAGE"],"min_level":"WARNING"}]}'
         

         Where:

         • {cluster_id} is your CockroachDB Dedicated cluster ID as determined in step 3.
         • {secret_key} is your CockroachDB Dedicated API key. See API Access for instructions on generating this key.
         • {role_arn} is the ARN for the CockroachCloudLogExportRole role you copied in step 8.

9. Depending on the size of your cluster and how many regions it spans, the configuration may take a moment. You can monitor the ongoing status of the configuration using the following Cloud API command:

   curl --request GET \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/logexport \
     --header "Authorization: Bearer {secret_key}"
   

   Run the command periodically until the command returns a status of ENABLED, at which point the configuration across all nodes is complete, and logs will begin appearing in CloudWatch under the log group you created in step 1. Since the configuration is applied to cluster nodes in a rolling fashion, you may see some logs appear even before the GET command returns an ENABLED status.

10. Once log export has been enabled, you can access logs from your CockroachDB Dedicated cluster directly in AWS CloudWatch.

Perform the following steps to enable log export from your CockroachDB Dedicated cluster to GCP Cloud Logging.

1. Find your CockroachDB Dedicated organization ID in the CockroachDB Cloud organization settings page.

2. Find your CockroachDB Dedicated cluster ID:

   1. Visit the CockroachDB Cloud console cluster page.
   2. Click on the name of your cluster.
   3. Find your cluster ID in the URL of the single cluster overview page: https://cockroachlabs.cloud/cluster/{your_cluster_id}/overview.

3. Determine the GCP principal to grant permission to from your account. This principal is already created for you by Cockroach Labs; this step merely determines its account name. This command uses the third-party JSON parsing tool jq to isolate just the needed id (GCP cluster ID) and account_id (GCP account ID) fields, and combines them for you in the required form:

   curl --request GET \
     --url https://cockroachlabs.cloud/api/v1/clusters/{your_cluster_id} \
     --header 'Authorization: Bearer {secret_key}' | jq '("crl-logging-user-" + (.id | split("-"))[4] + "@" + .account_id + ".iam.gserviceaccount.com")'
   

   Where:

   • {your_cluster_id} is the cluster ID of your CockroachDB Dedicated cluster as determined in step 2.
   • {secret_key} is your API access key. See API Access for more details.

   The resulting GCP principal should resemble the following example:

   crl-logging-user-a1c42be2e53b@crl-prod-abc.iam.gserviceaccount.com
   

   This GCP principal refers to a resource that is owned by Cockroach Labs and is created automatically along with your cluster. You do not need to create this account in GCP; it is already present for use by your cluster.

4. Create a new role with the required permissions in your GCP project:

   1. In the GCP console, visit the IAM roles page for your project.
   2. Click + Create role.
   3. Give your role a title and ID of your choosing, then click + Add permissions.
   4. Search for logging.logEntries.create in the Filter field, check the checkbox next to the resulting match, then click Add.
   5. Click the Create button.

5. Add your cluster's GCP principal to the role you just created.

   1. In the GCP console, visit the IAM admin page for your project.
   2. Click the + Grant Access button.
   3. In the box labeled New principals, enter the name of your cluster's GCP principal you determined in step 3.
   4. In the Select a role dropdown, select the role you created in step 4.
   5. Click SAVE.

6. Use one of the following Cloud API commands to enable log export for your CockroachDB Dedicated cluster. The first presents a basic configuration, where all logs are sent to GCP Cloud Logging using the default settings. The second allows for more detailed customization of the logging configuration, such as the ability to send certain log channels to specific target log groups, or the ability to redact sensitive log entries.

   1. To enable log export for your CockroachDB Dedicated cluster with default logging configuration, issue the following Cloud API command:

      curl --request POST \
        --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/logexport \
        --header "Authorization: Bearer {secret_key}" \
        --data '{"type": "GCP_CLOUD_LOGGING", "log_name": "{log_name}", "auth_principal": "{gcp_project_id}"}'
      

      Where:

      • {cluster_id} is your CockroachDB Dedicated cluster ID as determined in step 3.
      • {secret_key} is your CockroachDB Dedicated API key. See API Access for instructions on generating this key.
      • {log_name} is a string of your choosing to represent logs written from your CockroachDB Dedicated cluster. This name will appear in the name of each log written to GCP Cloud Logging.
      • {gcp_project_id} is your GCP project ID, as shown in your GCP Cloud Console Settings page.

   2. To enable log export for your CockroachDB Dedicated cluster with custom logging configuration:

      1. Consult the log export entry on the CockroachDB Cloud API Reference and select the Schema tab to view the supported log configuration options, and determine the customized logging configuration you would like to use.

         For example, consider the following configuration:

         {
          "type": "GCP_CLOUD_LOGGING",
          "log_name": "default",
          "auth_principal": "{gcp_project_id}",
          "redact": true,
          "region": "",
          "omitted_channels": [ "SESSIONS", "SQL_PERF"],
          "groups": [
                  {
                      "log_name": "sql",
                      "channels": ["SQL_SCHEMA", "SQL_EXEC"],
                      "redact": false
                  },
                  {
                      "log_name": "devops",
                      "channels": ["OPS", "HEALTH", "STORAGE"],
                      "min_level": "WARNING"
                  }
          ]
         }
         

         This configuration:

         • Enables redaction globally for all log entries emitted to GCP Cloud Logging.
         • Does not send log entries in the SESSIONS and SQL_PERF logging channels.
         • Sends log entries in the SQL_SCHEMA and SQL_EXEC logging channels to a GCP Cloud Logging log group named sql, and overrides (disables) the global redaction configuration for just these two log channels only.
         • Sends log entries in the OPS, HEALTH, and STORAGE logging channels to a GCP Cloud Logging log group named devops, but only for those entries that are of log severity level WARNING or higher.
         • Sends log entries in all other logging channels to the default GCP Cloud Logging log group.

      2. Once you have determined the configuration you'd like to use, edit the configuration to be a single line, the required form for passing to the configuration command in the next step. To accomplish this easily, use a third party minifier, such as json minifier. The above configuration becomes the following single line, suitable for the next step's POST command:

         {"type":"GCP_CLOUD_LOGGING","log_name":"default","auth_principal":"{gcp_project_id}","redact":true,"region":"","omitted_channels":["SESSIONS","SQL_PERF"],"groups":[{"log_name":"sql","channels":["SQL_SCHEMA","SQL_EXEC"],"redact":false},{"log_name":"devops","channels":["OPS","HEALTH","STORAGE"],"min_level":"WARNING"}]}
         

      3. Then, to enable log export for your CockroachDB Dedicated cluster with the above example custom logging configuration, issue the following Cloud API command:

         curl --request POST \
           --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/logexport \
           --header "Authorization: Bearer {secret_key}" \
           --data '{"type":"GCP_CLOUD_LOGGING","log_name":"default","auth_principal":"{gcp_project_id}","redact":true,"region":"","omitted_channels":["SESSIONS","SQL_PERF"],"groups":[{"log_name":"sql","channels":["SQL_SCHEMA","SQL_EXEC"],"redact":false},{"log_name":"devops","channels":["OPS","HEALTH","STORAGE"],"min_level":"WARNING"}]}'
         

         Where:

         • {cluster_id} is your CockroachDB Dedicated cluster ID as determined in step 3.
         • {secret_key} is your CockroachDB Dedicated API key. See API Access for instructions on generating this key.
         • {gcp_project_id} is your GCP project ID, as shown in your GCP Cloud Console Settings page.

7. Depending on the size of your cluster and how many regions it spans, the configuration may take a moment. You can monitor the ongoing status of the configuration using the following Cloud API command:

   curl --request GET \
     --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/logexport \
     --header "Authorization: Bearer {secret_key}"
   

   Run the command periodically until the command returns a status of ENABLED, at which point the configuration across all nodes is complete, and logs will begin appearing in GCP Cloud Logging. Since the configuration is applied to cluster nodes in a rolling fashion, you may see some logs appear even before the GET command returns an ENABLED status.

8. Once log export has been enabled, you can access logs from your CockroachDB Dedicated cluster directly in GCP Cloud Logging's Log Explorer.

Perform the following steps to enable log export from your CockroachDB Dedicated cluster to Azure Monitor.

1. Find your CockroachDB Dedicated cluster ID:

   1. Visit the CockroachDB Cloud console cluster page.
   2. Click on the name of your cluster.
   3. Find your cluster ID in the URL of the single cluster overview page and note the value: https://cockroachlabs.cloud/cluster/{your_cluster_id}/overview.

2. Create a Log Analytics workspace in Azure Portal.

   1. Navigate to the Log Analytics workspace. Under Settings, click Agents.
   2. Click Log Analytics agent instructions section header.
   3. In that section, note the values for the Workspace ID and Primary key (or Secondary key)

3. Use one of the following Cloud API commands to enable log export for your CockroachDB Dedicated cluster.

• The first presents a basic configuration, where all logs are sent to Azure Monitor using the default settings.
• The second allows for more detailed customization of the logging configuration, such as the ability to send certain log channels to specific target log tables, or the ability to redact sensitive log entries.

Default

To enable log export for your CockroachDB Dedicated cluster with default logging configuration, issue the following Cloud API command:

curl --request POST \
--url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/logexport \
--header "Authorization: Bearer {secret_key}" \
--data '{"type": "AZURE_LOG_ANALYTICS", "log_name": "{log_prefix}", "auth_principal": "{workspace_id}", "azure_shared_key": "{primary_or_secondary_key}"}'

Where:

• {cluster_id} is your CockroachDB Dedicated cluster ID as previously noted.
• {secret_key} is your CockroachDB Dedicated API key. For instructions on generating this key, refer to API Access for instructions on generating this key.
• {log_prefix} is a user-specified prefix for the table that will be created in the Log Analytics workspace.
• {auth_principal} is the ID for the Log Analytics workspace, Workspace ID as previously noted.
• {azure_shared_key} is the Primary key or Secondary key as previously noted.

Custom

To enable log export for your CockroachDB Dedicated cluster with custom logging configuration:

1. Consult the log export entry on the CockroachDB Cloud API Reference to determine the customized logging configuration you would like to use. Under the REQUEST section, select the SCHEMA tab to view the supported log configuration options.

   For example, consider the following configuration:

   {
       "type": "AZURE_LOG_ANALYTICS",
       "log_name": "default",
       "auth_principal": "{workspace_id}",
       "azure_shared_key": "{primary_or_secondary_key}",
       "redact": true,
       "region": "",
       "omitted_channels": [ "SESSIONS", "SQL_PERF"],
       "groups": [
               {
                   "log_name": "sql",
                   "channels": ["SQL_SCHEMA", "SQL_EXEC"],
                   "redact": false
               },
               {
                   "log_name": "devops",
                   "channels": ["OPS", "HEALTH", "STORAGE"],
                   "min_level": "WARNING"
               }
       ]
   }
   

   This configuration:

   • Enables redaction globally for all log entries emitted to Azure Monitor.
   • Does not send log entries in the SESSIONS and SQL_PERF logging channels.
   • Sends log entries in the SQL_SCHEMA and SQL_EXEC logging channels to an Azure Monitor log table prefixed with sql, and overrides (disables) the global redaction configuration for only these two log channels.
   • Sends log entries in the OPS, HEALTH, and STORAGE logging channels to an Azure Monitor log table prefixed with devops, but only for those entries that are of log severity level WARNING or higher.
   • Sends log entries in all other logging channels to the Azure Monitor log table prefixed with default.

2. Once you have determined the configuration you'd like to use, edit the configuration to be a single line, the required form for passing to the configuration command in the next step. To accomplish this, use a third party minifier, such as json minifier. The preceding example configuration becomes the following single line, suitable for the next step's POST command:

   {"type":"AZURE_LOG_ANALYTICS","log_name":"default","auth_principal":"{workspace_id}","azure_shared_key":"{primary_or_secondary_key}","redact":true,"region":"","omitted_channels":["SESSIONS","SQL_PERF"],"groups":[{"log_name":"sql","channels":["SQL_SCHEMA","SQL_EXEC"],"redact":false},{"log_name":"devops","channels":["OPS","HEALTH","STORAGE"],"min_level":"WARNING"}]}
   

3. To enable log export for your CockroachDB Dedicated cluster with the example custom logging configuration, issue the following Cloud API command:

   curl --request POST \
       --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/logexport \
       --header "Authorization: Bearer {secret_key}" \
       --data '{"type":"AZURE_LOG_ANALYTICS","log_name":"default","auth_principal":"{workspace_id}","azure_shared_key":"{primary_or_secondary_key}","redact":true,"region":"","omitted_channels":["SESSIONS","SQL_PERF"],"groups":[{"log_name":"sql","channels":["SQL_SCHEMA","SQL_EXEC"],"redact":false},{"log_name":"devops","channels":["OPS","HEALTH","STORAGE"],"min_level":"WARNING"}]}'
   

   Where:

   • {cluster_id} is your CockroachDB Dedicated cluster ID as previously noted.
   • {secret_key} is your CockroachDB Dedicated API key. See API Access for instructions on generating this key.
   • {auth_principal} is the ID for the Log Analytics workspace, Workspace ID as previously noted.
   • {azure_shared_key} is the Primary key or Secondary key as previously noted.

Verify

• Depending on the size of your cluster and how many regions it spans, the configuration may take a moment. You can monitor the ongoing status of the configuration using the following Cloud API command:

  curl --request GET \
    --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/logexport \
    --header "Authorization: Bearer {secret_key}"
  

  When the command returns a status of ENABLED, the configuration has been applied to all nodes, and logs will begin appearing in Azure Monitor. Since the configuration is applied to cluster nodes one at a time, logs may begin to appear even before the status is ENABLED.

• Once log export has been enabled, you can access logs from your CockroachDB Dedicated cluster directly in Azure Portal.

  • Navigate to the Log Analytics workspace created previously.
  • In the left menu, under Settings, click Tables. In the Table names column, there should be a table name with the following pattern: {log_prefix}_{region}_CL. For example, crdb_eastus_CL where {log_prefix} is the string passed to log_name in the Cloud API command (crdb in this case), {region} is the region of your cluster (eastus in this case), and CL is the suffix for a classic custom table.
  • To view the logs in this table, in the left menu, click Logs. In a New Query window, enter the table name and click Run.