The cockroach debug zip command connects to your cluster and gathers information from each active node into a single .zip file (inactive nodes are not included). For details on the .zip contents, see Files.
You can use the cockroach debug merge-logs command in conjunction with cockroach debug zip to merge the collected logs into one file, making them easier to parse.
The files produced by cockroach debug zip can contain highly sensitive, identifiable information, such as usernames, hashed passwords, and possibly table data. Use the --redact-logs flag to redact sensitive data from log files and crash reports before sharing them with Cockroach Labs.
Details
Use cases
There are two scenarios in which debug zip is useful:
To collect all of your nodes' logs, which you can then parse to locate issues. You can optionally use the flags to retrieve only the log files. For more information about logs, see Logging. Also note:
- Nodes that are currently down cannot deliver their logs over the network. For these nodes, you must log on to the machine where the
cockroachprocess would otherwise be running, and gather the files manually. - Nodes that are currently up but disconnected from other nodes (e.g., because of a network partition) may not be able to respond to
debug ziprequests forwarded by other nodes, but can still respond to requests for data when asked directly. In such situations, we recommend using the--hostflag to pointdebug zipat each of the disconnected nodes until data has been gathered for the entire cluster.
- Nodes that are currently down cannot deliver their logs over the network. For these nodes, you must log on to the machine where the
If you experience severe or difficult-to-reproduce issues with your cluster, Cockroach Labs might ask you to send us your cluster's debugging information using
cockroach debug zip.
Files
Log files, heap profiles, CPU profiles, and goroutine dumps can greatly increase the size of the cockroach debug zip output. If you are collecting data from a large cluster, we recommend first experimenting with cockroach debug list-files and the --exclude-files, --include-files, --files-from, and/or --files-until flags to limit the .zip file size.
These files collected by cockroach debug zip, which are found in the individual node directories, can be filtered using the --exclude-files, --include-files, --files-from, and/or --files-until flags:
| Information | Filename |
|---|---|
| Log files | cockroach-{log-file-group}.{host}.{user}.{start timestamp in UTC}.{process ID}.log |
| Goroutine dumps | goroutine_dump.{date-and-time}.{metadata}.double_since_last_dump.{metadata}.txt.gz |
| Heap profiles | memprof.{date-and-time}.{heapsize}.pprof |
| Memory statistics | memstats.{date-and-time}.{heapsize}.txt |
| CPU profiles | cpuprof.{date-and-time} |
The following information is also contained in the .zip file, and cannot be filtered:
- Cluster events
- Database details
- Schema change events
- Database, table, node, and range lists
- Node details
- Node liveness
- Gossip data
- Stack traces
- Range details
- Jobs
- Cluster Settings
- Metrics
- Problem ranges
- Thread stack traces (Linux only)
- CPU profiles
- A script (
hot-ranges.sh) that summarizes the hottest ranges (ranges receiving a high number of reads or writes)
Subcommands
While the cockroach debug command has a few subcommands, users are expected to use only the zip, encryption-active-key, merge-logs, list-files, and ballast subcommands.
The other debug subcommands are useful only to CockroachDB's developers and contributors.
Synopsis
$ cockroach debug zip {ZIP file destination} {flags}
The following flags must apply to an active CockroachDB node. If no nodes are live, you must start at least one node.
Flags
The debug zip subcommand supports the following general-use, client connection, and logging flags.
General
| Flag | Description |
|---|---|
--cpu-profile-duration |
Fetch CPU profiles from the cluster with the specified sample duration in seconds. The debug zip command will block for the duration specified. A value of 0 disables this feature.Default: 5 |
--concurrency |
The maximum number of nodes to concurrently poll for data. This can be any value between 1 and 15. |
--exclude-files |
Files to exclude from the generated .zip. This can be used to limit the size of the generated .zip, and affects logs, heap profiles, goroutine dumps, and/or CPU profiles. The files are specified as a comma-separated list of glob patterns. For example:--exclude-files=*.logNote that this flag is applied after --include_files. Use cockroach debug list-files with this flag to see a list of files that will be contained in the .zip. |
--exclude-nodes |
Specify nodes to exclude from inspection as a comma-separated list or range of node IDs. For example:--exclude-nodes=1,10,13-15 |
--files-from |
Start timestamp for log file, goroutine dump, and heap profile collection. This can be used to limit the size of the generated .zip, which is increased by these files. The timestamp uses the format YYYY-MM-DD, followed optionally by HH:MM:SS or HH:MM. For example:--files-from='2021-07-01 15:00'When specifying a narrow time window, we recommend adding extra seconds/minutes to account for uncertainties such as clock drift. Default: 48 hours before now |
--files-until |
End timestamp for log file, goroutine dump, and heap profile collection. This can be used to limit the size of the generated .zip, which is increased by these files. The timestamp uses the format YYYY-MM-DD, followed optionally by HH:MM:SS or HH:MM. For example:--files-until='2021-07-01 16:00'When specifying a narrow time window, we recommend adding extra seconds/minutes to account for uncertainties such as clock drift. Default: 24 hours beyond now (to include files created during .zip creation) |
--include-files |
Files to include in the generated .zip. This can be used to limit the size of the generated .zip, and affects logs, heap profiles, goroutine dumps, and/or CPU profiles. The files are specified as a comma-separated list of glob patterns. For example:--include-files=*.pprofNote that this flag is applied before --exclude-files. Use cockroach debug list-files with this flag to see a list of files that will be contained in the .zip. |
--nodes |
Specify nodes to inspect as a comma-separated list or range of node IDs. For example:--nodes=1,10,13-15 |
--redact-logs |
Redact sensitive data from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed above) collected by the debug zip command may still contain sensitive information. |
--timeout |
Return an error if the command does not conclude within a specified nonzero value. The timeout is suffixed with s (seconds), m (minutes), or h (hours). For example:--timeout=2m |
Client connection
| Flag | Description |
|---|---|
--cert-principal-map |
A comma-separated list of <cert-principal>:<db-principal> mappings. This allows mapping the principal in a cert to a DB principal such as node or root or any SQL user. This is intended for use in situations where the certificate management system places restrictions on the Subject.CommonName or SubjectAlternateName fields in the certificate (e.g., disallowing a CommonName like node or root). If multiple mappings are provided for the same <cert-principal>, the last one specified in the list takes precedence. A principal not specified in the map is passed through as-is via the identity function. A cert is allowed to authenticate a DB principal if the DB principal name is contained in the mapped CommonName or DNS-type SubjectAlternateName fields. |
--certs-dir |
The path to the certificate directory containing the CA and client certificates and client key. Env Variable: COCKROACH_CERTS_DIRDefault: ${HOME}/.cockroach-certs/ |
--cluster-name |
The cluster name to use to verify the cluster's identity. If the cluster has a cluster name, you must include this flag. For more information, see cockroach start. |
--disable-cluster-name-verification |
Disables the cluster name check for this command. This flag must be paired with --cluster-name. For more information, see cockroach start. |
--host |
The server host and port number to connect to. This can be the address of any node in the cluster. Env Variable: COCKROACH_HOSTDefault: localhost:26257 |
--insecure |
Use an insecure connection. Env Variable: COCKROACH_INSECUREDefault: false |
--url |
A connection URL to use instead of the other arguments. Env Variable: COCKROACH_URLDefault: no URL |
Logging
By default, this command logs messages to stderr. This includes events with WARNING severity and higher.
If you need to troubleshoot this command's behavior, you can customize its logging behavior.
Examples
Generate a debug zip file
Generate the debug zip file for an insecure cluster:
$ cockroach debug zip ./cockroach-data/logs/debug.zip --insecure --host=200.100.50.25
Generate the debug zip file for a secure cluster:
$ cockroach debug zip ./cockroach-data/logs/debug.zip --host=200.100.50.25
Secure examples assume you have the appropriate certificates in the default certificate directory, ${HOME}/.cockroach-certs/.
Generate a debug zip file with logs only
Generate a debug zip file containing only log files:
$ cockroach debug zip ./cockroach-data/logs/debug.zip --include-files=*.log
Redact sensitive information from the logs
Example of a log string without redaction enabled:
server/server.go:1423 ⋮ password of user ‹admin› was set to ‹"s3cr34?!@x_"›
Enable log redaction:
$ cockroach debug zip ./cockroach-data/logs/debug.zip --redact-logs --insecure --host=200.100.50.25
server/server.go:1423 ⋮ password of user ‹×› was set to ‹×›
See also
Was this helpful?
