The Runtime dashboard in the DB Console lets you monitor runtime metrics for you cluster, such as node count, memory usage, and CPU time.
To view this dashboard, access the DB Console, click Metrics on the left-hand navigation bar, and select Dashboard > Runtime.
Dashboard navigation
Use the Graph menu to display metrics for your entire cluster or for a specific node.
To the right of the Graph and Dashboard menus, a time interval selector allows you to filter the view for a predefined or custom time interval. Use the navigation buttons to move to the previous, next, or current time interval. When you select a time interval, the same interval is selected in the SQL Activity pages. However, if you select 10 or 30 minutes, the interval defaults to 1 hour in SQL Activity pages.
When viewing graphs, two perpendicular lines will appear at your mouse cursor providing further insight into the data. The metric values are displayed in the legend under the graph. Click anywhere within the graph to pin the values in place, decoupling the values from your mouse movements. Click anywhere within the graph to cause the values to change with your mouse movements once more.
Hovering your mouse cursor over the graph title will display a tooltip with a description and the metrics used to create the graph.
The Runtime dashboard displays the following time series graphs:
Live Node Count
In the node view as well as the cluster view, the graph shows the number of live nodes in the cluster.
A dip in the graph indicates decommissioned nodes, dead nodes, or nodes that are not responding. To troubleshoot the dip in the graph, refer to the Summary panel.
Memory Usage
In the node view, the graph shows the memory in use for the selected node.
In the cluster view, the graph shows the memory in use across all nodes in the cluster.
On hovering over the graph, the values for the following metrics are displayed:
| Metric | Description |
|---|---|
| RSS | Total memory in use by CockroachDB. |
| Go Allocated | Memory allocated by the Go layer. |
| Go Total | Total memory managed by the Go layer. |
| CGo Allocated | Memory allocated by the C layer. |
| CGo Total | Total memory managed by the C layer. |
Expected values for a healthy cluster: Go Allocated will depend on workload but should not exceed --max-sql-memory by more than 100%. CGo Allocated should not exceed the --cache size and CGo Total should not exceed the --cache size by more than 15%.
Runnable Goroutines per CPU
This graph shows the number of Goroutines waiting to run per CPU, as tracked by the sys.runnable.goroutines.per.cpu metric. This graph should rise and fall based on CPU load. Values greater than 50 are considered high. A value greater than the value set in admission.kv_slot_adjuster.overload_threshold (default = 32), sampled at 1ms, is used by admission control to throttle regular CPU work.
- In the node view, the graph shows the number of Goroutines waiting per CPU on the selected node.
- In the cluster view, the graph shows the number of Goroutines waiting per CPU across all nodes in the cluster.
CPU Time
- In the node view, the graph shows the CPU time used by CockroachDB user and system-level operations for the selected node.
- In the cluster view, the graph shows the CPU time used by CockroachDB user and system-level operations across all nodes in the cluster.
On hovering over the CPU Time graph, the values for the following metrics are displayed:
| Metric | Description |
|---|---|
| User CPU Time | Total CPU seconds per second used by the CockroachDB process across all nodes. |
| Sys CPU Time | Total CPU seconds per second used for CockroachDB system-level operations across all nodes. |
Clock Offset
- In the node view, the graph shows the mean clock offset of the node against the rest of the cluster.
- In the cluster view, the graph shows the mean clock offset of each node against the rest of the cluster.
Other graphs
The Runtime dashboard shows other time series graphs that are important for CockroachDB developers:
- Goroutine Count
- GC Runs
- GC Pause Time
For monitoring CockroachDB, it is sufficient to use the Live Node Count, Memory Usage, CPU Time, and Clock Offset graphs.
Summary and events
Summary panel
A Summary panel of key metrics is displayed to the right of the timeseries graphs.
| Metric | Description |
|---|---|
| Total Nodes | The total number of nodes in the cluster. Decommissioned nodes are not included in this count. |
| Capacity Used | The storage capacity used as a percentage of usable capacity allocated across all nodes. |
| Unavailable Ranges | The number of unavailable ranges in the cluster. A non-zero number indicates an unstable cluster. |
| Queries per second | The total number of SELECT, UPDATE, INSERT, and DELETE queries executed per second across the cluster. |
| P99 Latency | The 99th percentile of service latency. |
If you are testing your deployment locally with multiple CockroachDB nodes running on a single machine (this is not recommended in production), you must explicitly set the store size per node in order to display the correct capacity. Otherwise, the machine's actual disk capacity will be counted as a separate store for each node, thus inflating the computed capacity.
Events panel
Underneath the Summary panel, the Events panel lists the 5 most recent events logged for all nodes across the cluster. To list all events, click View all events.
The following types of events are listed:
- Database created
- Database dropped
- Table created
- Table dropped
- Table altered
- Index created
- Index dropped
- View created
- View dropped
- Schema change reversed
- Schema change finished
- Node joined
- Node decommissioned
- Node restarted
- Cluster setting changed
See also
Was this helpful?
