Why is my process hanging when I try to start it in the background?
The first question that needs to be asked is whether or not you have previously run a multi-node cluster using the same data directory. If you haven't, then you should check out our Cluster Setup Troubleshooting docs. If you have previously started and stopped a multi-node cluster and are now trying to bring it back up, you're in the right place.
In order to keep your data consistent, CockroachDB only works when at least a
majority of its nodes are running. This means that if only one node of a three
node cluster is running, that one node will not be able to do anything. The
--background flag of cockroach start causes the start
command to wait until the node has fully initialized and is able to start
serving queries.
Together, these two facts mean that the --background flag will cause
cockroach start to hang until a majority of nodes are running. In order to
restart your cluster, you should either use multiple terminals so that you can
start multiple nodes at once or start each node in the background using your
shell's functionality (e.g., cockroach start &) instead of the --background
flag.
Why is memory usage increasing despite lack of traffic?
Like most databases, CockroachDB caches the most recently accessed data in memory so that it can provide faster reads, and its periodic writes of timeseries data cause that cache size to increase until it hits its configured limit. For information about manually controlling the cache size, see Recommended Production Settings.
Why is disk usage increasing despite lack of writes?
The timeseries data used to power the graphs in the admin UI is stored within the cluster and accumulates for 30 days before it starts getting truncated. As a result, for the first 30 days or so of a cluster's life you will see a steady increase in disk usage and the number of ranges in the cluster even if you aren't writing data to it yourself.
As of the 1.0 release, there is no way to change the number of days before timeseries data gets truncated. As a workaround, however, you can start each node with the COCKROACH_METRICS_SAMPLE_INTERVAL environment variable set higher than its default of 10s to store fewer data points. For example, you could set it to 1m to only collect data every 1 minute, which would result in storing 6x less timeseries data than the default setting.
Why does CockroachDB collect anonymized cluster usage details by default?
Collecting information about CockroachDB's real world usage helps us prioritize the development of product features. We choose our default as "opt-in" to strengthen the information we receive from our collection efforts, but we also make a careful effort to send only anonymous, aggregate usage statistics. See Diagnostics Reporting for a detailed look at what information is sent and how to opt-out.
What happens when node clocks are not properly synchronized?
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. While serializable consistency is maintained regardless of clock skew, skew outside the configured clock offset bounds can result in violations of single-key linearizability between causally dependent transactions. It's therefore important to prevent clocks from drifting too far by running NTP or other clock synchronization software on each node.
The one rare case to note is when a node's clock suddenly jumps beyond the maximum offset before the node detects it. Although extremely unlikely, this could occur, for example, when running CockroachDB inside a VM and the VM hypervisor decides to migrate the VM to different hardware with a different time. In this case, there can be a small window of time between when the node's clock becomes unsynchronized and when the node spontaneously shuts down. During this window, it would be possible for a client to read stale data and write data derived from stale reads.
For guidance on synchronizing clocks, see the tutorial for your deployment environment:
| Environment | Featured Approach |
|---|---|
| On-Premises | Use NTP with Google's external NTP service. |
| AWS | Use the Amazon Time Sync Service. |
| Azure | Disable Hyper-V time synchronization and use NTP with Google's external NTP service. |
| Digital Ocean | Use NTP with Google's external NTP service. |
| GCE | Use NTP with Google's internal NTP service. |
How can I tell how well node clocks are synchronized?
As explained in more detail in our monitoring documentation, each CockroachDB node exports a wide variety of metrics at http://<host>:<http-port>/_status/vars in the format used by the popular Prometheus timeseries database. Two of these metrics export how close each node's clock is to the clock of all other nodes:
| Metric | Definition |
|---|---|
clock_offset_meannanos |
The mean difference between the node's clock and other nodes' clocks in nanoseconds |
clock_offset_stddevnanos |
The standard deviation of the difference between the node's clock and other nodes' clocks in nanoseconds |
As described in the above answer, a node will shut down if the mean offset of its clock from the other nodes' clocks exceeds 80% of the maximum offset allowed. It's recommended to monitor the clock_offset_meannanos metric and alert if it's approaching the 80% threshold of your cluster's configured max offset.
See Also
Was this helpful?
