This page shows you how to reproduce CockroachDB TPC-C performance benchmarking results. Across all scales, CockroachDB can process tpmC (new order transactions per minute) at near maximum efficiency. Start by choosing the scale you're interested in:
Workload | Cluster size | Warehouses | Data size |
---|---|---|---|
Local | 3 nodes on your laptop | 10 | 2 GB |
Local (multi-region) | 9 in-memory nodes on your laptop using cockroach demo |
10 | 2 GB |
Small | 3 nodes on c5d.4xlarge machines |
2500 | 200 GB |
Medium | 15 nodes on c5d.4xlarge machines |
13,000 | 1.04 TB |
Large | 81 nodes on c5d.9xlarge machines |
140,000 | 11.2 TB |
Before you begin
Review TPC-C concepts
TPC-C provides the most realistic and objective measure for OLTP performance at various scale factors. Before you get started, consider reviewing what TPC-C is and how it is measured.
Request a trial license
Reproducing these TPC-C results involves using CockroachDB's partitioning feature to ensure replicas for any given section of data are located on the same nodes that will be queried by the load generator for that section of data. Partitioning helps distribute the workload evenly across the cluster.
The partitioning feature requires an Enterprise license, so request a 30-day trial license before you get started.
You should receive your trial license via email within a few minutes. You'll enable your license once your cluster is up-and-running.
Step 1. Set up the environment
Provision VMs
Create 16 VM instances, 15 for CockroachDB nodes and 1 for the TPC-C workload.
- Create all instances in the same region and the same security group.
- Use the
c5d.4xlarge
machine type. - Use local SSD instance store volumes. Local SSDs are low latency disks attached to each VM, which maximizes performance. This configuration best resembles what a bare metal deployment would look like, with machines directly connected to one physical disk each. We do not recommend using network-attached block storage.
Note the internal IP address of each instance. You'll need these addresses when starting the CockroachDB nodes.
This configuration is intended for performance benchmarking only. For production deployments, there are other important considerations, such as security, load balancing, and data location techniques to minimize network latency. For more details, see the Production Checklist.
Configure your network
CockroachDB requires TCP communication on two ports:
26257
for inter-node communication (i.e., working as a cluster) and for the TPC-C workload to connect to nodes8080
for exposing your DB Console
Create inbound rules for your security group:
Inter-node and TPCC-to-node communication
Field | Recommended Value |
---|---|
Type | Custom TCP Rule |
Protocol | TCP |
Port Range | 26257 |
Source | The name of your security group (e.g., sg-07ab277a) |
DB Console
Field | Recommended Value |
---|---|
Type | Custom TCP Rule |
Protocol | TCP |
Port Range | 8080 |
Source | Your network's IP ranges |
Step 2. Start CockroachDB
The --insecure
flag used in this tutorial is intended for non-production testing only. To run CockroachDB in production, use a secure cluster instead.
SSH to the first VM where you want to run a CockroachDB node.
Download the CockroachDB archive for Linux, extract the binary, and copy it into the
PATH
:$ curl https://binaries.cockroachdb.com/cockroach-v24.1.3.linux-amd64.tgz \ | tar -xz
$ cp -i cockroach-v24.1.3.linux-amd64/cockroach /usr/local/bin/
If you get a permissions error, prefix the command with
sudo
.Run the
cockroach start
command:$ cockroach start \ --insecure \ --advertise-addr=<node1 internal address> \ --join=<node1 internal address>,<node2 internal address>,<node3 internal address> \ --cache=.25 \ --locality=rack=0
Each node will start with a locality that includes an artificial "rack number" (e.g.,
--locality=rack=0
). Use 5 racks for 15 nodes so that 3 nodes will be assigned to each rack.Repeat steps 1 - 3 for the other 14 VMs for CockroachDB nodes. Each time, be sure to:
- Adjust the
--advertise-addr
flag. - Set the
--locality
flag to the appropriate "rack number".
- Adjust the
On any of the VMs with the
cockroach
binary, run the one-timecockroach init
command to join the first nodes into a cluster:$ cockroach init --insecure --host=<address of any node on --join list>
Step 3. Configure the cluster
You'll be importing a large TPC-C data set. To speed that up, you can tweak some cluster settings. You'll also need to enable the Enterprise license you requested earlier.
SSH to any VM with the
cockroach
binary.Launch the built-in SQL shell:
$ cockroach sql --insecure --host=<address of any node>
Adjust some cluster settings:
> SET CLUSTER SETTING rocksdb.ingest_backpressure.l0_file_count_threshold = 100; SET CLUSTER SETTING rocksdb.ingest_backpressure.pending_compaction_threshold = '5 GiB'; SET CLUSTER SETTING schemachanger.backfiller.max_buffer_size = '5 GiB'; SET CLUSTER SETTING kv.snapshot_rebalance.max_rate = '128 MiB'; SET CLUSTER SETTING rocksdb.min_wal_sync_interval = '500us'; SET CLUSTER SETTING kv.range_merge.queue_enabled = false;
Enable the trial license you requested earlier:
> SET CLUSTER SETTING cluster.organization = '<your organization>';
> SET CLUSTER SETTING enterprise.license = '<your license key>';
Exit the SQL shell:
> \q
Step 4. Import the TPC-C dataset
CockroachDB comes with a number of built-in workloads for simulating client traffic. This step features CockroachDB's version of the TPC-C workload.
SSH to the VM where you want to run TPC-C.
Download the CockroachDB archive for Linux, extract the binary, and copy it into the
PATH
:$ curl https://binaries.cockroachdb.com/cockroach-v24.1.3.linux-amd64.tgz \ | tar -xz
$ cp -i cockroach-v24.1.3.linux-amd64/cockroach /usr/local/bin/
If you get a permissions error, prefix the command with
sudo
.Import the TPC-C dataset:
$ cockroach workload fixtures import tpcc \ --partitions=5 \ --warehouses=13000 \ 'postgres://root@<address of any CockroachDB node>:26257?sslmode=disable'
This will load 1.04 TB of data for 13,000 "warehouses". This can take around 1 hour to complete.
You can monitor progress on the Jobs screen of the DB Console. Open the DB Console by pointing a browser to the address in the
admin
field in the standard output of any node on startup.
Step 5. Allow the cluster to rebalance
Next, partition your database to divide all of the TPC-C tables and indexes into 5 partitions, one per rack, and then use zone configurations to pin those partitions to a particular rack.
Still on the same VM, briefly run TPC-C to let the cluster balance and the leases settle. Bump the file descriptor limits with
ulimit
to the high value shown in the following snippet, since the workload generators create a lot of database connections.$ ulimit -n 100000 && cockroach workload run tpcc \ --partitions=5 \ --warehouses=13000 \ --duration=1m \ --ramp=1ms \ 'postgres://root@<address of any CockroachDB node>:26257?sslmode=disable'
Wait for range rebalancing to finish.
This will likely take 10s of minutes. To watch the progress, go to the Metrics > Queues > Replication Queue graph in the DB Console. Once the Replication Queue gets to
0
for all actions and stays there, you can move on to the next step.
Step 6. Run the benchmark
Back on the VM with the
workload
binary, create anaddrs
file containing connection strings to all 15 CockroachDB nodes:postgres://root@<node 1 internal address>:26257?sslmode=disable postgres://root@<node 2 internal address>:26257?sslmode=disable postgres://root@<node 3 internal address>:26257?sslmode=disable postgres://root@<node 4 internal address>:26257?sslmode=disable ...
Run TPC-C for 30 minutes:
$ ulimit -n 100000 && cockroach workload run tpcc \ --partitions=5 \ --warehouses=13000 \ --ramp=1m \ --duration=30m \ $(cat addrs)
Step 7. Interpret the results
Once the workload has finished running, you will see a result similar to the following. The efficiency and latency can be combined to determine whether this was a passing run. You should expect to see an efficiency number above 95%, well above the required minimum of 85%, and p95 latencies well below the required maximum of 10 seconds.
_elapsed_______tpmC____efc__avg(ms)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms)
1800.0s 160420.0 96.0% 303.1 285.2 570.4 671.1 939.5 4160.7
See also
Hardware
CockroachDB works well on commodity hardware in public cloud, private cloud, on-prem, and hybrid environments. For hardware recommendations, see our Production Checklist.
Cockroach Labs creates a yearly cloud report focused on evaluating hardware performance. For more information, see the 2022 Cloud Report.
Performance Tuning
For guidance on tuning a real workload's performance, see SQL Best Practices, and for guidance on techniques to minimize network latency in multi-region or global clusters, see Multi-Region Capabilities Overview.