If you need a location to store files for the IMPORT
process or CockroachDB enterprise backups, but do not have access to (or simply cannot use) cloud storage providers, you can easily create your own file server. You can then use this file server by leveraging support for our HTTP Export Storage API.
This is especially useful for:
- Implementing a compatibility layer in front of custom or proprietary storage providers for which CockroachDB does not yet have built-in support
- Using on-premises storage
HTTP Export Storage API
CockroachDB tasks that require reading or writing external files (such as IMPORT
and BACKUP
) can use the HTTP Export Storage API by prefacing the address with http
, e.g., http://fileserver/mnt/cockroach-exports
.
This API uses the GET
, PUT
and DELETE
methods. This behaves like you would expect typical HTTP requests to work. After a PUT
request to some path, a subsequent GET
request should return the content sent in the PUT
request body, at least until a DELETE
request is received for that path.
Examples
You can use any file server software that supports GET
, PUT
and DELETE
methods, but we've included code samples for common ones:
cockroach
as file servers. Using machines that are running cockroach as file servers could negatively impact performance if I/O operations exceed capacity.Using Caddy as a File Server
Download a
caddy
binary that includes thehttp.upload
plugin.Run
caddy
with anupload
directive, either in the command line or viaCaddyfile
.Command line example (with no TLS):
caddy -root /mnt/cockroach-exports "upload / {" 'to "/mnt/cockroach-exports"' 'yes_without_tls' "}"
Caddyfile
example (using a key and cert):tls key cert root "/mnt/cockroach-exports" upload / { to "/mnt/cockroach-exports" }
Using nginx as a File Server
Install
nginx
with thewebdav
module (often included in-full
or similarly named packages in various distributions).In the
nginx.conf
file, add adav_methods PUT DELETE
directive. For example:events { worker_connections 1024; } http { server { listen 20150; location / { dav_methods PUT DELETE; root /mnt/cockroach-exports; sendfile on; sendfile_max_chunk 1m; } } }