Skip to content
shippy

Server configuration

This page lists every setting of shippyd, the server on the box, with its default. Use it when you write shippyd.toml, tune a box, or hand out tokens.

Every setting has three names that mean the same thing: a key in the config file, a SHIPPYD_* variable and a command-line flag. A wrong value is refused the same way wherever it came from, with an error that names where it was, and shippyd does not start.

Quick start

shippyd config print-default > /etc/shippy/shippyd.toml   # every key, its default, its variable
sudoedit /etc/shippy/shippyd.toml
shippyd config check                                       # what a start would find; starts nothing

Where settings come from

Loading diagram...
  1. The file. --config <file>, else SHIPPYD_CONFIG, else /etc/shippy/shippyd.toml when it exists. No file is fine if the required settings come from elsewhere.
  2. The environment goes over the file. The systemd unit also reads /etc/shippy/shippyd.env. An empty variable counts as unset.
  3. Flags go over both.

A key in the file that is not a setting is an error, so a typo is never silently ignored. A change to the file needs systemctl restart shippyd. Only the tokens file is followed live.

Required settings

shippyd will not start without these three. shippyd config print-default writes them with the values shown.

KeyVariableFlagValue in the default file
listenSHIPPYD_LISTEN--listen127.0.0.1:7447
store.pathSHIPPYD_STORE--store/var/lib/shippy/store
tokens.fileSHIPPYD_TOKENS--tokens/etc/shippy/tokens.toml
  • listen is an address and a port. Inside a tailnet, use the box's Tailscale address.
  • store.path is the store directory. The sandbox's scratch space is inside it, so its inputs are hard links: the filesystem must allow executables (no noexec). A tmpfs keeps the cache in memory.
  • tokens.file holds the hashes of accepted tokens. Keep it out of /usr and the toolchains' directories, which the sandbox can read.

All settings

Durations are written like 30s, 15m, 12h, 30d. Sizes like 200GB or 64GiB.

Top level and [store]

SettingDefaultWhat it does
listen (SHIPPYD_LISTEN, --listen)requiredAddress and port to serve on.
store.path (SHIPPYD_STORE, --store)requiredThe store directory.
store.max_size (SHIPPYD_MAX_STORE_SIZE, --max-store-size)autoHow large the store may get: a size, off, or auto (three quarters of the store's own size plus the free space of its filesystem).
store.max_age (SHIPPYD_MAX_AGE, --max-age)offAlso remove results nobody used for this long, and claims and packs this old.
store.evict_grace (SHIPPYD_EVICT_GRACE, --evict-grace)15mNever remove what was stored or used within this long. It does not protect what an untrusted principal sent when the store is over its limit.
store.principal_share (SHIPPYD_PRINCIPAL_SHARE, --principal-share)10Percent (1 to 100) of max_size one untrusted principal may hold in claims and blobs no public result names. A request past it gets 507.
store.evict_every (SHIPPYD_EVICT_EVERY, --evict-every)30sHow often the store is checked against its limits.

See eviction.

[exec]

A claim is a result an untrusted principal uploaded. The box re-runs its recipe (the full description of one rustc call) in a sandbox before anyone else gets it. These settings control those re-runs.

SettingDefaultWhat it does
exec.verify (SHIPPYD_VERIFY, --verify, --no-verify)truefalse keeps claims unverified: only their claimants see them.
exec.workers (SHIPPYD_VERIFY_WORKERS, --verify-workers)half the coresSandboxed re-runs of claims at once.
exec.queue (SHIPPYD_VERIFY_QUEUE, --verify-queue)16384Claims waiting for a re-run in memory, for everyone. One more waits on disk.
exec.queue_per_principal (SHIPPYD_VERIFY_QUEUE_PER_PRINCIPAL, --verify-queue-per-principal)4096The same, for one principal.
exec.timeout (SHIPPYD_COMPILE_TIMEOUT, --exec-timeout, --exec-timeout-secs)15mHow long one sandboxed compilation may take.
exec.local (SHIPPYD_EXEC_LOCAL, --exec-local)truefalse: the box runs no recipe itself, only its workers do.

[workers]

How the box treats its worker machines. See workers.

SettingDefaultWhat it does
workers.audit (SHIPPYD_WORKERS_AUDIT, --workers-audit)5%Percent (0 to 100) of workers' runs that are run a second time elsewhere and compared.
workers.audit_first (SHIPPYD_WORKERS_AUDIT_FIRST, --workers-audit-first)5How many of a worker's first runs are always run a second time.
workers.wait (SHIPPYD_WORKERS_WAIT, --workers-wait)30sHow long a run waits for a worker that could take it to appear.
workers.retries (SHIPPYD_WORKERS_RETRIES, --workers-retries)2How often a run goes to another worker after one lost it.
workers.heartbeat (SHIPPYD_WORKERS_HEARTBEAT, --workers-heartbeat)5sHow often a worker says a lease is alive. A lease is lost after three times this without a word.
workers.max_slots (SHIPPYD_WORKERS_MAX_SLOTS, --workers-max-slots)64The most runs one worker is leased at once, whatever it says it takes.

[worker]

Read by shippyd worker alone, on a worker machine. A worker also reads store.path (its cache of inputs and scratch space), exec.timeout, [toolchains], [log], [lifecycle] and [env].

SettingDefaultWhat it does
worker.store (SHIPPYD_WORKER_STORE, --worker-store)requiredThe box's address, such as http://box:7447.
worker.token_file (SHIPPYD_WORKER_TOKEN_FILE, --worker-token-file)requiredThe file that holds the worker's token. The worker refuses one in a directory every sandbox can read (/usr, a toolchain's directory, the entries of /etc a sandbox has).
worker.slots (SHIPPYD_WORKER_SLOTS, --worker-slots)half the coresRuns at once.
worker.cache_size (SHIPPYD_WORKER_CACHE_SIZE, --worker-cache-size)20GiBThe most its cache of inputs may hold.

[toolchains]

A claim is re-run with the exact compiler its recipe names. These say which toolchains rustup may install on the box.

SettingDefaultWhat it does
toolchains.install (SHIPPYD_INSTALL_TOOLCHAINS, --install-toolchains)[]Toolchains to install when anyone's recipe names one: a release (1.100.0), a dated channel (nightly-2026-11-10) or a channel (stable, beta, nightly). A list in the file, comma separated elsewhere. Without it only a trusted principal's recipe installs one. Through a whole channel, an untrusted principal's recipe has only a toolchain near the newest one of that channel here installed: a stable release no older than two before the newest, a nightly or a beta of the twelve weeks before the newest (before today, where none of that channel is here), no stable release at all where none is here, and none into the last quarter of the places: those are for a trusted principal's recipe and for a toolchain named here itself.
toolchains.max_installed (SHIPPYD_MAX_TOOLCHAINS, --max-toolchains)8Install nothing while this many toolchains are installed.

[tokens], [log], [metrics], [lifecycle]

SettingDefaultWhat it does
tokens.file (SHIPPYD_TOKENS, --tokens)requiredThe tokens file that shippyd token writes.
log.format (SHIPPYD_LOG_FORMAT, --log-format)texttext for people, json for one object per line.
log.level (SHIPPYD_LOG, --log-level)infotrace, debug, info, warn or error.
metrics.listen (SHIPPYD_METRICS_LISTEN, --metrics-listen)noneA second address serving /metrics, /healthz and /readyz without a token, such as 127.0.0.1:9447. Keep it on loopback.
lifecycle.shutdown_grace (SHIPPYD_SHUTDOWN_GRACE, --shutdown-grace)30sAfter SIGTERM, how long running requests and sandboxes get to finish.
lifecycle.selftest_every (SHIPPYD_SELFTEST_EVERY, --selftest-every)5mHow often the sandbox self-test behind /readyz runs again.

The [env] table

Some parts of the server read the environment themselves: the sandbox's limits, remote execution's quotas and build scripts. Put those variables in the file's [env] table. shippyd copies them into its own environment before anything starts, unless the real environment already has them. Only SHIPPYD_* names are allowed.

[env]
SHIPPYD_SANDBOX_MEMORY = "16GiB"
SHIPPYD_EXEC_SLOTS = 48

Sandbox limits

What one sandboxed compilation may use. A run past a limit is ended and its claim stays a claim. Memory and CPU limits need a cgroup per run (the systemd unit's Delegate=yes).

SettingDefaultWhat it does
SHIPPYD_SANDBOX_MEMORYa quarter of the machine's memory, 2 to 32 GiBMemory of the whole run, its /tmp included.
SHIPPYD_SANDBOX_CPUSno limitCPUs a run uses at once, such as 4 or 2.5.
SHIPPYD_SANDBOX_JOBSno cap but the box's coresThreads a run's compiler is lent through its jobserver. Never more than SHIPPYD_SANDBOX_CPUS rounded up.
SHIPPYD_SANDBOX_PIDS4096Processes and threads of a run.
SHIPPYD_SANDBOX_OPEN_FILES16384Open files per process.
SHIPPYD_SANDBOX_OUTPUT_BYTES8GiBWhat a run may leave in its output directories.
SHIPPYD_SANDBOX_OUTPUT_FILES20000How many files it may leave there.
SHIPPYD_SANDBOX_TMP_BYTES2GiBA run's /tmp, which is memory.
SHIPPYD_SANDBOX_STRICT_INPUTS11: only the inputs the recipe declares. 0 also adds undeclared dependency metadata that public results lead to. Leave it on.
SHIPPYD_SANDBOX_SECCOMP10 runs the sandbox without its system call filter.
SHIPPYD_CGROUPthe parent of shippyd's own cgroupThe delegated cgroup each run's group is created in.
SHIPPYD_IMAGE_IDcomputed from the system directoriesA name for the system image verified results are sealed with, so boxes provisioned alike agree.

Remote execution

Limits of /v1/exec, the route that compiles a client's cache misses (remote execution). Sizes are bytes.

SettingDefaultWhat it does
SHIPPYD_EXEC_SLOTSone per coreCompilations at once, for everyone.
SHIPPYD_EXEC_PRINCIPAL_SLOTSall the slotsCompilations at once for one principal.
SHIPPYD_EXEC_PRINCIPAL_QUEUE256Requests one principal may have waiting. One more gets 429.
SHIPPYD_EXEC_QUEUE4096Requests waiting, for everyone.
SHIPPYD_EXEC_PRINCIPAL_STREAMS256Answers one principal may have open as streams beyond its requests that run or wait. One more gets 429.
SHIPPYD_EXEC_RETRY_AFTER5Seconds a refused client is told to stay away.
SHIPPYD_EXEC_MAX_RECIPE_BYTES16777216The longest recipe.
SHIPPYD_EXEC_MAX_INLINE_BYTES67108864Inputs one request may bring in its own body.
SHIPPYD_EXEC_MAX_INPUTS100000Files a recipe may declare.
SHIPPYD_EXEC_MAX_INPUT_BYTES34359738368Their size together.
SHIPPYD_EXEC_MAX_LISTED_BYTES4294967296Packed directories one request may make the box read to find its inputs. Past it: 413.
SHIPPYD_EXEC_MAX_LISTED_ENTRIES2097152The entries of those directories.
SHIPPYD_NARROW10 compiles a request that declares a whole package with all of its files, without first finding what the compiler reads, and once more when the request set variables the crate does not read. With 0 every file a package was declared with is listed for everyone who builds the crate, read or not: leave it on where principals must not learn of each other's checkouts.

Build scripts

The box runs build scripts in its sandbox (build scripts). A box with exec.local = false runs none.

SettingDefaultWhat it does
SHIPPYD_SCRIPTS10 runs no build scripts; clients run their own.
SHIPPYD_SCRIPT_TIMEOUT1800Seconds one script may run.
SHIPPYD_SCRIPT_JOBShalf of shippyd's CPUs, 2 to 32NUM_JOBS for a script, and its CPUs with a cgroup per run.
SHIPPYD_SCRIPT_OUTPUT_BYTES8GiBWhat one script may leave in OUT_DIR.
SHIPPYD_SCRIPT_OUTPUT_FILES200000How many files and directories.

Connections

SettingDefaultWhat it does
SHIPPYD_MAX_CONNECTIONS8192Connections the server holds at once. One more waits to be accepted.
SHIPPYD_MAX_CONNECTIONS_PER_ADDRESShalf of the aboveConnections at once from one address (an IPv4 address, an IPv6 /64). One more is closed as it is accepted. Behind a proxy every client has the proxy's address: set it to SHIPPYD_MAX_CONNECTIONS.

A request's body that stands still for a minute, or comes slower than four kilobytes a second after its first minute of waiting, is given up and its request answered 400.

Token roles

A principal is one named identity (a person, a CI pool, a scraper) with one role. It may have several tokens. Results and claims are kept per principal, so use one principal per person or machine pool.

shippyd token add alice                       # untrusted, the default
shippyd token add ci-main --role trusted
shippyd token add prometheus --role metrics
shippyd token add worker-1 --role worker
RoleWhat it may do
untrustedEverything a developer or an agent needs. Its uploads are claims, visible to itself only, until the box has compiled them in its sandbox. The default.
trustedIts uploads are public as they are. Give it only to CI that builds protected branches on machines you control.
metricsGET /metrics and nothing else.
workerA worker machine of the shared pool: it leases sandbox runs and hands back their results, and reaches nothing else, /metrics included.
trusted-workerThe same, for the trusted pool, which runs only what trusted principals ask for.
Loading diagram...

--require-trusted-pool on a trusted token has it served only what a trusted principal uploaded or the trusted pool produced.

The tokens file

[[token]]
id = "3f9a1c2e"                 # the first hex digits of the hash; token revoke takes it
principal = "alice"
role = "untrusted"              # trusted, untrusted, metrics, worker or trusted-worker
sha256 = "3f9a1c2e..."
created = "2026-11-13T20:00:00Z"
expires = "2027-02-11T20:00:00Z"  # optional

The file holds only the SHA-256 of each token, so a copy of it lets nobody in. Tokens start with shp_. A running shippyd looks at the file on every request: a new token works and a revoked one is refused from the next request. SIGHUP (systemctl reload shippyd) reads it again anyway. A file that cannot be read or parsed lets nobody in until it is fixed.

Go deeper