Run a box

This is the handbook for the person who runs a production shippyd box. It covers installing one from zero, giving people access, watching it, fixing the common problems and upgrading it. If you only want to try a shared cache, start with Share a cache.

What runs where

Install from zero

SHIPPY_TOOLCHAINS="1.100.0 1.99.0" SHIPPY_STORE_DEVICE=/dev/nvme1n1 \
  bash deploy/bootstrap.sh --dry-run

sudo SHIPPY_TOOLCHAINS="1.100.0 1.99.0" \
     SHIPPY_STORE_DEVICE=/dev/nvme1n1 \
     TS_AUTHKEY=tskey-auth-... \
     bash deploy/bootstrap.sh

curl 127.0.0.1:9447/readyz

Where the store lives

Keep the store on disk (the NVMe) at /var/lib/shippy/store, not on tmpfs:

• A reboot, kernel update or crash empties a tmpfs, and then everything compiles again for everyone.
• Reads come from memory either way: the kernel's page cache holds what is read often.
• A full tmpfs fails uploads and sandbox runs at once. A full disk degrades into eviction.
• shippyd syncs nothing, so writes cost the same as on tmpfs.

The filesystem must not be mounted noexec, and one filesystem must hold the whole store (sandbox inputs are hard links). ext4 as bootstrap makes it is fine.

Network

Inside the tailnet the box speaks plain HTTP; WireGuard encrypts the path. ufw lets port 7447 in from tailscale0 only. Every request carries a token, so bootstrap.sh opens the port to the networks of SHIPPY_ALLOW_CIDRS only when SHIPPY_ALLOW_PLAIN_HTTP=1 says they may carry tokens unencrypted. If the port must be reachable from outside the tailnet, put Caddy in front with deploy/caddy/Caddyfile: clients require a public certificate for an https remote. Behind a proxy every client has the proxy's address, so set SHIPPYD_MAX_CONNECTIONS_PER_ADDRESS to SHIPPYD_MAX_CONNECTIONS (server configuration).

Configuration basics

shippyd config print-default   # every key, its default and its variable
sudo shippyd config check      # read the file as a start would, start nothing
sudo systemctl restart shippyd # apply a change to shippyd.toml

• One file: /etc/shippy/shippyd.toml. Then the environment (SHIPPYD_*, also from /etc/shippy/shippyd.env) overrides it, then command-line flags.
• A wrong value is an error naming the key and where it came from, and shippyd does not start.
• Sandbox limits and remote-execution quotas are SHIPPYD_* variables. Put them in the file's [env] table.
• A change to the file needs a restart. Only the tokens file is followed live.

The settings you are most likely to touch:

Every key is in Server config.

Tokens

sudo shippyd token add alice                      # a person: untrusted
sudo shippyd token add alice-laptop2 --expires 90d
sudo shippyd token add ci-main --role trusted     # CI on protected branches only
sudo shippyd token add prometheus --role metrics  # reads /metrics, nothing else
sudo shippyd token list
sudo shippyd token revoke alice                   # every token of the principal
sudo shippyd token revoke 3f9a1c2e                # one token, by its id

The token is printed once, alone on standard output, so it can be piped into a secret store. Changes apply with the next request; no restart. A tokens file that cannot be parsed lets nobody in (alert ShippydTokensFileBroken). See Share a cache for what each role means.

Offboard someone with shippyd token revoke <principal>, and shippyd admin purge --principal <principal> if their claims should go too.

The warmer

shippy-warm builds every new commit of the branches it watches, through shippy, so the box has a commit's crates before people pull it.

sudo install -m 0640 -o root -g shippy-warm deploy/shippy-warm.toml /etc/shippy-warm/shippy-warm.toml
sudo shippyd token add warmer | sudo install -m 0640 -o root -g shippy-warm /dev/stdin /etc/shippy-warm/token
sudoedit /etc/shippy-warm/shippy-warm.toml
sudo -u shippy-warm shippy-warm config check --config /etc/shippy-warm/shippy-warm.toml
sudo systemctl enable --now shippy-warm

On the box, SHIPPY_WARM=1 bash deploy/bootstrap.sh creates its user, toolchains and unit; it can also run on a machine of its own.

• Put in commands what people and CI actually run, with the same environment (for example SOURCE_DATE_EPOCH). A build warms another only when rustc is invoked the same way.
• The code of a build can read the token it is built with. Give a repository whose code is not to act as the others' on the box a token of its own (token_file in its [[repo]]). Each repository has its own local store either way.
• It checks each branch every poll (60 s) and when POST /trigger says so. One build runs at a time, in bubblewrap, from a fresh checkout.
• A failed commit is retried after retry_min, doubling up to retry_max.

curl -s 127.0.0.1:9448/status | jq '.targets | map_values({built, pending, failures, tip})'
sudo less /var/lib/shippy-warm/logs/tempo.main.log

Watch it

• Metrics. GET /metrics on port 7447 needs a token of any role but a worker's; give the scraper its own --role metrics token. 127.0.0.1:9447 serves the same page without a token. deploy/prometheus/scrape.yml has both, alerts.yml the rules, and deploy/grafana/shippy.json a dashboard.
• Probes. /healthz means the process answers. /readyz means it can do its work: the store's directories exist, the tokens file parses, and the last sandbox self-test passed. The self-test compiles a crate at start and every five minutes.
• Logs. stderr, to the journal. With log.format = "json" every request is one JSON line. No token or file content is ever logged, and an event is one line: line breaks and terminal escape sequences in what a principal wrote (an output's name, a compiler's message) are written out, in the log and in what shippyd admin prints.

journalctl -u shippyd -o cat | jq 'select(.level != "info")'
journalctl -u shippyd -o cat | jq 'select(.msg == "request" and .status >= 500)'

Key metrics:

Alerts

Worker alerts are on the Workers page.

Troubleshooting

Install a toolchain for the service:

sudo -u shippy env RUSTUP_HOME=/var/lib/shippy/rustup CARGO_HOME=/var/lib/shippy/cargo \
  /var/lib/shippy/cargo/bin/rustup toolchain install 1.100.0 --profile minimal

Check what the sandbox's C toolchain sees:

cd "$(mktemp -d)" && echo 'int main(void){return 0;}' > t.c
env -i PATH=/usr/local/bin:/usr/bin:/bin cc -fuse-ld=lld t.c -o t && ./t

Upgrade and roll back

tar -xzf shippy-<tag>-x86_64-unknown-linux-gnu.tar.gz
sudo bash shippy-<tag>-x86_64-unknown-linux-gnu/deploy/upgrade.sh
sudo bash deploy/upgrade.sh --rollback    # back to the release before
bash deploy/upgrade.sh --dry-run          # what it would do

upgrade.sh checks the configuration with the new shippyd first, installs the binaries beside the old ones, restarts and waits for /readyz. If the new release does not get ready, it puts the old one back.

• Builds go on during the restart: a wrapper that cannot reach the box compiles locally and uploads later. No client needs restarting.
• Upgrade the box first, then the clients. A release that changes RECIPE_VERSION makes old results unreachable for new clients; expect a cold cache.
• After a good upgrade, it keeps the running release, the one before and the three newest others (SHIPPY_KEEP_RELEASES=<n> to change).

A suspected poisoned result

A client got something it should not have: a wrong .rlib, a binary that misbehaves, a mismatch alert that looks deliberate.

sudo -u shippy shippyd admin who --key <action-key>

sudo shippyd token revoke <principal>

sudo -u shippy shippyd admin purge --key <action-key> --dry-run
sudo -u shippy shippyd admin purge --key <action-key>

An untrusted principal cannot make its upload anyone else's result. A poisoned public result points at a trusted token, the box itself, or a bug; the first is by far the likeliest.

Capacity

• Store. One whole dev build's results: tempo 8.7 GB, reth 4.3 GB. Weekly growth is the growth of shippyd_store_bytes plus increase(shippyd_evicted_bytes_total[7d]). Size the disk so eviction removes less than the store per week.
• Claims. A waiting claim keeps its outputs and inputs, so untrusted uploads cost about twice their size until verified.
• Sandbox runs. exec.workers defaults to half the cores. A shippyd_verify_queue that drains between pushes is enough; one that grows all day is not.
• Memory. Sandbox compilers (a large crate's rustc takes several GB) plus the page cache that serves the store.
• Network. A cold build downloads its whole target directory.
• The warmer on the same machine takes cores from the sandbox. Move it to its own machine when ShippydVerifyQueueDeep fires during its builds.

Backups

Do not back up the store. Every blob and result can be built again; a box that lost its store is slow for a day, not broken.

Back up these, where you keep other infrastructure config:

• /etc/shippy/tokens.toml: losing it locks everyone out until tokens are handed out again. It holds hashes, so a copy lets nobody in.
• /etc/shippy/shippyd.toml
• /etc/shippy-warm/

Go deeper

• docs/operations.md: sandbox limits, cgroups, threads, claim verdicts, what survives what, the unit's hardening
• deploy/bootstrap.sh
• deploy/shippyd.toml
• docs/threat-model.md