Workers

A worker is a separate machine that runs the box's sandbox jobs for it. Read this when one box no longer has enough cores, or when you want the box itself to run no untrusted code at all.

The box still holds the cache, the tokens and the decisions. Workers only run recipes. With exec.local = false the box runs nothing itself: no procedural macro, recipe source or compiler a client chose ever runs on the machine that holds the cache.

What a worker is

A worker is the same program, started as shippyd worker. It has:

• the box's sandbox (bubblewrap, the system call filter, the limits, a cgroup per run);
• the toolchains;
• a cache of inputs, but no store of results;
• one credential: a token with the role worker or trusted-worker.

It listens on nothing. It only makes outgoing requests to the box.

The loop

A lease is one recipe handed to one worker, with the list of input blobs it needs and how often to say it is still alive.

1. Poll. The worker asks for work. The request is held open until a run comes up. It says what the worker is: its system image, linker, toolchains and how many runs it takes at once.
2. Lease. The box hands out a recipe only if the principal it is for may read every input.
3. Fetch. The worker downloads inputs it lacks, each checked against its digest.
4. Run. It runs the recipe through the same executor code the box uses: same checks, same sandbox with only the declared inputs.
5. Heartbeat. It sends the crate's metadata as soon as rustc writes it, so a waiting client can start dependent crates, and says every few seconds that the lease is alive.
6. Complete. It hands back the outputs and rustc's messages, or why the run failed.

The box treats a completion like anything from outside. It checks that the lease is that worker's and alive, that the outputs are exactly files rustc writes for this crate, and it hashes every byte itself. The key, trust and producer name on the result are the box's own. Which worker ran each result goes to <store>/workers/runs.log, and shippyd admin who --key shows it.

Trust: pools and audits

A worker whose sandbox was escaped could hand back wrong outputs. Hashing does not help, because the box hashes what it is given. Three things limit the damage.

Pools. Each worker is in the shared pool (--role worker) or the trusted pool (--role trusted-worker).

So code from someone without a trusted token never reaches a trusted worker. Use the trusted pool, with --require-trusted-pool, for release builds:

sudo shippyd token add release-ci --role trusted --require-trusted-pool

Audits. A share of completed runs (workers.audit, 5%) and the first five runs of every worker (workers.audit_first) are run a second time before anything is published, by another executor. A secret of the box picks which runs, after they are handed back, so a worker cannot tell an audit from a normal lease. Two runs of one recipe in the same sandbox write the same bytes: the two results may differ only in the name rustc makes up for an invocation with -C incremental, which is in the names of a library's objects.

Three different answers mean the recipe does not reproduce: nobody is suspended. A worker whose result nobody else gave is asked for the run once more, and suspended only when it gives the same again: a recipe whose runs differ (a proc macro that reads the clock) looks like a lie the first time and not the second. A recipe can also tell one machine from the next, so the recipes of one untrusted principal have one worker suspended at a time, and never the last that runs a pool's recipes on a box that runs none itself. Past that an audit suspends nobody, quarantines the key if the disagreement is not settled, and counts it (shippyd_worker_suspensions_withheld_total).

A suspended worker is leased nothing and loses the leases it holds, and a quarantined key is run by nobody, until an operator clears them. A fleet of one worker is not audited.

No state. A worker keeps blobs by digest and nothing else. Each run gets a scratch directory, removed when it ends. Reimage workers on a schedule, and after every suspension.

sudo -u shippy shippyd admin quarantine   # the keys, who ran them, what differs
sudo -u shippy shippyd admin workers      # the fleet, and who is suspended

sudo -u shippy shippyd admin workers --clear <worker>
sudo -u shippy shippyd admin quarantine --clear <action-key>

Add a worker

sudo SHIPPY_ROLE=worker SHIPPY_BOX=http://box:7447 SHIPPY_IMAGE_ID=ubuntu-24.04-2026-11 \
     SHIPPY_TOOLCHAINS="1.100.0 1.99.0" TS_AUTHKEY=tskey-auth-... \
     bash deploy/bootstrap.sh

sudo shippyd token add worker-7 --role worker   # or --role trusted-worker

sudo install -m 0640 -o root -g shippy /dev/stdin /etc/shippy/worker-token
sudo systemctl start shippyd-worker
journalctl -u shippyd-worker -f   # "the sandbox works: asking the box for work"

Put the same toolchains on every worker of a pool: a recipe waits for a worker that has its compiler.

Stop running recipes on the box, once workers are there: set local = false under [exec] in /etc/shippy/shippyd.toml and restart. The box then needs no sandbox of its own. With no usable worker, every /v1/exec request is refused at once with 429 and clients compile for themselves.

Take a worker out by stopping its unit. Runs in flight get lifecycle.shutdown_grace to finish; the rest go to other workers. Upgrade by running bootstrap.sh again from the new release (with SHIPPY_ROLE=worker), or by reimaging. Upgrade the box first and its workers right after: the two speak one release's protocol.

When things fail

Settings

On the box (shippyd.toml):

On a worker (shippyd-worker.toml):

Sizing

• Slots. Half the cores (the default) keeps a worker responsive. A fleet of mostly small crates can go to one per core; then set SHIPPYD_SANDBOX_CPUS.
• Memory. Slots times SHIPPYD_SANDBOX_MEMORY should fit the machine, with room for the page cache.
• Cache. It only saves fetching, so any size works. A few times one target's build outputs avoids fetching the same libraries again.
• Network. Every missing input and every output crosses the link once. Keep workers beside the box.
• How many. The box's /v1/exec slots are the workers' slots added up, each worker counted for at most workers.max_slots (64). Claims are re-run by the box's exec.workers threads, each holding one lease at a time: raise it as the fleet grows.

Watching the fleet

shippyd admin workers prints the fleet as the box last saw it: who is there, pool, slots, runs, last contact, who is suspended or on another image. GET /v1/stats has the same under workers.

A worker has no port and no /metrics of its own; its log is its journal, one line per lease.

Go deeper

• docs/workers.md, including what each compromise gets and what is not built
• docs/threat-model.md
• deploy/shippyd-worker.toml