Skip to content
shippy

Remote execution

With remote execution on, a crate the cache misses is compiled by the box, in its sandbox, unless it is a dependency your machine has a core for or a crate you are rebuilding. Use it when your machine has few cores, or (with always) when you want what you build to be in the cache for everyone at once.

SHIPPY_EXEC=auto shippy build

Or set it for a project in shippy.toml:

[remote]
url = "http://box:7447"
exec = true

The wrapper sends the crate's recipe (the exact rustc invocation and the digest of every input) to the box. The box runs it in its sandbox, publishes the result, and streams the outputs back. Your machine restores them as it restores a cache hit. A machine with five cores then builds with its own cores and the box's.

Nothing about trust changes. A result is public because the box's own sandbox made it from the inputs its key names, exactly as when the box re-runs an uploaded claim (an untrusted upload, see Trust).

Pick a mode

SettingWhat the box compiles
SHIPPY_EXEC=auto (or 1), or exec = trueThe misses the cache covers (libraries, proc macros and, unless SHIPPY_LINKED=0, executables and C-compatible libraries), with two exceptions: a dependency compiles here while your machine runs fewer compilers than it has cores, and a workspace crate being rebuilt in this target directory always compiles here.
SHIPPY_EXEC=always, or exec = "always"Every such miss, whatever your machine could do itself.
SHIPPY_EXEC=deps, or exec = "deps"Only crates from a registry or a git repository.
unset, 0, false, offNothing: misses compile here, as with the cache alone.

How a miss goes

Loading diagram...

In plain steps:

  1. The wrapper builds the recipe and looks in the local store.
  2. On a miss it sends POST /v1/exec with the recipe and the inputs the box most likely lacks, up to 4 MiB of them.
  3. If the box already has a public result for that key, it answers with it at once. That counts as a hit.
  4. Otherwise the request waits for a slot and runs in the sandbox. The answer is a stream: a heartbeat every five seconds, the crate's metadata as soon as rustc writes it, then the result and its output blobs.
  5. The wrapper puts the metadata in place as soon as it arrives, so cargo can start the crates that wait for it. At the end it restores the other outputs and replays rustc's messages.

The common miss is one request. After an edit, a crate is one request too: the wrapper sends the edited file along.

Which files a request declares

The sandbox holds exactly the files the recipe declares, but what rustc reads is known only after it runs. So the wrapper declares, in order of preference:

  • The files the crate read last time, with their content as it is now. The wrapper remembers them after any compilation, here or on the box for this machine's request. A manifest someone else listed on the box is never where they come from.
  • The whole package, when nothing is remembered, as on a machine that has never compiled the crate: every file of the package directory, whatever is below it, without .git and target directories. The box then runs a quick pass (--emit=dep-info) to find what rustc reads, and compiles with exactly that. The result is published under the narrower key, so other checkouts with a different README still hit.
  • The package without the packages below it, when the whole package is past SHIPPY_EXEC_MAX_BYTES or SHIPPY_EXEC_MAX_FILES, as the root package of a large workspace is: its directory is the whole checkout. Directories with a Cargo.toml of their own and cache directories (CACHEDIR.TAG) are left out, but only when the crate loads no procedural macro. rustc reads files by name, so a file it needs and does not find fails the compilation with a message that names it, and nothing left out can change the result. A procedural macro can list a directory and embed whatever is there, so a crate that loads one is sent with its whole package or compiled here. Files below the package that your machine already knows the crate reads are declared anyway.
  • What the compiler asks for: when rustc names a missing file or variable, the wrapper adds it and asks once more.

The cost of that rule: on a machine that has never compiled it, such as a CI runner, the root crate of a workspace past the limits compiles on that machine when it loads a procedural macro. The session log says here package-too-large. A package that fits the limits is always sent whole.

A file outside the crate's own directories is never declared. A crate that reads one is compiled here. Variables are declared with your machine's values, so only the ones your machine knows the crate reads are: never one a manifest someone else listed names, and never a SHIPPY_* variable.

Whatever a request declares, the box publishes its run only under a manifest of what the compiler read: the files and variables its dep-info names (and the package's other files only for a crate that does not compile without them, which the box finds out from the compiler). A request that names files of its package the compiler did not read, for a crate that compiles without them, is answered and not published: that run is its requester's alone. A request that sets a variable the crate does not read, or names a file outside the package that it does not read, is refused after the run. The wrapper then asks again with the whole package and says that it declares one, and the crate compiles here only when that is refused too. A request that says it declares a whole package and only set such a variable is not refused: the box compiles it once more without the variable and answers with that run.

Here or on the box

The box's sandbox starts empty, so it always compiles a crate from nothing. Your machine keeps rustc's incremental state for workspace crates, and an incremental rebuild after an edit beats the box however many cores it has. With auto, each miss goes where it is likely to finish first.

Loading diagram...

A core to spare. A dependency, or any crate built without incremental compilation, still compiles here when your machine runs fewer compilers than it has cores (SHIPPY_LOCAL_JOBS). A compiler's front end is one thread wherever it runs, so the crate is done here no later than on the box, and sooner by what the box adds to each compilation: the request, a pass that finds what the crate reads, the outputs sent back. The box compiles what your machine has no core for, and a chain of dependencies that compile one at a time runs at your machine's own speed.

What you compile here is uploaded as a claim, where the box's own run would have been public at once. A dependency's claim is byte for byte what the box writes, so it becomes everyone's once the box has re-run it. A workspace crate compiled here incrementally would never match, and it and everything built on it would stay yours. So a workspace crate your machine has not built yet goes to the box even when a core is free: the next machine to build the same commit then restores the whole build, and your machine makes the crate's incremental state after the build. With SHIPPY_LOCAL_JOBS=0 nothing limits your machine's compilers, nothing says whether a core is free, and these misses go to the box.

One exception: if the box was clearly faster at this crate (it took at most half as long as your machine did, both from nothing), the first two rebuilds in a target directory still go to the box, and then they stay here.

A crate that was restored or compiled by the box has no state here when the build ends, so shippy compiles it once more in the background for its state (What is cached). After that an edit is rebuilt here, incrementally. Where that is turned off or has not reached a crate yet, its first rebuild runs here from nothing, and every rebuild after that is incremental. Each line of the decision is written to the session log (~/.cache/shippy/sessions/) as route <crate> ... <remote|here> <why>.

Switches

On the machine that builds:

SettingDefaultWhat it does
SHIPPY_EXEC, or exec under [remote]offauto, always or deps (above).
SHIPPY_EXEC_JOBS, or exec_jobseight per core, at most 64How many crates cargo works on at once. -j, CARGO_BUILD_JOBS and build.jobs win.
SHIPPY_LOCAL_JOBSone per coreHow many compilers run on this machine at once, and how many jobs a local compiler or build script is told to start.
SHIPPY_EXEC_TIMEOUT900Seconds a remote compilation may take, waiting for a slot included, before the crate compiles here.
SHIPPY_EXEC_RECHECK=1offWhen the box's compiler fails, compile here too and report this machine's result.
SHIPPY_EXEC_INLINE4 MiBThe most bytes of inputs a request brings with it. 0 brings nothing and looks up first.
SHIPPY_EXEC_MAX_BYTES, SHIPPY_EXEC_MAX_FILES64 MiB and 20,000 files (1 GiB and 50,000 for a downloaded package)The most a crate declares when its whole package is sent. A larger package compiles here, or is sent without the packages below it when the crate loads no procedural macro.
SHIPPY_BROKER=0onEvery wrapper connects to the box itself instead of sharing the front end's connections.
SHIPPY_BROKER_CONNECTIONS8Connections that lookups, blobs and uploads share. Compilations hold their own beyond these.

The box has its own limits (SHIPPYD_EXEC_SLOTS, per-principal slots and queues, size caps). See Run a box and Server config.

Failures and fallbacks

The remote path never fails a build that rustc would pass. Whatever goes wrong ends with your machine's compiler.

What happensWhat the wrapper does
The box cannot be reached, or the answer breaksCompiles here, and leaves the box alone for 30 seconds.
429 (the box is busy, or you are past your queue)Compiles here. No wrapper of this store asks the box to compile for Retry-After seconds. Lookups go on.
424 (inputs missing)Sends what is missing and asks again, at most twice.
413, 507 or another errorCompiles here. 507 means the inputs would take you past your share of the box's store.
The box uses another linker or lacks the toolchainCompiles here. shippy doctor says whether the box's linker matches this machine's.
No answer within SHIPPY_EXEC_TIMEOUT, or 30 seconds without a heartbeatCompiles here.
A box from before /v1/exec answers 404 or 415Compiles here, and remembers that for an hour.

When the compiler says no. A compile error counts as the crate's own only when the sandbox held the whole package. Then the wrapper shows rustc's messages once and exits with rustc's exit code, without compiling here. If the error looks like the sandbox caused it (a missing file, an undefined variable, a proc macro that panicked, a crashed compiler), or the package was sent without the packages below it, your machine compiles and its result counts. SHIPPY_EXEC_RECHECK=1 always compiles here after a remote failure.

Your machine while the box compiles

A wrapper waiting for the box uses none of your cores. So the front end lets cargo work on more crates than you have cores: eight per core, at most 64, unless you chose a number with -j, CARGO_BUILD_JOBS or build.jobs.

What compiles locally (a miss that found a core free, a crate the box refused, a rebuild, a test, a build script's executable) first takes one of SHIPPY_LOCAL_JOBS slots. The slots belong to the local store, so two builds on one machine share them. Local compilers and build scripts are pointed at a jobserver sized to your machine, not to cargo's raised job count, so a C build does not start 40 compilers on five cores.

Why so many cargo jobs

A crate waiting for the box holds one of cargo's jobs and no core. With few jobs, crates the box could compile queue behind local ones. On a 5-core machine against a 48-thread box, a cold build took 236.5 seconds with 20 jobs and 218.7 with 64.

What it does not do

  • It does not run tests on the box.
  • Build scripts of dependencies run on the box only when their runs are cached and shared. See Build scripts.
  • A request is never answered from someone's unverified claim, only from a public result or the box's own run.

Go deeper

Edit this page on GitHub