The wire: prefetch and uploads
A thousand crates times two requests each, from a new process every time, is too slow, especially over a long link. So shippy fetches a whole build before cargo starts, and uploads while cargo runs. This page explains how.
Before cargo starts: the prefetch
A build id names "a build like this one" without any machine's paths: the toolchain, the content of Cargo.lock, the cargo arguments, the directory cargo runs in, the switches that change keys, and environment that reaches every key (SOURCE_DATE_EPOCH, RUSTFLAGS). It does not cover your sources: two commits with one lock file need mostly the same results.
A pack is the list of results a whole build used, sent to the box after the build passes. A build counts as whole only when the target directory was empty at the start, so cargo ran the wrapper for every crate. Partial rebuilds never send a list.
When you ask for a pack, you get your own list first, then the trusted list, then other principals' entries whose results are already public. The box keeps the lists of your last few different builds of one build id, and when two of them take turns (two cargo homes, a branch and main) it answers both, so neither build pushes the other's results off the list. Those earlier lists are yours alone: another principal is shown your last list only, and for an untrusted principal the whole file counts against its share of the store.
A rebuild after an edit pays one small request: the box tags each list, and a request naming the current tag gets 304 Not Modified.
Compression
Between two sites the link is the limit, not the box. Over a 90 ms round trip, one stream carried 43 MB/s and eight carried 337 MB/s, and zstd level 1 shrank tempo's files 4.1 times at about 400 MB/s per core.
So the client asks for one zstd stream per connection, unless the box is on loopback or answered within 4 ms, where compressing would be the bottleneck. SHIPPY_COMPRESS=1 or 0 decides by hand. Files stay uncompressed in the store, because the sandbox hard-links them.
While cargo runs: uploads
- Cargo never waits for an upload round, and nothing a round does changes the build's exit code.
- The box only asks for what it needs: blobs it lacks, and blobs it has only from someone else (to read a blob, you must show you have it).
- Each frame certifies itself. A frame whose bytes don't match its digest is dropped, and the box carries on with the next one.
- A result is offered only once all its blobs are there. Otherwise it waits for the next round, or for
shippy push, which sends whatever is still queued.
When the box is busy or down
| Situation | What happens |
|---|---|
| Box unreachable | One failed connect, then the wrappers leave it alone for 30 seconds. The build goes on locally. |
| Prefetch fails | One line on stderr; each crate asks for itself. |
Box answers 503 to a stream | The client retries three times, waiting longer each time, then fetches blob by blob. A busy box never turns a hit into a compilation. |
Box too old for a route (404) | The client falls back to one item per request for the next hour. |
Proxies and TLS
The client follows http_proxy, https_proxy and no_proxy. A box at an https address must have a certificate from a public authority: the client carries Mozilla's list and doesn't read the machine's own.
Turning it off
SHIPPY_PREFETCH=0, or prefetch = false under [remote], turns off both the fetch and the list. SHIPPY_PREFETCH_JOBS and SHIPPY_PUSH_JOBS set the number of connections (default 16).