Trust: claims and verification

A shared cache is only useful if many machines can write to it, and only safe if none of them can slip in a bad result. shippy solves this by making the shared answer something only the box itself can produce. This page explains how.

The problem

A blob (a stored file) can't lie: its name is the hash of its bytes, and the box re-hashes every upload. What can lie is the mapping from a key to a result: "the crate with these inputs compiles to this file". A malicious or broken machine could upload any file under any key.

Bazel and sccache answer by letting only CI on the main branch write. That shuts out every developer and agent. shippy lets everyone write, but treats most writes as claims.

Who is who

Every machine talks to the box with a token. The token belongs to a principal (a named user of the box) and has a role:

Worker machines have roles of their own (worker, trusted-worker); see worker machines.

The life of an untrusted upload

1. Upload. Alice's upload is stored as a claim. It is returned only to Alice: her own machines reuse her builds at once, and nobody else ever sees them.
2. Re-run. The box re-runs the claim's recipe in a sandbox that holds the declared inputs and nothing else: no network, a clean environment, inputs hard-linked read-only from the store.
3. Publish. What the sandbox produces becomes the public result, whatever Alice uploaded. Her bytes are only compared with it, so the box can count how many claims matched.

Why this is safe

A claim can't smuggle an input. If a recipe leaves out a file the crate really reads, that file isn't in the sandbox, the compile fails, and nothing is published. So a public result depends only on the inputs its key names.

A public manifest says what the box's own compiler read. A manifest is shown to everyone who builds the same crate, and leads a client whose files and environment match it to a result. So what a sender writes in a manifest decides only what its own sandbox holds. The box publishes a run under a manifest only when the manifest names every file and variable the run's dep-info names, each variable with the value the sandbox had, and nothing the run did not read. The rest of a package's files are in a public manifest only for a crate that does not compile without them, which the box learns from the compiler and not from the request: a run that held such files and did not need them is answered to whoever asked, and is nobody else's. A claim whose manifest is anything else stays a claim. No manifest names one of shippy's own variables (SHIPPY_*).

A manifest only steers a client to a key, and the client checks where it leads. Your machine does not rest on what the box says of a manifest. It takes the result only when the result's own dep-info (the compiler's account of what it read) names no variable that has another value on your machine, so a result compiled with a variable unset is not restored where you set it. Of several manifests that match, yours come first, then a trusted principal's result, then one the box verified: never the one that lists more files. And a manifest someone else listed never says which files or variables your own requests to the box declare, so nobody can make your machine send a variable by naming it.

Your token stays out of the build. The bearer token is in no compiler's and no build script's environment, and no manifest names a SHIPPY_* variable.

Code that runs at compile time is confined. Proc macros and the compiler itself run inside the sandbox. A malicious proc macro can only change the output of crates that list it as an input, and its digest is already part of their keys.

The sandbox is locked down. It is bubblewrap with every namespace unshared, the toolchain and system directories read-only, the inputs read-only, and only the output directory writable. A system call filter refuses what no compiler needs, and each run has a budget of memory, processes, CPU time and bytes written.

Who can read what

A digest is not a secret: a public manifest lists the digests of the sources it was compiled from. So the box never serves a blob just because someone names its digest. It serves a blob to a principal only when:

• they sent those bytes themselves (the box marks each blob with every principal it came from), or
• they ask through a result they may see: the blob is an output or the recipe of the public result, or of their own claim.

A blob you may not read is answered exactly like one the box doesn't have. Nothing tells you whether somebody else's file is on the box.

Compiling is a way of reading, so the same rule decides what a compilation is given. A request to compile that names an input you may not read is told to send it, in the same words and in the same place in the list as an input the box lacks, and its size counts for nothing. The box asks when the request arrives and again when the run starts, so a file somebody else uploads while your request waits never ends up in your build.

Nobody can push your results out

Every edit adds a manifest under the same pre-key, so the list a lookup sees is bounded. The bounds are set so an untrusted principal can only ever crowd out its own entries:

• Manifests of trusted results are one list, changed only by trusted uploads. A lookup sees the 32 newest.
• Each untrusted principal has its own share of 16 per pre-key. One more replaces its own oldest, nobody else's.
• A lookup sees the trusted list, the asker's own share, and up to 32 from other principals, taken in turns.

Toolchains

A recipe names its compiler, and only a toolchain whose rustc -vV prints exactly that text may run it. The box runs anyone's recipe with toolchains it already has, but installs a new one only for a trusted principal's recipe or one the operator listed (--install-toolchains). An untrusted claim for a missing toolchain stays a claim.

What this does not cover

• Results are not signed. The box, and the network path to it, are trusted for integrity.
• A public result is public whole. Its outputs and compiler messages (which quote source lines) are served to anyone who may see it.
• Build-script runs are public only when the box ran the script itself. See build scripts.

The full list, with the test for each defence, is in the threat model.

Go deeper

• DESIGN.md, "Trust"
• docs/threat-model.md