#Reproducible builds

1 messages · Page 1 of 1 (latest)

hoary pike
#

We've been discussing reproducible builds here and there, and I think the topic warrants a thread dedicated to it.

There's also this thread https://discord.com/channels/897514728459468821/1218934322745315389 that is focused on trusted builds where a source to wasm mapping can be proven without actually needing reproducible builds, since the mapping just needs to occur once to be verifiable, by using a single system, GitHub Actions, to act as a centralised source of truth on what code maps to what wasm.

This thread is separately focused on how can we make builds of Soroban contracts reproducible, so that within a system like the GitHub one above, or on an independent local machine, it is possible to go a step further and guarantee with a high degree of certainty you could reproduce the exact same wasm binary at some later point.

#

cc @uncut blade

#

The tooling does already today store some meta information in contracts about their build environment, and we could expand that.

#

The custom section is a stream of SCMetaEntry XDRs.

# 
$ wasm-cs target/wasm32-unknown-unknown/release/test_empty.wasm ls
contractspecv0 (120 bytes)
contractenvmetav0 (12 bytes)
contractmetav0 (104 bytes)
# 
$ wasm-cs target/wasm32-unknown-unknown/release/test_empty.wasm read contractmetav0
Length: 104 (0x68) bytes
0000:   00 00 00 00  00 00 00 05  72 73 76 65  72 00 00 00   ........rsver...
0010:   00 00 00 06  31 2e 37 36  2e 30 00 00  00 00 00 00   ....1.76.0......
0020:   00 00 00 08  72 73 73 64  6b 76 65 72  00 00 00 35   ....rssdkver...5
0030:   32 30 2e 35  2e 30 23 31  30 30 31 38  64 38 35 35   20.5.0#10018d855
0040:   66 39 64 33  65 38 32 35  35 33 64 65  33 65 31 64   f9d3e82553de3e1d
0050:   62 33 32 38  35 64 65 65  33 35 38 31  34 61 39 2d   b3285dee35814a9-
0060:   64 69 72 74  79 00 00 00                             dirty...
# 
    | stellar-xdr decode --type ScMetaEntry --input stream --output json-formatted
{
  "sc_meta_v0": {
    "key": "rsver",
    "val": "1.76.0"
  }
}
{
  "sc_meta_v0": {
    "key": "rssdkver",
    "val": "20.5.0#10018d855f9d3e82553de3e1db3285dee35814a9-dirty"
  }
}```
uncut blade
#

I have started doing some hacking in this direction, teaching soroban-cli to reproduce builds based on metadata in the contract. To start I'm just pulling out the already embedded rust version from some local wasm, building a local project with that toolchain, and comparing. After that there are many obvious directions: downloading wasm from chain, embedding repo/revision/project-name data so the source can be located automatically; embedding more toolchain data like the wasm-opt revision, configuration data like cargo features and compiler flags; add a pre-deploy reproduction option (perhaps in a container) to ensure a build can be reproduced.

uncut blade
#

I have continued hacking on an experiment in making soroban contracts reproducible.
In this experiment I have modified soroban-cli, when running contract build and contract optimize,
to embed additional metadata about the build into the contract; I then added a soroban contract repro
command that looks at the metadata in a given wasm, locates the source,
and attempts to perform the build with similar toolchain setup as the original.

As of now it can

  • embed the git(hub) repo and commit
  • embed the path to the manifest file and the package name
  • embed the soroban-cli version
  • embed whether wasm-opt was run
  • (the rust version is already embedded by the sdk)
  • clone and checkout the source repo
  • build the correct package with the correct toolchain
  • run wasm-opt
  • check that the repro contract is the same as the original

It doesn't yet

  • attempt to repro with the correct version of soroban-cli/wasm-opt

My next steps are to

  • write a test suite that builds and repros all the soroban-example projects
    with a variety of settings
gusty knotBOT
#

Remember you can open a Github issue right here from Discord with the /create command.

Stella Issue Detector
uncut blade
hoary pike
#

With a tooling interface like that, where a .wasm is downloaded, info extracted, code downloaded and compiled, there opportunity for developers to run the repro on random untrusted contracts is pretty high. Rustc isn't safe to use to build untrusted code because compilation is a direct access to arbitrary code execution.

#

I guess these two goals are at odds. On the one hand the main place to use this feature is with untrusted code. But compiling untrusted rust code is unsafe.

#

And it's deceiving because downloading and executing the untrusted wasm is intended to be safe, but then there's this escape hatch where all the safeties are off.

#

Maybe we can address that with a big enough warning. A warning that displays the code URL, and git commit hash, and says "review the code before hitting the go for launch button" or something like that.

uncut blade
#

That's definitely a concern for arbitrary people verifying arbitrary contracts. Anybody verifying untrusted contracts would have to do it in a sandbox. The tooling could potentially attempt to do the sandboxing for you.

hoary pike
#

Would docker be sufficient?

#

I don't think so.

uncut blade
#

I think docker is not really sufficient - last time I looked into docker for security sandboxing it wasn't advised, mostly because it wasn't designed for security as I understand it.

#

In the meantime I have created some scripts that build then reproduce all of the soroban-examples in a few different configurations. It seems to work when done on the same platform, but when I do the build on mac, then the repro on linux, or vice versa, there are a few discrepancies.
This is disappointing as I was hoping for wasm, the toolchains would produce identical results; and it's possible they do and there is just a bug in my own repro code, but I have yet to debug.

gusty knotBOT
#

Remember you can open a Github issue right here from Discord with the /create command.

Stella Issue Detector
uncut blade
#

So besides debugging that, I was considering starting a project to reproduce the top N production soroban contracts. These wouldn't have the embedded matadata to locate the source and configure them correctly, but I could try to figure out how to reproduce them myself and keep a little database of their configurations. This is just to have some dogfooding of the reproduction tooling and some real-world results.
Eventually I'd expect other tools like block explorers to automatically verify reproducibility of major contracts, perhaps using tooling provided by stellar.

ruby violet
hoary pike
#

Could it be that the different host platforms has different defaults for some compiler options?

uncut blade
#

@hoary pike I have debugged the issue of reproducing contracts across platforms and have discovered that rustc is embedding absolute paths to source files for the purposes of displaying panic messages. This doesn't manifest for simple contracts, but it does manifest when calling a crate dependency that itself contains a panic.
This is something I had assumed rust had already solved, but not quite yet. A full solution within cargo seems to be unstable.
Here are two related links:

3127-trim-paths - The Rust RFC Book
GitHub
Tracking Issue for trim-paths RFC 3127 · Issue #111540 · rust-lang/...

This is a tracking issue for the RFC 3127 (rust-lang/rfcs#3127). This enhancement adds the --remap-path-scope command-line flag to control the scoping of how paths get remapped in the resulting bin...

GitHub
Strip absolute local paths from binaries · Issue #1445 · stellar/st...

What problem does your feature solve? For builds to be reproducible, the binaries cannot contain absolute filesystem paths. Today there are cases where rustc embeds absolute paths even when debugin...

gusty knotBOT
#

Remember you can open a Github issue right here from Discord with the /create command.

Stella Issue Detector
hoary pike
#

The requirement to use RUSTFLAGS is unfortunate. Instead of asserting it's already set, are there safe ways to merge an existing set of rust flags with parameters we want to provide? I guess that's problematic or at best makes assumptions.

uncut blade
uncut blade
#

I've continued working in this direction and discovered that there are more issues (other than panic message file paths) with contracts not reproducing across different host platforms. I am still debugging these.

hoary pike
uncut blade
#

I've found that when building e.g. the liquidity_pool example on macos and linux with the same version of the rust compiler and same lockfile, that definitions in the wasm are in slightly different orders; and that if I don't strip symbols, then the type hash in the symbols generated on macos is different from the type hash in the symbols generated on linux. And interestingly this is only true for soroban_sdk symbols - core library symbols have the same type hashes when built on both platforms.

hoary pike
# uncut blade I've found that when building e.g. the liquidity_pool example on macos and linux...

Looks like people reported this difference between macos and linux in 2022 on this issue: https://github.com/rust-lang/rust/issues/97919

GitHub
Mismatched wasm output on darwin vs Ubuntu · Issue #97919 · rust-la...

I tried this code: // Crate A use std::cell::{RefCell}; use std::collections::HashMap; use internet_identity_interface as other; #[export_name = "the name"] fn dummy() { let _err_info = &...

uncut blade
#

I added a comment there. Seems like this is a pretty deep problem.

uncut blade
hoary pike
#

That means we probably have another ~8 weeks before we see that show up in stable?

#

I saw in the comments you figured out the path remapping successfully. Is that something you'd be up for contributing to the stellar contract build command?

hoary pike
uncut blade
#

I am doing some testing to figure out where in the relaase train that cargo fix is and how it impacts soroban reproducibility. cargo updates seem to be pulled into the rust distribution irregularly.

hoary pike
uncut blade
#

I've done some testing with a nightly toolchain that contains that fix. I've found that some of the crates now have identical metadata hashes, but many still have different hashes. I am trying to figure out which is the first dependency that has a different hash and figure out where the difference is.

#

Pretty big rabbit hole!

uncut blade
#

My current hypothesis is that proc macro dependencies introduce platform-dependent data into the hash chain. In our case that includes wasm_bindgen_macros and soroban_sdk_macros. I am building cargo so I can instrument it and see if that is the case and if I can propose a strawman patch to fix it.
For now I am still assuming that we want to be able to reproduce builds regardless of the build plattform, but this issue at least would be moot if we assume builds are done on linux.

hoary pike
#

I think it's worth going deeper, but it could be reasonable to say reproducible builds are platform specific and rely on a docker linux container. However, I worry that makes deployment inconvenient though, since a deployed contract would only be verifiable if it had been built and deployed via the container.

#

We shouldn't be using wasm_bindgen_macros in the sdk at all.

#

With the soroban_sdk_macros proc-macros, would it only be if the generated code was different? Or does the very presence of a proc-macro alter other aspects of the build?

eternal goblet
#

Im blockchain and frontend developer.

uncut blade
# hoary pike With the `soroban_sdk_macros` proc-macros, would it only be if the generated cod...

The presence of any proc macro or build script causes the the hashed metadata passed to the wasm crate to be different. I've left a comment on the cargo issue tracker about it https://github.com/rust-lang/cargo/issues/8140#issuecomment-2285036273 and am hoping they'll give some guidance on an acceptable solution.

GitHub
Non-reproducible -C metadata=hash passed to rustc depending on the ...

Problem Tock is an operating system written in Rust for embedded platforms (e.g. thumbv7em-none-eabi target). We've been trying to make builds reproducible, with good progress (tock/tock#1666)....

uncut blade
uncut blade
#

ok, at this point, the absolute path issue (https://github.com/stellar/stellar-cli/issues/1445) and this cargo build script/proc macro issue are the only obstacles i'm currently aware of.
Next I'm going to do a new round of testing with patches to cargo and stellar-cli that fix both issues, and see if there are any further reproducibility failures in soroban-examples. I'll also develop a mergable patch for the absolute paths issue submit a pr to soroban-cli.

GitHub
Strip absolute local paths from binaries · Issue #1445 · stellar/st...

What problem does your feature solve? For builds to be reproducible, the binaries cannot contain absolute filesystem paths. Today there are cases where rustc embeds absolute paths even when debugin...

chrome grove
# uncut blade ok, at this point, the absolute path issue (https://github.com/stellar/stellar-c...

I posted this to your GH issue but wanted to reply here as well for transparency sake 🙂

This may be a long shot and I'm not entirely sure how you'd go about testing it on your end, but zig released https://github.com/rust-cross/cargo-zigbuild?tab=readme-ov-file as a drop in solution to rust cross compilation issues that I think may be relevant to the issues you're facing. And full disclaimer, I am by no means a rust developer, this is just pure anectdotal speculation on my end based on what I know about Zig

Here are a few links that I found interesting that may be of relevance as well.
https://docs.rs/cargo-zigbuild/latest/cargo_zigbuild/struct.Rustc.html
https://actually.fyi/posts/zig-makes-rust-cross-compilation-just-work/
https://ziglang.org/learn/overview/#cross-compiling-is-a-first-class-use-case

some other maybe useful links about Zig
https://www.quicknode.com/guides/cross-chain/layerzero/what-is-layerzero
https://youtu.be/5_oqWE9otaE?t=953 <--zig creator discussing cargo_zigbuild

uncut blade
#

I have testing reproducing all of soroban-examples with patches for the two known issues, and it worked, but with one pretty big caveat:

uncut blade
#

@hoary pike the contractimport! macro imports metadata from another contract. If that other contract hasn't been built reproducibly then the current contract won't either. One reason is that the sha256 hash of the wasm contract is an input to the macro and is presumably embedded in the built contract in some way, but there are many obstacles:

This is an external dependency that the build system doesn't know anything about - it's just a file on disk and we don't know where it came from. So if I want to reproduce an arbitrary wasm that used contractimport I have to figure out what the wasm dependencies were, where to get them, how to build them, where to put them on disk for the macro to find them.

From perusing the implementation of contractimpl (https://github.com/stellar/rs-soroban-sdk/blob/main/soroban-spec-rust/src/lib.rs#L59-L65) it looks to me like the entirety of the wasm dependency might be embedded in the wasm being built. Is that actually what the linked code is doing @hoary pike ? If so we could probably look through it and figure out what it is, read the metadata to locate the dependency's source code. It's not quite enough to decide where to put it on disk, but suggests some kind of solution.

hoary pike
#

I wouldn’t have thought contractimport! would be an issue because hopefully that .wasm has been comitted to version control. If we wanted to make sure to increase our ability to look at the code and confirm we had the right binary we could require the presence of the sha256 on the outer instead of generating it for the inner. (The inner sha256 existed only because the generated code is also used for the clis generation of rust bindings (which I don’t think gets used very much)

hoary pike
uncut blade
uncut blade
#

I am working on the two known patches needed and expect to have PRs submitted this week.

I have also prototyped a tool that can build contracts with extra metadata, and when given an on-chain contract_id, download them, locate the source, and reproduce them.

It does require the two patches to work reliably.

uncut blade