I'm proposing that we look at adding Go and Rust to the coreboot toolchain.
Go: We’re already using a number of things written in Go as a part of the build, and they currently use the system toolchain, leaving droppings outside of the coreboot and $obj folders as a part of the build. Additionally, we want the binary we produce to be reproducible - I’m not sure if that’s currently a problem or not. SBOM, STM
Rust: Rust is starting to be used more and more widely as a system programming language. The Linux kernel is accepting Rust. Oxide’s entire codebase is in Rust. The Oreboot project is using Rust. While I’m not personally a fan of Rust, I can see the need to support it coming. Ron has already posted a patch to add Rust support, but I think it needs to be reworked so that everything is contained in the XGCC directory, has a stable version for coreboot, and doesn't conflict with any system tools. https://review.coreboot.org/c/coreboot/+/74124
Neither of these is particularly urgent in my opinion, but I want to get the discussion started. I've added this as a topic to the next coreboot leadership meeting.
Martin
We’re already using a number of things written in Go as a part of the build
Only a few utils are written in Go. I don't need Go installed to actually build coreboot.
leaving droppings outside of the coreboot and $obj folders as a part of the build
With Rust, this is controlled by setting the `RUSTUP_HOME` and `CARGO_HOME` paths. How is it handled with Go?
Ron has already posted a patch to add Rust support, but I think it needs to be reworked
CB:74124 doesn't add Rust support. It doesn't even install Rust. It just installs some random tools assuming rustup is already installed on the system.
but I want to get the discussion started
Having the toolchains in the SDK is going to make them required for building even if they're not used, much like Ada, so code will at least compile. I detest Go, but it would mean I could e.g. enable SBOM without having to install Go globally on my system.
Will coreboot use prebuilt binaries, like installing Rust via rustup (does Go have something comparable?), or building the binaries itself like GCC? I assume the former, since these are more modern languages that support such distribution and management.
Hi Martin,
sorry in advance if I'll ask stupid questions :) It doesn't feel like I understood what is on your mind.
On 03.11.23 00:18, Martin Roth via coreboot wrote:
I'm proposing that we look at adding Go and Rust to the coreboot toolchain.
What's "the coreboot toolchain" in this context?
Later you mentioned xgcc, that's the cross toolchain we use to build coreboot itself, not its tools. Hence my confusion, do you expect Go and Rust in coreboot itself, like Linux does? (I would welcome the latter. But trying to see things realistic, that would require a much bigger effort and discussions much beyond the toolchain. Also, depen- ding on how we would use Rust, that may change requirements of the toolchain.)
Go: We’re already using a number of things written in Go as a part of the build,
Well, some builds, but not "the build". And I think we should discuss this separately before we make it an official requirement.
and they currently use the system toolchain, leaving droppings outside of the coreboot and $obj folders as a part of the build.
That sounds like something to tune in the Makefiles. And if not possible yet, maybe we should help the Go developers to make it possible.
Additionally, we want the binary we produce to be reproducible - I’m not sure if that’s currently a problem or not. SBOM, STM
Now I'm lost. I thought we only use it to build tools. If the output of the tools would change because of the toolchain used to build them, wouldn't that be a real bug (either in the tool or toolchain) and not just a reproducibility issue?
Rust: Rust is starting to be used more and more widely as a system programming language. The Linux kernel is accepting Rust. Oxide’s entire codebase is in Rust. The Oreboot project is using Rust. While I’m not personally a fan of Rust, I can see the need to support it coming. Ron has already posted a patch to add Rust support, but I think it needs to be reworked so that everything is contained in the XGCC directory, has a stable version for coreboot, and doesn't conflict with any system tools. https://review.coreboot.org/c/coreboot/+/74124
Did try that patch now and it raises a lot of questions. EDIT: now that I went through everything below, I think the most important question is Would we want to use `rustup`?
It started somewhat like this:
rustup run --install 1.67 cargo install --version 0.3.4 cargo-binutils make: rustup: No such file or directory
Trying to install `rustup`:
:: rustup and rust are in conflict. Remove rust? [y/N]
So to use this, I have to kill my local Rust toolchain?
Then, `rustup` ran for a while, told me what it downloaded and compiled. And I thought, nice, it's compiling from source. But isn't the Rust com- piler written in Rust? didn't I just delete my Rust compiler? I suppose the toolchain was downloaded as binary.
Then it went on with the next `rustup` command. Looked about the same, but what? aren't these the same packages it's compiling? Looking at the running processes, they were. But to my surprise nothing changed in my .cargo directory. Could still be that something was statically linked and the artifacts removed. But it seems like an inefficient way to use software anyway. IMO ok for somebody typing these commands into a terminal, but a bit too much to put into a script for anybody to use without additional thought.
This story reminded me of two things:
1. Currently we have a chain of trust: * our Git repository and the checked out commit (hash) * hashes for files to download (e.g. for the GCC sources) - those are checked into the repository If we surrender the control about what to download (and install and run) to another program, we should probably warn the user. I had a quick look at `rustup` and `cargo` and couldn't find an answer if things can be specified by hash. I hope they can.
2. So far we built everything (that was added on top of the user's OS distribution) from source. I think we should be proud of that.
Neither of these is particularly urgent in my opinion, but I want to get the discussion started. I've added this as a topic to the next coreboot leadership meeting.
I would just wait until somebody committed to maintaining coreboot needs it and then assist them the best we can to add support.
Nico
Would we want to use `rustup`?
Yes.
So to use this, I have to kill my local Rust toolchain?
That's not a "local" rust toolchain. That's the distro-provided global toolchain.
You would remove the distro packages of `rustc` and `cargo` and use rustup instead. Rustup is a toolchain manager; It allows you to install multiple toolchain versions side-by-side, instead of just the 1 version the distro provides.
(I would hope Go has something comparable for managing multiple toolchain installs.)
And I thought, nice, it's compiling from source. But isn't the Rust com- piler written in Rust? didn't I just delete my Rust compiler? I suppose the toolchain was downloaded as binary.
- rustup downloads toolchains as binaries - cargo downloads and builds crates from source
Then it went on with the next `rustup` command. Looked about the same, but what? aren't these the same packages it's compiling? Looking at the running processes, they were. But to my surprise nothing changed in my .cargo directory. Could still be that something was statically linked and the artifacts removed.
Crates are mostly statically linked. Even if different crates use the same dependency, they are part of separate projects are so are rebuild. Other considerations are they may use different versions of the same dep, or enable different features. `cargo install` will compile crates in `/tmp`.
- Currently we have a chain of trust:
If we surrender the control about what to download (and install and run) to another program, we should probably warn the user. I had a quick look at `rustup` and `cargo` and couldn't find an answer if things can be specified by hash. I hope they can.
- our Git repository and the checked out commit (hash)
- hashes for files to download (e.g. for the GCC sources)
- those are checked into the repository
It's not hashes, but projects can use a `rust-toolchain.toml` file to control the exact Rust version used. e.g.:
[toolchain] channel = "nightly-2023-09-07" components = ["clippy", "rust-src", "rustfmt"]
So regardless of the distro you're on, the same version of the toolchain is used.
- So far we built everything (that was added on top of the user's OS distribution) from source. I think we should be proud of that.
Building from source is just a means to an end: Ensuring the same version of the toolchain is always used. `rust-toolchain.toml` accomplishes this for Rust projects.
Hi Tim,
thanks for the insight!
On 03.11.23 18:04, Tim Crawford via coreboot wrote:
Would we want to use `rustup`?
Yes.
So to use this, I have to kill my local Rust toolchain?
That's not a "local" rust toolchain. That's the distro-provided global toolchain.
Yes, but I deleted my local copy. Not all the copies in the world ;) I meant local as "not the rust(up) project's one".
You would remove the distro packages of `rustc` and `cargo` and use rustup instead. Rustup is a toolchain manager; It allows you to install multiple toolchain versions side-by-side, instead of just the 1 version the distro provides.
(I would hope Go has something comparable for managing multiple toolchain installs.)
I would hope this wouldn't be necessary and one toolchain could compile all the Rust programs (we'll need).
- Currently we have a chain of trust:
If we surrender the control about what to download (and install and run) to another program, we should probably warn the user. I had a quick look at `rustup` and `cargo` and couldn't find an answer if things can be specified by hash. I hope they can.
- our Git repository and the checked out commit (hash)
- hashes for files to download (e.g. for the GCC sources)
- those are checked into the repository
It's not hashes, but projects can use a `rust-toolchain.toml` file to control the exact Rust version used. e.g.:
Do you happen to know what cryptographic mechanisms are used to ensure that what the server delivers is what was asked for?
Nico
Hi
From https://go.dev/blog/compat "Go programs that work today should continue to work even as future releases of Go 1 arise." A minimal version can be specified in the go.mod file. As go is only used in tooling and that the output is specified to not depend on the version. The benefit of caring about reproducibility of the go toolchain and its outputs is IMO less important. Also getting the right toolchain might just become a trivial issue in the future: https://go.dev/blog/toolchain . Rustup worked like that for some time.
For rust I'd recommend making rustup a dependency. You can specify a toolchain version which the rust code should automatically take care of reproducibility. Getting the 'right compiler' is a trivial issue in rust, so I'd add no explicit dealing except by what is provided by the package manager (cargo). Getting rustup is a one-liner btw, it's a cool tool!
About rust in coreboot. I like it a lot! The tooling is excellent and makes importing new code a bliss compared to C. The language itself has a steep learning curve, but is very elegant once you know it in my opinion. I already linked some rust code into a coreboot stage and wrote a blog post about it: https://blog.aheymans.xyz/rust_coreboot.html I believe a proper integration of rust is a very achievable goal.
1. Currently we have a chain of trust:
* our Git repository and the checked out commit (hash) * hashes for files to download (e.g. for the GCC sources) - those are checked into the repository If we surrender the control about what to download (and install and run) to another program, we should probably warn the user. I had a quick look at `rustup` and `cargo` and couldn't find an answer if things can be specified by hash. I hope they can.
Is rustup-init the anwser? https://rust-lang.github.io/rustup/installation/other.html has hashes?
Arthur
On Fri, Nov 3, 2023 at 12:19 AM Martin Roth via coreboot < coreboot@coreboot.org> wrote:
I'm proposing that we look at adding Go and Rust to the coreboot toolchain.
Go: We’re already using a number of things written in Go as a part of the build, and they currently use the system toolchain, leaving droppings outside of the coreboot and $obj folders as a part of the build. Additionally, we want the binary we produce to be reproducible - I’m not sure if that’s currently a problem or not. SBOM, STM
Rust: Rust is starting to be used more and more widely as a system programming language. The Linux kernel is accepting Rust. Oxide’s entire codebase is in Rust. The Oreboot project is using Rust. While I’m not personally a fan of Rust, I can see the need to support it coming. Ron has already posted a patch to add Rust support, but I think it needs to be reworked so that everything is contained in the XGCC directory, has a stable version for coreboot, and doesn't conflict with any system tools. https://review.coreboot.org/c/coreboot/+/74124
Neither of these is particularly urgent in my opinion, but I want to get the discussion started. I've added this as a topic to the next coreboot leadership meeting.
Martin _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
There is also a rust frontend for GCC. Not sure how far it is regarding development: https://rust-gcc.github.io/ So maybe for Rust we can just use GCC. I am guessing the same will come for clang eventually (basically just another language frontend and maybe LLVM backend). Maybe worth looking into.
On November 8, 2023 8:29:30 PM GMT+01:00, Arthur Heymans arthur@aheymans.xyz wrote:
Hi
From https://go.dev/blog/compat "Go programs that work today should continue to work even as future releases of Go 1 arise." A minimal version can be specified in the go.mod file. As go is only used in tooling and that the output is specified to not depend on the version. The benefit of caring about reproducibility of the go toolchain and its outputs is IMO less important. Also getting the right toolchain might just become a trivial issue in the future: https://go.dev/blog/toolchain . Rustup worked like that for some time.
For rust I'd recommend making rustup a dependency. You can specify a toolchain version which the rust code should automatically take care of reproducibility. Getting the 'right compiler' is a trivial issue in rust, so I'd add no explicit dealing except by what is provided by the package manager (cargo). Getting rustup is a one-liner btw, it's a cool tool!
About rust in coreboot. I like it a lot! The tooling is excellent and makes importing new code a bliss compared to C. The language itself has a steep learning curve, but is very elegant once you know it in my opinion. I already linked some rust code into a coreboot stage and wrote a blog post about it: https://blog.aheymans.xyz/rust_coreboot.html I believe a proper integration of rust is a very achievable goal.
- Currently we have a chain of trust:
* our Git repository and the checked out commit (hash) * hashes for files to download (e.g. for the GCC sources) - those are checked into the repository If we surrender the control about what to download (and install and run) to another program, we should probably warn the user. I had a quick look at `rustup` and `cargo` and couldn't find an answer if things can be specified by hash. I hope they can.Is rustup-init the anwser? https://rust-lang.github.io/rustup/installation/other.html has hashes?
Arthur
On Fri, Nov 3, 2023 at 12:19 AM Martin Roth via coreboot < coreboot@coreboot.org> wrote:
I'm proposing that we look at adding Go and Rust to the coreboot toolchain.
Go: We’re already using a number of things written in Go as a part of the build, and they currently use the system toolchain, leaving droppings outside of the coreboot and $obj folders as a part of the build. Additionally, we want the binary we produce to be reproducible - I’m not sure if that’s currently a problem or not. SBOM, STM
Rust: Rust is starting to be used more and more widely as a system programming language. The Linux kernel is accepting Rust. Oxide’s entire codebase is in Rust. The Oreboot project is using Rust. While I’m not personally a fan of Rust, I can see the need to support it coming. Ron has already posted a patch to add Rust support, but I think it needs to be reworked so that everything is contained in the XGCC directory, has a stable version for coreboot, and doesn't conflict with any system tools. https://review.coreboot.org/c/coreboot/+/74124
Neither of these is particularly urgent in my opinion, but I want to get the discussion started. I've added this as a topic to the next coreboot leadership meeting.
Martin _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
-
Arthur Heymans -
Martin Roth -
Max -
Nico Huber -
Tim Crawford