> What coreboot problems that we have seen in the past are we actually solving with these rewrites?
To be a bit more blunt, what is expected to be improved by writing it in python?
Utilities, eg. to analyse blobs?
Menu and configuration of builds?
Packing blobs for flashing?

I could see some things being harmless, like using python to drive menus and check dependencies. I could potentially see a case that it's preferable to complicated #ifdef logic and for handling of board variants.

On the other hand, I could definitely see python adding new ways that reproducible builds fail, unless you compile a specific python interpreter on the spot. Who knows what subtle differences there might be in python's handling of binary data, especially on different arches and systems with different endianness? Maybe I'm paranoid, or maybe this stuff could be caught well enough in automated testing (and $someone could fix it).

Perhaps user-driven utils like those that extract blobs or examine MSRs to discover interrupt routing on proprietary firmware are a safer middle ground between the two extremes, but those are already written in C aren't they?

>Python is already being used, in the "util" directory. util/qualcomm has ~3500 LoC in Python [1] (this is not just a "small snippet" of code).
How was it introduced? Was this from a vendor dump? Those seem to occasionally bend the rules of what would be "ideal."

-Matt


On Wed, Sep 29, 2021 at 5:15 PM Ricardo Quesada via coreboot <coreboot@coreboot.org> wrote:
Thanks Patrick for bringing this in.

Regarding Pytest for utils (https://review.coreboot.org/c/coreboot/+/57869 ), what's the recommended way to test the tools that are placed in "util/*" ?
I noticed that most of them don't have any kind of end-to-end test (I couldn't find a single one, but perhaps I missed them).

Using PyTest (or any other easy to install / easy to use framework) to create tests for util/* seems natural to me. 
Although any other test framework can be used. Suggestions?

Besides all the things that have been said in favor or against Python, I'd like to add that:
   Python is already being used, in the "util" directory. util/qualcomm has ~3500 LoC in Python [1] (this is not just a "small snippet" of code).
   (compare it with PyTests for elogutil[2]: It has ~160 LoC ).

[1]: https://github.com/coreboot/coreboot/tree/master/util/qualcomm
[2]: https://review.coreboot.org/c/coreboot/+/57869


Thanks.


On Wed, Sep 29, 2021 at 12:09 PM Stefan Reinauer <stefan.k.reinauer@gmail.com> wrote:
On Wed, Sep 29, 2021 at 7:44 AM Jack Rosenthal <jrosenth@google.com> wrote:
Overall I think introducing Python to the build would provide net benefit, mainly from Kconfiglib, but could also find other good uses in e2e tests like Ricardo was working on. Most people's Linux distros ship with a Python interpreter too, so most developers would be unlikely to notice the extra dependency introduction.

In terms of Kconfiglib, we have a lot to gain by switching away from the Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've forked from the Linux tree and hacked in our own customizations (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into a miniature Python script that we use to handle our custom header format, and a stable API to work off of so that we can uprev Kconfiglib without needing to change our scripts.

Looking at the current Kconfiglib implementation we would be replacing the C code with 21873 lines of Python code that is now taking the code to deviate from what the Linux kernel is doing. I am having a hard time seeing a "net benefit" in this scenario. Given the mess that Python 2 to Python 3 conversion has been (and still is), this is just inviting a lot of trouble into what has been a fairly stable part of coreboot for the last decade.

In terms of Kconfiglib's stability and track record: I think it has it covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already without any issues at all.

I am failing to see how anybody involved in coreboot would sign up for and commit to porting 20k lines of Python code to the next version, when it arrives. My indication is that not even the python code that is currently in the tree has been ported to python3 yet.
 
At a minimum, I think we should consider introducing Python on an optional basis (i.e., the C Kconfig implementation only gets used if a Python interpreter is unavailable), but making it required would be even better.

Come on, we do not need two types of Kconfig parsers in the tree. Let's focus coreboot on actually booting cores, not collections of implementations in different programming languages. There are better projects for that, such as https://github.com/mame/quine-relay

What coreboot problems that we have seen in the past are we actually solving with these rewrites?

Stefan
 

On Wed, Sep 29, 2021 at 5:39 AM Rao G <grao.v80@gmail.com> wrote:
Hi Patrick,

That's good to hear, would there be change to "make menuconfig" with kconfiglib 

No, we'd make it so that all the "make *config" commands run the Kconfiglib alternatives without any user action required.

The "menuconfig" interface in specific is very similar to the lxdialog-based interface that the C Kconfig uses, except it's a bit more polished and refined feeling.
 

Thanks
Ranga

On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot <coreboot@coreboot.org> wrote:
Hi everybody,

Historically, coreboot avoided depending on python too much (we got rid of an entire python based configuration and build system, for example), with few minor exceptions.

The main reason has been that while python code is quick to slap together, it has demonstrated a penchant for breaking in all kinds of mysterious ways (python 2->3 really was just a slightly bigger instance of what's going on in python all the time), and its users demonstrate a disregard for their fellow developers as demonstrated by endless stack traces on trivial errors (or is the language too complicated to properly catch them all?)

While probably nice for one-off prototypes, long term maintenance is a concern: this project has over 20 years of history under its belt, with more to come.

That said, python makes its way back into the tree every now and then (typically as small snippets to compute and add hashes to binaries as needed by ARM SoCs). Uncanny, but typically not a big deal.

There are two bigger initiatives proposed that would significantly increase our python footprint:
1. Replacing Linux's kconfig with kconfiglib (https://review.coreboot.org/c/coreboot/+/48679)
2. Using pytest for end-to-end testing utilities (https://review.coreboot.org/c/coreboot/+/57869)

Compared to the "inject a hash value at a fixed location" scripts, these would probably be here to stay, and sufficiently integrated that everybody will have to deal with them.

People spending time working on python code when it has no chance to land isn't a good use of their time and we should avoid that in the project.

People spending time arguing that python shouldn't be used (to avoid the other outcome) even though the project's culture shifted and is now accepting Python isn't creating a great community for anybody.

To avoid these scenarios, could we possibly nail down the policy on python in coreboot?


Regards,
Patrick
--
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
_______________________________________________
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-leave@coreboot.org
 
_______________________________________________
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-leave@coreboot.org