Hi everybody,
Historically, coreboot avoided depending on python too much (we got rid of an entire python based configuration and build system, for example), with few minor exceptions.
The main reason has been that while python code is quick to slap together, it has demonstrated a penchant for breaking in all kinds of mysterious ways (python 2->3 really was just a slightly bigger instance of what's going on in python all the time), and its users demonstrate a disregard for their fellow developers as demonstrated by endless stack traces on trivial errors (or is the language too complicated to properly catch them all?)
While probably nice for one-off prototypes, long term maintenance is a concern: this project has over 20 years of history under its belt, with more to come.
That said, python makes its way back into the tree every now and then (typically as small snippets to compute and add hashes to binaries as needed by ARM SoCs). Uncanny, but typically not a big deal.
There are two bigger initiatives proposed that would significantly increase our python footprint: 1. Replacing Linux's kconfig with kconfiglib ( https://review.coreboot.org/c/coreboot/+/48679) 2. Using pytest for end-to-end testing utilities ( https://review.coreboot.org/c/coreboot/+/57869)
Compared to the "inject a hash value at a fixed location" scripts, these would probably be here to stay, and sufficiently integrated that everybody will have to deal with them.
People spending time working on python code when it has no chance to land isn't a good use of their time and we should avoid that in the project.
People spending time arguing that python shouldn't be used (to avoid the other outcome) even though the project's culture shifted and is now accepting Python isn't creating a great community for anybody.
To avoid these scenarios, could we possibly nail down the policy on python in coreboot?
Regards, Patrick
Hi Patrick,
That's good to hear, would there be change to "make menuconfig" with kconfiglib
Thanks Ranga
On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot < coreboot@coreboot.org> wrote:
Hi everybody,
Historically, coreboot avoided depending on python too much (we got rid of an entire python based configuration and build system, for example), with few minor exceptions.
The main reason has been that while python code is quick to slap together, it has demonstrated a penchant for breaking in all kinds of mysterious ways (python 2->3 really was just a slightly bigger instance of what's going on in python all the time), and its users demonstrate a disregard for their fellow developers as demonstrated by endless stack traces on trivial errors (or is the language too complicated to properly catch them all?)
While probably nice for one-off prototypes, long term maintenance is a concern: this project has over 20 years of history under its belt, with more to come.
That said, python makes its way back into the tree every now and then (typically as small snippets to compute and add hashes to binaries as needed by ARM SoCs). Uncanny, but typically not a big deal.
There are two bigger initiatives proposed that would significantly increase our python footprint:
- Replacing Linux's kconfig with kconfiglib (
https://review.coreboot.org/c/coreboot/+/48679) 2. Using pytest for end-to-end testing utilities ( https://review.coreboot.org/c/coreboot/+/57869)
Compared to the "inject a hash value at a fixed location" scripts, these would probably be here to stay, and sufficiently integrated that everybody will have to deal with them.
People spending time working on python code when it has no chance to land isn't a good use of their time and we should avoid that in the project.
People spending time arguing that python shouldn't be used (to avoid the other outcome) even though the project's culture shifted and is now accepting Python isn't creating a great community for anybody.
To avoid these scenarios, could we possibly nail down the policy on python in coreboot?
Regards, Patrick -- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Overall I think introducing Python to the build would provide net benefit, mainly from Kconfiglib, but could also find other good uses in e2e tests like Ricardo was working on. Most people's Linux distros ship with a Python interpreter too, so most developers would be unlikely to notice the extra dependency introduction.
In terms of Kconfiglib, we have a lot to gain by switching away from the Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've forked from the Linux tree and hacked in our own customizations (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into a miniature Python script https://chromium.googlesource.com/chromiumos/platform/depthcharge/+/refs/heads/main/util/autoheader.py that we use to handle our custom header format, and a stable API to work off of so that we can uprev Kconfiglib without needing to change our scripts.
In terms of Kconfiglib's stability and track record: I think it has it covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already without any issues at all.
At a minimum, I think we should consider introducing Python on an optional basis (i.e., the C Kconfig implementation only gets used if a Python interpreter is unavailable), but making it required would be even better.
On Wed, Sep 29, 2021 at 5:39 AM Rao G grao.v80@gmail.com wrote:
Hi Patrick,
That's good to hear, would there be change to "make menuconfig" with kconfiglib
No, we'd make it so that all the "make *config" commands run the Kconfiglib alternatives without any user action required.
The "menuconfig" interface in specific is very similar to the lxdialog-based interface that the C Kconfig uses, except it's a bit more polished and refined feeling.
Thanks Ranga
On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot < coreboot@coreboot.org> wrote:
Hi everybody,
Historically, coreboot avoided depending on python too much (we got rid of an entire python based configuration and build system, for example), with few minor exceptions.
The main reason has been that while python code is quick to slap together, it has demonstrated a penchant for breaking in all kinds of mysterious ways (python 2->3 really was just a slightly bigger instance of what's going on in python all the time), and its users demonstrate a disregard for their fellow developers as demonstrated by endless stack traces on trivial errors (or is the language too complicated to properly catch them all?)
While probably nice for one-off prototypes, long term maintenance is a concern: this project has over 20 years of history under its belt, with more to come.
That said, python makes its way back into the tree every now and then (typically as small snippets to compute and add hashes to binaries as needed by ARM SoCs). Uncanny, but typically not a big deal.
There are two bigger initiatives proposed that would significantly increase our python footprint:
- Replacing Linux's kconfig with kconfiglib (
https://review.coreboot.org/c/coreboot/+/48679) 2. Using pytest for end-to-end testing utilities ( https://review.coreboot.org/c/coreboot/+/57869)
Compared to the "inject a hash value at a fixed location" scripts, these would probably be here to stay, and sufficiently integrated that everybody will have to deal with them.
People spending time working on python code when it has no chance to land isn't a good use of their time and we should avoid that in the project.
People spending time arguing that python shouldn't be used (to avoid the other outcome) even though the project's culture shifted and is now accepting Python isn't creating a great community for anybody.
To avoid these scenarios, could we possibly nail down the policy on python in coreboot?
Regards, Patrick -- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Am Mi., 29. Sept. 2021 um 16:43 Uhr schrieb Jack Rosenthal < jrosenth@google.com>:
Overall I think introducing Python to the build would provide net benefit, mainly from Kconfiglib, but could also find other good uses in e2e tests like Ricardo was working on. Most people's Linux distros ship with a Python interpreter too, so most developers would be unlikely to notice the extra dependency introduction.
This is assuming that all python environments are alike. Experience disagrees.
In terms of Kconfiglib, we have a lot to gain by switching away from the Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've forked from the Linux tree and hacked in our own customizations (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into a miniature Python script https://chromium.googlesource.com/chromiumos/platform/depthcharge/+/refs/heads/main/util/autoheader.py that we use to handle our custom header format, and a stable API to work off of so that we can uprev Kconfiglib without needing to change our scripts.
The customizations are a set of patches with simple maintenance (as documented in https://review.coreboot.org/c/coreboot/+/57879).
In terms of Kconfiglib's stability and track record: I think it has it
covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already without any issues at all.
This project deals in 20 year timelines. Zephyr is at 5.5 years, while depthcharge is used exclusively in a tightly controlled environment (and even there we had - and have - our share of pythonic problems).
At a minimum, I think we should consider introducing Python on an optional
basis (i.e., the C Kconfig implementation only gets used if a Python interpreter is unavailable), but making it required would be even better.
That idea is... horrible. Instead of only bringing in the python dependency we'd also create an environmental dependency that can silently introduce behavioral differences.
This project has the ability of building bit-identical firmware images on hosts spanning all kinds of CPU register sizes, endianness and operating system families. If you ever heard of "hermetic builds" being a good idea: the trustworthiness of our build is stronger - and part of making that possible is being (relatively) careful with picking our dependencies.
Patrick
On Wed, Sep 29, 2021 at 9:57 AM Patrick Georgi pgeorgi@google.com wrote:
Am Mi., 29. Sept. 2021 um 16:43 Uhr schrieb Jack Rosenthal < jrosenth@google.com>:
Overall I think introducing Python to the build would provide net benefit, mainly from Kconfiglib, but could also find other good uses in e2e tests like Ricardo was working on. Most people's Linux distros ship with a Python interpreter too, so most developers would be unlikely to notice the extra dependency introduction.
This is assuming that all python environments are alike. Experience disagrees.
Kconfiglib only uses Python standard libraries, and works with both Python 3.3+ and 2.7. I think it'd be pretty hard to find a Python environment it didn't work with.
In terms of Kconfiglib, we have a lot to gain by switching away from the Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've forked from the Linux tree and hacked in our own customizations (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into a miniature Python script https://chromium.googlesource.com/chromiumos/platform/depthcharge/+/refs/heads/main/util/autoheader.py that we use to handle our custom header format, and a stable API to work off of so that we can uprev Kconfiglib without needing to change our scripts.
The customizations are a set of patches with simple maintenance (as documented in https://review.coreboot.org/c/coreboot/+/57879).
This is true, but the maintenance burden could be even less.
Additionally, the last Kconfig uprev didn't go so well, at least from the Chrome OS perspective, we saw a build flake that sat around for a few weeks bothering people.
In terms of Kconfiglib's stability and track record: I think it has it
covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already without any issues at all.
This project deals in 20 year timelines. Zephyr is at 5.5 years, while depthcharge is used exclusively in a tightly controlled environment (and even there we had - and have - our share of pythonic problems).
IMO 5.5 years of stability is a pretty good point to start considering adopting a new technology. If we wait for 20 years of it, there'll be something better than Kconfiglib out there ;)
At a minimum, I think we should consider introducing Python on an optional
basis (i.e., the C Kconfig implementation only gets used if a Python interpreter is unavailable), but making it required would be even better.
That idea is... horrible. Instead of only bringing in the python dependency we'd also create an environmental dependency that can silently introduce behavioral differences.
This project has the ability of building bit-identical firmware images on hosts spanning all kinds of CPU register sizes, endianness and operating system families. If you ever heard of "hermetic builds" being a good idea: the trustworthiness of our build is stronger - and part of making that possible is being (relatively) careful with picking our dependencies.
Fair, require python then. Most people have it on their systems anyways.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Am Mi., 29. Sept. 2021 um 19:47 Uhr schrieb Jack Rosenthal < jrosenth@google.com>:
At a minimum, I think we should consider introducing Python on an optional
basis (i.e., the C Kconfig implementation only gets used if a Python interpreter is unavailable), but making it required would be even better.
That idea is... horrible. Instead of only bringing in the python dependency we'd also create an environmental dependency that can silently introduce behavioral differences.
Fair, require python then. Most people have it on their systems anyways.
Well, the question I put up on the mailing list, for the community to decide, is if we should lift our quasi-ban on python in the project. I added you and the other contributors proposing python changes in Cc to this email as a courtesy because you're not all registered on the project's discussion venue.
The default alternative to "let's do something horrible" isn't "let's do this other thing instead", it's "don't change anything at all". Don't look further than the requirement for proposed changes to make it through review: Without "don't change anything" as default, we could just open up write access to the main repo and branch for everybody and everything - we don't.
Only if we decide that extensive python use in the coreboot tree is now okay, kconfiglib is even eligible for discussion.
So far the only argument in favor of python in _any_ way has been "it's installed everywhere". That was true 20 years ago as well, and yet still we had a fair amount of trouble with python. Look no further than the recent uptick in support requests on IRC by people trying to build coreboot + "stable" Tianocore: It fails because python moved on while the tagged Tianocore version did not and its build tooling relies on python. So I agree that "installed everywhere" is a necessary condition, but given how badly python's maintainers historically treated their user base, I don't think it's a sufficient condition, by far.
tl;dr: people of coreboot, opinions?
Patrick
On Wed, Sep 29, 2021 at 7:44 AM Jack Rosenthal jrosenth@google.com wrote:
Overall I think introducing Python to the build would provide net benefit, mainly from Kconfiglib, but could also find other good uses in e2e tests like Ricardo was working on. Most people's Linux distros ship with a Python interpreter too, so most developers would be unlikely to notice the extra dependency introduction.
In terms of Kconfiglib, we have a lot to gain by switching away from the Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've forked from the Linux tree and hacked in our own customizations (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into a miniature Python script https://chromium.googlesource.com/chromiumos/platform/depthcharge/+/refs/heads/main/util/autoheader.py that we use to handle our custom header format, and a stable API to work off of so that we can uprev Kconfiglib without needing to change our scripts.
Looking at the current Kconfiglib implementation we would be replacing the C code with 21873 lines of Python code that is now taking the code to deviate from what the Linux kernel is doing. I am having a hard time seeing a "net benefit" in this scenario. Given the mess that Python 2 to Python 3 conversion has been (and still is), this is just inviting a lot of trouble into what has been a fairly stable part of coreboot for the last decade.
In terms of Kconfiglib's stability and track record: I think it has it
covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already without any issues at all.
I am failing to see how anybody involved in coreboot would sign up for and commit to porting 20k lines of Python code to the next version, when it arrives. My indication is that not even the python code that is currently in the tree has been ported to python3 yet.
At a minimum, I think we should consider introducing Python on an optional basis (i.e., the C Kconfig implementation only gets used if a Python interpreter is unavailable), but making it required would be even better.
Come on, we do not need two types of Kconfig parsers in the tree. Let's focus coreboot on actually booting cores, not collections of implementations in different programming languages. There are better projects for that, such as https://github.com/mame/quine-relay
What coreboot problems that we have seen in the past are we actually solving with these rewrites?
Stefan
On Wed, Sep 29, 2021 at 5:39 AM Rao G grao.v80@gmail.com wrote:
Hi Patrick,
That's good to hear, would there be change to "make menuconfig" with kconfiglib
No, we'd make it so that all the "make *config" commands run the Kconfiglib alternatives without any user action required.
The "menuconfig" interface in specific is very similar to the lxdialog-based interface that the C Kconfig uses, except it's a bit more polished and refined feeling.
Thanks Ranga
On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot < coreboot@coreboot.org> wrote:
Hi everybody,
Historically, coreboot avoided depending on python too much (we got rid of an entire python based configuration and build system, for example), with few minor exceptions.
The main reason has been that while python code is quick to slap together, it has demonstrated a penchant for breaking in all kinds of mysterious ways (python 2->3 really was just a slightly bigger instance of what's going on in python all the time), and its users demonstrate a disregard for their fellow developers as demonstrated by endless stack traces on trivial errors (or is the language too complicated to properly catch them all?)
While probably nice for one-off prototypes, long term maintenance is a concern: this project has over 20 years of history under its belt, with more to come.
That said, python makes its way back into the tree every now and then (typically as small snippets to compute and add hashes to binaries as needed by ARM SoCs). Uncanny, but typically not a big deal.
There are two bigger initiatives proposed that would significantly increase our python footprint:
- Replacing Linux's kconfig with kconfiglib (
https://review.coreboot.org/c/coreboot/+/48679) 2. Using pytest for end-to-end testing utilities ( https://review.coreboot.org/c/coreboot/+/57869)
Compared to the "inject a hash value at a fixed location" scripts, these would probably be here to stay, and sufficiently integrated that everybody will have to deal with them.
People spending time working on python code when it has no chance to land isn't a good use of their time and we should avoid that in the project.
People spending time arguing that python shouldn't be used (to avoid the other outcome) even though the project's culture shifted and is now accepting Python isn't creating a great community for anybody.
To avoid these scenarios, could we possibly nail down the policy on python in coreboot?
Regards, Patrick -- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Thanks Patrick for bringing this in.
Regarding Pytest for utils (https://review.coreboot.org/c/coreboot/+/57869 ), what's the recommended way to test the tools that are placed in "util/*" ? I noticed that most of them don't have any kind of end-to-end test (I couldn't find a single one, but perhaps I missed them).
Using PyTest (or any other easy to install / easy to use framework) to create tests for util/* seems natural to me. Although any other test framework can be used. Suggestions?
Besides all the things that have been said in favor or against Python, I'd like to add that: Python is already being used, in the "util" directory. util/qualcomm has ~3500 LoC in Python [1] (this is not just a "small snippet" of code). (compare it with PyTests for elogutil[2]: It has ~160 LoC ).
[1]: https://github.com/coreboot/coreboot/tree/master/util/qualcomm [2]: https://review.coreboot.org/c/coreboot/+/57869
Thanks.
On Wed, Sep 29, 2021 at 12:09 PM Stefan Reinauer < stefan.k.reinauer@gmail.com> wrote:
On Wed, Sep 29, 2021 at 7:44 AM Jack Rosenthal jrosenth@google.com wrote:
Overall I think introducing Python to the build would provide net benefit, mainly from Kconfiglib, but could also find other good uses in e2e tests like Ricardo was working on. Most people's Linux distros ship with a Python interpreter too, so most developers would be unlikely to notice the extra dependency introduction.
In terms of Kconfiglib, we have a lot to gain by switching away from the Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've forked from the Linux tree and hacked in our own customizations (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into a miniature Python script https://chromium.googlesource.com/chromiumos/platform/depthcharge/+/refs/heads/main/util/autoheader.py that we use to handle our custom header format, and a stable API to work off of so that we can uprev Kconfiglib without needing to change our scripts.
Looking at the current Kconfiglib implementation we would be replacing the C code with 21873 lines of Python code that is now taking the code to deviate from what the Linux kernel is doing. I am having a hard time seeing a "net benefit" in this scenario. Given the mess that Python 2 to Python 3 conversion has been (and still is), this is just inviting a lot of trouble into what has been a fairly stable part of coreboot for the last decade.
In terms of Kconfiglib's stability and track record: I think it has it
covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already without any issues at all.
I am failing to see how anybody involved in coreboot would sign up for and commit to porting 20k lines of Python code to the next version, when it arrives. My indication is that not even the python code that is currently in the tree has been ported to python3 yet.
At a minimum, I think we should consider introducing Python on an optional basis (i.e., the C Kconfig implementation only gets used if a Python interpreter is unavailable), but making it required would be even better.
Come on, we do not need two types of Kconfig parsers in the tree. Let's focus coreboot on actually booting cores, not collections of implementations in different programming languages. There are better projects for that, such as https://github.com/mame/quine-relay
What coreboot problems that we have seen in the past are we actually solving with these rewrites?
Stefan
On Wed, Sep 29, 2021 at 5:39 AM Rao G grao.v80@gmail.com wrote:
Hi Patrick,
That's good to hear, would there be change to "make menuconfig" with kconfiglib
No, we'd make it so that all the "make *config" commands run the Kconfiglib alternatives without any user action required.
The "menuconfig" interface in specific is very similar to the lxdialog-based interface that the C Kconfig uses, except it's a bit more polished and refined feeling.
Thanks Ranga
On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot < coreboot@coreboot.org> wrote:
Hi everybody,
Historically, coreboot avoided depending on python too much (we got rid of an entire python based configuration and build system, for example), with few minor exceptions.
The main reason has been that while python code is quick to slap together, it has demonstrated a penchant for breaking in all kinds of mysterious ways (python 2->3 really was just a slightly bigger instance of what's going on in python all the time), and its users demonstrate a disregard for their fellow developers as demonstrated by endless stack traces on trivial errors (or is the language too complicated to properly catch them all?)
While probably nice for one-off prototypes, long term maintenance is a concern: this project has over 20 years of history under its belt, with more to come.
That said, python makes its way back into the tree every now and then (typically as small snippets to compute and add hashes to binaries as needed by ARM SoCs). Uncanny, but typically not a big deal.
There are two bigger initiatives proposed that would significantly increase our python footprint:
- Replacing Linux's kconfig with kconfiglib (
https://review.coreboot.org/c/coreboot/+/48679) 2. Using pytest for end-to-end testing utilities ( https://review.coreboot.org/c/coreboot/+/57869)
Compared to the "inject a hash value at a fixed location" scripts, these would probably be here to stay, and sufficiently integrated that everybody will have to deal with them.
People spending time working on python code when it has no chance to land isn't a good use of their time and we should avoid that in the project.
People spending time arguing that python shouldn't be used (to avoid the other outcome) even though the project's culture shifted and is now accepting Python isn't creating a great community for anybody.
To avoid these scenarios, could we possibly nail down the policy on python in coreboot?
Regards, Patrick -- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
What coreboot problems that we have seen in the past are we actually
solving with these rewrites? To be a bit more blunt, what is expected to be improved by writing it in python? Utilities, eg. to analyse blobs? Menu and configuration of builds? Packing blobs for flashing?
I could see some things being harmless, like using python to drive menus and check dependencies. I could potentially see a case that it's preferable to complicated #ifdef logic and for handling of board variants.
On the other hand, I could definitely see python adding new ways that reproducible builds fail, unless you compile a specific python interpreter on the spot. Who knows what subtle differences there might be in python's handling of binary data, especially on different arches and systems with different endianness? Maybe I'm paranoid, or maybe this stuff could be caught well enough in automated testing (and $someone could fix it).
Perhaps user-driven utils like those that extract blobs or examine MSRs to discover interrupt routing on proprietary firmware are a safer middle ground between the two extremes, but those are already written in C aren't they?
Python is already being used, in the "util" directory. util/qualcomm has
~3500 LoC in Python [1] (this is not just a "small snippet" of code). How was it introduced? Was this from a vendor dump? Those seem to occasionally bend the rules of what would be "ideal."
-Matt
On Wed, Sep 29, 2021 at 5:15 PM Ricardo Quesada via coreboot < coreboot@coreboot.org> wrote:
Thanks Patrick for bringing this in.
Regarding Pytest for utils (https://review.coreboot.org/c/coreboot/+/57869 ), what's the recommended way to test the tools that are placed in "util/*" ? I noticed that most of them don't have any kind of end-to-end test (I couldn't find a single one, but perhaps I missed them).
Using PyTest (or any other easy to install / easy to use framework) to create tests for util/* seems natural to me. Although any other test framework can be used. Suggestions?
Besides all the things that have been said in favor or against Python, I'd like to add that: Python is already being used, in the "util" directory. util/qualcomm has ~3500 LoC in Python [1] (this is not just a "small snippet" of code). (compare it with PyTests for elogutil[2]: It has ~160 LoC ).
Thanks.
On Wed, Sep 29, 2021 at 12:09 PM Stefan Reinauer < stefan.k.reinauer@gmail.com> wrote:
On Wed, Sep 29, 2021 at 7:44 AM Jack Rosenthal jrosenth@google.com wrote:
Overall I think introducing Python to the build would provide net benefit, mainly from Kconfiglib, but could also find other good uses in e2e tests like Ricardo was working on. Most people's Linux distros ship with a Python interpreter too, so most developers would be unlikely to notice the extra dependency introduction.
In terms of Kconfiglib, we have a lot to gain by switching away from the Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've forked from the Linux tree and hacked in our own customizations (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into a miniature Python script https://chromium.googlesource.com/chromiumos/platform/depthcharge/+/refs/heads/main/util/autoheader.py that we use to handle our custom header format, and a stable API to work off of so that we can uprev Kconfiglib without needing to change our scripts.
Looking at the current Kconfiglib implementation we would be replacing the C code with 21873 lines of Python code that is now taking the code to deviate from what the Linux kernel is doing. I am having a hard time seeing a "net benefit" in this scenario. Given the mess that Python 2 to Python 3 conversion has been (and still is), this is just inviting a lot of trouble into what has been a fairly stable part of coreboot for the last decade.
In terms of Kconfiglib's stability and track record: I think it has it
covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already without any issues at all.
I am failing to see how anybody involved in coreboot would sign up for and commit to porting 20k lines of Python code to the next version, when it arrives. My indication is that not even the python code that is currently in the tree has been ported to python3 yet.
At a minimum, I think we should consider introducing Python on an optional basis (i.e., the C Kconfig implementation only gets used if a Python interpreter is unavailable), but making it required would be even better.
Come on, we do not need two types of Kconfig parsers in the tree. Let's focus coreboot on actually booting cores, not collections of implementations in different programming languages. There are better projects for that, such as https://github.com/mame/quine-relay
What coreboot problems that we have seen in the past are we actually solving with these rewrites?
Stefan
On Wed, Sep 29, 2021 at 5:39 AM Rao G grao.v80@gmail.com wrote:
Hi Patrick,
That's good to hear, would there be change to "make menuconfig" with kconfiglib
No, we'd make it so that all the "make *config" commands run the Kconfiglib alternatives without any user action required.
The "menuconfig" interface in specific is very similar to the lxdialog-based interface that the C Kconfig uses, except it's a bit more polished and refined feeling.
Thanks Ranga
On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot < coreboot@coreboot.org> wrote:
Hi everybody,
Historically, coreboot avoided depending on python too much (we got rid of an entire python based configuration and build system, for example), with few minor exceptions.
The main reason has been that while python code is quick to slap together, it has demonstrated a penchant for breaking in all kinds of mysterious ways (python 2->3 really was just a slightly bigger instance of what's going on in python all the time), and its users demonstrate a disregard for their fellow developers as demonstrated by endless stack traces on trivial errors (or is the language too complicated to properly catch them all?)
While probably nice for one-off prototypes, long term maintenance is a concern: this project has over 20 years of history under its belt, with more to come.
That said, python makes its way back into the tree every now and then (typically as small snippets to compute and add hashes to binaries as needed by ARM SoCs). Uncanny, but typically not a big deal.
There are two bigger initiatives proposed that would significantly increase our python footprint:
- Replacing Linux's kconfig with kconfiglib (
https://review.coreboot.org/c/coreboot/+/48679) 2. Using pytest for end-to-end testing utilities ( https://review.coreboot.org/c/coreboot/+/57869)
Compared to the "inject a hash value at a fixed location" scripts, these would probably be here to stay, and sufficiently integrated that everybody will have to deal with them.
People spending time working on python code when it has no chance to land isn't a good use of their time and we should avoid that in the project.
People spending time arguing that python shouldn't be used (to avoid the other outcome) even though the project's culture shifted and is now accepting Python isn't creating a great community for anybody.
To avoid these scenarios, could we possibly nail down the policy on python in coreboot?
Regards, Patrick -- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Thanks for bringing this to the list, Patrick.
Stefan Reinauer wrote:
Given the mess that Python 2 to Python 3 conversion has been (and still is), this is just inviting a lot of trouble into what has been a fairly stable part of coreboot for the last decade.
I strongly agree.
On a more general note I find it super sad that distributions purge Python 2 only because it is EOL upstream when it still has many use cases. It's not a problem for me but it does demonstrate how little critical thinking happens within distributions.
So projects must be extra critical about dependencies.
I prefer changing nothing (change != progress) or banning Python for the coreboot build. Utilities (and payloads) not needed for the actual coreboot build can continue to use it.
openssl + dd could probably calculate and inject hashes.
Thanks
//Peter
As a rule of thumb, any project involving a substantial amount of Python always ends up needing a Docker container to build. So I'm in the "no" camp for making Python a dependency, however I think it's fine to keep things as-is where it can be used for helper scripts and utilities for specific purposes such that they aren't critical to building the tree.
On Wed, Sep 29, 2021 at 2:58 AM Patrick Georgi via coreboot < coreboot@coreboot.org> wrote:
That said, python makes its way back into the tree every now and then (typically as small snippets to compute and add hashes to binaries as needed by ARM SoCs). Uncanny, but typically not a big deal.
...
To avoid these scenarios, could we possibly nail down the policy on python in coreboot?
The policy should be simple: The CI system (Jenkins) must be able to build every target in its default configuration.
If we introduce Python as a dependency, then all Python in the tree must be compatible with whatever version Jenkins uses. And if we're going to impose the burden of fixing Python on everyone, then all developers must have the ability to install a compatible version in their OS. Given the experiences many of us in this thread have had and how widely distros vary in Python support, I don't see this as tenable.
Another thing to keep in mind is that we have these sorts of helper scripts from multiple vendors/parties over several years, and we'll likely see more in the future. Pushing them all to use whatever version(s) of Python we decide to build with does not seem realistic.
All that said, I'm fine with Python being used for helper scripts and such as we've done in the past. It gives developers/vendors/etc. freedom to use whatever works for their purposes without imposing a huge burden on everyone else.
As a rule of thumb, any project involving a substantial amount of Python always ends up needing a Docker container to build. So I'm in the "no" camp for making Python a dependency, however I think it's fine to keep things as-is where it can be used for helper scripts and utilities for specific purposes such that they aren't critical to building the tree.
I'm on the same side here. Building the documentation with python sphinx is a pain and I ended up needing docker. The same can be said about edk2/tianocore which also uses a lot of python in critical parts of its build system.
At a minimum, I think we should consider introducing Python on an optional
basis (i.e., the C Kconfig implementation only gets used if a Python interpreter is unavailable), but making it required would be even better.
No. I don't like 'optional' things which do more or less the same as existing code. It's a great way to make the maintenance cost explode. This is true in general (not just python tooling): adding alternative codepaths to do more or less the same as existing code creates a combinatorial explosion. We have (had) 2 different bootblocks codepathds, multiple different cache as ram teardown mechanism, 2 resource different allocators, 2 MP init codepaths, 3 SMM init codepaths, ... Tooling is not the place you want to add new optional codepaths (maybe toolchains can be an exception as different compilers catch different errors which improves code quality). 'Optional' things under the pretext to not break existing things looks like a nice idea at first sight, but it's a very bad strategy in the long run. I'd much rather stick to 1 thing, even if it breaks the master branch, quickly fix things that broke and move on. I don't want to hijack the thread which is about python, but 'optional' python is worse than both no python or mandatory python.
On Thu, Sep 30, 2021 at 7:01 AM David Hendricks david.hendricks@gmail.com wrote:
As a rule of thumb, any project involving a substantial amount of Python always ends up needing a Docker container to build. So I'm in the "no" camp for making Python a dependency, however I think it's fine to keep things as-is where it can be used for helper scripts and utilities for specific purposes such that they aren't critical to building the tree.
On Wed, Sep 29, 2021 at 2:58 AM Patrick Georgi via coreboot < coreboot@coreboot.org> wrote:
That said, python makes its way back into the tree every now and then (typically as small snippets to compute and add hashes to binaries as needed by ARM SoCs). Uncanny, but typically not a big deal.
...
To avoid these scenarios, could we possibly nail down the policy on python in coreboot?
The policy should be simple: The CI system (Jenkins) must be able to build every target in its default configuration.
If we introduce Python as a dependency, then all Python in the tree must be compatible with whatever version Jenkins uses. And if we're going to impose the burden of fixing Python on everyone, then all developers must have the ability to install a compatible version in their OS. Given the experiences many of us in this thread have had and how widely distros vary in Python support, I don't see this as tenable.
Another thing to keep in mind is that we have these sorts of helper scripts from multiple vendors/parties over several years, and we'll likely see more in the future. Pushing them all to use whatever version(s) of Python we decide to build with does not seem realistic.
All that said, I'm fine with Python being used for helper scripts and such as we've done in the past. It gives developers/vendors/etc. freedom to use whatever works for their purposes without imposing a huge burden on everyone else. _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
On Thu, Sep 30, 2021 at 2:27 AM Arthur Heymans arthur@aheymans.xyz wrote:
As a rule of thumb, any project involving a substantial amount of Python
always ends up needing a Docker container to build. So I'm in the "no" camp for making Python a dependency, however I think it's fine to keep things as-is where it can be used for helper scripts and utilities for specific purposes such that they aren't critical to building the tree.
I'm on the same side here. Building the documentation with python sphinx is a pain and I ended up needing docker. The same can be said about edk2/tianocore which also uses a lot of python in critical parts of its build system.
For just Kconfiglib, the requirements are Python 2.7 or Python 3.2+, including no additional library installation (standard libraries only). Using docker in this scenario would be completely ridiculous.
Looking at the current Kconfiglib implementation we would be replacing the
C code with 21873 lines of Python code that is now taking the code to deviate from what the Linux kernel is doing. I am having a hard time seeing a "net benefit" in this scenario. Given the mess that Python 2 to Python 3 conversion has been (and still is), this is just inviting a lot of trouble into what has been a fairly stable part of coreboot for the last decade.
21,873 lines of code is including the tests. Wouldn't it be nice if the C Kconfig implementation even had tests? :P
IMO, any codebase is significantly easier and safer to maintain if there are tests.
In terms of Kconfiglib's stability and track record: I think it has it
covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already without any issues at all.
I am failing to see how anybody involved in coreboot would sign up for and commit to porting 20k lines of Python code to the next version, when it arrives. My indication is that not even the python code that is currently in the tree has been ported to python3 yet.
As for the idea of a Python 4 you seem to have here (or if it does come, repeating the massive language differences we had between 2 and 3), it's unlikely to happen. Guido says that a "Python 4" at the scale Python 3 was is unlikely to happen https://www.techrepublic.com/article/programming-languages-why-python-4-0-will-probably-never-arrive-according-to-its-creator/ .
Am Do., 30. Sept. 2021 um 15:22 Uhr schrieb Jack Rosenthal < jrosenth@google.com>:
IMO, any codebase is significantly easier and safer to maintain if there are tests.
Since we kinda-sorta support SPARK in our toolchain (not for the host though at this time), maybe we should evaluate doing a rewrite in SPARK then: Formal verification beats spot tests which may or may not cover the troublesome situations. ;-) (What I mean is that it's always possible to up the ante, not that we should actually do that. I'd rather get rid of the kconfig language, and even that idea has dubious value in this context.)
As for the idea of a Python 4 you seem to have here (or if it does come,
repeating the massive language differences we had between 2 and 3), it's unlikely to happen. Guido says that a "Python 4" at the scale Python 3 was is unlikely to happen https://www.techrepublic.com/article/programming-languages-why-python-4-0-will-probably-never-arrive-according-to-its-creator/ .
If Python 3->4 is only half as painful as 2->3, this would still be true - and still mean misery for 5+ years. No thanks.
Patrick
On Thu, Sep 30, 2021 at 7:43 AM Patrick Georgi pgeorgi@google.com wrote:
Am Do., 30. Sept. 2021 um 15:22 Uhr schrieb Jack Rosenthal < jrosenth@google.com>:
IMO, any codebase is significantly easier and safer to maintain if there are tests.
Since we kinda-sorta support SPARK in our toolchain (not for the host though at this time), maybe we should evaluate doing a rewrite in SPARK then: Formal verification beats spot tests which may or may not cover the troublesome situations. ;-) (What I mean is that it's always possible to up the ante, not that we should actually do that. I'd rather get rid of the kconfig language, and even that idea has dubious value in this context.)
Yeah. Let's stay practical. Removing the Kconfig language entirely from the tree wouldn't exactly be feasible.
Both the C Kconfig and the Kconfiglib implementations are implementations that exist today, let's evaluate which one of them is better from a maintenance & stability perspective.
As for the idea of a Python 4 you seem to have here (or if it does come,
repeating the massive language differences we had between 2 and 3), it's unlikely to happen. Guido says that a "Python 4" at the scale Python 3 was is unlikely to happen https://www.techrepublic.com/article/programming-languages-why-python-4-0-will-probably-never-arrive-according-to-its-creator/ .
If Python 3->4 is only half as painful as 2->3, this would still be true - and still mean misery for 5+ years. No thanks.
I don't even think even half is in domain... if you give the article to read, Guido basically says we might just stay on Python 3 forever.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Hi Jack,
On 30.09.21 15:22, Jack Rosenthal via coreboot wrote:
On Thu, Sep 30, 2021 at 2:27 AM Arthur Heymans arthur@aheymans.xyz wrote:
As a rule of thumb, any project involving a substantial amount of Python
always ends up needing a Docker container to build. So I'm in the "no" camp for making Python a dependency, however I think it's fine to keep things as-is where it can be used for helper scripts and utilities for specific purposes such that they aren't critical to building the tree.
I'm on the same side here. Building the documentation with python sphinx is a pain and I ended up needing docker. The same can be said about edk2/tianocore which also uses a lot of python in critical parts of its build system.
For just Kconfiglib, the requirements are Python 2.7 or Python 3.2+, including no additional library installation (standard libraries only). Using docker in this scenario would be completely ridiculous.
please keep in mind that this thread tries to discuss a general python requirement. Not Kconfiglib in particular. I suppose nobody doubts that Kconfiglib is one of the better/easier to integrate python projects.
IMHO, we should discuss every added dependency separately. E.g. not just all of python at once but every piece of code that we can't fix in our own repository. I guess, in the case of Kconfiglib that would be:
* python (the interpreter) * std libs (I guess part of the above) * Kconfiglib itself unless we pull it into our repo like we did with Kconfig
If we decided now that we'd allow the above that shouldn't mean that we can add any python related dependency in the future without further discussion.
For any piece of code that we can't fix in our own repository, we should have people that know the respective upstreaming process and can react rather quickly to any issue, IMO.
Looking at the current Kconfiglib implementation we would be replacing the
C code with 21873 lines of Python code that is now taking the code to deviate from what the Linux kernel is doing. I am having a hard time seeing a "net benefit" in this scenario. Given the mess that Python 2 to Python 3 conversion has been (and still is), this is just inviting a lot of trouble into what has been a fairly stable part of coreboot for the last decade.
21,873 lines of code is including the tests. Wouldn't it be nice if the C Kconfig implementation even had tests? :P
Not sure, I never looked close enough at the code to tell if it's bad enough to justify the overhead of tests.
IMO, any codebase is significantly easier and safer to maintain if there are tests.
In my experience that is only true for projects that don't focus enough on code reviews and refactoring. I often see the necessity of tests as lack of review. In some projects tests are even added to provide false confidence that one would need less review. IMO, the opposite is the case. Tests encourage less experienced developers to push patches with- out looking much into the code, because all tests are green, right? In such a scenario the effort to integrate a patch shifts from authors to reviewers, IMO not good.
Nico
On Thu, Sep 30, 2021 at 8:37 AM Nico Huber nico.h@gmx.de wrote:
Hi Jack,
On 30.09.21 15:22, Jack Rosenthal via coreboot wrote:
On Thu, Sep 30, 2021 at 2:27 AM Arthur Heymans arthur@aheymans.xyz
wrote:
As a rule of thumb, any project involving a substantial amount of Python
always ends up needing a Docker container to build. So I'm in the "no"
camp
for making Python a dependency, however I think it's fine to keep
things
as-is where it can be used for helper scripts and utilities for
specific
purposes such that they aren't critical to building the tree.
I'm on the same side here. Building the documentation with python sphinx is a pain and I ended up needing docker. The same can be said about edk2/tianocore which also uses a lot of
python
in critical parts of its build system.
For just Kconfiglib, the requirements are Python 2.7 or Python 3.2+, including no additional library installation (standard libraries only). Using docker in this scenario would be completely ridiculous.
please keep in mind that this thread tries to discuss a general python requirement. Not Kconfiglib in particular. I suppose nobody doubts that Kconfiglib is one of the better/easier to integrate python projects.
IMHO, we should discuss every added dependency separately. E.g. not just all of python at once but every piece of code that we can't fix in our own repository. I guess, in the case of Kconfiglib that would be:
Agreed. Some sort of specification of what the dependency is we are adding is absolutely required.
As an example, we could say "Python 3.6 or newer, no additional libraries required". (this is just an example, the actual version numbers etc we settle on should be discussed).
I bet most distributions are shipping with 3.6+ these days too...
- python (the interpreter)
- std libs (I guess part of the above)
- Kconfiglib itself unless we pull it into our repo like we did with Kconfig
I believe the previous discussion around Kconfiglib was to pull it into the coreboot repo directly so we don't need to make it a dependency for people. We don't need to clone the entire repository into our repos, everything is pretty much self-contained to kconfiglib.py, as well as any UIs we want to enable (e.g., menuconfig.py and guiconfig.py).
Looking at the current Kconfiglib implementation we would be replacing
the
C code with 21873 lines of Python code that is now taking the code to deviate from what the Linux kernel is doing. I am having a hard time
seeing
a "net benefit" in this scenario. Given the mess that Python 2 to
Python 3
conversion has been (and still is), this is just inviting a lot of
trouble
into what has been a fairly stable part of coreboot for the last decade.
21,873 lines of code is including the tests. Wouldn't it be nice if the C Kconfig implementation even had tests? :P
Not sure, I never looked close enough at the code to tell if it's bad enough to justify the overhead of tests.
IMO, any codebase is significantly easier and safer to maintain if there are tests.
In my experience that is only true for projects that don't focus enough on code reviews and refactoring. I often see the necessity of tests as lack of review. In some projects tests are even added to provide false confidence that one would need less review. IMO, the opposite is the case. Tests encourage less experienced developers to push patches with- out looking much into the code, because all tests are green, right? In such a scenario the effort to integrate a patch shifts from authors to reviewers, IMO not good.
Nico
Tests and code review serve two separate functions: tests allowing automatic validation that a change won't break everything, and code review to keep the code-quality high of the code that lands. Obviously, as you point out, there is some overlap, and tests cannot ever be 100% perfect. But enough with the philosophy.
With respect to Kconfig, we (at Google) encountered a lovely build flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal < jrosenth@google.com>:
With respect to Kconfig, we (at Google) encountered a lovely build flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
Let me put the record straight:
- The last kconfig "uprev" hasn't been a simple update the way https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire build system integration to ease maintenance - That issue sprang up because before the kconfig update, we were shipping prebuilt parser files (C code) while now we made bison and flex hard dependencies for our build - Tests covering the C code wouldn't have helped, because the issue wasn't in the C code - The issue we were facing has been an external dependency (namely: the Chromium OS development environment shipping a broken version of bison(1)) - Fixing bison in CrOS wouldn't have helped any because we have to assume that other users come with the same kind of broken bison tool - The fix has been to ship pre-built files again to remove an external dependency
The alternative that we did actually consider was to add support for building bison and flex to util/crossgcc/buildgcc. For three files that's sheer madness, so we went back to precompiled files instead.
In relation to your proposal to adopt kconfiglib: We can run into any kind of external tool demonstrating weird behavior. That's true for bison (as seen here) just as it can be true for arbitrary python versions (even when specified to be "python 3.6+" or whatever): Linux distributions do strange things to their packages, and we're not a Linux-only project, so even official, unchanged, straight-from-the-server python might behave unexpectedly on less well exercised platforms.
The best way to reduce issues on that end is to avoid external dependencies - like bison, like flex... like python. I'd _love_ to avoid the dependency on the host compiler as well, but that's one of those "sheer madness" moments when you support a multitude of operating systems on as many architectures.
Introducing kconfiglib (and through it, a deep reliance on python) just doesn't seem to provide comparable benefits.
Patrick
Speaking as the person who wrote the first few config tools in python, and was happy to see the python dependency gone, I think bringing python back in would be a mistake.
Every single python test framework I've worked with works until there is a problem, and I then find myself having to walk back a python call trace, because what inevitably breaks is the test framework tooling. It's why so many projects are removing python test frameworks.
If you want a good test framework, get the linuxboot fork of facebook's contest github.com/linuxboot/contest, written in Go, in use at scale near you.
It's easy to let the joy of building a build system overwhelm the actual goals of a project. coreboot is not about being a build system. It's easy to fall into the trap of creating an ever more complex system that is more than is needed.
On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot coreboot@coreboot.org wrote:
Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal jrosenth@google.com:
With respect to Kconfig, we (at Google) encountered a lovely build flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
Let me put the record straight:
- The last kconfig "uprev" hasn't been a simple update the way https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire build system integration to ease maintenance
- That issue sprang up because before the kconfig update, we were shipping prebuilt parser files (C code) while now we made bison and flex hard dependencies for our build
- Tests covering the C code wouldn't have helped, because the issue wasn't in the C code
- The issue we were facing has been an external dependency (namely: the Chromium OS development environment shipping a broken version of bison(1))
- Fixing bison in CrOS wouldn't have helped any because we have to assume that other users come with the same kind of broken bison tool
- The fix has been to ship pre-built files again to remove an external dependency
The alternative that we did actually consider was to add support for building bison and flex to util/crossgcc/buildgcc. For three files that's sheer madness, so we went back to precompiled files instead.
In relation to your proposal to adopt kconfiglib: We can run into any kind of external tool demonstrating weird behavior. That's true for bison (as seen here) just as it can be true for arbitrary python versions (even when specified to be "python 3.6+" or whatever): Linux distributions do strange things to their packages, and we're not a Linux-only project, so even official, unchanged, straight-from-the-server python might behave unexpectedly on less well exercised platforms.
The best way to reduce issues on that end is to avoid external dependencies - like bison, like flex... like python. I'd _love_ to avoid the dependency on the host compiler as well, but that's one of those "sheer madness" moments when you support a multitude of operating systems on as many architectures.
Introducing kconfiglib (and through it, a deep reliance on python) just doesn't seem to provide comparable benefits.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Hi all,
Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way to go forward? I just need to run one of the Coreboot's utils (in this case "elogtool"), and make sure the output is the expected one.
Should I use Contest, as suggested by Ron Minnich?
Thanks!
On Thu, Sep 30, 2021 at 10:18 AM ron minnich rminnich@gmail.com wrote:
Speaking as the person who wrote the first few config tools in python, and was happy to see the python dependency gone, I think bringing python back in would be a mistake.
Every single python test framework I've worked with works until there is a problem, and I then find myself having to walk back a python call trace, because what inevitably breaks is the test framework tooling. It's why so many projects are removing python test frameworks.
If you want a good test framework, get the linuxboot fork of facebook's contest github.com/linuxboot/contest, written in Go, in use at scale near you.
It's easy to let the joy of building a build system overwhelm the actual goals of a project. coreboot is not about being a build system. It's easy to fall into the trap of creating an ever more complex system that is more than is needed.
On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot coreboot@coreboot.org wrote:
Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
jrosenth@google.com>:
With respect to Kconfig, we (at Google) encountered a lovely build
flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
Let me put the record straight:
- The last kconfig "uprev" hasn't been a simple update the way
https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire build system integration to ease maintenance
- That issue sprang up because before the kconfig update, we were
shipping prebuilt parser files (C code) while now we made bison and flex hard dependencies for our build
- Tests covering the C code wouldn't have helped, because the issue
wasn't in the C code
- The issue we were facing has been an external dependency (namely: the
Chromium OS development environment shipping a broken version of bison(1))
- Fixing bison in CrOS wouldn't have helped any because we have to
assume that other users come with the same kind of broken bison tool
- The fix has been to ship pre-built files again to remove an external
dependency
The alternative that we did actually consider was to add support for
building bison and flex to util/crossgcc/buildgcc. For three files that's sheer madness, so we went back to precompiled files instead.
In relation to your proposal to adopt kconfiglib: We can run into any
kind of external tool demonstrating weird behavior. That's true for bison (as seen here) just as it can be true for arbitrary python versions (even when specified to be "python 3.6+" or whatever): Linux distributions do strange things to their packages, and we're not a Linux-only project, so even official, unchanged, straight-from-the-server python might behave unexpectedly on less well exercised platforms.
The best way to reduce issues on that end is to avoid external
dependencies - like bison, like flex... like python.
I'd _love_ to avoid the dependency on the host compiler as well, but
that's one of those "sheer madness" moments when you support a multitude of operating systems on as many architectures.
Introducing kconfiglib (and through it, a deep reliance on python) just
doesn't seem to provide comparable benefits.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Hi Ricardo,
sorry for the late response, and that your project fell a bit by the wayside. I guess discussion configuration frameworks is more attractive to this community than testing frameworks (which also explains why we have ~3 config frameworks and only ~1 testing frameworks ;-) )
So yes, testing is something we really need to improve on. I'm not sure if "contest" is the right solution to your particular problem though. The first thing I see when opening up its page is something about mysql, and scrolling down, something about submitting jobs to a server. That seems more like a potential replacement to our Jenkins install (qa.coreboot.org) rather than something to easily write end-to-end tests for our userland tools.
Looking for options, my first instinct was to go for expect(1), but that's really best for interactive uses - might be interesting if we ever grow interactive tools, but so far we manage to stick to clean and tidy CLI. Then I ran into https://github.com/bats-core/bats-core. That seems maintained, reasonably minimal in its dependencies, it emits TAP which is as good as JUnit in terms of Jenkins-integration (so we can have qa.coreboot.org parse the results and give meaningful feedback on review.coreboot.org), and it seems to be fairly widely used for similar things to what you're doing, see https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points to many, many code examples, e.g. https://github.com/docker/machine/blob/master/test/integration/core/core-com... which should cover the basic "call some command, see what it did" scenario quite nicely.
Of course in the end it's all a matter of taste, and that's why I opened the can of worms again that is Python-use in coreboot land. As python hasn't seen a warmer reception than last time, I'd look for alternatives. Maybe Bats could do? Of course, I haven't _actually_ used it yet and if writing tests with that makes you want to scream and run away, we'd have to look for other options (please, don't run away!)
Regards, Patrick
Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada < ricardoq@google.com>:
Hi all,
Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way to go forward? I just need to run one of the Coreboot's utils (in this case "elogtool"), and make sure the output is the expected one.
Should I use Contest, as suggested by Ron Minnich?
Thanks!
On Thu, Sep 30, 2021 at 10:18 AM ron minnich rminnich@gmail.com wrote:
Speaking as the person who wrote the first few config tools in python, and was happy to see the python dependency gone, I think bringing python back in would be a mistake.
Every single python test framework I've worked with works until there is a problem, and I then find myself having to walk back a python call trace, because what inevitably breaks is the test framework tooling. It's why so many projects are removing python test frameworks.
If you want a good test framework, get the linuxboot fork of facebook's contest github.com/linuxboot/contest, written in Go, in use at scale near you.
It's easy to let the joy of building a build system overwhelm the actual goals of a project. coreboot is not about being a build system. It's easy to fall into the trap of creating an ever more complex system that is more than is needed.
On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot coreboot@coreboot.org wrote:
Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
jrosenth@google.com>:
With respect to Kconfig, we (at Google) encountered a lovely build
flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
Let me put the record straight:
- The last kconfig "uprev" hasn't been a simple update the way
https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire build system integration to ease maintenance
- That issue sprang up because before the kconfig update, we were
shipping prebuilt parser files (C code) while now we made bison and flex hard dependencies for our build
- Tests covering the C code wouldn't have helped, because the issue
wasn't in the C code
- The issue we were facing has been an external dependency (namely: the
Chromium OS development environment shipping a broken version of bison(1))
- Fixing bison in CrOS wouldn't have helped any because we have to
assume that other users come with the same kind of broken bison tool
- The fix has been to ship pre-built files again to remove an external
dependency
The alternative that we did actually consider was to add support for
building bison and flex to util/crossgcc/buildgcc. For three files that's sheer madness, so we went back to precompiled files instead.
In relation to your proposal to adopt kconfiglib: We can run into any
kind of external tool demonstrating weird behavior. That's true for bison (as seen here) just as it can be true for arbitrary python versions (even when specified to be "python 3.6+" or whatever): Linux distributions do strange things to their packages, and we're not a Linux-only project, so even official, unchanged, straight-from-the-server python might behave unexpectedly on less well exercised platforms.
The best way to reduce issues on that end is to avoid external
dependencies - like bison, like flex... like python.
I'd _love_ to avoid the dependency on the host compiler as well, but
that's one of those "sheer madness" moments when you support a multitude of operating systems on as many architectures.
Introducing kconfiglib (and through it, a deep reliance on python) just
doesn't seem to provide comparable benefits.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
I set up a raspberry pi as a device under test with contest, and ran the contest controller in a mode that needed no mysql, and it took about 14 minutes.
I'd still recommend looking at contest. I've found it very easy to use and I am very new to it.
On Tue, Oct 12, 2021 at 7:02 AM Patrick Georgi pgeorgi@google.com wrote:
Hi Ricardo,
sorry for the late response, and that your project fell a bit by the wayside. I guess discussion configuration frameworks is more attractive to this community than testing frameworks (which also explains why we have ~3 config frameworks and only ~1 testing frameworks ;-) )
So yes, testing is something we really need to improve on. I'm not sure if "contest" is the right solution to your particular problem though. The first thing I see when opening up its page is something about mysql, and scrolling down, something about submitting jobs to a server. That seems more like a potential replacement to our Jenkins install (qa.coreboot.org) rather than something to easily write end-to-end tests for our userland tools.
Looking for options, my first instinct was to go for expect(1), but that's really best for interactive uses - might be interesting if we ever grow interactive tools, but so far we manage to stick to clean and tidy CLI. Then I ran into https://github.com/bats-core/bats-core. That seems maintained, reasonably minimal in its dependencies, it emits TAP which is as good as JUnit in terms of Jenkins-integration (so we can have qa.coreboot.org parse the results and give meaningful feedback on review.coreboot.org), and it seems to be fairly widely used for similar things to what you're doing, see https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points to many, many code examples, e.g. https://github.com/docker/machine/blob/master/test/integration/core/core-com... which should cover the basic "call some command, see what it did" scenario quite nicely.
Of course in the end it's all a matter of taste, and that's why I opened the can of worms again that is Python-use in coreboot land. As python hasn't seen a warmer reception than last time, I'd look for alternatives. Maybe Bats could do? Of course, I haven't _actually_ used it yet and if writing tests with that makes you want to scream and run away, we'd have to look for other options (please, don't run away!)
Regards, Patrick
Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada ricardoq@google.com:
Hi all,
Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way to go forward? I just need to run one of the Coreboot's utils (in this case "elogtool"), and make sure the output is the expected one.
Should I use Contest, as suggested by Ron Minnich?
Thanks!
On Thu, Sep 30, 2021 at 10:18 AM ron minnich rminnich@gmail.com wrote:
Speaking as the person who wrote the first few config tools in python, and was happy to see the python dependency gone, I think bringing python back in would be a mistake.
Every single python test framework I've worked with works until there is a problem, and I then find myself having to walk back a python call trace, because what inevitably breaks is the test framework tooling. It's why so many projects are removing python test frameworks.
If you want a good test framework, get the linuxboot fork of facebook's contest github.com/linuxboot/contest, written in Go, in use at scale near you.
It's easy to let the joy of building a build system overwhelm the actual goals of a project. coreboot is not about being a build system. It's easy to fall into the trap of creating an ever more complex system that is more than is needed.
On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot coreboot@coreboot.org wrote:
Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal jrosenth@google.com:
With respect to Kconfig, we (at Google) encountered a lovely build flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
Let me put the record straight:
- The last kconfig "uprev" hasn't been a simple update the way https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire build system integration to ease maintenance
- That issue sprang up because before the kconfig update, we were shipping prebuilt parser files (C code) while now we made bison and flex hard dependencies for our build
- Tests covering the C code wouldn't have helped, because the issue wasn't in the C code
- The issue we were facing has been an external dependency (namely: the Chromium OS development environment shipping a broken version of bison(1))
- Fixing bison in CrOS wouldn't have helped any because we have to assume that other users come with the same kind of broken bison tool
- The fix has been to ship pre-built files again to remove an external dependency
The alternative that we did actually consider was to add support for building bison and flex to util/crossgcc/buildgcc. For three files that's sheer madness, so we went back to precompiled files instead.
In relation to your proposal to adopt kconfiglib: We can run into any kind of external tool demonstrating weird behavior. That's true for bison (as seen here) just as it can be true for arbitrary python versions (even when specified to be "python 3.6+" or whatever): Linux distributions do strange things to their packages, and we're not a Linux-only project, so even official, unchanged, straight-from-the-server python might behave unexpectedly on less well exercised platforms.
The best way to reduce issues on that end is to avoid external dependencies - like bison, like flex... like python. I'd _love_ to avoid the dependency on the host compiler as well, but that's one of those "sheer madness" moments when you support a multitude of operating systems on as many architectures.
Introducing kconfiglib (and through it, a deep reliance on python) just doesn't seem to provide comparable benefits.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
-- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Both bats and expect seem problematic for Ricardo's use case ... generating the elog binary format in these tools seems difficult, bash wasn't really meant for generating C structs.
One huge advantage of pytest is that it's fairly industry standard at this point. There's a good number of people who have used it, and there's a large community around it (i.e., finding Q&A on sites like stack overflow is possible). I wasn't able to find a single stack overflow question about ConTest, although I may be looking in the wrong places (looks like they do have a slack group chat).
I believe Ricardo had an open question on the CL if it was OK to submit as an optional test suite for elogtool. I.e., tests don't need to run on CI tools or anything, if there's developers that want to run the tests they can if they have Python.
On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi pgeorgi@google.com wrote:
Hi Ricardo,
sorry for the late response, and that your project fell a bit by the wayside. I guess discussion configuration frameworks is more attractive to this community than testing frameworks (which also explains why we have ~3 config frameworks and only ~1 testing frameworks ;-) )
So yes, testing is something we really need to improve on. I'm not sure if "contest" is the right solution to your particular problem though. The first thing I see when opening up its page is something about mysql, and scrolling down, something about submitting jobs to a server. That seems more like a potential replacement to our Jenkins install (qa.coreboot.org) rather than something to easily write end-to-end tests for our userland tools.
Looking for options, my first instinct was to go for expect(1), but that's really best for interactive uses - might be interesting if we ever grow interactive tools, but so far we manage to stick to clean and tidy CLI. Then I ran into https://github.com/bats-core/bats-core. That seems maintained, reasonably minimal in its dependencies, it emits TAP which is as good as JUnit in terms of Jenkins-integration (so we can have qa.coreboot.org parse the results and give meaningful feedback on review.coreboot.org), and it seems to be fairly widely used for similar things to what you're doing, see https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points to many, many code examples, e.g. https://github.com/docker/machine/blob/master/test/integration/core/core-com... which should cover the basic "call some command, see what it did" scenario quite nicely.
Of course in the end it's all a matter of taste, and that's why I opened the can of worms again that is Python-use in coreboot land. As python hasn't seen a warmer reception than last time, I'd look for alternatives. Maybe Bats could do? Of course, I haven't _actually_ used it yet and if writing tests with that makes you want to scream and run away, we'd have to look for other options (please, don't run away!)
Regards, Patrick
Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada < ricardoq@google.com>:
Hi all,
Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way to go forward? I just need to run one of the Coreboot's utils (in this case "elogtool"), and make sure the output is the expected one.
Should I use Contest, as suggested by Ron Minnich?
Thanks!
On Thu, Sep 30, 2021 at 10:18 AM ron minnich rminnich@gmail.com wrote:
Speaking as the person who wrote the first few config tools in python, and was happy to see the python dependency gone, I think bringing python back in would be a mistake.
Every single python test framework I've worked with works until there is a problem, and I then find myself having to walk back a python call trace, because what inevitably breaks is the test framework tooling. It's why so many projects are removing python test frameworks.
If you want a good test framework, get the linuxboot fork of facebook's contest github.com/linuxboot/contest, written in Go, in use at scale near you.
It's easy to let the joy of building a build system overwhelm the actual goals of a project. coreboot is not about being a build system. It's easy to fall into the trap of creating an ever more complex system that is more than is needed.
On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot coreboot@coreboot.org wrote:
Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
jrosenth@google.com>:
With respect to Kconfig, we (at Google) encountered a lovely build
flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
Let me put the record straight:
- The last kconfig "uprev" hasn't been a simple update the way
https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire build system integration to ease maintenance
- That issue sprang up because before the kconfig update, we were
shipping prebuilt parser files (C code) while now we made bison and flex hard dependencies for our build
- Tests covering the C code wouldn't have helped, because the issue
wasn't in the C code
- The issue we were facing has been an external dependency (namely:
the Chromium OS development environment shipping a broken version of bison(1))
- Fixing bison in CrOS wouldn't have helped any because we have to
assume that other users come with the same kind of broken bison tool
- The fix has been to ship pre-built files again to remove an external
dependency
The alternative that we did actually consider was to add support for
building bison and flex to util/crossgcc/buildgcc. For three files that's sheer madness, so we went back to precompiled files instead.
In relation to your proposal to adopt kconfiglib: We can run into any
kind of external tool demonstrating weird behavior. That's true for bison (as seen here) just as it can be true for arbitrary python versions (even when specified to be "python 3.6+" or whatever): Linux distributions do strange things to their packages, and we're not a Linux-only project, so even official, unchanged, straight-from-the-server python might behave unexpectedly on less well exercised platforms.
The best way to reduce issues on that end is to avoid external
dependencies - like bison, like flex... like python.
I'd _love_ to avoid the dependency on the host compiler as well, but
that's one of those "sheer madness" moments when you support a multitude of operating systems on as many architectures.
Introducing kconfiglib (and through it, a deep reliance on python)
just doesn't seem to provide comparable benefits.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der
Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
-- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
contest is a coming standard. The companies I work with have moved away from python test frameworks. Familiarity did not breed respect for python test frameworks.
On Tue, Oct 12, 2021 at 7:31 AM Jack Rosenthal jrosenth@google.com wrote:
Both bats and expect seem problematic for Ricardo's use case ... generating the elog binary format in these tools seems difficult, bash wasn't really meant for generating C structs.
One huge advantage of pytest is that it's fairly industry standard at this point. There's a good number of people who have used it, and there's a large community around it (i.e., finding Q&A on sites like stack overflow is possible). I wasn't able to find a single stack overflow question about ConTest, although I may be looking in the wrong places (looks like they do have a slack group chat).
I believe Ricardo had an open question on the CL if it was OK to submit as an optional test suite for elogtool. I.e., tests don't need to run on CI tools or anything, if there's developers that want to run the tests they can if they have Python.
On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi pgeorgi@google.com wrote:
Hi Ricardo,
sorry for the late response, and that your project fell a bit by the wayside. I guess discussion configuration frameworks is more attractive to this community than testing frameworks (which also explains why we have ~3 config frameworks and only ~1 testing frameworks ;-) )
So yes, testing is something we really need to improve on. I'm not sure if "contest" is the right solution to your particular problem though. The first thing I see when opening up its page is something about mysql, and scrolling down, something about submitting jobs to a server. That seems more like a potential replacement to our Jenkins install (qa.coreboot.org) rather than something to easily write end-to-end tests for our userland tools.
Looking for options, my first instinct was to go for expect(1), but that's really best for interactive uses - might be interesting if we ever grow interactive tools, but so far we manage to stick to clean and tidy CLI. Then I ran into https://github.com/bats-core/bats-core. That seems maintained, reasonably minimal in its dependencies, it emits TAP which is as good as JUnit in terms of Jenkins-integration (so we can have qa.coreboot.org parse the results and give meaningful feedback on review.coreboot.org), and it seems to be fairly widely used for similar things to what you're doing, see https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points to many, many code examples, e.g. https://github.com/docker/machine/blob/master/test/integration/core/core-com... which should cover the basic "call some command, see what it did" scenario quite nicely.
Of course in the end it's all a matter of taste, and that's why I opened the can of worms again that is Python-use in coreboot land. As python hasn't seen a warmer reception than last time, I'd look for alternatives. Maybe Bats could do? Of course, I haven't _actually_ used it yet and if writing tests with that makes you want to scream and run away, we'd have to look for other options (please, don't run away!)
Regards, Patrick
Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada < ricardoq@google.com>:
Hi all,
Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way to go forward? I just need to run one of the Coreboot's utils (in this case "elogtool"), and make sure the output is the expected one.
Should I use Contest, as suggested by Ron Minnich?
Thanks!
On Thu, Sep 30, 2021 at 10:18 AM ron minnich rminnich@gmail.com wrote:
Speaking as the person who wrote the first few config tools in python, and was happy to see the python dependency gone, I think bringing python back in would be a mistake.
Every single python test framework I've worked with works until there is a problem, and I then find myself having to walk back a python call trace, because what inevitably breaks is the test framework tooling. It's why so many projects are removing python test frameworks.
If you want a good test framework, get the linuxboot fork of facebook's contest github.com/linuxboot/contest, written in Go, in use at scale near you.
It's easy to let the joy of building a build system overwhelm the actual goals of a project. coreboot is not about being a build system. It's easy to fall into the trap of creating an ever more complex system that is more than is needed.
On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot coreboot@coreboot.org wrote:
Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
jrosenth@google.com>:
With respect to Kconfig, we (at Google) encountered a lovely build
flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
Let me put the record straight:
- The last kconfig "uprev" hasn't been a simple update the way
https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire build system integration to ease maintenance
- That issue sprang up because before the kconfig update, we were
shipping prebuilt parser files (C code) while now we made bison and flex hard dependencies for our build
- Tests covering the C code wouldn't have helped, because the issue
wasn't in the C code
- The issue we were facing has been an external dependency (namely:
the Chromium OS development environment shipping a broken version of bison(1))
- Fixing bison in CrOS wouldn't have helped any because we have to
assume that other users come with the same kind of broken bison tool
- The fix has been to ship pre-built files again to remove an
external dependency
The alternative that we did actually consider was to add support for
building bison and flex to util/crossgcc/buildgcc. For three files that's sheer madness, so we went back to precompiled files instead.
In relation to your proposal to adopt kconfiglib: We can run into any
kind of external tool demonstrating weird behavior. That's true for bison (as seen here) just as it can be true for arbitrary python versions (even when specified to be "python 3.6+" or whatever): Linux distributions do strange things to their packages, and we're not a Linux-only project, so even official, unchanged, straight-from-the-server python might behave unexpectedly on less well exercised platforms.
The best way to reduce issues on that end is to avoid external
dependencies - like bison, like flex... like python.
I'd _love_ to avoid the dependency on the host compiler as well, but
that's one of those "sheer madness" moments when you support a multitude of operating systems on as many architectures.
Introducing kconfiglib (and through it, a deep reliance on python)
just doesn't seem to provide comparable benefits.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der
Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
-- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
--
Jack Rosenthal (he/him/his)
Software Engineer - Chrome OS
Google Boulder
jrosenth@google.com
I value feedback from others. Please feel free to contact me directly, or file it anonymously at go/jrosenth-feedback https://goto.google.com/jrosenth-feedback.
Thanks for the answers!
Regarding the end-to-end testing framework, perhaps that's a topic that needs further discussion / consensus (?). E.g: - discuss policy: what parts of coreboot should have end-to-end testing. Should it be forced? recommended? optional? - do some requirement gathering: not only from the firmware but also for the coreboot/utils - Update Jenkins accordingly
In the meantime, would it make sense, as Jack mentioned, to land my change [1] as it is? It is small/simple and it only has ~160 LoC Python. For comparison, other util are using Python: util/qualcomm has ~3500 LoC Python [2] I'll happily migrate + integrate my test once a end-to-end test has been chosen.
Thanks!
[1]: https://review.coreboot.org/c/coreboot/+/57869 [2]: https://github.com/coreboot/coreboot/tree/master/util/qualcomm
On Tue, Oct 12, 2021 at 8:30 AM ron minnich rminnich@gmail.com wrote:
contest is a coming standard. The companies I work with have moved away from python test frameworks. Familiarity did not breed respect for python test frameworks.
On Tue, Oct 12, 2021 at 7:31 AM Jack Rosenthal jrosenth@google.com wrote:
Both bats and expect seem problematic for Ricardo's use case ... generating the elog binary format in these tools seems difficult, bash wasn't really meant for generating C structs.
One huge advantage of pytest is that it's fairly industry standard at this point. There's a good number of people who have used it, and there's a large community around it (i.e., finding Q&A on sites like stack overflow is possible). I wasn't able to find a single stack overflow question about ConTest, although I may be looking in the wrong places (looks like they do have a slack group chat).
I believe Ricardo had an open question on the CL if it was OK to submit as an optional test suite for elogtool. I.e., tests don't need to run on CI tools or anything, if there's developers that want to run the tests they can if they have Python.
On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi pgeorgi@google.com wrote:
Hi Ricardo,
sorry for the late response, and that your project fell a bit by the wayside. I guess discussion configuration frameworks is more attractive to this community than testing frameworks (which also explains why we have ~3 config frameworks and only ~1 testing frameworks ;-) )
So yes, testing is something we really need to improve on. I'm not sure if "contest" is the right solution to your particular problem though. The first thing I see when opening up its page is something about mysql, and scrolling down, something about submitting jobs to a server. That seems more like a potential replacement to our Jenkins install ( qa.coreboot.org) rather than something to easily write end-to-end tests for our userland tools.
Looking for options, my first instinct was to go for expect(1), but that's really best for interactive uses - might be interesting if we ever grow interactive tools, but so far we manage to stick to clean and tidy CLI. Then I ran into https://github.com/bats-core/bats-core. That seems maintained, reasonably minimal in its dependencies, it emits TAP which is as good as JUnit in terms of Jenkins-integration (so we can have qa.coreboot.org parse the results and give meaningful feedback on review.coreboot.org), and it seems to be fairly widely used for similar things to what you're doing, see https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points to many, many code examples, e.g. https://github.com/docker/machine/blob/master/test/integration/core/core-com... which should cover the basic "call some command, see what it did" scenario quite nicely.
Of course in the end it's all a matter of taste, and that's why I opened the can of worms again that is Python-use in coreboot land. As python hasn't seen a warmer reception than last time, I'd look for alternatives. Maybe Bats could do? Of course, I haven't _actually_ used it yet and if writing tests with that makes you want to scream and run away, we'd have to look for other options (please, don't run away!)
Regards, Patrick
Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada < ricardoq@google.com>:
Hi all,
Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way to go forward? I just need to run one of the Coreboot's utils (in this case "elogtool"), and make sure the output is the expected one.
Should I use Contest, as suggested by Ron Minnich?
Thanks!
On Thu, Sep 30, 2021 at 10:18 AM ron minnich rminnich@gmail.com wrote:
Speaking as the person who wrote the first few config tools in python, and was happy to see the python dependency gone, I think bringing python back in would be a mistake.
Every single python test framework I've worked with works until there is a problem, and I then find myself having to walk back a python call trace, because what inevitably breaks is the test framework tooling. It's why so many projects are removing python test frameworks.
If you want a good test framework, get the linuxboot fork of facebook's contest github.com/linuxboot/contest, written in Go, in use at scale near you.
It's easy to let the joy of building a build system overwhelm the actual goals of a project. coreboot is not about being a build system. It's easy to fall into the trap of creating an ever more complex system that is more than is needed.
On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot coreboot@coreboot.org wrote:
Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
jrosenth@google.com>:
> > With respect to Kconfig, we (at Google) encountered a lovely build
flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
Let me put the record straight:
- The last kconfig "uprev" hasn't been a simple update the way
https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire build system integration to ease maintenance
- That issue sprang up because before the kconfig update, we were
shipping prebuilt parser files (C code) while now we made bison and flex hard dependencies for our build
- Tests covering the C code wouldn't have helped, because the issue
wasn't in the C code
- The issue we were facing has been an external dependency (namely:
the Chromium OS development environment shipping a broken version of bison(1))
- Fixing bison in CrOS wouldn't have helped any because we have to
assume that other users come with the same kind of broken bison tool
- The fix has been to ship pre-built files again to remove an
external dependency
The alternative that we did actually consider was to add support for
building bison and flex to util/crossgcc/buildgcc. For three files that's sheer madness, so we went back to precompiled files instead.
In relation to your proposal to adopt kconfiglib: We can run into
any kind of external tool demonstrating weird behavior. That's true for bison (as seen here) just as it can be true for arbitrary python versions (even when specified to be "python 3.6+" or whatever): Linux distributions do strange things to their packages, and we're not a Linux-only project, so even official, unchanged, straight-from-the-server python might behave unexpectedly on less well exercised platforms.
The best way to reduce issues on that end is to avoid external
dependencies - like bison, like flex... like python.
I'd _love_ to avoid the dependency on the host compiler as well, but
that's one of those "sheer madness" moments when you support a multitude of operating systems on as many architectures.
Introducing kconfiglib (and through it, a deep reliance on python)
just doesn't seem to provide comparable benefits.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der
Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
-- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
--
Jack Rosenthal (he/him/his)
Software Engineer - Chrome OS
Google Boulder
jrosenth@google.com
I value feedback from others. Please feel free to contact me directly, or file it anonymously at go/jrosenth-feedback https://goto.google.com/jrosenth-feedback.
Am Fr., 15. Okt. 2021 um 19:50 Uhr schrieb Ricardo Quesada < ricardoq@google.com>:
In the meantime, would it make sense, as Jack mentioned, to land my change [1] as it is? It is small/simple and it only has ~160 LoC Python. For comparison, other util are using Python: util/qualcomm has ~3500 LoC Python [2] I'll happily migrate + integrate my test once a end-to-end test has been chosen.
I'm fine getting it in for now, but it won't see testing every build.
Will wait for other people to chime in, though. If I don't see pushback against getting this in as-is (with the understanding that it might be out quickly again once we decide on a plan of action) by next Thursday at noon in Germany (UTC+0200), I'll see that I submit it.
Patrick
I would rather we not start depending on pytest. Just my take.There's a difference between a utility for one chipset and global infrastructure, and pytest could become the latter.
There's a pretty big testing activity starting up in OCP, involving many companies, and python is not something we will be using, for all the usual reasons. Some of those companies had python test infra and they are moving on. Contest is easy to set up, easy to run, it's getting contributors. I understand it's a commitment of a day or so to figure out, but it's worth it in our experience. It's just not that hard.
I believe starting down the python path is a bad start, and I'd rather not make it. I like to know programs have errors in the source code before I run them. I spend too much time debugging python call traces, having it work back to a missing module, to have confidence in it. Just recently I ran a program which has worked for months, and got this: Traceback (most recent call last): File "/Users/rminnich/projects/python-dlipower/scripts/dlipower", line 30, in <module> from dlipower import PowerSwitch ModuleNotFoundError: No module named 'dlipower'
What changed? nothing. Why did it break? I don't know. This sucks, but it super sucks when it happens on a batch test system and you're on a deadline. no no no no no.
Again, speaking as the person who wrote the first three device tree parsers in python, I was not as opposed to python for infra back then, but I am now. Python is always a headache every time I have to deal with it for things like test.
I realize "it's only 160 lines", now; but that's how these things always start. They don't end well.
On Fri, Oct 15, 2021 at 11:06 AM Patrick Georgi pgeorgi@google.com wrote:
Am Fr., 15. Okt. 2021 um 19:50 Uhr schrieb Ricardo Quesada ricardoq@google.com:
In the meantime, would it make sense, as Jack mentioned, to land my change [1] as it is? It is small/simple and it only has ~160 LoC Python. For comparison, other util are using Python: util/qualcomm has ~3500 LoC Python [2] I'll happily migrate + integrate my test once a end-to-end test has been chosen.
I'm fine getting it in for now, but it won't see testing every build.
Will wait for other people to chime in, though. If I don't see pushback against getting this in as-is (with the understanding that it might be out quickly again once we decide on a plan of action) by next Thursday at noon in Germany (UTC+0200), I'll see that I submit it.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Am Sa., 16. Okt. 2021 um 02:40 Uhr schrieb ron minnich rminnich@gmail.com:
Contest is easy to set up, easy to run, it's getting contributors. I understand it's a commitment of a day or so to figure out, but it's worth it in our experience. It's just not that hard.
I believe starting down the python path is a bad start, and I'd rather not make it.
It is, but that's the code proposed on the repo. I asked you about using Contest in a setting that comes without client/server architecture, SQL database and a dedicated system under test. Any news on that end?
I realize "it's only 160 lines", now; but that's how these things
always start. They don't end well.
It's not a license for adding more of the same.
I consider it: 1. A signal that we care for testing our tools (as in: we like contributions that improve our test coverage). 2. A signal of encouragement towards Ricardo who haplessly ran into the trap of coreboot discouraging (ahem) python (we should document that!) while trying to improve the project's general posture.
As soon as you bring up an alternative path that's acceptable to the project at large, Ricardo offered to rewrite this specific test in whatever we'll use officially (I suppose there's a caveat of it being "within reason": don't get out your brainf*ck-based e2e testing framework!), and I hereby offer to be the rewriter-of-last-resort in case Ricardo is gone by then.
As is, Content looks like a solution for a _different_ problem. As soon as we can clear up that confusion, I'm all for using it, and the faster we get that done, the faster this python code is removed again.
Therefore I guess one could say that I also consider this change (and my proposal of merging it):
3. A rather blunt tool to get you to resolve the open questions regarding Contest ;-)
Patrick
Still working on getting the setup, but 9elements reports: "We have it running as CI System attached to a Github repo and do build and boot testing on Qemu und real hardware with it."
On Sat, Oct 16, 2021 at 1:39 AM Patrick Georgi pgeorgi@google.com wrote:
Am Sa., 16. Okt. 2021 um 02:40 Uhr schrieb ron minnich rminnich@gmail.com:
Contest is easy to set up, easy to run, it's getting contributors. I understand it's a commitment of a day or so to figure out, but it's worth it in our experience. It's just not that hard.
I believe starting down the python path is a bad start, and I'd rather not make it.
It is, but that's the code proposed on the repo. I asked you about using Contest in a setting that comes without client/server architecture, SQL database and a dedicated system under test. Any news on that end?
I realize "it's only 160 lines", now; but that's how these things always start. They don't end well.
It's not a license for adding more of the same.
I consider it:
- A signal that we care for testing our tools (as in: we like contributions that improve our test coverage).
- A signal of encouragement towards Ricardo who haplessly ran into the trap of coreboot discouraging (ahem) python (we should document that!) while trying to improve the project's general posture.
As soon as you bring up an alternative path that's acceptable to the project at large, Ricardo offered to rewrite this specific test in whatever we'll use officially (I suppose there's a caveat of it being "within reason": don't get out your brainf*ck-based e2e testing framework!), and I hereby offer to be the rewriter-of-last-resort in case Ricardo is gone by then.
As is, Content looks like a solution for a _different_ problem. As soon as we can clear up that confusion, I'm all for using it, and the faster we get that done, the faster this python code is removed again.
Therefore I guess one could say that I also consider this change (and my proposal of merging it):
- A rather blunt tool to get you to resolve the open questions regarding Contest ;-)
Patrick
-
Arthur Heymans -
David Hendricks -
Jack Rosenthal -
Matt B -
Nico Huber -
Patrick Georgi -
Peter Stuge -
Rao G -
Ricardo Quesada -
ron minnich -
Stefan Reinauer