contest is a coming standard. The companies I work with have moved away from python test frameworks. Familiarity did not breed respect for python test frameworks.
On Tue, Oct 12, 2021 at 7:31 AM Jack Rosenthal jrosenth@google.com wrote:
Both bats and expect seem problematic for Ricardo's use case ... generating the elog binary format in these tools seems difficult, bash wasn't really meant for generating C structs.
One huge advantage of pytest is that it's fairly industry standard at this point. There's a good number of people who have used it, and there's a large community around it (i.e., finding Q&A on sites like stack overflow is possible). I wasn't able to find a single stack overflow question about ConTest, although I may be looking in the wrong places (looks like they do have a slack group chat).
I believe Ricardo had an open question on the CL if it was OK to submit as an optional test suite for elogtool. I.e., tests don't need to run on CI tools or anything, if there's developers that want to run the tests they can if they have Python.
On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi pgeorgi@google.com wrote:
Hi Ricardo,
sorry for the late response, and that your project fell a bit by the wayside. I guess discussion configuration frameworks is more attractive to this community than testing frameworks (which also explains why we have ~3 config frameworks and only ~1 testing frameworks ;-) )
So yes, testing is something we really need to improve on. I'm not sure if "contest" is the right solution to your particular problem though. The first thing I see when opening up its page is something about mysql, and scrolling down, something about submitting jobs to a server. That seems more like a potential replacement to our Jenkins install (qa.coreboot.org) rather than something to easily write end-to-end tests for our userland tools.
Looking for options, my first instinct was to go for expect(1), but that's really best for interactive uses - might be interesting if we ever grow interactive tools, but so far we manage to stick to clean and tidy CLI. Then I ran into https://github.com/bats-core/bats-core. That seems maintained, reasonably minimal in its dependencies, it emits TAP which is as good as JUnit in terms of Jenkins-integration (so we can have qa.coreboot.org parse the results and give meaningful feedback on review.coreboot.org), and it seems to be fairly widely used for similar things to what you're doing, see https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points to many, many code examples, e.g. https://github.com/docker/machine/blob/master/test/integration/core/core-com... which should cover the basic "call some command, see what it did" scenario quite nicely.
Of course in the end it's all a matter of taste, and that's why I opened the can of worms again that is Python-use in coreboot land. As python hasn't seen a warmer reception than last time, I'd look for alternatives. Maybe Bats could do? Of course, I haven't _actually_ used it yet and if writing tests with that makes you want to scream and run away, we'd have to look for other options (please, don't run away!)
Regards, Patrick
Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada < ricardoq@google.com>:
Hi all,
Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way to go forward? I just need to run one of the Coreboot's utils (in this case "elogtool"), and make sure the output is the expected one.
Should I use Contest, as suggested by Ron Minnich?
Thanks!
On Thu, Sep 30, 2021 at 10:18 AM ron minnich rminnich@gmail.com wrote:
Speaking as the person who wrote the first few config tools in python, and was happy to see the python dependency gone, I think bringing python back in would be a mistake.
Every single python test framework I've worked with works until there is a problem, and I then find myself having to walk back a python call trace, because what inevitably breaks is the test framework tooling. It's why so many projects are removing python test frameworks.
If you want a good test framework, get the linuxboot fork of facebook's contest github.com/linuxboot/contest, written in Go, in use at scale near you.
It's easy to let the joy of building a build system overwhelm the actual goals of a project. coreboot is not about being a build system. It's easy to fall into the trap of creating an ever more complex system that is more than is needed.
On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot coreboot@coreboot.org wrote:
Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
jrosenth@google.com>:
With respect to Kconfig, we (at Google) encountered a lovely build
flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
Let me put the record straight:
- The last kconfig "uprev" hasn't been a simple update the way
https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire build system integration to ease maintenance
- That issue sprang up because before the kconfig update, we were
shipping prebuilt parser files (C code) while now we made bison and flex hard dependencies for our build
- Tests covering the C code wouldn't have helped, because the issue
wasn't in the C code
- The issue we were facing has been an external dependency (namely:
the Chromium OS development environment shipping a broken version of bison(1))
- Fixing bison in CrOS wouldn't have helped any because we have to
assume that other users come with the same kind of broken bison tool
- The fix has been to ship pre-built files again to remove an
external dependency
The alternative that we did actually consider was to add support for
building bison and flex to util/crossgcc/buildgcc. For three files that's sheer madness, so we went back to precompiled files instead.
In relation to your proposal to adopt kconfiglib: We can run into any
kind of external tool demonstrating weird behavior. That's true for bison (as seen here) just as it can be true for arbitrary python versions (even when specified to be "python 3.6+" or whatever): Linux distributions do strange things to their packages, and we're not a Linux-only project, so even official, unchanged, straight-from-the-server python might behave unexpectedly on less well exercised platforms.
The best way to reduce issues on that end is to avoid external
dependencies - like bison, like flex... like python.
I'd _love_ to avoid the dependency on the host compiler as well, but
that's one of those "sheer madness" moments when you support a multitude of operating systems on as many architectures.
Introducing kconfiglib (and through it, a deep reliance on python)
just doesn't seem to provide comparable benefits.
Patrick
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der
Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
-- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
--
Jack Rosenthal (he/him/his)
Software Engineer - Chrome OS
Google Boulder
jrosenth@google.com
I value feedback from others. Please feel free to contact me directly, or file it anonymously at go/jrosenth-feedback https://goto.google.com/jrosenth-feedback.