Both bats and expect seem problematic for Ricardo's use case ... generating the elog binary format in these tools seems difficult, bash wasn't really meant for generating C structs.

One huge advantage of pytest is that it's fairly industry standard at this point. There's a good number of people who have used it, and there's a large community around it (i.e., finding Q&A on sites like stack overflow is possible). I wasn't able to find a single stack overflow question about ConTest, although I may be looking in the wrong places (looks like they do have a slack group chat).

I believe Ricardo had an open question on the CL if it was OK to submit as an optional test suite for elogtool. I.e., tests don't need to run on CI tools or anything, if there's developers that want to run the tests they can if they have Python.



On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi <pgeorgi@google.com> wrote:
Hi Ricardo,

sorry for the late response, and that your project fell a bit by the wayside. I guess discussion configuration frameworks is more attractive to this community than testing frameworks (which also explains why we have ~3 config frameworks and only ~1 testing frameworks ;-) )

So yes, testing is something we really need to improve on. I'm not sure if "contest" is the right solution to your particular problem though. The first thing I see when opening up its page is something about mysql, and scrolling down, something about submitting jobs to a server. That seems more like a potential replacement to our Jenkins install (qa.coreboot.org) rather than something to easily write end-to-end tests for our userland tools.

Looking for options, my first instinct was to go for expect(1), but that's really best for interactive uses - might be interesting if we ever grow interactive tools, but so far we manage to stick to clean and tidy CLI. Then I ran into https://github.com/bats-core/bats-core. That seems maintained, reasonably minimal in its dependencies, it emits TAP which is as good as JUnit in terms of Jenkins-integration (so we can have qa.coreboot.org parse the results and give meaningful feedback on review.coreboot.org), and it seems to be fairly widely used for similar things to what you're doing, see https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points to many, many code examples, e.g. https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats which should cover the basic "call some command, see what it did" scenario quite nicely.

Of course in the end it's all a matter of taste, and that's why I opened the can of worms again that is Python-use in coreboot land. As python hasn't seen a warmer reception than last time, I'd look for alternatives.
Maybe Bats could do? Of course, I haven't _actually_ used it yet and if writing tests with that makes you want to scream and run away, we'd have to look for other options (please, don't run away!)


Regards,
Patrick

Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada <ricardoq@google.com>:
Hi all,

Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way to go forward?
I just need to run one of the Coreboot's utils (in this case "elogtool"), and make sure the output is the expected one.

Should I use Contest, as suggested by Ron Minnich?

Thanks!



On Thu, Sep 30, 2021 at 10:18 AM ron minnich <rminnich@gmail.com> wrote:
Speaking as the person who wrote the first few config tools in python,
and was happy to see the python dependency gone, I think bringing
python back in would be a mistake.

Every single python test framework I've worked with works until there
is a problem, and I then find myself having to walk back a python call
trace, because what inevitably breaks is the test framework tooling.
It's why so many projects are removing python test frameworks.

If you want a good test framework, get the linuxboot fork of
facebook's contest github.com/linuxboot/contest, written in Go, in use
at scale near you.

It's easy to let the joy of building a build system overwhelm the
actual goals of a project. coreboot is not about being a build system.
It's easy to fall into the trap of creating an ever more complex
system that is more than is needed.

On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
<coreboot@coreboot.org> wrote:
>
> Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <jrosenth@google.com>:
>>
>> With respect to Kconfig, we (at Google) encountered a lovely build flake after the Kconfig uprev last month in the coreboot tree that took a couple of weeks to sort out and resolve. Some sort of automated validation that the code is working could have possibly helped. Of course, the C implementation of Kconfig has no tests at all. Some tests is better than nothing.
>
>
> Let me put the record straight:
>
> - The last kconfig "uprev" hasn't been a simple update the way https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire build system integration to ease maintenance
> - That issue sprang up because before the kconfig update, we were shipping prebuilt parser files (C code) while now we made bison and flex hard dependencies for our build
> - Tests covering the C code wouldn't have helped, because the issue wasn't in the C code
> - The issue we were facing has been an external dependency (namely: the Chromium OS development environment shipping a broken version of bison(1))
> - Fixing bison in CrOS wouldn't have helped any because we have to assume that other users come with the same kind of broken bison tool
> - The fix has been to ship pre-built files again to remove an external dependency
>
> The alternative that we did actually consider was to add support for building bison and flex to util/crossgcc/buildgcc. For three files that's sheer madness, so we went back to precompiled files instead.
>
> In relation to your proposal to adopt kconfiglib: We can run into any kind of external tool demonstrating weird behavior. That's true for bison (as seen here) just as it can be true for arbitrary python versions (even when specified to be "python 3.6+" or whatever): Linux distributions do strange things to their packages, and we're not a Linux-only project, so even official, unchanged, straight-from-the-server python might behave unexpectedly on less well exercised platforms.
>
> The best way to reduce issues on that end is to avoid external dependencies - like bison, like flex... like python.
> I'd _love_ to avoid the dependency on the host compiler as well, but that's one of those "sheer madness" moments when you support a multitude of operating systems on as many architectures.
>
> Introducing kconfiglib (and through it, a deep reliance on python) just doesn't seem to provide comparable benefits.
>
>
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
> _______________________________________________
> coreboot mailing list -- coreboot@coreboot.org
> To unsubscribe send an email to coreboot-leave@coreboot.org


--
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado


--

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrosenth@google.com

I value feedback from others. Please feel free to contact me directly, or file it anonymously at go/jrosenth-feedback.