Hi all,
I haven't tracked all recent development closely, so here's a quick question to everyone currently working on the master branch:
How is the current state wrt. to stability and a potential release from the master branch? Also, how does it look compared to the state half a year ago? I know some issues were fixed, but also new ones came up. What do people think, are we closer to / farther away from a release now?
Nico
Hello,
A quick answer first: We are closer to the release, and the state of the master is better than a half year ago.
More details. In general, the progress is visible here https://ticket.coreboot.org/issues/353
There is one issue that came up around May https://ticket.coreboot.org/issues/390 . This one is new. It does not prevent from using any functionality, and also if the user does not specify `--progress` they won’t even notice.
On the other hand, several issues have been fixed. Dummyflasher is fixed, i2c programmers got allow_brick param and were improved.
Manpages are added to all programmers that were missing them (I think, to all programmers, hopefully I am not forgetting anything).
Lots of code improvements: for example we have way less global state than it used to be 1/2yr ago, and we have more unit tests of all types.
Meson build file got a great upgrade. For this change I can mention: it was merged into chromium flashrom ~2weeks ago and no issues observed so far. Also there is now Documentation/ for how to build with meson.
Surely there were more improvements. I am just listing what is on the top of my mind.
If you ask me to pick what’s very important to fix I would say https://ticket.coreboot.org/issues/370 I remember several patches on that, from different people. Potentially, the bug is [technically] fixed, but we need to coordinate who is rebasing on the top of whom, and complete the review.
Hi all,
On 14.10.22 03:53, Anastasia Klimchuk wrote:
hmmm, I may not have made myself clear with my question. I didn't mean to ask for an overall state but wrt. stability (iow. absence of code flaws) in particular. And if/how we can trust that there are no or few enough flaws.
More details. In general, the progress is visible here https://ticket.coreboot.org/issues/353
Yes, this is the progress of tickets. But how to put this... I know the state as far as we measure it. What I'm concerned about is what we don't measure.
Maybe this helps to understand the idea of the release process better: In the Development Guidelines[1] it says
"Branching for a new release can happen at any point in time when a commit (branch point) on master seems to be in good shape and was reasonably tested after previous invasive changes."
The last part tries to protect us from very new regressions / issues that nobody found (measured) yet. And this was written in a time when flashrom builds from the master branch were much more tested. We could tell people that it's safe to use the master branch (which it isn't anymore). And even the people who used releases implicitly tested a state of the code for us that was much closer to master.
Also, what led to the long list in #353 was a development process that created issues faster than they were fixed. Many of the issues were only made visible by a re-review. Can we say that the ratio of creating and fixing issues is more balanced now? How can we trust that if we'd re-review again, we wouldn't end up with an even longer list?
Can the latter be proved? I'd say yes, but it seems like more effort then re-writing the whole patch. I never fully reviewed the patch but in the few places I looked into I saw wrong calculations and overflows. And those run independently from the `--progress` switch. And please remember we are talking about C code. To say that it doesn't affect existing functionality would require a proof of defined behavior. Because if there is undefined behavior, flashrom could just crash.
Also worth to remember: flashrom often runs with root privileges. So every coding mistake is a security issue unless we can prove that it isn't.
The former is in the category of "invasive changes" that I'd like to avoid before a release. I see a lot of refactorings in the name of less global state. IMO that's a nice long-term goal, but shouldn't stall the project. Just very recently, there was a patch train that unnecessarily introduced a lot of NULL pointers (I commented how to do it without and that we'd have to be careful otherwise). Things were merged (without being careful), reverted, fixed up (I warned about NULL pointers again), merged again (again without looking for NULL pointers left), fixed up again. How can we know now that it's settled? How to trust such a development process?
(This case might actually be why I started to wonder how the measured progress relates to the actual progress on the master branch.)
The story is much more complex. Every time somebody looks into it, more even older issues pop up. I'd say revert to the state of the last release, and then after a new release, fix remaining issues and only then add new things and the refactorings again. IMO this driver was patched to a dead end, or at least a state where the easiest way for- ward is to go a few steps back first. Last time Edward drew attention to it, I really tried to find a solution. But even comparing the state right after the problematic patch to the current state was too frus- trating: Due to all the refactorings I couldn't see (within reasonable time) what changed in terms of functionality.
Maybe we should have a rule for this? "If there are any known issues within a file or area of the code, no refactoring must take place until they are fixed."?
Nico
[1] https://www.flashrom.org/Development_Guidelines#Release_branches_(e.g._1.0.x)
Nico what was the goal why you started this thread?
I thought for a moment, you have a *quick* question. So I answered. Now you seem to be unhappy to learn that we are closer to the release than 1/2 yr ago. You are saying "But I don't believe you", what was the point of asking then?
Hi Anastasia,
On 16.10.22 23:41, Anastasia Klimchuk wrote:
Nico what was the goal why you started this thread?
I wanted to evaluate how immediate a release is. Also, maybe subconsciously (on my end), to raise awareness among developers that we want to do releases.
I learned some new things between my first two emails, overall that has changed my intentions to raising awareness.
It seems hard to assess to me. I want to make sure that we don't miss anything. For instance, the progress in the tickets: A lot have been closed. But some of them look like they were low-hanging fruits, i.e. easy to fix. While bigger issues remain and at least one new issue popped up, as you mentioned. I find it very hard to measure.
You are saying "But I don't believe you", what was the point of asking then?
I did not say that nor did I intend to make it look like this. I still feel misunderstood and that we are talking past each other. I didn't mean to ask "how many tickets are open" or "what was merged". These are things that are easily visible. I guess what I really want to know is "did the regression rate decline?", "does it look like the review process changed enough so that it could?".
Nico
PS. Anastasia, there is a lot of "you, you, you" in your email. This can get very emotional. I don't see a need to get personal nor to fight.
Hi all,
Perhaps a way forward would be to tag a rc release, ask distributors to package and test it (e.g. pushing to Fedora Rawhide, but not Fedora 36), and then push the actual official release a week or two later?
Certainly aiming to do releases monthly is much healthier than doing releases every few years. I think it's much more achievable to set the expectation to the end user "sorry for the regression! it'll be fixed in your distro in ~3 weeks, in the meantime use the previous release" than trying to squash every bug and add all the features before tagging a mythical beautiful bug-free "feature complete" release.
Richard.
On Mon, 17 Oct 2022 at 10:19, Nico Huber nico.h@gmx.de wrote:
I very much like Richard's pragmatic approach here.
It would be my view that we should re-evaluate branch critical bugs (sb600spi map issue + build system stuff are the two the most prominent in my mind at the moment) and just cut a release branch, stabilise that with some cherry-picks of any residue items. Forge forwards from that with a more regular cadence exactly how Richard suggests. If some critical issue comes up we can do a point release.
Kindest regards, Edward.
On Mon, 17 Oct 2022 at 20:27, Richard Hughes hughsient@gmail.com wrote:
If it helps, a monthly release seems to be the sweet spot for fwupd, from a "writing release notes" and "getting fixes into users hands" point of view. There's no point having a "no regressions" rule as this is software, and even the most benign of changes can have some unknown side effect -- it's much better to just make releases "cheap" and do lots of them. In fwupd the main branch is always "releasable" so if we have a high-priority fix or security issue we just merge, write release notes, tag, push outside the usual release cadence.
Richard
On Mon, 17 Oct 2022 at 12:24, Edward O'Callaghan quasisec@google.com wrote:
Hi all (again),
We (Angel) just took the time to reorder the previous replies so that we can respond to them in-line.
On Mon, Oct 17, 2022 at 11:34 AM Richard Hughes hughsient@gmail.com wrote:
We (Angel) agree that striving for a "feature complete" release is not good: in a way, it resembles maladaptive daydreaming. Having a faster-paced release cadence is a good idea, too. However, flashrom isn't the kind of project where regressions can be taken lightly, especially those which can render users' hardware unusable. It's important to not forget that flashrom's userbase is extremely wide, with users that just want to update their BIOS without having to install Windows, experienced coreboot developers, and anything in between and even out-of-bounds. Some people even use flashrom on DOS: if there's a bug that corrupts the flash chip contents, it may not be possible to get a different flashrom version without rebooting the system (in other words, they are doomed).
Sounds good, thank you. Not sure which are the most pressing concerns, but definitely not everything currently listed in https://ticket.coreboot.org/issues/353 is required to make a release. Another option would be to make a new branch for the next (probably v1.3, but it could be something else like v1.2.1 too?) release: one could base the next release off the master branch with the potentially broken parts removed, or one could base it off the v1.2 branch and cherry-pick the most important changes from master. At first, we (Angel) were unsure which approach would be better, but the second approach sounds more appealing the more we think about it: it would allow us (the flashrom community) to calmly re-review (and possibly re-design) all the changes since the v1.2 release. It will be a *lot* of work, but it will allow us (the flashrom community) to address any concerns about review thoroughness (for lack of a better word) and settle them down once and for all.
This is what coreboot strives for, too: coreboot's master branch is supposed to work at all times, and releases are just "glorified" commits from the master branch. The release cadence for coreboot is about 3 months (it used to be 6 months), which should also work for flashrom. However, flashrom's master branch isn't "releasable" at the moment even though it should be.
Richard
Best regards, Angel
Hi Richard,
thanks for your thoughts. This has woken some more ideas in my mind, how to express current flashrom troubles.
On 17.10.22 11:26, Richard Hughes wrote:
I very much agree to a shorter release cycle. Maybe not monthly, but every 3 months would be worth a shot, IMO. When we still did releases, it was about 1 per year. But back then we had little time to spare for releases. Regressions can be much more severe though, and with the cur- rent development the story might look like: "sorry for the regression! you may have to acquire a hot-air soldering station now to fix your PC. when you are done, please try to help us to debug the issue. although, even if you do, we may not fix the issue in a release if there's a chance that something would regress for a flashrom stakeholder.". So to say, we have other problems that need to be addressed before we can dream about a release cycle.
The problems seem to run very deep. It's not easy to put into words. I'll try to sketch one thought briefly as I don't have much time: One issue is a denial of service on many levels. One level (kind of my DoS) is the bug fixing. Because I've seen it burning out people (including myself) when one tries to clean up behind others, I argue against fixing bugs that other people introduced. This has went so far that I even stopped pushing reverts for others. Because last time I pushed a revert of a three-line patch with three flawed lines, people made a drama of it.
Nico
On Tue, 18 Oct 2022 at 14:22, Nico Huber nico.h@gmx.de wrote:
I don't think that's true. The only people using flashrom at scale is Google, and they're quite certainly testing every flashrom version with every firmware version with every hardware version before deploying anything. In the community the only people _really_ using flashrom are the ones with SPI clips and screws already removed, if we're being completely honest. Normal people don't use flashrom in production in any appreciable number.
But flashrom isn't your baby, it's not your responsibility. Advice yes, being condescending no.
Richard.
Hi Richard,
Maybe it's not intentional, but your email sounds rather unpleasant and very confusing. What would you like to achieve?
Also, out of curiosity, did you see https://mail.coreboot.org/hyperkitty/list/flashrom@flashrom.org/message/65BN... before writing your reply? Maybe you missed it. Perhaps you'd want to read it: the first paragraph contains information that seems to contradict some of your claims. It could be that we're talking past each other.
On Tue, Oct 18, 2022 at 2:50 PM Richard Hughes hughsient@gmail.com wrote:
Let's make sure we're on the same page, as misunderstandings have already costed the flashrom project way too much time and effort. Could you please explain what "at scale" means for you?
Yes, Google most likely checks that flashrom works for *their* use-cases before deploying anything.
Um, this is not true. There are people out there using flashrom on DOS to update their BIOS, and a significant number of users who come asking for help on IRC don't even know what a SPI clip is. Do you mean that these people are not _really_ using flashrom?
Also... Doesn't fwupd use flashrom? Do all fwupd users have a SPI clip at hand, in case something goes wrong? It would be interesting to know how fwupd handles the possibility of device(s) no longer booting after a failed firmware update.
Normal people don't use flashrom in production in any appreciable number.
Hmmm, this seems to be another potential source of misunderstandings. What do you mean with "in production"?
Huh, why do you say so?
Who do you think is responsible for flashrom, then? Whose responsibility is flashrom?
Richard.
Best regards, Angel
On Tue, 18 Oct 2022 at 16:25, Angel Pons th3fanbus@gmail.com wrote:
Maybe it's not intentional, but your email sounds rather unpleasant and very confusing. What would you like to achieve?
I have no agenda, I'm just sick of seeing all the behaviour on this list. There's a certain toxicity.
Yes, I saw that.
More than 10,000 users.
Um, this is not true. There are people out there using flashrom on DOS
I'm sure there *are* people using flashrom on DOS but I'm not sure it's something you should optimize a project for.
Also... Doesn't fwupd use flashrom?
Yes, but in the same way as Google; i.e. we only allow-list specific hardware (that we test) that uses the internal programmer for a subset of SPI chips. All the other fwupd functionality is native as the impedance mismatch between fwupd and libflashrom is striking. We did play with using other programmers but it didn't work very well.
Not on a test system, i.e. one with paying customers. StarLabs probably wins here, but they're one of the systems we do end-to-end testing with.
Who do you think is responsible for flashrom, then? Whose responsibility is flashrom?
Well it's not mine: I'm seeing hostility, ego and stop energy -- and that's not a place I really want to be.
Richard.
The initial list of tickets for the release was created from Nico’s list of issues to check before the release. (see earlier thread about the release). Some of these are bugs, some others were in the form “what do we do with this?”. I understand that since a few months ago things could have changed, and if as of today we all look at the list and we all feel like some of the tickets are not blocking release, that’s fine. We can unlink them from parent 353 and just leave in bugtracker as a ticket.
Re: bug vs feature and other fields. Please change to whatever you feel is more appropriate. It doesn’t matter for me which type the ticket has. Important info for me is parent ticket (or absence of it) and clear description.
As I posted at that time on the mailing list in “Release preparations”:
Surely all of the descriptions can be improved, or more info can be added to issues. Especially if you are taking an issue and start working on it.
We can also go once again through the list and decide whether all the remaining ones are indeed blocking. We can do this on 2rd November UTC (next scheduled flashrom meeting).
Even though we (Angel) are not licensed therapists, we are willing to listen to you.
Angel, thank you so much! This is really nice of you! It is so important to support each other. I think I am fine. I am very lucky to have a great team at work, and great upstream teammates! But I am on the channel as usual :) Surely we will talk!
Hi list,
We (Angel) sense that this thread might potentially drift towards an unpleasant direction and we think the outcome wouldn't be forward-useful, i.e. it would make things worse. How are all of you doing? Is anyone feeling hurt?
On Sat, Oct 15, 2022 at 11:53 AM Nico Huber nico.h@gmx.de wrote:
In order to make a release, stability (iow. absence of code flaws) is more important than having more features. So, we (Angel) would suggest prioritizing bug fixes over new features, at least to get the release done. Another option would be to branch off current master, radically deal with existing bugs (e.g. delete the problematic sb600spi code, revert the progress state patches) and use this branch for the flashrom v1.3 release. Then we can decide how to proceed for future releases without the pressure of not having done a release in a long time.
Do you have any examples of things that we do not measure? Hmmm, looks like they're explained below.
In other words, it's best to submit new features and potentially breaking changes after a release. Otherwise, things in a release don't get tested properly. On the coreboot side, we had to make the 4.8.1 release because 4.8 was broken: a small patch got submitted just before the 4.8 release that broke loading several payloads, and people who tried the 4.8 release quickly noticed that it did not boot...
We (Angel) found several patches with bugs after a re-review or a post-submit review in the past. Some of these bugs could have been oversights, but other bugs were too conspicuous to be accidental oversights. It felt as if those reviews (mainly those for CrOS flashrom upstreaming patches) were heavily expedited for some reason that we (Angel, and possibly the upstream flashrom community) don't know about. If anyone knows, we'd appreciate if they could help understand why things happened this way, especially if they can shed some light regarding what happened inside Google at that time.
We fully agree with Nico's view. We hadn't thought about the safety and security implications of such changes. Hmmm, maybe we should rewrite flashrom in at-least-Ada then? :D
Ah yes, that was not good... https://review.coreboot.org/66659 caused issues, https://review.coreboot.org/67621 reverted CB:66659, then https://review.coreboot.org/67094 and https://review.coreboot.org/67649 tried to do the same as CB:66659 but better, but still ended up causing problems that https://review.coreboot.org/67752 had to fix. We (Angel) are not convinced that there won't be any more issues with NULL pointers. The reason? The way https://review.coreboot.org/q/topic:programmer_init_param_globals did things is particularly error-prone. https://review.coreboot.org/66654 adds NULL pointers in a bunch of places, and it's very easy to lose track of them. Nico had already suggested an approach to avoid having to deal with NULL pointers: make the patches introducing `programmer_cfg` in reverse order. Roughly, this would mean starting with https://review.coreboot.org/67649 and ending with https://review.coreboot.org/66654 instead, so that `programmer_cfg` pointers are valid from the very beginning.
Can we (everyone reading this) learn from this case? We (Angel) think so: this can be avoided by reviewing the *design* and/or "plan of attack" (the "enemy" being "anything in flashrom that can be improved" (let's not attack others please)) in advance. This can be done by discussing the plans in flashrom's communication channels (e.g. our mailing list or the #flashrom IRC channel on Libera), or in the form of RFC (Request For Comments) changes (although authors of RFC changes *need* to be OK with having to completely redesign the implementation).
At least to us (Angel), what needs to be done to address #370 is not clear at all. Yes, sb600spi is buggy, but how should we proceed? We (Angel) believe that the most straightforward to deal with the sb600spi mess would be to revert the buggy features, sanitize the codebase (make sure refactoring patches didn't introduce any other regressions, fix any issues with the original codebase), and only then consider reintroducing the new features.
For potentially harmful issues, like the risk of soft bricking machines with >16 MiB flash, definitely.
On Sun, Oct 16, 2022 at 9:41 PM Anastasia Klimchuk aklm@chromium.org wrote:
Nico what was the goal why you started this thread?
I thought for a moment, you have a *quick* question. So I answered.
It could be that the question itself is quick, but its answer is not. Or it might be a way to introduce the question to the readers. We (Angel) introduce some of our questions with the self-deprecating "Stupid question" preamble, even if we're sure that the question itself is not at all stupid. Using "Stupid question" to preface (some of) our questions inspires others to critically assess what we're asking, which can help them put aside any grudges they might have against us. We also use this because we love humor: humor is one of the healthiest ways to cope with the trials and tribulations of life.
Now you seem to be unhappy to learn that we are closer to the release than 1/2 yr ago.
Are we actually closer to making a release? It's interesting that #353 is classified as a feature; we (Angel) would believe that not making a new release in over 2 years is more of a bug than a feature... There's also several refactoring tasks in #353 which seem to be orthogonal to the problem of "release flashrom v1.3 someday". We probably said earlier in this email that it would be a good idea to try to release v1.3 without the more invasive/controversial features and refactorings, not only from a technical perspective (users get to use a newer flashrom version with some important bug fixes, e.g. the emergency help text directs them to the correct IRC server) but also from an emotional perspective (not having to worry about the pressure of not having done any releases in over two years). This doesn't mean that these features will be scrapped; rather, we will be able to work on them much more effectively. It's just another form of "divide and conquer"!
You are saying "But I don't believe you", what was the point of asking then?
Anastasia, the following paragraph is specifically for you. Apologies if it makes you feel attacked in any way: we don't want to hurt you, but us (Angel) having autism makes it harder to understand people outside the spectrum.
You know that, a few days ago, we (Angel) finally managed to understand what really happened w.r.t. flashrom in the last two years or so. The only reason why we were able to do so is because we got rid of all prejudices we had regarding Google people, especially Edward. It was only after clearing those nasty prejudice-clouds that we were able to "see" what actually happened, and how far from reality our prejudices were. Although we do not recommend caustically ranting and raving on Gerrit (like we did...), we'd strongly encourage you to speak out your feelings to others (either publicly or privately) as a means of catharsis: to bring any buried grudges and other unpleasant feelings into your consciousness and release them permanently. In a way, the idea is to do something similar to Marie Kondo's https://en.wikipedia.org/wiki/Marie_Kondo#KonMari_method but applied to "things" in one's mind: quickly and completely discard whatever that doesn't "spark joy", and focus on the things that "spark joy". Even though we (Angel) are not licensed therapists, we are willing to listen to you.
-- Anastasia.
Best regards, Angel
-
Anastasia Klimchuk -
Angel Pons -
Edward O'Callaghan -
Nico Huber -
Richard Hughes