When attempting to use buildrom I'm seeing the " undefined reference to `__stack_chk_fail'" message. I see that this was a topic of discussion back in January but haven't found anything to say how or if this was resolved. At the risk of scratching an old itch I have to ask if this was resolved or if a solution to the problem is still needed.
I'm building on Ubuntu 6.10 which installs a gcc configured for stack checking. I know about the -fno-stack-protector option when compiling and have used this as a work-around but I'm thinking that if this was/is a problem then other host distro dependent issues perhaps will or still be lurking to catch someone.
Is it a viable solution to have buildrom build a cross-tool chain? Imho: This would be best in order to have repeatable results. Having config options to control the gcc and binutils versions cross-host build repeatability would be improved.
Steve
On 30/11/07 15:29 -0800, Steve Isaacs wrote:
Is it a viable solution to have buildrom build a cross-tool chain? Imho: This would be best in order to have repeatable results. Having config options to control the gcc and binutils versions cross-host build repeatability would be improved.
This is exactly the direction I want to avoid going in. We really don't want to be in the business of building a complete development environment. Its just too much complexity for what we will gain from it.
We have to reasonably expect that LinuxBIOS and the payloads will correctly compile with any give x86 development environment. The purpose of buildrom is just to make that process easier - 1 step instead of 2 or 6. Its a tool, not a requirement. If we create our own development environment, then it becomes a requirement, and instead of fixing compiler problems that we _should_ be fixing, we'll be relying more on hacking on the buildrom development environment, and we lose.
Jordan
On Fri, 2007-11-30 at 16:57 -0700, Jordan Crouse wrote:
On 30/11/07 15:29 -0800, Steve Isaacs wrote:
Is it a viable solution to have buildrom build a cross-tool chain? Imho: This would be best in order to have repeatable results. Having config options to control the gcc and binutils versions cross-host build repeatability would be improved.
This is exactly the direction I want to avoid going in. We really don't want to be in the business of building a complete development environment. Its just too much complexity for what we will gain from it.
I understand your sentiment and if I try to look at things from your perspective I can agree. However, I've seen it mentioned that it is the desire of the LinuxBIOS project to get more board manufacturer's involved. For that to happen LinuxBIOS cannot be a moving target.
Look at it from a manufacturer's perspective. The environments in which I have worked require a stringent qualification, test and release process in order to better understand what is being released. Once released, products can be in maintenance for years and service contracts typically determine who gets an update and when. During this time build servers can be replaced and tools typically move through several revisions.
Parts we place on our boards go through a strict qualification process and once on the board are not changed unless a severe problem is discovered (reduced risk because of test process), the part is no longer available (low risk because of the qualification process) or the board itself is end-of-life.
The same needs to be true for software. Willy-nilly use of a compiler because it happens to be the latest version introduces variables into the code that increases the test problem. Without rigorous qualification of the new compiler flaws can be introduced into the software that can find their way to the field in tens, hundreds or, for some manufacturer's, thousands of copies. Deployment on these scales cost money, the qualification process costs money. Redeployment can cost even more if one considers the disruption of other projects. In order to protect the bottom line manufacturer's typically avoid these costs unless absolutely necessary. And this does not consider the effect such things can have on the manufacturer's reputation in a competitive world.
The key to keeping these costs low is repeatability. A repeatable process is needed in order to reliably run repeatable tests so that when the results do not repeat, a flaw has been discovered (regression testing). This does not mean that a product needs to be perfect, none are. What I'm trying to convey is that known flaws are easier to deal with than unknown flaws. Workarounds are acceptable in most cases. Workarounds are necessary because of not being able to spend the funds to qualify a new tool or restart a full test cycle. Moving targets break workarounds. Repeatability helps maintain the viability of workarounds.
Back to my point. It boils down to cost and risk is a factor that effects cost. In order for LinuxBIOS to be used instead of a proprietary BIOS it must be shown that LinuxBIOS is a lower risk since direct cost is nearly non-existent. Without a repeatable build process risk remains high. Therefore, I believe that like it or not if you want LinuxBIOS to be widely accepted it must be able to demonstrate repeatability. Proprietary providers recognize this and this is why they tout support and controlled releases and get paid for it. imho: Without this for most LinuxBIOS will remain a hobby.
So, if LinuxBIOS does not want to be in the business of building tools then I suggest a project be found that will provide the tools into which LinuxBIOS can be integrated or visa versa. This way LinuxBIOS can also avoid being in the business of beta testing new tools and focus even more on the business of producing a reliable and repeatable BIOS.
Steve
On Mon, Dec 03, 2007 at 08:34:00AM -0800, Steve Isaacs wrote:
I believe that like it or not if you want LinuxBIOS to be widely accepted it must be able to demonstrate repeatability.
Excellent point! I agree with you completely.
We should definately look at automating the build process and even build environment, because it is a sensitive component in the build process, and we should make LB smart enough to build with a build environment other than the one that starts with /usr/bin/gcc.
However - I want it to remain optional rather than a requirement, so that the entry level stays as low as possible for as many as possible.
But one does not, and should not, exclude the other! :)
//Peter
On Mon, 2007-12-03 at 18:08 +0100, Peter Stuge wrote:
However - I want it to remain optional rather than a requirement, so that the entry level stays as low as possible for as many as possible.
I agree. A fully native build should be supported for when it's appropriate and to keep the startup checklist shorter.
Actually, entry level for me jumped rather high because of my use of of a distro that installs a compiler configured for using stack checking as a default. I was faced with either installing my own toolchain or modifying the build for LinuxBIOS or switching distros. Note that's a distro issue, not a toolchain issue. With the toolchain builder already integrated I'd have changed my config and moved on rather than potentially accumulating overtime dealing with this issue. For me, distro variations -- not compiler bugs -- have been the biggest problem.
I'll toss this on the table as a possible approach to a solution:
- Buildrom be enhanced to --optionally-- download and install in a user's sandbox a toolchain builder.
- Problems found in the toolchain builder be fed back to the toolchain builder project thus satisfying Jordan's point of having a responsibility to fix the toolset when broken. Crosstool builders often employ target specific patches when necessary.
- Configuration options be added to buildrom to select whether or not to build native or use cross tools (yes, this is a cross tool environment since I'm building on a host for a target of a different architecture).
- Additional options be added to control the cross-tool version to build (both for buildrom and the cross-tool builder itself).
- The cross tool builder config utility be used for control of toolchain versions and other options.
The obvious question is which toolchain builder? I might suggest using crosstool (by Dan Kegel) which has been widely used in the embedded world. This has more recently been revised into crosstool-NG which adds a menuconfig style interface. I like this because it would be visually and functionally consistent with buildrom. I'm unsure as to the status of either of these but they look promising for this purpose. Comments?
Other distro and embedded build systems exist (and several use crosstool) but I would not recommend integrating LinuxBIOS into any of them for obvious reasons. I'm sure others would agree that it is not in the best interest of LinuxBIOS to have it driven by the requirements of another project.
Steve
On 03/12/07 10:32 -0800, Steve Isaacs wrote:
On Mon, 2007-12-03 at 18:08 +0100, Peter Stuge wrote:
However - I want it to remain optional rather than a requirement, so that the entry level stays as low as possible for as many as possible.
I agree. A fully native build should be supported for when it's appropriate and to keep the startup checklist shorter.
Actually, entry level for me jumped rather high because of my use of of a distro that installs a compiler configured for using stack checking as a default. I was faced with either installing my own toolchain or modifying the build for LinuxBIOS or switching distros. Note that's a distro issue, not a toolchain issue. With the toolchain builder already integrated I'd have changed my config and moved on rather than potentially accumulating overtime dealing with this issue. For me, distro variations -- not compiler bugs -- have been the biggest problem.
I'm already on board with my opinion on a cross compiler, no need to rehash that here.
But the thing about _this_ bug is that this can be fixed, and if you use buildrom it _should_ be fixed. This is our fault, and we need to remedy it immediately. What platform are you trying to build for?
Jordan
On Mon, 2007-12-03 at 11:52 -0700, Jordan Crouse wrote:
I'm already on board with my opinion on a cross compiler, no need to rehash that here.
But the thing about _this_ bug is that this can be fixed, and if you use buildrom it _should_ be fixed. This is our fault, and we need to remedy it immediately. What platform are you trying to build for?
I'm working on an adaptation for a board of our design using a chipset not currently supported by LinuxBIOS. I say "currently" because one reason I'm participating in this forum is so the work I do can be submitted back to the project meaning the chips I add can one day become part of the LinuxBIOS source set.
Steve
On 03/12/07 11:29 -0800, Steve Isaacs wrote:
On Mon, 2007-12-03 at 11:52 -0700, Jordan Crouse wrote:
I'm already on board with my opinion on a cross compiler, no need to rehash that here.
But the thing about _this_ bug is that this can be fixed, and if you use buildrom it _should_ be fixed. This is our fault, and we need to remedy it immediately. What platform are you trying to build for?
I'm working on an adaptation for a board of our design using a chipset not currently supported by LinuxBIOS. I say "currently" because one reason I'm participating in this forum is so the work I do can be submitted back to the project meaning the chips I add can one day become part of the LinuxBIOS source set.
Depending on which Config.lb you based your code on, you might be happy or sad. In buildrom, we "fix" the stack problem by passing in -fno-stack-protector through the CPU_OPT environment variable, which then gets included in the make rules. But we have found that on some platforms. CPU_OPT isn't used by every rule, and we've been trying to fix those as they happen. The problem is that LinuxBIOS compiler arguments are pretty long, and its a pain to go through them all to figure out which files don't have -fno-stack-protector attached to them. The best way to do it is to figure out which files are the offenders (either through the linker logs or by running nm on all the .o files and looking for the stack protector symbol) and find the rules and fix them.
Jordan
On Mon, 2007-12-03 at 13:19 -0700, Jordan Crouse wrote:
they happen. The problem is that LinuxBIOS compiler arguments are pretty long, and its a pain to go through them all to figure out which files don't have -fno-stack-protector attached to them. The best way to do it is to figure out which files are the offenders (either through the linker logs or by running nm on all the .o files and looking for the stack protector symbol) and find the rules and fix them.
I'll do that. That should get me past the problem quickest.
Thanks,
Steve
* Steve Isaacs yasteve@gmail.com [071203 19:32]:
- Buildrom be enhanced to --optionally-- download and install in a
user's sandbox a toolchain builder.
I don't think buildrom is the place for this. It's pretty much out of the scope for the project.
Next time a distribution ships a broken glibc, we start adding that, too. If you use a distribution with a broken toolchain, you should visit http://kegel.com/crosstool/ and build a new toolchain _once_.
I agree we should have an FAQ entry to point people to crosstool though.
- Problems found in the toolchain builder be fed back to the toolchain
builder project thus satisfying Jordan's point of having a responsibility to fix the toolset when broken. Crosstool builders often employ target specific patches when necessary.
We also need feedback on our own bugs and issues before or while we start looking in other directions. We have a fully working test system that nobody is using anymore: http://www.linuxbios.org/Distributed_and_Automated_Testsystem If you have supported nodes, that you do not need anymore, please contact me for putting them into the test system.
Also, we should try to merge autobuild and buildrom. I never understood why a new tool was build instead of enhancing an already existing one. qa.linuxbios.org is fully using abuild. We drop a couple of potential problems if we start using one solution for building linuxbios.
On 03/12/07 22:02 +0100, Stefan Reinauer wrote:
- Problems found in the toolchain builder be fed back to the toolchain
builder project thus satisfying Jordan's point of having a responsibility to fix the toolset when broken. Crosstool builders often employ target specific patches when necessary.
We also need feedback on our own bugs and issues before or while we start looking in other directions. We have a fully working test system that nobody is using anymore: http://www.linuxbios.org/Distributed_and_Automated_Testsystem If you have supported nodes, that you do not need anymore, please contact me for putting them into the test system.
Testing isn't the problem here. Nobody has mentioned that this is a problem with LinuxBIOS - buildrom isn't about LinuxBIOS and neither is the Ubuntu stack problem. No amount of automated testing could help this problem, unless you happen to be building on the distribution in question, and its the correct version and everything else. Its great that we have all these tools, but they don't help with this particular problem, and never will.
Also, we should try to merge autobuild and buildrom. I never understood why a new tool was build instead of enhancing an already existing one. qa.linuxbios.org is fully using abuild. We drop a couple of potential problems if we start using one solution for building linuxbios.
Let me say this one more time - and I'll put it in capital letters. BUILDROM IS A TOOL. The only way that the LinuxBIOS team supports building LinuxBIOS is from the command line. *THAT* is the solution for building LinuxBIOS. Everything else is an tool for performing some custom task. abuild has its place. buidrom has its place. LBDistro has its place. The responsibilities and functionality of these tools are on the creators and maintainers of those tools, not the LinuxBIOS community as a whole. The only deliverable that this group needs to care about is making LinuxBIOS work.
Jordan
--- Jordan Crouse Systems Software Development Engineer Advanced Micro Devices, Inc.
* Peter Stuge peter@stuge.se [071203 18:08]:
On Mon, Dec 03, 2007 at 08:34:00AM -0800, Steve Isaacs wrote:
I believe that like it or not if you want LinuxBIOS to be widely accepted it must be able to demonstrate repeatability.
Excellent point! I agree with you completely.
We should definately look at automating the build process and even build environment, because it is a sensitive component in the build process, and we should make LB smart enough to build with a build environment other than the one that starts with /usr/bin/gcc.
We already have all this: That's why I developed the LinuxBIOS test system that started producing binary images at a central point:
http://qa.linuxbios.org/log_test.php?timestamp=20061109-173426&device=ep...
Of course everyone is free to use the same tool chain for all their builds. But for sure, the result can not be good if we try to build every software development and testing process out there into linuxbios. At some point there will always be a special restriction that a commercial vendor will have to make. Like running on certain svn revisions, basically forking, rather than staying on HEAD all the time.
However - I want it to remain optional rather than a requirement, so that the entry level stays as low as possible for as many as possible.
If someone feels the need for a cross development environment -- there's crosstool. You can easily build RPMs for all kinds of cross compiler versions for all kinds of platforms.
It is an interesting project, and we recommend everyone using LinuxBIOS in a commercial environment to use it. But that belongs into the corporate software development process, and not into linuxbios.
Just change your target config.lb to your known good tool chain and you're fine.
In addition, as a side note: I have not seen any miscompilations with any toolchain of (Open)SUSE Linux since 2003 whereas Redhat/Fedora was troublesome with every second toolchain release. It seems RH/F is quite a bit more "progressive" when it comes down to toolchains. So if you want a stable system, you should also take into consideration that choosing a distribution as experimental as FC8, you will run into such problems.
Stefan
On Mon, 2007-12-03 at 21:53 +0100, Stefan Reinauer wrote:
We already have all this: That's why I developed the LinuxBIOS test system that started producing binary images at a central point:
http://qa.linuxbios.org/log_test.php?timestamp=20061109-173426&device=ep...
Did I miss something? Is this something that others can use? btw: the URL didn't point to anything other than a test report.
If someone feels the need for a cross development environment -- there's crosstool. You can easily build RPMs for all kinds of cross compiler versions for all kinds of platforms.
It is an interesting project, and we recommend everyone using LinuxBIOS in a commercial environment to use it. But that belongs into the corporate software development process, and not into linuxbios.
I agree, I've used crosstool before on different embedded projects. I suppose my suggestion to integrate crosstool into buildrom doesn't fly. I'll accept that. Give me some time to get a feel for what is LinuxBIOS responsibility and what isn't.
Just change your target config.lb to your known good tool chain and you're fine.
I did some digging in the Config.lb files and couldn't find an example. Do you have one? Is there any doc on how to control things like CC, CFLAGS and the like? I didn't realize things could be shunted this way.
Steve
* Steve Isaacs yasteve@gmail.com [071203 22:19]:
Did I miss something? Is this something that others can use? btw: the URL didn't point to anything other than a test report.
In theory, yes. Anyone can hook up a mainboard into the test system. And anyone else can then submit bios images to be tested to those distributed machines. Also, the bios images built with every svn commit are sent to all connected machines, making sure there are no regressions. Unfortunately the system is currently unused when I quickly needed that one bios savior for 39SF020 parts in another project.
I agree, I've used crosstool before on different embedded projects. I suppose my suggestion to integrate crosstool into buildrom doesn't fly. I'll accept that. Give me some time to get a feel for what is LinuxBIOS responsibility and what isn't.
Sorry if I sounded bitchy. I don't want to be. Though I think we should try to follow the rules that all good things are simple.
I did some digging in the Config.lb files and couldn't find an example. Do you have one? Is there any doc on how to control things like CC, CFLAGS and the like? I didn't realize things could be shunted this way.
You can add the following lines to your target/<vendor>/<board>/Config.lb:
option CC="i386-elf-gcc" # this one is CFLAGS. option CPU_OPT="-O2 -Wl,..." option HOSTCC="gcc"
Stefan
On Mon, Dec 03, 2007 at 10:36:51PM +0100, Stefan Reinauer wrote:
You can add the following lines to your target/<vendor>/<board>/Config.lb:
option CC="i386-elf-gcc" # this one is CFLAGS. option CPU_OPT="-O2 -Wl,..." option HOSTCC="gcc"
Yep, this should be in the wiki (FAQ?) if it's not already there.
Uwe.
On Mon, 2007-12-03 at 22:45 +0100, Uwe Hermann wrote:
On Mon, Dec 03, 2007 at 10:36:51PM +0100, Stefan Reinauer wrote:
You can add the following lines to your target/<vendor>/<board>/Config.lb:
option CC="i386-elf-gcc" # this one is CFLAGS. option CPU_OPT="-O2 -Wl,..." option HOSTCC="gcc"
Yep, this should be in the wiki (FAQ?) if it's not already there.
Uwe.
I just now looked and didn't see anything. Of course, being relatively new here, this looks like a forest full of trees to me. I need time for my eyes to adjust.
So, I summarize. If I need a consistent and repeatable build I use the toolchain of my choice and use the above to point the LinuxBIOS build to that toolchain. Also, as I suspected -- crosstool is a good choice.
btw: this is the info I needed -- just didn't know it. Thanks,
Steve
On Mon, Dec 03, 2007 at 02:11:50PM -0800, Steve Isaacs wrote:
So, I summarize. If I need a consistent and repeatable build I use the toolchain of my choice and use the above to point the LinuxBIOS build to that toolchain.
Yep, pretty much. For a product you probably also want to stick with a certain tested svn version of LinuxBIOS (not trunk, which is a moving target).
(This doesn't mean trunk is unstable. Any breakage is considered a no-no and is a bug; all commits should keep trunk in a building/working state. But using a specific svn version is still a bit safer)
HTH, Uwe.
On 03/12/07 08:34 -0800, Steve Isaacs wrote:
On Fri, 2007-11-30 at 16:57 -0700, Jordan Crouse wrote:
On 30/11/07 15:29 -0800, Steve Isaacs wrote:
Is it a viable solution to have buildrom build a cross-tool chain? Imho: This would be best in order to have repeatable results. Having config options to control the gcc and binutils versions cross-host build repeatability would be improved.
This is exactly the direction I want to avoid going in. We really don't want to be in the business of building a complete development environment. Its just too much complexity for what we will gain from it.
I understand your sentiment and if I try to look at things from your perspective I can agree. However, I've seen it mentioned that it is the desire of the LinuxBIOS project to get more board manufacturer's involved. For that to happen LinuxBIOS cannot be a moving target.
Look at it from a manufacturer's perspective. The environments in which I have worked require a stringent qualification, test and release process in order to better understand what is being released. Once released, products can be in maintenance for years and service contracts typically determine who gets an update and when. During this time build servers can be replaced and tools typically move through several revisions.
You don't have to worry - as you can infer from the e-mail address, I am keenly aware of the challenges that we have to face for commercial adoption. It is one of my primary goals to ensure that LinuxBIOS becomes and stays a viable option for commercial vendors.
That said, you have to understand that LinuxBIOS and Linux and indeed all of open source will continue to constantly be a moving target. Every day the open source environment presents us with a slightly different view of the world as it twists and turns and evolves.
When I write code, I imagine that every open source project is a train, constantly rolling toward an unknown destination. At every stop, new people are introduced to the project and get on board. They may have heard about the project at a conference, or on a slashdot posting, or just stumbled across it on freshmeat. However they got here, something compelled them to board the train. Whats more, they are going to bring some sort of baggage with them - different distributions (or even operating systems), different languages, and different experiences.
Our job is to make sure that every new person that boards the train is immediately satisfied; because their first perceptions of the code are what determines if they stay or if they immediately get off the train.
This is why we need to make sure that LinuxBIOS works right out of the box with as many different pieces of "baggage" as possible. Buildrom is an excellent tool to help accomplish that, but it can not and should not be essential for all those new passengers to get satisfaction from their first LinuxBIOS experience.
Commercial vendors and manufacturers are no exception - every commercial project has to jump on the train at some point. The code is evaluated and a snapshot of all the current versions, behaviors and bugs in the code is constructed. Once the code has been qualified, it is always wise to freeze the code where it stands, anybody who disagrees obviously hasn't been in the trenches when trying to bring up a new platform. I have numerous projects sitting on my desk right now, not a single one of them that is running kernel version 2.6.24-rcX (heck, none of them are even running 2.6.23).
But this doesn't mean that the train doesn't keep on going without us - the code continues to evolve and prepare itself for the next new passenger.
This is why we (the project) doesn't want to start dictating terms like compiler versions and the like. Once we do that, we start to lose new users and contributors, as the price for admission starts to go up (especially as we start ignoring new features and flaws in newer compilers). Instead, we need to focus our energies on making sure that LinuxBIOS continues to work for every new passenger that gets on our train, and that regardless of what point our commercial vendors and friends decide to freeze, that they'll get the best possible experience from the compilers and applications that they have chosen.
All that said, I think its probably time that we fix this problem once and for all. Ron and Stefan - you are the remaining "experts" on the v2 configuration system - how can we sneak in the $(try-option) calls from buildrom (which were in turned borrowed from the kernel) into LinuxBIOS to put this thing to bed once and for all?
Jordan
On Mon, 2007-12-03 at 10:40 -0700, Jordan Crouse wrote:
You don't have to worry - as you can infer from the e-mail address, I am keenly aware of the challenges that we have to face for commercial adoption. It is one of my primary goals to ensure that LinuxBIOS becomes and stays a viable option for commercial vendors.
I'm sorry, I didn't mean to imply you are not aware of the issues. I just wanted to change the focus a bit. I did notice your e-mail address and was tempted to use that as a case in point but chose not to. I have a lot of respect for your employer.
That said, you have to understand that LinuxBIOS and Linux and indeed all of open source will continue to constantly be a moving target. Every day the open source environment presents us with a slightly different view of the world as it twists and turns and evolves.
Most products, open-source or not, are moving targets in the end. Commercial products are continually repaired and enhanced to satisfy customer requirements. But, for commercial viability, plateaus have to be established and maintained.
When I write code, I imagine that every open source project is a train, constantly rolling toward an unknown destination. At every stop, new people are introduced to the project and get on board. They may have heard about the project at a conference, or on a slashdot posting, or just stumbled across it on freshmeat. However they got here, something compelled them to board the train. Whats more, they are going to bring some sort of baggage with them - different distributions (or even operating systems), different languages, and different experiences.
I like your analogy. It reflects how the company that pays my salary decided to "board the train" as you put it. As a new passenger here I'll have to defer to your experience. It is not my intent to derail the ideals of an open source project.
Our job is to make sure that every new person that boards the train is immediately satisfied; because their first perceptions of the code are what determines if they stay or if they immediately get off the train.
Please remember that it is --my-- job to satisfy customer requirements on or before the deadline. These obviously are conflicting perspectives. A solution everyone can enjoy requires time coupled with a willingness to wander. This may not always be conducive to meeting a deadline which requires focus and limited time. I'm hoping I can suggest a means to mitigate both.
This is why we (the project) doesn't want to start dictating terms like compiler versions and the like.
I would never suggest dictating terms. If that were the case I would not be a "passenger" now. I was simply trying to suggest a means for reducing risk by increasing confidence through having repeatable releases. Nothing more.
Steve
btw: the opinions I expressed here are my own.
-
Jordan Crouse -
Peter Stuge -
Stefan Reinauer -
Steve Isaacs -
Uwe Hermann