John Kollasch just pointed out to me that it looks like the reason the cn700 is booting so slowly (>15min) could be that it's running out of flash. I've also noticed this:
LAR: Attempting to open 'normal/stage2/segment0'. LAR: Start 0xfff00000 len 0x100000 LAR: seen member normal/option_table@0xfff00000, size 932 LAR: seen member normal/initram/segment0@0xfff00400, size 6216 LAR: seen member normal/stage2/segment0@0xfff01ca0, size 1
With a 1MB flash part, so 0xfff00000 is the ROM. Is this how it's supposed to work, and if not, how do we fix it?
Thanks, Corey
On Tue, Dec 23, 2008 at 8:40 PM, Corey Osgood corey.osgood@gmail.com wrote:
John Kollasch just pointed out to me that it looks like the reason the cn700 is booting so slowly (>15min) could be that it's running out of flash. I've also noticed this:
LAR: Attempting to open 'normal/stage2/segment0'. LAR: Start 0xfff00000 len 0x100000 LAR: seen member normal/option_table@0xfff00000, size 932 LAR: seen member normal/initram/segment0@0xfff00400, size 6216 LAR: seen member normal/stage2/segment0@0xfff01ca0, size 1
With a 1MB flash part, so 0xfff00000 is the ROM. Is this how it's supposed to work, and if not, how do we fix it?
I think the problem is you are not caching flash or anything else for that matter.
I wonder if moving mtrr out of stage0 was the right move?
initram has to run from flash. It is execute in place. Stage2 is copied to dram.
ron
On Tue, Dec 23, 2008 at 11:59 PM, ron minnich rminnich@gmail.com wrote:
On Tue, Dec 23, 2008 at 8:40 PM, Corey Osgood corey.osgood@gmail.com wrote:
John Kollasch just pointed out to me that it looks like the reason the
cn700
is booting so slowly (>15min) could be that it's running out of flash.
I've
also noticed this:
LAR: Attempting to open 'normal/stage2/segment0'. LAR: Start 0xfff00000 len 0x100000 LAR: seen member normal/option_table@0xfff00000, size 932 LAR: seen member normal/initram/segment0@0xfff00400, size 6216 LAR: seen member normal/stage2/segment0@0xfff01ca0, size 1
With a 1MB flash part, so 0xfff00000 is the ROM. Is this how it's
supposed
to work, and if not, how do we fix it?
I think the problem is you are not caching flash or anything else for that matter.
What should I be doing where? I can enable, disable, or make the shadow ram dance like a monkey (yeah, Via hardware's got some odd features), but it makes no real difference in the boot speed.
I wonder if moving mtrr out of stage0 was the right move?
The only other option AFAIK is to go back to ROMCC, CAR uses those MTRRs and enabling them breaks CAR.
initram has to run from flash. It is execute in place. Stage2 is copied to dram.
Is the whole flash copied to dram once, or are the segments copied as their needed? LAR is saying it's opening the segments directly from flash (0xfff00000).
Thanks, Corey
On Wed, Dec 24, 2008 at 12:13 AM, Corey Osgood corey.osgood@gmail.comwrote:
On Tue, Dec 23, 2008 at 11:59 PM, ron minnich rminnich@gmail.com wrote:
On Tue, Dec 23, 2008 at 8:40 PM, Corey Osgood corey.osgood@gmail.com wrote:
John Kollasch just pointed out to me that it looks like the reason the
cn700
is booting so slowly (>15min) could be that it's running out of flash.
I've
also noticed this:
LAR: Attempting to open 'normal/stage2/segment0'. LAR: Start 0xfff00000 len 0x100000 LAR: seen member normal/option_table@0xfff00000, size 932 LAR: seen member normal/initram/segment0@0xfff00400, size 6216 LAR: seen member normal/stage2/segment0@0xfff01ca0, size 1
With a 1MB flash part, so 0xfff00000 is the ROM. Is this how it's
supposed
to work, and if not, how do we fix it?
I think the problem is you are not caching flash or anything else for that matter.
What should I be doing where? I can enable, disable, or make the shadow ram dance like a monkey (yeah, Via hardware's got some odd features), but it makes no real difference in the boot speed.
Sorry if this came off as rude, it was supposed to be a joke. I'm still confused on caching though, where/how it should be done, I'm trying to find how it's done on geode and coming up empty handed.
I wonder if moving mtrr out of stage0 was the right move?
The only other option AFAIK is to go back to ROMCC, CAR uses those MTRRs and enabling them breaks CAR.
initram has to run from flash. It is execute in place. Stage2 is copied to dram.
Is the whole flash copied to dram once, or are the segments copied as their needed? LAR is saying it's opening the segments directly from flash (0xfff00000).
Nevermind that question, I just (finally) found the code that handles that, and that is how it works, the segment is loaded from the ROM right before it's run. Is there any advantage/disadvantage to doing it that way instead of copying the whole LARball once and walking on RAM?
Thanks, Corey
On Tue, Dec 23, 2008 at 10:02 PM, Corey Osgood corey.osgood@gmail.com wrote:
Nevermind that question, I just (finally) found the code that handles that, and that is how it works, the segment is loaded from the ROM right before it's run. Is there any advantage/disadvantage to doing it that way instead of copying the whole LARball once and walking on RAM?
You can't copy the larball unless you have a mbyte of shadow ram somewhere.
Plus, a typical filo + coreboot is at around 64k -- you don't want to copy a whole Mbyte just to get at 64k.
We need to solve your caching problem. I'm a bit concerned about the "move mtrrs out of stage1" patch. I understand your goals with it and it might even be right but ... I'm wondering if you are caching ROM. I may have erred in acking that patch. Sorry. It did not impact kontron (I tested it), and geode has no mtrr's, so it was hard for me to see any negative impact. Myles, you have any thoughts here?
ron
On 24.12.2008 08:46, ron minnich wrote:
On Tue, Dec 23, 2008 at 10:02 PM, Corey Osgood corey.osgood@gmail.com wrote:
Nevermind that question, I just (finally) found the code that handles that, and that is how it works, the segment is loaded from the ROM right before it's run. Is there any advantage/disadvantage to doing it that way instead of copying the whole LARball once and walking on RAM?
You can't copy the larball unless you have a mbyte of shadow ram somewhere.
Plus, a typical filo + coreboot is at around 64k -- you don't want to copy a whole Mbyte just to get at 64k.
We need to solve your caching problem. I'm a bit concerned about the "move mtrrs out of stage1" patch. I understand your goals with it and it might even be right but ... I'm wondering if you are caching ROM. I may have erred in acking that patch. Sorry. It did not impact kontron (I tested it), and geode has no mtrr's, so it was hard for me to see any negative impact. Myles, you have any thoughts here?
Ron, this one is rather simple once you see the pitfalls. - The MTRR setup functions in stage1 were never active in any target because they killed CAR. That's why removing them was OK. - C7 CAR does not enable any caching yet (neither in stage0 (for easier debugging back then and also because of strange failures) nor in stage1) unless I'm mistaken.
Regards, Carl-Daniel
On Tue, Dec 23, 2008 at 11:56 PM, Carl-Daniel Hailfinger c-d.hailfinger.devel.2006@gmx.net wrote:
Ron, this one is rather simple once you see the pitfalls.
- The MTRR setup functions in stage1 were never active in any target
because they killed CAR. That's why removing them was OK.
But we ought to be able to use them after car is disabled, i.e. after initram is done, right? This is purely an academic question at this point.
- C7 CAR does not enable any caching yet (neither in stage0 (for easier
debugging back then and also because of strange failures) nor in stage1) unless I'm mistaken.
I am glad that things make sense. Now we just need to fix the C7 :-)
thanks
ron
On Wed, Dec 24, 2008 at 3:02 AM, ron minnich rminnich@gmail.com wrote:
On Tue, Dec 23, 2008 at 11:56 PM, Carl-Daniel Hailfinger c-d.hailfinger.devel.2006@gmx.net wrote:
Ron, this one is rather simple once you see the pitfalls.
- The MTRR setup functions in stage1 were never active in any target
because they killed CAR. That's why removing them was OK.
But we ought to be able to use them after car is disabled, i.e. after initram is done, right? This is purely an academic question at this point.
I put early_mtrr_init() into stage1_phase3(), it didn't seem to make any sort of difference in boot speed, but it didn't seem to break anything either.
- C7 CAR does not enable any caching yet (neither in stage0 (for easier
debugging back then and also because of strange failures) nor in stage1) unless I'm mistaken.
I am glad that things make sense. Now we just need to fix the C7 :-)
*phew* I was worried that was how v3 was always going to run. As for those times Ron, stage0 takes <1 second, initram is ~5 seconds, and stage2 is a loooong time. Just loading stage2/segment0 is around 30 seconds, but now we know why. I'm hoping you guys can cook up a fix, I've tinkered with that code a bunch of times tonight and all I've done is break stuff.
-Corey
On Wed, Dec 24, 2008 at 12:14 AM, Corey Osgood corey.osgood@gmail.com wrote:
*phew* I was worried that was how v3 was always going to run. As for those times Ron, stage0 takes <1 second, initram is ~5 seconds,
the initram seems way too long. But that stage 2 tells me you have some real problems with your dram setup.
I know nothing about this part. I wonder if you could dump your mtrr settings and let us see them.
ron
On Wed, Dec 24, 2008 at 5:09 AM, ron minnich rminnich@gmail.com wrote:
On Wed, Dec 24, 2008 at 12:14 AM, Corey Osgood corey.osgood@gmail.com wrote:
*phew* I was worried that was how v3 was always going to run. As for
those
times Ron, stage0 takes <1 second, initram is ~5 seconds,
the initram seems way too long. But that stage 2 tells me you have some real problems with your dram setup.
It could have been less, timing output from a serial port with a stopwatch isn't very exact. I don't think there's anything wrong with the dram setup, I ran memtest86 overnight without any errors.
I know nothing about this part. I wonder if you could dump your mtrr
settings and let us see them.
Sure, what point do you want them from? End of stage1?
-Corey
On Wed, Dec 24, 2008 at 5:20 AM, Corey Osgood corey.osgood@gmail.comwrote:
On Wed, Dec 24, 2008 at 5:09 AM, ron minnich rminnich@gmail.com wrote:
On Wed, Dec 24, 2008 at 12:14 AM, Corey Osgood corey.osgood@gmail.com wrote:
*phew* I was worried that was how v3 was always going to run. As for
those
times Ron, stage0 takes <1 second, initram is ~5 seconds,
the initram seems way too long. But that stage 2 tells me you have some real problems with your dram setup.
It could have been less, timing output from a serial port with a stopwatch isn't very exact. I don't think there's anything wrong with the dram setup, I ran memtest86 overnight without any errors.
I know nothing about this part. I wonder if you could dump your mtrr
settings and let us see them.
Sure, what point do you want them from? End of stage1?
Here's a boot log that dumps them right after serial comes up, again just before disable_car(), and once more before loading stage2. After that mtrrs aren't touched until stage 2 phase 6 AFAIK. I wish I knew more about this stuff, I should probably make some time to read those Intel x86 programming guides.
Thanks, Corey
On Wed, Dec 24, 2008 at 6:25 AM, Corey Osgood corey.osgood@gmail.comwrote:
On Wed, Dec 24, 2008 at 5:20 AM, Corey Osgood corey.osgood@gmail.comwrote:
On Wed, Dec 24, 2008 at 5:09 AM, ron minnich rminnich@gmail.com wrote:
On Wed, Dec 24, 2008 at 12:14 AM, Corey Osgood corey.osgood@gmail.com wrote:
*phew* I was worried that was how v3 was always going to run. As for
those
times Ron, stage0 takes <1 second, initram is ~5 seconds,
the initram seems way too long. But that stage 2 tells me you have some real problems with your dram setup.
It could have been less, timing output from a serial port with a stopwatch isn't very exact. I don't think there's anything wrong with the dram setup, I ran memtest86 overnight without any errors.
I know nothing about this part. I wonder if you could dump your mtrr
settings and let us see them.
Sure, what point do you want them from? End of stage1?
Here's a boot log that dumps them right after serial comes up, again just before disable_car(), and once more before loading stage2. After that mtrrs aren't touched until stage 2 phase 6 AFAIK. I wish I knew more about this stuff, I should probably make some time to read those Intel x86 programming guides.
Alright, it was ridiculously simple. I added #defines for XIP_ROM_SIZE/BASE in stage0.S like core2 has, and now it boots MUCH MUCH faster, until it gets into phase6_init and does the MTRR init that I brought in from v2, after that it slows back down. I'll send a patch to set that properly and to fix it in core2 (currently hardcoded to a 1MB rom) soonish. Is the proper solution to the MTRR problem to teach that mtrr_init() not to mess up the caching MTRR, to set the caching back up after init, or just to not bother with the late init?
Thanks, Corey
On Tue, Jan 6, 2009 at 2:59 AM, Corey Osgood corey.osgood@gmail.com wrote:
it was ridiculously simple. I added #defines for XIP_ROM_SIZE/BASE in stage0.S like core2 has, and now it boots MUCH MUCH faster, until it gets into phase6_init and does the MTRR init that I brought in from v2, after that it slows back down. I'll send a patch to set that properly and to fix it in core2 (currently hardcoded to a 1MB rom) soonish. Is the proper solution to the MTRR problem to teach that mtrr_init() not to mess up the caching MTRR, to set the caching back up after init, or just to not bother with the late init?
I think we need to get this right, but you could try skipping the late init to see what happens.
I expect it will slow down once linux starts.
ron
On 06.01.2009 17:55, ron minnich wrote:
On Tue, Jan 6, 2009 at 2:59 AM, Corey Osgood corey.osgood@gmail.com wrote:
it was ridiculously simple. I added #defines for XIP_ROM_SIZE/BASE in stage0.S like core2 has, and now it boots MUCH MUCH faster, until it gets into phase6_init and does the MTRR init that I brought in from v2, after that it slows back down. I'll send a patch to set that properly and to fix it in core2 (currently hardcoded to a 1MB rom) soonish. Is the proper solution to the MTRR problem to teach that mtrr_init() not to mess up the caching MTRR, to set the caching back up after init, or just to not bother with the late init?
I think we need to get this right, but you could try skipping the late init to see what happens.
I expect it will slow down once linux starts.
The situation is a bit complicated. As long as CAR is active, we don't want the cacheable area (CAR+ROM) to be bigger than the cache to prevent cache evictions of CAR contents. That means we can cache the boot block and maybe initram. In that situation, having initram directly before the boot block is a huge speed benefit. Once CAR is no longer active, we immediately want to mark the whole ROM and RAM as cacheable to speed up decompression.
Regards, Carl-Daniel
On Tue, Jan 6, 2009 at 11:47 AM, Carl-Daniel Hailfinger c-d.hailfinger.devel.2006@gmx.net wrote:
As long as CAR is active, we don't want the cacheable area (CAR+ROM) to be bigger than the cache to prevent cache evictions of CAR contents. That means we can cache the boot block and maybe initram. In that situation, having initram directly before the boot block is a huge speed benefit. Once CAR is no longer active, we immediately want to mark the whole ROM and RAM as cacheable to speed up decompression.
Yes. And you want to have those MTRRs in a state that they'll be correct even if the OS you boot doesn't touch them.
One logical place is at the end of initram. I'd like to get the discussion to some agreement on where this code should be, and then we can worry about writing the code. Maybe I am missing part of the discussion but we seem to be going in circles at times :-)
As for the CAR+ROM area caching, no argument; what I think we're saying is we may need the ability to control where a LAR file ends up in ROM so we can limit the cacheable area. Does that make sense?
ron
On Wed, Jan 7, 2009 at 12:45 AM, ron minnich rminnich@gmail.com wrote:
On Tue, Jan 6, 2009 at 11:47 AM, Carl-Daniel Hailfinger c-d.hailfinger.devel.2006@gmx.net wrote:
As long as CAR is active, we don't want the cacheable area (CAR+ROM) to be bigger than the cache to prevent cache evictions of CAR contents.
Ah, that explains the reboots I'm seeing if the lar isn't zero-filled.
That means we can cache the boot block and maybe initram. In that situation, having initram directly before the boot block is a huge speed benefit. Once CAR is no longer active, we immediately want to mark the whole ROM and RAM as cacheable to speed up decompression.
Yes. And you want to have those MTRRs in a state that they'll be correct even if the OS you boot doesn't touch them.
One logical place is at the end of initram. I'd like to get the discussion to some agreement on where this code should be, and then we can worry about writing the code. Maybe I am missing part of the discussion but we seem to be going in circles at times :-)
Probably my fault, I've just been skimming through the list the last couple months while I was in class, and I head back next week, so I'd like to get this running before then ;) Although the end of initram makes sense (I was going to tuck it into disable_car()), if we only mark the first couple MB of ram and all of ROM as cachable there (some safe value that every system should have and coreboot will always fit inside) and then mark all of it later when we add the ram resources, we eliminate the need for chipset-specific code to tell coreboot the ram size twice. Does that make sense, or will it just complicate things more?
As for the CAR+ROM area caching, no argument; what I think we're
saying is we may need the ability to control where a LAR file ends up in ROM so we can limit the cacheable area. Does that make sense?
Perfect sense.
Thanks, Corey
On Wed, Jan 7, 2009 at 1:35 AM, Corey Osgood corey.osgood@gmail.com wrote:
On Wed, Jan 7, 2009 at 12:45 AM, ron minnich rminnich@gmail.com wrote:
On Tue, Jan 6, 2009 at 11:47 AM, Carl-Daniel Hailfinger c-d.hailfinger.devel.2006@gmx.net wrote:
As long as CAR is active, we don't want the cacheable area (CAR+ROM) to be bigger than the cache to prevent cache evictions of CAR contents.
Ah, that explains the reboots I'm seeing if the lar isn't zero-filled.
That means we can cache the boot block and maybe initram. In that situation, having initram directly before the boot block is a huge speed benefit. Once CAR is no longer active, we immediately want to mark the whole ROM and RAM as cacheable to speed up decompression.
Yes. And you want to have those MTRRs in a state that they'll be correct even if the OS you boot doesn't touch them.
One logical place is at the end of initram. I'd like to get the discussion to some agreement on where this code should be, and then we can worry about writing the code. Maybe I am missing part of the discussion but we seem to be going in circles at times :-)
Probably my fault, I've just been skimming through the list the last couple months while I was in class, and I head back next week, so I'd like to get this running before then ;) Although the end of initram makes sense (I was going to tuck it into disable_car()), if we only mark the first couple MB of ram and all of ROM as cachable there (some safe value that every system should have and coreboot will always fit inside) and then mark all of it later when we add the ram resources, we eliminate the need for chipset-specific code to tell coreboot the ram size twice. Does that make sense, or will it just complicate things more?
And actually, we already have a config variable and function to do that in stage1, the following should make it work (this is inside stage1_phase3). All that would be left would be to teach ram_resource to re-set the mtrrs, or use the mtrr setup functions I pulled in from v2. The next question concerns that though, everything I can find suggests that coreboot uses the range from 0MB-CBMEMK. What happens if/when it hits a reserved range? IE the range from 768k-1MB which is always reserved on vt8237r, IIRC.
Thanks, Corey
Index: arch/x86/stage1.c =================================================================== --- arch/x86/stage1.c (revision 1101) +++ arch/x86/stage1.c (working copy) @@ -332,6 +332,10 @@ /* Provide an easy way to check whether RAM is available. */ global_vars()->ram_available = 1;
+#ifdef CONFIG_MTRR + cache_cbmem(MTRR_TYPE_WRBACK); +#endif + // location and size of image. init_archive(&archive);
As for the CAR+ROM area caching, no argument; what I think we're
saying is we may need the ability to control where a LAR file ends up in ROM so we can limit the cacheable area. Does that make sense?
Perfect sense.
Thanks, Corey
On Wed, Jan 7, 2009 at 7:07 AM, Corey Osgood corey.osgood@gmail.com wrote:
And actually, we already have a config variable and function to do that in stage1, the following should make it work (this is inside stage1_phase3). All that would be left would be to teach ram_resource to re-set the mtrrs, or use the mtrr setup functions I pulled in from v2. The next question concerns that though, everything I can find suggests that coreboot uses the range from 0MB-CBMEMK. What happens if/when it hits a reserved range? IE the range from 768k-1MB which is always reserved on vt8237r, IIRC.
I'm not sure under what circumstances coreboot would use the reserved range. ron
On Tue, Dec 23, 2008 at 9:13 PM, Corey Osgood corey.osgood@gmail.com wrote:
What should I be doing where? I can enable, disable, or make the shadow ram dance like a monkey (yeah, Via hardware's got some odd features), but it makes no real difference in the boot speed.
I don't know but 15 mins means for certain you are not caching. I would be interested to know how long each stage -- stage0, initram, stage2 -- takes, can you get that info?
I wonder if moving mtrr out of stage0 was the right move?
The only other option AFAIK is to go back to ROMCC, CAR uses those MTRRs and enabling them breaks CAR.
No, we don't need to go back to romcc, we need to figure this out.
First off, MTRRs will need to be properly set up once car is disabled. I have this suspicion that you've removed the post-mtrr, pre-stage2 setup of mtrrs that is needed to make things go fast. I could be wrong. But there is no question that your caching is not right.
Is the whole flash copied to dram once, or are the segments copied as their needed? LAR is saying it's opening the segments directly from flash (0xfff00000).
no, it's not really saying that. It's telling you where it is looking and where it found them. But stage2 is copied to RAM and executed. And if stage 2 is taking more than, say, 30 seconds, then you have a very serious problem. How long does stage 2 take?
But those numbers are what you get when no caching is turned on at all.
ron
-
Carl-Daniel Hailfinger -
Corey Osgood -
ron minnich