On 20.06.2009 21:38, ron minnich wrote:
On Fri, Jun 19, 2009 at 9:02 AM, Carl-Daniel Hailfinger c-d.hailfinger.devel.2006@gmx.net wrote:
However, this locking has to work when some CPUs are in CAR and others are in RAM. And that's the big problem.
Let's go back to this original statement. I think it's not correct. We had decided, on opteron, that bsp does all memory setup -- i.e. sets up all memory controllers on all sockets. IIRC the opteron code starts APs up when all of memory is functional. I believe our Intel startup does this too: BSP sets up memory, and when APs start, memory is working and usable for stack etc.
Unless I'm misreading the K8/Fam10 CAR code in v2, they are a bit peculiar. Please correct me if I'm wrong. 1. K8 code does not care at all if it runs on BSP or AP. Each core which enters v2/src/cpu/amd/car/cache_as_ram.inc locks the cache, clobbers the same area (all of CAR), then proceeds to call cache_as_ram_main() on each core which entered. This means if any AP ever enters v2/src/cpu/amd/car/cache_as_ram.inc, we either have each of them overwriting the stack of all other APs if they can access RAM, or they are indeed in CAR mode when the BSP is already in RAM mode (bad). 2. Fam10 code cares if it runs on BSP or AP. It takes care to give every core its own stack area (different locations), does not clobber the CAR area at all, then proceeds to call cache_as_ram_main() on each core which entered.
So if there is some code which makes sure no AP ever enters v2/src/cpu/amd/car/cache_as_ram.inc, we can kill off all AP handling code in there. However, if APs can enter v2/src/cpu/amd/car/cache_as_ram.inc, we either have some real corruption race conditions on K8 or your statement above (regarding usable memory) contradicts the code.
If what I am saying is true, that all APs run AFTER the BSP has set up memory, then there is no issue: all locking can be based in memory, since NO AP needs to run in CAR mode -- memory is working. IIRC this is how K8 actually works.
Then the only issue for locking is to find out if we are in CAR (ignore locking) or in RAM (use locking).
So can we confirm my idea, that no AP need ever run in CAR? If they do need to run in CAR, why?
Please see above. I'm now thoroughly confused, but if the situation is better than I hoped, I'm the first to celebrate.
It is really great that we talk about this, now I'm starting to understand multicore startup. Thanks!
Regards, Carl-Daniel