Large mmio resources and 4G+ of RAM...

[Eric W. Biederman]

ron minnich <rminnich at lanl.gov> writes:

> On 1 Jun 2004, Eric W. Biederman wrote:
> 
> > ron minnich <rminnich at lanl.gov> writes:
> > 
> > > just FYI, making BARS live above the 32-bit limit will break every single
> > > linux cluster here at LANL, and will also disable Plan 9.
> > 
> > Only for the hardware where the BARs move.  
> 
> guess I missed something in your writeup. Are you saying that moving all
> BARs above 2^32 won't cause trouble for a 32-bit mode linux or freebsd or
> plan 9 or ...

You can't move all BARs just 64bit BARs.  So you will lose some devices
like infiniband but not everything.  And because of limited bars to
filter things through on logical pci-pci bridges I can't throw
every 64bit BAR above 4G.
 
> > > With BARS under the 32-bit limit, you can boot anything. With BARS above 
> > > that limit, you immediately limit what you can boot. 
> > 
> > Yes.  And largely I see that as a good thing.
> 
> But do the potential users of linuxbios systems that as a good thing? I
> understand the motivation, but sometimes customers (including us) do silly
> things, such as run K8s in 32-bit mode (it's actually a good thing as a K8
> is a better Xeon than a Xeon, at least for our programs).

I had customers 3 years ago who wanted 6GB of ram.  I have been
telling salesmen that they can't sell nodes with more than 4GB of
RAM several times a year since then, because you can't use more
RAM then that in 2 MPI processes.  Customers coming from 64bit boxes
don't get why commodity machines have the silly size limits on things,
that the hardware can reasonably do.

So the demand exists.

> As long as it is an option I don't mind however. 

At this stage it would be silly not to have an option.
 
> > > In principle, I like your BAR fix, but setting up BARS that are optimized
> > > for 64-bit kernels should be a (normally disabled)  option.
> > 
> > For old clusters I agree that it should be normally disabled.
> 
> for our new cluster (Lightning) first boot is into 32-bit linux for now, 
> and will likely stay that way for a while as that system runs as 32-bits 
> (not my decision ...)
> 
> This change can hurt new clusters. 

We are still in the transition where 64bit support is maturing but if we
don't push forward we will hurt more.  x86 boxes have been > 32bit for
quite a while.  

It is a lot easier for me to explain why things suck in a 32bit kernel
and not in a 64bit kernel then the other way around.

As for things like beoboot some news.

1) Kexec is slowly making it's way into the kernel, the system call
   number as of 2.6.7-rc1 is reserved in Linus's tree. 

2) Getting a kexec that will work on x86-64 is one of the things high
   on my TODO list..  It really isn't going to be hard I just need to 
   make a little time.

3) There already exist kexec ports to ppc32 and ppc64.

I have no intention to push stable production machines to 64bits, 
I might encourage but things that work don't need to be messed with.
For new machines especially looking into next year I want 64bit
kernels.  Even if they have a 32bit user space I want 64bit kernels.

Eric