Re: [coreboot] 16 GPUs on one board

[Zoran Stojsavljevic]

> Yep, I am another crypto currencies miner.  But in all truth,
> I find the hardware challenge more fun then the bitcoin stuff.

Thank you for conforming. Nothing wrong with it, as far as I can tell. :-)

But for the sake of time, you should have this setup working ASAP.
This is the aim, isn't it?
> Power is not the issue (any more).  I have 2Kw worth of PSU.
> 2X HP Common slot 750W PSU + Thermaltake 500W PSU.
> Currently, with all 8 cards running full tilt, across two motherboards,
> and I am drawing 960~1000W.  Those numbers are according to a
> Kill-A-Watt meter.

Thank you for the update.
> Hardware wise, this is all x86_64.

Which CPU are you using there? i3? i5? i7? Which code name, or Core number?

And how much of system memory are you using? I guess, not less then
8GB (in two $GB DIMMs).
> Arthur: Thanks for the details.  I have a board that with give me a
> "missing memory"  beep code with more then 6 GPUs.  Now I
> understand why!

So, each of the newest GTX 1070s need lot of memory from the host?! I
always thought that all these GFX cards have their own GFX memory,
dedicated to themselves?! And, YES, there is some system (buffer)
memory dedicated to GFX processing.
> How can I track down how much system DRAM a GPU is using?
> These are all the newest Nvidia Pascal based cards.  Mostly GTX 1070's.

I can tell, I am also interested in this to know!
> Is this just a BIOS level issue?  Or is there some hardware component I
> should be aware of?

BIOS issue? Might be. You need to find video memory buffer dedicated
for GFX cards. in this lieu, here is one perfect article for you:
http://smallbusiness.chron.com/change-memory-allocated-graphics-card-58676.h...

Namely paragraph: Changing the Memory Allocation

Typical values listed in the BIOS these days are 32MB, 64MB, 128MB,
256MB, 512MB and 1024MB. For your configuration, it is obvious that
you need to set in BIOS the maximum size: 1024MB (since you have 16
PCIe GFX cards to support)!

I hope this helps (waiting for you to report after you change the GFX
system (buffer) memory area)!

Zoran
_______

On Fri, Jan 5, 2018 at 5:51 AM, Adam Talbot  wrote:
> Yep, I am another crypto currencies miner.  But in all truth, I find the
> hardware challenge more fun then the bitcoin stuff.
>
> Power is not the issue (any more).  I have 2Kw worth of PSU. 2X HP Common
> slot 750W PSU + Thermaltake 500W PSU.  Currently, with all 8 cards running
> full tilt, across two motherboards, and I am drawing 960~1000W.  Those
> numbers are according to a Kill-A-Watt meter.
>
> Hardware wise, this is all x86_64.
>
> Arthur: Thanks for the details.  I have a board that with give me a "missing
> memory"  beep code with more then 6 GPUs.  Now I understand why!
>
> How can I track down how much system DRAM a GPU is using?  These are all the
> newest Nvidia Pascal based cards.  Mostly GTX 1070's.
>
> On an interesting note, one of my oldest motherboards, a Gigabyte
> GA-970A-UD3 will boot with all 8 cards, but gives me the no VGA beep code.
> Serial console for the win!
>
> Is this just a BIOS level issue?  Or is there some hardware component I
> should be aware of?
>
> Thanks for the help.
> -Adam
>
> On Thu, Jan 4, 2018 at 8:14 PM, Zoran Stojsavljevic
>  wrote:
> > > I am totally off the deep end and don't know where else to turn
> > > for help/advice.  I am trying to get 16 GPU's on one motherboard.
> >
> > Hmmmm. Yet another crypto currencies miner. ;-)
> > > Whenever I attach more then 3~5 GPU's to a single motherboard,
> > > it fails to post.  To make matters worse, my post code reader(s) don't
> > > seem to give me any good error codes.  Or at least nothing I can go on.
> >
> > You should have at minimum 1KW PSU for this job. At least... I guess,
> > even more (for 16 discrete GPUs) 2 x 1KW would be reasonable.
> >
> > Zoran
> > _______
> >
> > On Thu, Jan 4, 2018 at 8:38 PM, Adam Talbot  wrote:
> > > -Coreboot
> > > I am totally off the deep end and don't know where else to turn for
> > > help/advice.  I am trying to get 16 GPU's on one motherboard. Whenever I
> > > attach more then 3~5 GPU's to a single motherboard, it fails to post.
> > > To
> > > make matters worse, my post code reader(s) don't seem to give me any
> > > good
> > > error codes.  Or at least nothing I can go on.
> > >
> > > I am using PLX PEX8614 chips (PCIe 12X switch) to take 4 lanes and pass
> > > them
> > > to 8 GPU's, 1 lane per GPU. Bandwidth is not an issues as all my code
> > > runs
> > > native on the GPUs. Depending on the motherboard, I can get up to 5
> > > GPU's to
> > > post.  After many hours of debugging, googling, and trouble shooting, I
> > > am
> > > out of ideas.
> > >
> > > At this point I have no clue. I think there is a hardware, and a BIOS
> > > component? Can you help me understand the post process and where the
> > > hang up
> > > is occurring?  Do you think Coreboot will get around this hangup and, if
> > > so,
> > > can you advise a motherboard for me to test with?
> > >
> > > Its been a long time sense I last compiled linuxbios. ;-)
> > >
> > > Thanks
> > > -Adam