Re: [coreboot] 16 GPUs on one board

[Adam Talbot]

Yep, I am another crypto currencies miner.  But in all truth, I find the
hardware challenge more fun then the bitcoin stuff.

Power is not the issue (any more).  I have 2Kw worth of PSU. 2X HP Common
slot 750W PSU + Thermaltake 500W PSU.  Currently, with all 8 cards running
full tilt, across two motherboards, and I am drawing 960~1000W.  Those
numbers are according to a Kill-A-Watt meter.

Hardware wise, this is all x86_64.

Arthur: Thanks for the details.  I have a board that with give me a
"missing memory"  beep code with more then 6 GPUs.  Now I understand why!

How can I track down how much system DRAM a GPU is using?  These are all
the newest Nvidia Pascal based cards.  Mostly GTX 1070's.

On an interesting note, one of my oldest motherboards, a
Gigabyte GA-970A-UD3 will boot with all 8 cards, but gives me the no VGA
beep code.  Serial console for the win!

Is this just a BIOS level issue?  Or is there some hardware component I
should be aware of?

Thanks for the help.
-Adam

On Thu, Jan 4, 2018 at 8:14 PM, Zoran Stojsavljevic <
[e-mail]> wrote:
> > I am totally off the deep end and don't know where else to turn
> > for help/advice.  I am trying to get 16 GPU's on one motherboard.
>
> Hmmmm. Yet another crypto currencies miner. ;-)
> > Whenever I attach more then 3~5 GPU's to a single motherboard,
> > it fails to post.  To make matters worse, my post code reader(s) don't
> > seem to give me any good error codes.  Or at least nothing I can go on.
>
> You should have at minimum 1KW PSU for this job. At least... I guess,
> even more (for 16 discrete GPUs) 2 x 1KW would be reasonable.
>
> Zoran
> _______
>
> On Thu, Jan 4, 2018 at 8:38 PM, Adam Talbot  wrote:
> > -Coreboot
> > I am totally off the deep end and don't know where else to turn for
> > help/advice.  I am trying to get 16 GPU's on one motherboard. Whenever I
> > attach more then 3~5 GPU's to a single motherboard, it fails to post.  To
> > make matters worse, my post code reader(s) don't seem to give me any good
> > error codes.  Or at least nothing I can go on.
> >
> > I am using PLX PEX8614 chips (PCIe 12X switch) to take 4 lanes and pass
>
> them
> > to 8 GPU's, 1 lane per GPU. Bandwidth is not an issues as all my code
>
> runs
> > native on the GPUs. Depending on the motherboard, I can get up to 5
>
> GPU's to
> > post.  After many hours of debugging, googling, and trouble shooting, I
>
> am
> > out of ideas.
> >
> > At this point I have no clue. I think there is a hardware, and a BIOS
> > component? Can you help me understand the post process and where the
>
> hang up
> > is occurring?  Do you think Coreboot will get around this hangup and, if
>
> so,
> > can you advise a motherboard for me to test with?
> >
> > Its been a long time sense I last compiled linuxbios. ;-)
> >
> > Thanks
> > -Adam