Yep, I am another crypto currencies miner.  But in all truth, I find the hardware challenge more fun then the bitcoin stuff.

Power is not the issue (any more).  I have 2Kw worth of PSU. 2X HP Common slot 750W PSU + Thermaltake 500W PSU.  Currently, with all 8 cards running full tilt, across two motherboards, and I am drawing 960~1000W.  Those numbers are according to a Kill-A-Watt meter.

Hardware wise, this is all x86_64.  

Arthur: Thanks for the details.  I have a board that with give me a "missing memory"  beep code with more then 6 GPUs.  Now I understand why!

How can I track down how much system DRAM a GPU is using?  These are all the newest Nvidia Pascal based cards.  Mostly GTX 1070's.

On an interesting note, one of my oldest motherboards, a Gigabyte GA-970A-UD3 will boot with all 8 cards, but gives me the no VGA beep code.  Serial console for the win!

Is this just a BIOS level issue?  Or is there some hardware component I should be aware of?

Thanks for the help.
-Adam


On Thu, Jan 4, 2018 at 8:14 PM, Zoran Stojsavljevic <zoran.stojsavljevic@gmail.com> wrote:
> I am totally off the deep end and don't know where else to turn
> for help/advice.  I am trying to get 16 GPU's on one motherboard.

Hmmmm. Yet another crypto currencies miner. ;-)

> Whenever I attach more then 3~5 GPU's to a single motherboard,
> it fails to post.  To make matters worse, my post code reader(s) don't
> seem to give me any good error codes.  Or at least nothing I can go on.

You should have at minimum 1KW PSU for this job. At least... I guess,
even more (for 16 discrete GPUs) 2 x 1KW would be reasonable.

Zoran
_______

On Thu, Jan 4, 2018 at 8:38 PM, Adam Talbot <ajtalbot1@gmail.com> wrote:
> -Coreboot
> I am totally off the deep end and don't know where else to turn for
> help/advice.  I am trying to get 16 GPU's on one motherboard. Whenever I
> attach more then 3~5 GPU's to a single motherboard, it fails to post.  To
> make matters worse, my post code reader(s) don't seem to give me any good
> error codes.  Or at least nothing I can go on.
>
> I am using PLX PEX8614 chips (PCIe 12X switch) to take 4 lanes and pass them
> to 8 GPU's, 1 lane per GPU. Bandwidth is not an issues as all my code runs
> native on the GPUs. Depending on the motherboard, I can get up to 5 GPU's to
> post.  After many hours of debugging, googling, and trouble shooting, I am
> out of ideas.
>
> At this point I have no clue. I think there is a hardware, and a BIOS
> component? Can you help me understand the post process and where the hang up
> is occurring?  Do you think Coreboot will get around this hangup and, if so,
> can you advise a motherboard for me to test with?
>
> Its been a long time sense I last compiled linuxbios. ;-)
>
> Thanks
> -Adam
>
> --
> coreboot mailing list: coreboot@coreboot.org
> https://mail.coreboot.org/mailman/listinfo/coreboot