Thank you, Timothy.
(2) get the PIKE 2008 card => will SATA3 work without non-free firmware?
No.
Ok, too bad. I thought the non-free firmware would only be needed for the SAS part, but I assume the LSI controller handles both SAS and SATA.
(3) put in some PCIe SATA3 card => any recommended chips that respect freedom?
There are very few. You can try some of the Marvell devices but you will still be limited by the host side bus as these old Opterons only support PCIe v2.
Yes, I was aware of the limitation. As far as I understand, the PCIe v2 x16 slot would allow for 8 GB/s - but that should be enough, even for a modern m.2 SSD. I just ordered a PCIe v2 x16 device with an Asmedia chipset to give it a try - seems to be supported in Linux and costs just 7€ incl. shipping.
(4) get a m.2 SSD instead together with some PCIe adapter => the cards don't have a co-processor, right?
Yes, they do. NVMe devices have an integrated proprietary controller to manage data storage / wear levelling.
(5) stay with SATA2 and live with the limited speed
Any recommendations for a freedom-respecting choice?
To be blunt, even your hard disks have an integrated (and hackable!) proprietary controller. I'd suggest going with m.2 and:
Yes, but you could say this about any 2.5" SATA SSD/HDD as well. Important for me is that the firmware runs isolated (on the drive) and does not have access to the host. As far as I understood this is the actual concern regarding the PIKE cards, right?
I haven't made up my mind to go with the m.2 option yet, because (1) the ssd cards are way more expensive than 2,5" sata drives and (2) was not sure what speed they can actually achieve with an adapter plugged to PCIe v2. Please correct me if I'm wrong, but, as far as I understood the firmware of the m.2 cards has direct pcie bus access because there is no controller (such as the marvell or asmedia) inbetween - right? If so, that would be a third reason to go for the sata3 pcie card for me.
Also, mandatory plug for Talos II here: these bottlenecks disappear on newer hardware and you don't have to accept the ME/PSP to get access to modern speeds. The KGPE-D16 is the last and most powerful owner-controllable x86 machine, but it is definitely showing its age in some areas.
I agree, SATA 2 and PCIe v2 are really getting old these days. Even if the G34 CPUs are somewhat powerful, that doesn't help much if the peripherials slow down the system.
Cheers, Daniel