Assigning regular devices such as a graphics card to a VM works just fine, but when I try to assign an intel i350 virtual function I receive the error
"error: internal error: Invalid device 0000:07:10.0 iommu_group file /sys/bus/pci/devices/0000:07:10.0/iommu_group is not a symlink"
The PF's are split in to seperate IOMMU groups like every other pci-e device but for some while the VF's appear in lspci they aren't placed in IOMMU groups at all not even in the ones that the PF's are in.
Weird.
Note: PCI-e root ports all report ACS support and so does the NIC and I am using the /etc/modprobe.d/blahblah.conf max_vfs option to enable SR-IOV.
Help from intel is blood from a stone, so I am wondering if anyone else has this successfully working? (I just bought a null modem cable and I now have free time to supply whichever logs are needed)
It seems the PCI-e root ports on the KGPE-D16 have ARI which is needed for SR-IOV, however it is not reported via # lspci -vv I assume that is why the VF's are not assigned to IOMMU groups and thus can't be assigned.
I am running coreboot v4.5 (I forgot to note this as it is another late night)
Tim was the ARI feature implemented when you guys did the programming? https://support.amd.com/TechDocs/44549.pdf
Logs (thought I would separate them)
ARI-fwd+ on the root port devcap and devctl but it doesn't have ARI listed in capabilities, I am not sure if lspci should report that on a root port as there is so little documentation for IOMMU/SR-IOV on the internet
07:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter I350-T4 Physical Slot: 13 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin D routed to IRQ 53 NUMA node: 0 Region 0: Memory at fa800000 (32-bit, non-prefetchable) [size=1M] Region 3: Memory at fa90c000 (32-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] MSI-X: Enable+ Count=10 Masked- Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Device Serial Number (removed) Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- IOVSta: Migration- Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 03 VF offset: 128, stride: 4, Device ID: 1520 Supported Page Size: 00000553, System Page Size: 00000001 Region 0: Memory at fa9d0000 (32-bit, non-prefetchable) VF Migration: offset: 00000000, BIR: 0 Capabilities: [1a0 v1] Transaction Processing Hints Device specific mode supported Steering table in TPH capability structure Capabilities: [1d0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Kernel driver in use: igb Kernel modules: igb ============================ root port:
00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP2 Port 0) (prog-if 00 [Normal decode]) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 35 NUMA node: 0 Bus: primary=00, secondary=07, subordinate=07, sec-latency=0 Memory behind bridge: fa500000-fa9fffff Prefetchable memory behind bridge: 00000000f9100000-00000000f92fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+ SltCap: AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+ Slot #13, PowerLimit 0.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn+ PwrFlt- MRL- PresDet- CmdCplt+ HPIrq+ LinkChg+ Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+ RootCap: CRSVisible+ RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+ LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00000 Data: 40a2 Capabilities: [b0] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 5a10 Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+ Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [190 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport Kernel modules: shpchp
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/21/2017 02:57 AM, Taiidan@gmx.com wrote:
It seems the PCI-e root ports on the KGPE-D16 have ARI which is needed for SR-IOV, however it is not reported via # lspci -vv I assume that is why the VF's are not assigned to IOMMU groups and thus can't be assigned.
I am running coreboot v4.5 (I forgot to note this as it is another late night)
Tim was the ARI feature implemented when you guys did the programming? https://support.amd.com/TechDocs/44549.pdf
We did not explicitly enable nor verify the ARI feature as we did not need it for our internal machines. As always, this is something that we could definitely work on under contract, and I am always happy to review patches on Gerrit if you choose to tackle this on your own!
- -- Timothy Pearson Raptor Engineering +1 (415) 727-8645 (direct line) +1 (512) 690-0200 (switchboard) https://www.raptorengineering.com
On 01/23/2017 09:23 AM, Timothy Pearson wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/21/2017 02:57 AM, Taiidan@gmx.com wrote:
It seems the PCI-e root ports on the KGPE-D16 have ARI which is needed for SR-IOV, however it is not reported via # lspci -vv I assume that is why the VF's are not assigned to IOMMU groups and thus can't be assigned.
I am running coreboot v4.5 (I forgot to note this as it is another late night)
Tim was the ARI feature implemented when you guys did the programming? https://support.amd.com/TechDocs/44549.pdf
We did not explicitly enable nor verify the ARI feature as we did not need it for our internal machines. As always, this is something that we could definitely work on under contract, and I am always happy to review patches on Gerrit if you choose to tackle this on your own!
Timothy Pearson Raptor Engineering +1 (415) 727-8645 (direct line) +1 (512) 690-0200 (switchboard) https://www.raptorengineering.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJYhhHzAAoJEK+E3vEXDOFbwP0IAIbCI5HLMEYNmDOeYyO/IYQT eVFMs+Bd2Q9bJA+x6e0D8MXL6LQzCOjxcg8qbeS4UuI6Kq6HszHqmKKH6rusmPqD Ew2DCffmoQnvsshmL/YvqPSL1SZhYWjjNCdUrBNCAvJAZLEh9Tef4eKpVp9aHjt4 WBvrttU7uFyTf5zbWAPsOFfe2aQ0TNGlCgl+EOqJQWeIaHw+8gu6MPDOIc7aorJ/ 2wC6ReI9iybAAlI0ZfYvsYFtuypqmJEOYmC/9uaU8xYtKOvrNumRPeJw1gB/m1yD Z1joJMC9WQ/1b9+L9D9wRnhFrYp00e5GVZYXWbSM31v81zjg+GKThsAZYwpTnO4= =haoQ -----END PGP SIGNATURE-----
TY for info.
I finally found some real info on ARI. https://us-east.manta.joyent.com/jmc/public/opensolaris/ARChive/FWARC/2010/0... Apparently SR-IOV doesn't actually require it, it is only needed for more than 8 total functions per device. I will continue troubleshooting and get back to you, SR-IOV is an important feature for today's modern servers so I will try to figure it out.
I would DIY but hardware engineering is currently above my paygrade, but when I get a job I am definitely going to do some bankrollin'
-
Taiidan＠gmx.com -
Timothy Pearson