Dear coreboot folks,
Am Montag, den 24.06.2013, 13:45 -0600 schrieb David Hubbard:
To answer http://review.coreboot.org/3517 about what the vendor BIOS does for the IOMMU, here are my kernel logs (with the IOMMU enabled)
Please note: Command line: ... iommu=pt
I had to do that to work around (more like, bury my head in the sand) the problem with the on-board RTL8111/8168F, which trips over the IOMMU when utilization gets near 100% gigabit speed. I put the details on kernel bugzilla https://bugzilla.kernel.org/show_bug.cgi?id=55841
thank you a lot for collecting logs while Rudolf is busy. Also thank for sending the logs with Radeon instead FGLRX. For the archive, I would deem it a good thing to CC the list too.
Am Freitag, den 28.06.2013, 19:29 -0600 schrieb David Hubbard:
[…]
Attached are two logs, the first one is the one you requested (with the parameter "iommu=pt" removed from the kernel command line). It does look like there's an interesting line in there for PCI device 1:0.0, the APU integrated GPU:
[ 0.525376] AMD-Vi: Using passthrough domain for device 0000:01:00.0
The second log is with "iommu=pt" added back. In both cases, I loaded the radeon open source driver this time, instead of fglrx.
1. Please keep all in mind, that Rudolf tested with Linux 3.10-rc1 and you are using 3.8.11.
2. Looking through the log, the vendor BIOS is not perfect either. ;-)
[…] [ 0.000000] Checking aperture... [ 0.000000] No AGP bridge found [ 0.000000] Node 0: aperture @ 0 size 32 MB [ 0.000000] Your BIOS doesn't leave a aperture memory hole [ 0.000000] Please enable the IOMMU option in the BIOS setup [ 0.000000] This costs you 64 MB of RAM [ 0.000000] Mapping aperture over 65536 KB of RAM @ 84000000 [ 0.000000] Memory: 15783604k/18071552k available (3442k kernel code, 1869616k absent, 418332k reserved, 3061k data, 516k init) [ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 […]
So back to the problem, the solution to the PAGE_FAULT with coreboot seems “simple”. Using the vendor BIOS the graphics device (1:00.0) is put into passthrough domain automatically while with coreboot it is not. `amd_iommu.c` prints the message. (The line numbers are from Linux 3.10-rc7.)
AMD-Vi: Using passthrough domain for device 0000:01:00.0
$ nl -ba drivers/iommu/amd_iommu.c […] 3028 /* 3029 * The function for pre-allocating protection domains. 3030 * 3031 * If the driver core informs the DMA layer if a driver grabs a device 3032 * we don't need to preallocate the protection domains anymore. 3033 * For now we have to. 3034 */ 3035 static void __init prealloc_protection_domains(void) 3036 { 3037 struct iommu_dev_data *dev_data; 3038 struct dma_ops_domain *dma_dom; 3039 struct pci_dev *dev = NULL; 3040 u16 devid; 3041 3042 for_each_pci_dev(dev) { 3043 3044 /* Do we handle this device? */ 3045 if (!check_device(&dev->dev)) 3046 continue; 3047 3048 dev_data = get_dev_data(&dev->dev); 3049 if (!amd_iommu_force_isolation && dev_data->iommu_v2) { 3050 /* Make sure passthrough domain is allocated */ 3051 alloc_passthrough_domain(); 3052 dev_data->passthrough = true; 3053 attach_device(&dev->dev, pt_domain); 3054 pr_info("AMD-Vi: Using passthrough domain for device %s\n", 3055 dev_name(&dev->dev)); 3056 } […]
As no options were passed, the variable `dev_data->iommu_v2` seems to make the difference.
$ nl -ba drivers/iommu/amd_iommu_init.c […] 1185 if (!(iommu->cap & (1 << IOMMU_CAP_IOTLB))) 1186 amd_iommu_iotlb_sup = false; 1187 1188 /* read extended feature bits */ 1189 low = readl(iommu->mmio_base + MMIO_EXT_FEATURES); 1190 high = readl(iommu->mmio_base + MMIO_EXT_FEATURES + 4); 1191 1192 iommu->features = ((u64)high << 32) | low; […] 1214 if (iommu_feature(iommu, FEATURE_GT) && 1215 iommu_feature(iommu, FEATURE_PPR)) { 1216 iommu->is_iommu_v2 = true; 1217 amd_iommu_v2_present = true; 1218 } […]
`iommu_feature()` is implemented as follows.
$ nl -ba drivers/iommu/amd_iommu_proto.h […] 72 static inline bool is_rd890_iommu(struct pci_dev *pdev) 73 { 74 return (pdev->vendor == PCI_VENDOR_ID_ATI) && 75 (pdev->device == PCI_DEVICE_ID_RD890_IOMMU); 76 } 77 78 static inline bool iommu_feature(struct amd_iommu *iommu, u64 f) 79 { 80 if (!(iommu->cap & (1 << IOMMU_CAP_EFR))) 81 return false; 82 83 return !!(iommu->features & f); 84 } […]
So if the former is correct, for some reason `FEATURE_GT` or `FEATURE_PPR` are not enable when running with coreboot.
$ nl -ba drivers/iommu/amd_iommu_types.h […] 83 /* Extended Feature Bits */ 84 #define FEATURE_PREFETCH (1ULL<<0) 85 #define FEATURE_PPR (1ULL<<1) 86 #define FEATURE_X2APIC (1ULL<<2) 87 #define FEATURE_NX (1ULL<<3) 88 #define FEATURE_GT (1ULL<<4) 89 #define FEATURE_IA (1ULL<<6) 90 #define FEATURE_GA (1ULL<<7) 91 #define FEATURE_HE (1ULL<<8) 92 #define FEATURE_PC (1ULL<<9) 93 94 #define FEATURE_PASID_SHIFT 32 95 #define FEATURE_PASID_MASK (0x1fULL << FEATURE_PASID_SHIFT) 96 97 #define FEATURE_GLXVAL_SHIFT 14 98 #define FEATURE_GLXVAL_MASK (0x03ULL << FEATURE_GLXVAL_SHIFT) 99 100 #define PASID_MASK 0x000fffff […]
Probably adding some debug output to find out what features are enabled and disabled would be good. But maybe somebody already knows the solution from the above information.
Thanks,
Paul
-
Paul Menzel