xHCI support for x86

[Pitrolle Jean-Jacques]

      Hello *,
I try to integrate coreboot *libpayload usb stack* in a custom binary 
for x86.
I already succeed integration of *ehci* for *qemu* and *core 2 duo* 
platforms.

But things seems to be not so easy for *xhci*.
When I try to run coreboot master branch (with hash 8bf3f7a) on qemu 
(version 2.7.0)
with following line :
  % qemu-system-x86_64 -bios build/coreboot.rom -serial stdio -drive 
if=none,id=usbstick,file=/tmp/qemu_usb.disk -usb -device 
nec-usb-xhci,id=xhci -device usb-storage,bus=xhci.0,drive=usbstick

I get these output lines concerning xHCI controller (nec-usb-xhci) :
8<---
[..]
00:04.0 0194:1033.0 xHCI controller
xhci_init: regbase: 0xfe070000
xhci_init: caplen:  0x40
xhci_init: rtsoff:  0x1000
xhci_init: dboff:   0x2000
xhci_init: hciversion: 0.0
xhci_init: Unsupported xHCI version
[..]
--->8

The same result is reached with qemu-system-i386...

Is it the expected result on qemu?

I don't have any motherboard supported by coreboot, so I try a custom 
binary  on
a *Xeon* processor (processor D-1500).
The xHCI version is supported (hciversion: 1.0) but things turn bad when 
starting
xHCI controller i.e xhci->opreg->usbcmd |= USBCMD_RS in *xhci_start* : 
the program hangs...
The registers usbcmd and usbsts before this command seems to have proper 
values (a usb key
is connected.)
8<---
[..]
  xhci_start - usbcmd:0 - usbsts:11
[..]
--->8

This seems to be related to interrupter configuration. When I comment 
out the two lines
in *xhci_reinit*
8<---
[..]
     xhci->hcrreg->intrrs[0].erstba_lo = virt_to_phys(xhci->ev_ring_table);
     xhci->hcrreg->intrrs[0].erstba_hi = 0;
[..]
--->8
xhci_start works as expected i.e not hangs.

But the sequence following with *NOOP* to test command ring and event 
ring mechanism fails.
8<---
[..]
NOOP run #0
Transfer TRB (@0x8051700):
  PTR_L  0x00000000
  PTR_H  0x00000000
  STATUS 0x00000000
  CNTRL  0x00005c00
  TL     0x0000
  TDS    0x0000
  C      0x0000
  ISP    0x0000
  CH     0x0000
  IOC    0x0000
  IDT    0x0000
  TT     0x0017
  DIR    0x0000
Command 23 (@0x8051700)
Warning: Timed out waiting for TRB_EV_CMD_CMPL.
Command ring is running
[..]
--->8

As I understand it is mandatory to have the first interrupter configure 
to manage
at least one event ring (4.9.4 Event Ring Management - eXtensible Host 
Controller Interface
Revision 1.1).

Does someone succeed with xHCI support for x86 board?
Did I miss something concerning xHCI configuration?

Thanks.
Have a nice day.

-- 
Jean-Jacques PITROLLE
Phone: [phone]
SYSGO S.A.S

[Nico Huber]

      Hi,

On 23.11.2016 11:09, Pitrolle Jean-Jacques wrote:
  Hello *,
 I try to integrate coreboot *libpayload usb stack* in a custom binary
 for x86.
 I already succeed integration of *ehci* for *qemu* and *core 2 duo*
 platforms.
 
 But things seems to be not so easy for *xhci*.
 When I try to run coreboot master branch (with hash 8bf3f7a) on qemu
 (version 2.7.0)
 with following line :
  % qemu-system-x86_64 -bios build/coreboot.rom -serial stdio -drive
 if=none,id=usbstick,file=/tmp/qemu_usb.disk -usb -device
 nec-usb-xhci,id=xhci -device usb-storage,bus=xhci.0,drive=usbstick
 
 I get these output lines concerning xHCI controller (nec-usb-xhci) :
 8<---
 [..]
 00:04.0 0194:1033.0 xHCI controller
 xhci_init: regbase: 0xfe070000
 xhci_init: caplen:  0x40
 xhci_init: rtsoff:  0x1000
 xhci_init: dboff:   0x2000
 xhci_init: hciversion: 0.0
 xhci_init: Unsupported xHCI version
 [..]
 --->8
 
 The same result is reached with qemu-system-i386...
 
 Is it the expected result on qemu?
it looks odd to me. We wrote the ancestor of this xHCI driver against
qemu but stopped testing it when we switched to real hardware. I would
have expected an hciversion around 0.9*. The libpayload driver currently
checks for hciversion > 0.96. The values above hciversion look sane, so
I suspect  qemu to tell a bad version.

  
 I don't have any motherboard supported by coreboot, so I try a custom
 binary  on
How is this custom binary build? In which environment does it run? It's
hard to make assumptions about the driver without knowing the details.

  a *Xeon* processor (processor D-1500).
So you are using the xHCI controller built into the SoC? or an extension
card?

  The xHCI version is supported (hciversion: 1.0) but
things turn bad when
 starting
 xHCI controller i.e xhci->opreg->usbcmd |= USBCMD_RS in *xhci_start* :
 the program hangs...
 The registers usbcmd and usbsts before this command seems to have proper
 values (a usb key
 is connected.)
 8<---
 [..]
  xhci_start - usbcmd:0 - usbsts:11
 [..]
 --->8
 
 This seems to be related to interrupter configuration. When I comment
 out the two lines
 in *xhci_reinit*
 8<---
 [..]
     xhci->hcrreg->intrrs[0].erstba_lo = virt_to_phys(xhci->ev_ring_table);
I'd suspect that the address returned by virt_to_phys() is wrong or not
accessible by the controller.

      xhci->hcrreg->intrrs[0].erstba_hi = 0;
Are you sure it's a 32-bit address?

  [..]
 --->8
 xhci_start works as expected i.e not hangs.
 
 But the sequence following with *NOOP* to test command ring and event
 ring mechanism fails.
 8<---
 [..]
 NOOP run #0
 Transfer TRB (@0x8051700):
  PTR_L  0x00000000
  PTR_H  0x00000000
  STATUS 0x00000000
  CNTRL  0x00005c00
  TL     0x0000
  TDS    0x0000
  C      0x0000
  ISP    0x0000
  CH     0x0000
  IOC    0x0000
  IDT    0x0000
  TT     0x0017
  DIR    0x0000
 Command 23 (@0x8051700)
 Warning: Timed out waiting for TRB_EV_CMD_CMPL.
 Command ring is running
 [..]
 --->8
 
 As I understand it is mandatory to have the first interrupter configure
 to manage
 at least one event ring (4.9.4 Event Ring Management - eXtensible Host
 Controller Interface
 Revision 1.1).
 
 Does someone succeed with xHCI support for x86 board?
 Did I miss something concerning xHCI configuration?
It's working in coreboot payloads in many production systems.

Nico

[Pitrolle Jean-Jacques]

      Hello *,
Sorry for delay, but i have to understand a lot things before answering (:

On 23/11/2016 14:25, Nico Huber wrote:
  Hi,

 On 23.11.2016 11:09, Pitrolle Jean-Jacques wrote:
  Hello *,
 I try to integrate coreboot *libpayload usb stack* in a custom binary
 for x86.
 I already succeed integration of *ehci* for *qemu* and *core 2 duo*
 platforms.

 But things seems to be not so easy for *xhci*.
 When I try to run coreboot master branch (with hash 8bf3f7a) on qemu
 (version 2.7.0)
 with following line :
   % qemu-system-x86_64 -bios build/coreboot.rom -serial stdio -drive
 if=none,id=usbstick,file=/tmp/qemu_usb.disk -usb -device
 nec-usb-xhci,id=xhci -device usb-storage,bus=xhci.0,drive=usbstick

 I get these output lines concerning xHCI controller (nec-usb-xhci) :
 8<---
 [..]
 00:04.0 0194:1033.0 xHCI controller
 xhci_init: regbase: 0xfe070000
 xhci_init: caplen:  0x40
 xhci_init: rtsoff:  0x1000
 xhci_init: dboff:   0x2000
 xhci_init: hciversion: 0.0
 xhci_init: Unsupported xHCI version
 [..]
 --->8

 The same result is reached with qemu-system-i386...

 Is it the expected result on qemu?
 it looks odd to me. We wrote the ancestor of
this xHCI driver against
 qemu but stopped testing it when we switched to real hardware. I would
 have expected an hciversion around 0.9*. The libpayload driver currently
 checks for hciversion > 0.96. The values above hciversion look sane, so
 I suspect  qemu to tell a bad version.
The functions that access virtual xHCI only works for 4 bytes access.
For example, if i read capability register
   - access to offset 0, size access 4 bytes : 1000040
   - access to offset 2, size access 2 bytes : 00  *wrong value*
   - access to offset 2 and 3, size access 1 bytes: 0 - 0 *wrong value*
The 8bit/16bit reads are not properly honored.

Moreover, the previous *used* qemu (2.7.0) compiled by my own with 
custom configuration
seems to be broken (i don't go too deep in investigation right now)..
Everything is ok with *upstream* qemu (hash : 00227f - version : 2.7.91 
_ v2.8.0-rc1-dirty)
  I don't
have any motherboard supported by coreboot, so I try a custom
 binary  on
 How is this custom binary build? In which environment does it run?
It's
 hard to make assumptions about the driver without knowing the details.

  a *Xeon* processor (processor D-1500).
 So
you are using the xHCI controller built into the SoC? or an extension
 card?
I use xHCI controller built into SoC.
  The xHCI
version is supported (hciversion: 1.0) but things turn bad when
 starting
 xHCI controller i.e xhci->opreg->usbcmd |= USBCMD_RS in *xhci_start* :
 the program hangs...
 The registers usbcmd and usbsts before this command seems to have proper
 values (a usb key
 is connected.)
 8<---
 [..]
   xhci_start - usbcmd:0 - usbsts:11
 [..]
 --->8

 This seems to be related to interrupter configuration. When I comment
 out the two lines
 in *xhci_reinit*
 8<---
 [..]
      xhci->hcrreg->intrrs[0].erstba_lo = virt_to_phys(xhci->ev_ring_table);
 I'd suspect that the address returned by virt_to_phys() is wrong or not
 accessible by the controller.
Hmm, seems not as it work properly with rigth qemu
version.
      
xhci->hcrreg->intrrs[0].erstba_hi = 0;
 Are you sure it's a 32-bit
address?
I used a 32 bits compiler, so i expect it..
Do i need to setup something else (in bios for example)?
  [..]
 --->8
 xhci_start works as expected i.e not hangs.

 But the sequence following with *NOOP* to test command ring and event
 ring mechanism fails.
 8<---
 [..]
 NOOP run #0
 Transfer TRB (@0x8051700):
   PTR_L  0x00000000
   PTR_H  0x00000000
   STATUS 0x00000000
   CNTRL  0x00005c00
   TL     0x0000
   TDS    0x0000
   C      0x0000
   ISP    0x0000
   CH     0x0000
   IOC    0x0000
   IDT    0x0000
   TT     0x0017
   DIR    0x0000
 Command 23 (@0x8051700)
 Warning: Timed out waiting for TRB_EV_CMD_CMPL.
 Command ring is running
 [..]
 --->8

 As I understand it is mandatory to have the first interrupter configure
 to manage
 at least one event ring (4.9.4 Event Ring Management - eXtensible Host
 Controller Interface
 Revision 1.1).

 Does someone succeed with xHCI support for x86 board?
 Did I miss something concerning xHCI configuration?
 It's working in coreboot
payloads in many production systems.
With values read from registers dboff (0x3000)
and rtsoff (0x2000) and 
not values provided in
datasheet (*wrong values*) write to USBCMD register to start controller 
works as expected : there
is no weird hanging.
The problem came from *extended capability register* xCAP. In my case it 
is mandatory to release xHCI
controller from BIOS. "HC BIOS Owned Semaphore" must be set to 0 and "HC 
OS Owned Semaphore" must
be set to 1.
  Nico
So things go ahead :
   - enumeration of a usb-key on *qemu* works properly.
   - *but* i still have a problem on Xeon (Broadwell-DE SoC).

The problem concerns event ring during enable slot command :
8<---
[..]
Command 9 (@0x8052240)
Transfer TRB (@0x8052240):
  PTR_L  0x00000000
  PTR_H  0x00000000
  STATUS 0x00000000
  CNTRL  0x00002400
  TL     0x0000
  TDS    0x0000
  C      0x0000
  ISP    0x0000
  CH     0x0000
  IOC    0x0000
  IDT    0x0000
  TT     0x0009
  DIR    0x0000
_____address:8052240 phys:d79240 low:0 high:0 s:0 c:8052240
--->8

The dump of TRB is located in *xhci_post_command* just after xhci_debug 
and before TRB_SET.

The line "_____address:8052240 phys:d79240 low:0 high:0 s:0 c:8052240" 
is located
in *xhci_wait_for_command_done* just before while 
(xhci_wait_for_event_type(xhci, TRB_EV_CMD_CMPL, &timeout_us))

Do i need to flush cache if event and command rings are located in 
"normal" memory?
I use a custom allocator and for now it is not possible to allocate 
memory as "uncachable"

Is is possible to check proper initialization of event and command rings?
I see nothing relevant in xHCI standard concerning this.

Thanks.

-- 
Jean-Jacques PITROLLE
Phone: [phone]
SYSGO S.A.S

[Julius Werner]

  Do i need to flush cache if event and command rings
are located in "normal"
 memory?
All Intel chipsets I know of support full cache snooping for the
integrated XHCI controller, so you shouldn't need any cache
management. Or are you trying to use an external (e.g. PCIe) XHCI
controller? In that case I think(?) you should still be able to
configure it to cache snoop, but I'm not really an expert in x86
peripherals.

  I use a custom allocator and for now it is not
possible to allocate memory as "uncachable"
If your controller really can't cache snoop, then this is the only
supported mechanism to deal with it, sorry. You should be able to set
aside a separate memory region and configure it as uncacheable through
MTRRs or something. (Standard coreboot/libpayload does this by
configuring the memory region in coreboot and then informing
libpayload of it via the LB_TAB_DMA coreboot table entry.