Is there a way to enable a PCI-PCI bridge on bus 3 early?

[Keith Hui]

In my quest to get POST codes to show on my Asus P8Z77-M, and fixing
sb/bd82x6x [1] along the way, I'm still not quite getting them.

I have a ISA/PCI POST card, but it isn't giving me anything. I bought
a little "Debug Card Expert" off Amazon that looks like a
mini-PCI/mini-PCIe cross, but in today's testing I found out that it
isn't wired properly (i.e. isn't going to work) for PCIe at all, but I
did get it to show POST codes over LPC via the TPM header. I have
another of this same little PoS that broke in half on arrival. Now
that I know it only really works over LPC and only 9 signals, I'll
attempt to repair the broken traces and glue it together.

My board has a PCI-PCI bridge on bus:device.func 3:0.0 (so bus 3, this
will be important) that services the PCI slot. After applying [1] to
have the PCH send port 80 data to PCI as I specified in Kconfig, I
still don't get any POST code... and not over LPC either, so I know my
patch works.

Now I think I may need to enable CONFIG_EARLY_PCI_BRIDGE. Problem is
the bridge has to be a device on bus 0.

How can I enable this PCI bridge on bus 3 early so a POST card sitting
on it can get codes?

Thanks
Keith

[1] https://review.coreboot.org/c/coreboot/+/79725

[Nico Huber]

Hi Keith,

On 22.12.23 08:53, Keith Hui wrote:
> My board has a PCI-PCI bridge on bus:device.func 3:0.0 (so bus 3, this
> will be important) that services the PCI slot. After applying [1] to
> have the PCH send port 80 data to PCI as I specified in Kconfig, I
> still don't get any POST code... and not over LPC either, so I know my
> patch works.
>
> Now I think I may need to enable CONFIG_EARLY_PCI_BRIDGE. Problem is
> the bridge has to be a device on bus 0.

you'll have to enable all bridges on the path to your PCI device.
Starting with the bridge on bus 0. Note that all bus numbers except
bus 0 are decided by software. The EARLY_PCI_BRIDGE code is currently
not prepared for this. You'll have to
* configure more than one bus (repeat what the code currently does)
* configure PCI_SUBORDINATE_BUS everywhere to the highest bus number
* configure a port I/O range (code currently assumes MMIO)
  (I don't think there's a simple port 80h switch for PCI
   bridges, but could be wrong)

The code currently sets bus 15 (as a comment suggests, this is an
arbitrary number >0). For your case with the P8Z77-M, the procedure
could look like this:

For 00:1c.6:
* PCI_SECONDARY_BUS    => 15
* PCI_SUBORDINATE_BUS  => 16

For 0f:00.0:
* PCI_SECONDARY_BUS    => 16
* PCI_SUBORDINATE_BUS  => 16

For both bridges:
* PCI_IO_BASE          => 0x00
* PCI_IO_LIMIT         => 0x00 (lower bits are implied to be 1s)
* PCI_IO_BASE_UPPER16  => 0x00 (registers only exist if indicated
* PCI_IO_LIMIT_UPPER16 => 0x00  by PCI_IO_BASE[0..3] == 0x01)
* PCI_COMMAND          => PCI_COMMAND_IO

Then your device should show up as 10:00.0 (if the board uses
00 as slot number), in case you need to configure something there.
That's usually done in pci_early_device_probe().

It's probably not too much work to refactor the code to support
multiple bridges in a generic way. For instance, by allowing
CONFIG_EARLY_PCI_BRIDGE_DEVICE and _FUNCTION to be comma-
separated lists (having two arrays is a bit ugly, though...):

unsigned int bus = 0;
  const unsigned int max_subordinate = 255;
  unsigned int bridge_devs[] = { CONFIG_EARLY_PCI_BRIDGE_DEVICES };
  unsigned int bridge_funcs[] = { CONFIG_EARLY_PCI_BRIDGE_FUNCTIONS };
  _Static_assert (ARRAY_SIZE(bridge_devs) == ARRAY_SIZE(bridge_funcs));

for (int i = 0; i < ARRAY_SIZE(bridge_devs); ++i, ++bus) {
          const pci_devfn_t bridge = PCI_DEV(bus, bridge_devs[i],
bridge_funcs[i]);
          const unsigned int secondary_bus = bus + 1;
          /* do thing for each bridge */
  }

/* pci_early_device_probe() etc. */

Hope that helps,
Nico

[Keith Hui]

(Also including the list as I welcome more eyeballs.)

Hi Nico,

Thanks for your pointers.

On Fri, 22 Dec 2023 at 08:55, Nico Huber  wrote:
> Hi Keith,
>
> On 22.12.23 08:53, Keith Hui wrote:
> > My board has a PCI-PCI bridge on bus:device.func 3:0.0 (so bus 3, this
> > will be important) that services the PCI slot. After applying [1] to
> > have the PCH send port 80 data to PCI as I specified in Kconfig, I
> > still don't get any POST code... and not over LPC either, so I know my
> > patch works.
> >
> > Now I think I may need to enable CONFIG_EARLY_PCI_BRIDGE. Problem is
> > the bridge has to be a device on bus 0.
>
> you'll have to enable all bridges on the path to your PCI device.
> Starting with the bridge on bus 0. Note that all bus numbers except
> bus 0 are decided by software. The EARLY_PCI_BRIDGE code is currently
> not prepared for this. You'll have to
> configure more than one bus (repeat what the code currently does)
> configure PCI_SUBORDINATE_BUS everywhere to the highest bus number
> configure a port I/O range (code currently assumes MMIO)
> (I don't think there's a simple port 80h switch for PCI
>  bridges, but could be wrong)
>
> The code currently sets bus 15 (as a comment suggests, this is an
> arbitrary number >0). For your case with the P8Z77-M, the procedure
> could look like this:
>
> For 00:1c.6:
> PCI_SECONDARY_BUS    => 15
> PCI_SUBORDINATE_BUS  => 16
>
> For 0f:00.0:
> PCI_SECONDARY_BUS    => 16
> PCI_SUBORDINATE_BUS  => 16
>
> For both bridges:
> PCI_IO_BASE          => 0x00
> PCI_IO_LIMIT         => 0x00 (lower bits are implied to be 1s)
> PCI_IO_BASE_UPPER16  => 0x00 (registers only exist if indicated
> PCI_IO_LIMIT_UPPER16 => 0x00  by PCI_IO_BASE[0..3] == 0x01)
> PCI_COMMAND          => PCI_COMMAND_IO
>
> Then your device should show up as 10:00.0 (if the board uses
> 00 as slot number), in case you need to configure something there.
> That's usually done in pci_early_device_probe().

I came up with the code at the end of this email. However it renders
my board so inoperative I had to rescue myself. Can you see any
obvious errors in it?

I commented out the timeout part as I thought it was hanging me up. It was not.

Thanks
Keith

--------8<--------

void bootblock_mainboard_early_init(void)
{
    nuvoton_enable_serial(SERIAL_DEV, CONFIG_TTYS0_BASE);

if (CONFIG(POST_DEVICE_PCI_PCIE)) {
    /*
     * Enable PCIe root port 7 (00:1c.6) and the ASM1083 PCIe-PCI
bridge behind it
     * and have it accept I/O cycles. This should allow a POST card in
the PCI slot
     * to receive POST codes.
     *
     * Below is copied and modified from device/pci_early.c because
CONFIG_PCI_EARLY_BRIDGE
     * doesn't quite do what we need.
     */
        //int timeout;
        const int secondary = 3;
        pci_devfn_t bridge = PCH_PCIE_DEV(6);
        //pci_devfn_t p2p_bridge = PCI_DEV(secondary_bus, 0, 0);

/* Trigger secondary bus reset and program its bus number. */
        pci_s_assert_secondary_reset(bridge);
        pci_s_deassert_secondary_reset(bridge);
        pci_s_write_config8(bridge, PCI_SECONDARY_BUS, secondary);
        pci_s_write_config8(bridge, PCI_SUBORDINATE_BUS, secondary + 1);

u16 reg16;

/* Enable I/O window behind the bridge.
         * Base = 0x0000
         * Limit = 0x0FFF
         */
        pci_s_write_config16(bridge, PCI_IO_BASE, 0x0000);
        pci_or_config16(bridge, PCI_COMMAND, PCI_COMMAND_IO);

/* Enable subtractive decode (PECR3[SDE]). */
        pci_or_config8(bridge, 0xec, 0x01);
        /* Have to do it in D30F0 too, even though Z77 doesn't
natively do PCI. */
        pci_or_config32(PCI_DEV(0, 30, 0), 0x4c, (1 << 28));

/* Done with root port. Now do the 1083. */
        bridge = PCI_DEV(secondary, 0, 0);

/*        for (timeout = 20000; timeout; timeout--) {
            u32 id = pci_s_read_config32(bridge, PCI_VENDOR_ID);
            if (id != 0 && id != 0xffffffff && id != 0xffff0001)
                break;
            udelay(10);
        }*/

reg16 = pci_read_config16(bridge, PCI_CLASS_DEVICE);

if (reg16 != 0x0604)    /* PCI-PCI Bridge */
            return;

/* Trigger secondary bus reset and program its bus number. */
        pci_s_assert_secondary_reset(bridge);
        pci_s_deassert_secondary_reset(bridge);
        pci_s_write_config8(bridge, PCI_SECONDARY_BUS, secondary + 1);
        pci_s_write_config8(bridge, PCI_SUBORDINATE_BUS, secondary + 1);
        /* Set this second bridge to decode the same I/O ports. */
        pci_s_write_config16(bridge, PCI_IO_BASE, 0x0000);
        pci_or_config16(bridge, PCI_COMMAND, PCI_COMMAND_IO);
    }

}

[Nico Huber]

Hi again, and a happy new year!

On 03.01.24 05:44, Keith Hui wrote:
>     /* Trigger secondary bus reset and program its bus number. */
>     pci_s_assert_secondary_reset(bridge);
>     pci_s_deassert_secondary_reset(bridge);

I don't think this is actually necessary as everything should be
reset already. The actual procedure is more involved. There should
be a delay between assertion and de-assertion.
>     pci_s_write_config8(bridge, PCI_SECONDARY_BUS, secondary);
>     pci_s_write_config8(bridge, PCI_SUBORDINATE_BUS, secondary + 1);
>
>     u16 reg16;
> >     /* Enable I/O window behind the bridge.
>      * Base = 0x0000
>      * Limit = 0x0FFF
>      */
>     pci_s_write_config16(bridge, PCI_IO_BASE, 0x0000);
>     pci_or_config16(bridge, PCI_COMMAND, PCI_COMMAND_IO);

One thing I didn't consider, this will conflict with all the
other legacy I/O ports (e.g. RTC, UART). And it's not necessary,
as...
>     /* Enable subtractive decode (PECR3[SDE]). */
>     pci_or_config8(bridge, 0xec, 0x01);

...this should actually suffice for the root port. I didn't know
Intel implemented this. But given that they did, it's the best
option for your case. I don't know if you also have to set the
PCI_COMMAND_IO bit. If you do, set PCI_IO_BASE to 0x00ff (which
disables it to avoid conflicts by positive decoding).
>     /* Have to do it in D30F0 too, even though Z77 doesn't
>
> natively do PCI. */
>         pci_or_config32(PCI_DEV(0, 30, 0), 0x4c, (1 << 28));

But you should never enable this for more than one device on a
given bus. If your port 80 sink is behind the root port, don't
do this for the legacy PCI bridge.
>     /* Done with root port. Now do the 1083. */
>     bridge = PCI_DEV(secondary, 0, 0);
>
> /*        for (timeout = 20000; timeout; timeout--) {
>             u32 id = pci_s_read_config32(bridge, PCI_VENDOR_ID);
>             if (id != 0 && id != 0xffffffff && id != 0xffff0001)
>                 break;
>             udelay(10);
>         }*/
>     reg16 = pci_read_config16(bridge, PCI_CLASS_DEVICE);
>
>     if (reg16 != 0x0604)    /* PCI-PCI Bridge */
>         return;
>
>     /* Trigger secondary bus reset and program its bus number. */
>     pci_s_assert_secondary_reset(bridge);
>     pci_s_deassert_secondary_reset(bridge);
>     pci_s_write_config8(bridge, PCI_SECONDARY_BUS, secondary + 1);
>     pci_s_write_config8(bridge, PCI_SUBORDINATE_BUS, secondary + 1);
>     /* Set this second bridge to decode the same I/O ports. */
>     pci_s_write_config16(bridge, PCI_IO_BASE, 0x0000);
>     pci_or_config16(bridge, PCI_COMMAND, PCI_COMMAND_IO);

You should clear the PCI_SECONDARY_BUS registers afterwards as the
current early-PCI code does. This is to avoid conflicts with coreboot's
later PCI enumeration. Start with PCI_DEV(secondary, 0, 0), then
the root port.

Hope this helps,
Nico

[Keith Hui]

Happy new year to you too!

On Fri, 5 Jan 2024 at 09:58, Nico Huber  wrote:
> Hi again, and a happy new year!

With this code below I did just enough to enable subtractive decode in
the root port and legacy PCI bridge (datasheet does call for doing
both for legacy PCI mode) and nothing else. That ASM1083 just worked,
and I finally got POST codes on PCI slot.

However, I lost integrated graphics completely. A boot log is
attached. Can you see why this is happening?

Thanks
Keith

----8<----

void bootblock_mainboard_early_init(void)
{
    nuvoton_enable_serial(SERIAL_DEV, CONFIG_TTYS0_BASE);

if (CONFIG(POST_DEVICE_PCI_PCIE)) {
        u32 rpfn = RCBA32(RPFN);
        /* Unhide root port 7. */
        rpfn &= ~(1 << 27);
        RCBA32(RPFN) = rpfn;
        /* Get root port 7 function number. */
        unsigned int rpfunc = (rpfn >> 24) & 0x7;

pci_devfn_t bridge = PCH_PCIE_DEV(rpfunc);

/* Enable subtractive decode (PECR3[SDE]). */
        pci_or_config8(bridge, 0xec, 0x01);

/* Enable I/O but don't set a window. */
        pci_s_write_config16(bridge, PCI_IO_BASE, 0x00ff);
        pci_or_config16(bridge, PCI_COMMAND, PCI_COMMAND_IO);

/* Datasheet p.129: Have to enable subtractive decode in D30F0 too,
         * even though Z77 doesn't natively do PCI.
         */
        pci_or_config32(PCI_DEV(0, 30, 0), 0x4c, (1 << 28));

/* POST card in the PCI slot should now receive POST codes. */
    }
}

[Nico Huber]

On 11.01.24 07:03, Keith Hui wrote:
> Happy new year to you too!
>
> On Fri, 5 Jan 2024 at 09:58, Nico Huber  wrote:
> > Hi again, and a happy new year!
>
> With this code below I did just enough to enable subtractive decode in
> the root port and legacy PCI bridge (datasheet does call for doing
> both for legacy PCI mode) and nothing else. That ASM1083 just worked,
> and I finally got POST codes on PCI slot.

That is interesting, so the ASM1083 forwards port 80h without
configuration?
> However, I lost integrated graphics completely. A boot log is
> attached.

I fear it is not. Did you forget the attachment?

Can you see why this is happening?

At least I can't imagine anything that this little code could
do to VGA registers. Have you tried a graphics framebuffer?
Not sure if it can be configured without legacy i/o.
> Thanks
> Keith
>
> ----8<----
>
> void bootblock_mainboard_early_init(void)
> {
>     nuvoton_enable_serial(SERIAL_DEV, CONFIG_TTYS0_BASE);
> if (CONFIG(POST_DEVICE_PCI_PCIE)) {
>     u32 rpfn = RCBA32(RPFN);
>     /* Unhide root port 7. */
>     rpfn &= ~(1 << 27);
>     RCBA32(RPFN) = rpfn;
>     /* Get root port 7 function number. */
>     unsigned int rpfunc = (rpfn >> 24) & 0x7;
>
>     pci_devfn_t bridge = PCH_PCIE_DEV(rpfunc);
>
>     /* Enable subtractive decode (PECR3[SDE]). */
>     pci_or_config8(bridge, 0xec, 0x01);
>
>     /* Enable I/O but don't set a window. */
>     pci_s_write_config16(bridge, PCI_IO_BASE, 0x00ff);
>     pci_or_config16(bridge, PCI_COMMAND, PCI_COMMAND_IO);
>
>     /* Datasheet p.129: Have to enable subtractive decode in D30F0 too,
>      * even though Z77 doesn't natively do PCI.
>      */

I still read that it says only one device at a time. It might mention
D30F0 only due to copy-pasting from older documentation. OTOH, if this
indeed makes it work, I would be wrong.
>     pci_or_config32(PCI_DEV(0, 30, 0), 0x4c, (1 << 28));
>
>     /* POST card in the PCI slot should now receive POST codes. */
> }
>
> }

[Nico Huber]

On 11.01.24 14:43, Nico Huber via coreboot wrote:
> On 11.01.24 07:03, Keith Hui wrote:
> > However, I lost integrated graphics completely. A boot log is
> > attached.
>
> I fear it is not. Did you forget the attachment?

Found it now. It says early in ramstage:

[INFO ]  PCI: Static device PCI: 00:02.0 not found, disabling it.

So that's definitely an issue. But I really can't imagine
how this is happening, and why this particular device. As I
understand the hardware, all the decoding settings in the PCH
shouldn't affect the northbridge devices.

There's more

[DEBUG]  IGD decoded, subtracting 160M UMA and 0M GTT

0M isn't a good idea. This just prints what register settings
were found. Those were probably set in romstage (compiled into
bootblock with your settings if I interpreted the log correctly).

It looks all very odd and alas I have no clue how it could relate
to the subtractive-decode settings.

Nico

[Keith Hui]

On Thu, 11 Jan 2024 at 10:52, Nico Huber  wrote:
> > On Fri, 5 Jan 2024 at 09:58, Nico Huber  wrote:
> > > Hi again, and a happy new year!
> >
> > With this code below I did just enough to enable subtractive decode in
> > the root port and legacy PCI bridge (datasheet does call for doing
> > both for legacy PCI mode) and nothing else. That ASM1083 just worked,
> > and I finally got POST codes on PCI slot.
>
> That is interesting, so the ASM1083 forwards port 80h without
> configuration?

Pretty much. Documentation on this chip is very minimal - it only has
a register listing and their defaults, no explaining what any of them
do.

It too has a subtractive decode bit that defaults to off, but no
minute details around it.
> On 11.01.24 14:43, Nico Huber via coreboot wrote:
> > On 11.01.24 07:03, Keith Hui wrote:
> > > However, I lost integrated graphics completely. A boot log is
> > > attached.
> >
> > I fear it is not. Did you forget the attachment?
>
> Found it now. It says early in ramstage:
>
> [INFO ]  PCI: Static device PCI: 00:02.0 not found, disabling it.
>
> So that's definitely an issue. But I really can't imagine
> how this is happening, and why this particular device. As I
> understand the hardware, all the decoding settings in the PCH
> shouldn't affect the northbridge devices.
>
> There's more
>
> [DEBUG]  IGD decoded, subtracting 160M UMA and 0M GTT
>
> 0M isn't a good idea. This just prints what register settings
> were found. Those were probably set in romstage (compiled into
> bootblock with your settings if I interpreted the log correctly).
>
> It looks all very odd and alas I have no clue how it could relate
> to the subtractive-decode settings.

Me neither - I didn't know where this setting resides.

After giving an arbitrary out-of-the-way bus number (10) to the root
port, I finally got both POST codes and integrated graphic working.

Thanks Nico for your help so far - but wait, there's more!

Seems all the troubles I have encountered along the way:
System becoming inoperative - turns out it got hung up during PCI bus scan
Loss of integrated graphics
Loss of ability to soft reboot and power off
... are all the results of bus number conflict.

Another takeaway is I don't really need the legacy PCI bridge device
aka D30F0 at all. All my troubles old and new this guy doesn't seem to
make any difference.

Leaving this setup in, now I have a Sound Blaster Live! in the PCI
slot and it can't initialize.

Kernel reports:
pci 0000:0a:00.0: can't derive routing for PCI INT A
snd_emu10k1 0000:0b:00.0: PCI INT A: no GSI
genirq: Flags mismatch irq 0. 00000080 (snd_emu10k1) vs. 00015a00 (timer)
snd_emu10k1: probe of 0000:0b:00.0 failed with error -16
(that -16 is -EBUSY.)

Since I shut the I/O window on the (subtractive) root port and the PCI
code in ramstage didn't touch it, the sound card didn't have I/O
allocated and can't possibly work.

I also see that ramstage did not scan bus 10. Is it by design? Will I
have to probe for a device and abort this attempt if one is found? My
thinking is POST cards tend to be cheap designs that don't respond to
bus scans nor have a config space. I have no experience with PCIe POST
cards because I don't have one, and are more expensive starting at
$50.

How can I convince the PCI enumerator to work on this fudged root port
like any other?

Thanks
Keith