Re: [NixOS/nixpkgs] Build failure: diffoscobe broken by coreboot-utils 4.21 upgrade (Issue #256896)

[ehaouas]

Hi Ivan,

Please send an email to [e-mail]

Thx

De : "Ivan Jager"
A : "NixOS/nixpkgs" ,"Elyes HAOUAS" ,"Mention"
Envoyé: samedi 23 Septembre 2023 15:19
Objet : [NixOS/nixpkgs] Build failure: diffoscobe broken by coreboot-utils 4.21 upgrade (Issue #256896)

Steps To Reproduce

Steps to reproduce the behavior:

build diffoscope

Eg: nix-build -A diffoscope

Build log

https://gist.github.com/aij/ffc8e357eed2c6f4411910972ce0b920

Additional context

Broken since #251110 merged per git bisect

I looked into this a bit and I'm not sure if it's a diffoscope bug or a coreboot bug.

The failing tests generate test files with cbfstool (from coreboot) and the files generated by cbfstool 4.21 are using 4 bytes less space for the "header" (trailer) at the end of the file:

$ diffoscope coreboot1.rom.4.20 coreboot1.rom.4.21 
--- coreboot1.rom.4.20
+++ coreboot1.rom.4.21
@@ -2039,10 +2039,10 @@
 00007f60: ffff ffff ffff ffff ffff ffff ffff ffff  ................
 00007f70: ffff ffff ffff ffff ffff ffff ffff ffff  ................
 00007f80: ffff ffff ffff ffff ffff ffff ffff ffff  ................
 00007f90: ffff ffff ffff ffff ffff ffff ffff ffff  ................
 00007fa0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
 00007fb0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
 00007fc0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
-00007fd0: ffff ffff ffff ffff ffff ffff 4f52 4243  ............ORBC
-00007fe0: 3131 3132 0000 8000 0000 0000 0000 0040  1112...........@
-00007ff0: 0000 0000 0000 0001 ffff ffff dcff ffff  ................
+00007fd0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
+00007fe0: 4f52 4243 3131 3132 0000 8000 0000 0000  ORBC1112........
+00007ff0: 0000 0040 0000 0000 0000 0001 e0ff ffff  ...@............

As a result, CbfsFile.recognizes does not match.

My first thought was to change the matcher

@@ -139,7 +138,7 @@ class CbfsFile(File):
             rel_offset = struct.unpack("<i", f.read(4))[0]
             if (
                 rel_offset < 0
-                and -rel_offset > CBFS_HEADER_SIZE
+                and -rel_offset >= CBFS_HEADER_SIZE
                 and -rel_offset < size
             ):
                 f.seek(rel_offset, io.SEEK_END)

but then the rom2 tests still fail...

Looking into the relevant change on the coreboot side it looks like the change to the cbfs_header size may have been accidental. (C99 flexible arrays are nice for filename and data but not for fixed size padding...)

Anyway, even if this is a coreboot bug, it may still be useful to update diffutils to be able to recognize files produced by coreboot 4.21.

Notify maintainers

Diffoscope: @dezgeg @danielfullmer @RaitoBezarius

coreboot-utils: @felixsinger @yuka

Other: @ElyesH @jmbaur @felixsinger @lamby

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

ivan@tobati:~/nixops/aij/unstable$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.50, NixOS, 23.11 (Tapir), 23.11.git.bfb7dfec93f3`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - channels(root): `"nixos-21.11pre329341.9e70e9f732f"`
 - nixpkgs: `/etc/nixos/aij/unstable`

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID:

[NixOS/nixpkgs] Build failure: diffoscobe broken by coreboot-utils 4.21 upgrade (Issue #256896)

[Kyösti Mälkki]

Hi

Thanks, Ivan, for bisecting and also submitting the fix for this:

https://review.coreboot.org/c/coreboot/+/78239

Kyösti

On Mon, Oct 2, 2023 at 6:33 PM ehaouas  wrote:
> Hi Ivan,
>
> Please send an email to [e-mail]
>
> Thx
>
> De : "Ivan Jager"
> A : "NixOS/nixpkgs" ,"Elyes HAOUAS" ,"Mention"
> Envoyé: samedi 23 Septembre 2023 15:19
> Objet : [NixOS/nixpkgs] Build failure: diffoscobe broken by coreboot-utils
> 4.21 upgrade (Issue #256896)
>
> Steps To Reproduce
>
> Steps to reproduce the behavior:
> build diffoscope
> Eg: nix-build -A diffoscope
>
> Build log
>
> https://gist.github.com/aij/ffc8e357eed2c6f4411910972ce0b920
> Additional context
>
> Broken since #251110 https://github.com/NixOS/nixpkgs/pull/251110
> merged per git bisect
> https://github.com/NixOS/nixpkgs/issues/252045#issuecomment-1713881494
>
> I looked into this a bit and I'm not sure if it's a diffoscope bug or a
> coreboot bug.
>
> The failing tests generate test files with cbfstool (from coreboot) and
> the files generated by cbfstool 4.21 are using 4 bytes less space for the
> "header" (trailer) at the end of the file:
>
> $ diffoscope coreboot1.rom.4.20 coreboot1.rom.4.21 --- coreboot1.rom.4.20+++ coreboot1.rom.4.21@@ -2039,10 +2039,10 @@
>  00007f60: ffff ffff ffff ffff ffff ffff ffff ffff  ................
>  00007f70: ffff ffff ffff ffff ffff ffff ffff ffff  ................
>  00007f80: ffff ffff ffff ffff ffff ffff ffff ffff  ................
>  00007f90: ffff ffff ffff ffff ffff ffff ffff ffff  ................
>  00007fa0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
>  00007fb0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
>  00007fc0: ffff ffff ffff ffff ffff ffff ffff ffff  ................-00007fd0: ffff ffff ffff ffff ffff ffff 4f52 4243  ............ORBC-00007fe0: 3131 3132 0000 8000 0000 0000 0000 0040  1112...........@-00007ff0: 0000 0000 0000 0001 ffff ffff dcff ffff  ................+00007fd0: ffff ffff ffff ffff ffff ffff ffff ffff  ................+00007fe0: 4f52 4243 3131 3132 0000 8000 0000 0000  ORBC1112........+00007ff0: 0000 0040 0000 0000 0000 0001 e0ff ffff  ...@............
>
> As a result, CbfsFile.recognizes
> https://salsa.debian.org/reproducible-builds/diffoscope/-/blob/ac79ac5c3dfa8a2d1ec0e897e0940c3b997b284d/diffoscope/comparators/cbfs.py#L142
> does not match.
>
> My first thought was to change the matcher
>
> @@ -139,7 +138,7 @@ class CbfsFile(File):
>              rel_offset = struct.unpack("<i", f.read(4))[0]
>              if (
>                  rel_offset < 0-                and -rel_offset > CBFS_HEADER_SIZE+                and -rel_offset >= CBFS_HEADER_SIZE
>                  and -rel_offset < size
>              ):
>                  f.seek(rel_offset, io.SEEK_END)
>
> but then the rom2 tests still fail...
>
> Looking into the relevant change on the coreboot side
> https://github.com/coreboot/coreboot/commit/242bac0e162ca9a9258f61b0674b0dced5bcc01a
> it looks like the change to the cbfs_header size may have been
> accidental. (C99 flexible arrays are nice for filename and data but not
> for fixed size padding...)
>
> Anyway, even if this is a coreboot bug, it may still be useful to update
> diffutils to be able to recognize files produced by coreboot 4.21.
> Notify maintainers
>
> Diffoscope: @dezgeg https://github.com/dezgeg @danielfullmer
> https://github.com/danielfullmer @RaitoBezarius
> https://github.com/RaitoBezarius
>
> coreboot-utils: @felixsinger https://github.com/felixsinger @yuka
> https://github.com/yuka
>
> Other: @ElyesH https://github.com/ElyesH @jmbaur
> https://github.com/jmbaur @felixsinger https://github.com/felixsinger
> @lamby https://github.com/lamby
> Metadata
>
> Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
>
> ivan@tobati:~/nixops/aij/unstable$ nix-shell -p nix-info --run "nix-info -m" - system: `"x86_64-linux"` - host os: `Linux 6.1.50, NixOS, 23.11 (Tapir), 23.11.git.bfb7dfec93f3` - multi-user?: `yes` - sandbox: `yes` - version: `nix-env (Nix) 2.17.0` - channels(root): `"nixos-21.11pre329341.9e70e9f732f"` - nixpkgs: `/etc/nixos/aij/unstable`
>
> —
> Reply to this email directly, view it on GitHub
> https://github.com/NixOS/nixpkgs/issues/256896, or unsubscribe
> https://github.com/notifications/unsubscribe-auth/AIUYLHYAIZ432L4XL42F2H3X33OUNANCNFSM6AAAAAA5EGLQZY
> .
> You are receiving this because you were mentioned.Message ID:
>
> _______________________________________________
> coreboot mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]