This patch series changes the SeaBIOS build to no longer permit one to provide a custom version string via "make VERSION=xyz". In its place the build will allow extra information to be added to the standard version string via a new "EXTRAVERSION" variable. For example: "make EXTRAVERSION=-coreboot"
In order to facilitate "reproducible" builds, this patch series also changes the build to no longer include the system hostname nor build time on default "clean" builds. Specifically, if git is available, the git repo is not in a "dirty" state, and gcc/binutils versions are successfully extracted, then the default version string will just contain the git version information. However, should any of the preceding tests not succeed, then the version will continue to include the hostname and build time.
-Kevin
Kevin O'Connor (3): build: Rework version generation; don't allow make version override build: Report gcc and binutils versions in debug log build: Generate "reproducible" version strings on "clean" builds
Makefile | 16 ++++---- scripts/buildversion.py | 107 ++++++++++++++++++++++++++++++++++++++++++++++++ scripts/buildversion.sh | 31 -------------- src/output.c | 1 + src/util.h | 2 +- src/version.c | 5 +++ vgasrc/vgainit.c | 1 + vgasrc/vgaversion.c | 6 +++ 8 files changed, 130 insertions(+), 39 deletions(-) create mode 100644 scripts/buildversion.py delete mode 100755 scripts/buildversion.sh create mode 100644 src/version.c create mode 100644 vgasrc/vgaversion.c
Convert the script to generate the build version from a shell script to a python script.
Remove the ability to override the version at build time via "make VERSION=xyz". Replace it with ability to add extra version information at build time via "make EXTRAVERSION=xyz".
Signed-off-by: Kevin O'Connor kevin@koconnor.net --- Makefile | 14 ++++++----- scripts/buildversion.py | 66 +++++++++++++++++++++++++++++++++++++++++++++++++ scripts/buildversion.sh | 31 ----------------------- src/version.c | 4 +++ vgasrc/vgaversion.c | 5 ++++ 5 files changed, 83 insertions(+), 37 deletions(-) create mode 100644 scripts/buildversion.py delete mode 100755 scripts/buildversion.sh create mode 100644 src/version.c create mode 100644 vgasrc/vgaversion.c
diff --git a/Makefile b/Makefile index 3448598..842c261 100644 --- a/Makefile +++ b/Makefile @@ -51,6 +51,8 @@ DIRS=src src/hw src/fw vgasrc cc-option=$(shell if test -z "`$(1) $(2) -S -o /dev/null -xc /dev/null 2>&1`" \ ; then echo "$(2)"; else echo "$(3)"; fi ;)
+EXTRAVERSION= + CPPFLAGS = -P -MD -MT $@
COMMONCFLAGS := -I$(OUT) -Isrc -Os -MD -g \ @@ -154,10 +156,10 @@ $(OUT)romlayout.o: src/romlayout.S $(OUT)autoconf.h $(OUT)asm-offsets.h @echo " Compiling (16bit) $@" $(Q)$(CC) $(CFLAGS16) -c -D__ASSEMBLY__ $< -o $@
-$(OUT)romlayout16.lds: $(OUT)ccode32flat.o $(OUT)code32seg.o $(OUT)ccode16.o $(OUT)romlayout.o scripts/layoutrom.py scripts/buildversion.sh +$(OUT)romlayout16.lds: $(OUT)ccode32flat.o $(OUT)code32seg.o $(OUT)ccode16.o $(OUT)romlayout.o src/version.c scripts/layoutrom.py scripts/buildversion.py @echo " Building ld scripts" - $(Q)BUILD_VERSION="$(VERSION)" ./scripts/buildversion.sh $(OUT)version.c - $(Q)$(CC) $(CFLAGS32FLAT) -c $(OUT)version.c -o $(OUT)version.o + $(Q)$(PYTHON) ./scripts/buildversion.py -e "$(EXTRAVERSION)" $(OUT)autoversion.h + $(Q)$(CC) $(CFLAGS32FLAT) -c src/version.c -o $(OUT)version.o $(Q)$(LD) $(LD32BIT_FLAG) -r $(OUT)ccode32flat.o $(OUT)version.o -o $(OUT)code32flat.o $(Q)$(LD) $(LD32BIT_FLAG) -r $(OUT)ccode16.o $(OUT)romlayout.o -o $(OUT)code16.o $(Q)$(OBJDUMP) -thr $(OUT)code32flat.o > $(OUT)code32flat.o.objdump @@ -226,10 +228,10 @@ $(OUT)vgaentry.o: vgasrc/vgaentry.S $(OUT)autoconf.h $(OUT)asm-offsets.h @echo " Compiling (16bit) $@" $(Q)$(CC) $(CFLAGS16) -c -D__ASSEMBLY__ $< -o $@
-$(OUT)vgarom.o: $(OUT)vgaccode16.o $(OUT)vgaentry.o $(OUT)vgasrc/vgalayout.lds scripts/buildversion.sh +$(OUT)vgarom.o: $(OUT)vgaccode16.o $(OUT)vgaentry.o $(OUT)vgasrc/vgalayout.lds vgasrc/vgaversion.c scripts/buildversion.py @echo " Linking $@" - $(Q)BUILD_VERSION="$(VERSION)" ./scripts/buildversion.sh $(OUT)vgaversion.c VAR16 - $(Q)$(CC) $(CFLAGS16) -c $(OUT)vgaversion.c -o $(OUT)vgaversion.o + $(Q)$(PYTHON) ./scripts/buildversion.py -e "$(EXTRAVERSION)" $(OUT)autovgaversion.h + $(Q)$(CC) $(CFLAGS16) -c vgasrc/vgaversion.c -o $(OUT)vgaversion.o $(Q)$(LD) --gc-sections -T $(OUT)vgasrc/vgalayout.lds $(OUT)vgaccode16.o $(OUT)vgaentry.o $(OUT)vgaversion.o -o $@
$(OUT)vgabios.bin.raw: $(OUT)vgarom.o diff --git a/scripts/buildversion.py b/scripts/buildversion.py new file mode 100644 index 0000000..85d69b6 --- /dev/null +++ b/scripts/buildversion.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python +# Generate version information for a program +# +# Copyright (C) 2015 Kevin O'Connor kevin@koconnor.net +# +# This file may be distributed under the terms of the GNU GPLv3 license. +import sys, os, subprocess, time, socket, optparse + +VERSION_FORMAT = """ +/* DO NOT EDIT! This is an autogenerated file. See scripts/buildversion.py. */ +#define BUILD_VERSION "%s" +""" + +# Obtain version info from "git" program +def git_version(): + if not os.path.exists('.git'): + return "" + params = "git describe --tags --long --dirty".split() + try: + ver = subprocess.check_output(params).strip() + except: + return "" + return ver + +# Look for version in a ".version" file +def file_version(): + if not os.path.isfile('.version'): + return "" + try: + f = open('.version', 'r') + ver = f.readline().strip() + f.close() + except: + return "" + return ver + +# Generate an output file with the version information +def write_version(outfile, version): + sys.stdout.write("Version: %s\n" % (version,)) + f = open(outfile, 'w') + f.write(VERSION_FORMAT % (version,)) + f.close() + +def main(): + usage = "%prog [options] <outputheader.h>" + opts = optparse.OptionParser(usage) + opts.add_option("-e", "--extra", dest="extra", default="", + help="extra version string to append to version") + + options, args = opts.parse_args() + if len(args) != 1: + opts.error("Incorrect arguments") + outfile = args[0] + + ver = git_version() + if not ver: + ver = file_version() + if not ver: + ver = "?" + btime = time.strftime("%Y%m%d_%H%M%S") + hostname = socket.gethostname() + ver = "%s-%s-%s%s" % (ver, btime, hostname, options.extra) + write_version(outfile, ver) + +if __name__ == '__main__': + main() diff --git a/scripts/buildversion.sh b/scripts/buildversion.sh deleted file mode 100755 index 516aff5..0000000 --- a/scripts/buildversion.sh +++ /dev/null @@ -1,31 +0,0 @@ -#!/bin/sh -# Script to generate a C file with version information. -OUTFILE="$1" -VAR16MODE="$2" - -# Extract version info -if [ -z "$BUILD_VERSION" ]; then - if [ -d .git -o -f .git ]; then - VERSION="`git describe --tags --long --dirty`" - elif [ -f .version ]; then - VERSION="`cat .version`" - else - VERSION="?" - fi - VERSION="${VERSION}-`date +"%Y%m%d_%H%M%S"`-`hostname`" -else - VERSION="$BUILD_VERSION" -fi -echo "Version: ${VERSION}" - -# Build header file -if [ "$VAR16MODE" = "VAR16" ]; then - cat > ${OUTFILE} <<EOF -#include "types.h" -char VERSION[] VAR16 = "${VERSION}"; -EOF -else - cat > ${OUTFILE} <<EOF -char VERSION[] = "${VERSION}"; -EOF -fi diff --git a/src/version.c b/src/version.c new file mode 100644 index 0000000..d8c266f --- /dev/null +++ b/src/version.c @@ -0,0 +1,4 @@ +// Place build generated version into a C variable +#include "autoversion.h" + +char VERSION[] = BUILD_VERSION; diff --git a/vgasrc/vgaversion.c b/vgasrc/vgaversion.c new file mode 100644 index 0000000..02c8ea3 --- /dev/null +++ b/vgasrc/vgaversion.c @@ -0,0 +1,5 @@ +// Place build generated version into a C variable +#include "autovgaversion.h" +#include "types.h" + +char VERSION[] VAR16 = BUILD_VERSION;
Attempt to extract the gcc and binutils versions. Report that information in the debug log.
Signed-off-by: Kevin O'Connor kevin@koconnor.net --- Makefile | 6 +++--- scripts/buildversion.py | 47 +++++++++++++++++++++++++++++++++++++++++++---- src/output.c | 1 + src/util.h | 2 +- src/version.c | 1 + vgasrc/vgainit.c | 1 + vgasrc/vgaversion.c | 1 + 7 files changed, 51 insertions(+), 8 deletions(-)
diff --git a/Makefile b/Makefile index 842c261..46dbc96 100644 --- a/Makefile +++ b/Makefile @@ -22,8 +22,8 @@ LD=$(CROSS_PREFIX)ld OBJCOPY=$(CROSS_PREFIX)objcopy OBJDUMP=$(CROSS_PREFIX)objdump STRIP=$(CROSS_PREFIX)strip +CPP=$(CROSS_PREFIX)cpp PYTHON=python -CPP=cpp IASL:=iasl LD32BIT_FLAG:=-melf_i386
@@ -158,7 +158,7 @@ $(OUT)romlayout.o: src/romlayout.S $(OUT)autoconf.h $(OUT)asm-offsets.h
$(OUT)romlayout16.lds: $(OUT)ccode32flat.o $(OUT)code32seg.o $(OUT)ccode16.o $(OUT)romlayout.o src/version.c scripts/layoutrom.py scripts/buildversion.py @echo " Building ld scripts" - $(Q)$(PYTHON) ./scripts/buildversion.py -e "$(EXTRAVERSION)" $(OUT)autoversion.h + $(Q)$(PYTHON) ./scripts/buildversion.py -e "$(EXTRAVERSION)" -t "$(CC);$(AS);$(LD);$(OBJCOPY);$(OBJDUMP);$(STRIP);$(CPP)" $(OUT)autoversion.h $(Q)$(CC) $(CFLAGS32FLAT) -c src/version.c -o $(OUT)version.o $(Q)$(LD) $(LD32BIT_FLAG) -r $(OUT)ccode32flat.o $(OUT)version.o -o $(OUT)code32flat.o $(Q)$(LD) $(LD32BIT_FLAG) -r $(OUT)ccode16.o $(OUT)romlayout.o -o $(OUT)code16.o @@ -230,7 +230,7 @@ $(OUT)vgaentry.o: vgasrc/vgaentry.S $(OUT)autoconf.h $(OUT)asm-offsets.h
$(OUT)vgarom.o: $(OUT)vgaccode16.o $(OUT)vgaentry.o $(OUT)vgasrc/vgalayout.lds vgasrc/vgaversion.c scripts/buildversion.py @echo " Linking $@" - $(Q)$(PYTHON) ./scripts/buildversion.py -e "$(EXTRAVERSION)" $(OUT)autovgaversion.h + $(Q)$(PYTHON) ./scripts/buildversion.py -e "$(EXTRAVERSION)" -t "$(CC);$(AS);$(LD);$(OBJCOPY);$(OBJDUMP);$(STRIP);$(CPP)" $(OUT)autovgaversion.h $(Q)$(CC) $(CFLAGS16) -c vgasrc/vgaversion.c -o $(OUT)vgaversion.o $(Q)$(LD) --gc-sections -T $(OUT)vgasrc/vgalayout.lds $(OUT)vgaccode16.o $(OUT)vgaentry.o $(OUT)vgaversion.o -o $@
diff --git a/scripts/buildversion.py b/scripts/buildversion.py index 85d69b6..baaae57 100644 --- a/scripts/buildversion.py +++ b/scripts/buildversion.py @@ -4,11 +4,12 @@ # Copyright (C) 2015 Kevin O'Connor kevin@koconnor.net # # This file may be distributed under the terms of the GNU GPLv3 license. -import sys, os, subprocess, time, socket, optparse +import sys, os, subprocess, time, socket, optparse, re
VERSION_FORMAT = """ /* DO NOT EDIT! This is an autogenerated file. See scripts/buildversion.py. */ #define BUILD_VERSION "%s" +#define BUILD_TOOLS "%s" """
# Obtain version info from "git" program @@ -35,23 +36,61 @@ def file_version(): return ver
# Generate an output file with the version information -def write_version(outfile, version): +def write_version(outfile, version, toolstr): sys.stdout.write("Version: %s\n" % (version,)) f = open(outfile, 'w') - f.write(VERSION_FORMAT % (version,)) + f.write(VERSION_FORMAT % (version, toolstr)) f.close()
+re_gcc = re.compile(r'^(?P<prog>.*) (GCC) (?P<version>.*)$') +re_binutils = re.compile(r'^GNU (?P<prog>.*) version (?P<version>.*)$') + +# Run "tool --version" for each specified tool and extract versions +def tool_versions(tools): + tools = [t.strip() for t in tools.split(';')] + gcc = binutils = "" + success = 0 + for tool in tools: + try: + ver = subprocess.check_output([tool, '--version']) + except: + continue + ver = ver.split('\n')[0] + m = re_gcc.match(ver) + if m: + ver = m.group('version') + if gcc and gcc != ver: + gcc = "mixed" + continue + gcc = ver + success += 1 + continue + m = re_binutils.match(ver) + if m: + ver = m.group('version') + if binutils and binutils != ver: + binutils = "mixed" + continue + binutils = ver + success += 1 + cleanbuild = binutils and gcc and success == len(tools) + return cleanbuild, "gcc: %s binutils: %s" % (gcc, binutils) + def main(): usage = "%prog [options] <outputheader.h>" opts = optparse.OptionParser(usage) opts.add_option("-e", "--extra", dest="extra", default="", help="extra version string to append to version") + opts.add_option("-t", "--tools", dest="tools", default="", + help="list of build programs to extra version from")
options, args = opts.parse_args() if len(args) != 1: opts.error("Incorrect arguments") outfile = args[0]
+ cleanbuild, toolstr = tool_versions(options.tools) + ver = git_version() if not ver: ver = file_version() @@ -60,7 +99,7 @@ def main(): btime = time.strftime("%Y%m%d_%H%M%S") hostname = socket.gethostname() ver = "%s-%s-%s%s" % (ver, btime, hostname, options.extra) - write_version(outfile, ver) + write_version(outfile, ver, toolstr)
if __name__ == '__main__': main() diff --git a/src/output.c b/src/output.c index 45397b3..8a88388 100644 --- a/src/output.c +++ b/src/output.c @@ -30,6 +30,7 @@ void debug_banner(void) { dprintf(1, "SeaBIOS (version %s)\n", VERSION); + dprintf(1, "BUILD: %s\n", BUILDINFO); }
// Write a character to debug port(s). diff --git a/src/util.h b/src/util.h index 327abeb..cba3359 100644 --- a/src/util.h +++ b/src/util.h @@ -236,6 +236,6 @@ void vgahook_setup(struct pci_device *pci);
// version (auto generated file out/version.c) -extern const char VERSION[]; +extern const char VERSION[], BUILDINFO[];
#endif // util.h diff --git a/src/version.c b/src/version.c index d8c266f..a8a58cf 100644 --- a/src/version.c +++ b/src/version.c @@ -2,3 +2,4 @@ #include "autoversion.h"
char VERSION[] = BUILD_VERSION; +char BUILDINFO[] = BUILD_TOOLS; diff --git a/vgasrc/vgainit.c b/vgasrc/vgainit.c index 8d12261..9141470 100644 --- a/vgasrc/vgainit.c +++ b/vgasrc/vgainit.c @@ -150,6 +150,7 @@ vga_post(struct bregs *regs) { serial_debug_preinit(); dprintf(1, "Start SeaVGABIOS (version %s)\n", VERSION); + dprintf(1, "BUILD: %s\n", BUILDINFO); debug_enter(regs, DEBUG_VGA_POST);
if (CONFIG_VGA_PCI && !GET_GLOBAL(HaveRunInit)) { diff --git a/vgasrc/vgaversion.c b/vgasrc/vgaversion.c index 02c8ea3..1ef5ddb 100644 --- a/vgasrc/vgaversion.c +++ b/vgasrc/vgaversion.c @@ -3,3 +3,4 @@ #include "types.h"
char VERSION[] VAR16 = BUILD_VERSION; +char BUILDINFO[] VAR16 = BUILD_TOOLS;
If the build environment looks "clean" then don't add the build hostname or build time to the version string. This makes the default build string reproducible across builds.
Signed-off-by: Kevin O'Connor kevin@koconnor.net --- scripts/buildversion.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/scripts/buildversion.py b/scripts/buildversion.py index baaae57..d12d623 100644 --- a/scripts/buildversion.py +++ b/scripts/buildversion.py @@ -92,14 +92,16 @@ def main(): cleanbuild, toolstr = tool_versions(options.tools)
ver = git_version() + cleanbuild = cleanbuild and ver and 'dirty' not in ver if not ver: ver = file_version() if not ver: ver = "?" - btime = time.strftime("%Y%m%d_%H%M%S") - hostname = socket.gethostname() - ver = "%s-%s-%s%s" % (ver, btime, hostname, options.extra) - write_version(outfile, ver, toolstr) + if not cleanbuild: + btime = time.strftime("%Y%m%d_%H%M%S") + hostname = socket.gethostname() + ver = "%s-%s-%s" % (ver, btime, hostname) + write_version(outfile, ver + options.extra, toolstr)
if __name__ == '__main__': main()
On Tue, Oct 13, 2015 at 04:20:26PM -0400, Kevin O'Connor wrote:
This patch series changes the SeaBIOS build to no longer permit one to provide a custom version string via "make VERSION=xyz". In its place the build will allow extra information to be added to the standard version string via a new "EXTRAVERSION" variable. For example: "make EXTRAVERSION=-coreboot"
In order to facilitate "reproducible" builds, this patch series also changes the build to no longer include the system hostname nor build time on default "clean" builds. Specifically, if git is available, the git repo is not in a "dirty" state, and gcc/binutils versions are successfully extracted, then the default version string will just contain the git version information. However, should any of the preceding tests not succeed, then the version will continue to include the hostname and build time.
FYI, I committed this series.
-Kevin
Kevin,
what toolchain is needed to generate a clean build / without hostname/datestamp in the version string? Can coreboots toolchain be used? I'm using Ubuntu 15.04's default apparently (I'd have to check versions), which results in cleanbuild = false.
thanks, Matt
On 10/21/2015 12:38:25 PM, "Kevin O'Connor" kevin@koconnor.net wrote:
On Tue, Oct 13, 2015 at 04:20:26PM -0400, Kevin O'Connor wrote:
This patch series changes the SeaBIOS build to no longer permit one to provide a custom version string via "make VERSION=xyz". In its place the build will allow extra information to be added to the standard version string via a new "EXTRAVERSION" variable. For example: "make EXTRAVERSION=-coreboot"
In order to facilitate "reproducible" builds, this patch series also changes the build to no longer include the system hostname nor build time on default "clean" builds. Specifically, if git is available, the git repo is not in a "dirty" state, and gcc/binutils versions are successfully extracted, then the default version string will just contain the git version information. However, should any of the preceding tests not succeed, then the version will continue to include the hostname and build time.
FYI, I committed this series.
-Kevin
SeaBIOS mailing list SeaBIOS@seabios.org http://www.seabios.org/mailman/listinfo/seabios
On Wed, Oct 21, 2015 at 10:41:20PM +0000, Matt DeVillier wrote:
Kevin,
what toolchain is needed to generate a clean build / without hostname/datestamp in the version string? Can coreboots toolchain be used? I'm using Ubuntu 15.04's default apparently (I'd have to check versions), which results in cleanbuild = false.
Looks like there are greater differences in the various version strings than I was aware of. I think the patch below should fix it.
Thanks, -Kevin
commit f1356349b8c7b631b5c815d5d69f780fc733bcf6 Author: Kevin O'Connor kevin@koconnor.net Date: Wed Oct 21 20:35:50 2015 -0400
build: Be more permissive in buildversion.py version scan
There is some variation in version strings between various tool chain builds. Make the version tool scan more permissive to attempt to handle these variations.
Signed-off-by: Kevin O'Connor kevin@koconnor.net
diff --git a/scripts/buildversion.py b/scripts/buildversion.py index c3a83b0..da66151 100755 --- a/scripts/buildversion.py +++ b/scripts/buildversion.py @@ -4,7 +4,7 @@ # Copyright (C) 2015 Kevin O'Connor kevin@koconnor.net # # This file may be distributed under the terms of the GNU GPLv3 license. -import sys, os, subprocess, time, socket, optparse, re +import sys, os, subprocess, time, socket, optparse
VERSION_FORMAT = """ /* DO NOT EDIT! This is an autogenerated file. See scripts/buildversion.py. */ @@ -42,39 +42,33 @@ def write_version(outfile, version, toolstr): f.write(VERSION_FORMAT % (version, toolstr)) f.close()
-re_gcc = re.compile(r'^(?P<prog>.*) (GCC) (?P<version>.*)$') -re_binutils = re.compile(r'^GNU (?P<prog>.*) version (?P<version>.*)$') - # Run "tool --version" for each specified tool and extract versions def tool_versions(tools): tools = [t.strip() for t in tools.split(';')] - gcc = binutils = "" + versions = ['', ''] success = 0 for tool in tools: try: ver = subprocess.check_output([tool, '--version']).decode() except: continue - ver = ver.split('\n')[0] - m = re_gcc.match(ver) - if m: - ver = m.group('version') - if gcc and gcc != ver: - gcc = "mixed" - continue - gcc = ver - success += 1 + verstr = ver.split('\n')[0] + isbinutils = 0 + if verstr.startswith('GNU '): + isbinutils = 1 + verstr = verstr[4:] + if ' ' not in verstr: + continue + prog, ver = verstr.split(' ', 1) + if not prog or not ver: + continue + if versions[isbinutils] and versions[isbinutils] != ver: + vers[isbinutils] = "mixed" continue - m = re_binutils.match(ver) - if m: - ver = m.group('version') - if binutils and binutils != ver: - binutils = "mixed" - continue - binutils = ver - success += 1 - cleanbuild = binutils and gcc and success == len(tools) - return cleanbuild, "gcc: %s binutils: %s" % (gcc, binutils) + versions[isbinutils] = ver + success += 1 + cleanbuild = versions[0] and versions[1] and success == len(tools) + return cleanbuild, "gcc: %s binutils: %s" % (versions[0], versions[1])
def main(): usage = "%prog [options] <outputheader.h>"
On 10/21/2015 7:41:36 PM, "Kevin O'Connor" kevin@koconnor.net wrote:
On Wed, Oct 21, 2015 at 10:41:20PM +0000, Matt DeVillier wrote:
Kevin,
what toolchain is needed to generate a clean build / without hostname/datestamp in the version string? Can coreboots toolchain be used? I'm using Ubuntu 15.04's default apparently (I'd have to check versions), which results in cleanbuild = false.
Looks like there are greater differences in the various version strings than I was aware of. I think the patch below should fix it.
Thanks, -Kevin
Kevin,
this patch, along with the one you posted on IRC just prior, both produce a clean version string on my Ubuntu-based build machine
thanks, Matt
commit f1356349b8c7b631b5c815d5d69f780fc733bcf6 Author: Kevin O'Connor kevin@koconnor.net Date: Wed Oct 21 20:35:50 2015 -0400
build: Be more permissive in buildversion.py version scan There is some variation in version strings between various toolchain builds. Make the version tool scan more permissive to attempt to handle these variations.
Signed-off-by: Kevin O'Connor <kevin@koconnor.net>diff --git a/scripts/buildversion.py b/scripts/buildversion.py index c3a83b0..da66151 100755 --- a/scripts/buildversion.py +++ b/scripts/buildversion.py @@ -4,7 +4,7 @@ # Copyright (C) 2015 Kevin O'Connor kevin@koconnor.net # # This file may be distributed under the terms of the GNU GPLv3 license. -import sys, os, subprocess, time, socket, optparse, re +import sys, os, subprocess, time, socket, optparse
VERSION_FORMAT = """ /* DO NOT EDIT! This is an autogenerated file. See scripts/buildversion.py. */ @@ -42,39 +42,33 @@ def write_version(outfile, version, toolstr): f.write(VERSION_FORMAT % (version, toolstr)) f.close()
-re_gcc = re.compile(r'^(?P<prog>.*) (GCC) (?P<version>.*)$') -re_binutils = re.compile(r'^GNU (?P<prog>.*) version (?P<version>.*)$')
# Run "tool --version" for each specified tool and extract versions def tool_versions(tools): tools = [t.strip() for t in tools.split(';')]
- gcc = binutils = ""
- versions = ['', ''] success = 0 for tool in tools: try: ver = subprocess.check_output([tool,
'--version']).decode() except: continue
ver = ver.split('\n')[0]m = re_gcc.match(ver)if m:ver = m.group('version')if gcc and gcc != ver:gcc = "mixed"continuegcc = versuccess += 1
verstr = ver.split('\n')[0]isbinutils = 0if verstr.startswith('GNU '):isbinutils = 1verstr = verstr[4:]if ' ' not in verstr:continueprog, ver = verstr.split(' ', 1)if not prog or not ver:continueif versions[isbinutils] and versions[isbinutils] != ver:vers[isbinutils] = "mixed" continue
m = re_binutils.match(ver)if m:ver = m.group('version')if binutils and binutils != ver:binutils = "mixed"continuebinutils = versuccess += 1- cleanbuild = binutils and gcc and success == len(tools)
- return cleanbuild, "gcc: %s binutils: %s" % (gcc, binutils)
versions[isbinutils] = versuccess += 1- cleanbuild = versions[0] and versions[1] and success == len(tools)
- return cleanbuild, "gcc: %s binutils: %s" % (versions[0],
versions[1])
def main(): usage = "%prog [options] <outputheader.h>"
On Thu, Oct 22, 2015 at 10:21:49PM +0000, Matt DeVillier wrote:
On 10/21/2015 7:41:36 PM, "Kevin O'Connor" kevin@koconnor.net wrote:
On Wed, Oct 21, 2015 at 10:41:20PM +0000, Matt DeVillier wrote:
what toolchain is needed to generate a clean build / without hostname/datestamp in the version string? Can coreboots toolchain be used? I'm using Ubuntu 15.04's default apparently (I'd have to check versions), which results in cleanbuild = false.
Looks like there are greater differences in the various version strings than I was aware of. I think the patch below should fix it.
this patch, along with the one you posted on IRC just prior, both produce a clean version string on my Ubuntu-based build machine
Thanks. I committed the patch.
-Kevin
Hi,
In order to facilitate "reproducible" builds, this patch series also changes the build to no longer include the system hostname nor build time on default "clean" builds. Specifically, if git is available, the git repo is not in a "dirty" state, and gcc/binutils versions are successfully extracted, then the default version string will just contain the git version information. However, should any of the preceding tests not succeed, then the version will continue to include the hostname and build time.
FYI, I committed this series.
Hmm, just noticed that only git builds are considered being "clean". That implies builds from release tarballs will *not* be considered clean. Was that intentional?
cheers, Gerd
On Thu, Oct 22, 2015 at 09:56:44AM +0200, Gerd Hoffmann wrote:
Hi,
In order to facilitate "reproducible" builds, this patch series also changes the build to no longer include the system hostname nor build time on default "clean" builds. Specifically, if git is available, the git repo is not in a "dirty" state, and gcc/binutils versions are successfully extracted, then the default version string will just contain the git version information. However, should any of the preceding tests not succeed, then the version will continue to include the hostname and build time.
FYI, I committed this series.
Hmm, just noticed that only git builds are considered being "clean". That implies builds from release tarballs will *not* be considered clean. Was that intentional?
My thinking was that it is too easy for a ".version" file to be inadvertently incorrect. That is, if one pulls down a release tarball and then modifies some files, the version in the binary is not going to reflect the fact that changes were made.
I didn't think it would be too painful to require the git repo for those desiring a reproducible build. The "git describe" tool is quite useful for getting a meaningful hash of the repo.
Do you think that will be a problem?
-Kevin
Hi,
Hmm, just noticed that only git builds are considered being "clean". That implies builds from release tarballs will *not* be considered clean. Was that intentional?
My thinking was that it is too easy for a ".version" file to be inadvertently incorrect. That is, if one pulls down a release tarball and then modifies some files, the version in the binary is not going to reflect the fact that changes were made.
I didn't think it would be too painful to require the git repo for those desiring a reproducible build. The "git describe" tool is quite useful for getting a meaningful hash of the repo.
Do you think that will be a problem?
Linux distro builds usually use release tarballs (plus possibly patches) not git checkouts. Having timestamp and hostname back in the version string then is a step backwards. Will have only the effect that distros start patching the build system again ...
One option I see is to consider builds clean in case EXTRAVERSION is present, so distros can simply set EXTRAVERSION to the rpm release.
Or add a new variable specifically for package build versioning, and possibly even set that automatically. rpm sets some environment variables in the build environment ....
cheers, Gerd
On Thu, Oct 22, 2015 at 04:16:04PM +0200, Gerd Hoffmann wrote:
Hmm, just noticed that only git builds are considered being "clean". That implies builds from release tarballs will *not* be considered clean. Was that intentional?
My thinking was that it is too easy for a ".version" file to be inadvertently incorrect. That is, if one pulls down a release tarball and then modifies some files, the version in the binary is not going to reflect the fact that changes were made.
I didn't think it would be too painful to require the git repo for those desiring a reproducible build. The "git describe" tool is quite useful for getting a meaningful hash of the repo.
Do you think that will be a problem?
Linux distro builds usually use release tarballs (plus possibly patches) not git checkouts. Having timestamp and hostname back in the version string then is a step backwards. Will have only the effect that distros start patching the build system again ...
One option I see is to consider builds clean in case EXTRAVERSION is present, so distros can simply set EXTRAVERSION to the rpm release.
So, consider the build clean if git does not exist, ".version" does exist, and EXTRAVERSION is set? That seems reasonable, if a bit convoluted. The key would be letting distribution builders know about this and to document what information we expect to be contained in the EXTRAVERSION field.
Or add a new variable specifically for package build versioning, and possibly even set that automatically. rpm sets some environment variables in the build environment ....
That's also possible (buildversion.py could check for common package building environment variables), but I suspect that different distros use different variables.
-Kevin
Hi,
One option I see is to consider builds clean in case EXTRAVERSION is present, so distros can simply set EXTRAVERSION to the rpm release.
So, consider the build clean if git does not exist, ".version" does exist, and EXTRAVERSION is set? That seems reasonable, if a bit convoluted. The key would be letting distribution builders know about this and to document what information we expect to be contained in the EXTRAVERSION field.
Yes, that should be documented of course.
Or add a new variable specifically for package build versioning, and possibly even set that automatically. rpm sets some environment variables in the build environment ....
That's also possible (buildversion.py could check for common package building environment variables), but I suspect that different distros use different variables.
Depends on the packaging tool, not the distro, so it'll be the same for all rpm-based distros (redhat, suse, ...). dpkg has other variables, but again it should be same for all distros using it (debian, ubuntu, ...). So there shouldn't be *that* many different cases.
rpm sets these:
RPM_PACKAGE_VERSION (upstream version, should be identical to .version) RPM_PACKAGE_RELEASE (downstream version, gets bumbed each time something is changed on the package without upstream version change: patch added, config changed, rebuild with new gcc, whatever ...)
EXTRAVERSION="-${RPM_PACKAGE_RELEASE}" is what (rpm) distros should do for proper build tracking.
cheers, Gerd
On Thu, Oct 22, 2015 at 05:52:22PM +0200, Gerd Hoffmann wrote:
One option I see is to consider builds clean in case EXTRAVERSION is present, so distros can simply set EXTRAVERSION to the rpm release.
So, consider the build clean if git does not exist, ".version" does exist, and EXTRAVERSION is set? That seems reasonable, if a bit convoluted. The key would be letting distribution builders know about this and to document what information we expect to be contained in the EXTRAVERSION field.
Yes, that should be documented of course.
Okay - I sent a couple of patches to the list for review.
Or add a new variable specifically for package build versioning, and possibly even set that automatically. rpm sets some environment variables in the build environment ....
That's also possible (buildversion.py could check for common package building environment variables), but I suspect that different distros use different variables.
Depends on the packaging tool, not the distro, so it'll be the same for all rpm-based distros (redhat, suse, ...). dpkg has other variables, but again it should be same for all distros using it (debian, ubuntu, ...). So there shouldn't be *that* many different cases.
Well, there's also Arch and likely others. If someone wants to send a patch then we can take a look. My initial thought is that it seems a bit complex.
-Kevin
Kevin O'Connor wrote:
Depends on the packaging tool, not the distro,
..
So there shouldn't be *that* many different cases.
Well, there's also Arch and likely others. If someone wants to send a patch then we can take a look. My initial thought is that it seems a bit complex.
Yes. This belongs in packaging, not upstream.
It's not awesome that release tarballs aren't reproducible by default..
I mean, you do know that everyone else deals with the problem by replying "please try with latest upstream" to the first bug report with any ambiguity, right?
//Peter
-
Gerd Hoffmann -
Kevin O'Connor -
Matt DeVillier -
Peter Stuge