<p>Martin Roth has uploaded this change for <strong>review</strong>.</p><p><a href="https://review.coreboot.org/20874">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">[WIP] Update testing routines<br><br>Currently the only testing we had was 'what-jenkins-does' and<br>'make lint'. While the lint testing is suitable for developers,<br>the 'what-jenkins-does' target really isn't, as it was designed<br>specifically for testing on jenkins.<br><br>This adds the infrastructure for tests that are more suitable for the<br>developer.<br><br>TODO:<br>- Add save/restore of .config file<br>- Add additional QEMU testing<br>- Add additional abuild tests for flags that are typically disabled<br>- Get extended payload test working - Need to create config files<br>for each.<br><br>Change-Id: I19e1256d79531112ff84e47a307f55791533806f<br>Signed-off-by: Martin Roth <martinroth@google.com><br>---<br>M Makefile<br>M Makefile.inc<br>M payloads/libpayload/Makefile.inc<br>M util/testing/Makefile.inc<br>4 files changed, 110 insertions(+), 14 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://review.coreboot.org:29418/coreboot refs/changes/74/20874/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">diff --git a/Makefile b/Makefile<br>index ff9a6b8..343ec65 100644<br>--- a/Makefile<br>+++ b/Makefile<br>@@ -92,9 +92,7 @@<br> @echo ' distclean - Remove build artifacts and config files'<br> @echo ' doxygen - Build doxygen documentation for coreboot'<br> @echo ' doxyplatform - Build doxygen documentation for the current platform'<br>- @echo ' what-jenkins-does - Run platform build tests (Use CPUS=# for more cores)'<br> @echo ' printall - print makefile info for debugging'<br>- @echo ' lint / lint-stable - run coreboot lint tools (all / minimal)'<br> @echo ' gitconfig - set up git to submit patches to coreboot'<br> @echo ' ctags / ctags-project - make ctags file for all of coreboot or current board'<br> @echo ' cscope / cscope-project - make cscope.out file for coreboot or current board'<br>diff --git a/Makefile.inc b/Makefile.inc<br>index 9decd65..49344a6 100644<br>--- a/Makefile.inc<br>+++ b/Makefile.inc<br>@@ -75,7 +75,7 @@<br> <br> #######################################################################<br> # our phony targets<br>-PHONY+= clean-abuild coreboot lint lint-stable build-dirs build_complete<br>+PHONY+= clean-abuild coreboot build-dirs build_complete<br> <br> #######################################################################<br> # root source directories of coreboot<br>@@ -555,9 +555,6 @@<br> <br> update:<br> dongle.py -c /dev/term/1 $(obj)/coreboot.rom EOF<br>-<br>-lint lint-stable:<br>- util/lint/lint $@<br> <br> gitconfig:<br> [ -d .git ]<br>diff --git a/payloads/libpayload/Makefile.inc b/payloads/libpayload/Makefile.inc<br>index b88cd20..7787762 100644<br>--- a/payloads/libpayload/Makefile.inc<br>+++ b/payloads/libpayload/Makefile.inc<br>@@ -147,3 +147,12 @@<br> done<br> echo "</testsuite>" >> $@.tmp<br> mv $@.tmp $@<br>+<br>+test-configs:<br>+ for config in $(filter-out %.old,$(wildcard configs/*)); do \<br>+ $(MAKE) clean; \<br>+ cp "$$config" test_config; \<br>+ echo "*** Making libpayload config $$config ***"; \<br>+ $(MAKE) olddefconfig DOTCONFIG=test_config V=$(V) Q=$(Q) ; \<br>+ $(MAKE) V=$(V) Q=$(Q) CONFIG_LP_CCACHE=$(CONFIG_LP_CCACHE) DOTCONFIG=test_config; \<br>+ done<br>diff --git a/util/testing/Makefile.inc b/util/testing/Makefile.inc<br>index f5a4c5e..99f4d84 100644<br>--- a/util/testing/Makefile.inc<br>+++ b/util/testing/Makefile.inc<br>@@ -11,6 +11,24 @@<br> ## GNU General Public License for more details.<br> ##<br> <br>+test-help help::<br>+ @echo '*** coreboot test targets ***'<br>+ @echo ' what-jenkins-does - Run platform build tests with junit output'<br>+ @echo ' lint / lint-stable - run coreboot lint tools (all / minimal)'<br>+ @echo ' test-basic - Run stardard build tests. All expected to pass.'<br>+ @echo ' test-extended - Run extended tests.'<br>+ @echo ' test-lint - basic: Run stable and extended lint tests.'<br>+ @echo ' test-tools - basic: Tests a basic list of tools.'<br>+ @echo ' test-abuild - basic: Builds all platforms'<br>+ @echo ' test-payloads - basic: Builds internal payloads'<br>+ @echo ' test-cleanup - basic: Cleans coreboot directories'<br>+ @echo ' test-external-payloads - extended: Builds external payloads'<br>+ @echo ' test-qemu - extended: Builds & boots QEMU targets'<br>+<br>+# junit.xml is a helper target to wrap builds that don't create junit.xml output<br>+# BLD = Subdirectory the build lives in - The name of the build<br>+# BLD_DIR = Top path from coreboot to the build subdirectory<br>+# MAKETARGET = target to build<br> junit.xml:<br> echo "Building $(BLD)"<br> echo '<?xml version="1.0" encoding="utf-8"?><testsuite>' > $@.tmp<br>@@ -35,16 +53,32 @@<br> echo<br> <br> TOOLLIST= \<br>- cbmem \<br>- ectool \<br>- futility \<br>- inteltool \<br>- intelvbttool \<br>- nvramtool \<br>- superiotool \<br>- viatool<br>+cbmem \<br>+ectool \<br>+futility \<br>+inteltool \<br>+intelvbttool \<br>+nvramtool \<br>+superiotool \<br>+viatool<br>+<br>+TEST_PAYLOADLIST_INTERNAL= \<br>+coreinfo \<br>+nvramcui \<br>+<br>+TEST_PAYLOADLIST_EXTERNAL= \<br>+iPXE \<br>+Memtest86Plus \<br>+SeaBIOS \<br>+tint<br>+<br> JENKINS_PAYLOAD?=none<br>+TEST_PAYLOAD?=$(JENKINS_PAYLOAD)<br> CPUS?=4<br>+<br>+lint lint-stable:<br>+ util/lint/lint $@<br>+<br> what-jenkins-does:<br> util/lint/lint lint-stable --junit<br> util/abuild/abuild -B -J $(if $(JENKINS_NOCCACHE),,-y) -c $(CPUS) -z -p $(JENKINS_PAYLOAD) -x -X $(top)/abuild-chromeos.xml<br>@@ -54,3 +88,61 @@<br> unset COREBOOT_BUILD_DIR;$(MAKE) CPUS=$(CPUS) V=$(V) Q=$(Q) BLD_DIR=payloads/ BLD=nvramcui MFLAGS= MAKEFLAGS= MAKETARGET=all junit.xml<br> unset COREBOOT_BUILD_DIR;$(MAKE) CPUS=$(CPUS) V=$(V) Q=$(Q) BLD_DIR=payloads/ BLD=coreinfo MFLAGS= MAKEFLAGS= MAKETARGET=defaultbuild junit.xml<br> $(MAKE) CPUS=$(CPUS) V=$(V) Q=$(Q) BLD_DIR=util/ BLD=romcc MFLAGS= MAKEFLAGS= MAKETARGET=test junit.xml<br>+<br>+test-basic: test-lint test-tools test-abuild test-payloads test-cleanup<br>+<br>+test-extended: test-external_payloads test-qemu<br>+<br>+test-lint:<br>+ util/lint/lint lint-stable<br>+ util/lint/lint-015-final-newlines<br>+<br>+test-abuild:<br>+ util/abuild/abuild -B -e $(if $(TEST_NOCCACHE),,-y) -c $(CPUS) -p $(TEST_PAYLOAD) -x COREBOOT_BUILD_DIR=coreboot-builds-chromeos<br>+ util/abuild/abuild -B -e $(if $(TEST_NOCCACHE),,-y) -c $(CPUS) -p $(TEST_PAYLOAD)<br>+<br>+test-payloads:<br>+ $(MAKE) -C payloads/libpayload test-configs -j $(CPUS) V=$(V) Q=$(Q) MFLAGS= MAKEFLAGS= $(if $(TEST_NOCCACHE),,CONFIG_LP_CCACHE=y)<br>+ export MFLAGS=;export MAKEFLAGS=; \<br>+ $(foreach payload, $(TEST_PAYLOADLIST_INTERNAL), \<br>+ echo "*** Making payload $(payload) ***"; \<br>+ $(MAKE) -C payloads/$(payload) distclean ;\<br>+ $(MAKE) $(payload) -j $(CPUS) V=$(V) Q=$(Q)\<br>+ || exit 1; )<br>+<br>+test-external-payloads:<br>+ export MFLAGS=;export MAKEFLAGS=;\<br>+ $(foreach payload, $(TEST_PAYLOADLIST_EXTERNAL), \<br>+ echo "*** Making payload $$payload ***"; \<br>+ $(MAKE) -C payloads/exeternal/$(payload) clean; \<br>+ $(MAKE) $(payload) -j $(CPUS) V=$(V) Q=$(Q) \<br>+ || exit 1; )<br>+<br>+test-tools:<br>+ @echo "Build testing $(TOOLLIST)"<br>+ $(foreach tool, $(TOOLLIST), $(MAKE) -C util/$(tool) clean ; )<br>+ $(MAKE) -C util/romcc clean<br>+ $(foreach tool, $(TOOLLIST), echo "Building $(tool)";export MFLAGS= ;export MAKEFLAGS= ;$(MAKE) -C util/$(tool) all V=$(V) Q=$(Q) || exit 1; )<br>+ echo "Building romcc"<br>+ $(MAKE) -C util/romcc all test -j $(CPUS) V=$(V) Q=$(Q)<br>+<br>+test-qemu:<br>+ cp $(top)/configs/config.emulation_qemu_x86_i440fx $(top)/.config<br>+ echo "CONFIG_PAYLOAD_NONE=y" > $(top)/.config<br>+ make olddefconfig; make clean; make all<br>+ qemu-system-x86_64 -bios build/coreboot.rom -serial file:QEMU.LOG -daemonize -pidfile $(top)/QEMU.PID<br>+ sleep 5<br>+ kill $$(cat $(top)/QEMU.PID)<br>+ grep -q "Payload not loaded." $(top)/QEMU.LOG<br>+ rm $(top)/QEMU.PID $(top)/QEMU.LOG<br>+<br>+test-cleanup:<br>+ rm -rf coreboot-builds coreboot-builds-chromeos<br>+ $(MAKE) clean<br>+ $(MAKE) distclean<br>+ $(foreach tool, $(TOOLLIST), $(MAKE) -C util/$(tool) clean ; )<br>+ $(MAKE) -C util/romcc clean<br>+<br>+.PHONY: test-basic test-extended test-lint test-abuild test-payloads<br>+.PHONY: test-external_payloads test-tools test-qemu test-cleanup test-help<br>+.PHONY: lint lint-stable what-jenkins-does<br></pre><p>To view, visit <a href="https://review.coreboot.org/20874">change 20874</a>. To unsubscribe, visit <a href="https://review.coreboot.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://review.coreboot.org/20874"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: coreboot </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>
<div style="display:none"> Gerrit-Change-Id: I19e1256d79531112ff84e47a307f55791533806f </div>
<div style="display:none"> Gerrit-Change-Number: 20874 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: Martin Roth <martinroth@google.com> </div>