JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6e2
("build: create JSON files containing image info").
They are useful for firmware wizards and script checking for
reproducibility.
Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.
This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.
Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.
Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.
As before, this creation is enabled by default only if `BUILDBOT` is set.
Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
(backported from commit 07449f692ce4c4525e946401f4c3ed0cbbc8c4df)
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
The prepare target was added some 11 years ago to build tools and
toolchain and was recently extended to create buildinfo files for
reproducibility, meaning {feeds,version,config}.buildinfo.
As the buildbot workflow is more complex than the single prepare (kmod
feed insertion), prepare is only used to create those buildinfo files.
Running prepare however runs `target/compile` as well, taking time even
everything is already compiled.
Splitting this allows the buildbot to run only the `buildinfo` target
while others can still use the convenience feature `prepare`.
Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit 6caf437652d858e5795ee16bdaf9f0436d2488f9)
On a recent Gentoo Linux installation, invoking `make menuconfig`, `make
kernel_menuconfig` or `make kernel_nconfig` in the build system fails,
whereas for example `make menuconfig` in the kernel tree alone works as
expected.
This is happening because STAGING_PREFIX is not defined when kernel's
{menu,n}config target calls pkg-config from the toolchain/host and thus
pkg-config returns an empty value, and the fallback values in the kernel
config script are applied but those are off and the linking fails.
Solution is to use system's pkg-config for all ncurses based menu config
targets in order to provide proper compiler/linker flags.
Ref: FS#2423
Cc: Thomas Albers <thomas.gameiro@gmail.com>
Tested-by: Stijn Tintel <stijn@linux-ipv6.be>
Tested-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
(cherry picked from commit 965f341aa9fdb6e07d509d02a6ca188af050292a)
generate feeds.buildinfo and version.buildinfo in build dir after
containing the feed revisions (via ./scripts/feeds list -sf) as well as
the current revision of buildroot (via ./scripts/getver.sh).
With this information it should be possible to reproduce any build,
especially the release builds.
Usage would be to move feeds.buildinfo to feeds.conf and git checkout the
revision hash of version.buildinfo.
Content of feeds.buildinfo would look similar to this:
src-git routing https://git.openwrt.org/feed/routing.git^bf475d6
src-git telephony https://git.openwrt.org/feed/telephony.git^470eb8e
...
Content of version.buildinfo would look similar to this:
r10203+1-c12bd3a21b
Without the exact feed revision it is not possible to determine
installed package versions.
Also rename config.seed to config.buildinfo to follow the recommended
style of https://reproducible-builds.org/docs/recording/
Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit 454021581f630d5d04afeb8ff6581c1bda295c87)
In some places the output of commands, which include "cd" are used.
In case of CDPATH the new path is printed, which might not be expected.
Disable the variable to avoid these problem.
When CDPATH was set by the user to some value like "export CDPATH=."
the git checkout done by the build system did not work anymore, the
git cloning aborted with such an error message for example:
....
Packing checkout...
tar: /disk/fs1/tmp2/mehrtens/pon-ugw/ugw-haps/openwrt/tmp/dl/ppa-drv-1.0\n@1534240258: Cannot stat: No such file or directory
tar: Date sample file not found
Try 'tar --help' or 'tar --usage' for more information.
.....
To avoid this, this patch makes the build system unset CDPATH inside
the build system, so the build system will still work even when the
user set this variable in his local environment.
Signed-off-by: Thomas Langer <thomas.langer@intel.com>
Signed-off-by: Hauke Mehrtens <hauke.mehrtens@intel.com>
Acked-by: Hans Dedecker <dedeckeh@gmail.com>
In order to be able to better compare files to sync in the future, include
all BIN_DIR subdirectories in the checksum calculation.
To not break existing applications, restrict the recursive checksumming to
CONFIG_BUILDBOT for now.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Using a single host package staging dir (and build dir) significantly speeds up
builds when multiple targets are built in succession, especially for large host
packages like NodeJS.
$(STAGING_DIR)/host is kept in addition to $(STAGING_DIR_HOSTPKG) in most
places; it is still used as destination for host files in Build/InstallDev.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Ensure that BIN_DIR exists when the diffconfig target needs it.
Otherwise 'make diffconfig' fails after 'make clean'
Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
Add a "diffconfig" build target which stores the output of
"scripts/diffconfig.sh" as "config.seed" in the image output directory and
invoke that target by default.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
This is intended to be used for a wide array of package sanity checks.
The first check that is implemented is for the hash of downloaded files.
It checks:
- Missing hash
- Use of SHA256 instead of MD5
- dl/<file> hash not matching hash in makefile
- deprecated MD5SUM variable
The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt
is updated as well
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Add a new "checksum" make target which generates an sha256sums file over the
image files produced in bin/targets/ and automatically call it during make
world after the package index generation.
The advantage of this new target is that it is guaranteed to run after the
images, the SDK and the ImageBuilder archives have been generated to ensure
that they all end up in the checksum file. Fixes FS#51.
Uses sed to postprocess the OpenSSL digest output into an sha256sum command
compatible format.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Currently "make clean" only clears the build_dir/target*, but leaves
staging_dir/target* intact. "make clean" should also clean the
staging_dir/target* directories, as in the current situation some old
packages or libraries may be linked into the firmware from staging_dir
despite a "make clean".
The patch reorganises clean / dirclean functionality so that
* "make clean" also clears the staging_dir/target* in addition to
build_dir/target*.
* "make dirclean" clears toolchain and host(=tools) directories from both
build_dir and staging_dir
signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
SVN-Revision: 45973
Makefile: remove non-existent STAGING_DIR_TOOLCHAIN from dirclean
Openwrt's top level Makefile uses STAGING_DIR_TOOLCHAIN in the make dirclean
statement.
https://dev.openwrt.org/browser/trunk/Makefile#L55
rm -rf $(STAGING_DIR) $(STAGING_DIR_HOST) $(STAGING_DIR_TOOLCHAIN)
$(TOOLCHAIN_DIR) $(BUILD_DIR_HOST) $(BUILD_DIR_TOOLCHAIN)
As far as I can determine, no such variable has been defined. I made a search
in Openwrt source repository and the one line in Makefile's dirclean command
is the only place where that variable exists.
The item has been introduced to Makefile by r8362, but even at that time
neither Makefile nor rules.mk defined such a variable. Most likely the goal
has been to set both staging_dir/toolchain and build_dir/toolchain to be
cleaned, but one of the variables has been erroneous. The correct variable
for build_dir/toolchain has been then added by r13494.
References:
https://dev.openwrt.org/changeset/8362/https://dev.openwrt.org/browser/trunk/Makefile?rev=8362https://dev.openwrt.org/browser/trunk/rules.mk?rev=8362https://lists.openwrt.org/pipermail/openwrt-devel/2007-August/001159.htmlhttps://dev.openwrt.org/changeset/13494
In current code,
TOOLCHAIN_DIR = $(TOPDIR)/staging_dir/$(TOOLCHAIN_DIR_NAME)
BUILD_DIR_TOOLCHAIN = $(TOPDIR)/build_dir/$(TOOLCHAIN_DIR_NAME)
so the item STAGING_DIR_TOOLCHAIN in the rm command is unnecessary.
signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
SVN-Revision: 45736
Most of the time, we want to make sure OpenWrt has been configured and
setup before start running make. However, in case of package/symlinks,
forcing prereq as a dependency creates multiple issues:
*when executed on a clean workspace, it will prompt for user input
and open a menuconfig window before executing the feeds command
*the only way around that is to provide a .config. However, the "prereq"
target would then run a "make defconfig", which will remove all the
packages in the .config but from external feeds, as feeds have not been
installed yet.
The only way to currently work around this, is to generate a fake config
by running "make defconfig", then "make package/symlinks", copy the real
config (which at this point disregards the previously generated config),
and run make defconfig again. Something like this:
make defconfig
make package/symlinks
cp real.config .config
make defconfig
This change is removing the need for the first defconfig, making the
process more logical for OpenWrt users using the package/symlinks target.
Signed-off-by: Mathieu Olivari <mathieu@qca.qualcomm.com>
SVN-Revision: 45657
When using GREP_OPTIONS to supply default options to grep, the buildsystem might get broken (For example adding --color=always breaks it)
This patch will empty the GREP_OPTIONS to prevent the described (and any other) problems related to GREP_OPTIONS
Signed-off-by: Maarten Bezemer <m.m.bezemer@utwente.nl>
SVN-Revision: 22443
Paul Spooren