zstd with its default settings (compression level -3) compresses better
than bzip2 -9 (which is the default setting), and is an order of magnitude
faster.
I made the following measurements for the most common compression tools
(all standard Debian Buster versions, default flags unless noted
otherwise), using the debug information of a large x86-64 kernel with
ALL_KMODS:
* kernel-debug.tar: 376M
* kernel-debug.tar.gz: 101M, compressed in ~12s
* kernel-debug.tar.bz2: 91M, compressed in ~15s
* kernel-debug.tar.xz: 57M, compressed in ~101s
* kernel-debug.tar.zst: 86M, compressed in ~1s
With zstd, there is still some room for improvement by increasing the
compression, but the slight increase in compression ratio
(22.83% -> 19.46%) does not justify the significant increase in
compression time (about 5 times on my machine) in my opinion.
Note that multithreaded compression (-T argument) does not affect
reproducibility with zstd.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
(cherry picked from commit 4bd7990488b0ca7b5cae16f0a9147a4146759053)
Refreshed all patches.
Removed patches because included in upstream:
- 499-mtd-parser-cmdline-Fix-parsing-of-part-names-with-co.patch
- 0071-2-PCI-qcom-Fixed-IPQ806x-PCIE-reset-changes.patch
Compile-tested on: ipq40xx, lantiq/xrx200, x86/64, ipq806x
Runtime-tested on: ipq40xx, lantiq/xrx200, x86/64
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Multiple prereq checks are only required within the build system but not
for the ImageBuilder. These checks are excluded by using ifndef IB.
This commit merges the three ifndef IB blocks together.
Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit cc9d5b5a488d3a43068f621a1ef184f06e43bede)
The buildroot and SDK both require the compilers (gcc, g++) to be
installed on the host system, however the ImageBuilder uses precompiled
binaries.
This patch changes the prerequirements checks to skip the checking for
the compilers if running as ImageBuilder. A similar change has been
made for libncurses-dev in 4a1a58a3e2d2.
Signed-off-by: Sven Roederer <devel-sven@geroedel.de>
Acked-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit ae12a747cae3df16d84b7dc92f39427948d4e8e2)
The buildroot and SDK both require `libncurses-dev` to be installed on
the system, however the ImageBuilder uses precompiled binaries.
This patch changes the prerequirements checks to skip the
`libncurses-dev` part if running as ImageBuilder.
Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit 4a1a58a3e2d2360950ff23a05f1f55b779174d6a)
Some images are created using different filesystems, most popular
squashfs and ext4. To allow downstream projects to distinguesh between
those, add the `filesystem` information to created json files.
Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit bc0ffff36a0bd8043e958913f574cb8d2a79e93d)
Refreshed all patches.
Removed patches because included in upstream:
- 315-v5.10-usbnet-ipeth-fix-connectivity-with-ios-14.patch
Compile-tested on: ipq40xx, ath79, x86/64
Runtime-tested on: ipq40xx, ath79
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Currently it's assumed, that already downloaded tarballs are always
fine, so no checksum checking is performed and the tarball is used even
if it might be corrupted.
From now on, we're going to always check the downloaded tarballs before
considering them valid.
Steps to reproduce:
1. Remove cached tarball
rm dl/libubox-2020-08-06-9e52171d.tar.xz
2. Download valid tarball again
make package/libubox/download
3. Invalidate the tarball
sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile
4. Now compile with corrupt tarball source
make package/libubox/{clean,compile}
Signed-off-by: Petr Štetiar <ynezz@true.cz>
(cherry picked from commit 4e19cbc553350b8146985367ba46514cf50e3393)
Commit f98878e4c17d ("cmake.mk: set C/CXX compiler for host builds as
well") has introduced regression as it didn't taken usage of ccache into
the account so fix it by handling ccache use cases as well.
In order to get this working we need to export HOSTCXX_NOCACHE in
rules.mk as well.
Fixes: f98878e4c17d ("cmake.mk: set C/CXX compiler for host builds as well")
Reported-by: Ansuel Smith <ansuelsmth@gmail.com>
Tested-by: Ansuel Smith <ansuelsmth@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
(cherry picked from commit 524fb5646eec6147aadfdd508219f39bcf8ba8fc)
Without this, cmake will use whatever CC/CXX is set to, which could be
clang. In that case, at least libjson-c/host will fail to compile.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
(cherry picked from commit f98878e4c17d5f11e78994b4fc456e6b60b2660f)
Add package which provides size optimized wpad with support for just
WPA-PSK, SAE (WPA3-Personal), 802.11r and 802.11w.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
[adapt to recent changes, add dependency for WPA_WOLFSSL config]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
(cherry picked from commit c487cf8e94cbdf582dfc3c2bdaab913a146a2100)
With this commit the `profiles.json` contain both the target specific
`default_packages` as well as the device specific `device_packages` as a
array of strings.
This information is required for downstream projects like the various
web-based interactive firmware generators.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit 263f7e5bbd119ebed1f514c16f659a2e2a2b132c)
JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6e2
("build: create JSON files containing image info").
They are useful for firmware wizards and script checking for
reproducibility.
Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.
This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.
Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.
Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.
As before, this creation is enabled by default only if `BUILDBOT` is set.
Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
(backported from commit 07449f692ce4c4525e946401f4c3ed0cbbc8c4df)
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
Commit 881ed09ee6e2 ("build: create JSON files containing image info")
has removed the crucial empty new line from the image copy step
resulting in the following errors during make function expansion:
GZ_SUFFIX :=
bash: GZ_SUFFIX: command not found
Makefile:86: recipe for target 'openwrt-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin' failed
Fixes: 881ed09ee6e2 ("build: create JSON files containing image info")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
(cherry picked from commit 0fb23d67f00a18c3f9e712ca97cfc34b38071f4a)
The JSON info files contain details about the created firmware images
per device and are stored next to the created images.
The JSON files are stored as "$(IMAGE_PREFIX).json" and contain some
device/image meta data as well as a list of created firmware images.
An example of openwrt-ramips-rt305x-aztech_hw550-3g.json
{
"id": "aztech_hw550-3g",
"image_prefix": "openwrt-ramips-rt305x-aztech_hw550-3g",
"images": [
{
"name": "openwrt-ramips-rt305x-aztech_hw550-3g-squashfs-sysupgrade.bin",
"sha256": "db2b34b0ec4a83d9bf612cf66fab0dc3722b191cb9bedf111e5627a4298baf20",
"type": "sysupgrade"
}
],
"metadata_version": 1,
"supported_devices": [
"aztech,hw550-3g",
"hw550-3g"
],
"target": "ramips/rt305x",
"titles": [
{
"model": "HW550-3G",
"vendor": "Aztech"
},
{
"model": "ALL0239-3G",
"vendor": "Allnet"
}
],
"version_commit": "r10920+123-0cc87b3bac",
"version_number": "SNAPSHOT"
}
Signed-off-by: Paul Spooren <mail@aparcar.org>
(backported from commit 881ed09ee6e23f6c224184bb7493253c4624fb9f)
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
Refreshed all patches and removed upstreamed:
oxnas/001-irqchip-versatile-fpga-Handle-chained-IRQs-properly.patch
oxnas/002-irqchip-versatile-fpga-Apply-clear-mask-earlier.patch
Fixes: CVE-2020-12114 and CVE-2020-11669
Runtime-tested on: qemu-x86-64
Compile-tested on: ath79/generic, x86/64, imx6
Signed-off-by: Petr Štetiar <ynezz@true.cz>
There is a restriction in the number of parameters(10) that may be passed to
the SetupHostCommand macro so continually adding explicit gcc'n' version
checks ends up breaking the compiler check for the later versions and
oddballs like Darwin as was done in 835d1c68a0 which added gcc10.
Drop all the explicitly specified gcc version checks. If a suitable gcc
compiler is not found, it may be specified at the dependency checking
stage after which that version will be symlinked into the build staging
host directory.
eg. 'CC=gccfoo CXX=g++foo make prereq'
Signed-off-by: Kevin Darbyshire-Bryant <ldir@darbyshire-bryant.me.uk>
Acked-by: Jo-Philipp Wich <jo@mein.io>
(cherry picked from commit 1fb3c003d68d3feaf797e8b64edccc9fa622d250)
Lets add GCC 10 detection to the build system as distributions like Fedora 32 have started shipping with it.
Some tools like mtd-utils need work to compile under GCC10, but that will be next step.
Signed-off-by: Robert Marko <robert.marko@sartura.hr>
(cherry picked from commit 835d1c68a0f036c8b0d837a48b5a05fdfb2e8218)
Moritz Warning