The parallel-build failure gate only inspected $exit_code in run_on_finish. A
ForkManager child killed by a signal -- SIGKILL, or the OOM-killer under the
concurrent build load -- is reaped with $exit_code == 0 but $exit_signal != 0
(and possibly $core_dump). Such a worker therefore was NOT recorded as a
failure, so the parent could still index and GPG-sign a repository that is
missing the package that worker was building -- the exact partial-repo hazard
the gate was added to prevent, via a path it did not cover.
Capture $exit_signal and $core_dump from the run_on_finish callback and fail
the build when any of $exit_code, $exit_signal, or $core_dump is set.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Addresses three concurrency-safety defects raised in review of the parallel
buildrpms.pl work:
1. Per-target build lock was released immediately. The flock filehandle was a
lexical (my $blk) scoped to the guard block, so it was destroyed -- and the
lock dropped -- as soon as that block exited, before any worker forked. The
"intentionally leaked" comment did not match the code. Hold the handle in a
file-scoped $BUILD_LOCK_FH so the fd (and the lock) live for the whole
process; forked children inherit the fd but their exits never release it.
2. Build failures were silently swallowed. buildspkgs()/buildpkgs() called
sh_retry() in void context, so a mock build that failed all retries returned
non-zero into the void; the child then exited 0 and the parent's run_on_finish
ignored the exit code. The parent could therefore index and GPG-sign a repo
that was missing packages and still exit 0. Now sh_retry failures die in the
child, run_on_finish records any non-zero child, and the run aborts before
update_repo and again before signing if any child failed -- never publishing
a partial core.
3. Abort cleanup unmounted unrelated builds. sweep_mock_mounts() lazy-unmounted
every bind under /var/lib/mock, which on a shared host tears out the live
chroots of concurrent, unrelated builds. Scope it to this run's own chroots
(each chroot dir plus its -bootstrap sibling), passed in from abort_builds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
merge_core_repos already assembles the multi-arch core and runs the full
finalize tail (index -> sign -> final metadata -> write_release_alias, in the
required order so the xCAT-release-latest alias lands AFTER metadata and stays
out of the repo index). The preceding commit restored a separate finalize_core
sub purely to satisfy xcat_release_package.t, which grepped for
'sub finalize_core { ... $dir ... }'. That left two overlapping entry points:
--finalize-core (finalize one pre-assembled dir) and --merge-core-repos
(assemble N per-arch dirs THEN finalize) -- the former a strict subset of the
latter, with no in-tree or CI caller.
Consolidate on the single assemble+finalize path: inline the finalize tail into
merge_core_repos, drop sub finalize_core and the --finalize-core getopt/dispatch,
and update xcat_release_package.t to assert the stable alias is created after the
final metadata pass inside merge_core_repos ($out). Behaviour is unchanged (merge
already wrote the alias); only the redundant finalize_core interface is removed.
xcat_release_package.t 26/26; perl -c clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The CI buildrpms.pl port (d2d98b724) renamed the upstream sub finalize_core($dir)
to merge_core_repos($out) and inlined its index/sign/metadata/alias tail. The
xCAT-release logic (write the xCAT-release-latest stable bootstrap alias AFTER the
final metadata pass, so it stays out of the repo index) was preserved, but the rename
broke xCAT-test/unit/xcat_release_package.t, which statically pins:
sub finalize_core { ... write_repo_metadata_dir($dir); ... write_release_alias($dir); }
so 'assembled core repository creates the stable alias after final metadata' failed
(1 of 26), turning xcat_pr_test red on PR #7701.
Restore finalize_core($dir) as the shared finalize primitive (index -> sign ->
final metadata -> stable alias) and have merge_core_repos delegate to it on the
assembled multi-arch dir. Re-expose the --finalize-core CLI (single already-populated
dir) it also feeds. Behaviour is unchanged for both --merge-core-repos (CI) and
--finalize-core; only the shared code path is named again. xcat_release_package.t now
26/26; perl -c clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl built every package with mock -N (--no-clean), reusing the per-package
<pkg>-<target><ext> chroot across runs for flat disk. But a build aborted or killed
mid-flight leaves that chroot half-initialised with a corrupt rpmdb; the NEXT run
reused it and failed (cannot open Packages database .../usr/lib/sysimage/rpm),
producing an incomplete core (e.g. missing xCAT-test) that fails the deploy-time
completeness gate.
Re-init the buildroot (mock --init) right before building each package, after the
diskcache skip so it only runs when actually building. --init restores from mock's
root-cache tarball (cheap) so disk stays flat and builds stay fast; -N is kept on the
srpm/binary calls so they still reuse the freshly initialised root within the run.
This stabilises builds against corrupt state left by any previous failed/aborted build.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The 3 EL CD pipelines (xcat-core-devel-cd, xcat-core-stable-cd, xcat-dep-el-cd)
overlaid a pinned $CI/buildrpms.pl at build time because the tree's buildrpms.pl
lacked the options they depend on:
- a per-target flock guard alongside --mock-uniqueext, so concurrent same-target
builds do not corrupt each other's /var/lib/mock chroot namespace;
- --native-only (build only arch-native pkgs on the secondary arch) plus
--merge-core-repos/--output-dir/--input-core-repos, replacing --finalize-core,
to assemble one signed flat multi-arch core from per-arch build outputs;
- sh_retry() to absorb transient mock/nspawn flakes;
- a single --target guard and graceful mock cancellation
(sweep_mock_mounts/abort_builds) that unmounts chroots on abort.
The xCAT-release repository package (master/2.19 only) is preserved: its
write_release_alias() is invoked from the new merge_core_repos() and per-target.
Porting them in-tree lets CI drop the $CI/buildrpms.pl pin and run the three
pipelines in parallel without the cross-job serialize lock.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #7677 added xCAT-server/share/xcat/netboot/rh/compute.rhels9.ppc64le.pkglist
with a stray empty line at the end of the file. The reviewer asked for it to be
removed before merge, but it slipped through. A trailing blank line in a pkglist
is meaningless and inconsistent with the rest of the netboot package lists, so
remove it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The security response headers added to xcat.conf are wrapped in
<IfModule mod_headers.c>, so they only take effect where mod_headers is loaded.
EL loads it by default and the Debian postinst enables it with a2enmod, but SUSE
apache2 does not load it by default, so the headers stayed inactive there. Enable
it in the RPM %post for the apache2 case, guarded so it is a no-op on EL (where
a2enmod is absent and mod_headers is already loaded).
Verified on openSUSE Leap 15.6: mod_headers is off by default, the hardened
config still starts, and after a2enmod the served response carries all four
security headers. The need to load mod_headers on SUSE was identified in the
lenovobuild branch (660df81379 / xcat.conf.apach24.sles), here done via
a2enmod rather than a separate config file.
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
Add standard security response headers (X-Frame-Options, X-Content-Type-
Options, Content-Security-Policy, X-Permitted-Cross-Domain-Policies) to the
/install and /tftpboot directories, mask the server banner with
"ServerTokens Prod", and drop the Includes (SSI) and MultiViews options from
those file-serving directories. Indexes on /install/postscripts, /install/post
and the doc directory are left intact so directory browsing still works where
xCAT relies on it.
The Header directives are wrapped in <IfModule mod_headers.c> so a server
whose mod_headers is not loaded still starts cleanly instead of failing on an
unknown directive. On Debian/Ubuntu, where mod_headers is not enabled by
default, the xCAT and xCATsn package postinst scripts run "a2enmod headers"
before restarting Apache so the headers take effect there as well; on
RHEL/SLES the module is loaded by default and needs no action.
Recovered from the unmerged lenovobuild branch (originals 7ee0c129, 85c8bc09,
d4d1783a), adapted: the deprecated X-XSS-Protection header and the
mod_allowmethods-dependent AllowMethods directive are omitted, and the Header
directives use "set" rather than "append".
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão