The two builders wrote the same run-a-command-or-stop step in opposite
polarities. builddebs.pl used sh(...) == 0 or die; buildrpms.pl used
sh(...) and die, and also used the == 0 or die form for its sh_retry calls, so
both directions appeared in one file. The 'and die' spelling reads as though
the die is what happens next rather than what happens on failure, which is a
poor thing to have to re-read at every call site.
Add BuildUtils::sh_or_die and convert the fourteen plain sh() call sites to it.
The failure message now also carries the exit code, which every one of the old
spellings discarded -- a build that failed said only that a command failed, not
what it returned. The sh_retry sites keep their own form: retry is a different
operation and sh_retry is local to buildrpms.pl.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The two builders read and wrote files three different ways. buildrpms.pl used
File::Slurper, builddebs.pl opened and closed handles inline, and BuildUtils.pm
carried its own read_file/write_file pair to avoid the dependency. The result
was that the same operation was spelled differently in each place, and the
Version stamp was read by hand in both builders: buildrpms.pl chomped it ten
lines after reading it, which is how a stamp keeps a trailing newline nobody
notices until it reaches a package name.
Drop the hand-rolled pair and use File::Slurper throughout, and add
BuildUtils::read_line for the one-line Version and Release stamps -- it strips
the newline at the point of reading and reports an absent file as undef, which
is what builddebs.pl's fallback to snap_release needs.
File::Slurper is already required by buildrpms.pl and is already in the CI
package list, so this adds no new build-host prerequisite. The lock file still
uses a raw open: flock needs a real handle.
Behaviour note: Version is now read as its first line rather than the whole
file with the final newline removed. For the one-line file it is, these are the
same; a malformed multi-line Version now yields its first line instead of an
embedded newline.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddebs.pl stages the genesis bmcsetup and getipmi helpers into the xCAT
package by writing the rewritten text and then chmod'ing it, which is the third
copy of the write-then-make-executable pair the previous change collapsed. It
was left out because it uses 0755 while the published repo helper uses 0775.
Give write_script an optional mode, defaulting to the 0775 it already used, and
use it for the postscripts with 0755. The two modes stay exactly as they were
-- the postscripts are deliberately not group-writable -- and the test asserts
that the caller's mode is honoured rather than overwritten by the default.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Both builders ship a mklocalrepo.sh beside the packages they publish, and each
wrote it its own way: builddebs.pl opened, printed and closed the file by hand
and then chmod'ed it, while buildrpms.pl used File::Slurper and a separate
chmod. Writing the text and setting the executable bit are one operation -- a
copy published without 0775 is published broken -- but nothing tied them
together, so each caller had to remember the second step.
Add BuildUtils::write_script, which writes the file and sets the mode, and use
it from both builders. The mode is now asserted in build_utils.t instead of
being left to the callers to repeat.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Both builders wrote the same six fields beside a published repository --
VERSION, RELEASE, BUILD_TIME, BUILD_MACHINE, COMMIT_ID, COMMIT_ID_LONG -- each
assembling them by hand, and each shelling out to hostname to do it.
builddebs.pl also ran git rev-parse a second time to get the commit, although
it had already derived the same value into $GITINFO earlier in the file. It now
uses that.
The filenames and the time formats stay as they were. deploy.sh copies
builddebs.pl's buildinfo verbatim and cluster-test.pl parses buildrpms.pl's
buildinfo.txt, so both are a contract with consumers outside this repository;
the shared helper takes the format from its caller rather than picking one.
Confirmed byte for byte that each builder's stamp is unchanged.
Covered by tests: the field names and their order, the seven-character short
commit, and that the two callers' formats still differ.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddebs.pl spelled out the same slurp five times, and twice wrapped it in the
identical read, transform, write-back sequence -- once to pin the dependency
versions in debian/control and once to rewrite the changelog header. The two
blocks differed only in the file and the function applied to its contents.
BuildUtils.pm now provides read_file, write_file and rewrite_file. The last
leaves a file that is not there alone rather than creating it, which is what
both call sites guarded for with -f.
These use plain open and close rather than File::Slurper. buildrpms.pl already
depends on that module, but builddebs.pl does not, and putting it in the shared
module would oblige a deb build to install something it otherwise has no need
of.
Reading a missing file now names the file it could not read; one of the call
sites died with no message at all.
Covered by tests: a round trip, that rewrite_file applies its transform and
reports whether it acted, that it neither runs the transform nor creates the
file when there is nothing there, and that a failed read names the path.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Both builders defined their own sh() and usage(). The usage() copies differed
only in spelling, but the sh() copies disagreed about what they return:
buildrpms.pl shifted system()'s result into an exit code, builddebs.pl returned
the raw wait status. The same command therefore reported 1 from one builder and
256 from the other, and a caller comparing sh() against a particular code was
correct in only one of them. Both current call sites only test for zero, so
nothing was broken yet.
BuildUtils.pm now provides both, shifting the status as buildrpms.pl did.
pod2usage reads the POD of the running program, so each builder keeps its own
help text while sharing how it is printed and the status it exits with.
sh() echoed the command under the builder's own --verbose flag, which a shared
function cannot see. $BuildUtils::VERBOSE is set once after option parsing
instead of threading the flag through every call site.
Covered by tests: that sh() returns the exit code rather than the wait status
it is packed into, and that a verbose run still echoes the command.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
An xCAT installed from these debs could report no version at all: `lsxcatd -v`
printed a bare "Version" and there was no /opt/xcat/share/xcat/version to
consult.
xCAT::Version ships with #XCATVERSIONSUBHERE / #XCATSVNBUILDSUBHERE
placeholders that perl-xCAT/debian/rules fills by calling modifyUtils with the
version and the commit. It takes the commit from ../Gitinfo, falling back to
`git log`. Neither Debian builder wrote Gitinfo -- buildrpms.pl does, which is
why only the deb side was affected -- so a tree without a readable .git left
that argument empty. modifyUtils then hit `if [ -z "$2" ]` and returned 0
WITHOUT substituting, and neither caller checks the status, so the placeholders
shipped.
Two independent things had to be wrong for this to be silent, and both are
fixed:
modifyUtils now exits non-zero on a missing argument, so a build that cannot
stamp a version fails instead of producing one that cannot identify itself. Its
message was also wrong -- it checked $2 and complained about the version, which
is $1 -- so each argument now names itself.
builddebs.pl writes Gitinfo, as buildrpms.pl already does. BuildUtils::
git_revision prefers the checkout, falls back to an existing Gitinfo (a source
export carries the real revision that way, and overwriting it with a
placeholder would discard the only provenance the tree has), and only then
returns "unknown" -- never the empty string modifyUtils ignores.
"unknown" is now warned about rather than stamped quietly. The cause is usually
not a missing .git but git refusing one it considers dubiously owned: the tree
belongs to another user and the safe.directory exception sits in a config that
the build's own HOME override hides. That is exactly how this went unnoticed.
Not a regression -- build-ubunturepo produces byte-identical unsubstituted
placeholders in the same .git-less tree, verified by running it there. It is a
pre-existing hole that CD masks by writing Gitinfo in the pipeline.
Verified on xcat-master-ub: with a readable checkout the deb carries
'Version ' . '2.19.0' . ' (git commit 0030133c89)';
without one it carries '(git commit unknown)' and the build prints the warning
naming the tree, the command to run and the HOME to run it under. Nine
assertions drive modifyUtils itself -- stamping both forms, and each missing
argument -- and dropping either sed, or restoring the silent exit, reddens it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A second run in the same checkout died once the release string moved:
dpkg-genbuildinfo: error: cannot fstat file ../xcat_..._ppc64el.deb
debian/files accumulates one line per artifact and survives `dh_clean -d`,
which removes directories only. The next build's dpkg-genchanges reads the
stale entries and fstats artifacts collect_debs already moved away. The old
shell builder deleted debian/files explicitly; that step was not carried
over. --force does not help, since it only wipes the output repository.
Three further ways the build did not put the tree back as it found it:
xCAT/postscripts/{bmcsetup,getipmi} are TRACKED files that the xCAT build
rewrites from the genesis sources. They were recorded as created, so
cleanup deleted them from the checkout. They are claimed now.
Restoring a claimed file lost its mode: File::Copy::copy does not carry
permissions, so an executable came back 100644 with identical content --
visible only as a git mode change. backup_file/restore_file record and
reapply it.
debian/*.substvars are rewritten in place and several are tracked; they are
claimed too. debhelper's .debhelper/ and *.debhelper.log are never tracked
and are removed with the rest of the residue.
clean_debian_residue runs from collect_debs, which is OUTSIDE
with_prepared_tree -- the restore has already happened by then. That is why
it must not remove *.substvars: doing so would delete the tracked ones it
just put back. The claim mechanism handles those instead.
Verified on xcat-master-ub: two consecutive full 14-package builds in one
checkout both succeed, and a build against a git checkout now leaves zero
tracked files modified or deleted (was five, two of them deleted). The
remaining untracked residue -- pods/, share/, pod2htmd.tmp and the
substvars of packages that do not track one -- is inherited from
build-ubunturepo and unchanged here.
Also covers the guard that gives this PR its name: deleting
`return if $opts{source_only}` from buildall left buildrpms_source_only.t
green, so nothing checked that --source-only skips the binary rebuild. It
does now, and the mutation reddens two assertions. Same for the changelog
trailer: making its substitution global left the suite green because only
the older header was asserted, not its author and date.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Four fixes from @viniciusferrao's review plus the CI break his review predates.
orig tarball version. dpkg looks for <source>_<upstream>.orig.tar.gz with no
Debian revision, and the call site passed the full Version-Release. The rule now
lives in BuildUtils::upstream_version and orig_tarball_name applies it, so the
call site cannot get it wrong whichever string it is handed. Currently dormant --
every package is Format: 1.0, so the quilt branch does not run, which is why the
differential build did not catch it.
--dest could write to the filesystem root. Cwd::abs_path returns undef when a
PARENT component is missing (a missing leaf is fine), and the caller interpolated
that, so `--dest /no/such/parent/out` became `/debs` and `/xcat-core` at /.
Replaced with BuildUtils::resolve_dest, which is rel2abs and purely lexical --
correct for an output directory that does not exist yet.
Generated debian/control left behind. xCAT-genesis-scripts has no debian/control
of its own; it is generated from control-<arch>. The cleanup restored only files
that already existed, so the generated one stayed. Worse than dirty: ppc64el ran
last, so the restore put back the amd64 BACKUP and the leftover was the wrong
architecture's control, which a later single-arch build would have started from.
with_prepared_tree now records created files and removes them. Verified by a real
build: the checkout is byte-clean afterwards, matching the oracle.
CI install step. build-ubunturepo wrote its repo to $curdir/../../xcat-core,
which under GitHub's work/<repo>/<repo> layout IS $RUNNER_WORKSPACE, so
install_xcat's `./mklocalrepo.sh` happened to be in the directory it chdir'd to.
builddebs.pl writes inside the checkout instead -- that outside-the-checkout path
is what used to rm -rf the tree -- so install_xcat now names the script by its
real location and fails with a clear message if the build produced no repository.
This is what reddened xcat_pr_test at 2m13s; the builder itself was fine (the
exact CI invocation, `./builddebs.pl --force` with no --dest, returns 0 with all
14 packages).
The executable bit was already fixed before the review landed.
Both new helpers are tested and mutation-verified: not stripping the revision
reddens 3 assertions, swapping rel2abs back to abs_path reddens 2. Equivalence
re-measured after these changes -- all 14 packages identical to build-ubunturepo
in control and in every non-changelog file by md5.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
build-ubunturepo was 710 lines of shell doing the Debian half of what
buildrpms.pl does for rpms, with no code in common and a different CLI. It also
carried paths that are dead: GSA uploads, the PROMOTE/PREGA release flows, and a
-d mode that built an xcat-dep repository from a different project's packages.
builddebs.pl replaces it and mirrors buildrpms.pl -- Getopt::Long options, one
package list, build then index then sign -- so the two builders read the same way
and share BuildUtils.pm.
The design rests on one fact: xcat-core debs are Perl. They are byte-identical
for every Ubuntu release, so they are built ONCE and the same files are published
into every codename. Only xCAT, xCATsn and xCAT-genesis-scripts carry an
architecture, and there the difference is packaging metadata, not compiled
output. That is why this needs no sbuild and no per-codename chroot -- unlike
xcat-dep, whose packages are compiled and genuinely differ per release.
BuildUtils.pm holds what both builders need and what was worth making testable:
the Version-Release derivation from the commit time, the xCAT-probe helper
staging, the deb arch and dist tables, the debian/control version pinning, the
changelog rewrite, the reprepro conf generation, and the build lock. Every
function is pure or takes its side effect as an argument, so build_utils.t (45
assertions) drives each one rather than grepping a builder for evidence that it
is called. Verified by mutation: shrinking the arch table reddens 1, dropping
the /g from the control pin reddens 2.
The env-var CLI maps to options: BUILDALL=1 -> --force, GPGSIGN=1 -> --gpg-sign,
GPG_HOME -> --gpg-home, DEST -> --dest, DISTS -> --dist (repeatable). UP=0 has no
equivalent because uploading is gone -- the CD pipeline's deploy step publishes.
Callers updated: github_action_xcat_test.pl and travis.pl. The comment in
github_action_xcat_test.pl explaining why CI copies the tree before building is
corrected -- build-ubunturepo rm -rf'd $curdir/../../xcat-core, which under
GitHub's work/<repo>/<repo> layout is the checkout's own parent; builddebs.pl
writes under dist/debs inside the checkout and restores every file it edits, so
the copy is now only isolating the tests from build residue.
Two tests moved with it. build_ubunturepo_lock.t extracted the lock out of the
shell with a regex and ran that; the lock is now a function, so builddebs_lock.t
calls it -- and asserts what actually matters, that two builds of one checkout
fail fast while two builds of different checkouts run concurrently.
ubuntu_2604_pkglist.t asserted that resolute appeared in a shell fragment of
build-ubunturepo's source; it now asks BuildUtils for the release list and checks
a resolute stanza reaches conf/distributions. That assertion would have passed on
any file containing the fragment and broken on a reflow that changed nothing.
Verified: prove -r xCAT-test/unit fails on 6 files here against 7 on
upstream/master, the difference being apache_config_sources.t, fixed by the
preceding commit. The remaining 6 are missing DB modules on the machine that ran
it and are identical on both.
NOT done here, and required before this can merge: the Ubuntu core CD pipelines
still invoke ./build-ubunturepo (ci/ubuntu/Jenkinsfile.core-ubuntu-{devel,stable}
in VersatusHPC/xcat-core-ci-cd, and the inline script in each live Jenkins job).
Those must be switched to builddebs.pl in the same change window.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst