xCAT/xcat-core
2
0
Fork 0
mirror of https://github.com/xcat2/xcat-core.git synced 2026-09-05 04:27:55 +00:00
Code Issues Releases Wiki Activity
27,821 Commits 21 Branches 82 Tags
63c9ef8a1f794d4f2b204dea9cdedc848b7c2aff
Commit Graph

4 Commits

Author SHA1 Message Date
Daniel Hilst 63c9ef8a1f fix(build): stop builddebs.pl leaving the checkout unbuildable and dirty 
A second run in the same checkout died once the release string moved:

  dpkg-genbuildinfo: error: cannot fstat file ../xcat_..._ppc64el.deb

debian/files accumulates one line per artifact and survives `dh_clean -d`,
which removes directories only. The next build's dpkg-genchanges reads the
stale entries and fstats artifacts collect_debs already moved away. The old
shell builder deleted debian/files explicitly; that step was not carried
over. --force does not help, since it only wipes the output repository.

Three further ways the build did not put the tree back as it found it:

xCAT/postscripts/{bmcsetup,getipmi} are TRACKED files that the xCAT build
rewrites from the genesis sources. They were recorded as created, so
cleanup deleted them from the checkout. They are claimed now.

Restoring a claimed file lost its mode: File::Copy::copy does not carry
permissions, so an executable came back 100644 with identical content --
visible only as a git mode change. backup_file/restore_file record and
reapply it.

debian/*.substvars are rewritten in place and several are tracked; they are
claimed too. debhelper's .debhelper/ and *.debhelper.log are never tracked
and are removed with the rest of the residue.

clean_debian_residue runs from collect_debs, which is OUTSIDE
with_prepared_tree -- the restore has already happened by then. That is why
it must not remove *.substvars: doing so would delete the tracked ones it
just put back. The claim mechanism handles those instead.

Verified on xcat-master-ub: two consecutive full 14-package builds in one
checkout both succeed, and a build against a git checkout now leaves zero
tracked files modified or deleted (was five, two of them deleted). The
remaining untracked residue -- pods/, share/, pod2htmd.tmp and the
substvars of packages that do not track one -- is inherited from
build-ubunturepo and unchanged here.

Also covers the guard that gives this PR its name: deleting
`return if $opts{source_only}` from buildall left buildrpms_source_only.t
green, so nothing checked that --source-only skips the binary rebuild. It
does now, and the mutation reddens two assertions. Same for the changelog
trailer: making its substitution global left the suite green because only
the older header was asserted, not its author and date.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
 2026-09-02 12:39:44 -03:00
Daniel Hilst 2bfad97348 fix(build): address review on builddebs.pl, and repoint CI at the new repo 
Four fixes from @viniciusferrao's review plus the CI break his review predates.

orig tarball version. dpkg looks for <source>_<upstream>.orig.tar.gz with no
Debian revision, and the call site passed the full Version-Release. The rule now
lives in BuildUtils::upstream_version and orig_tarball_name applies it, so the
call site cannot get it wrong whichever string it is handed. Currently dormant --
every package is Format: 1.0, so the quilt branch does not run, which is why the
differential build did not catch it.

--dest could write to the filesystem root. Cwd::abs_path returns undef when a
PARENT component is missing (a missing leaf is fine), and the caller interpolated
that, so `--dest /no/such/parent/out` became `/debs` and `/xcat-core` at /.
Replaced with BuildUtils::resolve_dest, which is rel2abs and purely lexical --
correct for an output directory that does not exist yet.

Generated debian/control left behind. xCAT-genesis-scripts has no debian/control
of its own; it is generated from control-<arch>. The cleanup restored only files
that already existed, so the generated one stayed. Worse than dirty: ppc64el ran
last, so the restore put back the amd64 BACKUP and the leftover was the wrong
architecture's control, which a later single-arch build would have started from.
with_prepared_tree now records created files and removes them. Verified by a real
build: the checkout is byte-clean afterwards, matching the oracle.

CI install step. build-ubunturepo wrote its repo to $curdir/../../xcat-core,
which under GitHub's work/<repo>/<repo> layout IS $RUNNER_WORKSPACE, so
install_xcat's `./mklocalrepo.sh` happened to be in the directory it chdir'd to.
builddebs.pl writes inside the checkout instead -- that outside-the-checkout path
is what used to rm -rf the tree -- so install_xcat now names the script by its
real location and fails with a clear message if the build produced no repository.
This is what reddened xcat_pr_test at 2m13s; the builder itself was fine (the
exact CI invocation, `./builddebs.pl --force` with no --dest, returns 0 with all
14 packages).

The executable bit was already fixed before the review landed.

Both new helpers are tested and mutation-verified: not stripping the revision
reddens 3 assertions, swapping rel2abs back to abs_path reddens 2. Equivalence
re-measured after these changes -- all 14 packages identical to build-ubunturepo
in control and in every non-changelog file by md5.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
 2026-09-02 12:39:44 -03:00
Daniel Hilst 5abd4327c4 fix(build): keep build-ubunturepo as an oracle, and fix what running it found 
builddebs.pl was written against build-ubunturepo but never run beside it. Keeping
the old script in the tree for one more cycle makes the comparison repeatable, and
running the two on xcat-master-ub found four defects that no unit test would have.

  * builddebs.pl was not executable. github_action_xcat_test.pl runs
    `sudo ./builddebs.pl`, so CI would have failed with Permission denied.

  * .deb files were collected after every ARCHITECTURE. dpkg-genbuildinfo reads
    the sibling artifacts of the source package it is building, so moving the
    amd64 .deb away before ppc64el ran killed the second build with
      dpkg-genbuildinfo: error: cannot fstat file ../xcat_..._amd64.deb
    Collection now happens once a package's last architecture is done.

  * A checkout dirtied by an aborted run poisoned the next one: a leftover
    .changes made dpkg-genbuildinfo name an architecture the run had not reached.
    The dpkg output is now cleared at start.

  * The maintainer was xcat@xcat.org where build-ubunturepo uses
    xcat-build@xcat.org, and a Release file in the tree was ignored.
    buildrpms.pl WRITES Release, so a pipeline building both would have stamped
    its debs with a different release from its rpms. Both corrected.

Equivalence, measured rather than assumed. Both builders were run on
xcat-master-ub from the same tree:

  package set      identical, 14/14, same names
  control metadata identical, 14/14
  payload contents identical, 14/14
  changelog.Debian differs, 2 packages -- deliberately, see below

Byte-identity is NOT the criterion, because neither implementation has it: two
consecutive runs of build-ubunturepo on the same tree agree on 0 of 14 packages.
A .deb records the build wall-clock time in its ar member mtimes and
SOURCE_DATE_EPOCH does not reach dpkg-deb here, so every run differs from every
other. That is a pre-existing property of the Debian build, not a regression, and
worth fixing separately.

The changelog difference is build-ubunturepo being wrong. Its

    sed -i "s/^ -- .*/ -- $DEBFULLNAME <$DEBEMAIL>  $deterministic_date/"

carries no line address, so it rewrites EVERY trailer in debian/changelog:
"OCF xCAT <xcat@ocf.co.uk>  Mon, 25 Oct 2010" ships as "xCAT Build
<xcat-build@xcat.org>  Tue, 01 Sep 2026". It falsifies the authorship and dates of
the 2008 and 2010 releases. builddebs.pl rewrites only the top stanza. Matching
byte-for-byte would mean reproducing the defect, so this difference stays.

(xcat-vlan appeared to differ under `diff -r`; that was diff reporting dangling
symlinks in both trees. Its member listing is identical.)

build-ubunturepo is documented as retained-for-comparison and not to be extended.
It goes once the Ubuntu CD pipelines call builddebs.pl.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
 2026-09-02 12:39:43 -03:00
Daniel Hilst b8510e1be3 refactor(build): replace build-ubunturepo with builddebs.pl and BuildUtils.pm 
build-ubunturepo was 710 lines of shell doing the Debian half of what
buildrpms.pl does for rpms, with no code in common and a different CLI. It also
carried paths that are dead: GSA uploads, the PROMOTE/PREGA release flows, and a
-d mode that built an xcat-dep repository from a different project's packages.

builddebs.pl replaces it and mirrors buildrpms.pl -- Getopt::Long options, one
package list, build then index then sign -- so the two builders read the same way
and share BuildUtils.pm.

The design rests on one fact: xcat-core debs are Perl. They are byte-identical
for every Ubuntu release, so they are built ONCE and the same files are published
into every codename. Only xCAT, xCATsn and xCAT-genesis-scripts carry an
architecture, and there the difference is packaging metadata, not compiled
output. That is why this needs no sbuild and no per-codename chroot -- unlike
xcat-dep, whose packages are compiled and genuinely differ per release.

BuildUtils.pm holds what both builders need and what was worth making testable:
the Version-Release derivation from the commit time, the xCAT-probe helper
staging, the deb arch and dist tables, the debian/control version pinning, the
changelog rewrite, the reprepro conf generation, and the build lock. Every
function is pure or takes its side effect as an argument, so build_utils.t (45
assertions) drives each one rather than grepping a builder for evidence that it
is called. Verified by mutation: shrinking the arch table reddens 1, dropping
the /g from the control pin reddens 2.

The env-var CLI maps to options: BUILDALL=1 -> --force, GPGSIGN=1 -> --gpg-sign,
GPG_HOME -> --gpg-home, DEST -> --dest, DISTS -> --dist (repeatable). UP=0 has no
equivalent because uploading is gone -- the CD pipeline's deploy step publishes.

Callers updated: github_action_xcat_test.pl and travis.pl. The comment in
github_action_xcat_test.pl explaining why CI copies the tree before building is
corrected -- build-ubunturepo rm -rf'd $curdir/../../xcat-core, which under
GitHub's work/<repo>/<repo> layout is the checkout's own parent; builddebs.pl
writes under dist/debs inside the checkout and restores every file it edits, so
the copy is now only isolating the tests from build residue.

Two tests moved with it. build_ubunturepo_lock.t extracted the lock out of the
shell with a regex and ran that; the lock is now a function, so builddebs_lock.t
calls it -- and asserts what actually matters, that two builds of one checkout
fail fast while two builds of different checkouts run concurrently.
ubuntu_2604_pkglist.t asserted that resolute appeared in a shell fragment of
build-ubunturepo's source; it now asks BuildUtils for the release list and checks
a resolute stanza reaches conf/distributions. That assertion would have passed on
any file containing the fragment and broken on a reflow that changed nothing.

Verified: prove -r xCAT-test/unit fails on 6 files here against 7 on
upstream/master, the difference being apache_config_sources.t, fixed by the
preceding commit. The remaining 6 are missing DB modules on the machine that ran
it and are identical on both.

NOT done here, and required before this can merge: the Ubuntu core CD pipelines
still invoke ./build-ubunturepo (ci/ubuntu/Jenkinsfile.core-ubuntu-{devel,stable}
in VersatusHPC/xcat-core-ci-cd, and the inline script in each live Jenkins job).
Those must be switched to builddebs.pl in the same change window.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
 2026-09-02 12:39:43 -03:00