run_bounded signalled the process group on a timeout and on a cancellation, but
not when the leader itself died. A SIGKILL, or the OOM killer, takes the shell
and leaves schroot and qemu in the group, holding the chroot and writing into
staging after the call reports the build finished.
Those processes are not children of this one, so there is nothing to wait for.
Signal the group on the way out.
A child killed by a signal leaves 0 in the high byte of its wait status. The
per-codename worker loop read only that byte, so a cancelled or OOM-killed
codename was counted as built, and the run could reach validation with the
staging tree that worker never finished.
One helper decodes a wait status for both the worker loop and run_bounded, and
reports 128 plus the signal for a child the kernel killed.
The forwarding handler was in place before the fork, but the parent recorded the
worker pid after it. A signal in between reached a handler that did not know the
worker, so the orchestrator died while the worker kept building for hours and
held the per-architecture lock.
The mask discipline run_bounded already used is now a pair of helpers, and the
worker loop blocks the handled signals across both the fork and the
registration. The worker resets the inherited handlers before restoring the mask.
The handlers were installed after fork(), so a signal arriving in between killed
the wrapper under its inherited handler and left the new process group running.
INT, TERM and HUP are now blocked around the fork and delivered once the handler
is in place. The child restores the mask before exec, or the build would inherit
a blocked TERM and ignore the signal the forwarding depends on.
Both orchestrators fork a worker per codename or per build step, so a signal
sent to the orchestrator never reached the builds: run_bounded's forwarding
covers the build inside one worker, not the workers themselves. The
orchestrator exited and released its locks while its workers kept building,
and the next run raced processes it could not see.
One handler now passes INT, TERM and HUP to the live workers, waits for them
and re-raises the signal, shared by the sbuild loop and both ForkManager pools.
With no deadline the helper called system(), which leaves the build in the
orchestrator's process group and installs no signal handler. A cancellation
then killed the orchestrator, which released its locks while the build kept
writing into staging, and a build killed by a signal was reported as rc=0.
Both paths now fork; a timeout of 0 removes the deadline, nothing else.
The bounded build ran in its own process group, but a signal to the
orchestrator was not forwarded to it. On cancellation the orchestrator
exited and released its locks while schroot, mock and qemu kept running and
writing into staging, so the next run raced an orphan it could not see.
INT, TERM and HUP now reap the group before the process dies by the same
signal, which keeps the exit status honest.
Every per-package build ran through a bare system() call with no wall-clock
bound. Under qemu-user a build can deadlock -- a riscv64 goconserver `go build`
held both Go pids in futex_wait for 26 minutes with no CPU ticks and no open
socket -- and the step then never returns. The pipeline does not go red; it
stops, and a stopped run reads as "still running".
XCAT::BuildUtils::run_bounded runs the command in its own process group, kills
that group when a budget expires, and prints first what the manual
investigation had to collect by hand: the process tree, each pid's kernel wchan
and stack, its open socket count, and the CPU ticks the group used across a
20-second sample. Zero ticks names a deadlock; ticks name a build that is only
slow. BuildUtils::build_deb_in_chroot bounds every Ubuntu package build, and
mockbuild-all.pl bounds the dep and perl steps of a forcearch target.
The budget is 900 seconds for a native build and ten times that for a foreign
architecture, because qemu-user under TCG runs at roughly a tenth of native
speed. Both sit about five times above the slowest build measured on
xcat-master-ub: 3 minutes native, and 26 minutes for riscv64 ipmitool-xcat on
resolute with four codenames building at once. Native mock steps stay unbounded
-- no measurement of them exists, and a guessed budget would turn a trusted
cell red. sbuild-all.pl --build-timeout and mockbuild-all.pl --build-timeout
override the default; 0 removes the bound.
t/build_timeout.t fails without this change: run_bounded never returns and the
test reports the hang instead of blocking.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Tolerating individual dep-builder failures is deliberate: some packages are el-
or arch-pinned and are expected to fail on some targets. Tolerating all of them
is not -- it means the builder is unusable, the invocation produced nothing, and
whatever the run publishes came from somewhere other than this build.
Count the failures on both the serial and the parallel path and stop when they
account for every attempted step, before collection can take an earlier run's
artifacts for this one's. The rule itself lives in BuildUtils, where it can be
exercised without a builder.
Vinícius Ferrão
Daniel Hilst