The handlers were installed after fork(), so a signal arriving in between killed
the wrapper under its inherited handler and left the new process group running.
INT, TERM and HUP are now blocked around the fork and delivered once the handler
is in place. The child restores the mask before exec, or the build would inherit
a blocked TERM and ignore the signal the forwarding depends on.
Both orchestrators fork a worker per codename or per build step, so a signal
sent to the orchestrator never reached the builds: run_bounded's forwarding
covers the build inside one worker, not the workers themselves. The
orchestrator exited and released its locks while its workers kept building,
and the next run raced processes it could not see.
One handler now passes INT, TERM and HUP to the live workers, waits for them
and re-raises the signal, shared by the sbuild loop and both ForkManager pools.
With no deadline the helper called system(), which leaves the build in the
orchestrator's process group and installs no signal handler. A cancellation
then killed the orchestrator, which released its locks while the build kept
writing into staging, and a build killed by a signal was reported as rc=0.
Both paths now fork; a timeout of 0 removes the deadline, nothing else.
The bounded build ran in its own process group, but a signal to the
orchestrator was not forwarded to it. On cancellation the orchestrator
exited and released its locks while schroot, mock and qemu kept running and
writing into staging, so the next run raced an orphan it could not see.
INT, TERM and HUP now reap the group before the process dies by the same
signal, which keeps the exit status honest.
Every per-package build ran through a bare system() call with no wall-clock
bound. Under qemu-user a build can deadlock -- a riscv64 goconserver `go build`
held both Go pids in futex_wait for 26 minutes with no CPU ticks and no open
socket -- and the step then never returns. The pipeline does not go red; it
stops, and a stopped run reads as "still running".
XCAT::BuildUtils::run_bounded runs the command in its own process group, kills
that group when a budget expires, and prints first what the manual
investigation had to collect by hand: the process tree, each pid's kernel wchan
and stack, its open socket count, and the CPU ticks the group used across a
20-second sample. Zero ticks names a deadlock; ticks name a build that is only
slow. BuildUtils::build_deb_in_chroot bounds every Ubuntu package build, and
mockbuild-all.pl bounds the dep and perl steps of a forcearch target.
The budget is 900 seconds for a native build and ten times that for a foreign
architecture, because qemu-user under TCG runs at roughly a tenth of native
speed. Both sit about five times above the slowest build measured on
xcat-master-ub: 3 minutes native, and 26 minutes for riscv64 ipmitool-xcat on
resolute with four codenames building at once. Native mock steps stay unbounded
-- no measurement of them exists, and a guessed budget would turn a trusted
cell red. sbuild-all.pl --build-timeout and mockbuild-all.pl --build-timeout
override the default; 0 removes the bound.
t/build_timeout.t fails without this change: run_bounded never returns and the
test reports the hang instead of blocking.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Tolerating individual dep-builder failures is deliberate: some packages are el-
or arch-pinned and are expected to fail on some targets. Tolerating all of them
is not -- it means the builder is unusable, the invocation produced nothing, and
whatever the run publishes came from somewhere other than this build.
Count the failures on both the serial and the parallel path and stop when they
account for every attempted step, before collection can take an earlier run's
artifacts for this one's. The rule itself lives in BuildUtils, where it can be
exercised without a builder.
Vinícius Ferrão
Daniel Hilst