Daniel Hilst
ce25575d7f
dhcptest and provtest carried two copies of the config loader, the assertion language, the $step.field substitution, the TAP reporter, the error types and the CLI. The copies had already drifted: list equality, multi-line regex and CIDR membership behaved differently in each. blackboxtest keeps one of each. A step type table in steps.py declares the keys, required keys and reply fields of every type, and the runner dispatches to the DHCP client or to the dig, tftp, curl and xcatd clients. One scenario can now use a DHCP reply in a TFTP step. The shipped scenarios move unchanged to conf/dhcp/ and conf/prov/. Their scenario, step and assertion lists are identical to the old ones. The wire cases and fixtures move to autotest/testcase/blackboxtest/, and CI runs one offline phase instead of two. Removed because no scenario or fixture used them: the pretty and json reports, --pcap, --mac, the discover subcommand, the decline, inform and sleep steps, assert_all, select, and the keys dhcptest accepted and ignored. Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT-test
Unit tests that run from the source checkout are split by implementation language:
| Test type | Location | Runner |
|---|---|---|
| Perl unit tests | xCAT-test/unit/*.t |
prove -r xCAT-test/unit |
| Shell unit tests | xCAT-test/bats/*.bats |
bats -r xCAT-test/bats |
| Black-box service tests | xCAT-test/blackboxtest/ |
python3 -m unittest discover -s tests, then blackboxtest run <conf> on a management node |
| CLI functional tests | xCAT-test/autotest/testcase/ and xCAT-test/autotest/bundle/ |
xcattest -f <cluster.conf> -t <case> or xcattest -f <cluster.conf> -b <bundle> |
Use Perl .t tests for Perl modules, Perl scripts, templates, and repository
artifacts. Use BATS tests for shell-script behavior that can be exercised from
the checkout by sourcing a shell library or script and shadowing external
commands.
Shell behavior should not be tested by Perl tests that grep shell source. Put
those tests under xCAT-test/bats instead.
blackboxtest is the odd one out: its Python unit tests and blackboxtest validate run offline like the others, but its point is the third column --
asking a management node what a booting node asks (DHCP, DNS, TFTP, HTTP,
xcatd) and asserting on the answers. It talks to the services only as a client,
so it does not depend on which DHCP server answers or on the xCAT database.
See unit/README.md, bats/README.md and blackboxtest/README.md for the
detailed rules for each suite.