database/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2026-10-23 21:28:56 +03:00
Code Activity
Files
59bd2cd96cfba2b02b07e81ade57d9beb5f2812e
pgbackrest/test/ci.py
David Steele 59bd2cd96c Migrate the build and documentation code from C to Python. 
The build tool in src/build becomes build/build.py with its library in build/lib, and the C part of the documentation tool in doc/src becomes doc/doc.py with its library in doc/lib. They migrate together because doc/src was built on src/build, linking its config and help parsing along with its extensions to the core types. doc.pl runs doc.py rather than building and running a C binary, and meson runs build.py for the code it generates as it builds.

The three libraries form the hierarchy the C include path did: build/lib is the base, doc/lib sits on it, and test/lib sits on both. They are implicit namespace packages so a package spans them, and a module lives in exactly one library and holds the union of what the tools need from it, since Python resolves a module name to a single file and cannot extend one the way the C included a core module and added to it. define.yaml declares which libraries each tool may import and the test runner puts only those on the path, so a test of the build library cannot reach the doc or test library any more than the build tool can.

The declarations move with the code. config.yaml, error.yaml, and postgres.yaml are now at the root of build, and help.xml becomes doc/xml/reference.xml since it is documentation that the help happens to be generated from, so the generator is told where it is rather than knowing. dist.sh moves to build as well. The documentation tool's own option declarations are gone because they existed only to generate the option parser for the C binary and doc.py parses its command line directly.

Three modules in src/build/common were never build tool code. exec, regExp, and string extend the core types for the test harness and the unit tests, which were their only users, so they move to test/src/harness. The json and xml extensions had no users outside the two tools and are removed with them, and the tests for both are trimmed to the core functions that remain.

The help unit test built its fixture by calling the generator's C functions directly, which was the last thing outside the tools that depended on them. It now runs build.py help-data to write the packed help and loads that, so it still runs against the current declarations without needing the generated C, which may not have been written when the test runs.

The manual page wrapping is a port of the wrapper the binary uses for its own help, so the manual page and pgbackrest help break lines the same way, including where a word longer than a line is left unbroken.

libyaml is no longer needed to build pgBackRest since the only thing that read yaml was the C generator. It is dropped from the development container and from every CI platform, and meson no longer looks for it. Code generation needs nothing compiled first either, so a test run no longer builds a generator before it can select the tests.

The common source list is merged into the pgbackrest source list since it existed only so the generator could be built from the same files, and the entries that were out of alphabetical order are put in order while the list is being rewritten anyway.

Every file the generator writes is byte identical to what the C wrote, for the project declarations and for the documentation's own. The documentation is checked the same way: the HTML rendered on this branch is byte identical to the HTML rendered on integration, every file of it, which exercises the whole pipeline on the real content rather than only the four files the tool writes.

Every module in both libraries is declared in define.yaml and has full line and branch coverage. The coverage list is keyed by file rather than by module name, since the same name in two languages is two files and one of them would have silently lost its requirement while both existed.

TestError becomes ToolError, since it is the error type for three tools now rather than for the harness alone.
2026-07-30 19:37:43 +07:00

206 lines
7.9 KiB
Python
Executable File
Raw Blame History

#!/usr/bin/env python3
"""pgBackRest CI Wrapper.
Prepares a CI runner and then runs the tests or builds the documentation on it. A separate program from the test harness because it
is about the runner rather than about the tests, and because it runs before the runner has what the harness needs.
This is the only place that knows what a runner is missing, since a runner starts from a stock image rather than from one of the
test images the tests themselves run in.
All output, including errors, goes to stdout so the run reads in the order it happened."""
####################################################################################################################################
import argparse
import os
import signal
import sys
import time
import traceback
# Send everything written to stderr to stdout instead so the output is in the order it happened
sys.stderr = sys.stdout
# Do not cache bytecode. The harness runs from a copy of the repository that the linter then scans, so a __pycache__ written during
# import would be reported as an unexpected binary file. This must be set before the harness modules are imported below.
sys.dont_write_bytecode = True
# Each tool keeps its library beside itself and may use the libraries below it in the hierarchy, which for the harness is all of
# them. Insert them first, lowest last, so the harness modules are found before anything else on the path.
for lib in ("build", "doc", "test"):
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), lib, "lib"))
from common.error import EXIT_ERROR, ToolError # noqa: E402
from common.exec import exec_one # noqa: E402
from common.log import * # noqa: E402
from common.user import user_name # noqa: E402
from common.vm import * # noqa: E402
from config.config import PROJECT_EXE, project_version # noqa: E402
####################################################################################################################################
def cfg_load(arg_list, path_repo):
"""Parse the command line and initialize logging."""
parser = argparse.ArgumentParser(prog="ci.py", description="pgBackRest CI Wrapper")
parser.add_argument("--version", action="version", version="pgBackRest %s CI Wrapper" % project_version(path_repo))
parser.add_argument("target", choices=("doc", "test"), help="run the tests or build the documentation")
parser.add_argument("--vm", default=VM_NONE, metavar="VM", help="vm to run on")
parser.add_argument("--vm-arch", metavar="ARCH", help="vm architecture (defaults to the host architecture)")
parser.add_argument("--param", action="append", default=[], metavar="OPTION", help="option to pass to the test harness")
parser.add_argument("--sudo", action="store_true", help="leave passwordless sudo in place for the run")
parser.add_argument("--no-tempfs", dest="tempfs", action="store_false", help="do not mount a tmpfs for the test path")
config = parser.parse_args(arg_list)
config.repo_path = path_repo
log_init(INFO, True)
return config
####################################################################################################################################
def step(title, command_list, show_output=False):
"""Run the commands that make up one step of the run and report how long the step took."""
time_begin = time.time()
log(INFO, "Begin %s" % title)
for command in command_list:
log(INFO, " Exec %s" % command)
exec_one(command, show_output=show_output)
log(INFO, " End %s (%us)" % (title, int(time.time() - time_begin)))
####################################################################################################################################
def sudo_remove():
"""Remove passwordless sudo so a test cannot depend on having it.
A runner grants it to every user, which the test containers do not."""
step("remove sudo", ["sudo rm /etc/sudoers.d/%s" % user_name()])
####################################################################################################################################
def doc_build(config):
"""Build the documentation."""
# The documentation is the only thing left that runs on Perl
step("install documentation packages", ["sudo DEBIAN_FRONTEND=noninteractive apt-get install -y libxml-checker-perl"])
sudo_remove()
step(
"create link from home to repo for contributing doc",
["ln -s %s %s/%s" % (config.repo_path, os.environ["HOME"], PROJECT_EXE)],
)
step(
"release documentation",
["%s/doc/release.pl --build --no-gen --vm=%s" % (config.repo_path, config.vm)],
show_output=True,
)
####################################################################################################################################
def test_run(config):
"""Run the tests."""
vm_arch = "" if config.vm_arch is None else " --vm-arch=%s" % config.vm_arch
# Packages the tests need that the test images already have
package = "gcc ccache git rsync zlib1g-dev libssl-dev libxml2-dev libpq-dev pkg-config libssh2-1-dev valgrind"
# Extra packages required when testing without containers
if config.vm == VM_NONE:
package += " liblz4-dev liblz4-tool zstd libzstd-dev bzip2 libbz2-dev"
step(
"/tmp/pgbackrest owned by root so tests cannot use it",
["sudo mkdir -p /tmp/%s && sudo chown root:root /tmp/%s && sudo chmod 700 /tmp/%s" % ((PROJECT_EXE,) * 3)],
)
step(
"install test packages",
["sudo DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y %s" % package],
)
if not config.sudo:
sudo_remove()
# Build the containers
if config.vm != VM_NONE:
step(
"%s build" % config.vm,
["%s/test/test.py --vm-build --vm=%s%s" % (config.repo_path, config.vm, vm_arch)],
show_output=True,
)
step(
("no container" if config.vm == VM_NONE else config.vm) + " test",
[
"%s/test/test.py --vm-max=2 --vm=%s%s%s"
% (config.repo_path, config.vm, vm_arch, "".join(" --%s" % param for param in config.param))
],
show_output=True,
)
####################################################################################################################################
def command_run(config):
"""Prepare the runner and then run what was asked for on it."""
step(
"install common packages",
[
# A stale package index is not worth failing the run over since the install below will report what is missing
"sudo apt-get -qq update || true",
"sudo DEBIAN_FRONTEND=noninteractive apt-get install -y meson python3-yaml",
],
)
# The test path is a tmpfs so the tests are not limited by the runner's disk, which is slow and shared
if config.tempfs:
step(
"mount tmpfs",
["mkdir -p -m 770 test", "sudo mount -t tmpfs -o size=2048m tmpfs test", "df -h test"],
show_output=True,
)
if config.target == "doc":
doc_build(config)
else:
test_run(config)
log(INFO, "CI Complete")
return 0
####################################################################################################################################
def main():
"""Main."""
# Die silently on SIGPIPE as C programs do, rather than raising when output is piped to a command that exits early
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
path_repo = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
try:
return command_run(cfg_load(sys.argv[1:], path_repo))
except ToolError as error:
log(ERROR, error)
return error.status
except Exception:
# An unexpected exception is a bug here rather than a problem with the run, so show the traceback
print(traceback.format_exc())
return EXIT_ERROR
if __name__ == "__main__":
sys.exit(main())
View Git Blame Copy Permalink