You've already forked pgbackrest
mirror of
https://github.com/pgbackrest/pgbackrest.git
synced 2026-10-23 21:28:56 +03:00
fa3d1979b7
doc.pl and release.pl become doc.py and release.py and the Perl library in doc/lib/pgBackRestDoc is gone, which is 32 files and 11,059 lines replaced by about 3,850 lines of Python. Nothing in the project runs on Perl now. doc.pl and doc.py merge into one tool rather than one running the other. doc.py built the reference documents and the preprocessed user guide, wrote them to doc/output/xml, and doc.pl read them back; they are now passed in memory and those files are not written at all. That also removes a duplication the split had hidden: the preprocessor added a document's variables to its own store and left the declarations in the file, so the Perl parsed them again into a store of its own and the same variable existed twice with different values. Most of Doc.pm does not come across. Its parse step rebuilt the parsed tree into a structure of its own and its build step invented a second way to address that structure, so a document was reached through three accessors where the tree has one. What survives is about ninety lines in common/xml.py: a field is the text of a child that holds nothing else, the text of a node is the node itself for the tags that are text and a text child for the rest, and asking for a child rather than the list of them now reports a second one as the mistake in the document that it is. External entities are expanded before the parser sees a document, since the standard library does not fetch them. The release list is assembled from one file per release this way, 122 of them, and a declaration that is never referenced is reported rather than leaving the file it names silently unread. Nothing validated against a document type, so the document type declarations and the two dtd files are removed. A declaration that is not enforced does not get maintained. The two documents that assemble themselves from parts keep the declaration because the parts are declared inside it. The release notes become a document the tool builds, like the command and configuration references, rather than something the renderer reaches into a custom module to produce. A variable may still be evaluated rather than written out, but it is a Python expression against a short list of names rather than a Perl program. The three that ask for a fact about the environment keep doing so. The four that read the project version, the current stable version, and the release date are facts the tool computes, since working them out involved constructing library objects and reading the release list a second time from inside a variable body. exec and user move down from test/lib to doc/lib because the documentation needs both, the project version splits out into config/project, and eval and var_store move to common now that the manifest uses them as much as the preprocessor. The version is built from the parts in src/version.h rather than read from the string generated from them, which is what the Perl did and what a release needs, since a release clears the development suffix and the string is only regenerated by a test run. libxml-checker-perl is no longer installed on CI runners, in the development container, or in the documentation images, and the RHEL image no longer installs cpanminus and a toolchain to build XML::Checker::Parser from CPAN. The user guide never ran Perl, and the Debian image never had a parser and built the same guide. Every module in doc/lib is declared in define.yaml and has full line and branch coverage. Ubuntu 24.04 ships a user and a group at id 1000, so a build by a user with that id could not create the test user. The test image removes whatever holds the id before creating it, as it already did for the group, and the development container image removes both because adduser gives a new group the id of the user. The coverage summary listed the python under build/lib as modules with mangled names, since it stripped a fixed src/ prefix from paths that no longer start with one. It now summarizes what is under src, which is what the metric page says it reports. Deploying the website no longer removes the links at the top of the site or CNAME. The Perl used find -type f, which does not match a symbolic link, and the port used a test that follows one, which took out the guides kept under retired platform names. CNAME is a regular file and the Perl removed it every deploy, which was never intended. The rendered documentation is byte identical to what the Perl produced, for every page, the manual page, and the generated markdown. The user guide was also built end to end under Docker on both documented platforms, and rendering it again from the cache that build wrote produces the same pages, which exercises the whole path rather than only what can be checked without containers. A release was then simulated from beginning to end.
108 lines
4.3 KiB
Python
108 lines
4.3 KiB
Python
"""Code Format Command.
|
|
|
|
Formats the source to the project standards, or checks that it is already formatted, which is what CI runs. C is formatted with
|
|
uncrustify and python with black, and the execute permissions are checked afterward since a stray execute bit is easy to introduce
|
|
and hard to notice.
|
|
|
|
Only version controlled files are formatted, which is what git reports here, so scratch work in the working tree is left alone."""
|
|
|
|
####################################################################################################################################
|
|
import os
|
|
|
|
from common.error import ToolError
|
|
from common.exec import exec_one
|
|
from common.log import *
|
|
|
|
# Line length black formats python to, which is the line length the rest of the project uses
|
|
_LINE_LENGTH = 132
|
|
|
|
# Files that are allowed to be executable. Everything else that is version controlled must not be, since a stray execute bit is
|
|
# usually the result of an editor or a file system that does not preserve modes.
|
|
_FILE_EXECUTABLE_LIST = (
|
|
"build/build.py",
|
|
"doc/doc.py",
|
|
"doc/release.py",
|
|
"build/dist.sh",
|
|
"test/ci.py",
|
|
"test/smoke.py",
|
|
"test/test.py",
|
|
)
|
|
|
|
|
|
####################################################################################################################################
|
|
def _file_list(path_repo):
|
|
"""The version controlled files, which are the ones that get formatted.
|
|
|
|
Only regular files are returned. A file that is in the index but no longer in the working tree has nothing to format, and git
|
|
reports a symlink to a directory as a single entry rather than descending into it."""
|
|
|
|
file_list = exec_one("git -C %s ls-files -c --others --exclude-standard" % path_repo).splitlines()
|
|
|
|
return [file for file in file_list if os.path.isfile(os.path.join(path_repo, file))]
|
|
|
|
|
|
####################################################################################################################################
|
|
def _format_c(path_repo, file_list, check):
|
|
"""Format the C with uncrustify."""
|
|
|
|
command = "uncrustify -c %s/test/uncrustify.cfg%s" % (path_repo, " --check" if check else " --replace --no-backup")
|
|
|
|
for file in file_list:
|
|
# Skip anything that is not C
|
|
if not file.endswith((".c", ".h", ".c.inc")):
|
|
continue
|
|
|
|
# Skip the test template, which does not format correctly, and vendorized and generated files
|
|
if file == "test/src/test.c" or file.endswith((".vendor.h", ".vendor.c.inc", ".auto.c.inc")):
|
|
continue
|
|
|
|
command += " %s/%s" % (path_repo, file)
|
|
|
|
exec_one(command)
|
|
|
|
|
|
####################################################################################################################################
|
|
def _format_python(path_repo, file_list, check):
|
|
"""Format the python with black."""
|
|
|
|
command = "black --line-length=%u%s" % (_LINE_LENGTH, " --check --diff" if check else "")
|
|
|
|
for file in file_list:
|
|
if file.endswith(".py"):
|
|
command += " %s/%s" % (path_repo, file)
|
|
|
|
exec_one(command)
|
|
|
|
|
|
####################################################################################################################################
|
|
def _permission_check(path_repo, file_list):
|
|
"""Check that nothing is executable that is not meant to be, which formatting can munge."""
|
|
|
|
for file in file_list:
|
|
mode = os.stat(os.path.join(path_repo, file)).st_mode & 0o777
|
|
|
|
if mode & 0o111 and file not in _FILE_EXECUTABLE_LIST:
|
|
raise ToolError("expected mode '%04o' for '%s' but found '%04o'" % (mode & 0o666, file, mode))
|
|
|
|
|
|
####################################################################################################################################
|
|
def cmd_code_format(config):
|
|
"""Format the source, or check that it is already formatted."""
|
|
|
|
path_repo = config.repo_path
|
|
file_list = _file_list(path_repo)
|
|
|
|
log(INFO, "code format" + (" check" if config.check else ""))
|
|
|
|
try:
|
|
_format_c(path_repo, file_list, config.check)
|
|
_format_python(path_repo, file_list, config.check)
|
|
except ToolError as error:
|
|
if not config.check:
|
|
raise
|
|
|
|
# The check is what CI runs, so it says how to fix what it found rather than only what it found
|
|
raise ToolError("%s\nHINT: run 'test.py code-format' to format the code" % error)
|
|
|
|
_permission_check(path_repo, file_list)
|