You've already forked pgbackrest
mirror of
https://github.com/pgbackrest/pgbackrest.git
synced 2026-10-23 21:28:56 +03:00
59bd2cd96c
The build tool in src/build becomes build/build.py with its library in build/lib, and the C part of the documentation tool in doc/src becomes doc/doc.py with its library in doc/lib. They migrate together because doc/src was built on src/build, linking its config and help parsing along with its extensions to the core types. doc.pl runs doc.py rather than building and running a C binary, and meson runs build.py for the code it generates as it builds. The three libraries form the hierarchy the C include path did: build/lib is the base, doc/lib sits on it, and test/lib sits on both. They are implicit namespace packages so a package spans them, and a module lives in exactly one library and holds the union of what the tools need from it, since Python resolves a module name to a single file and cannot extend one the way the C included a core module and added to it. define.yaml declares which libraries each tool may import and the test runner puts only those on the path, so a test of the build library cannot reach the doc or test library any more than the build tool can. The declarations move with the code. config.yaml, error.yaml, and postgres.yaml are now at the root of build, and help.xml becomes doc/xml/reference.xml since it is documentation that the help happens to be generated from, so the generator is told where it is rather than knowing. dist.sh moves to build as well. The documentation tool's own option declarations are gone because they existed only to generate the option parser for the C binary and doc.py parses its command line directly. Three modules in src/build/common were never build tool code. exec, regExp, and string extend the core types for the test harness and the unit tests, which were their only users, so they move to test/src/harness. The json and xml extensions had no users outside the two tools and are removed with them, and the tests for both are trimmed to the core functions that remain. The help unit test built its fixture by calling the generator's C functions directly, which was the last thing outside the tools that depended on them. It now runs build.py help-data to write the packed help and loads that, so it still runs against the current declarations without needing the generated C, which may not have been written when the test runs. The manual page wrapping is a port of the wrapper the binary uses for its own help, so the manual page and pgbackrest help break lines the same way, including where a word longer than a line is left unbroken. libyaml is no longer needed to build pgBackRest since the only thing that read yaml was the C generator. It is dropped from the development container and from every CI platform, and meson no longer looks for it. Code generation needs nothing compiled first either, so a test run no longer builds a generator before it can select the tests. The common source list is merged into the pgbackrest source list since it existed only so the generator could be built from the same files, and the entries that were out of alphabetical order are put in order while the list is being rewritten anyway. Every file the generator writes is byte identical to what the C wrote, for the project declarations and for the documentation's own. The documentation is checked the same way: the HTML rendered on this branch is byte identical to the HTML rendered on integration, every file of it, which exercises the whole pipeline on the real content rather than only the four files the tool writes. Every module in both libraries is declared in define.yaml and has full line and branch coverage. The coverage list is keyed by file rather than by module name, since the same name in two languages is two files and one of them would have silently lost its requirement while both existed. TestError becomes ToolError, since it is the error type for three tools now rather than for the harness alone.
207 lines
8.2 KiB
Python
207 lines
8.2 KiB
Python
"""Xml Handler.
|
|
|
|
Thin helpers over the standard library element tree, shaped the way the documentation reads and writes xml.
|
|
|
|
Text is the awkward part. A document like the user guide is mixed content -- text and markup interleaved in one paragraph -- which the
|
|
element tree stores as the text before a node's first child plus the text trailing each child. So adding text to a node means appending
|
|
to whichever of those came last, and reading a node's text means gathering all of it. Both are done here so nothing else has to know.
|
|
|
|
Comments never appear in a parsed document, since the parser drops them, so nothing here has to skip them."""
|
|
|
|
####################################################################################################################################
|
|
import copy
|
|
import re
|
|
import xml.etree.ElementTree as etree
|
|
|
|
from common.error import ToolError
|
|
|
|
|
|
####################################################################################################################################
|
|
class XmlDocument:
|
|
"""An xml document, i.e. a root node plus the declarations that go above it."""
|
|
|
|
def __init__(self, root, dtd_name=None, dtd_file=None):
|
|
self.root = root
|
|
self.dtd_name = dtd_name # Document type, if any
|
|
self.dtd_file = dtd_file # File the document type is declared in
|
|
|
|
################################################################################################################################
|
|
def render(self):
|
|
"""Render the document as the xml text of it."""
|
|
|
|
result = '<?xml version="1.0" encoding="UTF-8"?>\n'
|
|
|
|
if self.dtd_name is not None:
|
|
result += '<!DOCTYPE %s SYSTEM "%s">\n' % (self.dtd_name, self.dtd_file)
|
|
|
|
return result + etree.tostring(self.root, encoding="unicode") + "\n"
|
|
|
|
|
|
# Document type a document declares, which is kept so a document that is read and written again declares the same
|
|
_DTD_EXP = re.compile(r"""<!DOCTYPE\s+(\S+)\s+SYSTEM\s+"([^"]+)"\s*>""")
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_document_new(root_name, dtd_name=None, dtd_file=None):
|
|
"""Build an empty document."""
|
|
|
|
return XmlDocument(etree.Element(root_name), dtd_name, dtd_file)
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_document_parse(content, path):
|
|
"""Parse a document, keeping the document type it declares."""
|
|
|
|
match = _DTD_EXP.search(content)
|
|
|
|
return XmlDocument(xml_parse(content, path), *(match.groups() if match is not None else (None, None)))
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_parse(content, path):
|
|
"""Parse xml, naming the file in any error since the message is otherwise only a line number."""
|
|
|
|
try:
|
|
return etree.fromstring(content)
|
|
except etree.ParseError as error:
|
|
raise ToolError("unable to parse '%s': %s" % (path, error))
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_attribute(node, name, error_on_missing=False):
|
|
"""Attribute of a node, or None when it is not there."""
|
|
|
|
result = node.get(name)
|
|
|
|
if result is None and error_on_missing:
|
|
raise ToolError("unable to find attribute '%s' in node '%s'" % (name, node.tag))
|
|
|
|
return result
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_attribute_set(node, name, value):
|
|
"""Set an attribute of a node."""
|
|
|
|
node.set(name, value)
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_attribute_remove(node, name):
|
|
"""Remove an attribute of a node, which is not an error when it is not there."""
|
|
|
|
node.attrib.pop(name, None)
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_child(node, name, error_on_missing=False):
|
|
"""First child of a node with a name, or None when there is none."""
|
|
|
|
result = node.find(name)
|
|
|
|
if result is None and error_on_missing:
|
|
raise ToolError("unable to find child '%s' in node '%s'" % (name, node.tag))
|
|
|
|
return result
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_child_list(node, name):
|
|
"""Every child of a node with a name."""
|
|
|
|
return node.findall(name)
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_add(node, name):
|
|
"""Add a child to a node."""
|
|
|
|
return etree.SubElement(node, name)
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_dup(node):
|
|
"""Copy a node and everything under it, so what is copied can be changed without changing what it came from."""
|
|
|
|
return copy.deepcopy(node)
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_content(node):
|
|
"""All the text under a node, with the markup dropped."""
|
|
|
|
return "".join(node.itertext())
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_content_add(node, content):
|
|
"""Add text to the end of what a node holds.
|
|
|
|
Text is added rather than set because a title is built from text and markup in turn, e.g. "Backup Command (" then an id then ")".
|
|
"""
|
|
|
|
if len(node) == 0:
|
|
node.text = (node.text or "") + content
|
|
else:
|
|
node[-1].tail = (node[-1].tail or "") + content
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_text_set(node, content):
|
|
"""Replace everything a node holds with text."""
|
|
|
|
for child in list(node):
|
|
node.remove(child)
|
|
|
|
node.text = content
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_child_add(node, source):
|
|
"""Add a copy of everything a source node holds to another node."""
|
|
|
|
xml_node_content_add(node, source.text or "")
|
|
|
|
for child in source:
|
|
added = xml_node_add(node, child.tag)
|
|
|
|
for name, value in child.attrib.items():
|
|
xml_node_attribute_set(added, name, value)
|
|
|
|
xml_node_child_add(added, child)
|
|
xml_node_content_add(node, child.tail or "")
|
|
|
|
|
|
####################################################################################################################################
|
|
def xml_node_child_replace(parent, node, source):
|
|
"""Replace a node with a copy of everything a source node holds.
|
|
|
|
The parent is passed because a node does not know what holds it, and what replaces it takes its place rather than going at the
|
|
end."""
|
|
|
|
index = list(parent).index(node)
|
|
|
|
# Build the replacement in a node of its own so the copy can be made with the same code that adds children anywhere else
|
|
replace = etree.Element(node.tag)
|
|
xml_node_child_add(replace, source)
|
|
|
|
# Whatever trailed the node goes after whatever now ends the replacement
|
|
tail = node.tail or ""
|
|
parent.remove(node)
|
|
|
|
for offset, child in enumerate(list(replace)):
|
|
parent.insert(index + offset, child)
|
|
|
|
# Text the replacement begins with belongs to what precedes the node, i.e. the parent or the child before it
|
|
if index == 0:
|
|
parent.text = (parent.text or "") + replace.text
|
|
else:
|
|
parent[index - 1].tail = (parent[index - 1].tail or "") + replace.text
|
|
|
|
if len(replace) > 0:
|
|
parent[index + len(replace) - 1].tail = (parent[index + len(replace) - 1].tail or "") + tail
|
|
elif index == 0:
|
|
parent.text = (parent.text or "") + tail
|
|
else:
|
|
parent[index - 1].tail = (parent[index - 1].tail or "") + tail
|