The deployed files are generated rather than hand written, and that option is
not one we can set there, so the templates described a restriction production
does not have. What a key may do is still bounded by restrict and the forced
command, which is the part that matters; what is lost is the second answer to
where it may be used from.
It has been failing check-python-style since it landed: five lines over the
limit, and two breaks before a binary operator, which count here because
.pycodestyle sets an explicit ignore list and so loses pycodestyle's own
default exemption for W503.
sync-ftp-to-s3.py and purge_pgadmin_releases.sh have been living in the
pgaweb checkout under tools/, which was never where they belonged: neither
touches Django, the site database or Varnish, and both work on the download
tree at /var/ftp/pgadmin4, so they are part of publishing rather than part of
the website. They move here, renamed to match the hyphenated names of the
scripts already in this directory, and the README gains them in the file
table, in the install line and in the prose that explains what lives where.
The sync-s3 verb now runs sync-ftp-to-s3.py out of PUBLISH_DIR instead of
TOOLS_DIR, which makes this a deployment-ordering change: the wrapper and the
copy of the script it names must both be installed on the download server
before the pgaweb side deletes its own copy, and the comment on the verb says
so. The three verbs that remain genuinely website operations, create-release,
load-docs and purge-cache, still reach into TOOLS_DIR and are untouched.
purge-pgadmin-releases.sh deliberately gets no verb. It is a by-hand
operation that deletes published content, it defaults to a dry run that
somebody is expected to read before re-running with --live, and the rebuild
scripts it pairs with are already documented as standalone; handing a
compromised runner the ability to delete releases would undo most of what the
wrapper is for.
Both scripts are otherwise unchanged apart from the filename in the purge
script's own header comment, and four lines in the Python rewrapped to 79
columns because pkg/ is covered by check-pep8 and the originals were not.
Every one of these is a way that a verb meant only to add something would
quietly change something already published, which is the failure the wrapper
exists to prevent, so they are worth having together.
Snapshot retention kept the newest five directories, and a directory is
something either key can create: five dated today with high suffixes filled
the keep set and every genuine snapshot was deleted. It now keeps every
directory whose date is among the newest five that are not in the future, so
retention is counted in days, and the intended consequence is that a day with
several builds keeps all of them.
release-fetch required only that v<VERSION> existed, and shutil.move onto an
existing file replaces it, so a release published months ago could be
overwritten and autoupdate-publish and sync-s3 would then carry the
replacement to the auto-updater and into an archive that has no undo. The
single-shot guarantee that release-create made for the first verb now covers
the whole sequence: it refuses before moving anything if .incoming exists or
if a content directory holds more than its maintainer marker, and refuses each
individual move whose destination is already there.
packages-fetch pulled with no --ignore-existing, so a staging package whose
filename matched a published one replaced it and the following rebuild signed
the new bytes. It now leaves the live copy alone and reports what it skipped,
since a collision during a real promotion means two builds have produced one
filename.
autoupdate-publish deleted the live archives before moving the new ones in,
which left a window with nothing there at all; the order is reversed and only
archives this release did not replace are removed, and once every move has
succeeded. --delay-updates leaves .~tmp~ behind in the live trees when a
transfer is interrupted, and both indexers descend into it, so the holding
directories are swept immediately before indexing rather than moved out of the
tree with --partial-dir, which would make the final rename a copy across
filesystems. DATESTAMP_RE and VERSION_RE gain re.ASCII, because \d matches any
Unicode decimal digit and int() parses those, so a fullwidth datestamp on the
filesystem read back as an ordinary date.
In the workflows, the repository variables naming the SSH target and the key
paths reached option positions, as the first positional argument to ssh and
inside the -e string rsync splits on whitespace. Only an administrator can set
those, so this is defence in depth: they are validated against a conservative
pattern in the step that already checks the keys are present, and the target
is passed after --. snapshot.yml passed the raw inputs.location to its publish
step rather than the validated output of the location job, which agreed only
by coincidence, and it now carries a flag for whether a location was named so
that an empty input still means the assembled directory name.
check-container-build.yml gains the top-level permissions: contents: read the
other workflows have, leaving the publish job's packages: write alone, and
promote-release.yml verifies the wheel's detached signature against the
published signing key before uploading it to PyPI, the mirror having been
trusted on TLS alone until now.
The README and the download server's key annotations are updated to match: the
guarantees they stated about single-shot publication and about retention were
overstated, and are now true.
The promotion validated its inputs in the promote job, which the registries
scope skips, so under that scope the typed-twice confirmation did not cover
the step that moves the latest tag; validation moves to a job both paths
depend on. The container tag loops no longer glob against the workspace, and
check-container-build validates the location it interpolates into a push tag
in a job holding the Docker Hub credentials.
The acceptance test built its apt sources line with sh -c, which re-parsed the
expanded base URL as source: the one place in the pipeline where a
caller-supplied value reached a shell. The outer shell writes the file now,
and the URL is checked once, against a host allowlist, before any job
downloads and installs what that host serves as root.
The actions in jobs holding registry or trusted-publishing credentials are
pinned by commit rather than by a movable tag, as the SonarQube action already
was, and so is the helm installer piped into a shell beside them.
Also: the README understated the signing verbs, which select by extension
across the tree rather than by anything tied to a build.
snapshot-purge sorted directory names and kept the last five, so a name that
merely sorted late survived and everything real fell off the end: five
snapshot-create calls with a future datestamp, which the verb accepts up to
two days ahead, were enough to have the next purge delete every genuine
snapshot. Anything dated after today is now left out of the retention set
entirely, and the ordering is by date and then by suffix as a number, which
also fixes 2026-09-22-10 sorting before 2026-09-22-2 and so a day with more
than nine builds keeping the older ones.
packages-fetch pulled staging's Packages files and repodata into the live
tree, whilst the signed Release, Release.gpg and InRelease they are checked
against live a directory further up and were left as they were. Every live
apt repository therefore failed on a hash sum mismatch from the moment the
fetch landed until that codename's rebuild caught up, which with five
sequential rebuilds each taking the lock is minutes. The docstring already
claimed the indices were left behind; now they are.
The pull key is renamed to pga-pull to match the other two.
The machine names were only ever in comments and two filenames, but the
repository is public and the buildfarm never was, so there is no reason to hand
a reader the mapping from internal name to public service. The two
authorized_keys templates become .staging and .download, and the prose refers
to the staging server, the download server and the runner.
The pull key the download server uses is renamed to match, so it needs moving
on that host: ~pgaupload/.ssh/id_ed25519_paxsor_pull -> id_ed25519_pull. The
runner's own key paths come from repository variables, so the defaults here
changing affects nothing.
It was the last SITE value the repository had to get right, and it was a
fully qualified internal hostname in a public tree.
It is read here rather than accepted from the caller, which is the part
that matters: the client names a verb and a datestamp and never a host,
a path, a binary or a signing key, and that is what stops a compromised
runner asking for a signature over something it fetched from a source of
its own choosing. A workflow variable would put it on the wrong side of
that line. It is validated as a hostname on the way in, since it becomes
part of an rsync remote specification.
Every directory under a release carries a CURRENT_MAINTAINER naming the
support address, which the PostgreSQL mirror network expects and which
the buildfarm wrote with an echo in each of nine job definitions. The
rehearsal produced a release directory without them.
It is a file beside the scripts now, so changing the support address is
one reviewed commit.
Two faults the first working rehearsal exposed.
rsync without -r reports 'skipping directory .' and transfers nothing,
so release-fetch created the directory, sorted no files and reported
success. The comment claiming -r was omitted on purpose was simply
wrong; the trees are excluded by pattern instead.
The auto-updater archives were being moved by release-fetch, which
writes them outside v<VERSION>. That contradicted the whole point of the
release-directory scope, which is that deleting one directory undoes it.
They are now autoupdate-publish's job, and it runs with the packages,
where the promotion stops being reversible anyway.
PULL_HOST carried the short name, one of the SITE values never confirmed
against the servers, and the first real release-fetch failed on it. The
firewall now permits the pull and the host key is recorded on the
download server under both names; the fully qualified one is used here so
that what this file names and what known_hosts records cannot drift.
The last piece, replacing pgadmin4-all-publish-qa-to-prod and the deb
and rpm promotion jobs, plus update-download-archive and
pgadmin4-website-create-release.
It builds nothing. It takes a candidate that already exists on the
development server, having been built, signed, indexed and installed
from by the acceptance test, and moves it across: the release directory,
the downloads sorted into it, the packages into the live repository
trees, then the indices, the READMEs, the website and finally the S3
archive. The order is deliberate, since an interruption before the
indices leaves packages nothing points at, which is recoverable, whereas
the reverse is a repository describing packages that are not there.
Two changes to the wrapper it depends on.
release-fetch assumed the staging build was divided into docs, macos,
pip, source and windows. It is not: a staging build is flat, and it is
promotion that sorts the downloads into those directories, which the
buildfarm did with a series of scp commands carrying shell globs. That
sorting is now a table, the destination comes from each file's own name,
and an unrecognised file stops the publication rather than being dropped
silently. It also handles the auto-updater ZIPs, which replace the
previous release's rather than accumulating beside them, and which
nothing here had carried across at all.
rebuild-readme is new, and closes the gap left when the README generator
went in: nothing regenerated the production platform tables, only the
staging and snapshot ones. Those carry the release history, so they
change with every release.
Ten fixes from a review pass with fresh eyes. The ones that mattered:
rebuild-yum-repo.sh signed repomd.xml with no -u, so it used whatever
the keyring's default key resolved to. That is the packaging key today
only because there is one secret key in it. A repository whose metadata
is signed by the wrong key is one every client with gpgcheck=1 rejects,
produced by a run that exits 0.
The snapshot publish gate required only deb and rpm, whilst its comment
justified exactly one exception. A failed docs, tarball or wheel job
published a snapshot missing those files, and a failed repo-rpm job was
worse: install-repo-readme.py then fails after the packages are uploaded
and signed, leaving half a snapshot on the download server.
The location input was interpolated into shell on the runner holding both
publishing keys. It now travels through the environment, and is validated
against the shape of a datestamp before anything downstream sees it, so a
bad one fails in five seconds rather than after an hour of building.
Make.bat tested %ERRORLEVEL% inside a parenthesised block, where it is
expanded at parse time, so a failed Authenticode signature on
pgAdmin4.exe was undetectable; the handler was PAUSE, which on a headless
signing host either returns at once or blocks until the job times out.
Both sites now fail the build.
A called workflow inherits the caller's ref, so once this is on master an
ordinary push would have shared a concurrency group with the nightly
snapshot's builds and cancelled them.
And PULL_KEY named a file that does not exist on the download server,
which would have failed the first release promotion.
The rest: if-no-files-found on every build artefact, so a build that
produces nothing fails rather than silently shipping a snapshot without
it; pinned syft and helm installers, which were piping a default branch
into a shell as root in the container that builds signed packages; the
Helm chart no longer built and discarded; and the comments that had
stopped describing the code.
The self-test grows from 56 cases to 70, covering the verbs that delete
things, arguments that look like options, and the SSH_ORIGINAL_COMMAND
entry point rather than only the dry-run path.
rpmsign refuses to run without %_gpg_name, and the buildfarm satisfied
that with a ~/.rpmmacros on each build agent. Signing has moved to the
download server, which has never signed an RPM and has no such file, so
the first real snapshot-sign failed there.
Passing it explicitly suits this better than adding the dotfile: the key
is already a constant in this file, for the same reason the client cannot
name one, and the result no longer depends on whose home directory the
wrapper runs in.
Verifying against paxsor and procyon before the first run turned up that
the flow had the wrong shape, which would have produced a snapshot
nobody could install from.
A staging build and a snapshot are flat: apt/, yum/, and the
downloadable files loose beside them. It is only a published release
that is divided into docs/, macos/, pip/, source/ and windows/, and the
promotion job sorts files into those on the way to v<VERSION>. We had
the release layout applied to all three, so stage-create and
snapshot-create made directories nothing uses, the assembled tree put
every file one level too deep, and sign_packages looked for things to
sign in directories that would have been empty. A snapshot would have
gone out unsigned.
Signing is now driven by extension and location rather than by a list of
directories, so it is right for both layouts: RPMs under yum/ signed in
place, and detached signatures for the tarball, wheel, documentation
tarball, PDF and ePub wherever they sit. Debs are left alone, apt
verifying them through the signed Release file, as are the disk images,
ZIPs and installer, which carry signatures of their own.
The documentation also gets versioned filenames, the build leaving them
as pgadmin4.pdf and pgAdmin4.epub whilst the download site has always
carried pgadmin4-<version>.pdf and .epub.
I had the two flows conflated. Release candidates are staged on the
development server and promoted; snapshots are published straight to the
download server under snapshots/<date>, which is what every
*-snapshot job in the buildfarm targets.
So the publish job talks to the download server, the wrapper grows the
snapshot verbs it needed and loses snapshot-fetch, which existed only to
copy a staging build that is never made, and the variable is named for
the server it actually names.
It also reverses my advice about the upload key on the download server,
which I had said to drop on the reasoning that publication always pulls
from procyon. It does not, for snapshots. The key comes back, confined
to the snapshots tree, which is the right root on two counts: it is
where snapshots belong, and it is the one part of the download site that
the S3 sync excludes, so nothing written through it can reach the
archive or the mirrors.
snapshot-purge replaces the buildfarm's ls | tail | xargs rm. It is the
only verb that deletes anything, so it takes no argument: it cannot be
asked to remove a particular snapshot, only to enforce the retention the
server already has, and it ignores directories that do not look like a
snapshot.
They are already on roadie. Putting copies in the secret store would
mean two places to rotate them and two places to leak them from, so the
workflow names the paths and never handles key material.
What changes with it is where the boundary sits. An environment secret
is gated by a deployment branch policy; a file on disk is gated by which
workflows may run on that machine. This repository is public, so roadie
needs to be in a runner group restricted to these workflows, with
nothing triggered by pull_request targeting it. That is the same bargain
the Windows signing host already makes with the Certum token, but it is
now load-bearing in a second place, so it is written down.
The environment stays for the branch policy and the deployment record.
The last missing link: assemble produced a tree and nothing carried it
anywhere. The publish job uploads it over the rrsync-confined key and
then asks pga-publish to sign and index it, so the runner names a verb
and a datestamp rather than a command, and the signing happens where the
key is.
The staging directory can be named explicitly, because the buildfarm
writes <date> on that same server and a test run has no business landing
on the directory Jenkins is still filling. Publication is off by default
for the same reason: it takes an explicit input, so a dispatch that only
wants to know whether everything builds cannot reach the servers at all.
Also adds the deployment notes, which is where the four files go on each
host, what the two configuration files hold, and why no key gets a
shell.
It was drafted from the Jenkins job definitions before anybody had
looked at the servers, and most of what it assumed about them was wrong.
The apt layout is per distribution release, with the packages sitting
directly in binary-<arch> and no shared pool, so packages-fetch and
stage-create were writing to paths that do not exist. The allowlists
were stale in both directions: plucky and RHEL 8 are not built, whilst
resolute, Fedora 43 and 44 were missing. sync-ftp-to-s3.py takes the
CloudFront distribution to invalidate rather than a choice of what to
sync, so that verb loses its argument entirely and reads the identifier
from the server, since the client has nothing to choose.
The indexing is no longer implemented here. It calls the scripts in this
directory instead, which are the same ones an administrator runs by hand
after purging old releases, so there is one implementation rather than
the four the buildfarm had. That also settles the doubled signing the
draft worried about: the scripts sign as part of indexing.
New verb: stage-sign. With the GPG key staying on the servers rather
than moving into Actions secrets, the workflow uploads unsigned
artefacts and asks for them to be signed here. A compromised workflow
can therefore ask for a signature over something it just uploaded, which
is real and acknowledged; what it cannot do is hold the key, sign out of
band, or sign with any other key.
The self-test covers 56 cases and passes.
The support matrix was maintained by hand in five Jenkins heredocs,
which is exactly why the published one had drifted: it stops at Noble
and still lists Mantic, whilst the job that writes it goes up to
Resolute.
It is now derived, and the two sources are deliberately different. Which
platforms are supported comes from the repositories present in the tree
being written, since that is what a user can actually install from.
Which releases each platform carried comes from the S3 archive rather
than the download server, because old releases are purged from the
server as they age: bookworm reads as 7.8 there against 7.0 in the
archive, so deriving history from the server would quietly rewrite it
every time somebody tidied up. The sync has no --delete, so the bucket
keeps everything.
Platforms that are no longer supported get their own table with the last
release each carried, which turns the standing note about the archive
into something a user on an old distribution can act on. For snapshot
and pre-release trees --no-archive drops that section and the release
columns with it, since every package there is from the one build.
Cross-checked against the hand-written tables: every value they assert
is reproduced, including bookworm 7.0, jammy 6.9, bullseye 4.30 to 9.14
and fedora-42 9.3 to 9.17. Two they got wrong come out right, RHEL 10
having been copied from the RHEL 9 row as 6.12 when it is 9.6.
Two traps worth naming. aws s3 sync follows symlinks, so the EL
compatibility links exist in the bucket as full copies and would
otherwise invent platforms like fedora-31Client. And Debian versions
carry a packaging revision now, as in 9.16-1.resolute, which sorts below
9.3 unless it is stripped first.
It lived unversioned in pgaupload's home directory on paxsor, which also
made it writable by the account the publishing scripts run as, despite
being an input to what gets indexed and therefore to what gets signed.
Taking it here settles both: changing the repository's Origin, Label or
architectures becomes a reviewed commit, and the copy on the server can
become root owned.
The default architecture list drops i386 to match. apt-ftparchive has
been writing dists/pgadmin4/main/binary-i386/Packages for years, but
Architectures in the Release file has only ever said 'amd64 all', so no
client has been able to reach that index through the signed metadata.
Nothing that works today stops working.
The sequence that indexes and signs an APT repository existed in four
places that differed only in which directory they ran against: inline in
the pgadmin4-deb-build, pgadmin4-deb-snapshot and
pgadmin4-deb-publish-qa-to-prod Jenkins jobs, and in rebuild-apt-repo.sh
in the pgaweb repository. The YUM equivalent existed in five, and the
repository READMEs were heredocs in five more jobs.
These are those sequences, once each, with the tree root as an argument
so that the staging, snapshot and production trees share one
implementation, and with the architecture parameterised rather than
x86_64 spliced into the paths, since we will want to publish for arm64
before long. The APT script passes the architecture list to
apt-ftparchive as an override rather than letting it come from
aptftp.conf, so the set indexed and the set advertised in the signed
Release file cannot drift apart; a package whose architecture is missing
from Release is ignored by every client, silently.
Run with no options they behave exactly as the pgaweb copies did, so
rebuilding a repository by hand after purging old releases is unchanged.
aptftp.conf still needs to be brought across from the download server.
