feat(ci): rebuild supported releases when package updates are available (#4839)

* ci: rebuild supported releases when package updates are available

Check the published image of each supported release every hour and only
rebuild when `apt-get upgrade` would change something in it, instead of
rebuilding blindly once a week. Scheduled runs skip an image younger than
3 hours. Manual runs still rebuild unconditionally unless
`only-if-updates` is set.

* docs: describe rebuilds on package updates instead of weekly rebuilds

* ci: ignore packages that never clear when checking for updates

dovecot-fts-xapian is built from source as 1.9.1-1~bpo12+1 in v16, which
apt always sorts below the Debian package, so it would trigger a rebuild
on every cooldown. List the pending packages in the log.

* ci: schedule the check at minute 17 instead of the start of the hour

* docs: link the changelog entry to the PR

* docs: fix the changelog link text

* ci: do not trigger release rebuilds on Rspamd updates

v16 installs Rspamd from the rspamd.com apt repository, so every upstream
release showed up as a pending update and would be published to the floating
tags. Rebuilds should only pick up Debian updates. From v17 on, Rspamd is a
pinned `.deb` without an apt repository.

* ci: attempt a scheduled release rebuild only once per set of pending updates

The cooldown is based on the age of the published image, but a failed rebuild
publishes nothing. A rebuild that keeps failing was therefore started again on
every hourly run, and an update that never clears every 3 hours.

Each attempt is now recorded as an artifact named after the release and a hash
of the pending updates (`name=version`, ignored packages left out). The
`resolve` job skips a release whose pending updates were already attempted. The
record is written by a new `record-attempt` job that runs in parallel to the
rebuild, so it exists whatever the outcome is. It expires after 7 days, so the
same updates are retried at most once a week. New updates change the name and
are attempted right away.

Also pass `--error-on=any` to `apt-get update` in the check: without it, apt
exits 0 when index downloads fail, which made an outage look like "nothing
pending" instead of failing the job.

* ci: check for package updates daily instead of hourly

The check runs at 06:17 UTC. A fix reaches the floating tags within about a day
plus the build time instead of within about an hour, which is sufficient for
Debian security updates and needs far fewer scheduled runs.

* ci: remove the cooldown for scheduled release rebuilds

The check runs once a day, which already limits how often the floating tags
move. A rebuild loop is prevented by the recorded attempts, so the age of the
published image no longer needs to be checked.

---------

Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
This commit is contained in:
Alan SavageandGeorg Lauterbach authored and GitHub committed 2026-11-06 00:29:10 +01:00
1 parent 626d893992
commit 172848ccda
5 files changed
+144 -16

No files matched your search

+132 -5
View File
@@ -1,4 +1,4 @@
name: 'Rebuild Supported Releases on Schedule'
name: 'Rebuild Supported Releases on Package Updates'
# Debian security fixes land continuously, but a release image is only built once.
# This rebuilds the newest release of every supported major version from its tag, ignoring
@@ -8,6 +8,11 @@ name: 'Rebuild Supported Releases on Schedule'
# release appeared in the meantime.
#
# Only the newest release of a major is rebuilt, as older ones must not move the floating tags.
#
# Scheduled runs (daily) first check the published image: a major is only rebuilt if
# `apt-get upgrade` would change something in it. There is no way to be notified of new Debian
# packages, so this is as close to "as soon as an update is available" as it gets.
# Manual runs always rebuild, unless `only-if-updates` is set.
on:
workflow_dispatch:
@@ -16,15 +21,24 @@ on:
description: 'Rebuild only this supported major version (default: all)'
required: false
type: string
only-if-updates:
description: 'Only rebuild majors whose published image has pending package updates that were not attempted yet (always the case for scheduled runs)'
required: false
type: boolean
default: false
dry-run:
description: 'Build and test, but do not push any image'
required: false
type: boolean
default: false
schedule:
- cron: 0 6 * * 5
# Not at minute 0: GitHub delays or drops scheduled runs
# at the start of the hour, when load is highest.
- cron: 35 4 * * *
permissions:
# Required to look up the artifacts that record rebuild attempts:
actions: read
contents: read
packages: write
@@ -33,8 +47,15 @@ env:
# Drop a major once its Debian release is end of life. Each line is gated on the tests of its own tag.
# `15` (Debian 12) is not listed, as its tests need `bitnami/openldap:latest`, which no longer exists.
SUPPORTED_MAJORS: '16'
# Packages that must not trigger a rebuild. Space separated.
# - `dovecot-fts-xapian` stays "upgradable" even after a rebuild: v16 builds it from source as
# `1.9.1-1~bpo12+1`, which apt sorts below Debian's `1.9.1-1.1`. Not needed for v17+, where Xapian was removed.
# - `rspamd`: v16 installs it from the rspamd.com apt repository, so every upstream release would show up as
# pending. Rebuilds should only pick up Debian updates. From v17 on, Rspamd is a pinned `.deb` (no apt repository).
IGNORED_PENDING_PACKAGES: 'dovecot-fts-xapian rspamd'
# Two overlapping runs would probe the same rebuild number and the second push would overwrite the first.
# GitHub keeps only one pending run per group, so a pending manual run can be replaced by a later scheduled one.
concurrency:
group: release-rebuild
cancel-in-progress: false
@@ -45,6 +66,7 @@ jobs:
runs-on: ubuntu-24.04
outputs:
matrix: ${{ steps.resolve.outputs.matrix }}
rebuild-needed: ${{ steps.resolve.outputs.rebuild-needed }}
steps:
# Required to see private packages (eg on forks) when probing for existing tags:
- name: 'Login to GitHub Container Registry'
@@ -54,15 +76,70 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
# Builds the matrix of `{tag, major, rebuild}`, one entry per supported major:
# Builds the matrix of `{tag, major, rebuild, attempt, pending}`, one entry per supported major that needs a rebuild:
# - `tag` is the newest non-prerelease release of that major.
# - `rebuild` is the first unused `X.Y.Z-rN` (N >= 1), found by probing the registry (no stored state).
# Only a "not found" answer means a tag is free. Any other error (network, rate limit, outage)
# fails the job, as continuing could overwrite an existing, supposedly immutable tag.
# - `attempt` is the name of the artifact that `record-attempt` creates, `pending` its content (the pending
# updates as `name=version`). Both are empty if the rebuild was not conditional on pending updates.
- name: 'Resolve releases and rebuild numbers'
id: resolve
shell: bash
run: |
# Whether the published image of a release has package updates pending that were not attempted
# yet (returns 0 if so). The check uses the amd64 image and what the image's own `apt-get upgrade` would do.
# A rebuild that fails publishes nothing, and an update may never clear. Either would be retried on every
# run, so each attempt is recorded (see `record-attempt`) under a name made of the release and the pending
# updates with their versions. The same updates are not attempted again until the record expires.
# On success, sets `ATTEMPT_KEY` (that name) and `ATTEMPT_PENDING` (the pending updates) for the caller.
function needs_rebuild() {
local TAG=$1 IMAGE_REF="${IMAGE}:${1#v}" OUTPUT ENTRY PACKAGES KEY ATTEMPTS PENDING=()
if ! OUTPUT=$(docker pull --quiet --platform linux/amd64 "${IMAGE_REF}" 2>&1); then
if grep -qiE 'not found|manifest unknown' <<< "${OUTPUT}"; then
echo "${IMAGE_REF}: not published yet (release publish in progress?), skipping"
return 1
fi
echo "Unexpected error while pulling ${IMAGE_REF}:" >&2
echo "${OUTPUT}" >&2
exit 1
fi
# A failing `apt-get update` (or `docker run`) must not look like "nothing pending", hence `pipefail`.
# Without `--error-on=any`, `apt-get update` exits 0 when index downloads fail.
# The lines to parse look like `Inst <name> [<installed>] (<candidate> <origin> [<arch>])`:
PACKAGES=$(docker run --rm --platform linux/amd64 --entrypoint bash "${IMAGE_REF}" \
-c 'apt-get -qq --error-on=any update >/dev/null && apt-get -s upgrade' \
| awk '/^Inst/ { for (i = 3; i <= NF; i++) if ($i ~ /^\(/) { print $2 "=" substr($i, 2); next } }') || exit 1
for ENTRY in ${PACKAGES}; do
if [[ " ${IGNORED_PENDING_PACKAGES} " != *" ${ENTRY%%=*} "* ]]; then
PENDING+=("${ENTRY}")
fi
done
if (( ${#PENDING[@]} == 0 )); then
echo "${IMAGE_REF}: up to date"
return 1
fi
KEY="release-rebuild-attempt-${TAG}-$(printf '%s\n' "${PENDING[@]}" | LC_ALL=C sort | sha256sum | cut -c1-16)" || exit 1
ATTEMPTS=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${KEY}" \
--jq '[.artifacts[] | select(.expired | not)] | length') || exit 1
if (( ATTEMPTS > 0 )); then
echo "${IMAGE_REF}: ${#PENDING[@]} pending package updates, but a rebuild for them was already attempted (${KEY}), skipping"
echo "To retry, dispatch the workflow manually without 'only-if-updates' or re-run the failed jobs of that run"
return 1
fi
echo "${IMAGE_REF}: ${#PENDING[@]} pending package updates, rebuilding"
printf ' %s\n' "${PENDING[@]}"
ATTEMPT_KEY=${KEY}
ATTEMPT_PENDING="${PENDING[*]}"
return 0
}
function next_rebuild() {
local VERSION=${1#v} N=1 OUTPUT
while true; do
@@ -97,6 +174,11 @@ jobs:
RELEASES=$(gh release list --repo "${GITHUB_REPOSITORY}" --limit 200 --exclude-drafts --exclude-pre-releases --json tagName --jq '.[].tagName' \
| grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V)
CHECK_UPDATES=false
if [[ ${EVENT_NAME} == schedule || ${INPUT_ONLY_IF_UPDATES} == true ]]; then
CHECK_UPDATES=true
fi
ENTRIES=()
for MAJOR in ${MAJORS}; do
TAG=$(grep -E "^v${MAJOR}\." <<< "${RELEASES}" | tail -1 || true)
@@ -105,21 +187,66 @@ jobs:
exit 1
fi
# The floating `X.Y.Z` tag, which `needs_rebuild` checks, is the newest build of the release:
ATTEMPT_KEY='' ATTEMPT_PENDING=''
if [[ ${CHECK_UPDATES} == true ]] && ! needs_rebuild "${TAG}"; then
continue
fi
REBUILD=$(next_rebuild "${TAG}")
ENTRIES+=("$(jq -cn --arg tag "${TAG}" --arg major "${MAJOR}" --arg rebuild "${REBUILD}" \
'{tag: $tag, major: $major, rebuild: $rebuild}')")
--arg attempt "${ATTEMPT_KEY}" --arg pending "${ATTEMPT_PENDING}" \
'{tag: $tag, major: $major, rebuild: $rebuild, attempt: $attempt, pending: $pending}')")
done
echo "matrix=$(printf '%s\n' "${ENTRIES[@]}" | jq -cs '{include: .}')" >>"${GITHUB_OUTPUT}"
# An empty matrix is an error in Actions, so the `rebuild` job is skipped instead:
if (( ${#ENTRIES[@]} == 0 )); then
echo 'No rebuild needed'
echo 'matrix={"include":[]}' >>"${GITHUB_OUTPUT}"
echo 'rebuild-needed=false' >>"${GITHUB_OUTPUT}"
else
echo "matrix=$(printf '%s\n' "${ENTRIES[@]}" | jq -cs '{include: .}')" >>"${GITHUB_OUTPUT}"
echo 'rebuild-needed=true' >>"${GITHUB_OUTPUT}"
fi
env:
GH_TOKEN: ${{ github.token }}
IMAGE: ${{ secrets.GHCR_REPOSITORY }}
INPUT_MAJOR: ${{ inputs.major }}
INPUT_ONLY_IF_UPDATES: ${{ inputs.only-if-updates }}
EVENT_NAME: ${{ github.event_name }}
# A rebuild that fails publishes nothing, and an update may never clear. Without a record of the attempt, every
# scheduled run would start another full build. The artifact is looked up by `resolve` (see `needs_rebuild`).
# It runs in parallel to `rebuild` on purpose, so the attempt is recorded whatever the outcome is.
# The record expires after `retention-days`, which is when the same updates are attempted again.
record-attempt:
name: 'Record Attempt for ${{ matrix.tag }}'
needs: resolve
if: needs.resolve.outputs.rebuild-needed == 'true'
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.resolve.outputs.matrix) }}
steps:
- name: 'Write the pending updates'
if: matrix.attempt != ''
run: tr ' ' '\n' <<< "${PENDING}" >attempt.txt
env:
PENDING: ${{ matrix.pending }}
- name: 'Upload the record'
if: matrix.attempt != ''
uses: actions/upload-artifact@v7
with:
name: ${{ matrix.attempt }}
path: attempt.txt
retention-days: 7
rebuild:
name: 'Rebuild ${{ matrix.tag }}'
needs: resolve
if: needs.resolve.outputs.rebuild-needed == 'true'
strategy:
# A failing line must not stop the others:
fail-fast: false
+1 -1
View File
@@ -28,7 +28,7 @@ All notable changes to this project will be documented in this file. The format
### CI
- A weekly workflow rebuilds the newest release of each supported major version and republishes its `X.Y.Z`, `X.Y` and `X` tags (and `latest` for the newest release), plus an immutable `X.Y.Z-rN` tag for rollbacks ([#4833](https://github.com/docker-mailserver/docker-mailserver/pull/4833))
- A daily workflow rebuilds the newest release of each supported major version when its published image has package updates pending, and republishes its `X.Y.Z`, `X.Y` and `X` tags (and `latest` for the newest release), plus an immutable `X.Y.Z-rN` tag for rollbacks ([#4833](https://github.com/docker-mailserver/docker-mailserver/pull/4833), [#4839](https://github.com/docker-mailserver/docker-mailserver/pull/4839))
- Moved `setup_cli.bats` to the serial test suite to prevent `setup.sh` from racing with teardown of another DMS test container ([#4701](https://github.com/docker-mailserver/docker-mailserver/issues/4701))
- Excluded `saslauthd` from the process kill/restart assertion because its forked parent process is not reliably reaped by the test
+6 -6
View File
@@ -2,13 +2,13 @@
## Supported Versions
Due to the way DMS is released, the most recent patches and the most current software is published on the `:edge` tag of the container image. Hence, security updates will land on this "rolling release tag". The newest release of each supported major version (currently `v16`, while its Debian base still receives security updates) is rebuilt weekly to include the current Debian security updates. Older tags need manual updating, as we do not usually release an updated image for an existing tag; this will only be done in case of _severe_ vulnerabilities.
Due to the way DMS is released, the most recent patches and the most current software is published on the `:edge` tag of the container image. Hence, security updates will land on this "rolling release tag". The newest release of each supported major version (currently `v16`, while its Debian base still receives security updates) is rebuilt as soon as package updates are available (checked daily), to include the current Debian security updates. Older tags need manual updating, as we do not usually release an updated image for an existing tag; this will only be done in case of _severe_ vulnerabilities.
| Image Tags | Latest Packages & Patches |
|---------------------------------------|:-----------------------------------:|
| `:edge` | :white_check_mark: |
| newest release of a supported major | :white_check_mark: (weekly rebuild) |
| other releases | :x: |
| Image Tags | Latest Packages & Patches |
|-------------------------------------|:--------------------------------------------:|
| `:edge` | :white_check_mark: |
| newest release of a supported major | :white_check_mark: (rebuilt on updates) |
| other releases | :x: |
## Reporting a Vulnerability
@@ -36,7 +36,7 @@ title: 'Maintenance | Update and Cleanup'
The tag for an image is never modified by `watchtower`, instead `watchtower` monitors the image digest associated to that image tag (_which will change to a new image digest if a new image release reassigns the tag_), when the digest for the tag changes this triggers a pull of the new image.
- Your container will not update to a new major release version (_unless using `:latest`_).
- The tags of the newest release of each supported major (eg `:16.0.1`, `:16.0`, `:16`, and `:latest` for the newest overall) are rebuilt weekly with the current Debian security fixes. Use the immutable `:16.0.1-rN` tag if you need to pin an exact build.
- The tags of the newest release of each supported major (eg `:16.0.1`, `:16.0`, `:16`, and `:latest` for the newest overall) are rebuilt with the current package updates (_Debian security fixes included_) when any are available. This is checked every day. Use the immutable `:16.0.1-rN` tag if you need to pin an exact build.
- Omit the minor or patch portion of a semver tag to receive updates for the omitted portion (_eg: `:16` will represent the latest minor + patch release, whereas `:16.0` would only receive patch updates instead of minor releases like `16.1`_).
!!! tip "Updating only specific containers"
+4 -3
View File
@@ -89,14 +89,15 @@ All workflows are using the tagging convention listed below. It is subsequently
|-------------------------|-------------------------------------------|
| `push` on `master` | `edge` |
| `push` a tag (`v1.2.3`) | `1.2.3`, `1.2`, `1`, `latest`, `1.2.3-r0` |
| Weekly rebuild | `1.2.3`, `1.2`, `1`, `latest`, `1.2.3-rN` |
| Package updates | `1.2.3`, `1.2`, `1`, `latest`, `1.2.3-rN` |
!!! info "Weekly rebuilds of supported releases"
!!! info "Rebuilds of supported releases on package updates"
Debian publishes security fixes continuously, but a release image is only built once. To pick those fixes up, the newest release of every supported major version (_currently `v16`_) is rebuilt every week without cached layers, tested, and published again:
Debian publishes security fixes continuously, but a release image is only built once. To pick those fixes up, the newest release of every supported major version (_currently `v16`_) is checked every day. If its published image has package updates pending (_from Debian_), it is rebuilt without cached layers, tested, and published again:
- The tags `X.Y.Z`, `X.Y` and `X` of the rebuilt release are moved to the rebuilt image. `latest` is only moved when the rebuilt release is the newest overall. Older minor and patch releases within a major are not rebuilt and keep their last image.
- Each build also receives an immutable `X.Y.Z-rN` tag (`-r0` is the original release build, rebuilds count up from `-r1`). Use it to pin an exact build or to roll back, or pin the image digest.
- A given set of pending updates is only attempted once, even if the rebuild fails or the updates stay pending afterwards. The next attempt happens when the set of pending updates changes, or after 7 days. To retry earlier, re-run the failed jobs of the workflow run or dispatch the workflow manually.
[github-ci]: https://github.com/docker-mailserver/docker-mailserver/actions
[dockerhub-image]: https://hub.docker.com/r/mailserver/docker-mailserver