From 172848ccdafd445034b463f5f59bf151fcdfc97d Mon Sep 17 00:00:00 2001 From: Alan Savage Date: Fri, 6 Nov 2026 00:29:10 +0100 Subject: [PATCH] feat(ci): rebuild supported releases when package updates are available (#4839) * ci: rebuild supported releases when package updates are available Check the published image of each supported release every hour and only rebuild when `apt-get upgrade` would change something in it, instead of rebuilding blindly once a week. Scheduled runs skip an image younger than 3 hours. Manual runs still rebuild unconditionally unless `only-if-updates` is set. * docs: describe rebuilds on package updates instead of weekly rebuilds * ci: ignore packages that never clear when checking for updates dovecot-fts-xapian is built from source as 1.9.1-1~bpo12+1 in v16, which apt always sorts below the Debian package, so it would trigger a rebuild on every cooldown. List the pending packages in the log. * ci: schedule the check at minute 17 instead of the start of the hour * docs: link the changelog entry to the PR * docs: fix the changelog link text * ci: do not trigger release rebuilds on Rspamd updates v16 installs Rspamd from the rspamd.com apt repository, so every upstream release showed up as a pending update and would be published to the floating tags. Rebuilds should only pick up Debian updates. From v17 on, Rspamd is a pinned `.deb` without an apt repository. * ci: attempt a scheduled release rebuild only once per set of pending updates The cooldown is based on the age of the published image, but a failed rebuild publishes nothing. A rebuild that keeps failing was therefore started again on every hourly run, and an update that never clears every 3 hours. Each attempt is now recorded as an artifact named after the release and a hash of the pending updates (`name=version`, ignored packages left out). The `resolve` job skips a release whose pending updates were already attempted. The record is written by a new `record-attempt` job that runs in parallel to the rebuild, so it exists whatever the outcome is. It expires after 7 days, so the same updates are retried at most once a week. New updates change the name and are attempted right away. Also pass `--error-on=any` to `apt-get update` in the check: without it, apt exits 0 when index downloads fail, which made an outage look like "nothing pending" instead of failing the job. * ci: check for package updates daily instead of hourly The check runs at 06:17 UTC. A fix reaches the floating tags within about a day plus the build time instead of within about an hour, which is sufficient for Debian security updates and needs far fewer scheduled runs. * ci: remove the cooldown for scheduled release rebuilds The check runs once a day, which already limits how often the floating tags move. A rebuild loop is prevented by the recorded attempts, so the age of the published image no longer needs to be checked. --------- Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --- .../workflows/scheduled_release_rebuild.yml | 137 +++++++++++++++++- CHANGELOG.md | 2 +- SECURITY.md | 12 +- .../maintenance/update-and-cleanup.md | 2 +- docs/content/usage.md | 7 +- 5 files changed, 144 insertions(+), 16 deletions(-) diff --git a/.github/workflows/scheduled_release_rebuild.yml b/.github/workflows/scheduled_release_rebuild.yml index 4bafe4f9..852d10e4 100644 --- a/.github/workflows/scheduled_release_rebuild.yml +++ b/.github/workflows/scheduled_release_rebuild.yml @@ -1,4 +1,4 @@ -name: 'Rebuild Supported Releases on Schedule' +name: 'Rebuild Supported Releases on Package Updates' # Debian security fixes land continuously, but a release image is only built once. # This rebuilds the newest release of every supported major version from its tag, ignoring @@ -8,6 +8,11 @@ name: 'Rebuild Supported Releases on Schedule' # release appeared in the meantime. # # Only the newest release of a major is rebuilt, as older ones must not move the floating tags. +# +# Scheduled runs (daily) first check the published image: a major is only rebuilt if +# `apt-get upgrade` would change something in it. There is no way to be notified of new Debian +# packages, so this is as close to "as soon as an update is available" as it gets. +# Manual runs always rebuild, unless `only-if-updates` is set. on: workflow_dispatch: @@ -16,15 +21,24 @@ on: description: 'Rebuild only this supported major version (default: all)' required: false type: string + only-if-updates: + description: 'Only rebuild majors whose published image has pending package updates that were not attempted yet (always the case for scheduled runs)' + required: false + type: boolean + default: false dry-run: description: 'Build and test, but do not push any image' required: false type: boolean default: false schedule: - - cron: 0 6 * * 5 + # Not at minute 0: GitHub delays or drops scheduled runs + # at the start of the hour, when load is highest. + - cron: 35 4 * * * permissions: + # Required to look up the artifacts that record rebuild attempts: + actions: read contents: read packages: write @@ -33,8 +47,15 @@ env: # Drop a major once its Debian release is end of life. Each line is gated on the tests of its own tag. # `15` (Debian 12) is not listed, as its tests need `bitnami/openldap:latest`, which no longer exists. SUPPORTED_MAJORS: '16' + # Packages that must not trigger a rebuild. Space separated. + # - `dovecot-fts-xapian` stays "upgradable" even after a rebuild: v16 builds it from source as + # `1.9.1-1~bpo12+1`, which apt sorts below Debian's `1.9.1-1.1`. Not needed for v17+, where Xapian was removed. + # - `rspamd`: v16 installs it from the rspamd.com apt repository, so every upstream release would show up as + # pending. Rebuilds should only pick up Debian updates. From v17 on, Rspamd is a pinned `.deb` (no apt repository). + IGNORED_PENDING_PACKAGES: 'dovecot-fts-xapian rspamd' # Two overlapping runs would probe the same rebuild number and the second push would overwrite the first. +# GitHub keeps only one pending run per group, so a pending manual run can be replaced by a later scheduled one. concurrency: group: release-rebuild cancel-in-progress: false @@ -45,6 +66,7 @@ jobs: runs-on: ubuntu-24.04 outputs: matrix: ${{ steps.resolve.outputs.matrix }} + rebuild-needed: ${{ steps.resolve.outputs.rebuild-needed }} steps: # Required to see private packages (eg on forks) when probing for existing tags: - name: 'Login to GitHub Container Registry' @@ -54,15 +76,70 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - # Builds the matrix of `{tag, major, rebuild}`, one entry per supported major: + # Builds the matrix of `{tag, major, rebuild, attempt, pending}`, one entry per supported major that needs a rebuild: # - `tag` is the newest non-prerelease release of that major. # - `rebuild` is the first unused `X.Y.Z-rN` (N >= 1), found by probing the registry (no stored state). # Only a "not found" answer means a tag is free. Any other error (network, rate limit, outage) # fails the job, as continuing could overwrite an existing, supposedly immutable tag. + # - `attempt` is the name of the artifact that `record-attempt` creates, `pending` its content (the pending + # updates as `name=version`). Both are empty if the rebuild was not conditional on pending updates. - name: 'Resolve releases and rebuild numbers' id: resolve shell: bash run: | + # Whether the published image of a release has package updates pending that were not attempted + # yet (returns 0 if so). The check uses the amd64 image and what the image's own `apt-get upgrade` would do. + # A rebuild that fails publishes nothing, and an update may never clear. Either would be retried on every + # run, so each attempt is recorded (see `record-attempt`) under a name made of the release and the pending + # updates with their versions. The same updates are not attempted again until the record expires. + # On success, sets `ATTEMPT_KEY` (that name) and `ATTEMPT_PENDING` (the pending updates) for the caller. + function needs_rebuild() { + local TAG=$1 IMAGE_REF="${IMAGE}:${1#v}" OUTPUT ENTRY PACKAGES KEY ATTEMPTS PENDING=() + + if ! OUTPUT=$(docker pull --quiet --platform linux/amd64 "${IMAGE_REF}" 2>&1); then + if grep -qiE 'not found|manifest unknown' <<< "${OUTPUT}"; then + echo "${IMAGE_REF}: not published yet (release publish in progress?), skipping" + return 1 + fi + echo "Unexpected error while pulling ${IMAGE_REF}:" >&2 + echo "${OUTPUT}" >&2 + exit 1 + fi + + # A failing `apt-get update` (or `docker run`) must not look like "nothing pending", hence `pipefail`. + # Without `--error-on=any`, `apt-get update` exits 0 when index downloads fail. + # The lines to parse look like `Inst [] ( [])`: + PACKAGES=$(docker run --rm --platform linux/amd64 --entrypoint bash "${IMAGE_REF}" \ + -c 'apt-get -qq --error-on=any update >/dev/null && apt-get -s upgrade' \ + | awk '/^Inst/ { for (i = 3; i <= NF; i++) if ($i ~ /^\(/) { print $2 "=" substr($i, 2); next } }') || exit 1 + + for ENTRY in ${PACKAGES}; do + if [[ " ${IGNORED_PENDING_PACKAGES} " != *" ${ENTRY%%=*} "* ]]; then + PENDING+=("${ENTRY}") + fi + done + + if (( ${#PENDING[@]} == 0 )); then + echo "${IMAGE_REF}: up to date" + return 1 + fi + + KEY="release-rebuild-attempt-${TAG}-$(printf '%s\n' "${PENDING[@]}" | LC_ALL=C sort | sha256sum | cut -c1-16)" || exit 1 + ATTEMPTS=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${KEY}" \ + --jq '[.artifacts[] | select(.expired | not)] | length') || exit 1 + if (( ATTEMPTS > 0 )); then + echo "${IMAGE_REF}: ${#PENDING[@]} pending package updates, but a rebuild for them was already attempted (${KEY}), skipping" + echo "To retry, dispatch the workflow manually without 'only-if-updates' or re-run the failed jobs of that run" + return 1 + fi + + echo "${IMAGE_REF}: ${#PENDING[@]} pending package updates, rebuilding" + printf ' %s\n' "${PENDING[@]}" + ATTEMPT_KEY=${KEY} + ATTEMPT_PENDING="${PENDING[*]}" + return 0 + } + function next_rebuild() { local VERSION=${1#v} N=1 OUTPUT while true; do @@ -97,6 +174,11 @@ jobs: RELEASES=$(gh release list --repo "${GITHUB_REPOSITORY}" --limit 200 --exclude-drafts --exclude-pre-releases --json tagName --jq '.[].tagName' \ | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V) + CHECK_UPDATES=false + if [[ ${EVENT_NAME} == schedule || ${INPUT_ONLY_IF_UPDATES} == true ]]; then + CHECK_UPDATES=true + fi + ENTRIES=() for MAJOR in ${MAJORS}; do TAG=$(grep -E "^v${MAJOR}\." <<< "${RELEASES}" | tail -1 || true) @@ -105,21 +187,66 @@ jobs: exit 1 fi + # The floating `X.Y.Z` tag, which `needs_rebuild` checks, is the newest build of the release: + ATTEMPT_KEY='' ATTEMPT_PENDING='' + if [[ ${CHECK_UPDATES} == true ]] && ! needs_rebuild "${TAG}"; then + continue + fi + REBUILD=$(next_rebuild "${TAG}") ENTRIES+=("$(jq -cn --arg tag "${TAG}" --arg major "${MAJOR}" --arg rebuild "${REBUILD}" \ - '{tag: $tag, major: $major, rebuild: $rebuild}')") + --arg attempt "${ATTEMPT_KEY}" --arg pending "${ATTEMPT_PENDING}" \ + '{tag: $tag, major: $major, rebuild: $rebuild, attempt: $attempt, pending: $pending}')") done - echo "matrix=$(printf '%s\n' "${ENTRIES[@]}" | jq -cs '{include: .}')" >>"${GITHUB_OUTPUT}" + # An empty matrix is an error in Actions, so the `rebuild` job is skipped instead: + if (( ${#ENTRIES[@]} == 0 )); then + echo 'No rebuild needed' + echo 'matrix={"include":[]}' >>"${GITHUB_OUTPUT}" + echo 'rebuild-needed=false' >>"${GITHUB_OUTPUT}" + else + echo "matrix=$(printf '%s\n' "${ENTRIES[@]}" | jq -cs '{include: .}')" >>"${GITHUB_OUTPUT}" + echo 'rebuild-needed=true' >>"${GITHUB_OUTPUT}" + fi env: GH_TOKEN: ${{ github.token }} IMAGE: ${{ secrets.GHCR_REPOSITORY }} INPUT_MAJOR: ${{ inputs.major }} + INPUT_ONLY_IF_UPDATES: ${{ inputs.only-if-updates }} + EVENT_NAME: ${{ github.event_name }} + + # A rebuild that fails publishes nothing, and an update may never clear. Without a record of the attempt, every + # scheduled run would start another full build. The artifact is looked up by `resolve` (see `needs_rebuild`). + # It runs in parallel to `rebuild` on purpose, so the attempt is recorded whatever the outcome is. + # The record expires after `retention-days`, which is when the same updates are attempted again. + record-attempt: + name: 'Record Attempt for ${{ matrix.tag }}' + needs: resolve + if: needs.resolve.outputs.rebuild-needed == 'true' + runs-on: ubuntu-24.04 + strategy: + fail-fast: false + matrix: ${{ fromJSON(needs.resolve.outputs.matrix) }} + steps: + - name: 'Write the pending updates' + if: matrix.attempt != '' + run: tr ' ' '\n' <<< "${PENDING}" >attempt.txt + env: + PENDING: ${{ matrix.pending }} + + - name: 'Upload the record' + if: matrix.attempt != '' + uses: actions/upload-artifact@v7 + with: + name: ${{ matrix.attempt }} + path: attempt.txt + retention-days: 7 rebuild: name: 'Rebuild ${{ matrix.tag }}' needs: resolve + if: needs.resolve.outputs.rebuild-needed == 'true' strategy: # A failing line must not stop the others: fail-fast: false diff --git a/CHANGELOG.md b/CHANGELOG.md index bdbca894..e3ed7cc8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,7 +28,7 @@ All notable changes to this project will be documented in this file. The format ### CI -- A weekly workflow rebuilds the newest release of each supported major version and republishes its `X.Y.Z`, `X.Y` and `X` tags (and `latest` for the newest release), plus an immutable `X.Y.Z-rN` tag for rollbacks ([#4833](https://github.com/docker-mailserver/docker-mailserver/pull/4833)) +- A daily workflow rebuilds the newest release of each supported major version when its published image has package updates pending, and republishes its `X.Y.Z`, `X.Y` and `X` tags (and `latest` for the newest release), plus an immutable `X.Y.Z-rN` tag for rollbacks ([#4833](https://github.com/docker-mailserver/docker-mailserver/pull/4833), [#4839](https://github.com/docker-mailserver/docker-mailserver/pull/4839)) - Moved `setup_cli.bats` to the serial test suite to prevent `setup.sh` from racing with teardown of another DMS test container ([#4701](https://github.com/docker-mailserver/docker-mailserver/issues/4701)) - Excluded `saslauthd` from the process kill/restart assertion because its forked parent process is not reliably reaped by the test diff --git a/SECURITY.md b/SECURITY.md index 8ce4b591..12e051d2 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -2,13 +2,13 @@ ## Supported Versions -Due to the way DMS is released, the most recent patches and the most current software is published on the `:edge` tag of the container image. Hence, security updates will land on this "rolling release tag". The newest release of each supported major version (currently `v16`, while its Debian base still receives security updates) is rebuilt weekly to include the current Debian security updates. Older tags need manual updating, as we do not usually release an updated image for an existing tag; this will only be done in case of _severe_ vulnerabilities. +Due to the way DMS is released, the most recent patches and the most current software is published on the `:edge` tag of the container image. Hence, security updates will land on this "rolling release tag". The newest release of each supported major version (currently `v16`, while its Debian base still receives security updates) is rebuilt as soon as package updates are available (checked daily), to include the current Debian security updates. Older tags need manual updating, as we do not usually release an updated image for an existing tag; this will only be done in case of _severe_ vulnerabilities. -| Image Tags | Latest Packages & Patches | -|---------------------------------------|:-----------------------------------:| -| `:edge` | :white_check_mark: | -| newest release of a supported major | :white_check_mark: (weekly rebuild) | -| other releases | :x: | +| Image Tags | Latest Packages & Patches | +|-------------------------------------|:--------------------------------------------:| +| `:edge` | :white_check_mark: | +| newest release of a supported major | :white_check_mark: (rebuilt on updates) | +| other releases | :x: | ## Reporting a Vulnerability diff --git a/docs/content/config/advanced/maintenance/update-and-cleanup.md b/docs/content/config/advanced/maintenance/update-and-cleanup.md index d3cfd67b..12a74a17 100644 --- a/docs/content/config/advanced/maintenance/update-and-cleanup.md +++ b/docs/content/config/advanced/maintenance/update-and-cleanup.md @@ -36,7 +36,7 @@ title: 'Maintenance | Update and Cleanup' The tag for an image is never modified by `watchtower`, instead `watchtower` monitors the image digest associated to that image tag (_which will change to a new image digest if a new image release reassigns the tag_), when the digest for the tag changes this triggers a pull of the new image. - Your container will not update to a new major release version (_unless using `:latest`_). - - The tags of the newest release of each supported major (eg `:16.0.1`, `:16.0`, `:16`, and `:latest` for the newest overall) are rebuilt weekly with the current Debian security fixes. Use the immutable `:16.0.1-rN` tag if you need to pin an exact build. + - The tags of the newest release of each supported major (eg `:16.0.1`, `:16.0`, `:16`, and `:latest` for the newest overall) are rebuilt with the current package updates (_Debian security fixes included_) when any are available. This is checked every day. Use the immutable `:16.0.1-rN` tag if you need to pin an exact build. - Omit the minor or patch portion of a semver tag to receive updates for the omitted portion (_eg: `:16` will represent the latest minor + patch release, whereas `:16.0` would only receive patch updates instead of minor releases like `16.1`_). !!! tip "Updating only specific containers" diff --git a/docs/content/usage.md b/docs/content/usage.md index 36d000ab..96ab19a1 100644 --- a/docs/content/usage.md +++ b/docs/content/usage.md @@ -89,14 +89,15 @@ All workflows are using the tagging convention listed below. It is subsequently |-------------------------|-------------------------------------------| | `push` on `master` | `edge` | | `push` a tag (`v1.2.3`) | `1.2.3`, `1.2`, `1`, `latest`, `1.2.3-r0` | -| Weekly rebuild | `1.2.3`, `1.2`, `1`, `latest`, `1.2.3-rN` | +| Package updates | `1.2.3`, `1.2`, `1`, `latest`, `1.2.3-rN` | -!!! info "Weekly rebuilds of supported releases" +!!! info "Rebuilds of supported releases on package updates" - Debian publishes security fixes continuously, but a release image is only built once. To pick those fixes up, the newest release of every supported major version (_currently `v16`_) is rebuilt every week without cached layers, tested, and published again: + Debian publishes security fixes continuously, but a release image is only built once. To pick those fixes up, the newest release of every supported major version (_currently `v16`_) is checked every day. If its published image has package updates pending (_from Debian_), it is rebuilt without cached layers, tested, and published again: - The tags `X.Y.Z`, `X.Y` and `X` of the rebuilt release are moved to the rebuilt image. `latest` is only moved when the rebuilt release is the newest overall. Older minor and patch releases within a major are not rebuilt and keep their last image. - Each build also receives an immutable `X.Y.Z-rN` tag (`-r0` is the original release build, rebuilds count up from `-r1`). Use it to pin an exact build or to roll back, or pin the image digest. + - A given set of pending updates is only attempted once, even if the rebuild fails or the updates stay pending afterwards. The next attempt happens when the set of pending updates changes, or after 7 days. To retry earlier, re-run the failed jobs of the workflow run or dispatch the workflow manually. [github-ci]: https://github.com/docker-mailserver/docker-mailserver/actions [dockerhub-image]: https://hub.docker.com/r/mailserver/docker-mailserver