ci: Add continuous integration for github repositories via cirrus-ci.

Currently FreeBSD, Linux, macOS and Windows (Visual Studio) are tested.

The main goal of this integration is to make it easier to test in-development
patches across multiple platforms. This includes improving the testing done
automatically by cfbot [1] for commitfest entries.  It is *not* the goal to
supersede the buildfarm.

cirrus-ci [2] was chosen because it was already in use for cfbot, allows using
full VMs, has good OS coverage and allows accessing the full test results
without authentication (like a github account).  It might be worth adding
support for further CI providers, particularly ones supporting other git
forges, in the future.

To keep CI times tolerable, most platforms use pre-generated images. Some
platforms use containers, others use full VMs.

For instructions on how to enable the CI integration in a repository and
further details, see src/tools/ci/README

[1] http://cfbot.cputube.org/
[2] https://cirrus-ci.org/

Author: Andres Freund <andres@anarazel.de>
Author: Thomas Munro <tmunro@postgresql.org>
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-By: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-By: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-By: Thomas Munro <tmunro@postgresql.org>
Reviewed-By: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Discussion: https://postgr.es/m/20211001222752.wrz7erzh4cajvgp6@alap3.anarazel.de

src/tools/ci/README

@@ -0,0 +1,63 @@
+Postgres Continuous Integration (CI)
+====================================
+
+Postgres has two forms of CI:
+
+1) All supported branches in the main postgres repository are continuously
+   tested via the buildfarm. As this covers only the main repository, it
+   cannot be used during development of features.
+
+   For details see https://buildfarm.postgresql.org/
+
+2) For not yet merged development work, CI can be enabled for some git hosting
+   providers. This allows developers to test patches on a number of platforms
+   before they are merged (or even submitted).
+
+
+Configuring CI on personal repositories
+=======================================
+
+Currently postgres contains CI support utilizing cirrus-ci. cirrus-ci
+currently is only available for github.
+
+
+Enabling cirrus-ci in a github repository
+=========================================
+
+To enable cirrus-ci on a repository, go to
+https://github.com/marketplace/cirrus-ci and select "Public
+Repositories". Then "Install it for free" and "Complete order". The next page
+allows to configure which repositories cirrus-ci has access to. Choose the
+relevant repository and "Install".
+
+See also https://cirrus-ci.org/guide/quick-start/
+
+Once enabled on a repository, future commits and pull-requests in that
+repository will automatically trigger CI builds. These are visible from the
+commit history / PRs, and can also be viewed in the cirrus-ci UI at
+https://cirrus-ci.com/github/<username>/<reponame>/
+
+
+Images used for CI
+==================
+
+To keep CI times tolerable, most platforms use pre-generated images. Some
+platforms use containers, others use full VMs. Images for both are generated
+separately from CI runs, otherwise each git repository that is being tested
+would need to build its own set of containers, which would be wasteful (both
+in space and time.
+
+These images are built, on a daily basis, from the specifications in
+github.com/anarazel/pg-vm-images/
+
+
+Controlling CI via commit messages
+==================================
+
+The behavior of CI can be controlled by special content in commit
+messages. Currently the following controls are available:
+
+- ci-os-only: {(freebsd|linux|macos|windows)}
+
+  Only runs CI on operating systems specified. This can be useful when
+  addressing portability issues affecting only a subset of platforms.

src/tools/ci/cores_backtrace.sh

@@ -0,0 +1,50 @@
+#! /bin/sh
+
+if [ $# -ne 2 ]; then
+    echo "cores_backtrace.sh <os> <directory>"
+    exit 1
+fi
+
+os=$1
+directory=$2
+
+case $os in
+    freebsd|linux|macos)
+    ;;
+    *)
+        echo "unsupported operating system ${os}"
+        exit 1
+    ;;
+esac
+
+first=1
+for corefile in $(find "$directory" -type f) ; do
+    if [ "$first" -eq 1 ]; then
+        first=0
+    else
+        # to make it easier to separate the different crash reports
+        echo -e '\n\n'
+    fi
+
+    if [ "$os" = 'macos' ]; then
+        lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit'
+    else
+        auxv=$(gdb --quiet --core ${corefile} --batch -ex 'info auxv' 2>/dev/null)
+        if [ $? -ne 0 ]; then
+            echo "could not process ${corefile}"
+            continue
+        fi
+
+        if [ "$os" = 'freebsd' ]; then
+            binary=$(echo "$auxv" | grep AT_EXECPATH | perl -pe "s/^.*\"(.*)\"\$/\$1/g")
+        elif [ "$os" = 'linux' ]; then
+            binary=$(echo "$auxv" | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g")
+        else
+            echo 'should not get here'
+            exit 1
+        fi
+
+        echo "dumping ${corefile} for ${binary}"
+        gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" "$binary" "$corefile" 2>/dev/null
+    fi
+done

src/tools/ci/gcp_freebsd_repartition.sh

@@ -0,0 +1,28 @@
+#!/bin/sh
+
+set -e
+set -x
+
+# The default filesystem on freebsd gcp images is very slow to run tests on,
+# due to its 32KB block size
+#
+# XXX: It'd probably better to fix this in the image, using something like
+# https://people.freebsd.org/~lidl/blog/re-root.html
+
+# fix backup partition table after resize
+gpart recover da0
+gpart show da0
+# kill swap, so we can delete a partition
+swapoff -a || true
+# (apparently we can only have 4!?)
+gpart delete -i 3 da0
+gpart add -t freebsd-ufs -l data8k -a 4096 da0
+gpart show da0
+newfs -U -b 8192 /dev/da0p3
+
+# Migrate working directory
+du -hs $CIRRUS_WORKING_DIR
+mv $CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR.orig
+mkdir $CIRRUS_WORKING_DIR
+mount -o noatime /dev/da0p3 $CIRRUS_WORKING_DIR
+cp -r $CIRRUS_WORKING_DIR.orig/* $CIRRUS_WORKING_DIR/

src/tools/ci/pg_ci_base.conf

@@ -0,0 +1,14 @@
+# Tends to produce too many core files, taking a long time
+restart_after_crash = false
+
+# So that tests using the "manually" started postgres on windows can use
+# prepared statements
+max_prepared_transactions = 10
+
+# Settings that make logs more useful
+log_autovacuum_min_duration = 0
+log_checkpoints = true
+log_connections = true
+log_disconnections = true
+log_line_prefix = '%m [%p][%b] %q[%a][%v:%x] '
+log_lock_waits = true

src/tools/ci/windows_build_config.pl

@@ -0,0 +1,13 @@
+use strict;
+use warnings;
+
+our $config;
+
+$config->{"tap_tests"} = 1;
+$config->{"asserts"} = 1;
+
+$config->{"openssl"} = "c:/openssl/1.1/";
+$config->{"perl"} = "c:/strawberry/$ENV{DEFAULT_PERL_VERSION}/perl/";
+$config->{"python"} = "c:/python/";
+
+1;