Merge pull request #7948 from certbot/snap-build-squashed

Despite this PR (only) being ~200 lines containing mostly code copied from another repo, there is a lot going on here. For the sake of making it both easier to review and to remember some of these things in the future by referring back to this PR, I've documented a lot of noteworthy things with section headers below. With that said, it's probably not necessary to read each section unless you're interested in that topic. The most noteworthy thing for the reviewer is **this PR should be merged and not squashed** to preserve authorship. To merge this code, once we're happy with this PR, I'll probably open a new PR squashing any commits I make in response in review comments back into a single commit to try to keep history somewhat clean. To help prevent this PR from being accidentally squashed, I'm making this a draft PR for now. ### Git history of https://github.com/basak/certbot-snap-build I think it is worth preserving the git history of https://github.com/basak/certbot-snap-build that this PR is based on in this repo to help us track why things were done a certain way. To do this while keeping our git history somewhat clean, I took the approach described at https://stackoverflow.com/questions/1425892/how-do-you-merge-two-git-repositories/21495718#21495718 to move all history of https://github.com/basak/certbot-snap-build into a `snap` directory. I then squashed all commits so that sequential commits from the same author are one commit. I probably could have reordered commits to try and squash things a little more, but I personally don't think it's worth the trouble. Finally, I merged this rewritten history into this branch of the Certbot repo. The contents of the `snap` directory are identical to the current contents of https://github.com/basak/certbot-snap-build before my final commit in this PR which makes the changes to make things work in this repo. ### Travis stages This is described in general at https://docs.travis-ci.com/user/build-stages/, but I don't think we should deploy the snap if any of our tests are failing. To accomplish this, I created a "Snap" stage that builds, tests, and deploys the snap which is only executed after a "Test" stage that contains all of our other tests. The "Snap" stage will not run until the "Test" stage completes successfully. ### snap/local This directory is ignored by `snapcraft` which I think makes it a good place to store `snap` specific scripts like `build_and_install.sh`. See https://bugs.launchpad.net/snapcraft/+bug/1792203 for more info. ### Why remove certbot-compatibility-test from apacheconftest toxenvs? Because it's not used. In theory, it could go in its own PR, but it'll create merge conflicts with this one so I'd personally prefer to include this simple change in this PR as well. ### Checklist for landing this PR - [x] Squash all of my commits into one commit - [x] Update the release instructions to have to move the snap to the beta channel - [x] Shut down Robie's nightly builds probably by updating his repo to say that the code has moved here and deleting everything
2026-01-26 07:41:33 +03:00 · 2020-04-24 14:13:09 -07:00
parent 4cce3458f3 335894ab3b
commit 01dc981a09
5 changed files with 189 additions and 8 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -51,3 +51,10 @@ tests/letstest/venv3/
 .certbot_test_workspace
 **/assets/pebble*
 **/assets/challtestsrv*
+
+# snap files
+.snapcraft
+parts
+prime
+stage
+*.snap
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,17 +11,23 @@ before_script:
  # Use Travis retry feature for farm tests since they are flaky
  - 'if [[ "$TOXENV" == "travis-test-farm"* ]]; then export TRAVIS_RETRY=travis_retry; fi'
  - export TOX_TESTENV_PASSENV=TRAVIS
+  - 'if [[ "$SNAP" == true ]]; then snap/local/build_and_install.sh; fi'

 # Only build pushes to the master branch, PRs, and branches beginning with
-# `test-`, `travis-test-`, or of the form `digit(s).digit(s).x`. This reduces
-# the number of simultaneous Travis runs, which speeds turnaround time on
-# review since there is a cap of on the number of simultaneous runs.
+# `test-`, `travis-test-`, or of the form `digit(s).digit(s).x` or
+# `vdigit(s).digit(s).digit(s)`. As documented at
+# https://docs.travis-ci.com/user/customizing-the-build/#safelisting-or-blocklisting-branches,
+# this includes tags so pushing tags of the form `vdigit(s).digit(s).digit(s)`
+# will also trigger tests. This reduces the number of simultaneous Travis runs,
+# which speeds turnaround time on review since there is a cap of on the number
+# of simultaneous runs.
 branches:
  # When changing these branches, please ensure the documentation under
  # "Running tests in CI" is still correct.
  only:
    - master
-    - /^\d+\.\d+\.x$/
+    - /^\d+\.\d+\.x$/  # this matches our point release branches
+    - /^v\d+\.\d+\.\d+$/  # this matches our release tags
    - /^(travis-)?test-.*$/

 # Jobs for the main test suite are always executed (including on PRs) except for pushes on master.
@@ -36,10 +42,16 @@ extended-test-suite: &extended-test-suite
 matrix:
  include:
    # Main test suite
-    - python: "2.7"
+    - stage: "Test"
+      python: "2.7"
      env: ACME_SERVER=pebble TOXENV=integration
      <<: *not-on-master

+    # As documented at
+    # https://docs.travis-ci.com/user/build-stages/#how-to-define-build-stages,
+    # the previous stage will be automatically applied to all subsequent jobs
+    # until a new stage is defined.
+
    # This job is always executed, including on master
    - python: "3.8"
      env: TOXENV=py38-cover FYI="py38 tests + code coverage"
@@ -218,6 +230,51 @@ matrix:
          packages:  # don't install nginx and apache
            - libaugeas0
      <<: *extended-test-suite
+    - stage: "Snap"
+      sudo: required
+      env: SNAP=true TOXENV=integration-external,apacheconftest-external-with-pebble
+      addons:
+        apt:
+          packages:
+          - nginx-light
+        snaps:
+          - name: snapcraft
+            channel: stable
+            confinement: classic
+          - name: lxd
+            channel: stable
+      git:
+        # By default, Travis clones the repo to a depth of 50 commits which can
+        # break the ability to use `git describe` to set the version of the
+        # snap. This setting removes the --depth flag from git commands solving
+        # this problem. See
+        # https://docs.travis-ci.com/user/customizing-the-build#git-clone-depth
+        # for more info.
+        depth: false
+      deploy:
+        # This section relies on credentials stored in a SNAP_TOKEN environment
+        # variable in Travis. See
+        # https://docs.travis-ci.com/user/deployment/snaps/ for more info.
+        # This credential has a maximum lifetime of 1 year and the current
+        # credential will expire on 4/22/2021. The value of SNAP_TOKEN will
+        # need to be updated to use a new credential before then to prevent
+        # automated deploys from breaking. Remembering to do this is also
+        # tracked by https://github.com/certbot/certbot/issues/7931.
+        'on':
+          # Deploy on release tags or nightly runs from any branch. We only try
+          # to deploy from the certbot/certbot repo to prevent errors if forks
+          # of this repo try to run tests.
+          all_branches: true
+          condition: -n $TRAVIS_TAG || $TRAVIS_EVENT_TYPE = cron
+          repo: certbot/certbot
+        provider: snap
+        snap: certbot_*.snap
+        channel: edge
+        # skip_cleanup is needed to prevent Travis from deleting the snaps we
+        # just built and tested. See
+        # https://docs.travis-ci.com/user/deployment#uploading-files-and-skip_cleanup.
+        skip_cleanup: true
+      <<: *extended-test-suite

 # container-based infrastructure
 sudo: false
--- a/snap/local/build_and_install.sh
+++ b/snap/local/build_and_install.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+set -ex
+
+if [[ -z "$TRAVIS" ]]; then
+    echo "This script makes global changes to the system it is run on so should only be run in CI."
+    exit 1
+fi
+
+sudo /snap/bin/lxd.migrate -yes
+sudo /snap/bin/lxd waitready
+sudo /snap/bin/lxd init --auto
+tools/strip_hashes.py letsencrypt-auto-source/pieces/dependency-requirements.txt > constraints.txt
+sudo snapcraft --use-lxd
+sudo snap install --dangerous --classic *.snap
--- a/snap/snapcraft.yaml
+++ b/snap/snapcraft.yaml
@@ -0,0 +1,88 @@
+name: certbot
+summary: Automatically configure HTTPS using Let's Encrypt
+description: |
+ The objective of Certbot, Let's Encrypt, and the ACME (Automated
+ Certificate Management Environment) protocol is to make it possible
+ to set up an HTTPS server and have it automatically obtain a
+ browser-trusted certificate, without any human intervention. This is
+ accomplished by running a certificate management agent on the web
+ server.
+ 
+ This agent is used to:
+   - Automatically prove to the Let's Encrypt CA that you control the website
+   - Obtain a browser-trusted certificate and set it up on your web server
+   - Keep track of when your certificate is going to expire, and renew it
+   - Help you revoke the certificate if that ever becomes necessary.
+confinement: classic
+grade: devel
+base: core18
+adopt-info: certbot
+
+apps:
+  certbot:
+    command: certbot
+    environment:
+      PATH: "$SNAP/bin:$SNAP/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
+      AUGEAS_LENS_LIB: "$SNAP/usr/share/augeas/lenses/dist"
+      LD_LIBRARY_PATH: "$SNAP/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH"
+  renew:
+    command: certbot -q renew
+    daemon: oneshot
+    environment:
+      PATH: "$SNAP/bin:$SNAP/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
+      AUGEAS_LENS_LIB: $SNAP/usr/share/augeas/lenses/dist
+      LD_LIBRARY_PATH: "$SNAP/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH"
+    passthrough:
+        # Run approximately twice a day with randomization
+        timer: 00:00~24:00/2
+
+parts:
+  python-augeas:
+    plugin: python
+    source: git://github.com/basak/python-augeas
+    source-branch: snap
+    python-version: python3
+    build-packages: [libaugeas-dev]
+  acme:
+    plugin: python
+    source: .
+    source-subdir: acme
+    constraints: [$SNAPCRAFT_PART_SRC/constraints.txt]
+    python-version: python3
+  certbot:
+    plugin: python
+    source: .
+    source-subdir: certbot
+    constraints: [$SNAPCRAFT_PART_SRC/constraints.txt]
+    python-version: python3
+    after: [acme]
+    override-pull: |
+        snapcraftctl pull
+        snapcraftctl set-version `cd $SNAPCRAFT_PART_SRC && git describe|sed s/^v//`
+    # Workaround for lack of site-packages leading to empty sitecustomize.py
+    stage:
+      - -usr/lib/python3.6/sitecustomize.py
+  certbot-apache:
+    plugin: python
+    source: .
+    source-subdir: certbot-apache
+    constraints: [$SNAPCRAFT_PART_SRC/constraints.txt]
+    python-version: python3
+    after: [python-augeas, certbot]
+    stage-packages: [libaugeas0]
+    stage:
+      # Prefer cffi
+      - -lib/python3.6/site-packages/augeas.py
+  certbot-nginx:
+    plugin: python
+    source: .
+    source-subdir: certbot-nginx
+    constraints: [$SNAPCRAFT_PART_SRC/constraints.txt]
+    python-version: python3
+    # This is the last step, compile pycache now as there should be no conflicts.
+    override-prime: |
+      snapcraftctl prime
+      ./usr/bin/python3 -m compileall -q .
+    # After certbot-apache to not rebuild duplicates (essentially sharing what was already staged,
+    # like zope)
+    after: [certbot-apache]
--- a/tox.ini
+++ b/tox.ini
@@ -138,15 +138,22 @@ commands =

 [testenv:apacheconftest]
 commands =
-    {[base]pip_install} acme certbot certbot-apache certbot-compatibility-test
+    {[base]pip_install} acme certbot certbot-apache
    {toxinidir}/certbot-apache/tests/apache-conf-files/apache-conf-test --debian-modules
 passenv =
    SERVER

+[testenv:apacheconftest-external-with-pebble]
+# Run apacheconftest with pebble and Certbot outside of tox's virtual
+# environment.
+commands =
+    {[base]pip_install} certbot-ci
+    {toxinidir}/certbot-apache/tests/apache-conf-files/apache-conf-test-pebble.py --debian-modules
+
 [testenv:apacheconftest-with-pebble]
 commands =
-    {[base]pip_install} acme certbot certbot-apache certbot-ci certbot-compatibility-test
-    {toxinidir}/certbot-apache/tests/apache-conf-files/apache-conf-test-pebble.py --debian-modules
+    {[base]pip_install} acme certbot certbot-apache
+    {[testenv:apacheconftest-external-with-pebble]commands}

 [testenv:nginxroundtrip]
 commands =
@@ -250,6 +257,14 @@ commands =
        --cov-config=certbot-ci/certbot_integration_tests/.coveragerc
    coverage report --include 'certbot/*' --show-missing --fail-under=62

+[testenv:integration-external]
+# Run integration tests with Certbot outside of tox's virtual environment.
+commands =
+    {[base]pip_install} certbot-ci
+    pytest certbot-ci/certbot_integration_tests \
+        --acme-server={env:ACME_SERVER:pebble}
+passenv = DOCKER_*
+
 [testenv:integration-certbot-oldest]
 commands =
    {[base]pip_install} certbot