`tox -e cover` fails for me on macOS. This is due to the differences in the code that is run when on Linux vs. other platforms in `certbot.util` and its tests. Diffing my local coverage with Travis, the only difference in lines missing test coverage is:
```
$ diff travis.txt local.txt
< certbot/certbot/util.py 238 24 90% 132-134, 210, 275-281, 318, 330-340, 347, 381-382
> certbot/certbot/util.py 239 34 86% 30, 132-134, 210, 275-281, 301, 305, 316-318, 330-340, 347, 367-373, 381-382
< certbot/tests/util_test.py 392 0 100%
> certbot/tests/util_test.py 375 26 93% 483-487, 492-502, 507-514, 548-550, 555-557
```
I think tests on `master` should not be failing locally for people.
While there would be other ways to fix this by adding `# pragma: no cover` lines or writing mocked out tests for other platforms, I personally just think dropping the minimum coverage one percentage point is fine at least for now.
Coming out of the conversation at #7863 in the linked Google Doc, we should always have at least 1 release between updating one of our plugins to stop using a deprecated acme/certbot API and removing it from acme/certbot. Doing this gives the plugin changes time to propagate rather than potentially having the plugin break because Certbot was updated before the plugin had made the necessary changes.
This comment here should help ensure this.
* Add pytest warnings warning.
* clarify comment
Fixes#7713.
As discussed in #7713, providing a Powershell script as hook for Certbot is not working currently. This is because hooks are run in a `cmd` environment, that recognizes only `.bat` files as valid scripts that can be run from their bare name on command line.
On the other hand, the Powershell both `.bat` and `.ps1` scripts as valid scripts.
This PR makes hooks command be executed by Powershell, instead of `cmd` as `Popen` does by default when `shell=true` is used. It also modifies the tests to handle this new environment, in particular in term of encoding (UTF-16-LE is the default one in Powershell).
* Run hooks in powershell on Windows
* Fix hook test
* Fallback to unittest.mock
* In fact, shell_cmd as a list of str could not work. Declare only str as acceptable input for shell_cmd.
* Added changelog
Despite this PR (only) being ~200 lines containing mostly code copied from another repo, there is a lot going on here. For the sake of making it both easier to review and to remember some of these things in the future by referring back to this PR, I've documented a lot of noteworthy things with section headers below. With that said, it's probably not necessary to read each section unless you're interested in that topic.
The most noteworthy thing for the reviewer is **this PR should be merged and not squashed** to preserve authorship. To merge this code, once we're happy with this PR, I'll probably open a new PR squashing any commits I make in response in review comments back into a single commit to try to keep history somewhat clean. To help prevent this PR from being accidentally squashed, I'm making this a draft PR for now.
### Git history of https://github.com/basak/certbot-snap-build
I think it is worth preserving the git history of https://github.com/basak/certbot-snap-build that this PR is based on in this repo to help us track why things were done a certain way. To do this while keeping our git history somewhat clean, I took the approach described at https://stackoverflow.com/questions/1425892/how-do-you-merge-two-git-repositories/21495718#21495718 to move all history of https://github.com/basak/certbot-snap-build into a `snap` directory. I then squashed all commits so that sequential commits from the same author are one commit. I probably could have reordered commits to try and squash things a little more, but I personally don't think it's worth the trouble. Finally, I merged this rewritten history into this branch of the Certbot repo.
The contents of the `snap` directory are identical to the current contents of https://github.com/basak/certbot-snap-build before my final commit in this PR which makes the changes to make things work in this repo.
### Travis stages
This is described in general at https://docs.travis-ci.com/user/build-stages/, but I don't think we should deploy the snap if any of our tests are failing. To accomplish this, I created a "Snap" stage that builds, tests, and deploys the snap which is only executed after a "Test" stage that contains all of our other tests. The "Snap" stage will not run until the "Test" stage completes successfully.
### snap/local
This directory is ignored by `snapcraft` which I think makes it a good place to store `snap` specific scripts like `build_and_install.sh`.
See https://bugs.launchpad.net/snapcraft/+bug/1792203 for more info.
### Why remove certbot-compatibility-test from apacheconftest toxenvs?
Because it's not used. In theory, it could go in its own PR, but it'll create merge conflicts with this one so I'd personally prefer to include this simple change in this PR as well.
### Checklist for landing this PR
- [x] Squash all of my commits into one commit
- [x] Update the release instructions to have to move the snap to the beta channel
- [x] Shut down Robie's nightly builds probably by updating his repo to say that the code has moved here and deleting everything
* merge .gitignore
* Move snapcraft.yml up one level.
* update source
* move test.sh to tox.ini
* use new tox.ini in .travis.yml
* move snap build code
* make script executable
* remove unused python3-dev
* don't use deprecated classic flag
* go back to stable channel
* add nginx in snap addons
* add deploy steps
* Add comments explaining external tox envs.
* error if not in CI
* don't use --depth
* remove old .travis.yml
* Add big comment about SNAP_TOKEN.
* Set all_branches: true.
* Add repo setting.
* run travis on tags
* Add more documenting comments to .travis.yml.
* snap: move snapcraft.yaml to snap directory
Signed-off-by: Sergio Schvezov <sergio.schvezov@canonical.com>
* snap: use a local plugin to get around the delivered plugin
Add a plugin to the project which behaves as expected until a version
of snapcraft satisfies the project needs.
Additional snapcraft.yaml changes were made to accommodate for the snap
to build.
Signed-off-by: Sergio Schvezov <sergio.schvezov@canonical.com>
* snap: compile pycache in the last step for the last part
Signed-off-by: Sergio Schvezov <sergio.schvezov@canonical.com>
* Remove legacy Store upload credentials
These have not been needed since 5d7969a.
* Work around dev part dependency failure case
Get pip to install certbot from its VCS repository directly during the
build of the nginx and apache plugin parts.
This works around issue #12 when it affects the interaction between the
apache or nginx plugin and certbot itself.
It does not work around the case where the same problem occurs in the
interaction between certbot and acme. This looks harder to work around
because pip's VCS URL handling doesn't appear to include a facility to
install from a subdirectory of a git repository and this is where the
acme source is located.
* Switch to using lxd for the snapcraft run
The Docker image is 16.04 only. Before we can switch to 18.04, we need
to remove Docker, which means using the snapcraft snap using Travis' new
snap support, and then using the lxd functionality in snapcraft so that
it is enabled to build in the appropriate environment.
* Switch build to use core 18
Fixes: #14
* Move constraints into a list
This seems to be a requirement either either newer snapcraft, snapcraft
in lxd or in the move to core18 (it isn't clear to me which). This fixes
the error:
Failed to load plugin: properties failed to load for certbot: The 'constraints' property does not match the required schema: '$SNAPCRAFT_PART_SRC/constraints.txt' is not of type 'array'
* version-script -> snapcraftctl set-version
The use of version-script seems to break with either newer snapcraft,
snapcraft with lxd or core18 (it's not clear to me which). The breakage
is related to "parts/certbot/src" not being found. This can be fixed
with $SNAPCRAFT_PART_SRC, but this doesn't seem to be defined at
"version-script time".
However https://snapcraft.io/docs/deprecation-notices/dn10 deprecates
the use of version-script, and if we convert to the recommended new way,
then we use override-pull instead and $SNAPCRAFT_PART_SRC is defined
there, so this conveniently fixes both problems at once.
* Do not explicitly install snapd
Since this is now handled by the Travis addon, we do not need to do it
explicitly.
* Add Travis notifications
* Adjust automatic snap deployment configuration
Travis now has a documented[1] "snap" provider and the previous
experimental mechanism seems to have stopped working, presumably because
it was deprecated in favour of this new mechanism.
[1] https://docs.travis-ci.com/user/deployment/snaps/
* Configure for python3
* Update tests
* Use appropriate virtualenv
* Install nginx for the integration tests
* Try use LD_LIBRARY_PATH to find augeas shared library in snap when python-augeas is invoked
* Update travis to use build-in setup capabilities
* Update .travis.yml
* Add acme build
* Update tests
* Try more recent dist
* Update command
* Clean tests
* Add back augeas
* Add env
* Revert to last working snapcraft config
* Add a gitignore
* Reintegrate acme. Declare augeas in certbot parts
* Use release version of certbot
* Try new approach
* Fix config
* Directly install version of python-augeas from pypi
* Restart from basic
* Clone only once certbot repository. Use pinned versions of dependencies from certbot-auto.
* Try relatively to source
* Use snapcraft env variables
* Strip hashes
* Fix path
* Redefine path
* Continue to prepare the runtime
* Fix command line
* Update .travis.yml
* Add back certbot-apache
* Update snapcraft.yaml
* Build snap against the latest release of certbot
* Add renewal timer
* Install libaugeas0 in python-augeas part build
This part needs libaugeas0 to build.
* Bump to 0.26.1
* Always act directly on upstream master
I want to keep this always working, so move to master. We can
reintroduce upstream stable releases when we are ready for general use.
Closes: #5
That particular issue seems to no longer happen. Presumably something
changed in upstream git or in PyPI. If it happens again, hopefully I'll
have CI against upstream master up by then and I'll be able to pin it
down.
* Add empty Travis build
* Add Travis automatic snap edge publication
* Add integration test
This uses upstream's test suite from their source tree to check the
built snap to make sure it behaves as expected, before attempting upload
to the store.
* Point Augeas to its lens library
Augeas defaults to looking in /usr/share/augeas/lenses, which in a snap
isn't found at this path, but inside $SNAP. So set AUGEAS_LENS_LIB to
where the lenses can be found within the snap.
This fixes the Apache plugin that uses Augeas.
In #7925 we accidently changed the logic here. Before it was:
type = cron OR (type = push AND branch NOT IN (apache-parser-v2, master))
now it's
type = cron OR (type = push AND branch = master)
We want to be able to run our full test suite on things like test-* branches. The reason we had been excluding master is it has the full test suite run on it (through what Travis calls cron) nightly and not running on every commit helps prevent us on waiting on CI since our nightly tests spin up so many jobs.
This PR changes things back to the intended behavior.
(We could talk about changing the condition to just type = cron OR type = push if we want, but I'd rather do that in a separate PR once things like test- branches are fixed.)
Fixes#7268
I removed the reference to automatically selecting which ACME protocol we use, since at some point we'll want to rip out the non-spec-compliant ACMEv1 code.
Python 2 is going to get harder and harder to install locally so I don't think we should assume/require devs to have it installed.
This PR builds on #7905 so our developer guide only has people use Python 3.
Part of #7886.
This PR conditionally installs `mock` in `certbot-dns-*/setup.py` based on setuptools version and python version, when possible. It then updates the tests to use `unittest.mock` when `mock` isn't available.
* Do not require mock in Python 3 in certbot-dns modules
* update changelog
* error when trying to build wheels with old setuptools
* add type: ignores
Part of #7886.
This PR conditionally installs mock in `apache/setup.py` based on setuptools version and python version, when possible. It then updates `apache` tests to use `unittest.mock` when `mock` isn't available.
* Conditionally install mock in apache
* error out on newer python and older setuptools
* error when trying to build wheels with old setuptools
* use unittest.mock when third-party mock isn't available in apache, with no cover and type ignore
This PR is exactly the same as #7895, but know we know a little bit more about what was going on with `mypy`.
Part of #7886.
This PR conditionally installs mock in `certbot/setup.py` based on setuptools version and python version, when possible. It then updates `certbot` tests to use `unittest.mock` when `mock` isn't available.
* Conditionally install mock in certbot
* use unittest.mock when third-party mock isn't available in certbot
* Add type:ignores because of https://github.com/python/mypy/issues/1153
* error out on newer python and older setuptools
* error when trying to build wheels with old setuptools
Part of #7886.
This PR conditionally installs mock in `acme/setup.py` based on setuptools version and python version, when possible. It then updates `acme` tests to use `unittest.mock` when `mock` isn't available.
Now with `type: ignore` as appropriate. Once the "future steps" of #7886 are finished, and mypy is on Python 3, the `pragma no cover`s and `type ignore`s will be gone.
* Conditionally install mock in acme
* error out on newer python and older setuptools
* error when trying to build wheels with old setuptools
* use unittest.mock when third-party mock isn't available in acme, with no cover and type ignore
This PR fixes the Travis failures that can be seen https://travis-ci.com/certbot/certbot/builds/160258644. Running the tests locally, it looks like Ubuntu has started shutting down the 19.04 repos which makes sense as this release has been EOL'd. See https://wiki.ubuntu.com/Releases.
I have the full suite including the test farm tests running at https://travis-ci.com/github/certbot/certbot/builds/160269969 with this change.
The issue of adding 19.10 to our test farm tests is tracked by #7851. I think that issue is important and it's in our current milestone, but I'd personally rather get our tests passing for now and try to expand them to run on other systems later.
* Revert "Do not require mock in Python 3 in certbot module (#7895)"
This reverts commit 77871ba71c.
* Revert "Do not require mock in Python 3 in acme module (#7894)"
This reverts commit cd0acf5dcc.
Part of #7886.
This PR conditionally installs mock in `certbot/setup.py` based on setuptools version and python version, when possible. It then updates `certbot` tests to use `unittest.mock` when `mock` isn't available.
* Conditionally install mock in certbot
* use unittest.mock when third-party mock isn't available in certbot
* Add type:ignores because of https://github.com/python/mypy/issues/1153
* error when trying to build wheels with old setuptools
Part of #7886.
This PR conditionally installs mock in acme/setup.py based on setuptools version and python version, when possible. It then updates acme tests to use unittest.mock when mock isn't available.
* Conditionally install mock in acme
* use unittest.mock when third-party mock isn't available in acme
* error when trying to build wheels with old setuptools
* Fix dangerous default argument
* Remove unused imports
* Remove unnecessary comprehension
* Use literal syntax to create data structure
* Use literal syntax instead of function calls to create data structure
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Fixes#7857.
* stop using urllib2 in test farm tests
* use six for urllib instead
* remove fabric lcd usage
* correct lcd removal
* remove fabric cd
* convert some remote calls to v2
* move more cxns to v2
* get run working with prefix
* get sudo commands working
* remove final fabric v1 references including local
* update requirements and README
* add new venv to gitignore
* update version used in travis
* remove deploy_script unused kwargs
* fix killboulder implementation so I can test creating a new boulder server
* hardcode the gopath due to broken env manamagement in fabric2
* Update letstest readme
* move the comment about hardcoding the ggopath
* catch BaseException instead of Exception
* work around fabric #2007
* use connections as context managers to ensure they're closed
* remove reference to virtualenv
Translate a proxy specified by an environment variable ("http_proxy"
or "HTTP_PROXY") into options recognized by "openssl ocsp". Support
is limited to HTTP proxies which don't require authentication.
Fixes#6150
