* IDisplay.notification: add `decorate` param.
The flag allows the caller to control whether the message will be
printed in a decorated way (wrapped by hlines) or in an undecorated
way (similar to print).
It is set to true by default, to reflect the existing behavior of the
function.
* IDisplay.notification: write message to debug log
In the same vein as IReporter, this ensures that all notifications which
are shown to the user also make an appearance in the debug log, which
will aid in troubleshooting.
* restore accidentally deleted newline in decoration
* add helper function for printing status messages
* register: use notify rather than logger
Undoes the change in #8393 in favor of the new helper
* comment .display and ._internal.log
Describing when it is suitable to use each
* add more comments to log.py
* make IDisplay.notification decorate arg private
* rename notify->print and move to .display.util
* rename .display.print back to .display.notify
because linters complain about print being a redefined builtin
* Add a new, simplified version of pipstrap.
* Use tools/pipstrap.py
* Uncomment code
* Refactor pip_install.py and provide hashes.
* Fix test_sdists.sh.
* Make code work on Python 2.
* Call strip_hashes.py using Python 3.
* Pin the oldest version of httplib2 used in distros
* Strip enum34 dependency.
* Remove pip pinnings from dev_constraints.txt
* Correct pipstrap docstring.
* Don't set working_dir twice.
* Add comments
* Remove python_version from mypy.ini.
* Fix magic_typing
* Ignore msvcrt usage.
* make mypy happier
* clean up changes
* Add type for reporter queue
* More mypy fixes
* Fix pyrfc3339 str.
* Remove unused import.
* Make certbot.util mypy work in both Pythons
* Fix typo
While reviewing https://github.com/certbot/certbot/pull/8404, it occurred to me that we're keeping both the generated files and the script used to generate them in `git`. Keeping both around seems unnecessary and is almost asking for the files to get out of sync at some point in the future. I fixed that by removing the files, adding them to `.gitignore`, and updating `build_remote.py` to generate them as needed.
* Remove generated files.
* Add generated files to gitignore.
* Reuse generate_dnsplugins_all.sh in build_remote
While working on https://github.com/certbot/certbot/issues/8400, I noticed our Fedora AMIs are quite out of date. I considered updating them and what we could do to avoid the AMIs becoming so out-of-date in the future, but I think we don't actually need these tests.
I pulled a new count of Certbot users by OS and we have less than 7,000 Fedora users meaning only ~0.26% of Certbot users run Fedora. (I think Fedora is a popular desktop OS, but not as popular of a server OS which is where Certbot normally runs.)
Also, Certbot is regularly updated on Fedora including Fedora Rawhide or the rolling release version of Fedora which is similar to Debian sid/unstable. Rawhide changes far too frequently for it to make sense for us to run tests there in my opinon, but that also means that many problems such as Certbot's unit tests failing to run because of Fedora changes will be caught there by our Fedora maintainers before we'd even see it. This is how https://github.com/certbot/certbot/issues/7106 became an issue and how I learned [Certbot worked on Python 3.9 before we could run tests on it](https://github.com/certbot/certbot/issues/8134#issuecomment-655106169).
Because of all this, I think we should just simplify things and remove these tests. If a problem arises in the future, we can always add them back.
Fixes#8409.
Change the line in the README to allow `sudo /snap/bin/lxd.migrate -yes` to fail (for example, if there's nothing to migrate), but the whole command to succeed.
I tested this on a clean Focal install and confirmed it works.
Fixes https://github.com/certbot/certbot/issues/8400.
I had to switch the package installed in `apacheconftest` to `libapache2-mod-wsgi-py3` because Ubuntu 20.10 removed the Python 2 version of this module.
I didn't add this AMI to `tests/letstest/auto_targets.yaml` because like Ubuntu 20.04, `certbot-auto` has never worked on the OS.
* Add Ubuntu 20.20 test farm tests
* Try Python 3 WSGI
This PR adds the following documentation improvements to fix https://github.com/certbot/certbot/issues/7958:
- Simplify building external plugins
- Separate out certbot snap instructions from plugin instructions
- Mention that dnsimple is just an example for the plugin instructions
- Mention remote build for other architectures
- Mention snap doc exists elsewhere in developer guide (`contributing.rst`)
* Set up generate_dnsplugins_all.sh for all files and parametrize snapcraft and postrefreshhook files
* Create constraints file in the generate_dnsplugins_all script
* Separate out plugin and certbot snaps and update instructions
* Add remote build instructions
* Add pointers to the README to contributing.rst
Fixes#8355
During the troubleshooting of #8355, I came to the conclusion that using buildkit was creating the problem. Without it all docker images are built correctly. Initially buildkit was enabled to avoid a building problem in Azure Pipeline, but I also found in my recent tests that this problem was not there anymore.
You can find more details about the troubleshooting and reasoning in #8355.
As a consequence, I disable the usage of buildkit in this PR which will solve the issue.
Fixes#8202
This PR adds an Azure Pipeline job to execute certbot plugins --prepare for each Docker image created during the CI on amd64.
* Prepare basic integration tests for certbot dockers
* Add a displayName for the integration tests task
* Add timeout to DNS query function calls
* Modify tests to account for new timeout variable
* Add change to CHANGELOG
* Add `dns.exception.Timeout` to exception handler
* Move changelog to 1.10.0
Fixes https://github.com/certbot/certbot/issues/8171.
See the comment at the top of the script to learn how to set things up and run this. Running the script between releases will have no effect on our snaps and it should fail when creating the GitHub release. The latter is described at https://github.com/certbot/certbot/pull/8189#discussion_r466707114.
* Rename create_github_release to finish_release
* Add initial version of snap release automation.
* Handle snapcraft login.
* Catch OSError raised when snapcraft doesn't exist.
* Update documentation.
* Only publish the Certbot snap for now.
* Fix typo.
* Document other exceptions.
* Document assertion
* Add status message before getting revisions.
* Publish all snaps.
With more and more of our wildcard instructions on https://certbot.eff.org telling people to use these plugins, I think we should get ready to move our DNS plugins to the stable channel. This PR removes grade: devel so the snap store doesn't prevent us from doing that when we want to. See #8128 where we did this to the Certbot snap for more info.
You can see the snap tests passing with this change at https://dev.azure.com/certbot/certbot/_build/results?buildId=2797&view=results.
Fixes https://github.com/certbot/certbot/issues/8292.
This uses the same approach that worked well for us in https://github.com/certbot/certbot/pull/7926. I'm sure we could delete more code or refactor things here, but I think we should make the most conservative changes we can to certbot-auto until we can just delete the entire thing.
I ran the full test suite on these changes at https://dev.azure.com/certbot/certbot/_build/results?buildId=2773&view=results and manually tested things on OpenSUSE and it worked as expected. certbot-auto refused to create new installations and refused to update old ones while continuing to allow the old version of Certbot to run.
* Deprecate cb-auto outside of Debian and RHEL.
* Don't deprecate Amazon Linux yet.
Partial fix for #8280
This PR refactors the bash script wrapper for snap (`/certbot.wrapper`) into certbot python codebase. Here are the keypoints of this refactoring:
* the wrapping is applied when `main` function from `certbot._internal.main` is called if environment variable `CERTBOT_SNAPPED` is `True`, which is set during the snap build
* the initial bash script wrapper is removed, simplifying `snap/snapcraft.yaml` by removing the `certbot.wrapper` part
* the dependency to `curl` and `jq` binaries are removed
* the failure during requesting the snapd socket is correctly handled, and displays an informative message in order to correct the situation, as required by #8280
One side note about the modifications done to `app.certbot.command` in `snapcraft.yaml`. Normally calling `bin/certbot` should be sufficient and it is effectively under a normal situation (`core` snap up-to-date). However in the same situation than when the problem occurs in #8280, using `bin/certbot` makes the snap raise an exception about `certbot.main` module that cannot be found.
It seems that when `core` snap is not up-to-date (in Debian for instance with default `snapd` installation), the shebang `/usr/bin/env python3` in the `bin/certbot` wrapper is wrongly resolved to the host Python, instead of the snap Python. It is working as expected if `core` snap is up-to-date. One way to fix that is to keep a bash script wrapper, because in this case, it is the `PATH` value that matters to resolve the Python interpreter, and `PATH` is correctly set up to resolve it from the snap first.
However to keep the simplification provided by the wrapper removal, I prefered to use `bin/python3 $SNAP/bin/certbot` as `command` to explicitly target the correct Python interpreter. Again normally it is not needed because everything is working correctly with a `core` snap up-to-date, but since the root purpose of all of this is to target bad situations, well, it is better to have a snap that is effectively able to start to display the informative message...
* Refactor the bash wrapper for snap execution as Python code into certbot
* Remove wrapper, finalize the python logic
* Organize code
* Improve error handling
* Update command
* Setup basic certbot logging before running the snap prepare logic
* Improve instructions
* Use logging facility
* Handle properly an exception in snap_config
* Use the python script call approach
* Update instructions to keep sync with https://github.com/certbot/website/pull/650
This reverts commit feca125437.
Since this change landed, ARM builds for many of the DNS plugins have failed every night. See https://dev.azure.com/certbot/certbot/_build?definitionId=5 or our public Mattermost channel.
I quickly tried to fix this myself and wasn't trivially able to do so. I tried setting `SNAPCRAFT_PYTHON_VENV_ARGS: --system-site-packages` and adding `python3-wheel` as a build dependency, but it didn't work for some reason. The `python3-wheel` package didn't seem to be installed.
I still suspect something like this is the approach we should take, however, I want to fix the failing tests now so things are no longer broken in `master` and those of us on the Certbot team at EFF stop getting spammed with 54 (!!) emails about failed builds from launchpad every night.
Unfortunately, while I was working on this the queue for ARM machines on Launchpad jumped up to an estimated ~20 hour wait, but I confirmed that this fixes the problem by building on an ARM AMI using the instructions at https://github.com/certbot/certbot/blob/master/tools/snap/README.md#use-testing-and-development. If whoever reviews this would like an ARM machine to test on themselves, please let me know.
* add set -e to all bash instances in deploy-stage.yml
* retry uploading snap if we fail
* Add the rest of the set -e calls for bash in azure while we're here
* use retry based on travis_retry
* add set -e to the script: sections that run on macOS/Linux
* actually don't fail on result
* reset result before running command because bash short circuits or conditionals
* remove inapplicable comment
