1.10.0 was a bad release and this breaks our oldest Boulder tests.
I bumped the version to 1.10.0 in #8852 to get access to a new public display_util API, but that was the release with the broken deprecation of `--manual-public-ip-logging-ok`. So let's bump it to 1.10.1.
Streamline and reorganize Certbot's CLI output.
This change is a substantial command-line UX overhaul,
based on previous user research. The main goal was to streamline
and clarify output. To see more verbose output, use the -v or -vv flags.
---
* nginx,apache: CLI logging changes
- Add "Successfully deployed ..." message using display_util
- Remove IReporter usage and replace with display_util
- Standardize "... could not find a VirtualHost ..." error
This changes also bumps the version of certbot required by certbot-nginx
and certbot-apache to take use of the new display_util function.
* fix certbot_compatibility_test
since the http plugins now require IDisplay, we need to inject it
* fix dependency version on certbot
* use better asserts
* try fix oldest deps
because certbot 1.10.0 depends on acme>=1.8.0, we need to use
acme==1.8.0 in the -oldest tests
* cli: redesign output of new certificate reporting
Changes the output of run, certonly and certonly --csr. No longer uses
IReporter.
* cli: redesign output of failed authz reporting
* fix problem sorting to be stable between py2 & 3
* add some catch-all error text
* cli: dont use IReporter for EFF donation prompt
* add per-authenticator hints
* pass achalls to auth_hint, write some tests
* exclude static auth hints from coverage
* dont call auth_hint unless derived from .Plugin
* dns fallback hint: dont assume --dns-blah works
--dns-blah won't work for third-party plugins, they need to be specified
using --authenticator dns-blah.
* add code comments about the auth_hint interface
* renew: don't restart the installer for dry-runs
Prevents Certbot from superfluously invoking the installer restart
during dry-run renewals. (This does not affect authenticator restarts).
Additionally removes some CLI output that was reporting the fullchain
path of the renewed certificate.
* update CHANGELOG.md
* cli: redesign output when cert installation failed
- Display a message when certificate installation begins.
- Don't use IReporter, just log errors immediately if restart/rollback
fails.
- Prompt the user with a command to retry the installation process once
they have fixed any underlying problems.
* vary by preconfigured_renewal
and move expiry date to be above the renewal advice
* update code comment
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* update code comment
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* fix lint
* derve cert name from cert_path, if possible
* fix type annotation
* text change in nginx hint
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* print message when restarting server after renewal
* log: print "advice" when exiting with an error
When running in non-quiet mode.
* try fix -oldest lock_test.py
* fix docstring
* s/Restarting/Reloading/ when notifying the user
* fix test name
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* type annotations
* s/using the {} plugin/installer: {}/
* copy: avoid "plugin" where possible
* link to user guide#automated-renewals
when not running with --preconfigured-renewal
* cli: reduce default logging verbosity
* fix lock_test: -vv is needed to see logger.debug
* Change comment in log.py to match the change to default verbosity
* Audit and adjust logging levels in apache module
* Audit and adjust logging levels in nginx module
* Audit, adjust logging levels, and improve logging calls in certbot module
* Fix tests to mock correct methods and classes
* typo in non-preconfigured-renewal message
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* fix test
* revert acme version bump
* catch up to python3 changes
* Revert "revert acme version bump"
This reverts commit fa83d6a51c.
* Change ocsp check error to warning since it's non-fatal
* Update storage_test in parallel with last change
* get rid of leading newline on "Deploying [...]"
* shrink renewal and installation success messages
* print logfile rather than logdir in exit handler
* Decrease logging level to info for idempotent operation where enhancement is already set
* Display cert not yet due for renewal message when renewing and no other action will be taken, and change cert to certificate
* also write to logger so it goes in the log file
* Don't double write to log file; fix main test
* cli: remove trailing newline on new cert reporting
* ignore type error
* revert accidental changes to dependencies
* Pass tests in any timezone by using utcfromtimestamp
* Add changelog entry
* fix nits
* Improve wording of try again message
* minor wording change to changelog
* hooks: send hook stdout to CLI stdout
includes both --manual and --{pre,post,renew} hooks
* update docstrings and remove TODO
* add a pending deprecation on execute_command
* add test coverage for both
* update deprecation text
Co-authored-by: ohemorange <ebportnoy@gmail.com>
Co-authored-by: Alex Zorin <alex@zorin.id.au>
Co-authored-by: alexzorin <alex@zor.io>
This is part of https://github.com/certbot/certbot/issues/8782. I took it on now because the currently pinned version of `pylint` doesn't work with newer versions of `poetry` which I wanted to upgrade as part of https://github.com/certbot/certbot/issues/8787.
To say a bit more about the specific changes in this PR:
* Newer versions of `pylint` complain if `Popen` isn't used as a context manager. Instead of making this change, I switched to using `subprocess.run` which is simpler and [recommended in the Python docs](https://docs.python.org/3/library/subprocess.html#using-the-subprocess-module). I also disabled this check in a few places where no longer using `Popen` would require significant refactoring.
* The deleted code in `certbot/certbot/_internal/renewal.py` is cruft since https://github.com/certbot/certbot/pull/8685.
* The unused argument to `enable_mod` in the Apache plugin is used in some over the override classes that subclass that class.
* unpin pylint and repin dependencies
* disable raise-missing-from
* disable wrong-input-order
* remove unused code
* misc lint fixes
* remove unused import
* various lint fixes
Fixes#8824
This PR makes the installer first delete (if exist) the previous `pkg` directory in the Certbot installation in order to avoid dependencies conflicts when a new version of Certbot (with new versions of dependencies) is intaller other an existing one.
I took the simplest approach here, which is to delete specifically the directories known to create conflicts, instead of more complex approaches that involve to factor in some way the complete uninstaller logic. This is because the complexity added without a clear improvement does not worth it in my opinion. More specifically:
* factorizing in some way the uninstaller section in the NSIS template make the installer use any potential new logic of a new installation of Certbot instead of the one applying for the current installation, and may create unexpected errors during installation or at runtime
* calling the existing `uninstaller.exe` would be better, but I could not find a proper way to let NSIS wait for the actual end of the uninstall logic, and again may create unexpected errors during installation or at runtime
* Cleanup Certbot pkg dir before installing to avoid dependencies conflicts
* Add a changelog
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Apache test farm tests started failing last night due to a change in pyenv. See https://dev.azure.com/certbot/certbot/_build/results?buildId=3948&view=logs&j=f67c2a39-2c4f-5190-915f-6f32a7a4306f&t=96f0f394-f513-5158-f5e7-a26e55aeadbf&l=26943.
I managed to fix that in d94f20f8b7, however, the OSes the tests were failing on were Debian 9 and Ubuntu 16.04. [Debian 9 reached its end-of-life in July 2020](https://wiki.debian.org/DebianReleases) and [Ubuntu 16.04 reached its end of standard support in April 2021](https://wiki.ubuntu.com/Releases). As shown at the same links, Debian 9 still has support from the LTS team and Ubuntu 16.04 has ESM support. Do we still want to support either of these OSes?
If so, we can use the commit I linked in the first sentence of the last paragraph, but I think supporting the OSes through their standard support is good enough. The Certbot team has enough on their plate and especially when the OSes are so old that we can't even use their packaged version of Python anymore which complicates our tests, I think we can just drop support and move on.
I don't have a strong opinion here though so if someone else does, let me know what you'd like to see or make the PR yourself based on the changes in my linked commit and I'll merge it.
You can see the tests passing with this change at https://dev.azure.com/certbot/certbot/_build/results?buildId=3955&view=results.
* Remove apache tests on old OSes
* remove unused pyenv code
I think we should use our `pip_install*` scripts wherever we can and I'm not quite sure yet if I'd call `repoze.sphinx.autointerface` unmaintained.
* use pip_install_editable
* update sphinx comment
Since Saturday the CI pipeline is failing due to several Sphinx errors. See https://dev.azure.com/certbot/certbot/_build/results?buildId=3928&view=logs&j=d74e04fe-9740-597d-e9fa-1d0400037dfd&t=dde413a4-f24c-59a0-9684-e33d79f9aa02
First, the build of certbot-dns-google is failing because of a particular configuration. It seems that this configuration has been written here to activate the support of the RST instruction `.. code-block:: json` in documentation. However, it does not seem to be necessary for a similar situation in certbot-dns-route53 documentation. So let's try to remove it and fix the Sphinx builds.
Second, Sphinx builds were not pinning dependencies, so Sphinx 4.x (that has been released yesterday) started to be used in the pipeline. Sadly this new version is not compatible with the plugin `repoze.sphinx.autointerface`, used to extract documentation from `zope.interface`. So I fixed the pinning and also explicitly pin Sphinx to 3.5.x for now.
Technically speaking the second action is sufficient to fix the first error, but I keep the dedicated solution because it improves the documentation in my opinion.
This situation could be fixed by not requiring `repoze.sphinx.autointerface`, but this is possible only if we remove `zope.interface` from Certbot. Luckily I started the work few days ago ;).
* Remove explicit lexer call in certbot-dns-google doc builds.
* Write a valid JSON file in the documentation
* Apply constraints to sphinx build environments
* Pin Sphinx to 3.5.4
* Update dependencies
* Pin traitlets
Fixes https://github.com/certbot/certbot/issues/8832.
[These instructions are creating confusion among users](https://github.com/certbot/certbot/issues/8832) and [frustration among packagers](https://pagure.io/fesco/issue/2570) for whom the warning at the top of the OS packaging section doesn't apply. Because of this, I think we should remove them in favor of our instruction generator and snap/docker/pip instructions.
I also told Fedora packagers that we could probably do this in response to them continuing to improve their Certbot packages which they've done through things like the renewal timer that is now enabled by default.
Fixes https://github.com/certbot/certbot/issues/8781.
This PR makes our test farm tests into a normal package so it and its dependencies can be tracked and installed like our other packages.
Other noteworthy changes in this PR:
* Rather than continuing to place logs in your CWD, they're placed in a temporary directory that is printed to the terminal.
* `tests/letstest/auto_targets.yaml` was deleted rather than renamed because the file is no longer used.
* make a letstest package
* remove deleted deps
* fix letstest install
* add __init__.py
* call main
* Explicitly mention activating venv
* rerename file
* fix version.py path
* clarify "this"
* Use >= instead of caret requirement
* Update assertTrue/False to Python 3 precise asserts
* Fix test failures
* Fix test failures
* More replacements
* Update to Python 3 asserts in acme-module
* Fix Windows test failure
* Fix failures
* Fix test failure
* More replacements
* Don't include the semgrep rules
* Fix test failure
* Move version.py to tests/letstest since it's used by test_sdists.sh
* Delete unused components of certbot-auto
* Remove test_leauto_upgrades.sh and references to it
* Remove test_letsencrypt_auto_certonly_standalone.sh and references to it
* Remove outstanding references to certbot-auto
* Remove references to letsencrypt-auto
* find certbot in the correct directory
* delete letsencrypt-auto-source line from .isort.cfg since that directory no longer contains any python code
* remove (-auto) from certbot(-auto)
* delete line from test
* Improve style for version.py
Fixes#8802.
Also removed the unused `kgs` cruft while I was here, since it's leftover from the [initial release commit](3c08b512c3) and I'm pretty sure we don't use that anymore.
* Expand manual DNS challenge instructions
* Less jargon
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* Less is more
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* Make more clear where to look at Googles Toolbox
* Reshuffle text
* Show verify instructions only on last dns-01 challenge
* Swap domain and value
* Remove '(also)'
* Fix DNS verify message for mixed challenge types
* Add a lengthy comment about why there's a full stop after `{domain}`
* Typo
Co-authored-by: ohemorange <ebportnoy@gmail.com>
In https://github.com/certbot/certbot/pull/8748#discussion_r605457670 we discussed about changing the dict used to set OS options for Apache configurators into a dedicated object.
* Create _OsOptions class to configure the os specific options of the Apache configurators
* Fix tests
* Clean imports
* Fix naming
* Fix compatibility tests
* Rename a class
* Ensure restart_cmd_alt is set for specific OSes.
* Add docstring
* Fix override
* Fix coverage
This is one of the things that newer versions of `pylint` complains about.
* git grep -l super\( | xargs sed -i 's/super([^)]*)/super()/g'
* fix spacing
I think this PR improves tools/snap/build_remote.py's output in a number of ways such as:
* Logs of snap builds were being deleted because they weren't being copied out of the temporary directory added in https://github.com/certbot/certbot/pull/8719.
* The lock should now always be acquired before printing output when multiple processes are running which helps prevent processes mixing their output with each other.
* Output is never buffered which ensures that repeated calls to `print` from the same process while it holds the output lock is kept together.
* The case where we printed output about the "chroot problem" and stopped retrying the build has been deleted because with the fix in https://github.com/certbot/certbot/pull/8719, we should be able to recover in this case.
* If the build failed for any reason, we dump as much output about the problem as we can. I think most times we won't need to read this output, but I personally prefer it being there in case we want it for some reason. Due to this change, I also simplified `_build_snap` and `_dump_results` a bit since `_build_snap` handles printing logs as needed.
* print more output
* lock when printing output
* clarify purpose of lock
* preserve logfiles
* python better
* consistently flush output
* remove workspaces dict
* rename variable
* remove unused variable
* don't use all which exits early
* fix typo
Built on top of #8748, this PR reenables mypy strict mode and adds the appropriate corrections to pass the types checks.
* Upgrade mypy
* First step for acme
* Cast for the rescue
* Fixing types for certbot
* Fix typing for certbot-nginx
* Finalize type fixes, configure no optional strict check for mypy in tox
* Align requirements
* Isort
* Pylint
* Protocol for python 3.6
* Use Python 3.9 for mypy, make code compatible with Python 3.8<
* Pylint and mypy
* Pragma no cover
* Pythonic NotImplemented constant
* More type definitions
* Add comments
* Simplify typing logic
* Use vararg tuple
* Relax constraints on mypy
* Add more type
* Do not silence error if target is not defined
* Conditionally import Protocol for type checking only
* Clean up imports
* Add comments
* Align python version linting with mypy and coverage
* Just ignore types in an unused module
* Add comments
* Fix lint
* Work in progress
* Finish type control
* Isort
* Fix pylint
* Fix imports
* Fix cli subparser
* Some fixes
* Coverage
* Remove --no-strict-optional (obviously...)
* Update certbot-apache/certbot_apache/_internal/configurator.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/certbot/_internal/display/completer.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Cleanup dns_google
* Improve lock controls and fix subparser
* Use the expected interfaces
* Fix code
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Fixes#8773
I took option 2 from the issue mentionned above (importing `typing-extensions` on dev dependencies) to avoid modifying certbot runtime requirements given that what needs to be added is useful for mypy only.
I did not change the Python version used to execute the linting and mypy on the standard tests, given that the tox `docker_dev` target already checks if the development environment is working for Python < 3.8.
