* Add generic methods to save some casts, and fix lint
* Update current and oldest pinning
* Fix classes
* Remove some todos thanks to josepy 1.11.0
* Cleanup some useless pylint disable
* Finish complete typing
* Better TypeVar names
* Upgrade pinning and fix some typing errors
* Use protocol
* Fix types in apache
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
The `# self.comment = comment` caught my eye while working on #9071 as well as the intermediate variables, which aren't really needed. As a result, I reformatted the code slightly in those places.
* Remove comment in AugeasCommentNode.__init__
* Replace some intermediate varibles with return-statements in apache augeas parser.
* more clean-up
Fixes https://github.com/certbot/certbot/issues/9058.
The changes to the CI config are equivalent to the ones made in https://github.com/certbot/certbot/pull/8460.
Other than ignoring some warnings raised by botocore, the main additional work that had to be done here was switching away from using `distutils.version.LooseVersion` since the entire `distutils` module was deprecated in Python 3.10. To do that, I took a few different approaches:
* If the version strings being parsed are from Python packages such as Certbot or setuptools, I switched to using [pkg_resources.parse_version](https://setuptools.pypa.io/en/latest/pkg_resources.html#parsing-utilities) from `setuptools`. This functionality has been available since [setuptools 8.0 from 2014](https://setuptools.pypa.io/en/latest/history.html#id865).
* If the version strings being parsed are not from Python packages, I added code equivalent to `distutils.version.LooseVersion` in `certbot.util.parse_loose_version`.
* The code for `CERTBOT_PIP_NO_BINARY` can be completely removed since that variable isn't used or referenced anywhere in this repo.
* add python 3.10 support
* make some version changes
* don't use looseversion in setup.py
* switch to pkg_resources
* deprecate get_strict_version
* fix route53 tests
* remove unused CERTBOT_PIP_NO_BINARY code
* stop using distutils in letstest
* add unit tests
* more changelog entries
As a follow-up to #8971, this PR removes all references to the old Zope interfaces, except the ones used to deprecate them and prepare for their removal.
In the process, some documentation and tests about the `Display` objects are simply removed since they are not relevant anymore given that they are removed from the public API.
* Cleanup some interfaces.IInstaller
* Cleanup IConfig doc
* Allmost complete removal
* Remove useless tests
* Fixes
* More cleanup
* More cleanup
* More cleanup
* Remove a non existent reference
* Better type
* Fix lint
Streamline and reorganize Certbot's CLI output.
This change is a substantial command-line UX overhaul,
based on previous user research. The main goal was to streamline
and clarify output. To see more verbose output, use the -v or -vv flags.
---
* nginx,apache: CLI logging changes
- Add "Successfully deployed ..." message using display_util
- Remove IReporter usage and replace with display_util
- Standardize "... could not find a VirtualHost ..." error
This changes also bumps the version of certbot required by certbot-nginx
and certbot-apache to take use of the new display_util function.
* fix certbot_compatibility_test
since the http plugins now require IDisplay, we need to inject it
* fix dependency version on certbot
* use better asserts
* try fix oldest deps
because certbot 1.10.0 depends on acme>=1.8.0, we need to use
acme==1.8.0 in the -oldest tests
* cli: redesign output of new certificate reporting
Changes the output of run, certonly and certonly --csr. No longer uses
IReporter.
* cli: redesign output of failed authz reporting
* fix problem sorting to be stable between py2 & 3
* add some catch-all error text
* cli: dont use IReporter for EFF donation prompt
* add per-authenticator hints
* pass achalls to auth_hint, write some tests
* exclude static auth hints from coverage
* dont call auth_hint unless derived from .Plugin
* dns fallback hint: dont assume --dns-blah works
--dns-blah won't work for third-party plugins, they need to be specified
using --authenticator dns-blah.
* add code comments about the auth_hint interface
* renew: don't restart the installer for dry-runs
Prevents Certbot from superfluously invoking the installer restart
during dry-run renewals. (This does not affect authenticator restarts).
Additionally removes some CLI output that was reporting the fullchain
path of the renewed certificate.
* update CHANGELOG.md
* cli: redesign output when cert installation failed
- Display a message when certificate installation begins.
- Don't use IReporter, just log errors immediately if restart/rollback
fails.
- Prompt the user with a command to retry the installation process once
they have fixed any underlying problems.
* vary by preconfigured_renewal
and move expiry date to be above the renewal advice
* update code comment
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* update code comment
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* fix lint
* derve cert name from cert_path, if possible
* fix type annotation
* text change in nginx hint
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* print message when restarting server after renewal
* log: print "advice" when exiting with an error
When running in non-quiet mode.
* try fix -oldest lock_test.py
* fix docstring
* s/Restarting/Reloading/ when notifying the user
* fix test name
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* type annotations
* s/using the {} plugin/installer: {}/
* copy: avoid "plugin" where possible
* link to user guide#automated-renewals
when not running with --preconfigured-renewal
* cli: reduce default logging verbosity
* fix lock_test: -vv is needed to see logger.debug
* Change comment in log.py to match the change to default verbosity
* Audit and adjust logging levels in apache module
* Audit and adjust logging levels in nginx module
* Audit, adjust logging levels, and improve logging calls in certbot module
* Fix tests to mock correct methods and classes
* typo in non-preconfigured-renewal message
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* fix test
* revert acme version bump
* catch up to python3 changes
* Revert "revert acme version bump"
This reverts commit fa83d6a51c.
* Change ocsp check error to warning since it's non-fatal
* Update storage_test in parallel with last change
* get rid of leading newline on "Deploying [...]"
* shrink renewal and installation success messages
* print logfile rather than logdir in exit handler
* Decrease logging level to info for idempotent operation where enhancement is already set
* Display cert not yet due for renewal message when renewing and no other action will be taken, and change cert to certificate
* also write to logger so it goes in the log file
* Don't double write to log file; fix main test
* cli: remove trailing newline on new cert reporting
* ignore type error
* revert accidental changes to dependencies
* Pass tests in any timezone by using utcfromtimestamp
* Add changelog entry
* fix nits
* Improve wording of try again message
* minor wording change to changelog
* hooks: send hook stdout to CLI stdout
includes both --manual and --{pre,post,renew} hooks
* update docstrings and remove TODO
* add a pending deprecation on execute_command
* add test coverage for both
* update deprecation text
Co-authored-by: ohemorange <ebportnoy@gmail.com>
Co-authored-by: Alex Zorin <alex@zorin.id.au>
Co-authored-by: alexzorin <alex@zor.io>
This is part of https://github.com/certbot/certbot/issues/8782. I took it on now because the currently pinned version of `pylint` doesn't work with newer versions of `poetry` which I wanted to upgrade as part of https://github.com/certbot/certbot/issues/8787.
To say a bit more about the specific changes in this PR:
* Newer versions of `pylint` complain if `Popen` isn't used as a context manager. Instead of making this change, I switched to using `subprocess.run` which is simpler and [recommended in the Python docs](https://docs.python.org/3/library/subprocess.html#using-the-subprocess-module). I also disabled this check in a few places where no longer using `Popen` would require significant refactoring.
* The deleted code in `certbot/certbot/_internal/renewal.py` is cruft since https://github.com/certbot/certbot/pull/8685.
* The unused argument to `enable_mod` in the Apache plugin is used in some over the override classes that subclass that class.
* unpin pylint and repin dependencies
* disable raise-missing-from
* disable wrong-input-order
* remove unused code
* misc lint fixes
* remove unused import
* various lint fixes
In https://github.com/certbot/certbot/pull/8748#discussion_r605457670 we discussed about changing the dict used to set OS options for Apache configurators into a dedicated object.
* Create _OsOptions class to configure the os specific options of the Apache configurators
* Fix tests
* Clean imports
* Fix naming
* Fix compatibility tests
* Rename a class
* Ensure restart_cmd_alt is set for specific OSes.
* Add docstring
* Fix override
* Fix coverage
This is one of the things that newer versions of `pylint` complains about.
* git grep -l super\( | xargs sed -i 's/super([^)]*)/super()/g'
* fix spacing
Built on top of #8748, this PR reenables mypy strict mode and adds the appropriate corrections to pass the types checks.
* Upgrade mypy
* First step for acme
* Cast for the rescue
* Fixing types for certbot
* Fix typing for certbot-nginx
* Finalize type fixes, configure no optional strict check for mypy in tox
* Align requirements
* Isort
* Pylint
* Protocol for python 3.6
* Use Python 3.9 for mypy, make code compatible with Python 3.8<
* Pylint and mypy
* Pragma no cover
* Pythonic NotImplemented constant
* More type definitions
* Add comments
* Simplify typing logic
* Use vararg tuple
* Relax constraints on mypy
* Add more type
* Do not silence error if target is not defined
* Conditionally import Protocol for type checking only
* Clean up imports
* Add comments
* Align python version linting with mypy and coverage
* Just ignore types in an unused module
* Add comments
* Fix lint
* Work in progress
* Finish type control
* Isort
* Fix pylint
* Fix imports
* Fix cli subparser
* Some fixes
* Coverage
* Remove --no-strict-optional (obviously...)
* Update certbot-apache/certbot_apache/_internal/configurator.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/certbot/_internal/display/completer.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Cleanup dns_google
* Improve lock controls and fix subparser
* Use the expected interfaces
* Fix code
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Fixes#8425
This PR upgrades mypy to the latest version available, 0.812.
Given the advanced type inference capabilities provided by this newer version, this PRs also fixes various type inconsistencies that are now detected. Here are the non obvious changes done to fix types:
* typing in mixins has been solved using `Protocol` classes, as recommended by mypy (https://mypy.readthedocs.io/en/latest/more_types.html#mixin-classes, https://mypy.readthedocs.io/en/stable/protocols.html)
* `cast` when we are playing with `Union` types
This PR also disables the strict optional checks that have been enable by default in recent versions of mypy. Once this PR is merged, I will create an issue to study how these checks can be enabled.
`typing.Protocol` is available only since Python 3.8. To keep compatibility with Python 3.6, I try to import the class `Protocol` from `typing`, and fallback to assign `object` to `Protocol` if that fails. This way the code is working with all versions of Python, but the mypy check can be run only with Python 3.8+ because it needs the protocol feature. As a consequence, tox runs mypy under Python 3.8.
Alternatives are:
* importing `typing_extensions`, that proposes backport of newest typing features to Python 3.6, but this implies to add a dependency to Certbot just to run mypy
* redesign the concerned classes to not use mixins, or use them differently, but this implies to modify the code itself even if there is nothing wrong with it and it is just a matter of instructing mypy to understand in which context the mixins can be used
* ignoring type for these classes with `# type: ignore` but we loose the benefit of mypy for them
* Upgrade mypy
* First step for acme
* Cast for the rescue
* Fixing types for certbot
* Fix typing for certbot-nginx
* Finalize type fixes, configure no optional strict check for mypy in tox
* Align requirements
* Isort
* Pylint
* Protocol for python 3.6
* Use Python 3.9 for mypy, make code compatible with Python 3.8<
* Pylint and mypy
* Pragma no cover
* Pythonic NotImplemented constant
* More type definitions
* Add comments
* Simplify typing logic
* Use vararg tuple
* Relax constraints on mypy
* Add more type
* Do not silence error if target is not defined
* Conditionally import Protocol for type checking only
* Clean up imports
* Add comments
* Align python version linting with mypy and coverage
* Just ignore types in an unused module
* Add comments
* Fix lint
Fixes#8427
This PR converts the Python 2 types hints into Python 3 types annotations. I have used the project https://github.com/ilevkivskyi/com2ann which has been designed for that specific purpose and did that very well.
The only remaining things to do were to fix broken type hints that became wrong code after migration, and to fix lines too long with the new syntax.
* Raw execution of com2ann
* Fixing broken type annotations
* Cleanup imports
* Remove python_version from mypy.ini.
* Fix magic_typing
* Ignore msvcrt usage.
* make mypy happier
* clean up changes
* Add type for reporter queue
* More mypy fixes
* Fix pyrfc3339 str.
* Remove unused import.
* Make certbot.util mypy work in both Pythons
* Fix typo
In #7771, the Apache configurator gained the ability to identify what
version of OpenSSL Apache's ssl_module is linked against. However, the
detection was only functional if the module was built as a DSO (which is
almost always the case).
This commit covers the case where the ssl_module is statically linked
within the Apache binary. It requires the user to specify the path to
the binary (with --apache-bin) and emits a warning if static linking is
detected but no path has been provided.
* Error out in apache installer when mod_ssl is not available
* Update to MisconfigurationError and add/fix tests
* Remove error cases we no longer hit and associated test
* mock out function to have consistent error across machines
* improve changelog message
* only check key in modules list, not value
* Fix dangerous default argument
* Remove unused imports
* Remove unnecessary comprehension
* Use literal syntax to create data structure
* Use literal syntax instead of function calls to create data structure
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* Load apacheconfig dependency, gate behind flag
* Bump apacheconfig dependency to latest version and install dev version of apache for coverage tests
* Move augeasnode_test tests to more generic parsernode_test
* Revert "Move augeasnode_test tests to more generic parsernode_test"
This reverts commit 6bb986ef78.
* Mock AugeasNode into DualNode's place, and run augeasnode tests exclusively on AugeasNode
* Don't calculate coverage for skeleton functions
* clean up helper function in augeasnode_test
Fixes#7350.
This PR changes the parsed modules from a `set` to a `dict`, with the filepath argument as the value. Accordingly, after calling `enable_mod` to enable `ssl_module`, modules now need to be re-parsed, so call `reset_modules`.
* Add mechanism for selecting apache config file, based on work done in #7191.
* Check OpenSSL version
* Remove os imports
* debian override still needs os
* Reformat remaining apache tests with modules dict syntax
* Clean up more apache tests
* Switch from property to method for openssl and add tests for coverage.
* Sometimes the dict location will be None in which case we should in fact return None
* warn thoroughly and consistently in openssl_version function
* update tests for new warnings
* read file as bytes, and factor out the open for testing
* normalize ssl_module_location path to account for being relative to server root
* Use byte literals in a python 2 and 3 compatible way
* string does need to be a literal
* patch builtins open
* add debug, remove space
* Add test to check if OpenSSL detection is working on different systems
* fix relative test location for cwd
* put </IfModule> on its own line in test case
* Revert test file to status in master.
* Call augeas load before reparsing modules to pick up the changes
* fix grep, tail, and mod_ssl location on centos
* strip the trailing whitespace from fedora
* just use LooseVersion in test
* call apache2ctl on debian systems
* Use sudo for apache2ctl command
* add check to make sure we're getting a version
* Add boolean so we don't warn on debian/ubuntu before trying to enable mod_ssl
* Reduce warnings while testing by setting mock _openssl_version.
* Make sure we're not throwing away any unwritten changes to the config
* test last warning case for coverage
* text changes for clarity
As pylint is evolving, it improves its accuracy, and several pylint error suppression (`# pylint: disable=ERROR) added in certbot codebase months or years ago are not needed anymore to make it happy.
There is a (disabled by default) pylint error to detect the useless suppressions (pylint-ception: `useless-suppression`). It is not working perfectly (it has also false-positives ...) but it is a good start to clean the codebase.
This PR removes several of these useless suppressions as detected by the current pylint version we use.
* Remove useless suppress
* Remove useless lines
Part of #7550
This PR makes appropriate corrections to run pylint on Python 3.
Why not keeping the dependencies unchanged and just run pylint on Python 3?
Because the old version of pylint breaks horribly on Python 3 because of unsupported version of astroid.
Why updating pylint + astroid to the latest version ?
Because this version only fixes some internal errors occuring during the lint of Certbot code, and is also ready to run gracefully on Python 3.8.
Why upgrading mypy ?
Because the old version does not support the new version of astroid required to run pylint correctly.
Why not upgrading mypy to its latest version ?
Because this latest version includes a new typshed version, that adds a lot of new type definitions, and brings dozens of new errors on the Certbot codebase. I would like to fix that in a future PR.
That said so, the work has been to find the correct set of new dependency versions, then configure pylint for sane configuration errors in our situation, disable irrelevant lintings errors, then fixing (or ignoring for good reason) the remaining mypy errors.
I also made PyLint and MyPy checks run correctly on Windows.
* Start configuration
* Reconfigure travis
* Suspend a check specific to python 3. Start fixing code.
* Repair call_args
* Fix return + elif lints
* Reconfigure development to run mainly on python3
* Remove incompatible Python 3.4 jobs
* Suspend pylint in some assertions
* Remove pylint in dev
* Take first mypy that supports typed-ast>=1.4.0 to limit the migration path
* Various return + else lint errors
* Find a set of deps that is working with current mypy version
* Update local oldest requirements
* Remove all current pylint errors
* Rebuild letsencrypt-auto
* Update mypy to fix pylint with new astroid version, and fix mypy issues
* Explain type: ignore
* Reconfigure tox, fix none path
* Simplify pinning
* Remove useless directive
* Remove debugging code
* Remove continue
* Update requirements
* Disable unsubscriptable-object check
* Disable one check, enabling two more
* Plug certbot dev version for oldest requirements
* Remove useless disable directives
* Remove useless no-member disable
* Remove no-else-* checks. Use elif in symetric branches.
* Add back assertion
* Add new line
* Remove unused pylint disable
* Remove other pylint disable
