1
0
mirror of https://github.com/certbot/certbot.git synced 2026-01-26 07:41:33 +03:00
Commit Graph

311 Commits

Author SHA1 Message Date
Adrien Ferrand
dac0b2c187 Typed jose fields (#9073)
* Add generic methods to save some casts, and fix lint

* Update current and oldest pinning

* Fix classes

* Remove some todos thanks to josepy 1.11.0

* Cleanup some useless pylint disable

* Finish complete typing

* Better TypeVar names

* Upgrade pinning and fix some typing errors

* Use protocol

* Fix types in apache

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2022-01-24 15:16:19 -08:00
alexzorin
7198f43008 apache: expose aug_save errors in the debug log (#9169)
Fixes #9168.

* apache: expose aug_save errors in the debug log

* logger arguments wrong way around

* log formatting

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2022-01-24 09:33:11 -08:00
Mads Jensen
7d9e9a4900 Add typing to certbot.apache (#9071)
* Add typing to certbot.apache

Co-authored-by: Adrien Ferrand <ferrand.ad@gmail.com>
2022-01-21 10:15:48 +01:00
Adrien Ferrand
16aad35d31 Fully type certbot-nginx module (#9124)
* Work in progress

* Fix type

* Work in progress

* Work in progress

* Work in progress

* Work in progress

* Work in progress

* Oups.

* Fix typing in UnspacedList

* Fix logic

* Finish typing

* List certbot-nginx as fully typed in tox

* Fix lint

* Fix checks

* Organize imports

* Fix typing for Python 3.6

* Fix checks

* Fix lint

* Update certbot-nginx/certbot_nginx/_internal/configurator.py

Co-authored-by: alexzorin <alex@zor.io>

* Update certbot-nginx/certbot_nginx/_internal/configurator.py

Co-authored-by: alexzorin <alex@zor.io>

* Fix signature of deploy_cert regarding the installer interface

* Update certbot-nginx/certbot_nginx/_internal/obj.py

Co-authored-by: alexzorin <alex@zor.io>

* Fix types

* Update certbot-nginx/certbot_nginx/_internal/parser.py

Co-authored-by: alexzorin <alex@zor.io>

* Precise type

* Precise _coerce possible inputs/outputs

* Fix type

* Update certbot-nginx/certbot_nginx/_internal/http_01.py

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* Fix type

* Remove an undesirable implementation.

* Fix type

Co-authored-by: alexzorin <alex@zor.io>
Co-authored-by: ohemorange <ebportnoy@gmail.com>
2022-01-12 16:36:51 -08:00
Mads Jensen
30b066f082 Remove outdated pylint comments (#9167)
* Remove outdated pylint: disable=unused-import annotations.

* remove # pylint: disable=ungrouped-imports annotations.

* Remove single pylint: disable = unused-argument in DeleteIfAppropriateTest.test_opt_in_deletion.
2022-01-09 22:51:06 +01:00
Mads Jensen
eeca208c8f Various clean-ups in certbot-apache. Use f-strings. (#9132)
* Various clean-ups in certbot-apache. Use f-strings.

* Smaller tweaks
2022-01-02 00:27:47 +01:00
Mads Jensen
402f18e039 Apache augeas clean up (#9114)
The `# self.comment = comment` caught my eye while working on #9071 as well as the intermediate variables, which aren't really needed. As a result, I reformatted the code slightly in those places.

* Remove comment in AugeasCommentNode.__init__

* Replace some intermediate varibles with return-statements in apache augeas parser.

* more clean-up
2021-12-02 08:45:16 -07:00
Adrien Ferrand
19147e1b8c Add type annotations to the certbot package (part 2) (#9085)
* Extract from #9084

* Cast/ignore types during the transition

* Clean up

* Fix assertion

* Update certbot/certbot/display/ops.py

Co-authored-by: alexzorin <alex@zor.io>

* Use sequence

* Improve documentation of "default" in display

* Fix contract

* Fix types

* Fix type

* Fix type

* Update certbot/certbot/display/ops.py

Co-authored-by: alexzorin <alex@zor.io>

Co-authored-by: alexzorin <alex@zor.io>
2021-11-24 18:33:09 +11:00
Brad Warren
d1821b3ad7 Pin back setuptools-rust (#9112)
* pin back setuptools-rust

* make pylint happy

This was taken from https://github.com/certbot/certbot/pull/9073.

* pin back josepy

* Apply lint's code style suggestions

* fix lint again

Co-authored-by: Erica Portnoy <ebportnoy@gmail.com>
2021-11-23 12:35:49 -08:00
Adrien Ferrand
d20e42562c Add type annotations to the certbot package (part 1) (#9084)
* Extract from #9084

* Cast/ignore types during the transition

* Fix after review

* Fix lint
2021-11-12 14:27:46 +11:00
Brad Warren
e8265dbf9c Add Python 3.10 support and tests (#9077)
Fixes https://github.com/certbot/certbot/issues/9058.

The changes to the CI config are equivalent to the ones made in https://github.com/certbot/certbot/pull/8460.

Other than ignoring some warnings raised by botocore, the main additional work that had to be done here was switching away from using `distutils.version.LooseVersion` since the entire `distutils` module was deprecated in Python 3.10. To do that, I took a few different approaches:

* If the version strings being parsed are from Python packages such as Certbot or setuptools, I switched to using [pkg_resources.parse_version](https://setuptools.pypa.io/en/latest/pkg_resources.html#parsing-utilities) from `setuptools`. This functionality has been available since [setuptools 8.0 from 2014](https://setuptools.pypa.io/en/latest/history.html#id865).
* If the version strings being parsed are not from Python packages, I added code equivalent to `distutils.version.LooseVersion` in `certbot.util.parse_loose_version`.
* The code for `CERTBOT_PIP_NO_BINARY` can be completely removed since that variable isn't used or referenced anywhere in this repo.

* add python 3.10 support

* make some version changes

* don't use looseversion in setup.py

* switch to pkg_resources

* deprecate get_strict_version

* fix route53 tests

* remove unused CERTBOT_PIP_NO_BINARY code

* stop using distutils in letstest

* add unit tests

* more changelog entries
2021-11-08 15:55:32 -08:00
Adrien Ferrand
a0f22d21ce Add type annotations to the acme project (#9036)
* Start more types

* Second run

* Work in progress

* Types in all acme module

* Various fixes

* Various fixes

* Final fixes

* Disallow untyped defs for acme project

* Fix coverage

* Remote unecessary type ignore

* Use Mapping instead of Dict as input whenever it is possible

* Update acme/acme/client.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update acme/acme/client.py

Co-authored-by: alexzorin <alex@zor.io>

* Various fixes

* Fix code

* Fix code

* Update acme/acme/client.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update acme/acme/challenges.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update acme/acme/client.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Fix deactivate_registration and simplify signature of update_registration

* Do not leak personal data during account deactivation

* Clean more Dicts

* New fix to not leak contact field in the account deactivation payload.

* Add ignore for python 3.6 type check

* Revert "Add ignore for python 3.6 type check"

This reverts commit da7338137b.

* Let's find a smarter way than "type: ignore"

* Update certbot/certbot/_internal/account.py

Co-authored-by: alexzorin <alex@zor.io>

* Fix an annotation

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Co-authored-by: alexzorin <alex@zor.io>
2021-10-25 09:43:21 +11:00
Stefan Weil
0d4f92fa81 Fix some typos (found by codespell) (#9017)
* Fix some typos (found by codespell)

Signed-off-by: Stefan Weil <sw@weilnetz.de>

* Remove typo fixes for some files which should not be modified

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2021-09-03 06:43:13 +10:00
Adrien Ferrand
143ea15253 Remove all non essential references to the old Zope interfaces (#8988)
As a follow-up to #8971, this PR removes all references to the old Zope interfaces, except the ones used to deprecate them and prepare for their removal.

In the process, some documentation and tests about the `Display` objects are simply removed since they are not relevant anymore given that they are removed from the public API.

* Cleanup some interfaces.IInstaller

* Cleanup IConfig doc

* Allmost complete removal

* Remove useless tests

* Fixes

* More cleanup

* More cleanup

* More cleanup

* Remove a non existent reference

* Better type

* Fix lint
2021-08-17 14:51:26 -07:00
Yaroslav Halchenko
5e87aee968 BF: apache cfg parsing - relax assumption that value cannot contain = (#8930)
* BF: apache cfg parsing - relax assumption that value cannot contain =

* Remove failing test_update_runtime_vars_bad_output

* Add test Define statements: with = in value, and an empty value

* update CHANGELOG

Co-authored-by: Alex Zorin <alex@zorin.id.au>
2021-08-13 07:57:24 +10:00
Adrien Ferrand
979e21dcbf Reimplement Certbot zope.interfaces into abstract base classes (#8950)
* Implement certbot services

* Various fixes

* Local oldest requirements

* Clean imports

* Add unit tests for certbot.services

* Clean code

* Protect against nullity of global services

* Fix CLI

* Fix tests

* Consistent test behavior

* Define new ABC classes

* Reimplement services with new ABC classes

* Adapt plugins discovery and selection

* Remove zope interfaces from plugins

* Re-enable delegation for simplicity

* Fix interfaces declaration

* Remove interface implementer

* Interfaces ordering

* Extract zope logic from discovery

* Cleanup imports

* Fixing tests

* Fix main_test

* Finish certbot unit tests

* Fix lint

* Various fixes thanks to mypy

* Fix lint

* Order imports

* Various fixes

* Clean code

* Remove reporter service, migrate display service in certbot.display.util.

* Fix test

* Fix apache compatibility test

* Fix oldest test

* Setup certbot.display.service module

* Reintegrate in util

* Fix imports

* Fix tests and documentation

* Refactor

* Cleanup

* Cleanup

* Clean imports

* Add unit tests

* Borrow sphinx build fix from #8863

* Align zope interfaces on ABC

* Various fixes

* Fix type

* Fix type

* Some cleanup

* Fix lint

* Update certbot/certbot/_internal/configuration.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/_internal/configuration.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Fix imports

* Fix Config contract (accounts_dir property)

* Remove unnecessary interface

* Set NamespaceConfig public, remove Config interface

* Remove Display ABC and implementation of IDisplay

* Clean lint

* Cleanup old decorators

* Contract on plugin constructor only

* Update certbot/certbot/tests/util.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/configuration.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/interfaces.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Some corrections

* Add changelog

* Fix --authenticators and --installers flags on plugins subcommand

* Fix multiheritance on the interface Plugin

* Update certbot/certbot/_internal/plugins/manual.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/_internal/plugins/disco.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Add warnings in logger also

* Add deprecation warnings also when plugins are verified.

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2021-07-29 13:45:29 -07:00
Adrien Ferrand
10eecf9c97 Deprecate zope.component in favor of an direct calls to functions from certbot.display.util module (#8835)
* Implement certbot services

* Various fixes

* Local oldest requirements

* Clean imports

* Add unit tests for certbot.services

* Clean code

* Protect against nullity of global services

* Fix CLI

* Fix tests

* Consistent test behavior

* Various fixes

* Clean code

* Remove reporter service, migrate display service in certbot.display.util.

* Fix test

* Fix apache compatibility test

* Fix oldest test

* Setup certbot.display.service module

* Reintegrate in util

* Fix imports

* Fix tests and documentation

* Refactor

* Cleanup

* Cleanup

* Clean imports

* Add unit tests

* Borrow sphinx build fix from #8863

* Fix type

* Add comment

* Do not reuse existing display service, which never exist at that time

* Make get_display() private

* Fix lint

* Make display internal

* Fix circular dependencies

* Fixing circular dependencies

* Rename patch methods and update docstring

* Update deprecation messages

* Update certbot/certbot/_internal/display/obj.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/tests/util.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/tests/util.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/tests/util.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/tests/util.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Add links

* Avoid relying on internal certbot packages from certbot-apache

* Keep same behavior for patch_get_utility*

* Better diff

* Add changelog

* Update certbot/certbot/tests/util.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2021-07-19 17:09:06 -07:00
alexzorin
a105b587ac apache: fix crash when authenticating empty vhosts (#8941)
Fixes #8940.
2021-07-15 11:12:14 -07:00
alexzorin
8e29063ba7 pylint: upgrade pinned verson and fix new lints (#8936)
While bumping pinned packages in #8928, we came across a new version of pylint (2.9.3). Upgrading to this version requires some changes to Certbot's code, which is what this change is about.

* pylint: upgrade pinned verson and fix new lints

* maxsplit should be 1, not -1, for rsplit
2021-07-15 11:03:39 -07:00
alexzorin
1e114b4ef8 apache: configure nameless vhosts during auth (#8898)
In the apache2 package on Debian-based distros, the default
000-default.conf virtual host does not include a ServerName.

Depending on the FQDN hostname of the machine and DNS setup, Apache
assigns a name to this unnamed vhost at runtime. As a result, the
Apache config end up with vhosts that have duplicative names.

Previously, Certbot did not identify that the nameless vhost could be
a match for the requested identifier, which would, depending on
configuration load order, cause the authenticator to fail.

This change causes Certbot to include all unnamed vhosts on top of
matched vhosts, during authentication. If no vhosts matched, the
existing behavior remains the same.

* apache: configure nameless vhosts during auth

* vhost is only unnamed if ServerName is not set

* also fix test to only match ServerName

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2021-06-21 07:18:29 -04:00
kartikynwa
d3555623ba certbot-apache: Add Void Linux overrides (#8891)
* certbot-apache: Add Void Linux overrides

* certbot-apache: Correct distro name to Void Linux
2021-06-12 17:02:16 +10:00
ohemorange
6f27c32db1 Command-line UX overhaul (#8852)
Streamline and reorganize Certbot's CLI output.

This change is a substantial command-line UX overhaul,
based on previous user research. The main goal was to streamline
and clarify output. To see more verbose output, use the -v or -vv flags.

---

* nginx,apache: CLI logging changes

- Add "Successfully deployed ..." message using display_util
- Remove IReporter usage and replace with display_util
- Standardize "... could not find a VirtualHost ..." error

This changes also bumps the version of certbot required by certbot-nginx
and certbot-apache to take use of the new display_util function.

* fix certbot_compatibility_test

since the http plugins now require IDisplay, we need to inject it

* fix dependency version on certbot

* use better asserts

* try fix oldest deps

because certbot 1.10.0 depends on acme>=1.8.0, we need to use
acme==1.8.0 in the -oldest tests

* cli: redesign output of new certificate reporting

Changes the output of run, certonly and certonly --csr. No longer uses
IReporter.

* cli: redesign output of failed authz reporting

* fix problem sorting to be stable between py2 & 3

* add some catch-all error text

* cli: dont use IReporter for EFF donation prompt

* add per-authenticator hints

* pass achalls to auth_hint, write some tests

* exclude static auth hints from coverage

* dont call auth_hint unless derived from .Plugin

* dns fallback hint: dont assume --dns-blah works

--dns-blah won't work for third-party plugins, they need to be specified
using --authenticator dns-blah.

* add code comments about the auth_hint interface

* renew: don't restart the installer for dry-runs

Prevents Certbot from superfluously invoking the installer restart
during dry-run renewals. (This does not affect authenticator restarts).

Additionally removes some CLI output that was reporting the fullchain
path of the renewed certificate.

* update CHANGELOG.md

* cli: redesign output when cert installation failed

- Display a message when certificate installation begins.
- Don't use IReporter, just log errors immediately if restart/rollback
  fails.
- Prompt the user with a command to retry the installation process once
  they have fixed any underlying problems.

* vary by preconfigured_renewal

and move expiry date to be above the renewal advice

* update code comment

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* update code comment

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* fix lint

* derve cert name from cert_path, if possible

* fix type annotation

* text change in nginx hint

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* print message when restarting server after renewal

* log: print "advice" when exiting with an error

When running in non-quiet mode.

* try fix -oldest lock_test.py

* fix docstring

* s/Restarting/Reloading/ when notifying the user

* fix test name

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* type annotations

* s/using the {} plugin/installer: {}/

* copy: avoid "plugin" where possible

* link to user guide#automated-renewals

when not running with --preconfigured-renewal

* cli: reduce default logging verbosity

* fix lock_test: -vv is needed to see logger.debug

* Change comment in log.py to match the change to default verbosity

* Audit and adjust logging levels in apache module

* Audit and adjust logging levels in nginx module

* Audit, adjust logging levels, and improve logging calls in certbot module

* Fix tests to mock correct methods and classes

* typo in non-preconfigured-renewal message

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* fix test

* revert acme version bump

* catch up to python3 changes

* Revert "revert acme version bump"

This reverts commit fa83d6a51c.

* Change ocsp check error to warning since it's non-fatal

* Update storage_test in parallel with last change

* get rid of leading newline on "Deploying [...]"

* shrink renewal and installation success messages

* print logfile rather than logdir in exit handler

* Decrease logging level to info for idempotent operation where enhancement is already set

* Display cert not yet due for renewal message when renewing and no other action will be taken, and change cert to certificate

* also write to logger so it goes in the log file

* Don't double write to log file; fix main test

* cli: remove trailing newline on new cert reporting

* ignore type error

* revert accidental changes to dependencies

* Pass tests in any timezone by using utcfromtimestamp

* Add changelog entry

* fix nits

* Improve wording of try again message

* minor wording change to changelog

* hooks: send hook stdout to CLI stdout

includes both --manual and --{pre,post,renew} hooks

* update docstrings and remove TODO

* add a pending deprecation on execute_command

* add test coverage for both

* update deprecation text

Co-authored-by: ohemorange <ebportnoy@gmail.com>

Co-authored-by: Alex Zorin <alex@zorin.id.au>
Co-authored-by: alexzorin <alex@zor.io>
2021-05-25 10:47:39 +10:00
Brad Warren
315ddb247f Upgrade pylint (#8855)
This is part of https://github.com/certbot/certbot/issues/8782. I took it on now because the currently pinned version of `pylint` doesn't work with newer versions of `poetry` which I wanted to upgrade as part of https://github.com/certbot/certbot/issues/8787.

To say a bit more about the specific changes in this PR:

* Newer versions of `pylint` complain if `Popen` isn't used as a context manager. Instead of making this change, I switched to using `subprocess.run` which is simpler and [recommended in the Python docs](https://docs.python.org/3/library/subprocess.html#using-the-subprocess-module). I also disabled this check in a few places where no longer using `Popen` would require significant refactoring.
* The deleted code in `certbot/certbot/_internal/renewal.py` is cruft since https://github.com/certbot/certbot/pull/8685.
* The unused argument to `enable_mod` in the Apache plugin is used in some over the override classes that subclass that class.

* unpin pylint and repin dependencies

* disable raise-missing-from

* disable wrong-input-order

* remove unused code

* misc lint fixes

* remove unused import

* various lint fixes
2021-05-24 10:02:55 -07:00
Adrien Ferrand
0dbe17bbd4 Define OS options by a dedicated object in Apache configurator (#8778)
In https://github.com/certbot/certbot/pull/8748#discussion_r605457670 we discussed about changing the dict used to set OS options for Apache configurators into a dedicated object.

* Create _OsOptions class to configure the os specific options of the Apache configurators

* Fix tests

* Clean imports

* Fix naming

* Fix compatibility tests

* Rename a class

* Ensure restart_cmd_alt is set for specific OSes.

* Add docstring

* Fix override

* Fix coverage
2021-04-13 11:18:49 -07:00
Brad Warren
7f9857a81b Use Python 3 style super (#8777)
This is one of the things that newer versions of `pylint` complains about.

* git grep -l super\( | xargs sed -i 's/super([^)]*)/super()/g'

* fix spacing
2021-04-08 13:04:51 -07:00
Adrien Ferrand
c438a397a0 Enable mypy strict mode (#8766)
Built on top of #8748, this PR reenables mypy strict mode and adds the appropriate corrections to pass the types checks.

* Upgrade mypy

* First step for acme

* Cast for the rescue

* Fixing types for certbot

* Fix typing for certbot-nginx

* Finalize type fixes, configure no optional strict check for mypy in tox

* Align requirements

* Isort

* Pylint

* Protocol for python 3.6

* Use Python 3.9 for mypy, make code compatible with Python 3.8<

* Pylint and mypy

* Pragma no cover

* Pythonic NotImplemented constant

* More type definitions

* Add comments

* Simplify typing logic

* Use vararg tuple

* Relax constraints on mypy

* Add more type

* Do not silence error if target is not defined

* Conditionally import Protocol for type checking only

* Clean up imports

* Add comments

* Align python version linting with mypy and coverage

* Just ignore types in an unused module

* Add comments

* Fix lint

* Work in progress

* Finish type control

* Isort

* Fix pylint

* Fix imports

* Fix cli subparser

* Some fixes

* Coverage

* Remove --no-strict-optional (obviously...)

* Update certbot-apache/certbot_apache/_internal/configurator.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/_internal/display/completer.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Cleanup dns_google

* Improve lock controls and fix subparser

* Use the expected interfaces

* Fix code

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2021-04-05 15:04:21 -07:00
Adrien Ferrand
06a53cb7df Upgrade to mypy 0.812 (#8748)
Fixes #8425

This PR upgrades mypy to the latest version available, 0.812.

Given the advanced type inference capabilities provided by this newer version, this PRs also fixes various type inconsistencies that are now detected. Here are the non obvious changes done to fix types:
* typing in mixins has been solved using `Protocol` classes, as recommended by mypy (https://mypy.readthedocs.io/en/latest/more_types.html#mixin-classes, https://mypy.readthedocs.io/en/stable/protocols.html)
* `cast` when we are playing with `Union` types

This PR also disables the strict optional checks that have been enable by default in recent versions of mypy. Once this PR is merged, I will create an issue to study how these checks can be enabled.

`typing.Protocol` is available only since Python 3.8. To keep compatibility with Python 3.6, I try to import the class `Protocol` from `typing`, and fallback to assign `object` to `Protocol` if that fails. This way the code is working with all versions of Python, but the mypy check can be run only with Python 3.8+ because it needs the protocol feature. As a consequence, tox runs mypy under Python 3.8.

Alternatives are:
* importing `typing_extensions`, that proposes backport of newest typing features to Python 3.6, but this implies to add a dependency to Certbot just to run mypy
* redesign the concerned classes to not use mixins, or use them differently, but this implies to modify the code itself even if there is nothing wrong with it and it is just a matter of instructing mypy to understand in which context the mixins can be used
* ignoring type for these classes with `# type: ignore` but we loose the benefit of mypy for them

* Upgrade mypy

* First step for acme

* Cast for the rescue

* Fixing types for certbot

* Fix typing for certbot-nginx

* Finalize type fixes, configure no optional strict check for mypy in tox

* Align requirements

* Isort

* Pylint

* Protocol for python 3.6

* Use Python 3.9 for mypy, make code compatible with Python 3.8<

* Pylint and mypy

* Pragma no cover

* Pythonic NotImplemented constant

* More type definitions

* Add comments

* Simplify typing logic

* Use vararg tuple

* Relax constraints on mypy

* Add more type

* Do not silence error if target is not defined

* Conditionally import Protocol for type checking only

* Clean up imports

* Add comments

* Align python version linting with mypy and coverage

* Just ignore types in an unused module

* Add comments

* Fix lint
2021-04-02 11:54:40 -07:00
Adrien Ferrand
dd6f2f565e Convert Python 2 type hints to Python 3 types annotations (#8640)
Fixes #8427

This PR converts the Python 2 types hints into Python 3 types annotations. I have used the project https://github.com/ilevkivskyi/com2ann which has been designed for that specific purpose and did that very well.

The only remaining things to do were to fix broken type hints that became wrong code after migration, and to fix lines too long with the new syntax.

* Raw execution of com2ann

* Fixing broken type annotations

* Cleanup imports
2021-03-10 11:51:27 -08:00
Adrien Ferrand
67b65bb2c0 Deprecate acme.typing_magic module, stop using it in certbot (#8643)
* Deprecate acme.magic_typing, stop to use it in certbot

* Isort

* Add a changelog entry

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2021-03-09 16:12:32 -08:00
Mads Jensen
a3abcc001a Removed a Python 2 fallback in certbot.Reverter. (#8694)
* Removed a Python 2 fallback in certbot.Reverter.

* Removed a Python < 3.6 fallback in certbot-apache._internal.parser.
2021-03-04 08:10:56 +11:00
Mads Jensen
67c2b27af7 Stop inheriting from object. It's unneeded on Python 3+. (#8675) 2021-02-25 14:59:00 -08:00
Mads Jensen
135187f03e Python 3 obsoletes explicit __ne__ methods (#8676)
This shouldn't be needed as of Python 3+.

https://stackoverflow.com/questions/4352244/should-ne-be-implemented-as-the-negation-of-eq-in-python#30676267
2021-02-25 14:50:54 -08:00
Brad Warren
3d0dad8718 Remove dependency on six (#8650)
Fixes https://github.com/certbot/certbot/issues/8494.

I left the `six` dependency pinned in `tests/letstest/requirements.txt` and `tools/oldest_constraints.txt` because `six` is still a transitive dependency with our current pinnings.

The extra moving around of imports is due to me using `isort` to help me keep dependencies in sorted order after replacing imports of `six`.

* remove some six usage in acme

* remove six from acme

* remove six.add_metaclass usage

* fix six.moves.zip

* fix six.moves.builtins.open

* six.moves server fixes

* 's/six\.moves\.range/range/g'

* stop using six.moves.xrange

* fix urllib imports

* s/six\.binary_type/bytes/g

* s/six\.string_types/str/g

* 's/six\.text_type/str/g'

* fix six.iteritems usage

* fix itervalues usage

* switch from six.StringIO to io.StringIO

* remove six imports

* misc fixes

* stop using six.reload_module

* no six.PY2

* rip out six

* keep six pinned in oldest constraints

* fix log_test.py

* update changelog
2021-02-09 11:43:15 -08:00
Antonio Larrosa
434ca1985f Change the SUSE override to use apachectl (#8592)
For some time, SUSE distributions have had both an apachectl
executable and an apache2ctl compat symlink so both could be used
but apachectl is preferred since that's the official upstream name.
This is currently the case in SLE 15 SP2 and openSUSE Leap 15.2
(and every release since SLE 12 SP1)

OTOH, openSUSE Tumbleweed removed the apache2ctl compat symlink
some weeks ago and both SLE/Leap will follow in one of the next
releases so it's better to change certbot to use the official name,
apachectl.
2021-01-08 09:49:21 -08:00
Adrien Ferrand
e9a96f5e2a Deprecate support of Apache 2.2 in certbot-apache (#8516)
Fixes #8462

* Deprecate support of Apache 2.2 in certbot-apache

* Add a changelog
2020-12-10 12:57:13 -08:00
Brad Warren
75365f1d4e Remove python_version setting from mypy.ini (#8426)
* Remove python_version from mypy.ini.

* Fix magic_typing

* Ignore msvcrt usage.

* make mypy happier

* clean up changes

* Add type for reporter queue

* More mypy fixes

* Fix pyrfc3339 str.

* Remove unused import.

* Make certbot.util mypy work in both Pythons

* Fix typo
2020-11-05 15:28:35 -08:00
Cameron Steel
9a72db5b9b Convert http links to https (#8287)
* Convert http links to https

* Fix remaining links
2020-09-23 19:36:55 +02:00
Mads Jensen
501df0dc4e Use in dict rather than "in dict.keys()". Fix linting warnings about "not in". (#8298)
* Fixed a few linting warnings for if not x in y.

These should have been caught by pylint, but weren't.

* Replaced "x in y.keys()" with "x in y".

It's much faster, and more Pythonic.
2020-09-19 11:35:49 +02:00
ohemorange
713b91495b Fix paths when calling out to programs outside of snap (#8108)
Fixes #8093.

This PR modifies and audits all uses of `subprocess` and `Popen` outside of tests, `certbot-ci/`, `certbot-compatibility-test/`, `letsencrypt-auto-source/`, `tools/`, and `windows-installer/`. Calls to outside programs have their `env` modified to remove the `SNAP` components of paths, if they exist. This includes any calls made from hooks, calls to `apachectl` and `nginx`, and to `openssl` from `ocsp.py`.

For testing manually, rsync flags will look something like:

```
rsync -avzhe ssh root@focal.domain:/home/certbot/certbot/certbot_*_amd64.snap .
rsync -avzhe ssh certbot_*_amd64.snap root@centos7.domain:/root/certbot/
```

With these modifications, `certbot plugins --prepare` now passes on Centos 7.

If I'm wrong and we package the `openssl` binary, the modifications should be removed from `ocsp.py`, and `env` should be passed into `run_script` rather than set internally in its calls from nginx and apache.

One caveat with this approach is the disconnect between why it's a problem (packaging) and where it's solved (internal to Certbot). I considered a wrapping approach, but we'd still have to audit specific calls. I think the best way to address this is robust testing; specifically, running the snap on other systems.

For hooks, all calls will remove the snap paths if they exist. This is probably fine, because even if the hook intends to call back into certbot, it can do that, it'll just create a new snap.

I'm not sure if we need these modifications for the Mac OS X/ Darwin calls, but they can't hurt.

* Add method to plugins util to get env without snap paths

* Use modified environment in Nginx plugin

* Pass through env to certbot.util.run_script

* Use modified environment in Apache plugin

* move env_no_snap_for_external_calls to certbot.util

* Set env internally to run_script, since we use that only to call out

* Add env to mac subprocess calls in certbot.util

* Add env to openssl call in ocsp.py

* Add env for hooks calls in certbot.compat.misc.

* Pass env into execute_command to avoid circular dependency

* Update hook test to assert called with env

* Fix mypy type hint to account for new param

* Change signature to include Optional

* go back to using CERTBOT_PLUGIN_PATH

* no need to modify PYTHONPATH in env

* robustly detect when we're in a snap

* Improve env util fxn docstring

* Update changelog

* Add unit tests for env_no_snap_for_external_calls

* Import compat.os
2020-06-25 15:36:29 -07:00
Adrien Ferrand
50fa04ba0c Implement umask for Windows (#7967)
This PR gets its root from an observation I did on current version of Certbot (1.3.0): the `renewal-hooks` directory in Certbot configuration directory is created on Windows with write permissions to everybody.

I thought it was a critical bug since this directory contains hooks that are executed by Certbot, and you certainly do not want this folder to be open to any malicious hook that could be inserted by everyone, then executed with administrator privileges by Certbot.

Turns out for this specific problem that the bug is not critical for the hooks, because the scripts are expected to be in subdirectories of `renewal-hooks` (namely `pre`, `post` and `deploy`), and these subdirectories have proper permissions because we set them explicitly when Certbot is starting.

Still, there is a divergence here between Linux and Windows: on Linux all Certbot directories without explicit permissions have at maximum `0o755` permissions by default, while on Windows it is a `0o777` equivalent. It is not an immediate security risk, but it is definitly error-prone, not expected, and so a potential breach in the future if we forget about it.

Root cause is that umask is not existing in Windows. Indeed under Linux the umask defines the default permissions when you create a file or a directory. Python takes that into account, with an API for `os.open` and `os.mkdir` that expose a `mode` parameter with default value of `0o777`. In practice it is never `0o777` (either you the the `mode` explictly or left the default one) because the effective mode is masked by the current umask value in the system: on Linux it is `0o022`, so files/directories have a maximum mode of `0o755` if you did not set the umask explicitly, and it is what it is observed for Certbot.

However on Windows, the `mode` value passed (and got from default) to the `open` and `mkdir` of `certbot.compat.filesystem` module is taken verbatim, since umask does not exit, and then is used to calculate the DACL of the newly created file/directory. So if the mode is not set explicitly, we end up with files and directories with `0o777` permissions.

This PR fixes this problem by implementing a umask behavior in the `certbot.compat.filesystem` module, that will be applied to any file or directory created by Certbot since we forbid to use the `os` module directly.

The implementation is quite straight-forward. For Linux the behavior is not changed. On Windows a `mask` parameter is added to the function that calculates the DACL, to be invoked appropriately when file or directory are created. The actual value of the mask is taken from an internal class of the `filesystem` module: its default value is `0o755` to match default umasks on Linux, and can be changed with the new method `umask` that have the same behavior than the original `os.umask`. Of course `os.umask` becomes a forbidden function and `filesystem.umask` must be used instead.

Existing code that is impacted have been updated, and new unit tests are created for this new function.

* Implement umask for Windows

* Set umask at the beginning of tests

* Fix lint, update local oldest requirements

* Update certbot-apache/setup.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Improve tests

* Adapt filesystem.makedirs for Windows

* Fix

* Update certbot-apache/setup.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Changelog entries

* Fix lint

* Update certbot/CHANGELOG.md

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2020-06-09 17:08:22 -07:00
alexzorin
d135e6140b apache: handle statically linked mod_ssl (#8007)
In #7771, the Apache configurator gained the ability to identify what
version of OpenSSL Apache's ssl_module is linked against. However, the
detection was only functional if the module was built as a DSO (which is
almost always the case).

This commit covers the case where the ssl_module is statically linked
within the Apache binary. It requires the user to specify the path to
the binary (with --apache-bin) and emits a warning if static linking is
detected but no path has been provided.
2020-06-04 10:34:10 -07:00
ohemorange
3ea5170647 Error out earlier in apache installer when mod_ssl is not available (#7984)
* Error out in apache installer when mod_ssl is not available

* Update to MisconfigurationError and add/fix tests

* Remove error cases we no longer hit and associated test

* mock out function to have consistent error across machines

* improve changelog message

* only check key in modules list, not value
2020-05-19 15:34:21 -07:00
Brad Warren
859dc38cb9 Consolidate cover envs and default to py3-cover (#7905)
* Consolidate cover envs and default to py3-cover

* use py38 for code coverage in Travis

* Disable coverage on Python < 3.6 line.
2020-04-16 08:59:40 -07:00
Karan Suthar
8e4dc0a48c Minor bugfixes (#7891)
* Fix dangerous default argument

* Remove unused imports

* Remove unnecessary comprehension

* Use literal syntax to create data structure

* Use literal syntax instead of function calls to create data structure

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2020-04-13 10:41:39 -07:00
m0namon
1285297b23 [Apache v2] Load apacheconfig tree and gate related tests (#7710)
* Load apacheconfig dependency, gate behind flag

* Bump apacheconfig dependency to latest version and install dev version of apache for coverage tests

* Move augeasnode_test tests to more generic parsernode_test

* Revert "Move augeasnode_test tests to more generic parsernode_test"

This reverts commit 6bb986ef78.

* Mock AugeasNode into DualNode's place, and run augeasnode tests exclusively on AugeasNode

* Don't calculate coverage for skeleton functions

* clean up helper function in augeasnode_test
2020-03-23 17:05:22 -07:00
ohemorange
9e3c348dff Disable TLS session tickets in Apache (#7771)
Fixes #7350.

This PR changes the parsed modules from a `set` to a `dict`, with the filepath argument as the value. Accordingly, after calling `enable_mod` to enable `ssl_module`, modules now need to be re-parsed, so call `reset_modules`.

* Add mechanism for selecting apache config file, based on work done in #7191.

* Check OpenSSL version

* Remove os imports

* debian override still needs os

* Reformat remaining apache tests with modules dict syntax

* Clean up more apache tests

* Switch from property to method for openssl and add tests for coverage.

* Sometimes the dict location will be None in which case we should in fact return None

* warn thoroughly and consistently in openssl_version function

* update tests for new warnings

* read file as bytes, and factor out the open for testing

* normalize ssl_module_location path to account for being relative to server root

* Use byte literals in a python 2 and 3 compatible way

* string does need to be a literal

* patch builtins open

* add debug, remove space

* Add test to check if OpenSSL detection is working on different systems

* fix relative test location for cwd

* put </IfModule> on its own line in test case

* Revert test file to status in master.

* Call augeas load before reparsing modules to pick up the changes

* fix grep, tail, and mod_ssl location on centos

* strip the trailing whitespace from fedora

* just use LooseVersion in test

* call apache2ctl on debian systems

* Use sudo for apache2ctl command

* add check to make sure we're getting a version

* Add boolean so we don't warn on debian/ubuntu before trying to enable mod_ssl

* Reduce warnings while testing by setting mock _openssl_version.

* Make sure we're not throwing away any unwritten changes to the config

* test last warning case for coverage

* text changes for clarity
2020-03-23 16:49:52 -07:00
Brad Warren
06599a1e18 Cleanup more pylint issues (#7848)
This PR builds on #7657 and cleans up additional unnecessary pylint comments and some stray comments referring to pylint: disable comments that have been deleted that I didn't notice in my review of that PR.

* Remove stray pylint link.

* Cleanup more pylint comments

* Cleanup magic_typing imports

* Remove unneeded pylint: enable comments
2020-03-16 09:43:48 -07:00
m0namon
f169c37153 Merge pull request #7742 from osirisinferi/force-non-restrictive-umask
Force non restrictive umask when creating challenge directory in Apache plugin
2020-02-26 17:09:20 -08:00
Brad Warren
42dda355c5 Correct AutoHSTS docs (#7767)
domains is a list of strings, not a single string.

* Correct AutoHSTS docs.

* Fix Apache enable_autohsts docs.
2020-02-18 14:54:07 -08:00
Adrien Ferrand
fc7e5e8e60 Remove useless pylint error suppression directives (#7657)
As pylint is evolving, it improves its accuracy, and several pylint error suppression (`# pylint: disable=ERROR) added in certbot codebase months or years ago are not needed anymore to make it happy.

There is a (disabled by default) pylint error to detect the useless suppressions (pylint-ception: `useless-suppression`). It is not working perfectly (it has also false-positives ...) but it is a good start to clean the codebase.

This PR removes several of these useless suppressions as detected by the current pylint version we use.

* Remove useless suppress

* Remove useless lines
2020-02-13 13:56:16 -08:00