Built on top of #8748, this PR reenables mypy strict mode and adds the appropriate corrections to pass the types checks.
* Upgrade mypy
* First step for acme
* Cast for the rescue
* Fixing types for certbot
* Fix typing for certbot-nginx
* Finalize type fixes, configure no optional strict check for mypy in tox
* Align requirements
* Isort
* Pylint
* Protocol for python 3.6
* Use Python 3.9 for mypy, make code compatible with Python 3.8<
* Pylint and mypy
* Pragma no cover
* Pythonic NotImplemented constant
* More type definitions
* Add comments
* Simplify typing logic
* Use vararg tuple
* Relax constraints on mypy
* Add more type
* Do not silence error if target is not defined
* Conditionally import Protocol for type checking only
* Clean up imports
* Add comments
* Align python version linting with mypy and coverage
* Just ignore types in an unused module
* Add comments
* Fix lint
* Work in progress
* Finish type control
* Isort
* Fix pylint
* Fix imports
* Fix cli subparser
* Some fixes
* Coverage
* Remove --no-strict-optional (obviously...)
* Update certbot-apache/certbot_apache/_internal/configurator.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/certbot/_internal/display/completer.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Cleanup dns_google
* Improve lock controls and fix subparser
* Use the expected interfaces
* Fix code
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Fixes#8773
I took option 2 from the issue mentionned above (importing `typing-extensions` on dev dependencies) to avoid modifying certbot runtime requirements given that what needs to be added is useful for mypy only.
I did not change the Python version used to execute the linting and mypy on the standard tests, given that the tox `docker_dev` target already checks if the development environment is working for Python < 3.8.
We were originally using `socket.errno` with a `type: ignore` and a comment suggesting that this attribute needs to be included in the typeshed. This is incorrect.
While it's true that [socket imports errno](43682f1e39/Lib/socket.py (L58)), it's not intended to be part of its API. https://docs.python.org/3/library/socket.html has no mention of it.
Instead, we should be using the standard `errno` module and remove this `type: ignore`.
Some are no longer needed and other's comments are out of date.
For the changes to the acme nonce errors, `Exception` doesn't take kwargs. The error message about this our own classes isn't super helpful:
```
In [2]: BadNonce('nonce', 'error', foo='bar')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-2-54555658ef99> in <module>
----> 1 BadNonce('nonce', 'error', foo='bar')
TypeError: __init__() got an unexpected keyword argument 'foo'
```
but if you try this on `Exception` which these classes inherit from, you get:
```
In [4]: Exception(foo='bar')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-4-028b924f74c5> in <module>
----> 1 Exception(foo='bar')
TypeError: Exception() takes no keyword arguments
```
See https://github.com/python/typeshed/pull/2348 for more info.
* remove outdated ignores
* update locking ignore comment
* don't accept kwargs
Fixes#8425
This PR upgrades mypy to the latest version available, 0.812.
Given the advanced type inference capabilities provided by this newer version, this PRs also fixes various type inconsistencies that are now detected. Here are the non obvious changes done to fix types:
* typing in mixins has been solved using `Protocol` classes, as recommended by mypy (https://mypy.readthedocs.io/en/latest/more_types.html#mixin-classes, https://mypy.readthedocs.io/en/stable/protocols.html)
* `cast` when we are playing with `Union` types
This PR also disables the strict optional checks that have been enable by default in recent versions of mypy. Once this PR is merged, I will create an issue to study how these checks can be enabled.
`typing.Protocol` is available only since Python 3.8. To keep compatibility with Python 3.6, I try to import the class `Protocol` from `typing`, and fallback to assign `object` to `Protocol` if that fails. This way the code is working with all versions of Python, but the mypy check can be run only with Python 3.8+ because it needs the protocol feature. As a consequence, tox runs mypy under Python 3.8.
Alternatives are:
* importing `typing_extensions`, that proposes backport of newest typing features to Python 3.6, but this implies to add a dependency to Certbot just to run mypy
* redesign the concerned classes to not use mixins, or use them differently, but this implies to modify the code itself even if there is nothing wrong with it and it is just a matter of instructing mypy to understand in which context the mixins can be used
* ignoring type for these classes with `# type: ignore` but we loose the benefit of mypy for them
* Upgrade mypy
* First step for acme
* Cast for the rescue
* Fixing types for certbot
* Fix typing for certbot-nginx
* Finalize type fixes, configure no optional strict check for mypy in tox
* Align requirements
* Isort
* Pylint
* Protocol for python 3.6
* Use Python 3.9 for mypy, make code compatible with Python 3.8<
* Pylint and mypy
* Pragma no cover
* Pythonic NotImplemented constant
* More type definitions
* Add comments
* Simplify typing logic
* Use vararg tuple
* Relax constraints on mypy
* Add more type
* Do not silence error if target is not defined
* Conditionally import Protocol for type checking only
* Clean up imports
* Add comments
* Align python version linting with mypy and coverage
* Just ignore types in an unused module
* Add comments
* Fix lint
I recently noticed that we only support versions of `setuptools` that support environment markers which allows us to simplify our `setup.py` files a bit.
In #8649 we added some code to trick pynsist and make it understand that `abi3` wheels for Windows are forward compatible, meaning that the cryptography wheel tagged `cp36-abi3` is in fact compatible with Python 3.6+, and not only Python 3.6.
Since pynsist 2.7 the tool now understand `abi3` wheels properly, and this trick is not needed anymore.
Please note that despite modifying the pynsist pinning in `dev_constraints.txt`, it will have no effect since pynsist currently escape the pinning system. This is handled in https://github.com/certbot/certbot/pull/8749.
* Pin pynsist
* Update dependencies
* Set windows installer a proper python project
* Optimize usage of the venvs
* Add windows-installer when venv is set up
* Fix call
* Remove env marker
Fixes#8700
Now that `snapcraft remote-build` truly uses new builds for each call, we can split the builds to have a dedicated Azure job for each target architecture. This PR does that.
* Split snap_build job on each architecture
* Also parallelize the publish_snap jobs over each architecture
Fixes#8661.
As mentioned in https://github.com/certbot/certbot/issues/8661#issuecomment-806168214, there are quite a few remaining references, but until we modify the release script, we still need those. The changes here and the list there were created by grepping for the following terms:
```
certbot-auto
cb-auto
cbauto
certbotauto
letsencrypt-auto
le-auto
leauto
letsencryptauto
LEAUTO
LE_AUTO
LETSENCRYPT_AUTO
LETSENCRYPTAUTO
CB_AUTO
CERTBOT_AUTO
CBAUTO
CERTBOTAUTO
```
* Remove references to certbot-auto from certbot code
* Remove references to LEAUTO
* Remove references to CERTBOT_AUTO
* Remove references to letsencrypt-auto
* Remove references to certbot-auto from docs and tools
* remove cli constants header files
* Remove Python virtual environment section
* Upgrade cryptography to 3.4.6
* Fix comment with instructions for how to use hashin
* run tools/rebuild_certbot_constraints.py
* add deps for building cryptography in snaps
* Update cryptography build dependencies for docker
* Update sources for test farm tests
* Remove rust if it's installed for test farm tests
* source bootstrap script and call sudo as needed
We observed recently several unexpected behavior during the execution of snap jobs in Azure. In particular it seems that `snapcraft remote-build` is tending to reattach to the latest builds on Launchpad triggered by the nightly builds on master, independently from the actual branch, status of the code, or targeted architectures.
Primarily if the builds on Launchpad are stalled for some reason, it blocks effectively any other Azure snap jobs until someone manually cancel the builds on Launchpad. Secondarily it means that the outcome of the builds may be inconsistent, because they can be the result of a build for the master source even if you are on a PR that modifieds these sources (including `snapcraft.yaml`).
After digging in `snapcraft` source code, I realized that the signature computed to understand if a build should be resumed, is not based one some hashes against the snapcraft working directory content, but is simply a hash of the working directory absolute path *itself*. It means that every builds triggered from the working directory `/my/path/certbot` for instance, are recognized as the same unique build on Launchpad side, and may be resumed if they already exist, and so independently from the source code, `snapcraft.yaml` or targeted archs.
For the record, relevant parts in `snapcraft` source code:
82024d3748/snapcraft/project/_project.py (L44)82024d3748/snapcraft/project/_project.py (L86-L89)82024d3748/snapcraft/cli/remote.py (L128-L132)
This PR makes effectively the resume build mechanism effectively a noop by moving the source code first in a temporary directory with random name before running `snapcraft remote-build`. This way the signature is never the same and builds are always recognized as brand new builds.
* Invalidate snapcraft remote-build cache by using a temporary workspace.
* Capture one more state in the build
* Precise the certificate naming convention mechanism in a note.
* Add certificate name convention in user guide, refer to it in compatibility page.
* Update certbot/docs/compatibility.rst
Co-authored-by: alexzorin <alex@zor.io>
* Update certbot/docs/using.rst
Co-authored-by: alexzorin <alex@zor.io>
* Update certbot/docs/using.rst
Co-authored-by: alexzorin <alex@zor.io>
* Improve the note about naming conventions
Co-authored-by: alexzorin <alex@zor.io>
While working on #8640, I realized that there were some hidden circular dependencies in certbot._internal.cli package. Then cerbot could break if the order of these imports changes.
This PR fixes that and apply isort on top of the result.
* Kill snapcraft build when a "Chroot problem" is encountered
* Display specific helper for "Chroot problem" status and cancel retry mechanism in this case.
* Isolate build tmp directories
* Configure XDG_CACHE_HOME
* Kill snapcraftctl with chroot problem is encountered
Fixes#8427
This PR converts the Python 2 types hints into Python 3 types annotations. I have used the project https://github.com/ilevkivskyi/com2ann which has been designed for that specific purpose and did that very well.
The only remaining things to do were to fix broken type hints that became wrong code after migration, and to fix lines too long with the new syntax.
* Raw execution of com2ann
* Fixing broken type annotations
* Cleanup imports
There are still some left, but the `modification_check` test fails. Some are still in `tools`, and they can probably be removed as well. `with_statement` was introduced officially in Python 2.5, so there's really old stuff in the code base.
Fixes https://github.com/certbot/certbot/issues/8690.
After this PR, we'll let the release script make its automated changes to certbot-auto as part of the 1.14.0 release and then never make any code changes to certbot-auto ever again!
* disable upgrades on debian
* update test_leauto_upgrades.sh
* update changelog
* revoke: try determine the server automatically
When revoking via --cert-name, use the server from the lineage (unless
overriden by the CLI).
* RenewableCert.storage might be None
* guard against an empty lineage server
