1
0
mirror of https://github.com/certbot/certbot.git synced 2026-01-26 07:41:33 +03:00
Commit Graph

82 Commits

Author SHA1 Message Date
Brad Warren
315ddb247f Upgrade pylint (#8855)
This is part of https://github.com/certbot/certbot/issues/8782. I took it on now because the currently pinned version of `pylint` doesn't work with newer versions of `poetry` which I wanted to upgrade as part of https://github.com/certbot/certbot/issues/8787.

To say a bit more about the specific changes in this PR:

* Newer versions of `pylint` complain if `Popen` isn't used as a context manager. Instead of making this change, I switched to using `subprocess.run` which is simpler and [recommended in the Python docs](https://docs.python.org/3/library/subprocess.html#using-the-subprocess-module). I also disabled this check in a few places where no longer using `Popen` would require significant refactoring.
* The deleted code in `certbot/certbot/_internal/renewal.py` is cruft since https://github.com/certbot/certbot/pull/8685.
* The unused argument to `enable_mod` in the Apache plugin is used in some over the override classes that subclass that class.

* unpin pylint and repin dependencies

* disable raise-missing-from

* disable wrong-input-order

* remove unused code

* misc lint fixes

* remove unused import

* various lint fixes
2021-05-24 10:02:55 -07:00
Brad Warren
934de48d44 fix typo (#8828) 2021-05-05 15:49:06 -07:00
Brad Warren
d3d9a05826 fix client email address (#8817)
client-dev@letsencrypt.org is no longer used by the Certbot team so this PR updates the email address in our packages to our current mailing list.
2021-05-03 12:38:54 -07:00
Brad Warren
7f9857a81b Use Python 3 style super (#8777)
This is one of the things that newer versions of `pylint` complains about.

* git grep -l super\( | xargs sed -i 's/super([^)]*)/super()/g'

* fix spacing
2021-04-08 13:04:51 -07:00
Adrien Ferrand
c438a397a0 Enable mypy strict mode (#8766)
Built on top of #8748, this PR reenables mypy strict mode and adds the appropriate corrections to pass the types checks.

* Upgrade mypy

* First step for acme

* Cast for the rescue

* Fixing types for certbot

* Fix typing for certbot-nginx

* Finalize type fixes, configure no optional strict check for mypy in tox

* Align requirements

* Isort

* Pylint

* Protocol for python 3.6

* Use Python 3.9 for mypy, make code compatible with Python 3.8<

* Pylint and mypy

* Pragma no cover

* Pythonic NotImplemented constant

* More type definitions

* Add comments

* Simplify typing logic

* Use vararg tuple

* Relax constraints on mypy

* Add more type

* Do not silence error if target is not defined

* Conditionally import Protocol for type checking only

* Clean up imports

* Add comments

* Align python version linting with mypy and coverage

* Just ignore types in an unused module

* Add comments

* Fix lint

* Work in progress

* Finish type control

* Isort

* Fix pylint

* Fix imports

* Fix cli subparser

* Some fixes

* Coverage

* Remove --no-strict-optional (obviously...)

* Update certbot-apache/certbot_apache/_internal/configurator.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/_internal/display/completer.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Cleanup dns_google

* Improve lock controls and fix subparser

* Use the expected interfaces

* Fix code

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2021-04-05 15:04:21 -07:00
Adrien Ferrand
06a53cb7df Upgrade to mypy 0.812 (#8748)
Fixes #8425

This PR upgrades mypy to the latest version available, 0.812.

Given the advanced type inference capabilities provided by this newer version, this PRs also fixes various type inconsistencies that are now detected. Here are the non obvious changes done to fix types:
* typing in mixins has been solved using `Protocol` classes, as recommended by mypy (https://mypy.readthedocs.io/en/latest/more_types.html#mixin-classes, https://mypy.readthedocs.io/en/stable/protocols.html)
* `cast` when we are playing with `Union` types

This PR also disables the strict optional checks that have been enable by default in recent versions of mypy. Once this PR is merged, I will create an issue to study how these checks can be enabled.

`typing.Protocol` is available only since Python 3.8. To keep compatibility with Python 3.6, I try to import the class `Protocol` from `typing`, and fallback to assign `object` to `Protocol` if that fails. This way the code is working with all versions of Python, but the mypy check can be run only with Python 3.8+ because it needs the protocol feature. As a consequence, tox runs mypy under Python 3.8.

Alternatives are:
* importing `typing_extensions`, that proposes backport of newest typing features to Python 3.6, but this implies to add a dependency to Certbot just to run mypy
* redesign the concerned classes to not use mixins, or use them differently, but this implies to modify the code itself even if there is nothing wrong with it and it is just a matter of instructing mypy to understand in which context the mixins can be used
* ignoring type for these classes with `# type: ignore` but we loose the benefit of mypy for them

* Upgrade mypy

* First step for acme

* Cast for the rescue

* Fixing types for certbot

* Fix typing for certbot-nginx

* Finalize type fixes, configure no optional strict check for mypy in tox

* Align requirements

* Isort

* Pylint

* Protocol for python 3.6

* Use Python 3.9 for mypy, make code compatible with Python 3.8<

* Pylint and mypy

* Pragma no cover

* Pythonic NotImplemented constant

* More type definitions

* Add comments

* Simplify typing logic

* Use vararg tuple

* Relax constraints on mypy

* Add more type

* Do not silence error if target is not defined

* Conditionally import Protocol for type checking only

* Clean up imports

* Add comments

* Align python version linting with mypy and coverage

* Just ignore types in an unused module

* Add comments

* Fix lint
2021-04-02 11:54:40 -07:00
Brad Warren
584a1a3ece simplify setup.py (#8760)
I recently noticed that we only support versions of `setuptools` that support environment markers which allows us to simplify our `setup.py` files a bit.
2021-04-02 10:37:48 -07:00
Brad Warren
54b0b98988 use worker terminology (#8728)
This will be needed for me to update `pytest-xdist` as part of https://github.com/certbot/certbot/issues/8705 since `pytest-xdist` removed the "slave" terminology. See https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst#deprecations-and-removals.
2021-03-23 11:29:01 -07:00
Adrien Ferrand
dd6f2f565e Convert Python 2 type hints to Python 3 types annotations (#8640)
Fixes #8427

This PR converts the Python 2 types hints into Python 3 types annotations. I have used the project https://github.com/ilevkivskyi/com2ann which has been designed for that specific purpose and did that very well.

The only remaining things to do were to fix broken type hints that became wrong code after migration, and to fix lines too long with the new syntax.

* Raw execution of com2ann

* Fixing broken type annotations

* Cleanup imports
2021-03-10 11:51:27 -08:00
Adrien Ferrand
67b65bb2c0 Deprecate acme.typing_magic module, stop using it in certbot (#8643)
* Deprecate acme.magic_typing, stop to use it in certbot

* Isort

* Add a changelog entry

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2021-03-09 16:12:32 -08:00
Mads Jensen
c02b2d30f2 Removed Python legacy __future__ imports (#8697)
There are still some left, but the `modification_check` test fails. Some are still in `tools`, and they can probably be removed as well. `with_statement` was introduced officially in Python 2.5, so there's really old stuff in the code base.
2021-03-05 16:53:20 -08:00
Mads Jensen
67c2b27af7 Stop inheriting from object. It's unneeded on Python 3+. (#8675) 2021-02-25 14:59:00 -08:00
Brad Warren
3d0dad8718 Remove dependency on six (#8650)
Fixes https://github.com/certbot/certbot/issues/8494.

I left the `six` dependency pinned in `tests/letstest/requirements.txt` and `tools/oldest_constraints.txt` because `six` is still a transitive dependency with our current pinnings.

The extra moving around of imports is due to me using `isort` to help me keep dependencies in sorted order after replacing imports of `six`.

* remove some six usage in acme

* remove six from acme

* remove six.add_metaclass usage

* fix six.moves.zip

* fix six.moves.builtins.open

* six.moves server fixes

* 's/six\.moves\.range/range/g'

* stop using six.moves.xrange

* fix urllib imports

* s/six\.binary_type/bytes/g

* s/six\.string_types/str/g

* 's/six\.text_type/str/g'

* fix six.iteritems usage

* fix itervalues usage

* switch from six.StringIO to io.StringIO

* remove six imports

* misc fixes

* stop using six.reload_module

* no six.PY2

* rip out six

* keep six pinned in oldest constraints

* fix log_test.py

* update changelog
2021-02-09 11:43:15 -08:00
Adrien Ferrand
7399807ff2 Drop Python 2 support (#8591)
Fixes #8389 #8584.

This PR makes the necessary modifications to officially drop Python 2 support in the Certbot project.

I did not remove the specific Python 2 compatibility branches that has been added in various places in the codebase, to reduce the size of this PR and this will be done in a future one

* Update classifiers and python_requires in setup.py

* Remove warnings about Python 2 deprecation

* Remove Azure jobs on Python 2.7

* Remove references to python 2 in documentation

* Pin dnspython to 2.1.0

* Update changelog

* Remove warning ignore
2021-01-25 15:07:43 -08:00
Adrien Ferrand
c0917a0302 Use os.path.normcase to have Windows compatible challenge paths on Windows (#8599)
* Use os.path.normcase to have Windows compatible challenge paths on Windows.

* Add integration test and fix lint
2021-01-13 14:38:57 -08:00
alexzorin
13d4a99251 test: certbot-ci crash due to no p521 on boulder (#8602)
* test: certbot-ci crash due to no p521 on boulder

The bugfix in #8598 added an integration test to request a certificate
for an EC P-521 key, which is unsupported when ACME_SERVER=boulder,
failing our nightly integration tests.

* add an integration test for all EC curves
2021-01-12 16:08:32 -08:00
Daniel Almasi
42f20455cd Fix EC curve name typo in crypto_util (#8598)
* Fix EC curve name typo in crypto_util

Fix typo of secp521r1 in crypto util module.
- secp521r1 is to be supported by certbot, but a typo of "SECP521R1" in the input validation section of the make_key function results in an error being thrown

* Add myself to authors.md 

Add myself to authors.md ^^

* Add test for secp521r1 key generation

Add test for secp521r1 key generation to cli-tests
2021-01-11 13:40:12 -08:00
Brad Warren
d3b82a4e8e Fix test farm tests by using a local Pebble instance (#8561)
[As discussed in Mattermost](https://opensource.eff.org/eff-open-source/pl/yhtp4qu4zpfczm5wxmzxhndrto), our Apache test farm tests are failing because the CA certificate in the old version of boulder we have pinned expired over the weekend. This PR fixes that by running a local Pebble instance instead of an external boulder instance.

* switch from external boulder to local pebble

* add --http-01-port to run_acme_server
2020-12-22 10:24:20 -08:00
Jacob Hoffman-Andrews
18faf4f7ab Edit certs -> certificates in user-facing text. (#8541)
* Edit certs -> certificates in user-facing text.

To reduce confusion, we should consistently use the full term.

* Edit certs->certificates in more user-facing text.

* fix failing lint (line too long)

* fix typo

Co-authored-by: Jacob Hoffman-Andrews <github@hoffman-andrews.com>

Co-authored-by: Alex Zorin <alex@zorin.id.au>
2020-12-21 16:00:31 -08:00
alexzorin
0465643d0a certbot-ci: fix integration-external tests (#8547)
In 96a05d9, mypy testing was added to certbot-ci, but introduced an
undeclared dependency on acme.magic_typing, resulting in a crash when
run under the integration-external tox environment.

This change uses the typing module in certbot-ci in place of
acme.magic_typing. It is already provided via dev_constraints.
2020-12-17 09:06:21 +01:00
Mads Jensen
96a05d946c Added certbot-ci to lint section. Silenced and fixed linting warnings. (#8450) 2020-12-16 20:34:12 +01:00
Mads Jensen
5871de0c07 Removed some unused imports. (#8424)
These were not annotated as something that should be ignored, and the test-suite
passes with these changes.
2020-12-04 14:29:58 +01:00
Adrien Ferrand
d476aa4389 Update both main VA and remote VA to use the provided DNS server (#8467) 2020-12-04 12:00:32 +11:00
Brad Warren
5f73274390 Fix add deprecated argument (#8500)
Fixes https://github.com/certbot/certbot/issues/8495.

To further explain the problem here, `modify_kwargs_for_default_detection` as called in `add` is simplistic and doesn't always work. See https://github.com/certbot/certbot/issues/6164 for one other example.

In this case, were bitten by the code d1e7404358/certbot/certbot/_internal/cli/helpful.py (L393-L395)

The action used for deprecated arguments isn't in `ZERO_ARG_ACTIONS` so it assumes that all deprecated flags take one parameter.

Rather than trying to fix this function (which I think can only realistically be fixed by https://github.com/certbot/certbot/issues/4493), I took the approach that was previously used in `HelpfulArgumentParser.add_deprecated_argument` of bypassing this extra logic entirely. I adapted that function to now call `HelpfulArgumentParser.add` as well for consistency and to make testing easier.

* Rename deprecated arg action class

* Skip extra parsing for deprecated arguments

* Add back test of --manual-public-ip-logging-ok

* Add changelog entry
2020-12-02 15:08:07 -08:00
alexzorin
f5a88ade54 nginx: fix Unicode crash on Python 2 (#8480)
* nginx: fix py2 unicode sandwich

The nginx parser would crash when saving configuraitons containing
Unicode, because py2's `str` type does not support Unicode.

This change fixes that crash by ensuring that a string type supporting
Unicode is used in both Python 2 and Python 3.

* nginx: add unicode to the integration test config

* update CHANGELOG
2020-11-27 18:15:27 +01:00
Brad Warren
9a4e95e25a Add Python 3.9 support and tests (#8460)
Fixes https://github.com/certbot/certbot/issues/8134.

* Test on Python 3.9.

* Mention Python 3.9 support in changelog.

* s/\( *'Pro.*3\.\)8\(',\)/\18\2\n\19\2/

* undo changes to tox.ini

* Move more tests to Python 3.9

* Update PyYAML and packages which pinned it back

* Upgrade typed-ast

* Use <= to "pin" dnspython

* Fix lint by telling pylint it cannot be trusted

* Disable mypy on RFC plugin

* add comment about <= support
2020-11-19 12:48:36 -08:00
Brad Warren
9ca7f76505 Merge pull request #8444 from certbot/ecdsa
Integrate the ECDSA certificates feature on master
2020-11-19 11:54:24 -08:00
Alex Zorin
e8139e80be certbot-ci: fix py2 crash in dns_server 2020-11-17 14:58:29 -08:00
Brad Warren
7ba35b4407 import print_function 2020-11-17 11:51:27 -08:00
alexzorin
90557921e3 Add certbot-dns-rfc2136 integration testing (#8448)
* tests: add certbot-dns-rfc2136 integration tests

* dont use 'with' form of socket.socket

fixes py2 crash

* address some feedback:

- conftest: make DNS server a global resource
- conftest: add dns_xdist parameter into node config
- conftest: add --dns-server=bind flag
- conftest: if configured, point the ACME server to the DNS server
- dnsserver: make it sort-of compatible with xdist (future-proofing)
- context: parameterize dns-rfc2136 credentials file (future proofing)
- context: reduce dns-rfc2136 propagation time to speed up tests
- tox: add a integration-dns-rfc2136 target
- rfc2136: add a test/zone for subdelegation
- rfc2136: skip tests if no DNS server is configured

* try add integration-dns-rfc2136 to CI

* mock recursive dns via RPZ

* update --dns-server args and tox.ini args

* address more feedback:

- dns_server: rename rfc2136 creds file to .tpl
- dns_server: dont vary dns server port, instead we will vary zone names (#8455)
- dns_server: log error if bind9 fails to stop cleanly
- dns_server: replace assert with raise
- context: remove redundant _worker_id
- context: remove redundant cleanup override
- context: fix seek/flush in credentials context manager
- context: rename skip_if_no_server -> ...bind_server
- context: add newline EOF

* conftest: document _setup_primary_node sideeffects

* ci: rfc2136-integration from standard->nightly

* fix _stop_bind (function was renamed to stop)

* ignore errors from shutil.rmtree during cleanup

* dns_server: check for crash while polling

* remove --dry-run from rfc2136 test
2020-11-17 09:27:27 +01:00
Adrien Ferrand
553d3279c6 Add --dns-server option in run_acme_server (#7722)
Fixes #7717

This PR adds a `--dns-server` option to the `run_acme_server` test tool, in order to provide an arbitrary DNS server to Pebble or Boulder for the integration tests.

I also take this occasion to make `run_acme_server` a real CLI tool using argparse, and set the `--server-type` (default `pebble`) option as well.

* Set --dns-server flag in run_acme_server

* Default to pebble

* Add documentation

* Configure also Boulder
2020-11-12 15:31:32 -08:00
Brad Warren
f15f4f9838 Add certbot renew --key-type test (#8447)
* Test certbot renew --key-type

* Fix typo
2020-11-12 00:06:50 +01:00
Adrien Ferrand
8f5787008d Handle unexpected key type migration. (#8435)
Fixes #8365

This PR adds a control when `certbot certonly` or `certbot run` are called for a certificate that already exists and would eventually be replaced. As described in #8365, this control is here to ensure that the user will not modify the key type of their certificate (eg. ECDSA to RSA) without an explicit approval (set explicitly `--cert-name` and `--key-type`), since RSA is the default if not specified.

* Handle unexpected key type migration.

* Update certbot-ci/certbot_integration_tests/certbot_tests/test_main.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2020-11-11 12:36:16 -08:00
alexzorin
bf20f39ceb cli: miscellaneous IReporter removals (#8436)
* certbot delete: use undecorated print

* certbot revoke: use undecorated print

* certbot revoke: remove ireporter usages

* eff: remove IReporter usages

* certbot unregister: remove IReporter usage

* certbot update_account: remove IReporter usages

* certbot run: remove IReporter in duplicate prompt

* fix test_revoke_multiple_lineages
2020-11-09 15:31:27 -08:00
Mads Jensen
47c1045f6d Implements support for ECDSA keys. Fixes #2163.
Thanks to @pahrohfit and @Tomoyuki-GH for previous efforts to implement
suport for this.

Co-Authored-By: Robert Dailey <rob@wargam.es>
Co-Authored-By: Tomoyuki-GH <55397638+Tomoyuki-GH@users.noreply.github.com>
2020-11-04 15:16:48 +01:00
alexzorin
4eb0b560c5 manual: deprecate --manual-public-ip-logging-ok (#8381)
* manual: deprecate --manual-public-ip-logging-ok

* remove unused cli.report_config_interaction code

Co-authored-by: ohemorange <ebportnoy@gmail.com>
2020-10-22 12:12:54 -07:00
Brad Warren
575092d603 Drop Python 3.5 support (#8206)
* delete classifiers

* update python_requires

* Update py35 Azure jobs

* Revert "Add warnings about Python 3.5 deprecation in Certbot (#8154)"

This reverts commit 270b5535e2.

* Update other Python 3.5 references.

* update changelog

* bump MIN_PYTHON_3_VERSION
2020-08-16 13:19:08 -07:00
ohemorange
9ee4831f78 Make externally snapped plugin updates more stable (#8145)
Fixes #7863.

Connect command is `sudo snap connect certbot-dns-dnsimple:certbot-metadata certbot:certbot-metadata`
Logs are `cat /var/snap/certbot-dns-dnsimple/current/debuglog`
Echos in hook are only printed to terminal when it exits 0; otherwise, check logs in `debuglog` mentioned above.

Manual tests include all iterations of connected, unconnected, installed for the first, second time, etc, with passing and failing version checks.

* Make dnsimple not update if certbot is too old

* create an interface to read cb version

* add missing newline

* fix syntax

* trying to figure out the consumer syntax

* trying to figure out the consumer syntax, again

* only check post first install

* valid setting name

* test for first install differently

* snapctl doesn't error if it fails I guess

* time to do some print debugging

* continue playing with syntax

* once again, fooled by bash int vs string comparisons!

* debugging

* if we use post and pre together we can do this

* is this how content interface syntax works

* it's a directory?

* more debug

* what's that error message again?

* try other syntax

* if it's not documented just guess at syntax

* actually, I think this is the syntax

* oops didn't set for new hook

* test passing information along connection

* interface attributes can only be set during the execution of prepare hooks

* just do it with main connection

* undo last few test changes

* Add some printing to make sure we understand what's going on

* create empty directory to bind to

* put mkdir in the correct part

* let's inspect the environment

* it can't run bash directly.

* perhaps only directories can be shared via the contente interface

* update name of folder

* echo to debug log to understand what's going on exactly. we have file access though!

* update grep for new file

* more printing

* echo to the debug log

* ok NOW all print statements are going to the log

* why does echo need two >s

* remove unnecessary extra check, just check if the init file is available

* check if certbot version will be available post-refresh after all

* pre-refresh hook is not necessary to get certbot version

* update mkdir so we don't have to clean each time

* try comparing version numbers in python

* it's python3

* we need different prints for if we succeed or if we fail.

* improve bash syntax

* remove some debugging code

* Remove debug script

* remove spaces for clarity

* consolidate parts and remove more test code

* s/certbot-version/certbot-metadata/g

* use sys.exit instead of exit

* find and save certbot version on the certbot side

* change presence test to new file

* switch to using packaging.version.parse instead of LooseVersion

* switch to requiring certbot version >= plugin version

* add plugin snap changes to generate script

* Add comment to generation file saying not to edit generated files manually

* Create post-refresh hook for all plugins with script

* generate files using new script

* update snapcraft.yaml files for plugins

* bin/sh comes first

* Add packaging to install_requires

* Check that refresh is allowed in integration test

* switch plug and slot names in integration test

* Update tools/generate_dnsplugins_postrefreshhook.sh

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* small bash fixes

* Update snap readme with new instructions

* Run tools/generate_dnsplugins_postrefreshhook.sh

* Update tools/snap/generate_dnsplugins_postrefreshhook.sh

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2020-07-22 17:07:07 -07:00
Adrien Ferrand
14dfbdbea5 Build snaps using the remote-build feature (#8153)
Snapcraft has a feature name `remote-build`. It allows to compile snaps using the Canonical dedicated build architecture for several architectures. Compared to the QEMU-enabled Docker approach used currently, the remote build has several advantages:
* the builds are done on the native architecture, making them basically faster than what can be achieved on QEMU
* it avoids to depend on `adferrand/snapcraft` (which could be otherwise be fixed with the merge of https://github.com/snapcore/snapcraft/pull/3144, but this will not happen in the short term)
* when everything is good, all snaps build can be run in parallel and then can be orchestrated by one single Azure Pipeline job, since the heavy tasks are done remotely.

This PR makes the necessary ajustements to use the remote build feature instead of the QEMU-enabled docker approach.

One complex task was to be able to compile the `certbot` snap on `arm64` and `armhf`. Indeed on these architectures the pre-compiled wheel for `cffi` is not available. So it needs to be compiled during the snap build. Sadly, the current version of the python plugin in snapcraft is limited by the fact that `wheels` is not installed in the virtual environment set up to build the python packages, and there is no easy way to change that except by overridding the whole build process.

In the long term, I think I will open a PR on `snapcraft` Git repository to provide a consistent solution. But for the short term, I used the possibility to provide arguments to the `venv` module, to add the flag `--system-site-packages`. With it, the virtual environment can use the system site package, where `wheel` is available.

The other significant additions are in `tools/snap/build_remote.py` script. If invoking the remote build on a local machine is quite straight-forward, it is another story on the CI because we need build auditability and resiliency during these non-interactive actions. In particular we should avoid as possible inconsistent results on the nightly pipeline and the release pipeline.

So this script wraps the `snapcraft` call into a retry logic, and improves its logs in the context of parallel builds.

For the minor modifications, it is mainly about ensuring that plugins can be built (some of them also need `cffi` for instance), and simplify the Azure Pipeline since all snaps are retrieved in one go.

Please note that the `test-` branches still run only the `amd64` architecture. Indeed I noticed that builds on `arm64` and `armhf` are tending to be very slow to start (up to 40 min) while the `amd64` ones wait at max 10 mins, and usually 30 seconds only when the overall load on Canonical side is low.

To work on `certbot/certbot` repository, one secured file needs to be added, because `snapcraft` needs to be authenticated against Launchpad with credentials allowing remote builds. To do so, from a local machine that have this capability, one can extract the existing file at `$HOME/.local/share/snapcraft/provider/launchpad/credentials`, and register it as a secured file in Azure Pipeline with the name `snapcraftRemoteBuildCredentials`.

* Define scripts

* Setup pipeline to use remote builds

* Focus on packaging builds

* Set credentials

* Setup git

* Launch all builds in parallel

* Add dev dependencies to build cffi and cryptography

* Convert to a python logic

* Reorganize the pipeline

* Handle the fact that snap builds may be taken from cache

* Generate constraints

* Exit code

* Check existence

* Try to handle better non zero exit code

* Add --system-site-packages to get wheel in the venv

* Add executable permissions

* Troubleshoot

* Dynamic display, take the maximum timeout for snap build job

* Allow retries if the remote build does not start

* Trigger only amd64 builds for test branches

* Exit properly

* Update snapcraft.yaml

* Fix snap run

* Set secured file name

* Update .azure-pipelines/templates/jobs/packaging-jobs.yml

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update .azure-pipelines/templates/jobs/packaging-jobs.yml

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update .azure-pipelines/templates/jobs/packaging-jobs.yml

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Move order in deps

* Reactivate all builds

* Use Manager() as a context manager

* Use Pool as a context manager

* Some nice refactorings

* Check snapcraft execution interruption with exit codes

* Use f-string and format expressions

* Start log

* Consistent use of single/double quotes

* Better loop to extract lines

* Retry on build failures

* Few optimizations

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2020-07-22 16:05:20 -07:00
Adrien Ferrand
b13dfc6437 Do not create the symlink for test assets on Windows if the asset path is already a symlink (#8159) 2020-07-21 01:01:09 +02:00
Adrien Ferrand
d434b92945 Build the DNS plugins snaps (#8129)
Fixes #8041

This PR makes Azure Pipeline build the DNS plugins snaps for the 3 architectures during the CI.

It leverages the existing logic for building the Certbot snap in order to deploy a QEMU environment with Docker, and leverages the local PyPI index to speed up the build when installing `cffi` and `cryptography`.

All DNS plugins snaps are constructed in one unique docker container, in order to save the time required to install the system dependencies upon first start of `snapcraft`, and so speed up significantly the build.

Finally, all `amd64` DNS plugins snaps are built within 6 minutes. For `arm64` and `armhf`, it is around 40 mins: this is quite fast in fact, considering that 14 DNS plugins snaps are built.

However, this is still an extremely heavy task to make the full 3 architectures builds, even for Azure Pipelines and its 10 parallel jobs capability. That is why I make the `arm64` and `armhf` builds be skipped for the `full-test-suite`, and let them run only for `nightly` and `release`. This means however that these builds will not be done for the release branches. If this is a problem, I can put a more elaborate suspend condition to triggers the builds in this case.

All snaps are stored in the pipeline artifacts storage, making them available for publication during a `release` pipeline.

The PR is set as Draft for now, because I use temporarily `pr_test-suite` to validate the packaging jobs when commits are pushed. Once the PR is ready, I will revert it back to the normal configuration (run the standard tests).

* Configure a script to build DNS snaps

* Focus on packaging

* Trigger all architectures

* Add extra index

* Prepare conditional suspend

* Set final suspend logic

* Set final suspend value

* Loop for publication

* Use python3

* Clean before build

* Add a test

* Add test job in Azure

* Preserve env

* Apply normal config for pipelines

* Skip QEMU jobs only for test branches

* Makes snap run tests depends also on the Certbot snap build

* Update .azure-pipelines/templates/jobs/packaging-jobs.yml

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update .azure-pipelines/templates/stages/deploy-stage.yml

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* More accurate way to get the plugin snap name

* Integrate DNS snap tests into certbot-ci

* Fixes

* Update certbot-ci/snap_integration_tests/conftest.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot-ci/snap_integration_tests/conftest.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Clean an _init_.py file

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2020-07-09 11:33:25 -07:00
alexzorin
f743dbec3a certbot: add --preferred-chain (#8080)
* acme: add support for alternative cert. chains

* certbot: add --preferred-chain

* remove support for issuer SKI matching

* show --preferred-chain in "run" help

* warn if no chain matched and it's not a dry-run

* fix existing failing tests

* add unit, integration tests

* bump acme dependency to dev version

* simplify test to avoid py2.7 recursion bug

* add preferred_chain to STR_CONFIG_ITEMS

* reduce preferred_chain warning to info level

* acme: fix some docstrings in .messages

* certbot: fix docstring in crypto_util

* try to fix certbot-nginx acme dep problem
2020-06-30 17:45:39 -07:00
Florian Klink
25e79e4aca tree-wide: use LooseVersion instead of StrictVersion (#8081)
According to `distutils/version.py`, StrictVersion is pretty strict in
what version numbers to accept:

> A version number consists of two or three dot-separated numeric
> components, with an optional "pre-release" tag on the end.  The
> pre-release tag consists of the letter 'a' or 'b' followed by a number.

This assumption already fails for some pretty basic python libraries
itself, like setuptools, also available in `46.1.3.post20200610`, a
completely valid version number according to
https://www.python.org/dev/peps/pep-0440/#post-releases.

There doesn't seem to be a particular reason on why StrictVersion has
been used here, so let's use LooseVersion, to be compatible with these
versions.

Co-authored-by: Adrien Ferrand <adferrand@users.noreply.github.com>
2020-06-19 17:11:35 +02:00
Adrien Ferrand
9cbb13ef04 Run hooks with Powershell on Windows (#7800)
Fixes #7713.

As discussed in #7713, providing a Powershell script as hook for Certbot is not working currently. This is because hooks are run in a `cmd` environment, that recognizes only `.bat` files as valid scripts that can be run from their bare name on command line.

On the other hand, the Powershell both `.bat` and `.ps1` scripts as valid scripts.

This PR makes hooks command be executed by Powershell, instead of `cmd` as `Popen` does by default when `shell=true` is used. It also modifies the tests to handle this new environment, in particular in term of encoding (UTF-16-LE is the default one in Powershell).

* Run hooks in powershell on Windows

* Fix hook test

* Fallback to unittest.mock

* In fact, shell_cmd as a list of str could not work. Declare only str as acceptable input for shell_cmd.

* Added changelog
2020-04-27 09:38:30 -07:00
Brad Warren
4cce3458f3 Avoid deleting the workspace twice. (#7923) 2020-04-23 23:29:16 +02:00
Karan Suthar
8e4dc0a48c Minor bugfixes (#7891)
* Fix dangerous default argument

* Remove unused imports

* Remove unnecessary comprehension

* Use literal syntax to create data structure

* Use literal syntax instead of function calls to create data structure

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2020-04-13 10:41:39 -07:00
Brad Warren
5992d521e2 Print boulder logs when boulder setup fails (#7885)
This is part of https://github.com/certbot/certbot/issues/7303.

* Print boulder logs if boulder fails to start

* Print description and fix command.

* Change output to stderr.
2020-04-07 15:26:19 -07:00
Brad Warren
6df90d17ae Wait 5 minutes for boulder to start. (#7864) 2020-03-25 14:50:13 -07:00
Adrien Ferrand
809cb516c9 Fix acme compliance to RFC 8555 (#7176)
This PR is an alternative to #7125.

Instead of disabling the strict mode on Pebble, this PR fixes the JWS payloads regarding RFC 8555 to be compliant, and allow certbot to work with Pebble v2.1.0+.

* Fix acme compliance to RFC 8555.

* Working mixin

* Activate back pebble strict mode

* Use mixin for type

* Update dependencies

* Fix also in fields_to_partial_json

* Update pebble

* Add changelog
2020-03-13 09:56:35 -07:00
Brad Warren
3147026211 Check OCSP as part of determining if the certificate is due for renewal (#7829)
Fixes #1028.

Doing this now because of https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/.

The new `ocsp_revoked_by_paths` function  is taken from https://github.com/certbot/certbot/pull/7649 with the optional argument removed for now because it is unused.

This function was added in this PR because `storage.py` uses `self.latest_common_version()` to determine which certificate should be looked at for determining renewal status at 9f8e4507ad/certbot/certbot/_internal/storage.py (L939-L947)

I think this is unnecessary and you can just look at the currently linked certificate, but I don't think we should be changing the logic that code has always had now.

* Check OCSP status as part of determining to renew

* add integration tests

* add ocsp_revoked_by_paths
2020-03-03 11:07:15 -08:00