1
0
mirror of https://github.com/certbot/certbot.git synced 2026-01-26 07:41:33 +03:00
Commit Graph

258 Commits

Author SHA1 Message Date
Adrien Ferrand
16aad35d31 Fully type certbot-nginx module (#9124)
* Work in progress

* Fix type

* Work in progress

* Work in progress

* Work in progress

* Work in progress

* Work in progress

* Oups.

* Fix typing in UnspacedList

* Fix logic

* Finish typing

* List certbot-nginx as fully typed in tox

* Fix lint

* Fix checks

* Organize imports

* Fix typing for Python 3.6

* Fix checks

* Fix lint

* Update certbot-nginx/certbot_nginx/_internal/configurator.py

Co-authored-by: alexzorin <alex@zor.io>

* Update certbot-nginx/certbot_nginx/_internal/configurator.py

Co-authored-by: alexzorin <alex@zor.io>

* Fix signature of deploy_cert regarding the installer interface

* Update certbot-nginx/certbot_nginx/_internal/obj.py

Co-authored-by: alexzorin <alex@zor.io>

* Fix types

* Update certbot-nginx/certbot_nginx/_internal/parser.py

Co-authored-by: alexzorin <alex@zor.io>

* Precise type

* Precise _coerce possible inputs/outputs

* Fix type

* Update certbot-nginx/certbot_nginx/_internal/http_01.py

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* Fix type

* Remove an undesirable implementation.

* Fix type

Co-authored-by: alexzorin <alex@zor.io>
Co-authored-by: ohemorange <ebportnoy@gmail.com>
2022-01-12 16:36:51 -08:00
Adrien Ferrand
89ccbccff0 Fully type all DNS plugins (#9125)
* Add types in all DNS plugins

* Order imports

* Fix type

* Update certbot-dns-route53/certbot_dns_route53/_internal/dns_route53.py

Co-authored-by: alexzorin <alex@zor.io>

* Clean up imports

Co-authored-by: alexzorin <alex@zor.io>
2021-12-14 12:38:14 +11:00
Adrien Ferrand
cb3e1403cd Fully type certbot-compatibility-test (#9133)
* Finish typing the module

* Use cast

* Precise type
2021-12-14 12:14:11 +11:00
Adrien Ferrand
97d9e2c97d Fully type lock_test.py (#9126)
* Type lock_test.py

* Reconfigure tox

* Fix imports
2021-12-13 14:01:31 +11:00
Adrien Ferrand
aeb7beb1b1 Fully type certbot-ci module (#9120)
* Fully type certbot-ci module

* Fix lint, focus lint

* Add trailing comma

* Remove unused private function

* Type properly for future usages

* Update certbot-ci/certbot_integration_tests/utils/acme_server.py

Co-authored-by: alexzorin <alex@zor.io>

* Cleanup files

* Fix import

* Fix mypy and lint

Co-authored-by: alexzorin <alex@zor.io>
2021-11-30 08:24:39 +11:00
Adrien Ferrand
86406ab63a Add type annotations to the certbot package (part 4) (#9087)
* Extract from #9084

* Cast/ignore types during the transition

* Remove useless casts and type ignore directives

* Fix lint

* Fix a cast

* Mandatory typing for certbot packages

* Update certbot/certbot/_internal/plugins/disco.py

Co-authored-by: alexzorin <alex@zor.io>

* Remove unused type import

* Fix iterator type

* Fix type

* Fix types in selection

Co-authored-by: alexzorin <alex@zor.io>
2021-11-26 09:00:03 +11:00
Adrien Ferrand
a0f22d21ce Add type annotations to the acme project (#9036)
* Start more types

* Second run

* Work in progress

* Types in all acme module

* Various fixes

* Various fixes

* Final fixes

* Disallow untyped defs for acme project

* Fix coverage

* Remote unecessary type ignore

* Use Mapping instead of Dict as input whenever it is possible

* Update acme/acme/client.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update acme/acme/client.py

Co-authored-by: alexzorin <alex@zor.io>

* Various fixes

* Fix code

* Fix code

* Update acme/acme/client.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update acme/acme/challenges.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update acme/acme/client.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Fix deactivate_registration and simplify signature of update_registration

* Do not leak personal data during account deactivation

* Clean more Dicts

* New fix to not leak contact field in the account deactivation payload.

* Add ignore for python 3.6 type check

* Revert "Add ignore for python 3.6 type check"

This reverts commit da7338137b.

* Let's find a smarter way than "type: ignore"

* Update certbot/certbot/_internal/account.py

Co-authored-by: alexzorin <alex@zor.io>

* Fix an annotation

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Co-authored-by: alexzorin <alex@zor.io>
2021-10-25 09:43:21 +11:00
alexzorin
694c03bd6a lower coverage threshold for rfc2136 integration (#9006) 2021-08-25 07:40:26 -07:00
Adrien Ferrand
acf48df979 Use latest version of mypy (#8992)
Fixes #8899

This PR removes the pinning upper limit of mypy currently set to <0.900 and adds the required types-* stub packages to make recent versions of mypy work.

* Unpin mypy

* Improve type in TempHandler

* Add types
2021-08-17 10:52:57 -07:00
Adrien Ferrand
8133d3e70a Fix python-augeas failure on Windows (v2) (#8951)
This PR is a new approach for fixing #8732 based on the discussions occurred in the first PR #8877.

This PR upgrades python-augeas to the latest version, and avoids tests failure of Windows because of this upgrade. To do so it leverages the [tox multi-platform feature](https://tox.readthedocs.io/en/latest/example/platform.html) and modifications to `tools/venv.py` in order to not install and not test `certbot-apache` on Windows.

* Unpin python-augeas and upgrade current pinnings

* Do not install certbot-apache in Windows dev environments

* Introduce tox specific win packages + remove certbot compatibility on windows

* Add libaugeas to sphinx build

* Redefine lint and mypy targets

* Keep the lint and mypy environments
2021-07-29 11:25:25 -07:00
Brad Warren
08839758bd Finish pinning system rewrite (#8934)
* add oldest pyproject.toml file that works

* make single oldest_constraints.txt file

* remove unused merge_requirements.py

* remove unused import

* make conditional right

* simplify pip_install.py

* fix typo

* bump min dns-lexicon dependency

* fix zope import warning

* pin back wheel

* refactor pinning script

* Add oldest script.

* add pip comment

* add pipstrap extra

* simplify pinning scripts

* remove pipstrap extra

* update contributing

* Add design doc

* Update tools/pinning/DESIGN.md

Co-authored-by: ohemorange <erica@eff.org>

* Update tools/pinning/DESIGN.md

Co-authored-by: ohemorange <erica@eff.org>

* Update tools/pinning/DESIGN.md

Co-authored-by: ohemorange <erica@eff.org>

* Update tools/pinning/DESIGN.md

Co-authored-by: ohemorange <erica@eff.org>

* rename normal to current

* no dummies

* script improvements

* mention need to update setup.py

* try and clarify poetry behavior

* tweak section title

Co-authored-by: ohemorange <erica@eff.org>
2021-07-22 12:00:30 -07:00
Brad Warren
18ea72faf1 Split out testing extras (#8893)
* split out test extras

* update extras and regenerate pinnings

* pin back mypy
2021-06-11 13:17:50 -07:00
Brad Warren
c372dd8aee Remove local-oldest-requirements files (#8863)
This is part of https://github.com/certbot/certbot/issues/8787. I got a +1 from our packagers at major distros in https://github.com/certbot/certbot/issues/8761.

* remove local-oldest-requirements files

* fix tests

* fix some oldest tests

* list packages on one line in tox.ini

* add changelog entry
2021-06-01 14:46:06 -07:00
Brad Warren
dd0e590de3 Make a test farm tests package (#8821)
Fixes https://github.com/certbot/certbot/issues/8781.

This PR makes our test farm tests into a normal package so it and its dependencies can be tracked and installed like our other packages.

Other noteworthy changes in this PR:

* Rather than continuing to place logs in your CWD, they're placed in a temporary directory that is printed to the terminal.
*  `tests/letstest/auto_targets.yaml` was deleted rather than renamed because the file is no longer used.

* make a letstest package

* remove deleted deps

* fix letstest install

* add __init__.py

* call main

* Explicitly mention activating venv

* rerename file

* fix version.py path

* clarify "this"

* Use >= instead of caret requirement
2021-05-03 17:42:30 -07:00
ohemorange
f339d23e54 Remove further references to certbot-auto in the repo (#8814)
* Move version.py to tests/letstest since it's used by test_sdists.sh

* Delete unused components of certbot-auto

* Remove test_leauto_upgrades.sh and references to it

* Remove test_letsencrypt_auto_certonly_standalone.sh and references to it

* Remove outstanding references to certbot-auto

* Remove references to letsencrypt-auto

* find certbot in the correct directory

* delete letsencrypt-auto-source line from .isort.cfg since that directory no longer contains any python code

* remove (-auto) from certbot(-auto)

* delete line from test

* Improve style for version.py
2021-04-27 15:27:21 -07:00
Brad Warren
c06e40dbef Update certbot-auto modification checks (#8805)
* revert changes to letsencrypt-auto-source/le-auto

* update modification tests
2021-04-26 13:50:10 -07:00
Adrien Ferrand
c438a397a0 Enable mypy strict mode (#8766)
Built on top of #8748, this PR reenables mypy strict mode and adds the appropriate corrections to pass the types checks.

* Upgrade mypy

* First step for acme

* Cast for the rescue

* Fixing types for certbot

* Fix typing for certbot-nginx

* Finalize type fixes, configure no optional strict check for mypy in tox

* Align requirements

* Isort

* Pylint

* Protocol for python 3.6

* Use Python 3.9 for mypy, make code compatible with Python 3.8<

* Pylint and mypy

* Pragma no cover

* Pythonic NotImplemented constant

* More type definitions

* Add comments

* Simplify typing logic

* Use vararg tuple

* Relax constraints on mypy

* Add more type

* Do not silence error if target is not defined

* Conditionally import Protocol for type checking only

* Clean up imports

* Add comments

* Align python version linting with mypy and coverage

* Just ignore types in an unused module

* Add comments

* Fix lint

* Work in progress

* Finish type control

* Isort

* Fix pylint

* Fix imports

* Fix cli subparser

* Some fixes

* Coverage

* Remove --no-strict-optional (obviously...)

* Update certbot-apache/certbot_apache/_internal/configurator.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/certbot/_internal/display/completer.py

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>

* Cleanup dns_google

* Improve lock controls and fix subparser

* Use the expected interfaces

* Fix code

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
2021-04-05 15:04:21 -07:00
Adrien Ferrand
06a53cb7df Upgrade to mypy 0.812 (#8748)
Fixes #8425

This PR upgrades mypy to the latest version available, 0.812.

Given the advanced type inference capabilities provided by this newer version, this PRs also fixes various type inconsistencies that are now detected. Here are the non obvious changes done to fix types:
* typing in mixins has been solved using `Protocol` classes, as recommended by mypy (https://mypy.readthedocs.io/en/latest/more_types.html#mixin-classes, https://mypy.readthedocs.io/en/stable/protocols.html)
* `cast` when we are playing with `Union` types

This PR also disables the strict optional checks that have been enable by default in recent versions of mypy. Once this PR is merged, I will create an issue to study how these checks can be enabled.

`typing.Protocol` is available only since Python 3.8. To keep compatibility with Python 3.6, I try to import the class `Protocol` from `typing`, and fallback to assign `object` to `Protocol` if that fails. This way the code is working with all versions of Python, but the mypy check can be run only with Python 3.8+ because it needs the protocol feature. As a consequence, tox runs mypy under Python 3.8.

Alternatives are:
* importing `typing_extensions`, that proposes backport of newest typing features to Python 3.6, but this implies to add a dependency to Certbot just to run mypy
* redesign the concerned classes to not use mixins, or use them differently, but this implies to modify the code itself even if there is nothing wrong with it and it is just a matter of instructing mypy to understand in which context the mixins can be used
* ignoring type for these classes with `# type: ignore` but we loose the benefit of mypy for them

* Upgrade mypy

* First step for acme

* Cast for the rescue

* Fixing types for certbot

* Fix typing for certbot-nginx

* Finalize type fixes, configure no optional strict check for mypy in tox

* Align requirements

* Isort

* Pylint

* Protocol for python 3.6

* Use Python 3.9 for mypy, make code compatible with Python 3.8<

* Pylint and mypy

* Pragma no cover

* Pythonic NotImplemented constant

* More type definitions

* Add comments

* Simplify typing logic

* Use vararg tuple

* Relax constraints on mypy

* Add more type

* Do not silence error if target is not defined

* Conditionally import Protocol for type checking only

* Clean up imports

* Add comments

* Align python version linting with mypy and coverage

* Just ignore types in an unused module

* Add comments

* Fix lint
2021-04-02 11:54:40 -07:00
Brad Warren
c668172ef0 merge dev and dev3 (#8639) 2021-02-04 21:31:47 +11:00
Brad Warren
2e33aec8a8 add tests with external mock library 2021-01-29 15:31:11 -08:00
Brad Warren
00235d3807 Switch oldest tests to Python 3 (#8590)
Fixes https://github.com/certbot/certbot/issues/8580.

With this PR, it should now be possible to run the oldest tests natively on Linux, at least when using an older version of Python 3, which hasn't been possible in a long time. Unfortunately, this isn't possible on macOS which I opened https://github.com/certbot/certbot/issues/8589 to track.

You can see the full test suite running with these changes at https://dev.azure.com/certbot/certbot/_build/results?buildId=3283&view=results.

I took the version numbers for the packages I updated by searching for the oldest version of the dependency I think we should try and support based on the updated comments at the top of `oldest_constraints.txt`. While kind of annoying, I think it'd be a good idea for the reviewer to double check that I didn't make a mistake with the versions I used here.

To find these versions, I used https://packages.ubuntu.com, https://packages.debian.org, and a CentOS 7 Docker image with EPEL 7 installed. For the latter, not all packages are available in Python 3 yet (which is something Certbot's EPEL package maintainers are working on) and in that case I didn't worry about the system because I think they can/will package the newest version available. If they end up hitting any issues here when trying to package Certbot on Python 3, we can always work with them to fix it.

* remove py27 from oldest name

* update min cryptography version

* remove run_oldest_tests.sh

* upgrade setuptools and pyopenssl

* update cffi, pyparsing, and idna

* expand oldest_constraints comments

* clarify oldest comment

* update min configobj version

* update min parsedatetime version

* quote tox env name

* use Python 3.6 in the oldest tests

* use Python 3.6 for oldest integration tests

* properly pin asn1crypto

* update min six version

* set basepython for a nicer error message

* remove outdated python 2 oldest constraints
2021-01-25 12:59:14 -08:00
Mads Jensen
96a05d946c Added certbot-ci to lint section. Silenced and fixed linting warnings. (#8450) 2020-12-16 20:34:12 +01:00
alexzorin
38017473c5 add coverage testing to dns-rfc2136 integration (#8469)
* add coverage testing to dns-rfc2136 integration

* add coverage rule for certbot/* as well
2020-12-06 09:23:33 +01:00
Adrien Ferrand
ff3a07dca3 Deprecate certbot-auto and remove tests
* Completely deprecate certbot-auto

* DeaDeactivate centos6/oraclelinux6 tests

* Remove tests assets

* Remove another test

* Revert "Remove tests assets"

This reverts commit e603afe6c4.
2020-12-02 09:48:57 -08:00
alexzorin
90557921e3 Add certbot-dns-rfc2136 integration testing (#8448)
* tests: add certbot-dns-rfc2136 integration tests

* dont use 'with' form of socket.socket

fixes py2 crash

* address some feedback:

- conftest: make DNS server a global resource
- conftest: add dns_xdist parameter into node config
- conftest: add --dns-server=bind flag
- conftest: if configured, point the ACME server to the DNS server
- dnsserver: make it sort-of compatible with xdist (future-proofing)
- context: parameterize dns-rfc2136 credentials file (future proofing)
- context: reduce dns-rfc2136 propagation time to speed up tests
- tox: add a integration-dns-rfc2136 target
- rfc2136: add a test/zone for subdelegation
- rfc2136: skip tests if no DNS server is configured

* try add integration-dns-rfc2136 to CI

* mock recursive dns via RPZ

* update --dns-server args and tox.ini args

* address more feedback:

- dns_server: rename rfc2136 creds file to .tpl
- dns_server: dont vary dns server port, instead we will vary zone names (#8455)
- dns_server: log error if bind9 fails to stop cleanly
- dns_server: replace assert with raise
- context: remove redundant _worker_id
- context: remove redundant cleanup override
- context: fix seek/flush in credentials context manager
- context: rename skip_if_no_server -> ...bind_server
- context: add newline EOF

* conftest: document _setup_primary_node sideeffects

* ci: rfc2136-integration from standard->nightly

* fix _stop_bind (function was renamed to stop)

* ignore errors from shutil.rmtree during cleanup

* dns_server: check for crash while polling

* remove --dry-run from rfc2136 test
2020-11-17 09:27:27 +01:00
Brad Warren
c102ca66c3 Write a replacement for pipstrap (#8401)
* Add a new, simplified version of pipstrap.

* Use tools/pipstrap.py

* Uncomment code

* Refactor pip_install.py and provide hashes.

* Fix test_sdists.sh.

* Make code work on Python 2.

* Call strip_hashes.py using Python 3.

* Pin the oldest version of httplib2 used in distros

* Strip enum34 dependency.

* Remove pip pinnings from dev_constraints.txt

* Correct pipstrap docstring.

* Don't set working_dir twice.

* Add comments
2020-11-06 11:17:41 +01:00
Brad Warren
cb916a0682 Deprecate certbot-auto on Debian systems (#8354)
Fixes #8294.

* Deprecate certbot-auto on Debian systems.

* Add changelog entry.

* Remove le_auto_xenial test.

* Update certbot-auto test farm tests.

* Add comments explaining expected behavior.
2020-10-20 16:25:20 -07:00
Brad Warren
160b209394 Automatically retry test farm tests (#8325)
Fixes #8317.

* move retry to script

* Retry test farm tests.

* Fix retry path.
2020-09-30 17:05:52 -07:00
Adrien Ferrand
79297ef5cb Invoke pipstrap in tox and during the CI (#8316)
Partial fix for #8256

This PR makes tox calls pipstrap before any commands is executed, and Azure Pipelines calls pipstrap when appropriate (when an actual call to pip is done). 

* Invoke pipstrap in tox and during the CI

* Set default value for PYTHON_VERSION and always set python interpreter

* Set Python for snaps_build also

* Fix the build for Windows installer

* Add a warning comment for pinned versions in pipstrap

* Rebuild letsencrypt-auto

* Same version than the installer build

* Let's update to latest pip for installer tests
2020-09-24 17:12:12 -07:00
Cameron Steel
9a72db5b9b Convert http links to https (#8287)
* Convert http links to https

* Fix remaining links
2020-09-23 19:36:55 +02:00
ohemorange
acb6d34c5f Update test farm tests to stop using certbot-auto (#8207)
* Create bootstrap script

* Delete a whole bunch of the bootstrap script

* modify test_tests to use new script

* put python version checking in back in

* add x

* call the venv creation from inside the bootstrap

* add targets back

* modify test_apache2 to use new format

* shouldn't need virtualenv on rhel

* readd targets

* Update test_sdists to use new script

* move setting up venv back out of script so it's not run with sudo

* take venv3.py call out of bootstrap in all scripts

* add additional python3-devel pkg name

* fix test_sdists

* enable additional rhel7 repos

* clean up code and comments

* Update tests and instructions to use auto_targets.yaml with test_leauto_upgrades.sh and test_letsencrypt_auto_certonly_standalone.sh

* only install python3-devel.x86_64 for rhel7

* Upgrade python version for debian in test_apache2.sh

* don't run test_tests or test_sdists on debian 9 or ubuntu 16.04

* Add 20.04 and 20.04 arm images to targets.yaml

* use pyenv to upgrade to python3.5

* remove arm64 instance because it's having auth trouble

* correct pyenv usage on ubuntu

* add arm64 target to targets.yaml

* replace debian 9 arm64 with ubuntu 20

* don't try to upgrade a perfectly good python version

* let's just add ubuntu20 to apache2_targets while we're here

* uncomment test_apache2

* move adding python3-devel.x86_64 to bootstrap_os_packages to avoid potential race condition

* no need to specify the arch once extra rhel7 repos enabled

* explicitly specify python3

* don't fail if we can't enable rhel7 extras

* capture python36-devel as well
2020-08-18 10:07:27 -07:00
Brad Warren
8574313841 Remove final jessie references outside of cb-auto. (#8194) 2020-08-07 13:35:21 -07:00
Adrien Ferrand
8a3a8c7097 Migrate the CI pipeline from Travis to Azure Pipeline (#8098)
Fixes #8071 and fixes https://github.com/certbot/certbot/issues/8110.

This PR migrates every job from Travis in Azure Pipeline.

This PR essentially converts the Travis jobs into Azure Pipeline with a complete iso-fonctionality (or I made a mistake). The jobs are added in the relevant existing pipelines (`main`, `nightly`, `advanced-test`, `release`). A global refactoring thanks to the templating system is done to reduce greatly the verbosity of the pipeline descriptions.

A specific feature (not present in Travis) is added: the stage `On_Failure`. Using directly the Mattermost API, it allows to notify pipeline failure in a Mattermost channel with a link to the failed pipelines without the need to authenticate to Microsoft.

See https://github.com/certbot/certbot/pull/8098#issuecomment-649873641 for the post merge actions to do at the end of this work.
2020-07-02 15:01:21 -07:00
Brad Warren
335894ab3b Merge snap code into the Certbot repo
* merge .gitignore

* Move snapcraft.yml up one level.

* update source

* move test.sh to tox.ini

* use new tox.ini in .travis.yml

* move snap build code

* make script executable

* remove unused python3-dev

* don't use deprecated classic flag

* go back to stable channel

* add nginx in snap addons

* add deploy steps

* Add comments explaining external tox envs.

* error if not in CI

* don't use --depth

* remove old .travis.yml

* Add big comment about SNAP_TOKEN.

* Set all_branches: true.

* Add repo setting.

* run travis on tags

* Add more documenting comments to .travis.yml.
2020-04-24 13:47:36 -07:00
Brad Warren
859dc38cb9 Consolidate cover envs and default to py3-cover (#7905)
* Consolidate cover envs and default to py3-cover

* use py38 for code coverage in Travis

* Disable coverage on Python < 3.6 line.
2020-04-16 08:59:40 -07:00
m0namon
1285297b23 [Apache v2] Load apacheconfig tree and gate related tests (#7710)
* Load apacheconfig dependency, gate behind flag

* Bump apacheconfig dependency to latest version and install dev version of apache for coverage tests

* Move augeasnode_test tests to more generic parsernode_test

* Revert "Move augeasnode_test tests to more generic parsernode_test"

This reverts commit 6bb986ef78.

* Mock AugeasNode into DualNode's place, and run augeasnode tests exclusively on AugeasNode

* Don't calculate coverage for skeleton functions

* clean up helper function in augeasnode_test
2020-03-23 17:05:22 -07:00
Brad Warren
fd64c8c33b Remove letshelp-certbot (#7761)
* remove references to letshelp

* remove letshelp files

* Remove line continuation

Co-authored-by: ohemorange <ebportnoy@gmail.com>
2020-02-14 17:19:19 -08:00
ohemorange
c5a2ba03da Merge pull request #7735 from certbot/apache-parser-v2
[Apache v2] Merge apache-parser-v2 feature branch back to master
2020-02-06 15:29:28 -08:00
Brad Warren
cc764b65c1 Set recreate = true in tox.ini. (#7746)
Fixes #7745.
2020-02-05 14:37:39 -08:00
Joona Hoikkala
882335c7ec Merge remote-tracking branch 'origin/master' into ap2_to_master 2020-01-30 17:08:16 +02:00
Adrien Ferrand
e84ed49c56 Fix certbot-auto regarding python 3.4 -> python 3.6 migration for CentOS 6 users (#7519)
* Revert "Add back Python 3.4 support (#7510)"

This reverts commit 9b848b1d65.

* Fix certbot-auto

* Use a more consistent way to enable rh-python36

* Avoid to call CompareVersions unecessarily

* Control rh-python36 exit code

* Fix travis config

* Remove vscode config

* Ignore vscode

* Fix merge conflicts regarding #7587 (#70)

* Add changelog entry

* Finish sentence

* Update certbot/CHANGELOG.md

Co-Authored-By: Joona Hoikkala <joohoi@users.noreply.github.com>

* Update letsencrypt-auto-source/tests/centos6_tests.sh

Co-Authored-By: Joona Hoikkala <joohoi@users.noreply.github.com>

* Update letsencrypt-auto-source/tests/centos6_tests.sh

Co-Authored-By: Joona Hoikkala <joohoi@users.noreply.github.com>

* Update letsencrypt-auto-source/tests/centos6_tests.sh

Co-Authored-By: Joona Hoikkala <joohoi@users.noreply.github.com>

* Update letsencrypt-auto-source/tests/centos6_tests.sh

Co-Authored-By: Joona Hoikkala <joohoi@users.noreply.github.com>

* Update letsencrypt-auto-source/tests/centos6_tests.sh

Co-Authored-By: Joona Hoikkala <joohoi@users.noreply.github.com>

* Update comments

* Improve warning message

* Update changelog

Co-authored-by: Joona Hoikkala <joohoi@users.noreply.github.com>
2020-01-13 09:24:41 +01:00
sydneyli
0e78436b05 [Apache v2] Add apacheconfig as a dependency (#7643)
* Add apacheconfig as a dependency.

* Change apacheconfig to a dev dependency

* Bump apacheconfig dep to 0.3.1
2020-01-07 09:57:43 -08:00
Adrien Ferrand
9e5bca4bbf Lint certbot code on Python 3, and update Pylint to the latest version (#7551)
Part of #7550

This PR makes appropriate corrections to run pylint on Python 3.

Why not keeping the dependencies unchanged and just run pylint on Python 3?
Because the old version of pylint breaks horribly on Python 3 because of unsupported version of astroid.

Why updating pylint + astroid to the latest version ?
Because this version only fixes some internal errors occuring during the lint of Certbot code, and is also ready to run gracefully on Python 3.8.

Why upgrading mypy ?
Because the old version does not support the new version of astroid required to run pylint correctly.

Why not upgrading mypy to its latest version ?
Because this latest version includes a new typshed version, that adds a lot of new type definitions, and brings dozens of new errors on the Certbot codebase. I would like to fix that in a future PR.

That said so, the work has been to find the correct set of new dependency versions, then configure pylint for sane configuration errors in our situation, disable irrelevant lintings errors, then fixing (or ignoring for good reason) the remaining mypy errors.

I also made PyLint and MyPy checks run correctly on Windows.

* Start configuration

* Reconfigure travis

* Suspend a check specific to python 3. Start fixing code.

* Repair call_args

* Fix return + elif lints

* Reconfigure development to run mainly on python3

* Remove incompatible Python 3.4 jobs

* Suspend pylint in some assertions

* Remove pylint in dev

* Take first mypy that supports typed-ast>=1.4.0 to limit the migration path

* Various return + else lint errors

* Find a set of deps that is working with current mypy version

* Update local oldest requirements

* Remove all current pylint errors

* Rebuild letsencrypt-auto

* Update mypy to fix pylint with new astroid version, and fix mypy issues

* Explain type: ignore

* Reconfigure tox, fix none path

* Simplify pinning

* Remove useless directive

* Remove debugging code

* Remove continue

* Update requirements

* Disable unsubscriptable-object check

* Disable one check, enabling two more

* Plug certbot dev version for oldest requirements

* Remove useless disable directives

* Remove useless no-member disable

* Remove no-else-* checks. Use elif in symetric branches.

* Add back assertion

* Add new line

* Remove unused pylint disable

* Remove other pylint disable
2019-12-10 14:12:50 -08:00
ohemorange
6c1dfe43c7 Refactor tests out of packaged module for apache plugin (#7607)
Part of #7593.

* Refactor tests out of packaged module for apache plugin

* Exclude pycache and .py[cod]

* Change tests path in tox.ini
2019-11-27 09:57:35 -08:00
ohemorange
4abd81e218 Refactor certbot/ and certbot/tests/ to use the same structure as the other packages (#7544)
Summary of changes in this PR:
- Refactor files involved in the `certbot` module to be of a similar structure to every other package; that is, inside a directory inside the main repo root (see below).
- Make repo root README symlink to `certbot` README.
- Pull tests outside of the distributed module.
- Make `certbot/tests` not be a module so that `certbot` isn't added to Python's path for module discovery.
- Remove `--pyargs` from test calls, and make sure to call tests from repo root since without `--pyargs`, `pytest` takes directory names rather than package names as arguments.
- Replace mentions of `.` with `certbot` when referring to packages to install, usually editably.
- Clean up some unused code around executing tests in a different directory.
- Create public shim around main and make that the entry point.

New directory structure summary:
```
repo root ("certbot", probably, but for clarity all files I mention are relative to here)
├── certbot
│   ├── setup.py
│   ├── certbot
│   │   ├── __init__.py
│   │   ├── achallenges.py
│   │   ├── _internal
│   │   │   ├── __init__.py
│   │   │   ├── account.py
│   │   │   ├── ...
│   │   ├── ...
│   ├── tests
│   │   ├── account_test.py
│   │   ├── display
│   │   │   ├── __init__.py
│   │   │   ├── ...
│   │   ├── ... # note no __init__.py at this level
│   ├── ...
├── acme
│   ├── ...
├── certbot-apache
│   ├── ...
├── ...
```

* refactor certbot/ and certbot/tests/ to use the same structure as the other packages

* git grep -lE "\-e(\s+)\." | xargs sed -i -E "s/\-e(\s+)\./-e certbot/g"

* git grep -lE "\.\[dev\]" | xargs sed -i -E "s/\.\[dev\]/certbot[dev]/g"

* git grep -lE "\.\[dev3\]" | xargs sed -i -E "s/\.\[dev3\]/certbot[dev3]/g"

* Remove replacement of certbot into . in install_and_test.py

* copy license back out to main folder

* remove linter_plugin.py and CONTRIBUTING.md from certbot/MANIFEST.in because these files are not under certbot/

* Move README back into main folder, and make the version inside certbot/ a symlink

* symlink certbot READMEs the other way around

* move testdata into the public api certbot zone

* update source_paths in tox.ini to certbot/certbot to find the right subfolder for tests

* certbot version has been bumped down a directory level

* make certbot tests directory not a package and import sibling as module

* Remove unused script cruft

* change . to certbot in test_sdists

* remove outdated comment referencing a command that doesn't work

* Install instructions should reference an existing file

* update file paths in Dockerfile

* some package named in tox.ini were manually specified, change those to certbot

* new directory format doesn't work easily with pyargs according to http://doc.pytest.org/en/latest/goodpractices.html#tests-as-part-of-application-code

* remove other instance of pyargs

* fix up some references in _release.sh by searching for ' . ' and manual check

* another stray . in tox.ini

* fix paths in tools/_release.sh

* Remove final --pyargs call, and now-unnecessary call to modules instead of local files, since that's fixed by certbot's code being one layer deeper

* Create public shim around main and make that the entry point

* without pyargs, tests cannot be run from an empty directory

* Remove cruft for running certbot directly from main

* Have main shim take real arg

* add docs/api file for main, and fix up main comment

* Update certbot/docs/install.rst

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Fix comments in readthedocs requirements files to refer to current package

* Update .[docs] reference in contributing.rst

* Move plugins tests to certbot tests directory

* add certbot tests to MANIFEST.in so packagers can run python setup.py test

* move examples directory inside certbot/

* Move CHANGELOG into certbot, and create a top-level symlink

* Remove unused sys and logging from main shim

* nginx http01 test no longer relies on certbot plugins common test
2019-11-25 14:28:05 -08:00
Brad Warren
9b848b1d65 Add back Python 3.4 support (#7510)
* Revert "Deprecation warnings for Python 3.4 (#7378)"

This reverts commit 6fcdfb0e50.

* Revert "Migrate certbot-auto users on CentOS 6 to Python 3.6 (#7268)"

This reverts commit e19b2e04c7.

* add changelog entry

* keep mona in authors
2019-11-05 16:45:08 -08:00
Adrien Ferrand
e19b2e04c7 Migrate certbot-auto users on CentOS 6 to Python 3.6 (#7268)
Fixes #7007

Python 3.4 is [EOL](https://www.python.org/dev/peps/pep-0429/), and only Python 3.x version available for CentOS 6 through EPEL is this version, and so is used by `certbot-auto`, the only official way to install Certbot on this platform.

This unpleasant situation becomes a little more uncomfortable, considering that the newest `pip` version (19.2) [just dropped Python 3.4 support](https://github.com/pypa/pip/issues/6685) and will refuse to start on this Python version. We can expect a lot of dependencies to follow this path now.

One direct result of this situation is that a fix to support correctly the ARM platforms requires to upgrade `pip` to 19.2 for `certbot-auto`. So this is not possible right now.

Then, let's upgrade Certbot instances on CentOS 6 to a supported version of Python 3.

This PR proposes a new bootstrap approach for CentOS 6 platform, `BootstrapRpmPython3Legacy`, that will install Python 3.6 from [SCL](https://www.softwarecollections.org) (the latest one available for now on CentOS 6). In term of Python 3 specific bootstrap methods, I take the occasion here to completely separate the bootstrap of CentOS 6 as a legacy system, from the RPM-based newest systems (like Fedora 29+) that are simply dropping support for Python 2.x. This is in prevision of future migration for all systems on Python 3.x, that is a different problematic than supporting old systems.

* Add logic

* Rebuilt letsencrypt-auto

* Fix logic

* Focus on specific packages

* Maintain PATH for further invocations of letsencrypt-auto after bootstrap.

* Various corrections

* Fix farm test for RHEL6

* Working centos6 letsencrypt-auto self tests

* Fix test_sdist for CentOS 6

* Corrections

* Work in progress

* Working configuration

* Fix typo

* Remove EPEL. Add a test.

* Update letsencrypt-auto-source/letsencrypt-auto.template

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Improvements after review

* Improvements

* Add a comment

* Add a test

* Update a test

* Corrections

* Update function return

* Work in progress

* Correct behavior on oracle linux 6.

* Corrections

* Rebuild script

* Add letsencrypt-auto tests for oraclelinux6

* Update tox.ini

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update letsencrypt-auto-source/letsencrypt-auto

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update letsencrypt-auto-source/letsencrypt-auto.template

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update letsencrypt-auto-source/letsencrypt-auto

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update letsencrypt-auto-source/letsencrypt-auto

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update letsencrypt-auto-source/letsencrypt-auto.template

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Remove specific code for scientific linux

* Change some variables names

* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Various corrections

* Fix tests

* Add a comment

* Update message

* Fix test message

* Update letsencrypt-auto-source/letsencrypt-auto.template

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update letsencrypt-auto-source/letsencrypt-auto

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update letsencrypt-auto-source/letsencrypt-auto

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update scripts

* More focused assertion

* Add back a test

* Update script

* Update letsencrypt-auto-source/letsencrypt-auto.template

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update letsencrypt-auto-source/letsencrypt-auto.template

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Check quiet mode

* Add changelog

* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
2019-10-30 10:39:45 -07:00
Andrzej Górski
9da07590bd Remove --fast from the test farm tests (#7427) 2019-10-08 21:24:55 +02:00
Adrien Ferrand
ab76834100 [Windows|Linux] Forbid os.stat and os.fstat (#7325)
Fixes #7212

This PR forbid os.stat and os.fstat, and fix or provide alternatives to avoid its usage in certbot outside of certbot.compat.filesystem.

* Reimplement private key mode propagation

* Remove other os.stat

* Remove last call of os.stat in certbot package

* Forbid stat and fstat

* Implement mode comparison checks

* Add unit tests

* Update certbot/compat/filesystem.py

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update certbot/compat/filesystem.py

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Handle case where multiple ace concerns a given SID in has_min_permissions

* Add a new test scenario

* Add a simple test for has_same_ownership

* Fix name function

* Add a comment explaining an ACE structure

* Move a test in its dedicated class

* Improve a message error

* Calculate has_min_permission result using effective permission rights to be more generic.

* Change an exception message

* Add comments, avoid to skip a test.

* Update certbot/compat/filesystem.py

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
2019-09-06 14:30:25 -07:00
Adrien Ferrand
b67fda8832 Fix integration tests on Windows (#7271)
* Fix account_tests

* Fix hook executable test

* Remove the temporary decorator @broken_on_windows

* Fix util_test

* No broken unit test on Windows anymore

* More elegant mock

* Fix context manager

* Fix lint

* Fix mypy

* Adapt coverage

* Corrections

* Fix lint

* Adapt coverage

* Update certbot/tests/compat/filesystem_test.py

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Update util_test.py

* Fix pylint

* Forbid os.access

* Update os_test.py

* Update os.py

* Fix lint

* Update filesystem.py

* Update filesystem.py

* Update filesystem.py

* Update os.py

* Start fixing tests

* Platform independent hooks

* Fix probe fd close

* Add broken_on_windows for integration tests

* Fix a lot of tests

* Use a python hook script, to prepare cross-platform

* New approach to be compliant with Linux and Windows on hook scripts

* New tests fixed

* Test for permissions on Windows

* Permissions comparison for Windows

* No broken tests in certbot core anymore

* Change mode

* Specific config for appveyor

* Use forked pebble for now

* Various fixes

* Assert file permissions for world on private keys

* Clean code

* Fix several things

* Add integration target

* Optimize integration env

* Re-enable all AppVeyor envs

* Use again official pebble

* Update pebble_artifacts.py

* Set PYTEST_ADDOPTS silently

* Update appveyor.yml

* Pin pywin32 for tests, give a minimal requirement for certbot.

* Remove injection of nginx in PATH

* Clean debug code

* Various cleanup, ensure to remove workspace after tests

* Update tox target

* Improve assertions. Control the keyword echoed in hooks

* Fix for virtualenv on Python 3.7.4 for Windows

* Update certbot-ci/certbot_integration_tests/certbot_tests/assertions.py

Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>

* Add conditionally pywin in certbot-ci like in certbot
2019-08-07 00:02:16 +02:00