I don't fully understand why, but since I updated my macbook to macOS Catalina, the test script currently fails to run for me with the versions of our dependencies we have pinned. Updating the dependencies solves the problem though and you can see Travis also successfully running tests with these new dependencies at https://travis-ci.com/certbot/certbot/builds/150573696.
Part of #7550
This PR makes appropriate corrections to run pylint on Python 3.
Why not keeping the dependencies unchanged and just run pylint on Python 3?
Because the old version of pylint breaks horribly on Python 3 because of unsupported version of astroid.
Why updating pylint + astroid to the latest version ?
Because this version only fixes some internal errors occuring during the lint of Certbot code, and is also ready to run gracefully on Python 3.8.
Why upgrading mypy ?
Because the old version does not support the new version of astroid required to run pylint correctly.
Why not upgrading mypy to its latest version ?
Because this latest version includes a new typshed version, that adds a lot of new type definitions, and brings dozens of new errors on the Certbot codebase. I would like to fix that in a future PR.
That said so, the work has been to find the correct set of new dependency versions, then configure pylint for sane configuration errors in our situation, disable irrelevant lintings errors, then fixing (or ignoring for good reason) the remaining mypy errors.
I also made PyLint and MyPy checks run correctly on Windows.
* Start configuration
* Reconfigure travis
* Suspend a check specific to python 3. Start fixing code.
* Repair call_args
* Fix return + elif lints
* Reconfigure development to run mainly on python3
* Remove incompatible Python 3.4 jobs
* Suspend pylint in some assertions
* Remove pylint in dev
* Take first mypy that supports typed-ast>=1.4.0 to limit the migration path
* Various return + else lint errors
* Find a set of deps that is working with current mypy version
* Update local oldest requirements
* Remove all current pylint errors
* Rebuild letsencrypt-auto
* Update mypy to fix pylint with new astroid version, and fix mypy issues
* Explain type: ignore
* Reconfigure tox, fix none path
* Simplify pinning
* Remove useless directive
* Remove debugging code
* Remove continue
* Update requirements
* Disable unsubscriptable-object check
* Disable one check, enabling two more
* Plug certbot dev version for oldest requirements
* Remove useless disable directives
* Remove useless no-member disable
* Remove no-else-* checks. Use elif in symetric branches.
* Add back assertion
* Add new line
* Remove unused pylint disable
* Remove other pylint disable
Summary of changes in this PR:
- Refactor files involved in the `certbot` module to be of a similar structure to every other package; that is, inside a directory inside the main repo root (see below).
- Make repo root README symlink to `certbot` README.
- Pull tests outside of the distributed module.
- Make `certbot/tests` not be a module so that `certbot` isn't added to Python's path for module discovery.
- Remove `--pyargs` from test calls, and make sure to call tests from repo root since without `--pyargs`, `pytest` takes directory names rather than package names as arguments.
- Replace mentions of `.` with `certbot` when referring to packages to install, usually editably.
- Clean up some unused code around executing tests in a different directory.
- Create public shim around main and make that the entry point.
New directory structure summary:
```
repo root ("certbot", probably, but for clarity all files I mention are relative to here)
├── certbot
│ ├── setup.py
│ ├── certbot
│ │ ├── __init__.py
│ │ ├── achallenges.py
│ │ ├── _internal
│ │ │ ├── __init__.py
│ │ │ ├── account.py
│ │ │ ├── ...
│ │ ├── ...
│ ├── tests
│ │ ├── account_test.py
│ │ ├── display
│ │ │ ├── __init__.py
│ │ │ ├── ...
│ │ ├── ... # note no __init__.py at this level
│ ├── ...
├── acme
│ ├── ...
├── certbot-apache
│ ├── ...
├── ...
```
* refactor certbot/ and certbot/tests/ to use the same structure as the other packages
* git grep -lE "\-e(\s+)\." | xargs sed -i -E "s/\-e(\s+)\./-e certbot/g"
* git grep -lE "\.\[dev\]" | xargs sed -i -E "s/\.\[dev\]/certbot[dev]/g"
* git grep -lE "\.\[dev3\]" | xargs sed -i -E "s/\.\[dev3\]/certbot[dev3]/g"
* Remove replacement of certbot into . in install_and_test.py
* copy license back out to main folder
* remove linter_plugin.py and CONTRIBUTING.md from certbot/MANIFEST.in because these files are not under certbot/
* Move README back into main folder, and make the version inside certbot/ a symlink
* symlink certbot READMEs the other way around
* move testdata into the public api certbot zone
* update source_paths in tox.ini to certbot/certbot to find the right subfolder for tests
* certbot version has been bumped down a directory level
* make certbot tests directory not a package and import sibling as module
* Remove unused script cruft
* change . to certbot in test_sdists
* remove outdated comment referencing a command that doesn't work
* Install instructions should reference an existing file
* update file paths in Dockerfile
* some package named in tox.ini were manually specified, change those to certbot
* new directory format doesn't work easily with pyargs according to http://doc.pytest.org/en/latest/goodpractices.html#tests-as-part-of-application-code
* remove other instance of pyargs
* fix up some references in _release.sh by searching for ' . ' and manual check
* another stray . in tox.ini
* fix paths in tools/_release.sh
* Remove final --pyargs call, and now-unnecessary call to modules instead of local files, since that's fixed by certbot's code being one layer deeper
* Create public shim around main and make that the entry point
* without pyargs, tests cannot be run from an empty directory
* Remove cruft for running certbot directly from main
* Have main shim take real arg
* add docs/api file for main, and fix up main comment
* Update certbot/docs/install.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Fix comments in readthedocs requirements files to refer to current package
* Update .[docs] reference in contributing.rst
* Move plugins tests to certbot tests directory
* add certbot tests to MANIFEST.in so packagers can run python setup.py test
* move examples directory inside certbot/
* Move CHANGELOG into certbot, and create a top-level symlink
* Remove unused sys and logging from main shim
* nginx http01 test no longer relies on certbot plugins common test
* Create _internal package for Certbot's non-public modules
* Move account.py to _internal
* Move auth_handler.py to _internal
* Move cert_manager.py to _internal
* Move client.py to _internal
* Move error_handler.py to _internal
* Move lock.py to _internal
* Move main.py to _internal
* Move notify.py to _internal
* Move ocsp.py to _internal
* Move renewal.py to _internal
* Move reporter.py to _internal
* Move storage.py to _internal
* Move updater.py to _internal
* update apache and nginx oldest requirements
* Keep the lock file as certbot.lock
* nginx oldest tests still need to rely on newer certbot
* python doesn't have good dependency resolution, so specify the transitive dependency
* update required minimum versions in nginx setup.py
Fixes#7007
Python 3.4 is [EOL](https://www.python.org/dev/peps/pep-0429/), and only Python 3.x version available for CentOS 6 through EPEL is this version, and so is used by `certbot-auto`, the only official way to install Certbot on this platform.
This unpleasant situation becomes a little more uncomfortable, considering that the newest `pip` version (19.2) [just dropped Python 3.4 support](https://github.com/pypa/pip/issues/6685) and will refuse to start on this Python version. We can expect a lot of dependencies to follow this path now.
One direct result of this situation is that a fix to support correctly the ARM platforms requires to upgrade `pip` to 19.2 for `certbot-auto`. So this is not possible right now.
Then, let's upgrade Certbot instances on CentOS 6 to a supported version of Python 3.
This PR proposes a new bootstrap approach for CentOS 6 platform, `BootstrapRpmPython3Legacy`, that will install Python 3.6 from [SCL](https://www.softwarecollections.org) (the latest one available for now on CentOS 6). In term of Python 3 specific bootstrap methods, I take the occasion here to completely separate the bootstrap of CentOS 6 as a legacy system, from the RPM-based newest systems (like Fedora 29+) that are simply dropping support for Python 2.x. This is in prevision of future migration for all systems on Python 3.x, that is a different problematic than supporting old systems.
* Add logic
* Rebuilt letsencrypt-auto
* Fix logic
* Focus on specific packages
* Maintain PATH for further invocations of letsencrypt-auto after bootstrap.
* Various corrections
* Fix farm test for RHEL6
* Working centos6 letsencrypt-auto self tests
* Fix test_sdist for CentOS 6
* Corrections
* Work in progress
* Working configuration
* Fix typo
* Remove EPEL. Add a test.
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Improvements after review
* Improvements
* Add a comment
* Add a test
* Update a test
* Corrections
* Update function return
* Work in progress
* Correct behavior on oracle linux 6.
* Corrections
* Rebuild script
* Add letsencrypt-auto tests for oraclelinux6
* Update tox.ini
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Remove specific code for scientific linux
* Change some variables names
* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Various corrections
* Fix tests
* Add a comment
* Update message
* Fix test message
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update scripts
* More focused assertion
* Add back a test
* Update script
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Check quiet mode
* Add changelog
* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
This PR builds off of #7240 to fix#7241.
The code in certbot-auto is unchanged which I +1. Someone else should give it a 2nd review.
For the code in the tests, you can see all tests passing (including test_tests.sh) at https://travis-ci.com/certbot/certbot/builds/122198270.
I created #7301 to track removing the temporary code in test_leauto_upgrades.sh as suggested at #7282 (comment).
One noteworthy thing here is I did not add the RHEL 8 AMI to the Apache tests due to #7273. This problem is not related to support in certbot-auto though, is an edge case, and I do not personally believe it should block this PR.
Inspired by the number of ARM users we have (and because I want to rip out the only 32 bit test we have which without this PR would remove all tests we have on non-x86_64 architectures), this test adds an ARM image to the test farm tests. The image ID was taken from https://wiki.debian.org/Cloud/AmazonEC2Image/Stretch, you can see tests passing at https://travis-ci.com/certbot/certbot/builds/116857897, and I ran test_tests.sh locally and it passed.
Since #7073 for Certbot and letsencrypt/boulder@3918714 for Boulder have landed, the bash scripts that remained after certbot-ci are not useful anymore outside of Certbot.
Only remaining place is the apacheconftest-with-pebble tox target, which leverages pebble-fetch.py script to expose a running ACME server to the apache-conf-test script.
This PR refactor apacheconftest-with-pebble to use certbot-ci instead. Finally, this PR remove the remaining integration tests bash scripts, that are _common.sh, boulder-fetch.py and pebble-fetch.py.
* Disconnect common and boulder-fetch
* Prepare reconnection of apacheconftest to new pebble deployment logic
* Finish the configuration for apacheconftest
* Add executable flag to python script
* Fix shebang
* Delete pebble-fetch.sh
Fixes#6955.
This updates the Fedora version used in our test farm tests to Fedora 30. The AMI ID comes from https://alt.fedoraproject.org/cloud/ where it is listed as their standard HVM AMI for the region we use us-east-1 (US East (N. Virginia)).
Unfortunately, there were a lot of small changes required for this. The big reason for this is on Fedora, there isn't a Python 2 executable installed. In fact, there's not even an executable named python. It's just python3. Rather than installing another Python in each test, I wrote a script that the test scripts can share to figure out the different paths and names that should be used in their script. (This isn't used in test_sdists.sh because the logic is a little different.)
Other changes here worth flagging are:
I changed the name of the variable RUN_PYTHON3_TESTS in test_leauto_upgrades.sh to RUN_RHEL6_TESTS. The tests that are run when this variable is set test the upgrade from Python 2 to Python 3 on RHEL 6. I think this new name is much better now that we also have Fedora running Python 3.
I made tools/simple_http_server.py work on Python 3.
You can see tests passing with these changes at https://travis-ci.com/certbot/certbot/builds/113821476. I also ran test_tests.sh and they passed.
* Update to Fedora 30 in test farm tests.
Fedora 28 is likely to reach its EOL soon.
* Add set_python_envvars.sh.
* Fix test_apache2.sh on python3 only distros.
* Fix test_leauto_upgrades.sh on python3 systems.
* Fix certonly_standalone tests with python3 only
* Fix test_sdists.sh on python3 only distros.
* Make simple_http_server.py work on Python 3.
* add comments
You can see the full test suite running at https://travis-ci.com/certbot/certbot/builds/112291892.
A few noteworthy things:
--fast is included because without, the tests would sometimes reach Travis' 50 minute timeout even with 1 test script per Travis build.
The only script that is run at release time which is not being run here is https://github.com/certbot/certbot/blob/master/tests/letstest/scripts/test_tests.sh because that script runs tests on the packages installed by certbot-auto which won't be updated until midway through a release.
We check TRAVIS_PULL_REQUEST and error out if it is not false for simplicity which should be fine because these tests are never run on PRs. The reason it's more complex to run test farm tests on PRs is the test farm tests need a named branch to pull from and Travis effectively merges the PR into the target branch before running tests complicating this.
I don't think this should block this PRs, but the one final change we may want to make to the current setup is #7071.
* Add encrypted private key.
* Add test farm tests to tox and travis.
* Change magic profile name.
* Further split test farm tests.
* Build local branch.
* more depth
* Set LOGDIR at top of script.
* Set sentinel at top of script.
* Don't use EC2 global to block on instance start.
* Remove global boto3 state.
* Pass in boulder_url.
* Create main function.
* Add link to reload docs.
* Connect certbot-ci to travis. Remove old bash files.
* Configure test-everything
* Protect against import error
* Remove unused ignore
* Better handling of urllib3
* Correct path
* Remove a warning
* Correct call
* Protect atexit register execution
* Update docs/contributing.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update docs/contributing.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Add again some bash scripts to avoid breaking to much retro-compatiblity on third party scripts
* Move boulder-v1 and boulder-v2 in nightly tests
* Separate oldest unit tests and oldest integration tests
* Remove try/except
* Test integration included in toxenv
* Add a wait to avoid a transient issue on OCSP status in oldest tests
* Clean travis.yml, split other tests
* Remove useless config
* Update .travis.yml
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update tox.ini
* Update tox.ini
* Remove pytest-sugar
* Remove empty pytest.ini, tests are working without it
Fixes#7012.
Apparently, the previous test we had here doesn't catch the case when certbot-auto prints blank lines. (I don't yet understand why so if someone does, please let me know!)
Regardless, I fixed up the test and verified it fails with the version of letsencrypt-auto in master and then fixed letsencrypt-auto so the test passes.
I ran test farm tests on the changes here and they passed on all instances.
* correct test
* fixes#7012
We made this change locally yesterday while preparing the release.
I tested this change on all AMIs currently in the test farm as well as Fedora 29 and this test passed on all instances.
Recent changes are no longer compatible with the old version of boulder used in the test farm tests. This PR updates the version of boulder used, and runs it with the new way of running boulder.
A new ami was created and is used here that uses Ubuntu 18.04, so that docker-compose can be installed more properly.
Removed commented-out section about rabbitmq that was already deprecated.
Switched to using the public DNS resolver 8.8.8.8 for the tests because the way to find the correct local resolver changed.
This PR attempts to better inform people about the problem identified at https://community.letsencrypt.org/t/certbot-auto-deployment-best-practices/91979/.
I was hesitant to add the flag --no-permissions-check, however, if there's some obscure distro out there (or custom user setup) that has a strange users and groups, I didn't want us to either:
Have to put out a bug fix release
Refuse to fix the problem and let them deal with warnings on every run
* add check_permissions.py
* Update letsencrypt-auto.template.
* build letsencrypt-auto
* Add test_permissions_warnings to auto_test
* Allow uid/gid < 1000.
* Add --no-permissions-check to Certbot.
* Add --no-permissions-check to certbot-auto.
* Add test farm test that letsencrypt-auto is quiet.
As a bonus, this new test will catch problems like the one that the caused
0.33.1 point release.
* Update CHANGELOG about permissions check.
* Update permissions comment.
* Fix symlink handling.
* Use a better default in auto_test.py.
Fixes#6861.
_venv_common.py is no longer executable. The reason for this is the venv creation logic is now different between Python 2 and Python 3. We could add code that branches on the Python version running the script, but I personally think that's unnecessary.
--setuptools and --no-site-packages is no longer passed to virtualenv either. These flags were made noops in virtualenv 1.10 and 1.7 respectively, but all of CentOS 6, 7, Debian 8+, and Ubuntu 14.04+ have new enough versions of virtualenv where these flags are no longer necessary. They are not even accepted as flags to Python 3's venv module.
Use of VENV_ARGS from test_sdists.sh was also removed because that environment variable hasn't done anything in a while.
I ran test farm tests on test_apache2.sh and test_sdists.sh with these changes and they passed.
* Fixes#6861.
* _venv_common is no longer executable.
This PR is a part of the tls-sni-01 removal plan described in #6849.
This PR removes --tls-sni-01-port, --tls-sni-01-address and tls-sni-01/tls-sni options from --preferred-challenges. They are replace by deprecation warning, indicating that these options will be removed soon.
This deprecation, instead of complete removal, is done to avoid certbot instances to hard fail if some automated scripts still use these flags for some users.
Once this PR lands, we can remove completely theses flags in one or two release.
* Remove tls-sni related flags in cli. Add a deprecation warning instead.
* Adapt tests to cli and renewal towards tls-sni flags deprecation
* Add https_port option. Make tls_sni_01_port show a deprecation warning, but silently modify https_port if set
* Migrate last items
* Fix lint
* Update certbot/cli.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Ensure to remove all occurences of tls-sni-01
* Remove unused parameter
* Revert modifications on cli-help.txt
* Use logger.warning instead of sys.stderr
* Update the logger warning message
* Remove standalone_supported_challenges option.
* Fix order of preferred-challenges
* Remove supported_challenges property
* Fix some tests
* Fix lint
* Fix tests
* Add a changelog
* Clean code, fix test
* Update CI
* Reload
* No hard date for tls-sni removal
* Remove useless cast to list
* Update certbot/tests/renewal_test.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Add entry to the changelog
* Add entry to the changelog
* Remove tls-sni from nginx config
* Add a dedicated configuration to define what is the HTTPS port for this certbot instance.
* Correct some tests
* Reestablish default vhost creation
* Clean tls references for nginx integration tests
* Associate https_port only to tests and nginx
This PR is a part of the tls-sni-01 removal plan described in #6849.
This PR removes the tls-sni-01 challenge tests during the integration tests. The approach I used here is not to remove completely the existing test code, but simply editing it to use a http-01 challenge. Indeed:
* the current integration tests are strongly coupled, and would require more modifications that it is worth, because ...
* the certbot-ci project, that has already no tls-sni tests, will soon replace completely the current integration tests code.
