I think this is causing failures in some of our tests so this PR reverts the change until we can fix the problem.
The 2nd commit is to keep the change using more idiomatic wording in the changelog for another change that got included in this PR.
* Revert "Use built-in support for OCSP in cryptography >= 2.5 (#6603)"
This reverts commit 2ddaf3db04.
* keep changelog correction
We always run a full set of CI tests before beginning the release process. The way this would work previously is we would either trigger tests on the `test-everything` branch to run through Travis' web UI or if it was a point release, create a new branch based on `test-everything` but modify `.travis.yml` so the branch that was pulled in to be tested was the point release branch instead of `master`.
This no longer works because the former `test-everything` tests are now only run when Travis automatically runs our tests nightly.
We could create and maintain a separate branch for the purpose of manually running all tests or remove the conditionals from the latest `.travis.yml` file every time before we want to run these tests, but there must be A Better Way™.
This PR makes the change that in addition to running all tests nightly, they would also run on pushes to tested branches other than master. These changes do not affect the tests run on PRs or on commits to `master`.
What is affected is commits to point release branches and branches named `test-*`. (See [.travis.yml](2ddaf3db04/.travis.yml (L177)) for what branches we run tests on.) Running all tests on point release branches automates the step of running our full test suite before doing a point release.
The changes to `test-*` could be a mixed bag, however, since we switched to travis-ci.com over 3 weeks ago, I'm the only one who has used this functionality and I personally prefer things this way. At the very least, since these branches don't seem to be widely used, I think we can make this change and reevaluate if it becomes a problem.
* Test all on push events to non-master branches.
* Move branches section up.
* expand comment
Attempts to configure all of the following VirtualHosts for answering the HTTP challenge:
* VirtualHosts that have the requested domain name in either `ServerName` or `ServerAlias` directive.
* VirtualHosts that have a wildcard name that would match the requested domain name.
This also applies to HTTPS VirtualHosts, making Apache plugin able to handle cases where HTTP redirection takes place in reverse proxy or similar, before reaching the Apache HTTPD.
Even though also HTTPS VirtualHosts are selected, Apache plugin tries to ensure that at least one of the selected VirtualHosts listens to HTTP-01 port (configured with `--http-01-port` CLI option). So in a case where only HTTPS VirtualHosts exist, but user wants to configure those, `--http-01-port` parameter needs to be set for the port configured to the HTTPS VirtualHost(s).
Fixes: #6730
* Select all matching VirtualHosts for HTTP-01 challenges instead of just one
* Finalize PR and add tests
* Changelog entry
In response to #6594. [Fixes #6594.]
To execute OCSP requests, certbot relies currently on a openssl binary execution. If openssl is not present in the PATH, the OCSP check will be silently ignored. Since version 2.4, cryptography has support for OCSP requests, without the need to have openssl binary available locally.
This PR takes advantage of it, and will use the built-in support of OCSP in cryptography for versions >= 2.4. Otherwise, fallback is done do a direct call to openssl binary, allowing oldest requirements to still work with legacy cryptography versions.
Update: requirement is now cryptography >= 2.5, to avoid to rely on a private method from cryptography.
* Implement logic using cryptography
* Working OSCP using pure cryptography
* Fix openssl usage in unit tests
* Reduce verbosity
* Add tests
* Improve naive skipIf
* Test resiliency
* Update ocsp.py
* Validate OCSP response. Unify OCSP URL get
* Improve resiliency checks, correct lint/mypy
* Improve hash selection
* Fix warnings when calling openssl bin
* Load OCSP tests assets as vectors.
* Update ocsp.py
* Protect against invalid ocsp response.
* Add checks to OCSP response
* Add more control on ocsp response
* Be lenient about assertion that next_update must be in the future, similarly to openssl.
* Construct a more advanced OCSP response mock to trigger more logic in ocsp module.
* Add test
* Refactor signature process to use crypto_util
* Fallback for cryptography 2.4
* Avoid a collision with a meteor.
* Correct method signature documentation
* Relax OCSP update interval
* Trigger built-in ocsp logic from cryptography with 2.5+
* Update pinned version of cryptography
* Update certbot/ocsp.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update ocsp.py
* Update ocsp_test.py
* Update CHANGELOG.md
* Update CHANGELOG.md
The existing `acme.TLSALPN01` challenge class did not have
a `response_cls`, meaning it was not possible to use a tls-alpn-01
challenge with `client.answer_challenge`.
To support the above a simple `TLSALPN01Response` class is added that
doesn't provide the ability to solve a tls-alpn-01 challenge end to end
(e.g. generating and serving the correct response certificate) but that
does allow the challenge to be initiated. This is sufficient for users
that have set up the challenge response independent of the `acme`
module code.
Resolves#6676
The account path used to store user credentials is calculated from the domain used to contact the relevant ACME CA server.
For instance, if the directory URL is https://my.domain.net/directory, then the account path will be $CONFIG_DIR/accounts/my.domain.net.
However, if non standard HTTP/HTTPS port need to be used, colons will be involved. For instance, https://my.domain.net:14000/directory will give $CONFIG_DIR/accounts/my.domain.net:14000.
Colons in paths are supported on POSIX systems, but not on Windows (it is reserved for the root drive letter).
This PR replaces colons by underscores for account paths on Windows, and leaves them untouched on Linux.
* Fix account path on Windows when colons are involved
* Protect colon in drive letter
* Refactor compat platform specific logic
It was pointed out to me that you can no longer run tox.cover.py directly to run coverage tests on a subset of the packages in this repo.
This happened after we did both of:
1. Factored out --pyargs from the different test files and put it in pytest.ini.
2. Moved the options we added to pytest.ini to tox.ini meaning that --pyargs is not set unless you run the file through tox.
I think the fact that we factored out --pyargs from the files that needed it was a mistake. --pytest is needed by tox.cover.py and install_and_test.py in order to work correctly.
I think CLI options like this which are needed for the file to function should be left in the file directly. Doing anything else in my opinion unnecessarily couples these scripts to other files making them more brittle and harder to maintain.
With that said, I also think CLI options which are not needed (such as --numprocesses) can be left to be optionally added through PYTEST_ADDOPTS.
* Add --pyargs to tox.cover.py.
* Add --pyargs to install_and_test.py.
* Remove --pyargs from tox.ini.
When certbot is executing, several resources are opened. It is typically file handles and locks on them. Of course, theses resources need to be cleanup. It is done in Certbot by registering cleanup functions through atexit module, that ensures theses functions will be called when Certbot exit. This allow to not care about resource cleanup everywhere in the code, as it is processed globally.
The problem with atexit is it cleanup functions are called when the Python program exit. If the program is Certbot itself when used, this is Pytest in unit test execution. So during a unit test execution, cleanup is not called after a test and before its tearDown, but when Pytest exit, so way after tests and their respective tearDown.
But many tearDown implies to delete folders where this kind of resources are hold.
This is never a problem on Linux, thanks to its non-blocking file handling. It is usually not a problem on Windows, despite its blocking approach. But if the tearDown requires folder cleanup, exceptions are raised, and currently hidden as warnings. There is currently 504 exceptions of this type in Certbot core tests on Windows.
This PR starts to correct this situation. To do so, some of the functions cleanup normally called through atexit, are explicitly called as part of the tearDown process of relevant test classes, before directory removal is done. Theses situations come all from the certbot.tests.util.TempDirTestCase, so the code is in this specific tearDown process.
As a consequence, exceptions drop from 504 to 64.
Then there are still a significant part of them, that will be handled in later mitigation.
* Call atexit handlers before test tearDown to reduce errors on Windows
* Clear locks dict after global releasing execution
* Remove last tearDown errors.
* Clean out mock on open.
* Remove a test
* Reenable some tests
This PR reactivates tls-sni-01 challenges on recent Boulder versions checkout for integration tests. This allows to continue testing this challenge until it is officially dropped from server (Boulder) and client (Certbot).
Reverts #6679.
Fixes#6106.
AMIs were taken from https://wiki.debian.org/Cloud/AmazonEC2Image/Stretch and https://cloud-images.ubuntu.com/locator/ec2/.
I didn't update the AMI for Fedora due to #6698.
These new AMIs pass on all test farm tests we run during the release process except Ubuntu 18.04 and 18.10 fail on test_apache2.sh. This is tracked at #6706. If this PR lands before this issue is resolved, we should list these systems as expected failures in the release notes.
Adding these AMIs slows down our tests significantly. I didn't measure it, but it feels 50-100% slower at least on my setup. I think it's worth it though.
* Update test farm targets.
* use different ubuntu ami
* Fix test_leauto_upgrades.sh on newer OSes.
Fixes#6697.
This PR updates the version of setuptools pinned in pipstrap which works around the problems we have seen on recent OSes.
Why did I pick this version of setuptools? Because it's the latest and greatest, [supports all versions of Python that we do](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py#L173), [has been out for a month and a half without the need for a point release](https://setuptools.readthedocs.io/en/latest/history.html), and has no non-optional dependencies.
For the last point about dependencies, I don't think we have too much to worry about. setuptools did have a period between versions 34.0.0 and 36.0.0 where they tried to have normal dependencies on other packages, but reverted these changes. See their [changelog for 36.0.0](https://setuptools.readthedocs.io/en/latest/history.html#v36-0-0).
You can also compare their [current setup.py file](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py) to the [setup.py file for the currently pinned version](https://github.com/pypa/setuptools/blob/v29.0.1/setup.py) and you'll see [not much has changed](https://pastebin.com/nQj6d7D8).
Not only that, but I have successfully tested these changes on:
* ubuntu18.10
* ubuntu18.04LTS
* ubuntu16.04LTS
* ubuntu14.04LTS
* ubuntu14.04LTS_32bit
* debian9
* debian8.1
* amazonlinux-2015.09.1
* amazonlinux-2015.03.1
* RHEL7
* fedora23
* fedora29
* centos7
* centos6
* freebsd11
* macOS
* Update setuptools to 40.6.3.
* Build letsencrypt-auto.
* update changelog
* Don't use pipstrap in Dockerfile.centos6.
(cherry picked from commit b7211c3f39)
Fixes#6697.
This PR updates the version of setuptools pinned in pipstrap which works around the problems we have seen on recent OSes.
Why did I pick this version of setuptools? Because it's the latest and greatest, [supports all versions of Python that we do](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py#L173), [has been out for a month and a half without the need for a point release](https://setuptools.readthedocs.io/en/latest/history.html), and has no non-optional dependencies.
For the last point about dependencies, I don't think we have too much to worry about. setuptools did have a period between versions 34.0.0 and 36.0.0 where they tried to have normal dependencies on other packages, but reverted these changes. See their [changelog for 36.0.0](https://setuptools.readthedocs.io/en/latest/history.html#v36-0-0).
You can also compare their [current setup.py file](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py) to the [setup.py file for the currently pinned version](https://github.com/pypa/setuptools/blob/v29.0.1/setup.py) and you'll see [not much has changed](https://pastebin.com/nQj6d7D8).
Not only that, but I have successfully tested these changes on:
* ubuntu18.10
* ubuntu18.04LTS
* ubuntu16.04LTS
* ubuntu14.04LTS
* ubuntu14.04LTS_32bit
* debian9
* debian8.1
* amazonlinux-2015.09.1
* amazonlinux-2015.03.1
* RHEL7
* fedora23
* fedora29
* centos7
* centos6
* freebsd11
* macOS
* Update setuptools to 40.6.3.
* Build letsencrypt-auto.
* update changelog
* Don't use pipstrap in Dockerfile.centos6.
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so after virtualenv upgrades pip, downgrade to the pinned version.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Always download the pinned version of pip in pipstrap
* Run build.py
* Update changelog
* Remove unused variable
* Run build.py
(cherry picked from commit 9746c310d8)
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so after virtualenv upgrades pip, downgrade to the pinned version.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Always download the pinned version of pip in pipstrap
* Run build.py
* Update changelog
* Remove unused variable
* Run build.py
This will immediately address the breakage reported in #6682 and tracked at #6685. Pip 19.0.0 and 19.0.1 don't allow commas in filenames, so don't use commas in filenames in certbot-apache test code.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
Steps to test:
push master to test box
run tools/venv.py
activate venv
pip --version: 19.0.1
pip install ./certbot-apache/: fails
push branch code to test box
confirm pip --version still 19.0.1
pip install ./certbot-apache/: success
* Rename old,default.conf to old-and-default.conf
* Update changelog
* sites-enabled should contain a symlink to sites-available
(cherry picked from commit 34d655151d)
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
(cherry picked from commit 658b7b9d47)
This will immediately address the breakage reported in #6682 and tracked at #6685. Pip 19.0.0 and 19.0.1 don't allow commas in filenames, so don't use commas in filenames in certbot-apache test code.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
Steps to test:
push master to test box
run tools/venv.py
activate venv
pip --version: 19.0.1
pip install ./certbot-apache/: fails
push branch code to test box
confirm pip --version still 19.0.1
pip install ./certbot-apache/: success
* Rename old,default.conf to old-and-default.conf
* Update changelog
* sites-enabled should contain a symlink to sites-available
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
