* Fix check permissions logic (#7034)
Fixes#7031
I use the same approach than in `CreateVenv()` and `CompareVersions()`: a new bash function `CheckPathPermissions()` is declared an execute a python script passed to the interpreter through stdin.
This allows:
* to not require the temp_dir that holds a temporary script to be executed
* to reduce at the bare minimum the change to make on the order of bash command to execute (including when the temp_dir is created)
* Fix check permissions logic in certbot-auto by making a temp dir useless
* Update CHANGELOG.md
(cherry picked from commit 71b1b8c2d9)
* Fixup changelog.
Fixes#7012.
Apparently, the previous test we had here doesn't catch the case when certbot-auto prints blank lines. (I don't yet understand why so if someone does, please let me know!)
Regardless, I fixed up the test and verified it fails with the version of letsencrypt-auto in master and then fixed letsencrypt-auto so the test passes.
I ran test farm tests on the changes here and they passed on all instances.
* correct test
* fixes#7012
(cherry picked from commit e15e848474)
This PR attempts to better inform people about the problem identified at https://community.letsencrypt.org/t/certbot-auto-deployment-best-practices/91979/.
I was hesitant to add the flag --no-permissions-check, however, if there's some obscure distro out there (or custom user setup) that has a strange users and groups, I didn't want us to either:
Have to put out a bug fix release
Refuse to fix the problem and let them deal with warnings on every run
* add check_permissions.py
* Update letsencrypt-auto.template.
* build letsencrypt-auto
* Add test_permissions_warnings to auto_test
* Allow uid/gid < 1000.
* Add --no-permissions-check to Certbot.
* Add --no-permissions-check to certbot-auto.
* Add test farm test that letsencrypt-auto is quiet.
As a bonus, this new test will catch problems like the one that the caused
0.33.1 point release.
* Update CHANGELOG about permissions check.
* Update permissions comment.
* Fix symlink handling.
* Use a better default in auto_test.py.
The changelog should still say <version> - master because it will be fixed up automatically by the release script at https://github.com/certbot/certbot/blob/master/tools/_release.sh#L69.
* Protect certbot-auto against non numerical version release in some RPM distributions (#6913)
Fixes#6912
Bash evaluate all condition in a predicate statement, eg. `"$SOMEVAR" = "test" -a "$ANOTHERVAR" = "test2"`, even if it is not necessary, for instance if the first condition is false in the example here.
As a consequence, on non-Fedora distributions, an evaluation of the distribution version could be done on non numeric value, eg. `"6.7" -eq "29"`, making certbot-auto failing in this case.
This PR fixes that, by evaluating the version on RPM distributions only if we are on Fedora. Otherwise, version will be "0".
(cherry picked from commit c2d9ea1f61)
* Update changelog about #6912 fix. (#6914)
(cherry picked from commit 30eafba997)
* cleanup changelog
Dependencies generated by the script introduced with #6839 were not including anymore the fix about enum34 for CentOS 6.
This PR reinserts this fix, and updates the script overrides to ensure that this fix will stay in next dependencies generation.
* Add the environment marker back. Ensure that it will stay by adding an override to dependencies generator.
* Add comments, for future fix
* Update letsencrypt-auto-source/rebuild_dependencies.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update comment
Currently, `tox -e le_auto_jessie` job fails. It breaks in particular the cron pipeline that test everything each night.
The failure occurs while setting up the Jessie Docker container to run the tests for certbot-auto, when `apt-get update` is invoked, with this error:
```
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/main/binary-amd64/Packages 404 Not Found
```
Indeed, if there are `stretch-updates`, `buster-updates` and so on in the repository, there is no `jessie-updates`. I do not know exactly the logic of Debian here, but as `*-updates` folders store stable updates, a distribution moving to LTS support like Jessie has no stable updates anymore. I suppose `jessie-updates have been decommissioned recently, and the official Docker has not been updated yet to use the LTS configuration for repositories.
This PR does that live in the Dockerfile, using official instructions from https://wiki.debian.org/LTS/Using, and fixes this specific job.
An example of a successful job with this modification can be found here: https://travis-ci.com/certbot/certbot/jobs/187864341
Fixes#6698
Fedora maintainers engaged a deprecation path for Python 2.x with Fedora 29. As a first step, python2-virtualenv does not install the virtualenv binary anymore, in favor of python3-virtualenv, and so the installation of Python 3 virtual environments by default.
However, certbot-auto installs python2-virtualenv for all recent RPM distributions, and relies of the execution of virtualenv, and this is failing the process.
Since the plan in the future is to remove Python 2.x from Fedora, this PR follows this logic to fix certbot-auto: started to Fedora 29, certbot-auto will install and execute certbot on Python 3. This implies to detect that we are on Fedora 29+, install python3-virtualenv that will install also Python 3 dependencies and virtualenv binary, then instruct the process to use Python 3. This is in fact similar to EOL distributions shipping with Python 2.6, and for which Python 3.4 from EPEL is installed and used.
Older versions of Fedora continue to use Python 2.x, and their process is untouched. Four scenarios are covered here:
fresh Fedora 28: old process is used, nothing changes
fresh Fedora 29: new process is used, Python 3 is installed, certbot runs on it
update Fedora 29 from 28, already installed certbot-auto without rebootstrapping required: existing venv continue to be used, certbot runs on it
update Fedora 29 from 28, already installed certbot-auto with rebootstrapping required: new process is used, installing python3-virtualenv, python3-devel and python3-rpm-macros, Python 3 is installed, certbot runs on it
* Add a step to handle python3 on fedora29
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update rpm_python3.sh
* Rebuild certbot-auto
* Empty commit to relaunch CI pipeline
* Add changelog
* Update CHANGELOG.md
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update CHANGELOG.md
Fixes#6228.
Since OpenSUSE Leap 15, python-virtualenv became a source package, breaking certbot-auto bootstrap on this version. Then python2-virtualenv must be used to create Python 2.x virtual environments.
This PR makes certbot-auto compatible to prior and after Leap 15, by testing the existence of python-virtualenv on current OpenSUSE system, and then use appropriate packages.
* Cover case of OpenSUSE Leap 15+ in certbot-auto
* Revert increment on bootstrap for OpenSUSE
* Fix configuration for Leap15+
* Add comment about explicit installation of python2-setuptools
* Update letsencrypt-auto-source/pieces/bootstrappers/suse_common.sh
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update letsencrypt-auto
* Fix pipstrap on windows
* Pipstrap pin setuptools version, so explicit install it is not needed anymore.
* Rebuild letsencrypt-auto source
* Use sys.executable in pipstrap to allow straightforward execution in a venv by choosing the python interpreter from this venv.
* Update letsencrypt-auto
* Simulate test-everything
* Revert "Simulate test-everything"
This reverts commit b62c4d719a.
* Clean pipstrap code
* Reenabling OCSP cryptography support
* Refactor the validation logic of OCSP response to match the OpenSSL one
* Prepare runtime for OCSP response test
* Move unrelated test to another relevant place
* Reimplement OCSP status checks in integration tests
* Clean script
* Protect OCSP check against connection errors
* Update tests/certbot-boulder-integration.sh
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Cleaning
* Add a specific script for letsencrypt-auto install+help
* Remove inconsistent assertion
* Add executable permissions
* Remove unused variable
* Move testdata
* Corrected cleanup code
* Empty commit
I think this is causing failures in some of our tests so this PR reverts the change until we can fix the problem.
The 2nd commit is to keep the change using more idiomatic wording in the changelog for another change that got included in this PR.
* Revert "Use built-in support for OCSP in cryptography >= 2.5 (#6603)"
This reverts commit 2ddaf3db04.
* keep changelog correction
In response to #6594. [Fixes #6594.]
To execute OCSP requests, certbot relies currently on a openssl binary execution. If openssl is not present in the PATH, the OCSP check will be silently ignored. Since version 2.4, cryptography has support for OCSP requests, without the need to have openssl binary available locally.
This PR takes advantage of it, and will use the built-in support of OCSP in cryptography for versions >= 2.4. Otherwise, fallback is done do a direct call to openssl binary, allowing oldest requirements to still work with legacy cryptography versions.
Update: requirement is now cryptography >= 2.5, to avoid to rely on a private method from cryptography.
* Implement logic using cryptography
* Working OSCP using pure cryptography
* Fix openssl usage in unit tests
* Reduce verbosity
* Add tests
* Improve naive skipIf
* Test resiliency
* Update ocsp.py
* Validate OCSP response. Unify OCSP URL get
* Improve resiliency checks, correct lint/mypy
* Improve hash selection
* Fix warnings when calling openssl bin
* Load OCSP tests assets as vectors.
* Update ocsp.py
* Protect against invalid ocsp response.
* Add checks to OCSP response
* Add more control on ocsp response
* Be lenient about assertion that next_update must be in the future, similarly to openssl.
* Construct a more advanced OCSP response mock to trigger more logic in ocsp module.
* Add test
* Refactor signature process to use crypto_util
* Fallback for cryptography 2.4
* Avoid a collision with a meteor.
* Correct method signature documentation
* Relax OCSP update interval
* Trigger built-in ocsp logic from cryptography with 2.5+
* Update pinned version of cryptography
* Update certbot/ocsp.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update ocsp.py
* Update ocsp_test.py
* Update CHANGELOG.md
* Update CHANGELOG.md
Fixes#6697.
This PR updates the version of setuptools pinned in pipstrap which works around the problems we have seen on recent OSes.
Why did I pick this version of setuptools? Because it's the latest and greatest, [supports all versions of Python that we do](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py#L173), [has been out for a month and a half without the need for a point release](https://setuptools.readthedocs.io/en/latest/history.html), and has no non-optional dependencies.
For the last point about dependencies, I don't think we have too much to worry about. setuptools did have a period between versions 34.0.0 and 36.0.0 where they tried to have normal dependencies on other packages, but reverted these changes. See their [changelog for 36.0.0](https://setuptools.readthedocs.io/en/latest/history.html#v36-0-0).
You can also compare their [current setup.py file](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py) to the [setup.py file for the currently pinned version](https://github.com/pypa/setuptools/blob/v29.0.1/setup.py) and you'll see [not much has changed](https://pastebin.com/nQj6d7D8).
Not only that, but I have successfully tested these changes on:
* ubuntu18.10
* ubuntu18.04LTS
* ubuntu16.04LTS
* ubuntu14.04LTS
* ubuntu14.04LTS_32bit
* debian9
* debian8.1
* amazonlinux-2015.09.1
* amazonlinux-2015.03.1
* RHEL7
* fedora23
* fedora29
* centos7
* centos6
* freebsd11
* macOS
* Update setuptools to 40.6.3.
* Build letsencrypt-auto.
* update changelog
* Don't use pipstrap in Dockerfile.centos6.
(cherry picked from commit b7211c3f39)
Fixes#6697.
This PR updates the version of setuptools pinned in pipstrap which works around the problems we have seen on recent OSes.
Why did I pick this version of setuptools? Because it's the latest and greatest, [supports all versions of Python that we do](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py#L173), [has been out for a month and a half without the need for a point release](https://setuptools.readthedocs.io/en/latest/history.html), and has no non-optional dependencies.
For the last point about dependencies, I don't think we have too much to worry about. setuptools did have a period between versions 34.0.0 and 36.0.0 where they tried to have normal dependencies on other packages, but reverted these changes. See their [changelog for 36.0.0](https://setuptools.readthedocs.io/en/latest/history.html#v36-0-0).
You can also compare their [current setup.py file](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py) to the [setup.py file for the currently pinned version](https://github.com/pypa/setuptools/blob/v29.0.1/setup.py) and you'll see [not much has changed](https://pastebin.com/nQj6d7D8).
Not only that, but I have successfully tested these changes on:
* ubuntu18.10
* ubuntu18.04LTS
* ubuntu16.04LTS
* ubuntu14.04LTS
* ubuntu14.04LTS_32bit
* debian9
* debian8.1
* amazonlinux-2015.09.1
* amazonlinux-2015.03.1
* RHEL7
* fedora23
* fedora29
* centos7
* centos6
* freebsd11
* macOS
* Update setuptools to 40.6.3.
* Build letsencrypt-auto.
* update changelog
* Don't use pipstrap in Dockerfile.centos6.
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so after virtualenv upgrades pip, downgrade to the pinned version.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Always download the pinned version of pip in pipstrap
* Run build.py
* Update changelog
* Remove unused variable
* Run build.py
(cherry picked from commit 9746c310d8)
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so after virtualenv upgrades pip, downgrade to the pinned version.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Always download the pinned version of pip in pipstrap
* Run build.py
* Update changelog
* Remove unused variable
* Run build.py
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
(cherry picked from commit 658b7b9d47)
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
These tests were running on Ubuntu Precise and Debian Wheezy which have reached their end of life and are no longer maintained by the respective distros. This updates the tests to a newer version of Debian and Ubuntu.
* Remove tests on the deprecated precise.
* Add tests for Xenial.
* update Jessie tests to use Wheezy
* update .travis.yml
GitHub notified us about a security vulnerability in our pinned version of `urllib3` earlier this week. It doesn't affect us, but we might as well upgrade anyway. I checked:
* There are no backwards incompatible features we care about listed at https://github.com/urllib3/urllib3/blob/master/CHANGES.rst.
* urllib3's dependencies don't also need to be updated according to https://github.com/urllib3/urllib3/blob/1.24.1/setup.py.
* The hashes match when obtained from different network vantage points.
