* acme: add support for alternative cert. chains
* certbot: add --preferred-chain
* remove support for issuer SKI matching
* show --preferred-chain in "run" help
* warn if no chain matched and it's not a dry-run
* fix existing failing tests
* add unit, integration tests
* bump acme dependency to dev version
* simplify test to avoid py2.7 recursion bug
* add preferred_chain to STR_CONFIG_ITEMS
* reduce preferred_chain warning to info level
* acme: fix some docstrings in .messages
* certbot: fix docstring in crypto_util
* try to fix certbot-nginx acme dep problem
Fixes#8093.
This PR modifies and audits all uses of `subprocess` and `Popen` outside of tests, `certbot-ci/`, `certbot-compatibility-test/`, `letsencrypt-auto-source/`, `tools/`, and `windows-installer/`. Calls to outside programs have their `env` modified to remove the `SNAP` components of paths, if they exist. This includes any calls made from hooks, calls to `apachectl` and `nginx`, and to `openssl` from `ocsp.py`.
For testing manually, rsync flags will look something like:
```
rsync -avzhe ssh root@focal.domain:/home/certbot/certbot/certbot_*_amd64.snap .
rsync -avzhe ssh certbot_*_amd64.snap root@centos7.domain:/root/certbot/
```
With these modifications, `certbot plugins --prepare` now passes on Centos 7.
If I'm wrong and we package the `openssl` binary, the modifications should be removed from `ocsp.py`, and `env` should be passed into `run_script` rather than set internally in its calls from nginx and apache.
One caveat with this approach is the disconnect between why it's a problem (packaging) and where it's solved (internal to Certbot). I considered a wrapping approach, but we'd still have to audit specific calls. I think the best way to address this is robust testing; specifically, running the snap on other systems.
For hooks, all calls will remove the snap paths if they exist. This is probably fine, because even if the hook intends to call back into certbot, it can do that, it'll just create a new snap.
I'm not sure if we need these modifications for the Mac OS X/ Darwin calls, but they can't hurt.
* Add method to plugins util to get env without snap paths
* Use modified environment in Nginx plugin
* Pass through env to certbot.util.run_script
* Use modified environment in Apache plugin
* move env_no_snap_for_external_calls to certbot.util
* Set env internally to run_script, since we use that only to call out
* Add env to mac subprocess calls in certbot.util
* Add env to openssl call in ocsp.py
* Add env for hooks calls in certbot.compat.misc.
* Pass env into execute_command to avoid circular dependency
* Update hook test to assert called with env
* Fix mypy type hint to account for new param
* Change signature to include Optional
* go back to using CERTBOT_PLUGIN_PATH
* no need to modify PYTHONPATH in env
* robustly detect when we're in a snap
* Improve env util fxn docstring
* Update changelog
* Add unit tests for env_no_snap_for_external_calls
* Import compat.os
Fixes#8103.
* Update the DNS plugin generator script to core20 syntax
* Generate new snapcraft.yamls for the DNS plugins
* Update certbot.wrapper to search for python3.8 paths
Fixes#7957
This PR makes snapcraft generate itself the dependencies constraints file during the snap build, instead of having an external script that does it before calling snapcraft.
This line seems to refer to [augeas.py](https://github.com/hercules-team/python-augeas/blob/v0.5.0/augeas.py) from the version of Augeas we normally have pinned. This was necessary when we were installing each Certbot component in separate parts and only combining them later to ensure that the Augeas fork (which uses cffi) was used instead of the unmodified pinned version of Augeas.
Since everything is installed in one part and we're removing the Augeas pinning now though, this line is no longer necessary. You can see the snap being built and tested successfully with this change at https://travis-ci.com/github/certbot/certbot/builds/172134518.
* Do not unstage generated cffi augeas
* Add comment about deleted line.
According to `distutils/version.py`, StrictVersion is pretty strict in
what version numbers to accept:
> A version number consists of two or three dot-separated numeric
> components, with an optional "pre-release" tag on the end. The
> pre-release tag consists of the letter 'a' or 'b' followed by a number.
This assumption already fails for some pretty basic python libraries
itself, like setuptools, also available in `46.1.3.post20200610`, a
completely valid version number according to
https://www.python.org/dev/peps/pep-0440/#post-releases.
There doesn't seem to be a particular reason on why StrictVersion has
been used here, so let's use LooseVersion, to be compatible with these
versions.
Co-authored-by: Adrien Ferrand <adferrand@users.noreply.github.com>
* Base logic
* Various controls when email is None
* Adapt eff tests
* Forward compatibility
* Also for csr
* Explicit regr or meta updates in account objects
* Adapt logic to ask for eff subscription during registering
* Adapt tests
* Move dry-run control
* Add some relevant controls on handle_subscription call checks
This will allow DNS plugin snaps to build if they rely on unreleased acme/certbot, and remove other copy of Certbot from externally snapped plugins. Fixes#8064 and fixes#7946. Implementation is based on the design [here](https://github.com/certbot/certbot/issues/8064#issuecomment-645513120).
To test, see reverted commit 8632064. Steps taken:
- added changes to setup.py and snapcraft.yaml
- successfully snapped, connected, ran `sudo certbot plugins --prepare`
- added temporary changes to have both certbot and certbot-dns-dnsimple use DNSAuthenticator2
- snapped and installed certbot, `certbot plugins` failed as expected.
- snapped and installed certbot-dns-dnsimple, `sudo certbot plugins --prepare` succeeded
- Inspected dns plugin's `bin` and `lib`; no `certbot` or `acme`, as expected.
```
$ ls /snap/certbot-dns-dnsimple/current/lib/python3.6/site-packages/
OpenSSL future-0.18.2.dist-info requests_file.py
PyYAML-5.3.1.dist-info idna setuptools
_cffi_backend.cpython-36m-x86_64-linux-gnu.so idna-2.9.dist-info setuptools-47.3.1.dist-info
certbot_dns_dnsimple lexicon six-1.15.0.dist-info
certbot_dns_dnsimple-1.6.0.dev0-py3.6.egg-info libfuturize six.py
certifi libpasteurize tldextract
certifi-2020.4.5.1.dist-info past tldextract-2.2.2.dist-info
cffi pip urllib3
cffi-1.14.0.dist-info pip-20.1.1.dist-info urllib3-1.25.9.dist-info
chardet pkg_resources wheel
chardet-3.0.4.dist-info pyOpenSSL-19.1.0.dist-info wheel-0.34.2.dist-info
cryptography pycparser yaml
cryptography-2.8.dist-info pycparser-2.20.dist-info zope
dns_lexicon-3.3.26.dist-info requests zope.interface-5.1.0-py3.6-nspkg.pth
easy_install.py requests-2.23.0.dist-info zope.interface-5.1.0.dist-info
future requests_file-1.5.1.dist-info
$ ls /snap/certbot-dns-dnsimple/current/bin/
chardetect futurize lexicon pasteurize tldextract
```
- reset to HEAD^
- snapped and installed certbot to not have the DNSAuthenticator2 changes, `certbot plugins` failed as expected.
* Don't include certbot deps when EXCLUDE_CERTBOT_DEPS is set
* Set EXCLUDE_CERTBOT_DEPS in certbot-dns-dnsimple/snap/snapcraft.yaml
fix branch name
grammar
remove readme link
Remove links to Robie's repo.
say you cant use docker
Commands not command
Update publishing permissions section.
Have Certbot trust plugins.
Do not run snapcraft with sudo.
Upgrade snap to be based on core20
This PR makes several changes to be built on top of the core20 base snap. Fixes#7854.
The main changes are to `snapcraft.yaml`. With Snapcraft 4.0/core20 base, the python plugin is a thin wrapper, basically creating a `venv` and installing the packages from the source. The trouble with this is that it runs pycache, creating caches that conflict from the different parts. So to solve that, we put everything in a single part. Other changes include:
- We use classic confinement, so we need to specify a bunch of python packages to `stage-packages`, as mentioned [here](https://forum.snapcraft.io/t/trouble-bundling-python-with-classic-confinement-in-core20-4-0-4/18234/2).
- The certbot executable now lives in `bin`, so specify running `certbot/bin`.
- Since `python-augeas` is now being pulled into the single part, remove the pinning from constraints so we can use the latest version directly from github.
- Precompile our `cryptography` and `cffi` wheels to be based on python3.8.
Separately, we had to upgrade the snapcraft docker image to be based on focal, due to the thin wrapper situation. This was accomplished [here](https://github.com/adferrand/snapcraft/pull/1).
* Get rid of a whole bunch of error message
* Remove some more overlaps
* don't use certbot from nginx and apache
* use python3 from bin
* certbot needs to be in bin
* try to exclude just the certbot folder
* try a couple things to use the python from the venv bin
* play around with which versions of things we want from each package
* ok, certbot-nginx does need to stage bin
* certbot needs to not stage bin. why does certbot not put certbot in bin?
* fail to inspect more versions of things in the container shell
* take cffi backend from python-augeas
* if we use certbot from bin things should work?
* why is bin not in path? no idea, but let's get it compiled then inspect things in the snap shell
* use snap.certbot instead of bin/certbot
* it does require bin/certbot. I don't know why.
* let's see if we can stick it all in one step
* try installing local subdirectories
* move python-augeas into the single part
* remove after
* put back python-augeas part for now; ERROR: Could not satisfy constraints for 'python-augeas': installation from path or url cannot be constrained to a version
* how was this previously working without git installed? install git.
* maybe it needs to already have python3-wheel installed
* maybe wheel will install first if I change it to -e
* no -e
* maybe try a different python3 package to stage
* this last change wasn't necessary
* remove the bin/ from renew
* nope, it does need bin/certbot
* back to wget
* stage a bare python3
* add all necessary python packages to stage-packages
* pretty sure we don't actually need wheel. let's try removing it!
* remove python-augeas, since we have it pinned to an older version in cb-auto that might work
* stage augeas
* still need libaugeas-dev
* ok let's try building
* combining into one part works! just make sure to unpin python-augeas when generating snap-constraints.txt
* change our scripts to unpin python-augeas
* Use ubuntu 20 in compile_native_wheels.sh
* .travis.yml should use python3-dev instead of python-dev
* jk! we don't need python3-dev in travis
* Update cffi and cryptography wheels for ubuntu20 version of python
* looks like we need python3-dev to build things
* Remove deprecated i386 wheels
I initially added this when the script was doing things like migrating all LXD containers to the snap. I think the external side effects are now pretty minimal thought so I think we can remove the need for this environment variable which makes it easier to use outside of CI for manual testing.
* Tweaks for improved Cloudflare API
* Update docs for dns-cloudflare
* Update tests and changelog
* Fix bad merge
* Fix error code for record add
* Improve error message
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
This PR proposes a way to build the certbot snap for several architectures, using a QEMU-base emulation approach, and several optimizations to keep the build time of each snap below 20 minutes.
Most of the reasoning for the approach proposed here is described in the original PR: https://github.com/basak/certbot-snap-build/pull/27
On top of it, I added a docker pull to a pre-compiled snapcraft docker, instead of compiling it during the Travis pipeline, in order to save 5 to 7 minutes more on each snap build. The snap images are compiled and stored here: https://hub.docker.com/repository/docker/adferrand/snapcraft. Depending on the time the PR will be reviewed, we can:
* continue to use `adferrand/snapcraft`
* move its logic to certbot scope and use something like `certbot/snapcraft`
* wait for https://github.com/snapcore/snapcraft/pull/3144 to be merged, and use `snapcore/snapcraft`.
* Backport https://github.com/basak/certbot-snap-build/pull/27 into Certbot project
* Fix build deps
* Integrate proactively #8012 to fix builds on non-amd64 archs
* Configure jobs on Travis
* Focus on snap builds. Disable temporarily some jobs. Disable deploy actions by security.
* Specify TARGET_ARCH during snap build
* Do not do anything if TOXENV is not set
* Various optimizations
* Use recent version of ubuntu for get correct features on snap out of the box
* Add up to date wheels
* Organizing scripts
* Set dest dir
* Get back original configuration for Travis
* Add comments
* Update common_libs.sh
* Use adferrand/snapcraft
* Test build
* Stable snapcraft
* Update build_and_install.sh
* Move back snap builds to the cron/release pipeline
* Update snap/local/compile_native_wheels.sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update snap/local/compile_native_wheels.sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update snap/local/compile_native_wheels.sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update snap/local/build_and_install.sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Enable i386 builds, various optimizations
* Update dependencies
* Configure a simple http server to serve the pre compiled wheels
* Fix wheels compilation
* Relax permissions
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Add snap plugin support
Switch to a PoC branch of certbot that supports the new
CERTBOT_PLUGIN_PATH and wrap the snap to set this variable correctly
based on the content interfaces connected.
(cherry picked from commit 7076a55fd82116d068e2aca7239209b7203917d2)
* Modify certbot.wrapper to append to PYTHONPATH instead of separate CERTBOT_PLUGIN_PATH variable
* Update certbot-wrapper to python3.6 version
* add source field
* Update changelog
* Use bash instead of sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Exit if something goes wrong
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* No leading : when modifying empty PYTHONPATH
* Improve bash handling of PYTHONPATH manipulation
Co-authored-by: Robie Basak <robie.basak@canonical.com>
Co-authored-by: Brad Warren <bmw@eff.org>
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
This PR gets its root from an observation I did on current version of Certbot (1.3.0): the `renewal-hooks` directory in Certbot configuration directory is created on Windows with write permissions to everybody.
I thought it was a critical bug since this directory contains hooks that are executed by Certbot, and you certainly do not want this folder to be open to any malicious hook that could be inserted by everyone, then executed with administrator privileges by Certbot.
Turns out for this specific problem that the bug is not critical for the hooks, because the scripts are expected to be in subdirectories of `renewal-hooks` (namely `pre`, `post` and `deploy`), and these subdirectories have proper permissions because we set them explicitly when Certbot is starting.
Still, there is a divergence here between Linux and Windows: on Linux all Certbot directories without explicit permissions have at maximum `0o755` permissions by default, while on Windows it is a `0o777` equivalent. It is not an immediate security risk, but it is definitly error-prone, not expected, and so a potential breach in the future if we forget about it.
Root cause is that umask is not existing in Windows. Indeed under Linux the umask defines the default permissions when you create a file or a directory. Python takes that into account, with an API for `os.open` and `os.mkdir` that expose a `mode` parameter with default value of `0o777`. In practice it is never `0o777` (either you the the `mode` explictly or left the default one) because the effective mode is masked by the current umask value in the system: on Linux it is `0o022`, so files/directories have a maximum mode of `0o755` if you did not set the umask explicitly, and it is what it is observed for Certbot.
However on Windows, the `mode` value passed (and got from default) to the `open` and `mkdir` of `certbot.compat.filesystem` module is taken verbatim, since umask does not exit, and then is used to calculate the DACL of the newly created file/directory. So if the mode is not set explicitly, we end up with files and directories with `0o777` permissions.
This PR fixes this problem by implementing a umask behavior in the `certbot.compat.filesystem` module, that will be applied to any file or directory created by Certbot since we forbid to use the `os` module directly.
The implementation is quite straight-forward. For Linux the behavior is not changed. On Windows a `mask` parameter is added to the function that calculates the DACL, to be invoked appropriately when file or directory are created. The actual value of the mask is taken from an internal class of the `filesystem` module: its default value is `0o755` to match default umasks on Linux, and can be changed with the new method `umask` that have the same behavior than the original `os.umask`. Of course `os.umask` becomes a forbidden function and `filesystem.umask` must be used instead.
Existing code that is impacted have been updated, and new unit tests are created for this new function.
* Implement umask for Windows
* Set umask at the beginning of tests
* Fix lint, update local oldest requirements
* Update certbot-apache/setup.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Improve tests
* Adapt filesystem.makedirs for Windows
* Fix
* Update certbot-apache/setup.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Changelog entries
* Fix lint
* Update certbot/CHANGELOG.md
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Issue #1123 discusses a feature that allows users to set the cipher
security level. That feature wasn't built. It didn't provide enough
user value to justify the corresponding increase in complexity. The
feature request and the associated discussion threads were closed.
However, the proposed API spec and the TODO section remained in the
cipher docs. They're a vestige of that issue from olden days and this PR
removes those last living traces...
Fixes#8027.
* Add support for NetBSD by telling certbot-nginx where the nginx
configuration directory is.
* Update the CHANGELOG.
* Pass the right type of sequence to "in". Thanks lint.
* Adjust the CHANGELOG.md entry following feedback from ohemorange.
Co-authored-by: Lloyd Parkes <lloyd@must-have-coffee.gen.nz>
In #7771, the Apache configurator gained the ability to identify what
version of OpenSSL Apache's ssl_module is linked against. However, the
detection was only functional if the module was built as a DSO (which is
almost always the case).
This commit covers the case where the ssl_module is statically linked
within the Apache binary. It requires the user to specify the path to
the binary (with --apache-bin) and emits a warning if static linking is
detected but no path has been provided.
This PR upgrades Certbot pinned dependencies through `letsencrypt-auto-source/rebuild_dependencies.py` while taking into account the problems detected in https://github.com/certbot/certbot/pull/8035:
* `cryptography` is pinned to `2.8` to continue to support OpenSSL 1.0.1 on non-x86 ancient Linux distributions (RHEL 6 + Debian 8)
* `parsedatetime` is pinned to `2.5` because of an incompatibility with Python 2.7 (see https://github.com/bear/parsedatetime/issues/246)
* `letsencrypt-auto-source/rebuild_dependencies.py` now takes into account the environment markers that are aded to `AUTHORITATIVE_CONSTRAINTS`: this is used for the `enum34` dependency, to not install it on Python 3.6+ and not break the distribution by swapping the built-in `enum` module during the setup of Certbot venv.
Fixes#8030
* Pin cryptography and parsedatetime
* Upgrade dependencies
* Remove authoritative constraint
* Upgrade dependencies
* Rebuild certbot-auto
* Update letsencrypt-auto-source/rebuild_dependencies.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Honor specific requirements in the AUTHORITATIVE_CONSTRAINTS
* Fix injection
* Update dependencies
* Update rebuild_dependencies.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
