This PR is the second part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This PR creates the module certbot.compat.os, that delegates everything to os, and that will be the safeguard against problematic methods of the standard module. On top of that, a quality check wrapper is called in the lint tox environment. This wrapper calls pylint and ensures that standard os module is no used directly in the certbot codebase.
Finally local oldest requirements are updated to ensure that tests will take the new logic when running.
* Add executable permissions
* Add the delegate certbot.compat.os module, add check coding style to enforce usage of certbot.compat.os instead of standard os
* Load certbot.compat.os instead of os
* Move existing compat test
* Update local oldest requirements
* Import sys
* Update account_test.py
* Update os.py
* Update os.py
* Update local oldest requirements
* Implement the new linter_plugin
* Fix local oldest for nginx
* Remove check coding style
* Update linter_plugin.py
* Add several comments
* Update the setup.py
* Add documentation
* Update acme dependencies
* Update certbot/compat/os.py
* Update docs/contributing.rst
* Update linter_plugin.py
* Handle os.path. Simplify checker.
* Add a comment to a reference implementation
* Update changelog
* Fix module registering
* Update docs/contributing.rst
* Update config and changelog
This PR adds a step to Apache plugin config_test when run on Fedora. Because Fedora now creates self signed certificate and related key material upon first startup of httpd. This was causing issues for users who run certbot-auto or install certbot (and mod_ssl) and run Certbot directly after.
Fixes: #6828
* Try to restart httpd on Fedora if config check fails
* Update CHANGELOG.md
In CentOS 6 default httpd configuration, the `LoadModule ssl_module ...` is handled in `conf.d/ssl.conf`. As the `VirtualHost` configuration files in `conf.d/` are loaded in alphabetical order, this means that all files that have `<IfModule mod_ssl.c>` and are loaded before `ssl.conf` are effectively ignored. This PR moves the `LoadModule ssl_module` to the main `httpd.conf` while leaving a conditional `LoadModule` directive in `ssl.conf`.
Features
- Reads the module configuration from `ssl.conf` in case some modifications to paths have been made by the user.
- Falls back to default paths if the directive doesn't exist.
- Moves the `LoadModule` directive in `ssl.conf` inside `<IfModule !mod_ssl.c>` to avoid printing warning messages of duplicate module loads.
- Adds `LoadModule ssl_module` inside of `<IfModule !mod_ssl.c>` to the top of the main `httpd.conf`.
- Ensures that these modifications are not made multiple times.
Fixes: #6606
* Fix CentOS6 installer issue
* Changelog entry
* Address review comments
* Do not enable mod_ssl if multiple different values were found
* Add test comment
* Address rest of the review comments
* Address review comments
* Better ifmodule argument checking
* Test fixes
* Make linter happy
* Raise an exception when differing LoadModule ssl_module statements are found
* If IfModule !mod_ssl.c with LoadModule ssl_module already exists in Augeas path, do not create new LoadModule directive
* Do not use deprecated assertion functions
* Address review comments
* Kick tests
* Revert "Kick tests"
This reverts commit 967bb574c2.
* Address review comments
* Add pydoc return value to create_ifmod
Add a new test to make sure that we are covering all the branches of get_virtual_hosts() regardless of the order that Augeas returns the found VirtualHost paths.
Fixes: #6813
* Add a test to ensure test coverage regardless of the order of returned vhosts
* Use deepcopy instead, and increase coverage requirement back to 100%
To fix one of the two uncovered lines in certbot-apache, given in #6880. Instead of adding a test to just increase the coverage, this fixes the uncovered line by removing the unused code.
* Add a dedicated configuration to define what is the HTTPS port for this certbot instance.
* Remove tls-sni in apache plugin
* Update constants.py
* Update interfaces.py
* Remove option
* Simplify a test
Apache plugin will now use command line default values from `ApacheConfingurator.OS_DEFAULTS` instead of respective distribution override when `CERTBOT_DOCS=1` environment variable is present.
Fixes: #6234
* Apache: respect CERTBOT_DOCS environment variable
* Move the tests to apache plugin
Attempts to configure all of the following VirtualHosts for answering the HTTP challenge:
* VirtualHosts that have the requested domain name in either `ServerName` or `ServerAlias` directive.
* VirtualHosts that have a wildcard name that would match the requested domain name.
This also applies to HTTPS VirtualHosts, making Apache plugin able to handle cases where HTTP redirection takes place in reverse proxy or similar, before reaching the Apache HTTPD.
Even though also HTTPS VirtualHosts are selected, Apache plugin tries to ensure that at least one of the selected VirtualHosts listens to HTTP-01 port (configured with `--http-01-port` CLI option). So in a case where only HTTPS VirtualHosts exist, but user wants to configure those, `--http-01-port` parameter needs to be set for the port configured to the HTTPS VirtualHost(s).
Fixes: #6730
* Select all matching VirtualHosts for HTTP-01 challenges instead of just one
* Finalize PR and add tests
* Changelog entry
This will immediately address the breakage reported in #6682 and tracked at #6685. Pip 19.0.0 and 19.0.1 don't allow commas in filenames, so don't use commas in filenames in certbot-apache test code.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
Steps to test:
push master to test box
run tools/venv.py
activate venv
pip --version: 19.0.1
pip install ./certbot-apache/: fails
push branch code to test box
confirm pip --version still 19.0.1
pip install ./certbot-apache/: success
* Rename old,default.conf to old-and-default.conf
* Update changelog
* sites-enabled should contain a symlink to sites-available
Fixes#6585.
I wrote up three suggestions for fixing this at https://github.com/certbot/certbot/issues/6585#issuecomment-448054502. I took the middle approach of requiring the user to provide an ACME server to use. I like this better than the other approaches which were:
> Resolve#5938 instead of this issue.
There is value in these tests as is over the compatibility tests in that they don't use Docker and run on different OSes.
> Spin up a local Python server to return the directory object.
Trying to set up a dummy ACME server seemed hacky and error prone.
Other notes about this PR are:
* I put the Pebble setup in `tox.ini` rather than `.travis.yml` as this seems much cleaner and more natural.
* I created a new `tox` environment called `apacheconftest-with-pebble` that reuses the code from `testenv:apacheconftest` so `apacheconftest` can continue to be used with servers other than Pebble like is done in our test farm tests.
* I chose the environment variable `SERVER` for consistency with our integration tests. I chose to not give this environment variable a default but to fail fast when it is not set.
* I ran test farm tests on this PR and they passed.
This PR adds two new command line parameters, --apache-ctlpath and --apache-binpath both of which are used to construct commands that we shell out for.
The way that we previously fetched values either from Certbot configuration object or the dictionary of distribution based constants is now also unified, and the active options are parsed in prepare() to make it easier to override needed values for the distributions needing this behavior.
Fixes: #5338
* Added the command line options and parsing
* Refactor existing code
* Distro override updates
* Handle vhost_root from cli
* Fix compatibility tests
* Add comment about changes to command line arguments
* Check None properly
* Made help texts consistent
* Keep the old defaults
* Move to shorter CLI parameter names
* No need for specific bin path, nor apache_cmd anymore
* Make sure that we use user provided vhost-root value
* Fix alt restart commands in overrides
* Fix version_cmd defaults in overrides
* Fix comparison
* Remove cruft, and use configuration object for parser parameter
If user provides a custom --apache-vhost-root path that's not parsed by Apache per default, Certbot fails the challenge validation. While the VirtualHost on custom path is correctly found, and edited, it's still not seen by Apache. This PR adds a temporary Include directive to the root Apache configuration when writing the challenge tokens to the VirtualHost.
interfaces.GenericUpdater and new enhancement interface updater functions get run on every invocation of Certbot with "renew" verb for every lineage. This causes performance problems for users with large configurations, because of plugin plumbing and preparsing happening in prepare() method of installer plugins. This PR moves the responsibility to call prepare() to the plugin (possibly) implementing a new style enhancement interface.
Fixes: #6153
* Do not call IPlugin.prepare() for updaters when running renew
* Check prepare called in tests
* Refine pydoc and make the function name more informative
* Verify the plugin type
* Remove apacheconftest packages.
The apacheconftests handle installing Apache dependencies, so let's remove it from the general case.
* We don't need to run dpkg -s in before_install.
* Remove augeas sources.
We only needed it for Ubuntu Precise which is dead and it doesn't work in Ubuntu Xenial.
* Upgrade Python 3.6 tests to 3.7.
Let's continue the approach of testing on the oldest and newest versions of Python 3. We will continue testing on Python 3.6 in the nightly tests.
* Revert "We don't need to run dpkg -s in before_install."
This reverts commit e5d35099a7.
* let apacheconftest handle deps
This PR adds the functionality to enhance Apache configuration to include HTTP Strict Transport Security header with a low initial max-age value.
The max-age value will get increased on every (scheduled) run of certbot renew regardless of the certificate actually getting renewed, if the last increase took place longer than ten hours ago. The increase steps are visible in constants.AUTOHSTS_STEPS.
Upon the first actual renewal after reaching the maximum increase step, the max-age value will be made "permanent" and will get value of one year.
To achieve accurate VirtualHost discovery on subsequent runs, a comment with unique id string will be added to each enhanced VirtualHost.
* AutoHSTS code rebased on master
* Fixes to match the changes in master
* Make linter happy with metaclass registration
* Address small review comments
* Use new enhancement interfaces
* New style enhancement changes
* Do not allow --hsts and --auto-hsts simultaneuously
* MyPy annotation fixes and added test
* Change oldest requrements to point to local certbot core version
* Enable new style enhancements for run and install verbs
* Test refactor
* New test class for main.install tests
* Move a test to a correct test class
* Remove unsupported pylint disable options
* star-args removed in Pylint 1.4.3
* abstract-class-little-used removed in Pylint 1.4.3
* Fixes new lint errors
* Copy dummy-variable-rgx expression to new ignored-argument-names expression to ignore unused funtion arguments
* Notable changes
* Refactor to satisfy Pylint no-else-return warning
* Fix Pylint inconsistent-return-statements warning
* Refactor to satisfy consider-iterating-dictionary
* Remove methods with only super call to satisfy useless-super-delegation
* Refactor too-many-nested-statements where possible
* Suppress type checked errors where member is dynamically added (notably derived from josepy.JSONObjectWithFields)
* Remove None default of func parameter for ExitHandler and ErrorHandler
Resolves#5973
* resolved mypy untyped defs in parser.py
* resolved mypy untyped defs in obj.py
* removed unused imports
* resolved mypy untyped defs in http_01.py
* resolved mypy untyped defs in tls_sni_01.py
* resolved mypy untyped defs in configurator.py
* address mypy too-many-arguments error in override_centos.py
* resolved mypy untyped defs in http_01_test.py
* removed unused 'conf' argument that was causing mypy method assignment error
* address mypy error where same variable reassigned to different type
* address pylint and coverage issues
* one character space change for formatting
* fix required acme version for certbot-apache
- Fix code to log separate error messages when either SSLCertificateFile or SSLCertificateKeyFile -
directives are not found.
- Update the section in install.rst where the relevant error is referenced.
- Edit a docstring where 'cert' previously referred to certificate.
- Edit test_deploy_cert_invalid_vhost in the test suite to cover changes.
Fixes#5525.
In `deploy_cert()` and `enhance()`, the user will be presented with a dialog to choose from the VirtualHosts that can be covered by the wildcard domain name. The (multiple) selection result will then be handled in a similar way that we previously handled a single VirtualHost that was returned by the `_find_best_vhost()`.
Additionally the selected VirtualHosts are added to a dictionary that maps selections to a wildcard domain to be reused in the later `enhance()` call and not forcing the user to select the same VirtualHosts again.
* Apache plugin wildcard support
* Present dialog only once per domain, added tests
* Raise exception if no VHosts selected for wildcard domain
Unfortunately, the way that Apache merges the configuration directives is different for mod_rewrite and <Location> / <Directory> directives.
To work around basic auth in VirtualHosts, the challenge override Include had to be split in two. The first part handles overrides for RewriteRule and the other part will handle overrides for <Directory> and <Location> directives.
* Fix docstring quote spacing
* Remove unneeded directives
* Enable mod_rewrite
* Remove ifmod rewrite
* Use stricter rewriterule
* Uncomment tests
* Fix order args
* Remove S which doesn't seem to work across contexts
* Use double backslash to make pylint
* Fix enmod test
* Fix http-01 tests
* Test for rewrite
* check for Include in vhost
* add test_same_vhost
* Don't add includes twice
* Include default vhosts in search
* Respect port in find_best_http_vhost
* Add find_best_http_vhost port test
* Filter by port in http01
* Add a simple version of HTTP01
* remove cert from chall name
* make directory work on 2.2
* cleanup challenges when finished
* import shutil
* fixup perform and cleanup tests
* Add tests for http_01.py
