diff --git a/.gitignore b/.gitignore index b653cb06c..48ec7910b 100644 --- a/.gitignore +++ b/.gitignore @@ -17,9 +17,11 @@ letsencrypt-auto-source/letsencrypt-auto.sig.lzma.base64 /.vagrant +tags + # editor temporary files *~ -*.swp +*.sw? \#*# .idea diff --git a/.travis.yml b/.travis.yml index 6f964dbec..5ccf39811 100644 --- a/.travis.yml +++ b/.travis.yml @@ -34,6 +34,8 @@ matrix: - python: "2.7" env: TOXENV=apacheconftest sudo: required + - python: "2.7" + env: TOXENV=nginxroundtrip - python: "2.7" env: TOXENV=py27 BOULDER_INTEGRATION=1 sudo: true @@ -53,6 +55,16 @@ matrix: services: docker before_install: addons: + - sudo: required + env: TOXENV=apache_compat + services: docker + before_install: + addons: + - sudo: required + env: TOXENV=nginx_compat + services: docker + before_install: + addons: - python: "2.7" env: TOXENV=cover - python: "3.3" diff --git a/README.rst b/README.rst index c71079f9a..72188608b 100644 --- a/README.rst +++ b/README.rst @@ -1,169 +1,19 @@ -.. notice for github users +.. This file contains of a series of comments that are used to include sections of this README in other files. Do not modify these comments unless you know what you are doing. tag:intro-begin -Disclaimer -========== +Certbot is part of EFF’s effort to encrypt the entire Internet. Secure communication over the Web relies on HTTPS, which requires the use of a digital certificate that lets browsers verify the identify of web servers (e.g., is that really google.com?). Web servers obtain their certificates from trusted third parties called certificate authorities (CAs). Certbot is an easy-to-use client that fetches a certificate from Let’s Encrypt—an open certificate authority launched by the EFF, Mozilla, and others—and deploys it to a web server. -Certbot (previously, the Let's Encrypt client) is **BETA SOFTWARE**. It -contains plenty of bugs and rough edges, and should be tested thoroughly in -staging environments before use on production systems. +Anyone who has gone through the trouble of setting up a secure website knows what a hassle getting and maintaining a certificate is. Certbot and Let’s Encrypt can automate away the pain and let you turn on and manage HTTPS with simple commands. Using Certbot and Let's Encrypt is free, so there’s no need to arrange payment. -For more information regarding the status of the project, please see -https://letsencrypt.org. Be sure to checkout the -`Frequently Asked Questions (FAQ) `_. +How you use Certbot depends on the configuration of your web server. The best way to get started is to use our `interactive guide `_. It generates instructions based on your configuration settings. In most cases, you’ll need `root or administrator access `_ to your web server to run Certbot. -About Certbot -============================== +If you’re using a hosted service and don’t have direct access to your web server, you might not be able to use Certbot. Check with your hosting provider for documentation about uploading certificates or using certificates issues by Let’s Encrypt. -Certbot is a fully-featured, extensible client for the Let's -Encrypt CA (or any other CA that speaks the `ACME -`_ -protocol) that can automate the tasks of obtaining certificates and -configuring webservers to use them. This client runs on Unix-based operating -systems. - -Until May 2016, Certbot was named simply ``letsencrypt`` or ``letsencrypt-auto``, -depending on install method. Instructions on the Internet, and some pieces of the -software, may still refer to this older name. - -Contributing ------------- - -If you'd like to contribute to this project please read `Developer Guide -`_. - -.. _installation: - -Installation ------------- - -If ``certbot`` (or ``letsencrypt``) is packaged for your Unix OS (visit -certbot.eff.org_ to find out), you can install it -from there, and run it by typing ``certbot`` (or ``letsencrypt``). Because -not all operating systems have packages yet, we provide a temporary solution -via the ``certbot-auto`` wrapper script, which obtains some dependencies from -your OS and puts others in a python virtual environment:: - - user@webserver:~$ wget https://dl.eff.org/certbot-auto - user@webserver:~$ chmod a+x ./certbot-auto - user@webserver:~$ ./certbot-auto --help - -.. hint:: The certbot-auto download is protected by HTTPS, which is pretty good, but if you'd like to - double check the integrity of the ``certbot-auto`` script, you can use these steps for verification before running it:: - - user@server:~$ wget -N https://dl.eff.org/certbot-auto.asc - user@server:~$ gpg2 --recv-key A2CFB51FA275A7286234E7B24D17C995CD9775F2 - user@server:~$ gpg2 --trusted-key 4D17C995CD9775F2 --verify certbot-auto.asc certbot-auto - -And for full command line help, you can type:: - - ./certbot-auto --help all - -``certbot-auto`` updates to the latest client release automatically. And -since ``certbot-auto`` is a wrapper to ``certbot``, it accepts exactly -the same command line flags and arguments. More details about this script and -other installation methods can be found `in the User Guide -`_. - -How to run the client ---------------------- - -In many cases, you can just run ``certbot-auto`` or ``certbot``, and the -client will guide you through the process of obtaining and installing certs -interactively. - -You can also tell it exactly what you want it to do from the command line. -For instance, if you want to obtain a cert for ``example.com``, -``www.example.com``, and ``other.example.net``, using the Apache plugin to both -obtain and install the certs, you could do this:: - - ./certbot-auto --apache -d example.com -d www.example.com -d other.example.net - -(The first time you run the command, it will make an account, and ask for an -email and agreement to the Let's Encrypt Subscriber Agreement; you can -automate those with ``--email`` and ``--agree-tos``) - -If you want to use a webserver that doesn't have full plugin support yet, you -can still use "standalone" or "webroot" plugins to obtain a certificate:: - - ./certbot-auto certonly --standalone --email admin@example.com -d example.com -d www.example.com -d other.example.net - - -Understanding the client in more depth --------------------------------------- - -To understand what the client is doing in detail, it's important to -understand the way it uses plugins. Please see the `explanation of -plugins `_ in -the User Guide. - -Links -===== - -Documentation: https://certbot.eff.org/docs - -Software project: https://github.com/certbot/certbot - -Notes for developers: https://certbot.eff.org/docs/contributing.html - -Main Website: https://letsencrypt.org/ - -IRC Channel: #letsencrypt on `Freenode`_ or #certbot on `OFTC`_ - -Community: https://community.letsencrypt.org - -ACME spec: http://ietf-wg-acme.github.io/acme/ - -ACME working area in github: https://github.com/ietf-wg-acme/acme - - -Mailing list: `client-dev`_ (to subscribe without a Google account, send an -email to client-dev+subscribe@letsencrypt.org) - -|build-status| |coverage| |docs| |container| - - - -.. |build-status| image:: https://travis-ci.org/certbot/certbot.svg?branch=master - :target: https://travis-ci.org/certbot/certbot - :alt: Travis CI status - -.. |coverage| image:: https://coveralls.io/repos/certbot/certbot/badge.svg?branch=master - :target: https://coveralls.io/r/certbot/certbot - :alt: Coverage status - -.. |docs| image:: https://readthedocs.org/projects/letsencrypt/badge/ - :target: https://readthedocs.org/projects/letsencrypt/ - :alt: Documentation status - -.. |container| image:: https://quay.io/repository/letsencrypt/letsencrypt/status - :target: https://quay.io/repository/letsencrypt/letsencrypt - :alt: Docker Repository on Quay.io - -.. _`installation instructions`: - https://letsencrypt.readthedocs.org/en/latest/using.html - -.. _watch demo video: https://www.youtube.com/watch?v=Gas_sSB-5SU - -System Requirements -=================== - -The Let's Encrypt Client presently only runs on Unix-ish OSes that include -Python 2.6 or 2.7; Python 3.x support will hopefully be added in the future. The -client requires root access in order to write to ``/etc/letsencrypt``, -``/var/log/letsencrypt``, ``/var/lib/letsencrypt``; to bind to ports 80 and 443 -(if you use the ``standalone`` plugin) and to read and modify webserver -configurations (if you use the ``apache`` or ``nginx`` plugins). If none of -these apply to you, it is theoretically possible to run without root privileges, -but for most users who want to avoid running an ACME client as root, either -`letsencrypt-nosudo `_ or -`simp_le `_ are more appropriate choices. - -The Apache plugin currently requires a Debian-based OS with augeas version -1.0; this includes Ubuntu 12.04+ and Debian 7+. +.. Do not modify this comment unless you know what you're doing. tag:intro-end +.. Do not modify this comment unless you know what you're doing. tag:features-begin Current Features -================ +===================== * Supports multiple web servers: @@ -187,8 +37,6 @@ Current Features command line. * Free and Open Source Software, made with Python. +.. Do not modify this comment unless you know what you're doing. tag:features-end -.. _Freenode: https://webchat.freenode.net?channels=%23letsencrypt -.. _OFTC: https://webchat.oftc.net?channels=%23certbot -.. _client-dev: https://groups.google.com/a/letsencrypt.org/forum/#!forum/client-dev -.. _certbot.eff.org: https://certbot.eff.org/ +For extensive documentation on using and contributing to Certbot, go to https://certbot.eff.org/docs. If you would like to contribute to the project or run the latest code from git, you should read our `developer guide `_. diff --git a/acme/acme/challenges.py b/acme/acme/challenges.py index c436cc631..6242c376c 100644 --- a/acme/acme/challenges.py +++ b/acme/acme/challenges.py @@ -14,7 +14,6 @@ from acme import crypto_util from acme import fields from acme import jose - logger = logging.getLogger(__name__) @@ -206,6 +205,74 @@ class KeyAuthorizationChallenge(_TokenChallenge): self.validation(account_key, *args, **kwargs)) +@ChallengeResponse.register +class DNS01Response(KeyAuthorizationChallengeResponse): + """ACME dns-01 challenge response.""" + typ = "dns-01" + + def simple_verify(self, chall, domain, account_public_key): + """Simple verify. + + :param challenges.DNS01 chall: Corresponding challenge. + :param unicode domain: Domain name being verified. + :param JWK account_public_key: Public key for the key pair + being authorized. + + :returns: ``True`` iff validation with the TXT records resolved from a + DNS server is successful. + :rtype: bool + + """ + if not self.verify(chall, account_public_key): + logger.debug("Verification of key authorization in response failed") + return False + + validation_domain_name = chall.validation_domain_name(domain) + validation = chall.validation(account_public_key) + logger.debug("Verifying %s at %s...", chall.typ, validation_domain_name) + + try: + from acme import dns_resolver + except ImportError: # pragma: no cover + raise errors.Error("Local validation for 'dns-01' challenges " + "requires 'dnspython'") + txt_records = dns_resolver.txt_records_for_name(validation_domain_name) + exists = validation in txt_records + if not exists: + logger.debug("Key authorization from response (%r) doesn't match " + "any DNS response in %r", self.key_authorization, + txt_records) + return exists + + +@Challenge.register # pylint: disable=too-many-ancestors +class DNS01(KeyAuthorizationChallenge): + """ACME dns-01 challenge.""" + response_cls = DNS01Response + typ = response_cls.typ + + LABEL = "_acme-challenge" + """Label clients prepend to the domain name being validated.""" + + def validation(self, account_key, **unused_kwargs): + """Generate validation. + + :param JWK account_key: + :rtype: unicode + + """ + return jose.b64encode(hashlib.sha256(self.key_authorization( + account_key).encode("utf-8")).digest()).decode() + + def validation_domain_name(self, name): + """Domain name for TXT validation record. + + :param unicode name: Domain name being validated. + + """ + return "{0}.{1}".format(self.LABEL, name) + + @ChallengeResponse.register class HTTP01Response(KeyAuthorizationChallengeResponse): """ACME http-01 challenge response.""" @@ -231,8 +298,8 @@ class HTTP01Response(KeyAuthorizationChallengeResponse): being authorized. :param int port: Port used in the validation. - :returns: ``True`` iff validation is successful, ``False`` - otherwise. + :returns: ``True`` iff validation with the files currently served by the + HTTP server is successful. :rtype: bool """ @@ -410,7 +477,7 @@ class TLSSNI01Response(KeyAuthorizationChallengeResponse): :returns: ``True`` iff client's control of the domain has been - verified, ``False`` otherwise. + verified. :rtype: bool """ diff --git a/acme/acme/challenges_test.py b/acme/acme/challenges_test.py index 04b7442b0..27976931a 100644 --- a/acme/acme/challenges_test.py +++ b/acme/acme/challenges_test.py @@ -77,6 +77,93 @@ class KeyAuthorizationChallengeResponseTest(unittest.TestCase): self.assertFalse(response.verify(self.chall, KEY.public_key())) +class DNS01ResponseTest(unittest.TestCase): + # pylint: disable=too-many-instance-attributes + + def setUp(self): + from acme.challenges import DNS01Response + self.msg = DNS01Response(key_authorization=u'foo') + self.jmsg = { + 'resource': 'challenge', + 'type': 'dns-01', + 'keyAuthorization': u'foo', + } + + from acme.challenges import DNS01 + self.chall = DNS01(token=(b'x' * 16)) + self.response = self.chall.response(KEY) + + def test_to_partial_json(self): + self.assertEqual(self.jmsg, self.msg.to_partial_json()) + + def test_from_json(self): + from acme.challenges import DNS01Response + self.assertEqual(self.msg, DNS01Response.from_json(self.jmsg)) + + def test_from_json_hashable(self): + from acme.challenges import DNS01Response + hash(DNS01Response.from_json(self.jmsg)) + + def test_simple_verify_bad_key_authorization(self): + key2 = jose.JWKRSA.load(test_util.load_vector('rsa256_key.pem')) + self.response.simple_verify(self.chall, "local", key2.public_key()) + + @mock.patch("acme.dns_resolver.txt_records_for_name") + def test_simple_verify_good_validation(self, mock_resolver): + mock_resolver.return_value = [self.chall.validation(KEY.public_key())] + self.assertTrue(self.response.simple_verify( + self.chall, "local", KEY.public_key())) + mock_resolver.assert_called_once_with( + self.chall.validation_domain_name("local")) + + @mock.patch("acme.dns_resolver.txt_records_for_name") + def test_simple_verify_good_validation_multiple_txts(self, mock_resolver): + mock_resolver.return_value = [ + "!", self.chall.validation(KEY.public_key())] + self.assertTrue(self.response.simple_verify( + self.chall, "local", KEY.public_key())) + mock_resolver.assert_called_once_with( + self.chall.validation_domain_name("local")) + + @mock.patch("acme.dns_resolver.txt_records_for_name") + def test_simple_verify_bad_validation(self, mock_dns): + mock_dns.return_value = ["!"] + self.assertFalse(self.response.simple_verify( + self.chall, "local", KEY.public_key())) + + +class DNS01Test(unittest.TestCase): + + def setUp(self): + from acme.challenges import DNS01 + self.msg = DNS01(token=jose.decode_b64jose( + 'evaGxfADs6pSRb2LAv9IZf17Dt3juxGJ+PCt92wr+oA')) + self.jmsg = { + 'type': 'dns-01', + 'token': 'evaGxfADs6pSRb2LAv9IZf17Dt3juxGJ-PCt92wr-oA', + } + + def test_validation_domain_name(self): + self.assertEqual('_acme-challenge.www.example.com', + self.msg.validation_domain_name('www.example.com')) + + def test_validation(self): + self.assertEqual( + "rAa7iIg4K2y63fvUhCfy8dP1Xl7wEhmQq0oChTcE3Zk", + self.msg.validation(KEY)) + + def test_to_partial_json(self): + self.assertEqual(self.jmsg, self.msg.to_partial_json()) + + def test_from_json(self): + from acme.challenges import DNS01 + self.assertEqual(self.msg, DNS01.from_json(self.jmsg)) + + def test_from_json_hashable(self): + from acme.challenges import DNS01 + hash(DNS01.from_json(self.jmsg)) + + class HTTP01ResponseTest(unittest.TestCase): # pylint: disable=too-many-instance-attributes diff --git a/acme/acme/dns_resolver.py b/acme/acme/dns_resolver.py new file mode 100644 index 000000000..f551c6095 --- /dev/null +++ b/acme/acme/dns_resolver.py @@ -0,0 +1,30 @@ +"""DNS Resolver for ACME client. +Required only for local validation of 'dns-01' challenges. +""" +import logging + +import dns.resolver +import dns.exception + +logger = logging.getLogger(__name__) + + +def txt_records_for_name(name): + """Resolve the name and return the TXT records. + + :param unicode name: Domain name being verified. + + :returns: A list of txt records, if empty the name could not be resolved + :rtype: list of unicode + + """ + try: + dns_response = dns.resolver.query(name, 'TXT') + except dns.resolver.NXDOMAIN as error: + return [] + except dns.exception.DNSException as error: + logger.error("Error resolving %s: %s", name, str(error)) + return [] + + return [txt_rec.decode("utf-8") for rdata in dns_response + for txt_rec in rdata.strings] diff --git a/acme/acme/dns_resolver_test.py b/acme/acme/dns_resolver_test.py new file mode 100644 index 000000000..53fc0cc77 --- /dev/null +++ b/acme/acme/dns_resolver_test.py @@ -0,0 +1,53 @@ +"""Tests for acme.dns_resolver.""" +import unittest +import mock + +from acme import dns_resolver + +try: + import dns +except ImportError: # pragma: no cover + dns = None + + +def create_txt_response(name, txt_records): + """ + Returns an RRSet containing the 'txt_records' as the result of a DNS + query for 'name'. + + This takes advantage of the fact that an Answer object mostly behaves + like an RRset. + """ + return dns.rrset.from_text_list(name, 60, "IN", "TXT", txt_records) + + +class TxtRecordsForNameTest(unittest.TestCase): + + @mock.patch("acme.dns_resolver.dns.resolver.query") + def test_txt_records_for_name_with_single_response(self, mock_dns): + mock_dns.return_value = create_txt_response('name', ['response']) + self.assertEqual(['response'], + dns_resolver.txt_records_for_name('name')) + + @mock.patch("acme.dns_resolver.dns.resolver.query") + def test_txt_records_for_name_with_multiple_responses(self, mock_dns): + mock_dns.return_value = create_txt_response( + 'name', ['response1', 'response2']) + self.assertEqual(['response1', 'response2'], + dns_resolver.txt_records_for_name('name')) + + @mock.patch("acme.dns_resolver.dns.resolver.query") + def test_txt_records_for_name_domain_not_found(self, mock_dns): + mock_dns.side_effect = dns.resolver.NXDOMAIN + self.assertEquals([], dns_resolver.txt_records_for_name('name')) + + @mock.patch("acme.dns_resolver.dns.resolver.query") + def test_txt_records_for_name_domain_other_error(self, mock_dns): + mock_dns.side_effect = dns.exception.DNSException + self.assertEquals([], dns_resolver.txt_records_for_name('name')) + + def run(self, result=None): + if dns is None: # pragma: no cover + print(self, "... SKIPPING, no dnspython available") + return + super(TxtRecordsForNameTest, self).run(result) diff --git a/acme/acme/fields.py b/acme/acme/fields.py index 002240b23..12d09acf4 100644 --- a/acme/acme/fields.py +++ b/acme/acme/fields.py @@ -24,7 +24,7 @@ class Fixed(jose.Field): def encode(self, value): if value != self.value: - logger.warn( + logger.warning( 'Overriding fixed field (%s) with %r', self.json_name, value) return value diff --git a/acme/setup.py b/acme/setup.py index ed133e128..94f78d4cd 100644 --- a/acme/setup.py +++ b/acme/setup.py @@ -35,6 +35,11 @@ if sys.version_info < (2, 7): else: install_requires.append('mock') +# dnspython 1.12 is required to support both Python 2 and Python 3. +dns_extras = [ + 'dnspython>=1.12', +] + dev_extras = [ 'nose', 'pep8', @@ -76,6 +81,7 @@ setup( include_package_data=True, install_requires=install_requires, extras_require={ + 'dns': dns_extras, 'dev': dev_extras, 'docs': docs_extras, }, diff --git a/certbot-apache/certbot_apache/configurator.py b/certbot-apache/certbot_apache/configurator.py index 5c86ab4fb..17ff6c8db 100644 --- a/certbot-apache/certbot_apache/configurator.py +++ b/certbot-apache/certbot_apache/configurator.py @@ -244,7 +244,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): if not path["cert_path"] or not path["cert_key"]: # Throw some can't find all of the directives error" - logger.warn( + logger.warning( "Cannot find a cert or key directive in %s. " "VirtualHost was not modified", vhost.path) # Presumably break here so that the virtualhost is not modified @@ -522,7 +522,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): try: args = self.aug.match(path + "/arg") except RuntimeError: - logger.warn("Encountered a problem while parsing file: %s, skipping", path) + logger.warning("Encountered a problem while parsing file: %s, skipping", path) return None for arg in args: addrs.add(obj.Addr.fromstring(self.parser.get_arg(arg))) @@ -538,6 +538,9 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): is_ssl = True filename = get_file_path(self.aug.get("/augeas/files%s/path" % get_file_path(path))) + if filename is None: + return None + if self.conf("handle-sites"): is_enabled = self.is_site_enabled(filename) else: @@ -1089,7 +1092,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): try: func(self.choose_vhost(domain), options) except errors.PluginError: - logger.warn("Failed %s for %s", enhancement, domain) + logger.warning("Failed %s for %s", enhancement, domain) raise def _enable_ocsp_stapling(self, ssl_vhost, unused_options): @@ -1276,9 +1279,9 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): # but redirect loops are possible in very obscure cases; see #1620 # for reasoning. if self._is_rewrite_exists(general_vh): - logger.warn("Added an HTTP->HTTPS rewrite in addition to " - "other RewriteRules; you may wish to check for " - "overall consistency.") + logger.warning("Added an HTTP->HTTPS rewrite in addition to " + "other RewriteRules; you may wish to check for " + "overall consistency.") # Add directives to server # Note: These are not immediately searchable in sites-enabled @@ -1801,25 +1804,25 @@ def get_file_path(vhost_path): :rtype: str """ - # Strip off /files - avail_fp = vhost_path[6:] - # This can be optimized... - while True: - # Cast all to lowercase to be case insensitive - find_if = avail_fp.lower().find("/ifmodule") - if find_if != -1: - avail_fp = avail_fp[:find_if] - continue - find_vh = avail_fp.lower().find("/virtualhost") - if find_vh != -1: - avail_fp = avail_fp[:find_vh] - continue - find_macro = avail_fp.lower().find("/macro") - if find_macro != -1: - avail_fp = avail_fp[:find_macro] - continue - break - return avail_fp + # Strip off /files/ + try: + if vhost_path.startswith("/files/"): + avail_fp = vhost_path[7:].split("/") + else: + return None + except AttributeError: + # If we recieved a None path + return None + + last_good = "" + # Loop through the path parts and validate after every addition + for p in avail_fp: + cur_path = last_good+"/"+p + if os.path.exists(cur_path): + last_good = cur_path + else: + break + return last_good def install_ssl_options_conf(options_ssl): diff --git a/certbot-apache/certbot_apache/constants.py b/certbot-apache/certbot_apache/constants.py index 9252814c4..ba545c613 100644 --- a/certbot-apache/certbot_apache/constants.py +++ b/certbot-apache/certbot_apache/constants.py @@ -2,7 +2,23 @@ import pkg_resources from certbot import util - +CLI_DEFAULTS_DEFAULT = dict( + server_root="/etc/apache2", + vhost_root="/etc/apache2/sites-available", + vhost_files="*", + version_cmd=['apache2ctl', '-v'], + define_cmd=['apache2ctl', '-t', '-D', 'DUMP_RUN_CFG'], + restart_cmd=['apache2ctl', 'graceful'], + conftest_cmd=['apache2ctl', 'configtest'], + enmod=None, + dismod=None, + le_vhost_ext="-le-ssl.conf", + handle_mods=False, + handle_sites=False, + challenge_location="/etc/apache2", + MOD_SSL_CONF_SRC=pkg_resources.resource_filename( + "certbot_apache", "options-ssl-apache.conf") +) CLI_DEFAULTS_DEBIAN = dict( server_root="/etc/apache2", vhost_root="/etc/apache2/sites-available", @@ -71,7 +87,25 @@ CLI_DEFAULTS_DARWIN = dict( MOD_SSL_CONF_SRC=pkg_resources.resource_filename( "certbot_apache", "options-ssl-apache.conf") ) +CLI_DEFAULTS_SUSE = dict( + server_root="/etc/apache2", + vhost_root="/etc/apache2/vhosts.d", + vhost_files="*.conf", + version_cmd=['apache2ctl', '-v'], + define_cmd=['apache2ctl', '-t', '-D', 'DUMP_RUN_CFG'], + restart_cmd=['apache2ctl', 'graceful'], + conftest_cmd=['apache2ctl', 'configtest'], + enmod="a2enmod", + dismod="a2dismod", + le_vhost_ext="-le-ssl.conf", + handle_mods=False, + handle_sites=False, + challenge_location="/etc/apache2/vhosts.d", + MOD_SSL_CONF_SRC=pkg_resources.resource_filename( + "certbot_apache", "options-ssl-apache.conf") +) CLI_DEFAULTS = { + "default": CLI_DEFAULTS_DEFAULT, "debian": CLI_DEFAULTS_DEBIAN, "ubuntu": CLI_DEFAULTS_DEBIAN, "centos": CLI_DEFAULTS_CENTOS, @@ -83,6 +117,8 @@ CLI_DEFAULTS = { "gentoo": CLI_DEFAULTS_GENTOO, "gentoo base system": CLI_DEFAULTS_GENTOO, "darwin": CLI_DEFAULTS_DARWIN, + "opensuse": CLI_DEFAULTS_SUSE, + "suse": CLI_DEFAULTS_SUSE, } """CLI defaults.""" @@ -115,13 +151,36 @@ HEADER_ARGS = {"Strict-Transport-Security": HSTS_ARGS, def os_constant(key): - """Get a constant value for operating system + """ + Get a constant value for operating system + :param key: name of cli constant :return: value of constant for active os """ + os_info = util.get_os_info() try: constants = CLI_DEFAULTS[os_info[0].lower()] except KeyError: - constants = CLI_DEFAULTS["debian"] + constants = os_like_constants() + if not constants: + constants = CLI_DEFAULTS["default"] return constants[key] + + +def os_like_constants(): + """ + Try to get constants for distribution with + similar layout and configuration, indicated by + /etc/os-release variable "LIKE" + + :returns: Constants dictionary + :rtype: `dict` + """ + + os_like = util.get_systemd_os_like() + if os_like: + for os_name in os_like: + if os_name in CLI_DEFAULTS.keys(): + return CLI_DEFAULTS[os_name] + return {} diff --git a/certbot-apache/certbot_apache/display_ops.py b/certbot-apache/certbot_apache/display_ops.py index 24aabaed4..d7b76f83d 100644 --- a/certbot-apache/certbot_apache/display_ops.py +++ b/certbot-apache/certbot_apache/display_ops.py @@ -91,7 +91,7 @@ def _vhost_menu(domain, vhosts): "non-interactive mode. Currently Certbot needs each vhost to be " "in its own conf file, and may need vhosts to be explicitly " "labelled with ServerName or ServerAlias directories.") - logger.warn(msg) + logger.warning(msg) raise errors.MissingCommandlineFlag(msg) return code, tag diff --git a/certbot-apache/certbot_apache/parser.py b/certbot-apache/certbot_apache/parser.py index 321546eb3..6bb6ff170 100644 --- a/certbot-apache/certbot_apache/parser.py +++ b/certbot-apache/certbot_apache/parser.py @@ -146,7 +146,7 @@ class ApacheParser(object): constants.os_constant("define_cmd")) # Small errors that do not impede if proc.returncode != 0: - logger.warn("Error in checking parameter list: %s", stderr) + logger.warning("Error in checking parameter list: %s", stderr) raise errors.MisconfigurationError( "Apache is unable to check whether or not the module is " "loaded because Apache is misconfigured.") diff --git a/certbot-apache/certbot_apache/tests/configurator_test.py b/certbot-apache/certbot_apache/tests/configurator_test.py index 0bbdac1aa..dc953174e 100644 --- a/certbot-apache/certbot_apache/tests/configurator_test.py +++ b/certbot-apache/certbot_apache/tests/configurator_test.py @@ -56,7 +56,7 @@ class MultipleVhostsTest(util.ApacheTest): mock_surgery.return_value = False with mock.patch.dict('os.environ', silly_path): self.assertRaises(errors.NoInstallationError, self.config.prepare) - self.assertEquals(mock_surgery.call_count, 1) + self.assertEqual(mock_surgery.call_count, 1) @mock.patch("certbot_apache.augeas_configurator.AugeasConfigurator.init_augeas") def test_prepare_no_augeas(self, mock_init_augeas): @@ -125,6 +125,12 @@ class MultipleVhostsTest(util.ApacheTest): self.assertTrue("google.com" in names) self.assertTrue("certbot.demo" in names) + def test_get_bad_path(self): + from certbot_apache.configurator import get_file_path + self.assertEqual(get_file_path(None), None) + self.assertEqual(get_file_path("nonexistent"), None) + self.assertEqual(self.config._create_vhost("nonexistent"), None) # pylint: disable=protected-access + def test_bad_servername_alias(self): ssl_vh1 = obj.VirtualHost( "fp1", "ap1", set([obj.Addr(("*", "443"))]), @@ -1242,8 +1248,8 @@ class MultipleVhostsTest(util.ApacheTest): mock_match = mock.Mock(return_value=["something"]) self.config.aug.match = mock_match # pylint: disable=protected-access - self.assertEquals(self.config._check_aug_version(), - ["something"]) + self.assertEqual(self.config._check_aug_version(), + ["something"]) self.config.aug.match.side_effect = RuntimeError self.assertFalse(self.config._check_aug_version()) diff --git a/certbot-apache/certbot_apache/tests/constants_test.py b/certbot-apache/certbot_apache/tests/constants_test.py index c040030df..1c842aee9 100644 --- a/certbot-apache/certbot_apache/tests/constants_test.py +++ b/certbot-apache/certbot_apache/tests/constants_test.py @@ -25,3 +25,20 @@ class ConstantsTest(unittest.TestCase): os_info.return_value = ('Nonexistent Linux', '', '') self.assertEqual(constants.os_constant("vhost_root"), "/etc/apache2/sites-available") + + @mock.patch("certbot.util.get_os_info") + def test_get_default_constants(self, os_info): + os_info.return_value = ('Nonexistent Linux', '', '') + with mock.patch("certbot.util.get_systemd_os_like") as os_like: + # Get defaults + os_like.return_value = False + c_hm = constants.os_constant("handle_mods") + c_sr = constants.os_constant("server_root") + self.assertFalse(c_hm) + self.assertEqual(c_sr, "/etc/apache2") + # Use darwin as like test target + os_like.return_value = ["something", "nonexistent", "darwin"] + d_vr = constants.os_constant("vhost_root") + d_em = constants.os_constant("enmod") + self.assertFalse(d_em) + self.assertEqual(d_vr, "/etc/apache2/other") diff --git a/certbot-apache/certbot_apache/tls_sni_01.py b/certbot-apache/certbot_apache/tls_sni_01.py index 06d9e9ebd..18179d080 100644 --- a/certbot-apache/certbot_apache/tls_sni_01.py +++ b/certbot-apache/certbot_apache/tls_sni_01.py @@ -129,7 +129,7 @@ class ApacheTlsSni01(common.TLSSNI01): # because it's a new vhost that's not configured yet (GH #677), # or perhaps because there were multiple sections # in the config file (GH #1042). See also GH #2600. - logger.warn("Falling back to default vhost %s...", default_addr) + logger.warning("Falling back to default vhost %s...", default_addr) addrs.add(default_addr) return addrs diff --git a/certbot-compatibility-test/Dockerfile b/certbot-compatibility-test/Dockerfile new file mode 100644 index 000000000..e445a3555 --- /dev/null +++ b/certbot-compatibility-test/Dockerfile @@ -0,0 +1,51 @@ +FROM debian:jessie +MAINTAINER Brad Warren + +# no need to mkdir anything: +# https://docs.docker.com/reference/builder/#copy +# If doesn't exist, it is created along with all missing +# directories in its path. + +# TODO: Install non-default Python versions for tox. +# TODO: Install Apache/Nginx for plugin development. +COPY certbot-auto /opt/certbot/src/certbot-auto +RUN /opt/certbot/src/certbot-auto -n --os-packages-only + +# the above is not likely to change, so by putting it further up the +# Dockerfile we make sure we cache as much as possible + +COPY setup.py README.rst CHANGES.rst MANIFEST.in linter_plugin.py tox.cover.sh tox.ini pep8.travis.sh .pep8 .pylintrc /opt/certbot/src/ + +# all above files are necessary for setup.py, however, package source +# code directory has to be copied separately to a subdirectory... +# https://docs.docker.com/reference/builder/#copy: "If is a +# directory, the entire contents of the directory are copied, +# including filesystem metadata. Note: The directory itself is not +# copied, just its contents." Order again matters, three files are far +# more likely to be cached than the whole project directory + +COPY certbot /opt/certbot/src/certbot/ +COPY acme /opt/certbot/src/acme/ +COPY certbot-apache /opt/certbot/src/certbot-apache/ +COPY certbot-nginx /opt/certbot/src/certbot-nginx/ +COPY certbot-compatibility-test /opt/certbot/src/certbot-compatibility-test/ + +RUN virtualenv --no-site-packages -p python2 /opt/certbot/venv && \ + /opt/certbot/venv/bin/pip install -U setuptools && \ + /opt/certbot/venv/bin/pip install -U pip && \ + /opt/certbot/venv/bin/pip install \ + -e /opt/certbot/src/acme \ + -e /opt/certbot/src \ + -e /opt/certbot/src/certbot-apache \ + -e /opt/certbot/src/certbot-nginx \ + -e /opt/certbot/src/certbot-compatibility-test \ + -e /opt/certbot/src[dev,docs] + +# install in editable mode (-e) to save space: it's not possible to +# "rm -rf /opt/certbot/src" (it's stays in the underlaying image); +# this might also help in debugging: you can "docker run --entrypoint +# bash" and investigate, apply patches, etc. + +WORKDIR /opt/certbot/src/certbot-compatibility-test/certbot_compatibility_test/testdata + +ENV PATH /opt/certbot/venv/bin:$PATH diff --git a/certbot-compatibility-test/Dockerfile-apache b/certbot-compatibility-test/Dockerfile-apache new file mode 100644 index 000000000..5c0495966 --- /dev/null +++ b/certbot-compatibility-test/Dockerfile-apache @@ -0,0 +1,6 @@ +FROM certbot-compatibility-test +MAINTAINER Brad Warren + +RUN apt-get install apache2 -y + +ENTRYPOINT [ "certbot-compatibility-test", "-p", "apache" ] diff --git a/certbot-compatibility-test/Dockerfile-nginx b/certbot-compatibility-test/Dockerfile-nginx new file mode 100644 index 000000000..4ade03065 --- /dev/null +++ b/certbot-compatibility-test/Dockerfile-nginx @@ -0,0 +1,6 @@ +FROM certbot-compatibility-test +MAINTAINER Brad Warren + +RUN apt-get install nginx -y + +ENTRYPOINT [ "certbot-compatibility-test", "-p", "nginx" ] diff --git a/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/Dockerfile b/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/Dockerfile deleted file mode 100644 index ea9bb857f..000000000 --- a/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/Dockerfile +++ /dev/null @@ -1,20 +0,0 @@ -FROM httpd -MAINTAINER Brad Warren - -RUN mkdir /var/run/apache2 - -ENV APACHE_RUN_USER=daemon \ - APACHE_RUN_GROUP=daemon \ - APACHE_PID_FILE=/usr/local/apache2/logs/httpd.pid \ - APACHE_RUN_DIR=/var/run/apache2 \ - APACHE_LOCK_DIR=/var/lock \ - APACHE_LOG_DIR=/usr/local/apache2/logs - -COPY certbot-compatibility-test/certbot_compatibility_test/configurators/apache/a2enmod.sh /usr/local/bin/ -COPY certbot-compatibility-test/certbot_compatibility_test/configurators/apache/a2dismod.sh /usr/local/bin/ -COPY certbot-compatibility-test/certbot_compatibility_test/testdata/rsa1024_key2.pem /usr/local/apache2/conf/ -COPY certbot-compatibility-test/certbot_compatibility_test/testdata/empty_cert.pem /usr/local/apache2/conf/ - -# Note: this only exposes the port to other docker containers. You -# still have to bind to 443@host at runtime. -EXPOSE 443 diff --git a/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/common.py b/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/common.py index ed3d9d67a..64170ca72 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/common.py +++ b/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/common.py @@ -1,5 +1,4 @@ """Provides a common base for Apache proxies""" -import re import os import shutil import subprocess @@ -17,10 +16,6 @@ from certbot_compatibility_test import util from certbot_compatibility_test.configurators import common as configurators_common -APACHE_VERSION_REGEX = re.compile(r"Apache/([0-9\.]*)", re.IGNORECASE) -APACHE_COMMANDS = ["apachectl", "a2enmod", "a2dismod"] - - @zope.interface.implementer(interfaces.IConfiguratorProxy) class Proxy(configurators_common.Proxy): # pylint: disable=too-many-instance-attributes @@ -32,37 +27,25 @@ class Proxy(configurators_common.Proxy): self.le_config.apache_le_vhost_ext = "-le-ssl.conf" self.modules = self.server_root = self.test_conf = self.version = None - self._apache_configurator = self._all_names = self._test_names = None patch = mock.patch( "certbot_apache.configurator.display_ops.select_vhost") mock_display = patch.start() mock_display.side_effect = le_errors.PluginError( "Unable to determine vhost") - def __getattr__(self, name): - """Wraps the Apache Configurator methods""" - method = getattr(self._apache_configurator, name, None) - if callable(method): - return method - else: - raise AttributeError() - def load_config(self): """Loads the next configuration for the plugin to test""" - config = super(Proxy, self).load_config() self._all_names, self._test_names = _get_names(config) server_root = _get_server_root(config) - # with open(os.path.join(config, "config_file")) as f: - # config_file = os.path.join(server_root, f.readline().rstrip()) shutil.rmtree("/etc/apache2") shutil.copytree(server_root, "/etc/apache2", symlinks=True) self._prepare_configurator() try: - subprocess.check_call("apachectl -k start".split()) + subprocess.check_call("apachectl -k restart".split()) except errors.Error: raise errors.Error( "Apache failed to load {0} before tests started".format( @@ -78,38 +61,16 @@ class Proxy(configurators_common.Proxy): # An alias self.le_config.apache_handle_modules = self.le_config.apache_handle_mods - self._apache_configurator = configurator.ApacheConfigurator( + self._configurator = configurator.ApacheConfigurator( config=configuration.NamespaceConfig(self.le_config), name="apache") - self._apache_configurator.prepare() + self._configurator.prepare() def cleanup_from_tests(self): """Performs any necessary cleanup from running plugin tests""" super(Proxy, self).cleanup_from_tests() mock.patch.stopall() - def get_all_names_answer(self): - """Returns the set of domain names that the plugin should find""" - if self._all_names: - return self._all_names - else: - raise errors.Error("No configuration file loaded") - - def get_testable_domain_names(self): - """Returns the set of domain names that can be tested against""" - if self._test_names: - return self._test_names - else: - return {"example.com"} - - def deploy_cert(self, domain, cert_path, key_path, chain_path=None, - fullchain_path=None): - """Installs cert""" - cert_path, key_path, chain_path = self.copy_certs_and_keys( - cert_path, key_path, chain_path) - self._apache_configurator.deploy_cert( - domain, cert_path, key_path, chain_path, fullchain_path) - def _get_server_root(config): """Returns the server root directory in config""" diff --git a/certbot-compatibility-test/certbot_compatibility_test/configurators/common.py b/certbot-compatibility-test/certbot_compatibility_test/configurators/common.py index 03128cc86..2a800c1c2 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/configurators/common.py +++ b/certbot-compatibility-test/certbot_compatibility_test/configurators/common.py @@ -5,6 +5,7 @@ import shutil import tempfile from certbot import constants +from certbot_compatibility_test import errors from certbot_compatibility_test import util @@ -31,6 +32,18 @@ class Proxy(object): self.args = args self.http_port = 80 self.https_port = 443 + self._configurator = self._all_names = self._test_names = None + + def __getattr__(self, name): + """Wraps the configurator methods""" + if self._configurator is None: + raise AttributeError() + + method = getattr(self._configurator, name, None) + if callable(method): + return method + else: + raise AttributeError() def has_more_configs(self): """Returns true if there are more configs to test""" @@ -63,3 +76,25 @@ class Proxy(object): chain = None return cert, key, chain + + def get_all_names_answer(self): + """Returns the set of domain names that the plugin should find""" + if self._all_names: + return self._all_names + else: + raise errors.Error("No configuration file loaded") + + def get_testable_domain_names(self): + """Returns the set of domain names that can be tested against""" + if self._test_names: + return self._test_names + else: + return {"example.com"} + + def deploy_cert(self, domain, cert_path, key_path, chain_path=None, + fullchain_path=None): + """Installs cert""" + cert_path, key_path, chain_path = self.copy_certs_and_keys( + cert_path, key_path, chain_path) + self._configurator.deploy_cert( + domain, cert_path, key_path, chain_path, fullchain_path) diff --git a/certbot-compatibility-test/certbot_compatibility_test/configurators/nginx/__init__.py b/certbot-compatibility-test/certbot_compatibility_test/configurators/nginx/__init__.py new file mode 100644 index 000000000..ed294abe6 --- /dev/null +++ b/certbot-compatibility-test/certbot_compatibility_test/configurators/nginx/__init__.py @@ -0,0 +1 @@ +"""Certbot compatibility test Nginx configurators""" diff --git a/certbot-compatibility-test/certbot_compatibility_test/configurators/nginx/common.py b/certbot-compatibility-test/certbot_compatibility_test/configurators/nginx/common.py new file mode 100644 index 000000000..3622bee41 --- /dev/null +++ b/certbot-compatibility-test/certbot_compatibility_test/configurators/nginx/common.py @@ -0,0 +1,84 @@ +"""Provides a common base for Nginx proxies""" +import os +import shutil +import subprocess + +import zope.interface + +from certbot import configuration +from certbot_nginx import configurator +from certbot_nginx import constants +from certbot_compatibility_test import errors +from certbot_compatibility_test import interfaces +from certbot_compatibility_test import util +from certbot_compatibility_test.configurators import common as configurators_common + + +@zope.interface.implementer(interfaces.IConfiguratorProxy) +class Proxy(configurators_common.Proxy): + # pylint: disable=too-many-instance-attributes + """A common base for Nginx test configurators""" + + def __init__(self, args): + """Initializes the plugin with the given command line args""" + super(Proxy, self).__init__(args) + + def load_config(self): + """Loads the next configuration for the plugin to test""" + config = super(Proxy, self).load_config() + self._all_names, self._test_names = _get_names(config) + + server_root = _get_server_root(config) + + # XXX: Deleting all of this is kind of scary unless the test + # instances really each have a complete configuration! + shutil.rmtree("/etc/nginx") + shutil.copytree(server_root, "/etc/nginx", symlinks=True) + + self._prepare_configurator() + + try: + subprocess.check_call("service nginx reload".split()) + except errors.Error: + raise errors.Error( + "Nginx failed to load {0} before tests started".format( + config)) + + return config + + def _prepare_configurator(self): + """Prepares the Nginx plugin for testing""" + for k in constants.CLI_DEFAULTS.keys(): + setattr(self.le_config, "nginx_" + k, constants.os_constant(k)) + + conf = configuration.NamespaceConfig(self.le_config) + zope.component.provideUtility(conf) + self._configurator = configurator.NginxConfigurator( + config=conf, name="nginx") + self._configurator.prepare() + + +def _get_server_root(config): + """Returns the server root directory in config""" + subdirs = [ + name for name in os.listdir(config) + if os.path.isdir(os.path.join(config, name))] + + if len(subdirs) != 1: + raise errors.Error("Malformed configuration directory {0}".format(config)) + + return os.path.join(config, subdirs[0].rstrip()) + + +def _get_names(config): + """Returns all and testable domain names in config""" + all_names = set() + for root, _dirs, files in os.walk(config): + for this_file in files: + for line in open(os.path.join(root, this_file)): + if line.strip().startswith("server_name"): + names = line.partition("server_name")[2].rpartition(";")[0] + for n in names.split(): + all_names.add(n) + non_ip_names = set(n for n in all_names if not util.IP_REGEX.match(n)) + return all_names, non_ip_names diff --git a/certbot-compatibility-test/certbot_compatibility_test/test_driver.py b/certbot-compatibility-test/certbot_compatibility_test/test_driver.py index 38abffb18..b5e023f36 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/test_driver.py +++ b/certbot-compatibility-test/certbot_compatibility_test/test_driver.py @@ -22,7 +22,8 @@ from certbot_compatibility_test import errors from certbot_compatibility_test import util from certbot_compatibility_test import validator -from certbot_compatibility_test.configurators.apache import common +from certbot_compatibility_test.configurators.apache import common as a_common +from certbot_compatibility_test.configurators.nginx import common as n_common DESCRIPTION = """ @@ -32,7 +33,7 @@ tests that the plugin supports are performed. """ -PLUGINS = {"apache": common.Proxy} +PLUGINS = {"apache": a_common.Proxy, "nginx": n_common.Proxy} logger = logging.getLogger(__name__) @@ -143,7 +144,7 @@ def test_deploy_cert(plugin, temp_dir, domains): for domain in domains: try: - plugin.deploy_cert(domain, cert_path, util.KEY_PATH, cert_path) + plugin.deploy_cert(domain, cert_path, util.KEY_PATH, cert_path, cert_path) plugin.save() # Needed by the Apache plugin except le_errors.Error as error: logger.error("Plugin failed to deploy ceritificate for %s:", domain) @@ -369,10 +370,10 @@ def main(): plugin.cleanup_from_tests() if overall_success: - logger.warn("All compatibility tests succeeded") + logger.warning("All compatibility tests succeeded") sys.exit(0) else: - logger.warn("One or more compatibility tests failed") + logger.warning("One or more compatibility tests failed") sys.exit(1) diff --git a/certbot-compatibility-test/certbot_compatibility_test/testdata/configs.tar.gz b/certbot-compatibility-test/certbot_compatibility_test/testdata/apache.tar.gz similarity index 100% rename from certbot-compatibility-test/certbot_compatibility_test/testdata/configs.tar.gz rename to certbot-compatibility-test/certbot_compatibility_test/testdata/apache.tar.gz diff --git a/certbot-compatibility-test/certbot_compatibility_test/testdata/nginx.tar.gz b/certbot-compatibility-test/certbot_compatibility_test/testdata/nginx.tar.gz new file mode 100644 index 000000000..4ca5cc977 Binary files /dev/null and b/certbot-compatibility-test/certbot_compatibility_test/testdata/nginx.tar.gz differ diff --git a/certbot-nginx/certbot_nginx/configurator.py b/certbot-nginx/certbot_nginx/configurator.py index d11b6bb14..a1c24b5c8 100644 --- a/certbot-nginx/certbot_nginx/configurator.py +++ b/certbot-nginx/certbot_nginx/configurator.py @@ -160,9 +160,9 @@ class NginxConfigurator(common.Plugin): stapling_directives = [] if self.version >= (1, 3, 7): stapling_directives = [ - ['\n', 'ssl_trusted_certificate', ' ', chain_path], - ['\n', 'ssl_stapling', ' ', 'on'], - ['\n', 'ssl_stapling_verify', ' ', 'on'], ['\n']] + ['\n ', 'ssl_trusted_certificate', ' ', chain_path], + ['\n ', 'ssl_stapling', ' ', 'on'], + ['\n ', 'ssl_stapling_verify', ' ', 'on'], ['\n']] if len(stapling_directives) != 0 and not chain_path: raise errors.PluginError( @@ -179,7 +179,7 @@ class NginxConfigurator(common.Plugin): vhost.filep, vhost.names) except errors.MisconfigurationError as error: logger.debug(error) - logger.warn( + logger.warning( "Cannot find a cert or key directive in %s for %s. " "VirtualHost was not modified.", vhost.filep, vhost.names) # Presumably break here so that the virtualhost is not modified @@ -337,10 +337,10 @@ class NginxConfigurator(common.Plugin): """ snakeoil_cert, snakeoil_key = self._get_snakeoil_paths() - ssl_block = [['\n', 'listen', ' ', '{0} ssl'.format(self.config.tls_sni_01_port)], - ['\n', 'ssl_certificate', ' ', snakeoil_cert], - ['\n', 'ssl_certificate_key', ' ', snakeoil_key], - ['\n', 'include', ' ', self.parser.loc["ssl_options"]]] + ssl_block = [['\n ', 'listen', ' ', '{0} ssl'.format(self.config.tls_sni_01_port)], + ['\n ', 'ssl_certificate', ' ', snakeoil_cert], + ['\n ', 'ssl_certificate_key', ' ', snakeoil_key], + ['\n ', 'include', ' ', self.parser.loc["ssl_options"]]] self.parser.add_server_directives( vhost.filep, vhost.names, ssl_block, replace=False) vhost.ssl = True @@ -385,7 +385,7 @@ class NginxConfigurator(common.Plugin): raise errors.PluginError( "Unsupported enhancement: {0}".format(enhancement)) except errors.PluginError: - logger.warn("Failed %s for %s", enhancement, domain) + logger.warning("Failed %s for %s", enhancement, domain) def _enable_redirect(self, vhost, unused_options): """Redirect all equivalent HTTP traffic to ssl_vhost. @@ -401,9 +401,10 @@ class NginxConfigurator(common.Plugin): :type unused_options: Not Available """ redirect_block = [[ - ['if', '($scheme != "https")'], - [['return', '301 https://$host$request_uri']] - ]] + ['\n ', 'if', ' ', '($scheme != "https") '], + [['\n ', 'return', ' ', '301 https://$host$request_uri'], + '\n '] + ], ['\n']] self.parser.add_server_directives( vhost.filep, vhost.names, redirect_block, replace=False) logger.info("Redirecting all traffic to ssl in %s", vhost.filep) diff --git a/certbot-nginx/certbot_nginx/constants.py b/certbot-nginx/certbot_nginx/constants.py index 5dde30efc..8cf1f6bc9 100644 --- a/certbot-nginx/certbot_nginx/constants.py +++ b/certbot-nginx/certbot_nginx/constants.py @@ -16,3 +16,17 @@ MOD_SSL_CONF_SRC = pkg_resources.resource_filename( "certbot_nginx", "options-ssl-nginx.conf") """Path to the nginx mod_ssl config file found in the Certbot distribution.""" + +def os_constant(key): + # XXX TODO: In the future, this could return different constants + # based on what OS we are running under. To see an + # approach to how to handle different OSes, see the + # apache version of this file. Currently, we do not + # actually have any OS-specific constants on Nginx. + """ + Get a constant value for operating system + + :param key: name of cli constant + :return: value of constant for active os + """ + return CLI_DEFAULTS[key] diff --git a/certbot-nginx/certbot_nginx/nginxparser.py b/certbot-nginx/certbot_nginx/nginxparser.py index 1859777d8..c7a9f0c75 100644 --- a/certbot-nginx/certbot_nginx/nginxparser.py +++ b/certbot-nginx/certbot_nginx/nginxparser.py @@ -25,11 +25,19 @@ class RawNginxParser(object): key = Word(alphanums + "_/+-.") dollar_var = Combine(Literal('$') + Regex(r"[^\{\};,\s]+")) condition = Regex(r"\(.+\)") - # Matches anything that is not a special character AND any chars in single - # or double quotes + # Matches anything that is not a special character, and ${SHELL_VARS}, AND + # any chars in single or double quotes # All of these COULD be upgraded to something like # https://stackoverflow.com/a/16130746 - value = Regex(r"((\".*\")?(\'.*\')?[^\{\};,]?)+") + dquoted = Regex(r'(\".*\")') + squoted = Regex(r"(\'.*\')") + nonspecial = Regex(r"[^\{\};,]") + varsub = Regex(r"(\$\{\w+\})") + # nonspecial nibbles one character at a time, but the other objects take + # precedence. We use ZeroOrMore to allow entries like "break ;" to be + # parsed as assignments + value = Combine(ZeroOrMore(dquoted | squoted | varsub | nonspecial)) + location = CharsNotIn("{};," + string.whitespace) # modifier for location uri [ = | ~ | ~* | ^~ ] modifier = Literal("=") | Literal("~*") | Literal("~") | Literal("^~") @@ -40,6 +48,7 @@ class RawNginxParser(object): assignment = space + key + Optional(space + value, default=None) + semicolon location_statement = space + Optional(modifier) + Optional(space + location + space) if_statement = space + Literal("if") + space + condition + space + charset_map_statement = space + Literal("charset_map") + space + value + space + value map_statement = space + Literal("map") + space + nonspace + space + dollar_var + space # This is NOT an accurate way to parse nginx map entries; it's almost @@ -52,24 +61,27 @@ class RawNginxParser(object): map_pattern = Regex(r'".*"') | Regex(r"'.*'") | nonspace map_entry = space + map_pattern + space + value + space + semicolon map_block = Group( - # key could for instance be "server" or "http", or "location" (in which case - # location_statement needs to have a non-empty location) Group(map_statement).leaveWhitespace() + left_bracket + Group(ZeroOrMore(Group(comment | map_entry)) + space).leaveWhitespace() + right_bracket) block = Forward() - block << Group( - # key could for instance be "server" or "http", or "location" (in which case - # location_statement needs to have a non-empty location) - (Group(space + key + location_statement) ^ Group(if_statement)).leaveWhitespace() + - left_bracket + - Group(ZeroOrMore(Group(comment | assignment) | block | map_block) + space).leaveWhitespace() - + right_bracket) + + # key could for instance be "server" or "http", or "location" (in which case + # location_statement needs to have a non-empty location) + + block_begin = (Group(space + key + location_statement) ^ + Group(if_statement) ^ + Group(charset_map_statement)).leaveWhitespace() + + block_innards = Group(ZeroOrMore(Group(comment | assignment) | block | map_block) + + space).leaveWhitespace() + + block << Group(block_begin + left_bracket + block_innards + right_bracket) script = OneOrMore(Group(comment | assignment) ^ block ^ map_block) + space + stringEnd - script.parseWithTabs() + script.parseWithTabs().leaveWhitespace() def __init__(self, source): self.source = source diff --git a/certbot-nginx/certbot_nginx/parser.py b/certbot-nginx/certbot_nginx/parser.py index e7992810a..aae6ce122 100644 --- a/certbot-nginx/certbot_nginx/parser.py +++ b/certbot-nginx/certbot_nginx/parser.py @@ -168,7 +168,7 @@ class NginxParser(object): self.parsed[item] = parsed trees.append(parsed) except IOError: - logger.warn("Could not open file: %s", item) + logger.warning("Could not open file: %s", item) except pyparsing.ParseException: logger.debug("Could not parse file: %s", item) return trees diff --git a/certbot-nginx/certbot_nginx/tests/configurator_test.py b/certbot-nginx/certbot_nginx/tests/configurator_test.py index 7529bc10b..fede3bc08 100644 --- a/certbot-nginx/certbot_nginx/tests/configurator_test.py +++ b/certbot-nginx/certbot_nginx/tests/configurator_test.py @@ -37,8 +37,8 @@ class NginxConfiguratorTest(util.NginxTest): errors.NoInstallationError, self.config.prepare) def test_prepare(self): - self.assertEquals((1, 6, 2), self.config.version) - self.assertEquals(5, len(self.config.parser.parsed)) + self.assertEqual((1, 6, 2), self.config.version) + self.assertEqual(5, len(self.config.parser.parsed)) @mock.patch("certbot_nginx.configurator.util.exe_exists") @mock.patch("certbot_nginx.configurator.subprocess.Popen") @@ -56,7 +56,7 @@ class NginxConfiguratorTest(util.NginxTest): self.config.version = None self.config.config_test = mock.Mock() self.config.prepare() - self.assertEquals((1, 6, 2), self.config.version) + self.assertEqual((1, 6, 2), self.config.version) @mock.patch("certbot_nginx.configurator.socket.gethostbyaddr") def test_get_all_names(self, mock_gethostbyaddr): @@ -415,7 +415,7 @@ class NginxConfiguratorTest(util.NginxTest): def test_redirect_enhance(self): expected = [ - ['if', '($scheme != "https")'], + ['if', '($scheme != "https") '], [['return', '301 https://$host$request_uri']] ] diff --git a/certbot-nginx/certbot_nginx/tests/nginxparser_test.py b/certbot-nginx/certbot_nginx/tests/nginxparser_test.py index a0a0736a5..5fa9a7d1e 100644 --- a/certbot-nginx/certbot_nginx/tests/nginxparser_test.py +++ b/certbot-nginx/certbot_nginx/tests/nginxparser_test.py @@ -134,8 +134,8 @@ class TestRawNginxParser(unittest.TestCase): parsed_new = load(handle) try: self.maxDiff = None - self.assertEquals(parsed[0], parsed_new[0]) - self.assertEquals(parsed[1:], parsed_new[1:]) + self.assertEqual(parsed[0], parsed_new[0]) + self.assertEqual(parsed[1:], parsed_new[1:]) finally: os.unlink(util.get_data_filename('nginx.new.conf')) @@ -150,7 +150,7 @@ class TestRawNginxParser(unittest.TestCase): parsed_new = load(handle) try: - self.assertEquals(parsed, parsed_new) + self.assertEqual(parsed, parsed_new) self.assertEqual(parsed_new, [ ['#', " Use bar.conf when it's a full moon!"], diff --git a/certbot-nginx/certbot_nginx/tests/parser_test.py b/certbot-nginx/certbot_nginx/tests/parser_test.py index 464d717ed..77b4a877a 100644 --- a/certbot-nginx/certbot_nginx/tests/parser_test.py +++ b/certbot-nginx/certbot_nginx/tests/parser_test.py @@ -117,9 +117,9 @@ class NginxParserTest(util.NginxTest): fooconf = [x for x in vhosts if 'foo.conf' in x.filep][0] self.assertEqual(vhost5, fooconf) localhost = [x for x in vhosts if 'localhost' in x.names][0] - self.assertEquals(vhost1, localhost) + self.assertEqual(vhost1, localhost) somename = [x for x in vhosts if 'somename' in x.names][0] - self.assertEquals(vhost2, somename) + self.assertEqual(vhost2, somename) def test_add_server_directives(self): nparser = parser.NginxParser(self.config_path, self.ssl_options) diff --git a/certbot-nginx/certbot_nginx/tls_sni_01.py b/certbot-nginx/certbot_nginx/tls_sni_01.py index ebc92f5e3..c7ec80931 100644 --- a/certbot-nginx/certbot_nginx/tls_sni_01.py +++ b/certbot-nginx/certbot_nginx/tls_sni_01.py @@ -91,10 +91,10 @@ class NginxTlsSni01(common.TLSSNI01): # Add the 'include' statement for the challenges if it doesn't exist # already in the main config included = False - include_directive = ['include', ' ', self.challenge_conf] + include_directive = ['\n', 'include', ' ', self.challenge_conf] root = self.configurator.parser.loc["root"] - bucket_directive = ['server_names_hash_bucket_size', ' ', '128'] + bucket_directive = ['\n', 'server_names_hash_bucket_size', ' ', '128'] main = self.configurator.parser.parsed[root] for key, body in main: diff --git a/certbot/auth_handler.py b/certbot/auth_handler.py index f5557d604..a94734572 100644 --- a/certbot/auth_handler.py +++ b/certbot/auth_handler.py @@ -1,8 +1,8 @@ """ACME AuthHandler.""" -import itertools import logging import time +import six import zope.component from acme import challenges @@ -141,7 +141,7 @@ class AuthHandler(object): """ active_achalls = [] - for achall, resp in itertools.izip(achalls, resps): + for achall, resp in six.moves.zip(achalls, resps): # This line needs to be outside of the if block below to # ensure failed challenges are cleaned up correctly active_achalls.append(achall) @@ -472,7 +472,7 @@ def _report_failed_challs(failed_achalls): problems.setdefault(achall.error.typ, []).append(achall) reporter = zope.component.getUtility(interfaces.IReporter) - for achalls in problems.itervalues(): + for achalls in six.itervalues(problems): reporter.add_message( _generate_failed_chall_msg(achalls), reporter.MEDIUM_PRIORITY) diff --git a/certbot/cli.py b/certbot/cli.py index 22787392d..fbbc75845 100644 --- a/certbot/cli.py +++ b/certbot/cli.py @@ -158,7 +158,7 @@ def possible_deprecation_warning(config): # need warnings return if "CERTBOT_AUTO" not in os.environ: - logger.warn("You are running with an old copy of letsencrypt-auto that does " + logger.warning("You are running with an old copy of letsencrypt-auto that does " "not receive updates, and is less reliable than more recent versions. " "We recommend upgrading to the latest certbot-auto script, or using native " "OS packages.") @@ -343,8 +343,10 @@ class HelpfulArgumentParser(object): self.determine_verb() help1 = self.prescan_for_flag("-h", self.help_topics) help2 = self.prescan_for_flag("--help", self.help_topics) - assert max(True, "a") == "a", "Gravity changed direction" - self.help_arg = max(help1, help2) + if isinstance(help1, bool) and isinstance(help2, bool): + self.help_arg = help1 or help2 + else: + self.help_arg = help1 if isinstance(help1, str) else help2 if self.help_arg is True: # just --help with no topic; avoid argparse altogether print(usage) diff --git a/certbot/client.py b/certbot/client.py index 81b7ccdc6..0f414b474 100644 --- a/certbot/client.py +++ b/certbot/client.py @@ -104,10 +104,10 @@ def register(config, account_storage, tos_cb=None): if not config.register_unsafely_without_email: msg = ("No email was provided and " "--register-unsafely-without-email was not present.") - logger.warn(msg) + logger.warning(msg) raise errors.Error(msg) if not config.dry_run: - logger.warn("Registering without email!") + logger.warning("Registering without email!") # Each new registration shall use a fresh new key key = jose.JWKRSA(key=jose.ComparableRSAKey( @@ -453,10 +453,10 @@ class Client(object): try: self.installer.enhance(dom, enhancement, options) except errors.PluginEnhancementAlreadyPresent: - logger.warn("Enhancement %s was already set.", + logger.warning("Enhancement %s was already set.", enhancement) except errors.PluginError: - logger.warn("Unable to set enhancement %s for %s", + logger.warning("Unable to set enhancement %s for %s", enhancement, dom) raise diff --git a/certbot/constants.py b/certbot/constants.py index fb278161d..1ddb9fedf 100644 --- a/certbot/constants.py +++ b/certbot/constants.py @@ -18,7 +18,7 @@ CLI_DEFAULTS = dict( os.path.join(os.environ.get("XDG_CONFIG_HOME", "~/.config"), "letsencrypt", "cli.ini"), ], - verbose_count=-(logging.INFO / 10), + verbose_count=-int(logging.INFO / 10), server="https://acme-v01.api.letsencrypt.org/directory", rsa_key_size=2048, rollback_checkpoints=1, diff --git a/certbot/hooks.py b/certbot/hooks.py index 1a3e4a98e..f37f81c6e 100644 --- a/certbot/hooks.py +++ b/certbot/hooks.py @@ -53,7 +53,7 @@ def post_hook(config, final=False): if not pre_hook.already: logger.info("No renewals attempted, so not running post-hook") if config.verb != "renew": - logger.warn("Sanity failure in renewal hooks") + logger.warning("Sanity failure in renewal hooks") return if final or config.verb != "renew": logger.info("Running post-hook command: %s", config.post_hook) diff --git a/certbot/main.py b/certbot/main.py index 8bccc524d..511046df0 100644 --- a/certbot/main.py +++ b/certbot/main.py @@ -773,5 +773,5 @@ def main(cli_args=sys.argv[1:]): if __name__ == "__main__": err_string = main() if err_string: - logger.warn("Exiting with message %s", err_string) + logger.warning("Exiting with message %s", err_string) sys.exit(err_string) # pragma: no cover diff --git a/certbot/plugins/disco.py b/certbot/plugins/disco.py index d88b871f6..59410757c 100644 --- a/certbot/plugins/disco.py +++ b/certbot/plugins/disco.py @@ -3,6 +3,7 @@ import collections import itertools import logging import pkg_resources +import six import zope.interface import zope.interface.verify @@ -194,12 +195,12 @@ class PluginsRegistry(collections.Mapping): def init(self, config): """Initialize all plugins in the registry.""" return [plugin_ep.init(config) for plugin_ep - in self._plugins.itervalues()] + in six.itervalues(self._plugins)] def filter(self, pred): """Filter plugins based on predicate.""" return type(self)(dict((name, plugin_ep) for name, plugin_ep - in self._plugins.iteritems() if pred(plugin_ep))) + in six.iteritems(self._plugins) if pred(plugin_ep))) def visible(self): """Filter plugins based on visibility.""" @@ -216,7 +217,7 @@ class PluginsRegistry(collections.Mapping): def prepare(self): """Prepare all plugins in the registry.""" - return [plugin_ep.prepare() for plugin_ep in self._plugins.itervalues()] + return [plugin_ep.prepare() for plugin_ep in six.itervalues(self._plugins)] def available(self): """Filter plugins based on availability.""" @@ -238,7 +239,7 @@ class PluginsRegistry(collections.Mapping): """ # use list instead of set because PluginEntryPoint is not hashable - candidates = [plugin_ep for plugin_ep in self._plugins.itervalues() + candidates = [plugin_ep for plugin_ep in six.itervalues(self._plugins) if plugin_ep.initialized and plugin_ep.init() is plugin] assert len(candidates) <= 1 if candidates: @@ -249,7 +250,7 @@ class PluginsRegistry(collections.Mapping): def __repr__(self): return "{0}({1})".format( self.__class__.__name__, ','.join( - repr(p_ep) for p_ep in self._plugins.itervalues())) + repr(p_ep) for p_ep in six.itervalues(self._plugins))) def __str__(self): if not self._plugins: diff --git a/certbot/plugins/manual.py b/certbot/plugins/manual.py index 9b722aef4..6c7b822ab 100644 --- a/certbot/plugins/manual.py +++ b/certbot/plugins/manual.py @@ -10,6 +10,7 @@ import sys import tempfile import time +import six import zope.component import zope.interface @@ -187,7 +188,7 @@ s.serve_forever()" """ #answer = zope.component.getUtility(interfaces.IDisplay).notification( # message=message, height=25, pause=True) sys.stdout.write(message) - raw_input("Press ENTER to continue") + six.moves.input("Press ENTER to continue") def cleanup(self, achalls): # pylint: disable=missing-docstring,no-self-use,unused-argument diff --git a/certbot/plugins/selection.py b/certbot/plugins/selection.py index 1d7fa323f..b16515d8f 100644 --- a/certbot/plugins/selection.py +++ b/certbot/plugins/selection.py @@ -84,7 +84,7 @@ def pick_plugin(config, default, plugins, question, ifaces): else: return plugin_ep.init() elif len(prepared) == 1: - plugin_ep = prepared.values()[0] + plugin_ep = list(prepared.values())[0] logger.debug("Single candidate plugin: %s", plugin_ep) if plugin_ep.misconfigured: return None @@ -174,7 +174,7 @@ def choose_configurator_plugins(config, plugins, verb): if verb == "install": need_inst = True if config.authenticator: - logger.warn("Specifying an authenticator doesn't make sense in install mode") + logger.warning("Specifying an authenticator doesn't make sense in install mode") # Try to meet the user's request and/or ask them to pick plugins authenticator = installer = None diff --git a/certbot/plugins/util.py b/certbot/plugins/util.py index cdba88a87..b97ca1afd 100644 --- a/certbot/plugins/util.py +++ b/certbot/plugins/util.py @@ -3,15 +3,27 @@ import logging import os import socket -import psutil import zope.component from certbot import interfaces from certbot import util +try: + import psutil + USE_PSUTIL = True +except ImportError: + USE_PSUTIL = False logger = logging.getLogger(__name__) +RENEWER_EXTRA_MSG = ( + " For automated renewal, you may want to use a script that stops" + " and starts your webserver. You can find an example at" + " https://certbot.eff.org/docs/using.html#renewal ." + " Alternatively you can use the webroot plugin to renew without" + " needing to stop and start your webserver.") + + def path_surgery(restart_cmd): """Attempt to perform PATH surgery to find restart_cmd @@ -38,9 +50,11 @@ def path_surgery(restart_cmd): return True else: expanded = " expanded" if any(added) else "" - logger.warn("Failed to find %s in%s PATH: %s", restart_cmd, expanded, path) + logger.warning("Failed to find %s in%s PATH: %s", restart_cmd, + expanded, path) return False + def already_listening(port, renewer=False): """Check if a process is already listening on the port. @@ -53,6 +67,50 @@ def already_listening(port, renewer=False): :param int port: The TCP port in question. :returns: True or False. + """ + + if USE_PSUTIL: + return already_listening_psutil(port, renewer=renewer) + else: + logger.debug("Psutil not found, using simple socket check.") + return already_listening_socket(port, renewer=renewer) + + +def already_listening_socket(port, renewer=False): + """Simple socket based check to find out if port is already in use + + :param int port: The TCP port in question. + :returns: True or False + """ + + try: + testsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) + try: + testsocket.bind(("", port)) + except socket.error: + display = zope.component.getUtility(interfaces.IDisplay) + extra = "" + if renewer: + extra = RENEWER_EXTRA_MSG + display.notification( + "Port {0} is already in use by another process. This will " + "prevent us from binding to that port. Please stop the " + "process that is populating the port in question and try " + "again. {1}".format(port, extra), height=13) + return True + finally: + testsocket.close() + except socket.error: + pass + return False + + +def already_listening_psutil(port, renewer=False): + """Psutil variant of the open port check + + :param int port: The TCP port in question. + :returns: True or False. + """ try: net_connections = psutil.net_connections() @@ -81,12 +139,7 @@ def already_listening(port, renewer=False): display = zope.component.getUtility(interfaces.IDisplay) extra = "" if renewer: - extra = ( - " For automated renewal, you may want to use a script that stops" - " and starts your webserver. You can find an example at" - " https://letsencrypt.org/howitworks/#writing-your-own-renewal-script" - ". Alternatively you can use the webroot plugin to renew without" - " needing to stop and start your webserver.") + extra = RENEWER_EXTRA_MSG display.notification( "The program {0} (process ID {1}) is already listening " "on TCP port {2}. This will prevent us from binding to " diff --git a/certbot/plugins/util_test.py b/certbot/plugins/util_test.py index fa8b364d9..a9466ed63 100644 --- a/certbot/plugins/util_test.py +++ b/certbot/plugins/util_test.py @@ -1,14 +1,22 @@ """Tests for certbot.plugins.util.""" import os import unittest +import sys import mock -import psutil + +try: + # Python 3.5+ + from importlib import reload as refresh # pylint: disable=no-name-in-module +except ImportError: + # Python 2-3.4 + from imp import reload as refresh + class PathSurgeryTest(unittest.TestCase): """Tests for certbot.plugins.path_surgery.""" - @mock.patch("certbot.plugins.util.logger.warn") + @mock.patch("certbot.plugins.util.logger.warning") @mock.patch("certbot.plugins.util.logger.debug") def test_path_surgery(self, mock_debug, mock_warn): from certbot.plugins.util import path_surgery @@ -16,20 +24,64 @@ class PathSurgeryTest(unittest.TestCase): with mock.patch.dict('os.environ', all_path): with mock.patch('certbot.util.exe_exists') as mock_exists: mock_exists.return_value = True - self.assertEquals(path_surgery("eg"), True) - self.assertEquals(mock_debug.call_count, 0) - self.assertEquals(mock_warn.call_count, 0) - self.assertEquals(os.environ["PATH"], all_path["PATH"]) + self.assertEqual(path_surgery("eg"), True) + self.assertEqual(mock_debug.call_count, 0) + self.assertEqual(mock_warn.call_count, 0) + self.assertEqual(os.environ["PATH"], all_path["PATH"]) no_path = {"PATH": "/tmp/"} with mock.patch.dict('os.environ', no_path): path_surgery("thingy") - self.assertEquals(mock_debug.call_count, 1) - self.assertEquals(mock_warn.call_count, 1) + self.assertEqual(mock_debug.call_count, 1) + self.assertEqual(mock_warn.call_count, 1) self.assertTrue("Failed to find" in mock_warn.call_args[0][0]) self.assertTrue("/usr/local/bin" in os.environ["PATH"]) self.assertTrue("/tmp" in os.environ["PATH"]) -class AlreadyListeningTest(unittest.TestCase): + +class AlreadyListeningTestNoPsutil(unittest.TestCase): + """Tests for certbot.plugins.already_listening when + psutil is not available""" + def setUp(self): + import certbot.plugins.util + # Ensure we get importerror + self.psutil = None + if "psutil" in sys.modules: + self.psutil = sys.modules['psutil'] + sys.modules['psutil'] = None + # Reload hackery to ensure getting non-psutil version + # loaded to memory + refresh(certbot.plugins.util) + + def tearDown(self): + # Need to reload the module to ensure + # getting back to normal + import certbot.plugins.util + sys.modules["psutil"] = self.psutil + refresh(certbot.plugins.util) + + @mock.patch("certbot.plugins.util.zope.component.getUtility") + def test_ports_available(self, mock_getutil): + import certbot.plugins.util as plugins_util + # Ensure we don't get error + with mock.patch("socket._socketobject.bind"): + self.assertFalse(plugins_util.already_listening(80)) + self.assertFalse(plugins_util.already_listening(80, True)) + self.assertEqual(mock_getutil.call_count, 0) + + @mock.patch("certbot.plugins.util.zope.component.getUtility") + def test_ports_blocked(self, mock_getutil): + sys.modules["psutil"] = None + import certbot.plugins.util as plugins_util + import socket + with mock.patch("socket._socketobject.bind", side_effect=socket.error): + self.assertTrue(plugins_util.already_listening(80)) + self.assertTrue(plugins_util.already_listening(80, True)) + with mock.patch("socket.socket", side_effect=socket.error): + self.assertFalse(plugins_util.already_listening(80)) + self.assertEqual(mock_getutil.call_count, 2) + + +class AlreadyListeningTestPsutil(unittest.TestCase): """Tests for certbot.plugins.already_listening.""" def _call(self, *args, **kwargs): from certbot.plugins.util import already_listening @@ -42,6 +94,7 @@ class AlreadyListeningTest(unittest.TestCase): # This tests a race condition, or permission problem, or OS # incompatibility in which, for some reason, no process name can be # found to match the identified listening PID. + import psutil from psutil._common import sconn conns = [ sconn(fd=-1, family=2, type=1, laddr=("0.0.0.0", 30), @@ -94,7 +147,7 @@ class AlreadyListeningTest(unittest.TestCase): raddr=(), status="LISTEN", pid=4416)] mock_net.return_value = conns mock_process.name.return_value = "inetd" - result = self._call(17) + result = self._call(17, True) self.assertTrue(result) self.assertEqual(mock_get_utility.call_count, 1) mock_process.assert_called_once_with(4416) @@ -124,6 +177,7 @@ class AlreadyListeningTest(unittest.TestCase): @mock.patch("certbot.plugins.util.psutil.net_connections") def test_access_denied_exception(self, mock_net): + import psutil mock_net.side_effect = psutil.AccessDenied("") self.assertFalse(self._call(12345)) diff --git a/certbot/reverter.py b/certbot/reverter.py index 6dde05077..1c404e29b 100644 --- a/certbot/reverter.py +++ b/certbot/reverter.py @@ -552,7 +552,7 @@ class Reverter(object): others.sort() if others[-1] != timestamp: timetravel = str(float(others[-1]) + 1) - logger.warn("Current timestamp %s does not correspond to newest reverter " + logger.warning("Current timestamp %s does not correspond to newest reverter " "checkpoint; your clock probably jumped. Time travelling to %s", timestamp, timetravel) timestamp = timetravel diff --git a/certbot/tests/client_test.py b/certbot/tests/client_test.py index 9156277a9..e7ae6bbd1 100644 --- a/certbot/tests/client_test.py +++ b/certbot/tests/client_test.py @@ -82,7 +82,7 @@ class RegisterTest(unittest.TestCase): self.config.register_unsafely_without_email = True self.config.dry_run = False self._call() - mock_logger.warn.assert_called_once_with(mock.ANY) + mock_logger.warning.assert_called_once_with(mock.ANY) def test_unsupported_error(self): from acme import messages diff --git a/certbot/tests/reverter_test.py b/certbot/tests/reverter_test.py index 58cc68dce..450cecacf 100644 --- a/certbot/tests/reverter_test.py +++ b/certbot/tests/reverter_test.py @@ -362,8 +362,8 @@ class TestFullCheckpointsReverter(unittest.TestCase): self.assertEqual(mock_logger.warning.call_count, 1) # Test Generic warning - mock_logger.warning.call_count = 0 self._setup_three_checkpoints() + mock_logger.warning.call_count = 0 self.reverter.rollback_checkpoints(4) self.assertEqual(mock_logger.warning.call_count, 1) diff --git a/certbot/tests/testdata/os-release b/certbot/tests/testdata/os-release index cd5297acf..15bc5fb3c 100644 --- a/certbot/tests/testdata/os-release +++ b/certbot/tests/testdata/os-release @@ -1,7 +1,7 @@ NAME="SystemdOS" VERSION="42.42.42 LTS, Unreal" ID=systemdos -ID_LIKE=debian +ID_LIKE="something nonexistent debian" VERSION_ID="42" HOME_URL="http://www.example.com/" SUPPORT_URL="http://help.example.com/" diff --git a/certbot/tests/util_test.py b/certbot/tests/util_test.py index 8e1b330ed..36676443a 100644 --- a/certbot/tests/util_test.py +++ b/certbot/tests/util_test.py @@ -359,6 +359,15 @@ class OsInfoTest(unittest.TestCase): with mock.patch('os.path.isfile', return_value=False): self.assertEqual(get_systemd_os_info(), ("", "")) + def test_systemd_os_release_like(self): + from certbot.util import get_systemd_os_like + + with mock.patch('os.path.isfile', return_value=True): + id_likes = get_systemd_os_like(test_util.vector_path( + "os-release")) + self.assertEqual(len(id_likes), 3) + self.assertTrue("debian" in id_likes) + @mock.patch("certbot.util.subprocess.Popen") def test_non_systemd_os_info(self, popen_mock): from certbot.util import (get_os_info, get_python_os_info, diff --git a/certbot/util.py b/certbot/util.py index 2b40a0f2c..998808be0 100644 --- a/certbot/util.py +++ b/certbot/util.py @@ -268,6 +268,19 @@ def get_systemd_os_info(filepath="/etc/os-release"): return (os_name, os_version) +def get_systemd_os_like(filepath="/etc/os-release"): + """ + Get a list of strings that indicate the distribution likeness to + other distributions. + + :param str filepath: File path of os-release file + :returns: List of distribution acronyms + :rtype: `list` of `str` + """ + + return _get_systemd_os_release_var("ID_LIKE", filepath).split(" ") + + def _get_systemd_os_release_var(varname, filepath="/etc/os-release"): """ Get single value from systemd /etc/os-release @@ -349,7 +362,7 @@ def safe_email(email): if EMAIL_REGEX.match(email) is not None: return not email.startswith(".") and ".." not in email else: - logger.warn("Invalid email address: %s.", email) + logger.warning("Invalid email address: %s.", email) return False @@ -409,6 +422,9 @@ def enforce_domain_sanity(domain): else: raise errors.ConfigurationError(str(error_fmt).format(domain)) + if six.PY3: + domain = domain.decode('ascii') + # Remove trailing dot domain = domain[:-1] if domain.endswith('.') else domain diff --git a/docs/ciphers.rst b/docs/ciphers.rst index 8996dd9ef..c429c185d 100644 --- a/docs/ciphers.rst +++ b/docs/ciphers.rst @@ -151,9 +151,7 @@ Resources for recommendations In the course of considering how to handle this issue, we received recommendations with sources of expert guidance on ciphersuites and other cryptographic parameters. We're grateful to everyone who contributed -suggestions. The recommendations we received are available at - -https://github.com/certbot/certbot/wiki/Ciphersuite-guidance +suggestions. The recommendations we received are available under Feedback_. Certbot users are welcome to review these authorities to better inform their own cryptographic parameter choices. We also @@ -205,3 +203,137 @@ so far is redirecting HTTP requests to HTTPS in web servers, the "redirect" enhancement). The changes here would probably be either a new "ciphersuite" enhancement in each plugin that provides an installer, or a family of enhancements, one per selectable ciphersuite configuration. + +Feedback +======== +We recieve lots of feedback on the type of ciphersuites that Let's Encrypt supports and list some coallated feedback below. This section aims to track suggestions and references that people have offered or identified to improve the ciphersuites that Let's Encrypt enables when configuring TLS on servers. + +Because of the Chatham House Rule applicable to some of the discussions, people are *not* individually credited for their suggestions, but most suggestions here were made or found by other people, and I thank them for their contributions. + +Some people provided rationale information mostly having to do with compatibility of particular user-agents (especially UAs that don't support ECC, or that don't support DH groups > 1024 bits). Some ciphersuite configurations have been chosen to try to increase compatibility with older UAs while allowing newer UAs to negotiate stronger crypto. For example, some configurations forego forward secrecy entirely for connections from old UAs, like by offering ECDHE and RSA key exchange, but no DHE at all. (There are UAs that can fail the negotiation completely if a DHE ciphersuite with prime > 1024 bits is offered.) + +References +---------- + +RFC 7575 +~~~~~~~~ + +IETF has published a BCP document, RFC 7525, "Recommendations for Secure Use of Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS)" + +https://datatracker.ietf.org/doc/rfc7525/ + +BetterCrypto.org +~~~~~~~~~~~~~~~~ + +BetterCrypto.org, a collaboration of mostly European IT security experts, has published a draft paper, "Applied Crypto Hardening" + +https://bettercrypto.org/static/applied-crypto-hardening.pdf + +FF-DHE Internet-Draft +~~~~~~~~~~~~~~~~~~~~~ + +Gillmor's Internet-Draft "Negotiated Discrete Log Diffie-Hellman Ephemeral Parameters for TLS" is being developed at the IETF TLS WG. It advocates using *standardized* DH groups in all cases, not individually-chosen ones (mostly because of the Triple Handshake attack which can involve maliciously choosing invalid DH groups). The draft provides a list of recommended groups, with primes beginning at 2048 bits and going up from there. It also has a new protocol mechanism for agreeing to use these groups, with the possibility of backwards compatibility (and use of weaker DH groups) for older clients and servers that don't know about this mechanism. + +https://tools.ietf.org/html/draft-ietf-tls-negotiated-ff-dhe-10 + +Mozilla +~~~~~~~ + +Mozilla's general server configuration guidance is available at https://wiki.mozilla.org/Security/Server_Side_TLS + +Mozilla has also produced a configuration generator: https://mozilla.github.io/server-side-tls/ssl-config-generator/ + +Dutch National Cyber Security Centre +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Dutch National Cyber Security Centre has published guidance on "ICT-beveiligingsrichtlijnen voor Transport Layer Security (TLS)" ("IT Security Guidelines for Transport Layer Security (TLS)"). These are available only in Dutch at + +https://www.ncsc.nl/dienstverlening/expertise-advies/kennisdeling/whitepapers/ict-beveiligingsrichtlijnen-voor-transport-layer-security-tls.html + +I have access to an English-language summary of the recommendations. + +Keylength.com +~~~~~~~~~~~~~ + +Damien Giry collects recommendations by academic researchers and standards organizations about keylengths for particular cryptoperiods, years, or security levels. The keylength recommendations of the various sources are summarized in a chart. This site has been updated over time and includes expert guidance from eight sources published between 2000 and 2015. + +http://www.keylength.com/ + +NIST +~~~~ +NISA published its "NIST Special Publication 800-52 Revision 1: Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations" + +http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-52r1.pdf + +and its "NIST Special Publication 800-57: Recommendation for Key Management – Part 1: General (Revision 3)" + +http://csrc.nist.gov/publications/nistpubs/800-57/sp800-57_part1_rev3_general.pdf + +ENISA +~~~~~ + +ENISA published its "Algorithms, Key Sizes and Parameters Report - 2013" + +https://www.enisa.europa.eu/activities/identity-and-trust/library/deliverables/algorithms-key-sizes-and-parameters-report + +WeakDH/Logjam +------------- + +The WeakDH/Logjam research has thrown into question the safety of some existing practice using DH ciphersuites, especially the use of standardized groups with a prime ≤ 1024 bits. The authors provided detailed guidance, including ciphersuite lists, at + +https://weakdh.org/sysadmin.html + +These lists may have been derived from Mozilla's recommendations. +One of the authors clarified his view of the priorities for various changes as a result of the research at + +https://www.ietf.org/mail-archive/web/tls/current/msg16496.html + +In particular, he supports ECDHE and also supports the use of the standardized groups in the FF-DHE Internet-Draft mentioned above (which isn't clear from the group's original recommendations). + +Particular sites' opinions or configurations +-------------------------------------------- + +Amazon ELB +~~~~~~~~~~ + +Amazon ELB explains its current ciphersuite choices at + +https://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-security-policy-table.html + +U.S. Government 18F +~~~~~~~~~~~~~~~~~~~ + +The 18F site (https://18f.gsa.gov/) is using + +:: + + ssl_ciphers 'kEECDH+ECDSA+AES128 kEECDH+ECDSA+AES256 kEECDH+AES128 kEECDH+AES256 kEDH+AES128 kEDH+AES256 DES-CBC3-SHA +SHA !aNULL !eNULL !LOW !MD5 !EXP !DSS !PSK !SRP !kECDH !CAMELLIA !RC4 !SEED'; + +Duraconf +~~~~~~~~ + +The Duraconf project collects particular configuration files, with an apparent focus on avoiding the use of obsolete symmetric ciphers and hash functions, and favoring forward secrecy while not requiring it. + +https://github.com/ioerror/duraconf + +Site scanning or rating tools +----------------------------- + +Qualys SSL Labs +~~~~~~~~~~~~~~~ + +Qualys offers the best-known TLS security scanner, maintained by Ivan Ristić. + +https://www.ssllabs.com/ + +Dutch NCSC +~~~~~~~~~~ + +The Dutch NCSC, mentioned above, has also made available its own site security scanner which indicates how well sites comply with the recommendations. + +https://en.internet.nl/ + +Java compatibility issue +------------------------ + +A lot of backward-compatibility concerns have to do with Java hard-coding DHE primes to a 1024-bit limit, accepting DHE ciphersuites in negotiation, and then aborting the connection entirely if a prime > 1024 bits is presented. The simple summary is that servers offering a Java-compatible DHE ciphersuite in preference to other Java-compatible ciphersuites, and then presenting a DH group with a prime > 1024 bits, will be completely incompatible with clients running some versions of Java. (This may also be the case with very old MSIE versions...?) There are various strategies for dealing with this, and maybe we can document the options here. diff --git a/docs/contributing.rst b/docs/contributing.rst index bdda0e2e6..a5b9b5688 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -6,9 +6,9 @@ Developer Guide :local: -.. _hacking: +.. _getting_started: -Hacking +Getting Started ======= Running a local copy of the client @@ -323,11 +323,7 @@ Steps: should run the integration tests, see `integration`_. See `Known Issues`_ for some common failures that have nothing to do with your code. 7. Submit the PR. -8. Did your tests pass on Travis? If they didn't, it might not be your fault! - See `Known Issues`_. If it's not a known issue, fix any errors. - -.. _Known Issues: - https://github.com/certbot/certbot/wiki/Known-issues +8. Did your tests pass on Travis? If they didn't, fix any errors. Updating the documentation ========================== diff --git a/docs/index.rst b/docs/index.rst index b541e376e..746080864 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -5,9 +5,11 @@ Welcome to the Certbot documentation! :maxdepth: 2 intro + install using contributing packaging + resources .. toctree:: :maxdepth: 1 diff --git a/docs/install.rst b/docs/install.rst new file mode 100644 index 000000000..e79a3b596 --- /dev/null +++ b/docs/install.rst @@ -0,0 +1,33 @@ +===================== +Quick Installation +===================== + +If ``certbot`` (or ``letsencrypt``) is packaged for your Unix OS (visit +certbot.eff.org_ to find out), you can install it +from there, and run it by typing ``certbot`` (or ``letsencrypt``). Because +not all operating systems have packages yet, we provide a temporary solution +via the ``certbot-auto`` wrapper script, which obtains some dependencies from +your OS and puts others in a python virtual environment:: + + user@webserver:~$ wget https://dl.eff.org/certbot-auto + user@webserver:~$ chmod a+x ./certbot-auto + user@webserver:~$ ./certbot-auto --help + +.. hint:: The certbot-auto download is protected by HTTPS, which is pretty good, but if you'd like to + double check the integrity of the ``certbot-auto`` script, you can use these steps for verification before running it:: + + user@server:~$ wget -N https://dl.eff.org/certbot-auto.asc + user@server:~$ gpg2 --recv-key A2CFB51FA275A7286234E7B24D17C995CD9775F2 + user@server:~$ gpg2 --trusted-key 4D17C995CD9775F2 --verify certbot-auto.asc certbot-auto + +And for full command line help, you can type:: + + ./certbot-auto --help all + +``certbot-auto`` updates to the latest client release automatically. And +since ``certbot-auto`` is a wrapper to ``certbot``, it accepts exactly +the same command line flags and arguments. More details about this script and +other installation methods can be found `in the User Guide +`_. + +.. _certbot.eff.org: https://certbot.eff.org/ diff --git a/docs/intro.rst b/docs/intro.rst index 2fffbec68..90c3761ec 100644 --- a/docs/intro.rst +++ b/docs/intro.rst @@ -1,6 +1,7 @@ ===================== -README / Introduction +Introduction ===================== .. include:: ../README.rst -.. include:: ../CHANGES.rst + :start-after: tag:intro-begin + :end-before: tag:intro-end diff --git a/docs/packaging.rst b/docs/packaging.rst index bd366dbaa..1a1b83f50 100644 --- a/docs/packaging.rst +++ b/docs/packaging.rst @@ -2,5 +2,83 @@ Packaging Guide =============== -Documentation can be found at -https://github.com/certbot/certbot/wiki/Packaging. +Releases +======== + +We release packages and upload them to PyPI (wheels and source tarballs). + +- https://pypi.python.org/pypi/acme +- https://pypi.python.org/pypi/certbot +- https://pypi.python.org/pypi/certbot-apache +- https://pypi.python.org/pypi/certbot-nginx + +The following scripts are used in the process: + +- https://github.com/letsencrypt/letsencrypt/blob/master/tools/release.sh + +We currently version with the following scheme: + +- ``0.1.0`` +- ``0.2.0dev`` for developement in ``master`` +- ``0.2.0`` (only temporarily in ``master``) +- ... + +Notes for package maintainers +============================= + +0. Please use our releases, not ``master``! + +1. Do not package ``certbot-compatibility-test`` or ``letshelp-certbot`` - it's only used internally. + +2. If you'd like to include automated renewal in your package ``certbot renew -q`` should be added to crontab or systemd timer. + +3. ``jws`` is an internal script for ``acme`` module and it doesn't have to be packaged - it's mostly for debugging: you can use it as ``echo foo | jws sign | jws verify``. + +4. Do get in touch with us. We are happy to make any changes that will make packaging easier. If you need to apply some patches don't do it downstream - make a PR here. + +Already ongoing efforts +======================= + + +Arch +---- + +From our official releases: +- https://www.archlinux.org/packages/community/any/python2-acme +- https://www.archlinux.org/packages/community/any/certbot +- https://www.archlinux.org/packages/community/any/certbot-apache +- https://www.archlinux.org/packages/community/any/certbot-nginx +- https://www.archlinux.org/packages/community/any/letshelp-certbot + +From ``master``: https://aur.archlinux.org/packages/certbot-git + +Debian (and its derivatives, including Ubuntu) +------ + +https://packages.debian.org/sid/certbot +https://packages.debian.org/sid/python-certbot +https://packages.debian.org/sid/python-certbot-apache + +Fedora +------ + +In Fedora 23+. + +- https://admin.fedoraproject.org/pkgdb/package/letsencrypt/ +- https://admin.fedoraproject.org/pkgdb/package/certbot/ +- https://admin.fedoraproject.org/pkgdb/package/python-acme/ + +FreeBSD +------- + +https://svnweb.freebsd.org/ports/head/security/py-certbot/ + +GNU Guix +-------- + +- https://www.gnu.org/software/guix/package-list.html#certbot + +OpenBSD +------- + +- http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/security/letsencrypt/client/ diff --git a/docs/resources.rst b/docs/resources.rst new file mode 100644 index 000000000..a284f4a3d --- /dev/null +++ b/docs/resources.rst @@ -0,0 +1,54 @@ +===================== +Resources +===================== + +Documentation: https://certbot.eff.org/docs + +Software project: https://github.com/certbot/certbot + +Notes for developers: https://certbot.eff.org/docs/contributing.html + +Main Website: https://letsencrypt.org/ + +Let's Encrypt FAQ: https://community.letsencrypt.org/t/frequently-asked-questions-faq/26#topic-title + +IRC Channel: #letsencrypt on `Freenode`_ or #certbot on `OFTC`_ + +Community: https://community.letsencrypt.org + +ACME spec: http://ietf-wg-acme.github.io/acme/ + +ACME working area in github: https://github.com/ietf-wg-acme/acme + + +Mailing list: `client-dev`_ (to subscribe without a Google account, send an +email to client-dev+subscribe@letsencrypt.org) + +|build-status| |coverage| |docs| |container| + + + +.. |build-status| image:: https://travis-ci.org/certbot/certbot.svg?branch=master + :target: https://travis-ci.org/certbot/certbot + :alt: Travis CI status + +.. |coverage| image:: https://coveralls.io/repos/certbot/certbot/badge.svg?branch=master + :target: https://coveralls.io/r/certbot/certbot + :alt: Coverage status + +.. |docs| image:: https://readthedocs.org/projects/letsencrypt/badge/ + :target: https://readthedocs.org/projects/letsencrypt/ + :alt: Documentation status + +.. |container| image:: https://quay.io/repository/letsencrypt/letsencrypt/status + :target: https://quay.io/repository/letsencrypt/letsencrypt + :alt: Docker Repository on Quay.io + +.. _`installation instructions`: + https://letsencrypt.readthedocs.org/en/latest/using.html + +.. _watch demo video: https://www.youtube.com/watch?v=Gas_sSB-5SU + +.. _Freenode: https://webchat.freenode.net?channels=%23letsencrypt +.. _OFTC: https://webchat.oftc.net?channels=%23certbot +.. _client-dev: https://groups.google.com/a/letsencrypt.org/forum/#!forum/client-dev diff --git a/docs/using.rst b/docs/using.rst index ccd1d8f97..15658b2c5 100644 --- a/docs/using.rst +++ b/docs/using.rst @@ -5,6 +5,27 @@ User Guide .. contents:: Table of Contents :local: + +System Requirements +=================== + +The Let's Encrypt Client presently only runs on Unix-ish OSes that include +Python 2.6 or 2.7; Python 3.x support will hopefully be added in the future. The +client requires root access in order to write to ``/etc/letsencrypt``, +``/var/log/letsencrypt``, ``/var/lib/letsencrypt``; to bind to ports 80 and 443 +(if you use the ``standalone`` plugin) and to read and modify webserver +configurations (if you use the ``apache`` or ``nginx`` plugins). If none of +these apply to you, it is theoretically possible to run without root privileges, +but for most users who want to avoid running an ACME client as root, either +`letsencrypt-nosudo `_ or +`simp_le `_ are more appropriate choices. + +The Apache plugin currently requires OS with augeas version 1.0; currently `it +supports +`_ +modern OSes based on Debian, Fedora, SUSE, Gentoo and Darwin. + + Getting Certbot =============== @@ -13,7 +34,7 @@ visit certbot.eff.org_. This is the easiest way to learn how to get Certbot up and running on your system. If you're offline, you can find some general -instructions `in the README / Introduction `__ +instructions under `Quick Installation `__. __ installation_ .. _certbot.eff.org: https://certbot.eff.org @@ -406,7 +427,7 @@ good reason to do so. You should definitely read the :ref:`where-certs` section, in order to know how to manage the certs -manually. https://github.com/certbot/certbot/wiki/Ciphersuite-guidance +manually. `Our ciphersuites page `__ provides some information about recommended ciphersuites. If none of these make much sense to you, you should definitely use the certbot-auto_ method, which enables you to use installer plugins @@ -564,3 +585,8 @@ Beyond the methods discussed here, other methods may be possible, such as installing Certbot directly with pip from PyPI or downloading a ZIP archive from GitHub may be technically possible but are not presently recommended or supported. + +.. include:: ../README.rst + :start-after: tag:features-begin + :end-before: tag:features-end +.. include:: ../CHANGES.rst diff --git a/letsencrypt-auto-source/letsencrypt-auto b/letsencrypt-auto-source/letsencrypt-auto index cf2de34ed..27b11c656 100755 --- a/letsencrypt-auto-source/letsencrypt-auto +++ b/letsencrypt-auto-source/letsencrypt-auto @@ -126,7 +126,7 @@ ExperimentalBootstrap() { $2 fi else - echo "WARNING: $1 support is very experimental at present..." + echo "FATAL: $1 support is very experimental at present..." echo "if you would like to work on improving it, please ensure you have backups" echo "and then run this script again with the --debug flag!" exit 1 diff --git a/letsencrypt-auto-source/letsencrypt-auto.template b/letsencrypt-auto-source/letsencrypt-auto.template index d049e9df8..991d9dd76 100755 --- a/letsencrypt-auto-source/letsencrypt-auto.template +++ b/letsencrypt-auto-source/letsencrypt-auto.template @@ -126,7 +126,7 @@ ExperimentalBootstrap() { $2 fi else - echo "WARNING: $1 support is very experimental at present..." + echo "FATAL: $1 support is very experimental at present..." echo "if you would like to work on improving it, please ensure you have backups" echo "and then run this script again with the --debug flag!" exit 1 diff --git a/setup.py b/setup.py index 21cda901a..6d0909ea8 100644 --- a/setup.py +++ b/setup.py @@ -40,10 +40,8 @@ install_requires = [ 'configobj', 'cryptography>=0.7', # load_pem_x509_certificate 'parsedatetime>=1.3', # Calendar.parseDT - 'psutil>=2.2.1', # 2.1.0 for net_connections and 2.2.1 resolves #1080 'PyOpenSSL', 'pyrfc3339', - 'python2-pythondialog>=3.2.2rc1', # Debian squeeze support, cf. #280 'pytz', # For pkg_resources. >=1.0 so pip resolves it to a version cryptography # will tolerate; see #2599: @@ -53,6 +51,12 @@ install_requires = [ 'zope.interface', ] +# Debian squeeze support, cf. #280 +if sys.version_info[0] == 2: + install_requires.append('python2-pythondialog>=3.2.2rc1') +else: + install_requires.append('pythondialog>=3.2.2rc1') + # env markers in extras_require cause problems with older pip: #517 # Keep in sync with conditional_requirements.py. if sys.version_info < (2, 7): @@ -70,6 +74,7 @@ dev_extras = [ 'coverage', 'nose', 'pep8', + 'psutil>=2.2.1', # for tests, optional 'pylint==1.4.2', # upstream #248 'tox', 'twine', diff --git a/tests/boulder-fetch.sh b/tests/boulder-fetch.sh index 469c5cd80..0c0a8009b 100755 --- a/tests/boulder-fetch.sh +++ b/tests/boulder-fetch.sh @@ -5,6 +5,6 @@ set -xe # Check out special branch until latest docker changes land in Boulder master. git clone -b docker-integration https://github.com/letsencrypt/boulder $BOULDERPATH cd $BOULDERPATH -sed -i 's/FAKE_DNS: .*/FAKE_DNS: 172.17.42.1/' docker-compose.yml +FAKE_DNS=$(ifconfig docker0 | grep "inet addr:" | cut -d: -f2 | awk '{ print $1}') +sed -i "s/FAKE_DNS: .*/FAKE_DNS: $FAKE_DNS/" docker-compose.yml docker-compose up -d - diff --git a/tests/letstest/scripts/boulder_install.sh b/tests/letstest/scripts/boulder_install.sh index 7e298783f..f997268bd 100755 --- a/tests/letstest/scripts/boulder_install.sh +++ b/tests/letstest/scripts/boulder_install.sh @@ -5,5 +5,6 @@ # Check out special branch until latest docker changes land in Boulder master. git clone -b docker-integration https://github.com/letsencrypt/boulder $BOULDERPATH cd $BOULDERPATH -sed -i 's/FAKE_DNS: .*/FAKE_DNS: 172.17.42.1/' docker-compose.yml +FAKE_DNS=$(ifconfig docker0 | grep "inet addr:" | cut -d: -f2 | awk '{ print $1}') +sed -i "s/FAKE_DNS: .*/FAKE_DNS: $FAKE_DNS/" docker-compose.yml docker-compose up -d diff --git a/tox.ini b/tox.ini index 689bbf513..f19c41dc7 100644 --- a/tox.ini +++ b/tox.ini @@ -13,7 +13,7 @@ envlist = py{26,33,34,35},cover,lint # packages installed separately to ensure that downstream deps problems # are detected, c.f. #1002 commands = - pip install -e acme[dev] + pip install -e acme[dns,dev] nosetests -v acme pip install -e .[dev] nosetests -v certbot @@ -35,26 +35,27 @@ deps = py{26,27}-oldest: psutil==2.1.0 py{26,27}-oldest: PyOpenSSL==0.13 py{26,27}-oldest: python2-pythondialog==3.2.2rc1 + py{26,27}-oldest: dnspython>=1.12 [testenv:py33] commands = - pip install -e acme[dev] + pip install -e acme[dns,dev] nosetests -v acme [testenv:py34] commands = - pip install -e acme[dev] + pip install -e acme[dns,dev] nosetests -v acme [testenv:py35] commands = - pip install -e acme[dev] + pip install -e acme[dns,dev] nosetests -v acme [testenv:cover] basepython = python2.7 commands = - pip install -e acme[dev] -e .[dev] -e certbot-apache -e certbot-nginx -e letshelp-certbot + pip install -e acme[dns,dev] -e .[dev] -e certbot-apache -e certbot-nginx -e letshelp-certbot ./tox.cover.sh [testenv:lint] @@ -64,7 +65,7 @@ basepython = python2.7 # duplicate code checking; if one of the commands fails, others will # continue, but tox return code will reflect previous error commands = - pip install -q -e acme[dev] -e .[dev] -e certbot-apache -e certbot-nginx -e certbot-compatibility-test -e letshelp-certbot + pip install -q -e acme[dns,dev] -e .[dev] -e certbot-apache -e certbot-nginx -e certbot-compatibility-test -e letshelp-certbot ./pep8.travis.sh pylint --reports=n --rcfile=.pylintrc certbot pylint --reports=n --rcfile=acme/.pylintrc acme/acme @@ -79,6 +80,10 @@ commands = pip install -e acme -e .[dev] -e certbot-apache -e certbot-nginx -e certbot-compatibility-test -e letshelp-certbot {toxinidir}/certbot-apache/certbot_apache/tests/apache-conf-files/apache-conf-test --debian-modules +[testenv:nginxroundtrip] +commands = + pip install -e acme[dev] -e .[dev] -e certbot-nginx + python certbot-compatibility-test/nginx/roundtrip.py certbot-compatibility-test/nginx/nginx-roundtrip-testdata [testenv:le_auto] # At the moment, this tests under Python 2.7 only, as only that version is @@ -89,3 +94,21 @@ commands = whitelist_externals = docker passenv = DOCKER_* + +[testenv:apache_compat] +commands = + docker build -t certbot-compatibility-test -f certbot-compatibility-test/Dockerfile . + docker build -t apache-compat -f certbot-compatibility-test/Dockerfile-apache . + docker run --rm -it apache-compat -c apache.tar.gz -vvvv +whitelist_externals = + docker +passenv = DOCKER_* + +[testenv:nginx_compat] +commands = + docker build -t certbot-compatibility-test -f certbot-compatibility-test/Dockerfile . + docker build -t nginx-compat -f certbot-compatibility-test/Dockerfile-nginx . + docker run --rm -it nginx-compat -c nginx.tar.gz -vvvv +whitelist_externals = + docker +passenv = DOCKER_*