Implement our fancy new --help output (#3883)

* Start reorganising -h output

* Fix the --debug flag

  - Currently exceptions are often caught and burried in log files, even
    if this flag is provided!

* Explain the insanity

* Parallalelise nosetests from tox (#3836)

* Parallalelise nosetests from tox

* Parallelise even more things, break even more things

* Now unbreak all the tests that aren't ready for ||ism

* Try to pass tests!

 - Remove non-working hack in reporter_test
 - also be selective about ||ism in the cover environment

* Try again

* certbot-apache tests also work, given enough time

* Nginx may need more time in Travis's cloud

* Unbreak reporter_test under ||ism

* More timeout

* Working again?

* This goes way faster

* Another big win

* Split a couple more large test suites

* A last improvement

* More ||ism!

* ||ise lint too

* Allow nosetests to figure out how many cores to use

* simplify merge

* Mark the new CLI tests as ||izable

* Simplify reporter_test changes

* Rationalise ||ism flags

* Re-up coverage

* Clean up reporter tests

* Stop modifying testdata during tests

* remove unused os

* Improve the "certbot certificates" output (#3846)

* Begin making "certbot certificates" future safe

* Handle the case where a renewal conf file has no "server" entry

* Improvements, tweaks

* Capitalise on things

* Print the command summary for -h and -h all, but not otherwise

Also, update nginx not installed CLI hint

* Add a "certificates" help section

* Clean up usage string construction

* Greatly improve "certbot -h TOPIC"

  - subcommands now get their own usage headings if they want them
  - added "certbot -h commands"

* A few more cli formatting tests

* Auto-populate the verb subgroups from the docs

* Show the new help output

* Lint, tweak

* More lint, and cleanup

* Infinite lint

* Add rename to command summary; sort "-h commands" output

* Use fancy string formatting

* More space

* Implement --help manage

Also, implement a general mechanism for documenting subcommands within
topics

* Remove one comma

* Only create weird parser structures if -h is provided :)

* Update sample cli out

* Lint

* Revert cli-help.txt to previous release version

* Grammar & style

certbot/cli.py

@@ -51,51 +51,55 @@ cli_command = LEAUTO if fragment in sys.argv[0] else "certbot"
 # to replace as much of it as we can...
 
 # This is the stub to include in help generated by argparse
-
 SHORT_USAGE = """
-  {0} [SUBCOMMAND] [options] [-d domain] [-d domain] ...
+  {0} [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...
 
 Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
 it will attempt to use a webserver both for obtaining and installing the
-cert. Major SUBCOMMANDS are:
+cert. """.format(cli_command)
 
-  (default) run        Obtain & install a cert in your current webserver
+# This section is used for --help and --help all ; it needs information
-  certonly             Obtain cert, but do not install it (aka "auth")
+# about installed plugins to be fully formatted
-  install              Install a previously obtained cert in a server
+COMMAND_OVERVIEW = """The most common SUBCOMMANDS and flags are:
-  renew                Renew previously obtained certs that are near expiry
-  revoke               Revoke a previously obtained certificate
-  register             Perform tasks related to registering with the CA
-  rollback             Rollback server configuration changes made during install
-  config_changes       Show changes made to server config during installation
-  update_symlinks      Update cert symlinks based on renewal config file
-  rename               Update a certificate's name
-  plugins              Display information about installed plugins
-  certificates         Display information about certs configured with Certbot
 
-""".format(cli_command)
+obtain, install, and renew certificates:
-
+    (default) run   Obtain & install a cert in your current webserver
-# This is the short help for certbot --help, where we disable argparse
+    certonly        Obtain or renew a cert, but do not install it
-# altogether
+    renew           Renew all previously obtained certs that are near expiry
-USAGE = SHORT_USAGE + """Choice of server plugins for obtaining and installing cert:
+   -d DOMAINS       Comma-separated list of domains to obtain a cert for
 
   %s
   --standalone      Run a standalone webserver for authentication
   %s
   --webroot         Place files in a server's webroot folder for authentication
-  --script          User provided shell scripts for authentication
+  --manual          Obtain certs interactively, or using shell script hoooks
 
-OR use different plugins to obtain (authenticate) the cert and then install it:
+   -n               Run non-interactively
+  --test-cert       Obtain a test cert from a staging server
+  --dry-run         Test "renew" or "certonly" without saving any certs to disk
 
-  --authenticator standalone --installer apache
+manage certificates:
+    certificates    Display information about certs you have from Certbot
+    revoke          Revoke a certificate (supply --cert-path)
+    rename          Rename a certificate
 
+manage your account with Let's Encrypt:
+    register        Create a Let's Encrypt ACME account
+  --agree-tos       Agree to the ACME server's Subscriber Agreement
+   -m EMAIL         Email address for important account notifications
+"""
+
+# This is the short help for certbot --help, where we disable argparse
+# altogether
+HELP_USAGE = """
 More detailed help:
 
-  -h, --help [topic]    print this message, or detailed help on a topic;
+  -h, --help [TOPIC]    print this message, or detailed help on a topic;
-                        the available topics are:
+                        the available TOPICS are:
 
-   all, automation, paths, security, testing, or any of the subcommands or
+   all, automation, commands, paths, security, testing, or any of the
-   plugins (certonly, renew, install, register, nginx, apache, standalone,
+   subcommands or plugins (certonly, renew, install, register, nginx,
-   webroot, script, etc.)
+   apache, standalone, webroot, script, etc.)
 """
 
 
@@ -141,19 +145,6 @@ def report_config_interaction(modified, modifiers):
         VAR_MODIFIERS.setdefault(var, set()).update(modifiers)
 
 
-def usage_strings(plugins):
-    """Make usage strings late so that plugins can be initialised late"""
-    if "nginx" in plugins:
-        nginx_doc = "--nginx           Use the Nginx plugin for authentication & installation"
-    else:
-        nginx_doc = "(nginx support is experimental, buggy, and not installed by default)"
-    if "apache" in plugins:
-        apache_doc = "--apache          Use the Apache plugin for authentication & installation"
-    else:
-        apache_doc = "(the apache plugin is not installed)"
-    return USAGE % (apache_doc, nginx_doc), SHORT_USAGE
-
-
 def possible_deprecation_warning(config):
     "A deprecation warning for users with the old, not-self-upgrading letsencrypt-auto."
     if cli_command != LEAUTO:
@@ -309,6 +300,82 @@ class HelpfulArgumentGroup(object):
         """Add a new command line argument to the argument group."""
         self._parser.add(self._topic, *args, **kwargs)
 
+# The attributes here are:
+# short: a string that will be displayed by "certbot -h commands"
+# opts:  a string that heads the section of flags with which this command is documented,
+#        both for "cerbot -h SUBCOMMAND" and "certbot -h all"
+# usage: an optional string that overrides the header of "certbot -h SUBCOMMAND"
+VERB_HELP = [
+    ("run (default)", {
+        "short": "Obtain/renew a certificate, and install it",
+        "opts": "Options for obtaining & installing certs",
+        "usage": SHORT_USAGE.replace("[SUBCOMMAND]", ""),
+        "realname": "run"
+    }),
+    ("certonly", {
+        "short": "Obtain or renew a certificate, but do not install it",
+        "opts": "Options for modifying how a cert is obtained",
+        "usage": ("\n\n  certbot certonly [options] [-d DOMAIN] [-d DOMAIN] ...\n\n"
+                  "This command obtains a TLS/SSL certificate without installing it anywhere.")
+    }),
+    ("renew", {
+        "short": "Renew all certificates (or one specifed with --cert-name)",
+        "opts": ("The 'renew' subcommand will attempt to renew all"
+                 " certificates (or more precisely, certificate lineages) you have"
+                 " previously obtained if they are close to expiry, and print a"
+                 " summary of the results. By default, 'renew' will reuse the options"
+                 " used to create obtain or most recently successfully renew each"
+                 " certificate lineage. You can try it with `--dry-run` first. For"
+                 " more fine-grained control, you can renew individual lineages with"
+                 " the `certonly` subcommand. Hooks are available to run commands"
+                 " before and after renewal; see"
+                 " https://certbot.eff.org/docs/using.html#renewal for more"
+                 " information on these."),
+        "usage": "\n\n  certbot renew [--cert-name NAME] [options]\n\n"
+    }),
+    ("certificates", {
+        "short": "List all certificates managed by Certbot",
+        "opts": "List all certificates managed by Certbot"
+    }),
+    ("revoke", {
+        "short": "Revoke a certificate specified with --cert-path",
+        "opts": "Options for revocation of certs"
+    }),
+    ("rename", {
+        "short": "Change a certificate's name (for management purposes)",
+        "opts": "Options changing certificate names"
+    }),
+    ("register", {
+        "short": "Register for account with Let's Encrypt / other ACME server",
+        "opts": "Options for account registration & modification"
+    }),
+    ("install", {
+        "short": "Install an arbitrary cert in a server",
+        "opts": "Options for modifying how a cert is deployed"
+    }),
+    ("config_changes", {
+        "short": "Show changes that Certbot has made to server configurations",
+        "opts": "Options for controlling which changes are displayed"
+    }),
+    ("rollback", {
+        "short": "Roll back server conf changes made during cert installation",
+        "opts": "Options for rolling back server configuration changes"
+    }),
+    ("plugins", {
+        "short": "List plugins that are installed and available on your system",
+        "opts": 'Options for for the "plugins" subcommand'
+    }),
+    ("update_symlinks", {
+        "short": "Recreate symlinks in your /live/ directory",
+        "opts": ("Recreates cert and key symlinks in {0}, if you changed them by hand "
+                 "or edited a renewal configuration file".format(
+                  os.path.join(flag_default("config_dir"), "live")))
+    }),
+
+]
+# VERB_HELP is a list in order to preserve order, but a dict is sometimes useful
+VERB_HELP_MAP = dict(VERB_HELP)
+
 
 class HelpfulArgumentParser(object):
     """Argparse Wrapper.
@@ -319,6 +386,7 @@ class HelpfulArgumentParser(object):
 
     """
 
+
     def __init__(self, args, plugins, detect_defaults=False):
         from certbot import main
         self.VERBS = {"auth": main.obtain_cert, "certonly": main.obtain_cert,
@@ -331,22 +399,12 @@ class HelpfulArgumentParser(object):
 
         # List of topics for which additional help can be provided
         HELP_TOPICS = ["all", "security", "paths", "automation", "testing"] + list(self.VERBS)
+        HELP_TOPICS += self.COMMANDS_TOPICS + ["manage"]
 
         plugin_names = list(plugins)
         self.help_topics = HELP_TOPICS + plugin_names + [None]
-        usage, short_usage = usage_strings(plugins)
-        self.parser = configargparse.ArgParser(
-            prog="certbot",
-            usage=short_usage,
-            formatter_class=argparse.ArgumentDefaultsHelpFormatter,
-            args_for_setting_config_path=["-c", "--config"],
-            default_config_files=flag_default("config_files"))
-
-        # This is the only way to turn off overly verbose config flag documentation
-        self.parser._add_config_file_help = False  # pylint: disable=protected-access
 
         self.detect_defaults = detect_defaults
-
         self.args = args
         self.determine_verb()
         help1 = self.prescan_for_flag("-h", self.help_topics)
@@ -355,13 +413,72 @@ class HelpfulArgumentParser(object):
             self.help_arg = help1 or help2
         else:
             self.help_arg = help1 if isinstance(help1, str) else help2
-        if self.help_arg is True:
+
-            # just --help with no topic; avoid argparse altogether
+        short_usage = self._usage_string(plugins, self.help_arg)
-            print(usage)
+
-            sys.exit(0)
         self.visible_topics = self.determine_help_topics(self.help_arg)
         self.groups = {}       # elements are added by .add_group()
-        self.defaults = {}  # elements are added by .parse_args()
+        self.defaults = {}     # elements are added by .parse_args()
+
+        self.parser = configargparse.ArgParser(
+            prog="certbot",
+            usage=short_usage,
+            formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+            args_for_setting_config_path=["-c", "--config"],
+            default_config_files=flag_default("config_files"),
+            config_arg_help_message="path to config file (default: {0})".format(
+                " and ".join(flag_default("config_files"))))
+
+        # This is the only way to turn off overly verbose config flag documentation
+        self.parser._add_config_file_help = False  # pylint: disable=protected-access
+
+    # Help that are synonyms for --help subcommands
+    COMMANDS_TOPICS = ["command", "commands", "subcommand", "subcommands", "verbs"]
+    def _list_subcommands(self):
+        longest = max(len(v) for v in VERB_HELP_MAP.keys())
+
+        text = "The full list of available SUBCOMMANDS is:\n\n"
+        for verb, props in sorted(VERB_HELP):
+            doc = props.get("short", "")
+            text += '{0:<{length}}     {1}\n'.format(verb, doc, length=longest)
+
+        text += "\nYou can get more help on a specific subcommand with --help SUBCOMMAND\n"
+        return text
+
+    def _usage_string(self, plugins, help_arg):
+        """Make usage strings late so that plugins can be initialised late
+
+        :param plugins: all discovered plugins
+        :param help_arg: False for none; True for --help; "TOPIC" for --help TOPIC
+        :rtype: str
+        :returns: a short usage string for the top of --help TOPIC)
+        """
+        if "nginx" in plugins:
+            nginx_doc = "--nginx           Use the Nginx plugin for authentication & installation"
+        else:
+            nginx_doc = "(the certbot nginx plugin is not installed)"
+        if "apache" in plugins:
+            apache_doc = "--apache          Use the Apache plugin for authentication & installation"
+        else:
+            apache_doc = "(the cerbot apache plugin is not installed)"
+
+        usage = SHORT_USAGE
+        if help_arg == True:
+            print(usage + COMMAND_OVERVIEW % (apache_doc, nginx_doc) + HELP_USAGE)
+            sys.exit(0)
+        elif help_arg in self.COMMANDS_TOPICS:
+            print(usage + self._list_subcommands())
+            sys.exit(0)
+        elif help_arg == "all":
+            # if we're doing --help all, the OVERVIEW is part of the SHORT_USAGE at
+            # the top; if we're doing --help someothertopic, it's OT so it's not
+            usage += COMMAND_OVERVIEW % (apache_doc, nginx_doc)
+        else:
+            custom = VERB_HELP_MAP.get(help_arg, {}).get("usage", None)
+            usage = custom if custom else usage
+
+        return usage
+
 
     def parse_args(self):
         """Parses command line arguments and returns the result.
@@ -566,7 +683,7 @@ class HelpfulArgumentParser(object):
         util.add_deprecated_argument(
             self.parser.add_argument, argument_name, num_args)
 
-    def add_group(self, topic, **kwargs):
+    def add_group(self, topic, verbs=(), **kwargs):
         """Create a new argument group.
 
         This method must be called once for every topic, however, calls
@@ -574,6 +691,8 @@ class HelpfulArgumentParser(object):
         clarity.
 
         :param str topic: Name of the new argument group.
+        :param str verbs: List of subcommands that should be documented as part of
+                          this help group / topic
 
         :returns: The new argument group.
         :rtype: `HelpfulArgumentGroup`
@@ -581,6 +700,9 @@ class HelpfulArgumentParser(object):
         """
         if self.visible_topics[topic]:
             self.groups[topic] = self.parser.add_argument_group(topic, **kwargs)
+            if self.help_arg:
+                for v in verbs:
+                    self.groups[topic].add_argument(v, help=VERB_HELP_MAP[v]["short"])
 
         return HelpfulArgumentGroup(self, topic)
 
@@ -621,32 +743,17 @@ class HelpfulArgumentParser(object):
 def _add_all_groups(helpful):
     helpful.add_group("automation", description="Arguments for automating execution & other tweaks")
     helpful.add_group("security", description="Security parameters & server settings")
-    helpful.add_group(
+    helpful.add_group("testing",
-        "testing", description="The following flags are meant for "
+        description="The following flags are meant for testing and integration purposes only.")
-        "testing purposes only! Do NOT change them, unless you "
-        "really know what you're doing!")
-    # VERBS
-    helpful.add_group(
-        "renew", description="The 'renew' subcommand will attempt to renew all"
-        " certificates (or more precisely, certificate lineages) you have"
-        " previously obtained if they are close to expiry, and print a"
-        " summary of the results. By default, 'renew' will reuse the options"
-        " used to create obtain or most recently successfully renew each"
-        " certificate lineage. You can try it with `--dry-run` first. For"
-        " more fine-grained control, you can renew individual lineages with"
-        " the `certonly` subcommand. Hooks are available to run commands"
-        " before and after renewal; see"
-        " https://certbot.eff.org/docs/using.html#renewal for more"
-        " information on these.")
-
-    helpful.add_group("certonly", description="Options for modifying how a cert is obtained")
-    helpful.add_group("install", description="Options for modifying how a cert is deployed")
-    helpful.add_group("revoke", description="Options for revocation of certs")
-    helpful.add_group("rollback", description="Options for reverting config changes")
-    helpful.add_group("plugins", description='Options for the "plugins" subcommand')
-    helpful.add_group("config_changes",
-                      description="Options for showing a history of config changes")
     helpful.add_group("paths", description="Arguments changing execution paths & servers")
+    helpful.add_group("manage",
+        description="Various subcommands and flags are available for managing your certificates:",
+        verbs=["certificates", "renew", "revoke", "rename"])
+
+    # VERBS
+    for verb, docs in VERB_HELP:
+        name = docs.get("realname", verb)
+        helpful.add_group(name, description=docs["opts"])
 
 
 def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: disable=too-many-statements
@@ -675,7 +782,7 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         None, "-t", "--text", dest="text_mode", action="store_true",
         help=argparse.SUPPRESS)
     helpful.add(
-        [None, "automation"], "-n", "--non-interactive", "--noninteractive",
+        [None, "automation", "run", "certonly"], "-n", "--non-interactive", "--noninteractive",
         dest="noninteractive_mode", action="store_true",
         help="Run without ever asking for user input. This may require "
               "additional command line flags; the client will try to explain "
@@ -688,15 +795,15 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
              "multiple -d flags or enter a comma separated list of domains "
              "as a parameter.")
     helpful.add(
-        [None, "run", "certonly"],
+        [None, "run", "certonly", "manage"],
         "--cert-name", dest="certname",
         metavar="CERTNAME", default=None,
         help="Certificate name to apply. Only one certificate name can be used "
-             "per Certbot run. To see certificate names, run 'certbot certificates'."
+             "per Certbot run. To see certificate names, run 'certbot certificates'. "
              "If there is no existing certificate with this name and "
              "domains are requested, create a new certificate with this name.")
     helpful.add(
-        "rename",
+        ["rename", "manage"],
         "--updated-cert-name", dest="new_certname",
         metavar="NEW_CERTNAME", default=None,
         help="New name for the certificate. Must be a valid filename.")
@@ -728,9 +835,9 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         help="With the register verb, indicates that details associated "
              "with an existing registration, such as the e-mail address, "
              "should be updated, rather than registering a new account.")
-    helpful.add(None, "-m", "--email", help=config_help("email"))
+    helpful.add(["register", "automation"], "-m", "--email", help=config_help("email"))
     helpful.add(
-        ["automation", "renew", "certonly", "run"],
+        ["automation", "certonly", "run"],
         "--keep-until-expiring", "--keep", "--reinstall",
         dest="reinstall", action="store_true",
         help="If the requested cert matches an existing cert, always keep the "
@@ -784,7 +891,7 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         help="(certbot-auto only) prevent the certbot-auto script from"
              " upgrading itself to newer released versions")
     helpful.add(
-        ["automation", "renew", "certonly"],
+        ["automation", "renew", "certonly", "run"],
         "-q", "--quiet", dest="quiet", action="store_true",
         help="Silence all output except errors. Useful for automation via cron."
              " Implies --non-interactive.")
@@ -801,11 +908,11 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         help=config_help("no_verify_ssl"),
         default=flag_default("no_verify_ssl"))
     helpful.add(
-        ["certonly", "renew", "run"], "--tls-sni-01-port", type=int,
+        ["testing", "standalone", "apache", "nginx"], "--tls-sni-01-port", type=int,
         default=flag_default("tls_sni_01_port"),
         help=config_help("tls_sni_01_port"))
     helpful.add(
-        ["certonly", "renew", "run", "manual"], "--http-01-port", type=int,
+        ["testing", "standalone", "manual"], "--http-01-port", type=int,
         dest="http01_port",
         default=flag_default("http01_port"), help=config_help("http01_port"))
     helpful.add(
@@ -859,7 +966,7 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         help="Require that all configuration files are owned by the current "
              "user; only needed if your config is somewhere unsafe like /tmp/")
     helpful.add(
-        ["manual", "standalone", "certonly", "renew", "run"],
+        ["manual", "standalone", "certonly", "renew"],
         "--preferred-challenges", dest="pref_challs",
         action=_PrefChallAction, default=[],
         help='A sorted, comma delimited list of the preferred challenge to '
@@ -950,10 +1057,8 @@ def _paths_parser(helpful):
     if verb == "help":
         verb = helpful.help_arg
 
-    cph = "Path to where cert is saved (with auth --csr), installed from or revoked."
+    cph = "Path to where cert is saved (with auth --csr), installed from, or revoked."
-    section = "paths"
+    section = ["paths", "install", "revoke", "certonly", "manage"]
-    if verb in ("install", "revoke", "certonly"):
-        section = verb
     if verb == "certonly":
         add(section, "--cert-path", type=os.path.abspath,
             default=flag_default("auth_cert_path"), help=cph)
@@ -975,7 +1080,7 @@ def _paths_parser(helpful):
     default_cp = None
     if verb == "certonly":
         default_cp = flag_default("auth_chain_path")
-    add("paths", "--fullchain-path", default=default_cp, type=os.path.abspath,
+    add(["install", "paths"], "--fullchain-path", default=default_cp, type=os.path.abspath,
         help="Accompanying path to a full certificate chain (cert plus chain).")
     add("paths", "--chain-path", default=default_cp, type=os.path.abspath,
         help="Accompanying path to a certificate chain.")
@@ -1006,10 +1111,10 @@ def _plugins_parsing(helpful, plugins):
         "plugins", "--configurator", help="Name of the plugin that is "
         "both an authenticator and an installer. Should not be used "
         "together with --authenticator or --installer.")
-    helpful.add(["plugins", "certonly", "run", "install"],
+    helpful.add(["plugins", "certonly", "run", "install", "config_changes"],
                 "--apache", action="store_true",
                 help="Obtain and install certs using Apache")
-    helpful.add(["plugins", "certonly", "run", "install"],
+    helpful.add(["plugins", "certonly", "run", "install", "config_changes"],
                 "--nginx", action="store_true",
                 help="Obtain and install certs using Nginx")
     helpful.add(["plugins", "certonly"], "--standalone", action="store_true",

certbot/main.py

@@ -712,6 +712,7 @@ def _handle_exception(exc_type, exc_value, trace, config):
                 with open(logfile, "w") as logfd:
                     traceback.print_exception(
                         exc_type, exc_value, trace, file=logfd)
+                assert "--debug" not in sys.argv  # config is None if this explodes
             except:  # pylint: disable=bare-except
                 sys.exit(tb_str)
             if "--debug" in sys.argv:

certbot/tests/cli_test.py

@@ -84,6 +84,8 @@ class ParseTest(unittest.TestCase):
         self.assertTrue("--manual-test-mode" in out)
         self.assertTrue("--text" not in out)
         self.assertTrue("--dialog" not in out)
+        self.assertTrue("%s" not in out)
+        self.assertTrue("{0}" not in out)
 
         out = self._help_output(['-h', 'nginx'])
         if "nginx" in self.plugins:
@@ -97,7 +99,7 @@ class ParseTest(unittest.TestCase):
         if "nginx" in self.plugins:
             self.assertTrue("Use the Nginx plugin" in out)
         else:
-            self.assertTrue("(nginx support is experimental" in out)
+            self.assertTrue("(the certbot nginx plugin is not" in out)
 
         out = self._help_output(['--help', 'plugins'])
         self.assertTrue("--manual-test-mode" not in out)
@@ -125,8 +127,10 @@ class ParseTest(unittest.TestCase):
         self.assertTrue("--key-path" not in out)
 
         out = self._help_output(['-h'])
-
+        self.assertTrue(cli.SHORT_USAGE in out)
-        self.assertTrue(cli.usage_strings(self.plugins)[0] in out)
+        self.assertTrue(cli.COMMAND_OVERVIEW[:100] in out)
+        self.assertTrue("%s" not in out)
+        self.assertTrue("{0}" not in out)
 
     def test_parse_domains(self):
         short_args = ['-d', 'example.com']