Try another approach (#7022)

In #7019, a solution has been integrated to fix oldest tests execution in the corner cases described in #7014.

However this solution was not very satisfactory, as it consists in making a --force-reinstall for all requirements on each oldest tests (apache, certbot, acme, each dns plugin ...). As a consequence, the overall execution time of these tests increased from 5 min to 10 min.

In this PR I propose a more elegant solution: instead of reinstalling all dependencies, we force reinstall only the requirements themselves describe in the relevant oldest-requirements.txt files. This way only the packages that are potentially ignored by pip because they exists locally (acme, certbot, ...) are reinstalled.

The result is the same than in #7019 (we are sure that all packages are really installed by pip), but the very limited number of force reinstalled package here make the impact on execution time negligible.

As a consequence, I revert back also the tox environments to execute all oldest tests together.

A successful execution of oldest tests using this PR material in the context of a point release can be seen here: https://travis-ci.org/adferrand/certbot/builds/527513101

.travis.yml

@@ -60,12 +60,7 @@ matrix:
       env: TOXENV=mypy
       <<: *not-on-master
     - python: "2.7"
-      env: TOXENV='py27-{acme,apache,certbot,nginx,postfix}-oldest'
+      env: TOXENV='py27-{acme,apache,certbot,dns,nginx,postfix}-oldest'
-      sudo: required
-      services: docker
-      <<: *not-on-master
-    - python: "2.7"
-      env: TOXENV='py27-dns-oldest'
       sudo: required
       services: docker
       <<: *not-on-master

tools/pip_install.py

@@ -99,25 +99,25 @@ def main(args):
         else:
             # Otherwise, we merge requirements to build the constraints and pin dependencies
             requirements = None
-            reinstall = False
             if os.environ.get('CERTBOT_OLDEST') == '1':
                 requirements = certbot_oldest_processing(tools_path, args, test_constraints)
-                # We need to --force-reinstall the tested distribution when using oldest
-                # requirements because of an error in these tests in particular situations
-                # described in https://github.com/certbot/certbot/issues/7014.
-                # However this slows down considerably the oldest tests (5 min -> 10 min),
-                # so we need to find a better mitigation in the future.
-                reinstall = True
             else:
                 certbot_normal_processing(tools_path, test_constraints)
 
             merge_requirements(tools_path, requirements, test_constraints, all_constraints)
-            if requirements:
+            if requirements:  # This branch is executed during the oldest tests
+                # First step, install the transitive dependencies of oldest requirements
+                # in respect with oldest constraints.
                 pip_install_with_print('--constraint "{0}" --requirement "{1}"'
                                        .format(all_constraints, requirements))
+                # Second step, ensure that oldest requirements themselves are effectively
+                # installed using --force-reinstall, and avoid corner cases like the one described
+                # in https://github.com/certbot/certbot/issues/7014.
+                pip_install_with_print('--force-reinstall --no-deps --requirement "{0}"'
+                                       .format(requirements))
 
-            pip_install_with_print('--constraint "{0}" {1} {2}'.format(
+            pip_install_with_print('--constraint "{0}" {1}'.format(
-                all_constraints, '--force-reinstall' if reinstall else '', ' '.join(args)))
+                all_constraints, ' '.join(args)))
     finally:
         if os.environ.get('TRAVIS'):
             print('travis_fold:end:install_certbot_deps')