* Validate OCSP response for responders that are not the certificate's issuer.
* Improve OCSP tests using a issuer/responder pair for OCSP responses
* Clean code
* Update ocsp_test.py
* Add various comments
* Add several cases of ocsp responder. More factories for the resilience tests.
* Update ocsp_test.py
Fixes#6955.
This updates the Fedora version used in our test farm tests to Fedora 30. The AMI ID comes from https://alt.fedoraproject.org/cloud/ where it is listed as their standard HVM AMI for the region we use us-east-1 (US East (N. Virginia)).
Unfortunately, there were a lot of small changes required for this. The big reason for this is on Fedora, there isn't a Python 2 executable installed. In fact, there's not even an executable named python. It's just python3. Rather than installing another Python in each test, I wrote a script that the test scripts can share to figure out the different paths and names that should be used in their script. (This isn't used in test_sdists.sh because the logic is a little different.)
Other changes here worth flagging are:
I changed the name of the variable RUN_PYTHON3_TESTS in test_leauto_upgrades.sh to RUN_RHEL6_TESTS. The tests that are run when this variable is set test the upgrade from Python 2 to Python 3 on RHEL 6. I think this new name is much better now that we also have Fedora running Python 3.
I made tools/simple_http_server.py work on Python 3.
You can see tests passing with these changes at https://travis-ci.com/certbot/certbot/builds/113821476. I also ran test_tests.sh and they passed.
* Update to Fedora 30 in test farm tests.
Fedora 28 is likely to reach its EOL soon.
* Add set_python_envvars.sh.
* Fix test_apache2.sh on python3 only distros.
* Fix test_leauto_upgrades.sh on python3 systems.
* Fix certonly_standalone tests with python3 only
* Fix test_sdists.sh on python3 only distros.
* Make simple_http_server.py work on Python 3.
* add comments
* Ignore editor backups when running hooks.
When processing hooks, certbot also runs editor backups even though
such files are outdated, clearly warranted correction and may quite
possibly be defective.
That behavior could lead to unexpected breakage, and perhaps even pose
security risks---for example, if a previous script was careless with
file permissions. As an aggravating factor, the backup runs after the
corrected version and could unintentionally override a fix the user
thought was properly implemented.
This commit causes editor backup files ending in tilde (~) to be
excluded when running hooks.
Additional information can be found here:
https://github.com/certbot/certbot/issues/7107https://community.letsencrypt.org/t/editor-backup-files-executed-as-renewal-hooks/94750
* Add unit test for hook scripts with filenames ending in tilde.
* Provide changelog entry for not running hook scripts ending in tilde.
* Add Felix Lechner to the list of contributors.
Following discussion in #6947 (comment), I have second thoughts about relying on acme in certbot-ci.
Indeed, I think it is a good design to not rely in tests on the code you are testing. Obviously in unit tests it is very difficult, since most of the time the unit that is tested needs input generated by other part of the code. However it is not really a problem in a unit test, as its purpose is to make assertions about a specific portion of the code, not the others parts.
In the scope of integration tests, the software tested is treated as a black box. In this case, having some parts of the test logic that use in fact part of the code in the black box, increase the risk that some assertions compared two results coming from the same flawed logic from the tested software.
Since using acme in certbot-ci is only saving few lines of code, I think it does not worth the risk and the added complexity to declare acme as a dependency. I prefer to duplicate these lines and keep certbot-ci free of any dependency coming from the certbot project.
You can see the full test suite running at https://travis-ci.com/certbot/certbot/builds/112291892.
A few noteworthy things:
--fast is included because without, the tests would sometimes reach Travis' 50 minute timeout even with 1 test script per Travis build.
The only script that is run at release time which is not being run here is https://github.com/certbot/certbot/blob/master/tests/letstest/scripts/test_tests.sh because that script runs tests on the packages installed by certbot-auto which won't be updated until midway through a release.
We check TRAVIS_PULL_REQUEST and error out if it is not false for simplicity which should be fine because these tests are never run on PRs. The reason it's more complex to run test farm tests on PRs is the test farm tests need a named branch to pull from and Travis effectively merges the PR into the target branch before running tests complicating this.
I don't think this should block this PRs, but the one final change we may want to make to the current setup is #7071.
* Add encrypted private key.
* Add test farm tests to tox and travis.
* Change magic profile name.
* Further split test farm tests.
* Build local branch.
* more depth
* Set LOGDIR at top of script.
* Set sentinel at top of script.
* Don't use EC2 global to block on instance start.
* Remove global boto3 state.
* Pass in boulder_url.
* Create main function.
* Add link to reload docs.
* Add an option to dns_rfc2136 plugin to explicitly specify an authorative base domain.
* Updated CHANGELOG mentioning added base domain option
* Made the comment on the new option more clear on auto-detection
* Updated comment on how the authorative base domain is determined
* Added certbot-dns-rfc2136 to list of changed modules in CHANGELOG
* Add an option to dns_rfc2136 plugin to explicitly specify an authorative base domain.
* Updated CHANGELOG mentioning added base domain option
* Made the comment on the new option more clear on auto-detection
* Updated comment on how the authorative base domain is determined
* Connect certbot-ci to travis. Remove old bash files.
* Configure test-everything
* Protect against import error
* Remove unused ignore
* Better handling of urllib3
* Correct path
* Remove a warning
* Correct call
* Protect atexit register execution
* Update docs/contributing.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update docs/contributing.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Add again some bash scripts to avoid breaking to much retro-compatiblity on third party scripts
* Move boulder-v1 and boulder-v2 in nightly tests
* Separate oldest unit tests and oldest integration tests
* Remove try/except
* Test integration included in toxenv
* Add a wait to avoid a transient issue on OCSP status in oldest tests
* Clean travis.yml, split other tests
* Remove useless config
* Update .travis.yml
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update tox.ini
* Update tox.ini
* Remove pytest-sugar
* Remove empty pytest.ini, tests are working without it
Revert #6702
After some discussions, we realized that changing the path for FreeBSD users, if it corresponds to the path used when Certbot is installed using ports, will break for users that installed it through certbot-auto.
Indeed in this case, the path used was the one for Linux. After #6702, Certbot would not find anymore the existing config path by default.
It would require, to be integrated, a proper documentation and a migration path. For now, it is preferable to revert it.
This reverts commit 7fe82cf1ac.
* Fix check permissions logic (#7034)
Fixes#7031
I use the same approach than in `CreateVenv()` and `CompareVersions()`: a new bash function `CheckPathPermissions()` is declared an execute a python script passed to the interpreter through stdin.
This allows:
* to not require the temp_dir that holds a temporary script to be executed
* to reduce at the bare minimum the change to make on the order of bash command to execute (including when the temp_dir is created)
* Fix check permissions logic in certbot-auto by making a temp dir useless
* Update CHANGELOG.md
(cherry picked from commit 71b1b8c2d9)
* Fixup changelog.
Fixes#7031
I use the same approach than in `CreateVenv()` and `CompareVersions()`: a new bash function `CheckPathPermissions()` is declared an execute a python script passed to the interpreter through stdin.
This allows:
* to not require the temp_dir that holds a temporary script to be executed
* to reduce at the bare minimum the change to make on the order of bash command to execute (including when the temp_dir is created)
* Fix check permissions logic in certbot-auto by making a temp dir useless
* Update CHANGELOG.md
In #7019, a solution has been integrated to fix oldest tests execution in the corner cases described in #7014.
However this solution was not very satisfactory, as it consists in making a --force-reinstall for all requirements on each oldest tests (apache, certbot, acme, each dns plugin ...). As a consequence, the overall execution time of these tests increased from 5 min to 10 min.
In this PR I propose a more elegant solution: instead of reinstalling all dependencies, we force reinstall only the requirements themselves describe in the relevant oldest-requirements.txt files. This way only the packages that are potentially ignored by pip because they exists locally (acme, certbot, ...) are reinstalled.
The result is the same than in #7019 (we are sure that all packages are really installed by pip), but the very limited number of force reinstalled package here make the impact on execution time negligible.
As a consequence, I revert back also the tox environments to execute all oldest tests together.
A successful execution of oldest tests using this PR material in the context of a point release can be seen here: https://travis-ci.org/adferrand/certbot/builds/527513101
