Current version of pywin32 used in certbot (225) does not have wheels available for Python 3.8. Installing certbot for development in this case requires to build from source. On Windows, this implies a Visual Studio C++ environment up and ready, which is absolutely not fun.
Let's upgrade to pywin32 227, that provides these wheels for all Python versions from 3.5 up to current dev status of 3.9.
This PR adds appropriate corrections to certbot dockerfile to work with the new layout (moving certbot python project in its own subdirectory).
This PR has been tested with success using the `build.sh` script on a fake v1.0 version of certbot published on my fork (https://github.com/adferrand/certbot/releases/tag/v1.0) instead of archives from `certbot/certbot`.
Clean up some places missed by #7544.
Found this when running test farm tests. They were working as of 5d90544, and I will truly shocked if subsequent changes (all to the windows installer) made them stop working.
* Release script needs to target new CHANGELOG location
* Clean up various other CHANGELOG path references
* Update windows paths for new certbot location
* Add certbot to packages list for windows installer
Summary of changes in this PR:
- Refactor files involved in the `certbot` module to be of a similar structure to every other package; that is, inside a directory inside the main repo root (see below).
- Make repo root README symlink to `certbot` README.
- Pull tests outside of the distributed module.
- Make `certbot/tests` not be a module so that `certbot` isn't added to Python's path for module discovery.
- Remove `--pyargs` from test calls, and make sure to call tests from repo root since without `--pyargs`, `pytest` takes directory names rather than package names as arguments.
- Replace mentions of `.` with `certbot` when referring to packages to install, usually editably.
- Clean up some unused code around executing tests in a different directory.
- Create public shim around main and make that the entry point.
New directory structure summary:
```
repo root ("certbot", probably, but for clarity all files I mention are relative to here)
├── certbot
│ ├── setup.py
│ ├── certbot
│ │ ├── __init__.py
│ │ ├── achallenges.py
│ │ ├── _internal
│ │ │ ├── __init__.py
│ │ │ ├── account.py
│ │ │ ├── ...
│ │ ├── ...
│ ├── tests
│ │ ├── account_test.py
│ │ ├── display
│ │ │ ├── __init__.py
│ │ │ ├── ...
│ │ ├── ... # note no __init__.py at this level
│ ├── ...
├── acme
│ ├── ...
├── certbot-apache
│ ├── ...
├── ...
```
* refactor certbot/ and certbot/tests/ to use the same structure as the other packages
* git grep -lE "\-e(\s+)\." | xargs sed -i -E "s/\-e(\s+)\./-e certbot/g"
* git grep -lE "\.\[dev\]" | xargs sed -i -E "s/\.\[dev\]/certbot[dev]/g"
* git grep -lE "\.\[dev3\]" | xargs sed -i -E "s/\.\[dev3\]/certbot[dev3]/g"
* Remove replacement of certbot into . in install_and_test.py
* copy license back out to main folder
* remove linter_plugin.py and CONTRIBUTING.md from certbot/MANIFEST.in because these files are not under certbot/
* Move README back into main folder, and make the version inside certbot/ a symlink
* symlink certbot READMEs the other way around
* move testdata into the public api certbot zone
* update source_paths in tox.ini to certbot/certbot to find the right subfolder for tests
* certbot version has been bumped down a directory level
* make certbot tests directory not a package and import sibling as module
* Remove unused script cruft
* change . to certbot in test_sdists
* remove outdated comment referencing a command that doesn't work
* Install instructions should reference an existing file
* update file paths in Dockerfile
* some package named in tox.ini were manually specified, change those to certbot
* new directory format doesn't work easily with pyargs according to http://doc.pytest.org/en/latest/goodpractices.html#tests-as-part-of-application-code
* remove other instance of pyargs
* fix up some references in _release.sh by searching for ' . ' and manual check
* another stray . in tox.ini
* fix paths in tools/_release.sh
* Remove final --pyargs call, and now-unnecessary call to modules instead of local files, since that's fixed by certbot's code being one layer deeper
* Create public shim around main and make that the entry point
* without pyargs, tests cannot be run from an empty directory
* Remove cruft for running certbot directly from main
* Have main shim take real arg
* add docs/api file for main, and fix up main comment
* Update certbot/docs/install.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Fix comments in readthedocs requirements files to refer to current package
* Update .[docs] reference in contributing.rst
* Move plugins tests to certbot tests directory
* add certbot tests to MANIFEST.in so packagers can run python setup.py test
* move examples directory inside certbot/
* Move CHANGELOG into certbot, and create a top-level symlink
* Remove unused sys and logging from main shim
* nginx http01 test no longer relies on certbot plugins common test
This PR creates a pipeline triggered on tag push matching v0.* (eg. v0.40.0).
Once triggered, this pipeline will build the windows installer, and run integration tests on it, like for the pipeline run nightly.
I also add a simple script to extract from CHANGELOG.md file to extract the relevant part to put it in the body of the GitHub release. I believe it makes things nicer.
* Create release pipeline
* Relax condition on tags
* Put beta keyword
* Update job name
* Fix release pipeline
Over the weekend, nightly tests on Windows failed for certbot-dns-google: https://dev.azure.com/certbot/web/build.aspx?pcguid=74ef9c03-9faf-405b-9d03-9acf8c43e8d6&builduri=vstfs%3a%2f%2f%2fBuild%2fBuild%2f72
The error occurred inside `oauth2client`'s locking code and the failure seems spurious as it did not reproduce this morning: https://dev.azure.com/certbot/certbot/_build/results?buildId=73
I could not find a relevant changelog entry in `oauth2client` saying they've fixed the problem, but the problematic code no longer exists in `oauth2client>=4.0`. This PR updates our minimum dependency required in an attempt to avoid spurious failures for us in the future. The only downside I am aware of is it'll make it harder for certbot-dns-google to be packaged in Debian Old Stable or Ubuntu 16.04, but I don't expect either of those things to happen anytime soon.
* bump oauth2client dep
* Update dev_constraints.txt.
* Add changelog entry for packagers.
* Add one Dockerfile for each supported architecture
* Update multi arch hooks
* Create multi arch scripts
* Update README.md
* WIP. Use build args instead of multiple Dockerfiles in build script
* WIP. Fix typo mistake
* Use build args instead of multiple Dockerfiles in build script
* WIP. Build all the architectures in one DockerHub build
* Add arm64v8 architecture
* WIP. Testing build all the architectures in one DockerHub build
* Revert "WIP. Testing build all the architectures in one DockerHub build"
This reverts commit 94a89398a4120b183d2851ac7cb9c93db0e3d187.
* Refactor tag docker images in hooks/post_push files
* Use variables instead of positional arguments
* Export externally used variables
* Use ${variable//search/replace} instead of echo $variable | sed.
* Update README.md
* Add Cleanup in build.sh script
* Fix tagging error in post_push hook
* Add "-ex" flags to bash script
* Test tagging images in build hook
* Tagging in hook/post_build instead of hook/post_push
* Push built architecture dependent image
* Use Dockerfile argument instead of fixed value
* Fix typo
* Use parameter instead of global variable
* Use custom "hook/push" to prevent duplicated push
* Make a short doctype for each function declared in common
This PR implements the item "register a scheduled task for certificate renewal" from the list of requirements described in #7365.
This PR adds required instructions in the NSIS installer for Certbot to create a task, named "Certbot Renew Task" in the Windows Scheduler. This task is run twice a day, to execute the command certbot renew and keep the certificates up-to-date.
Uninstalling Certbot will also remove this scheduled task.
* Implementation
* Corrections
* Update template.nsi
* Improve scripts
* Add a random delay of 12 hours
* Synchronize template against default one in pynsist 2.4
* Clean config of scheduled task
* Install only in AllUsers mode
* Add comments
* Remove the logic of single user install
* Get integration tests working on python 3.8
* Run unit tests on py38
* Update coveragercs to use coverage 4.5+ format
* remove line added to tox.ini
* update changelog
* xenial is the new travis default; no need to specify in .travis.yml
Smallest possible fix for #7106
* Replace platform.linux_dependencies with distro.linux_dependencies
* run build.py
* Add minimum version of 1.0.1
* Pin back requests package
* Update changelog
* Fix account_tests
* Fix hook executable test
* Remove the temporary decorator @broken_on_windows
* Fix util_test
* No broken unit test on Windows anymore
* More elegant mock
* Fix context manager
* Fix lint
* Fix mypy
* Adapt coverage
* Corrections
* Fix lint
* Adapt coverage
* Update certbot/tests/compat/filesystem_test.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update util_test.py
* Fix pylint
* Forbid os.access
* Update os_test.py
* Update os.py
* Fix lint
* Update filesystem.py
* Update filesystem.py
* Update filesystem.py
* Update os.py
* Start fixing tests
* Platform independent hooks
* Fix probe fd close
* Add broken_on_windows for integration tests
* Fix a lot of tests
* Use a python hook script, to prepare cross-platform
* New approach to be compliant with Linux and Windows on hook scripts
* New tests fixed
* Test for permissions on Windows
* Permissions comparison for Windows
* No broken tests in certbot core anymore
* Change mode
* Specific config for appveyor
* Use forked pebble for now
* Various fixes
* Assert file permissions for world on private keys
* Clean code
* Fix several things
* Add integration target
* Optimize integration env
* Re-enable all AppVeyor envs
* Use again official pebble
* Update pebble_artifacts.py
* Set PYTEST_ADDOPTS silently
* Update appveyor.yml
* Pin pywin32 for tests, give a minimal requirement for certbot.
* Remove injection of nginx in PATH
* Clean debug code
* Various cleanup, ensure to remove workspace after tests
* Update tox target
* Improve assertions. Control the keyword echoed in hooks
* Fix for virtualenv on Python 3.7.4 for Windows
* Update certbot-ci/certbot_integration_tests/certbot_tests/assertions.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Add conditionally pywin in certbot-ci like in certbot
AppVeyor recently upgrade the Python 3.7.x installed in their VM to 3.7.4. However, virtualenv 16.6.1 is broken on that specific version of Python for Windows.
This PR upgrade virtualenv installed for a dev/test environment from 16.6.1 to 16.6.2 in order to fix this issue, and repair the CI jobs execute by AppVeyor on PRs.
* Update virtualenv to the latest version.
* Use venv from pip and pin more packages.
* Pin codecov.
* update appveyor config
* Write the path separator backwards.
* s/pip_install.py install/pip_install.py
* Prefix tools\\pip_install.py with python exe.
* Upgrade py to fix AppVeyor failures.
* add back comment
* Update virtualenv with CERTBOT_NO_PIN.
* Pass -U to upgrade tox and deps.
* Upgrade virtualenv.
In this spirt of cleaning up some low hanging cruft, this fixes#4343.
There are no (recent) release tags on quay.io and the builds are just following master. See https://quay.io/repository/letsencrypt/letsencrypt?tab=tags.
Once this lands, I can disable the automated builds on quay.io and we can delete Dockerfile-old and tools/docker-warning.sh.
Fixes#6955.
This updates the Fedora version used in our test farm tests to Fedora 30. The AMI ID comes from https://alt.fedoraproject.org/cloud/ where it is listed as their standard HVM AMI for the region we use us-east-1 (US East (N. Virginia)).
Unfortunately, there were a lot of small changes required for this. The big reason for this is on Fedora, there isn't a Python 2 executable installed. In fact, there's not even an executable named python. It's just python3. Rather than installing another Python in each test, I wrote a script that the test scripts can share to figure out the different paths and names that should be used in their script. (This isn't used in test_sdists.sh because the logic is a little different.)
Other changes here worth flagging are:
I changed the name of the variable RUN_PYTHON3_TESTS in test_leauto_upgrades.sh to RUN_RHEL6_TESTS. The tests that are run when this variable is set test the upgrade from Python 2 to Python 3 on RHEL 6. I think this new name is much better now that we also have Fedora running Python 3.
I made tools/simple_http_server.py work on Python 3.
You can see tests passing with these changes at https://travis-ci.com/certbot/certbot/builds/113821476. I also ran test_tests.sh and they passed.
* Update to Fedora 30 in test farm tests.
Fedora 28 is likely to reach its EOL soon.
* Add set_python_envvars.sh.
* Fix test_apache2.sh on python3 only distros.
* Fix test_leauto_upgrades.sh on python3 systems.
* Fix certonly_standalone tests with python3 only
* Fix test_sdists.sh on python3 only distros.
* Make simple_http_server.py work on Python 3.
* add comments
* Connect certbot-ci to travis. Remove old bash files.
* Configure test-everything
* Protect against import error
* Remove unused ignore
* Better handling of urllib3
* Correct path
* Remove a warning
* Correct call
* Protect atexit register execution
* Update docs/contributing.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update docs/contributing.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Add again some bash scripts to avoid breaking to much retro-compatiblity on third party scripts
* Move boulder-v1 and boulder-v2 in nightly tests
* Separate oldest unit tests and oldest integration tests
* Remove try/except
* Test integration included in toxenv
* Add a wait to avoid a transient issue on OCSP status in oldest tests
* Clean travis.yml, split other tests
* Remove useless config
* Update .travis.yml
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update tox.ini
* Update tox.ini
* Remove pytest-sugar
* Remove empty pytest.ini, tests are working without it
In #7019, a solution has been integrated to fix oldest tests execution in the corner cases described in #7014.
However this solution was not very satisfactory, as it consists in making a --force-reinstall for all requirements on each oldest tests (apache, certbot, acme, each dns plugin ...). As a consequence, the overall execution time of these tests increased from 5 min to 10 min.
In this PR I propose a more elegant solution: instead of reinstalling all dependencies, we force reinstall only the requirements themselves describe in the relevant oldest-requirements.txt files. This way only the packages that are potentially ignored by pip because they exists locally (acme, certbot, ...) are reinstalled.
The result is the same than in #7019 (we are sure that all packages are really installed by pip), but the very limited number of force reinstalled package here make the impact on execution time negligible.
As a consequence, I revert back also the tox environments to execute all oldest tests together.
A successful execution of oldest tests using this PR material in the context of a point release can be seen here: https://travis-ci.org/adferrand/certbot/builds/527513101
Fixes#7014.
Using a --force-reinstall (only for oldest tests), dependencies are properly reinstalled. Since this action significantly increases the execution time of oldest tests, I split them into two parts to allow their parallel execution by Travis.
We will need to find a better way to solve this in the future.
An example of successful execution of oldest tests in the situation of a point release can be found here: https://travis-ci.org/adferrand/certbot/builds/527475532
* Fix for oldest requirements
* Split oldest tests
* Update a comment
Fixes#6861.
_venv_common.py is no longer executable. The reason for this is the venv creation logic is now different between Python 2 and Python 3. We could add code that branches on the Python version running the script, but I personally think that's unnecessary.
--setuptools and --no-site-packages is no longer passed to virtualenv either. These flags were made noops in virtualenv 1.10 and 1.7 respectively, but all of CentOS 6, 7, Debian 8+, and Ubuntu 14.04+ have new enough versions of virtualenv where these flags are no longer necessary. They are not even accepted as flags to Python 3's venv module.
Use of VENV_ARGS from test_sdists.sh was also removed because that environment variable hasn't done anything in a while.
I ran test farm tests on test_apache2.sh and test_sdists.sh with these changes and they passed.
* Fixes#6861.
* _venv_common is no longer executable.
When releasing 0.33.1 and resolving merge conflicts between the candidate-0.33.1 branch and master, I had merge conflicts in the local-oldest-requirements.txt files. This is because the point release branch does not contain modifications to these files that landed in master because it happens later in the release script in the commit bumping version numbers which is not included in the point release branch.
I think having to resolve these merge conflicts is unnecessary and even a slight problem because it means that the "oldest" tests on the point release branch may still be using the latest version of certain components when they actually should be using an older version.
I fixed this by moving this code earlier in the script so the local-oldest-requirements.txt files are updated at the same time as the setup.py files.
