Summary of changes in this PR:
- Refactor files involved in the `certbot` module to be of a similar structure to every other package; that is, inside a directory inside the main repo root (see below).
- Make repo root README symlink to `certbot` README.
- Pull tests outside of the distributed module.
- Make `certbot/tests` not be a module so that `certbot` isn't added to Python's path for module discovery.
- Remove `--pyargs` from test calls, and make sure to call tests from repo root since without `--pyargs`, `pytest` takes directory names rather than package names as arguments.
- Replace mentions of `.` with `certbot` when referring to packages to install, usually editably.
- Clean up some unused code around executing tests in a different directory.
- Create public shim around main and make that the entry point.
New directory structure summary:
```
repo root ("certbot", probably, but for clarity all files I mention are relative to here)
├── certbot
│ ├── setup.py
│ ├── certbot
│ │ ├── __init__.py
│ │ ├── achallenges.py
│ │ ├── _internal
│ │ │ ├── __init__.py
│ │ │ ├── account.py
│ │ │ ├── ...
│ │ ├── ...
│ ├── tests
│ │ ├── account_test.py
│ │ ├── display
│ │ │ ├── __init__.py
│ │ │ ├── ...
│ │ ├── ... # note no __init__.py at this level
│ ├── ...
├── acme
│ ├── ...
├── certbot-apache
│ ├── ...
├── ...
```
* refactor certbot/ and certbot/tests/ to use the same structure as the other packages
* git grep -lE "\-e(\s+)\." | xargs sed -i -E "s/\-e(\s+)\./-e certbot/g"
* git grep -lE "\.\[dev\]" | xargs sed -i -E "s/\.\[dev\]/certbot[dev]/g"
* git grep -lE "\.\[dev3\]" | xargs sed -i -E "s/\.\[dev3\]/certbot[dev3]/g"
* Remove replacement of certbot into . in install_and_test.py
* copy license back out to main folder
* remove linter_plugin.py and CONTRIBUTING.md from certbot/MANIFEST.in because these files are not under certbot/
* Move README back into main folder, and make the version inside certbot/ a symlink
* symlink certbot READMEs the other way around
* move testdata into the public api certbot zone
* update source_paths in tox.ini to certbot/certbot to find the right subfolder for tests
* certbot version has been bumped down a directory level
* make certbot tests directory not a package and import sibling as module
* Remove unused script cruft
* change . to certbot in test_sdists
* remove outdated comment referencing a command that doesn't work
* Install instructions should reference an existing file
* update file paths in Dockerfile
* some package named in tox.ini were manually specified, change those to certbot
* new directory format doesn't work easily with pyargs according to http://doc.pytest.org/en/latest/goodpractices.html#tests-as-part-of-application-code
* remove other instance of pyargs
* fix up some references in _release.sh by searching for ' . ' and manual check
* another stray . in tox.ini
* fix paths in tools/_release.sh
* Remove final --pyargs call, and now-unnecessary call to modules instead of local files, since that's fixed by certbot's code being one layer deeper
* Create public shim around main and make that the entry point
* without pyargs, tests cannot be run from an empty directory
* Remove cruft for running certbot directly from main
* Have main shim take real arg
* add docs/api file for main, and fix up main comment
* Update certbot/docs/install.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Fix comments in readthedocs requirements files to refer to current package
* Update .[docs] reference in contributing.rst
* Move plugins tests to certbot tests directory
* add certbot tests to MANIFEST.in so packagers can run python setup.py test
* move examples directory inside certbot/
* Move CHANGELOG into certbot, and create a top-level symlink
* Remove unused sys and logging from main shim
* nginx http01 test no longer relies on certbot plugins common test
Fixes#7184.
I updated #7358 to track the issue of unpinning all of these dependencies.
* pin back configargparse
* Pin back zope packages.
* update deps
* Add changelog entry.
* run build.py
When you try to run this script, it crashes with:
```
standard_init_linux.go:211: exec user process caused "exec format error"
```
This is caused by the script being written to have the contents:
```
\
#!/bin/sh
set -e
...
```
This fixes the problem by removing the slash and moving the shebang to the first line of the string.
Fixes#7007
Python 3.4 is [EOL](https://www.python.org/dev/peps/pep-0429/), and only Python 3.x version available for CentOS 6 through EPEL is this version, and so is used by `certbot-auto`, the only official way to install Certbot on this platform.
This unpleasant situation becomes a little more uncomfortable, considering that the newest `pip` version (19.2) [just dropped Python 3.4 support](https://github.com/pypa/pip/issues/6685) and will refuse to start on this Python version. We can expect a lot of dependencies to follow this path now.
One direct result of this situation is that a fix to support correctly the ARM platforms requires to upgrade `pip` to 19.2 for `certbot-auto`. So this is not possible right now.
Then, let's upgrade Certbot instances on CentOS 6 to a supported version of Python 3.
This PR proposes a new bootstrap approach for CentOS 6 platform, `BootstrapRpmPython3Legacy`, that will install Python 3.6 from [SCL](https://www.softwarecollections.org) (the latest one available for now on CentOS 6). In term of Python 3 specific bootstrap methods, I take the occasion here to completely separate the bootstrap of CentOS 6 as a legacy system, from the RPM-based newest systems (like Fedora 29+) that are simply dropping support for Python 2.x. This is in prevision of future migration for all systems on Python 3.x, that is a different problematic than supporting old systems.
* Add logic
* Rebuilt letsencrypt-auto
* Fix logic
* Focus on specific packages
* Maintain PATH for further invocations of letsencrypt-auto after bootstrap.
* Various corrections
* Fix farm test for RHEL6
* Working centos6 letsencrypt-auto self tests
* Fix test_sdist for CentOS 6
* Corrections
* Work in progress
* Working configuration
* Fix typo
* Remove EPEL. Add a test.
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Improvements after review
* Improvements
* Add a comment
* Add a test
* Update a test
* Corrections
* Update function return
* Work in progress
* Correct behavior on oracle linux 6.
* Corrections
* Rebuild script
* Add letsencrypt-auto tests for oraclelinux6
* Update tox.ini
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Remove specific code for scientific linux
* Change some variables names
* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Various corrections
* Fix tests
* Add a comment
* Update message
* Fix test message
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update scripts
* More focused assertion
* Add back a test
* Update script
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Check quiet mode
* Add changelog
* Update letsencrypt-auto-source/tests/oraclelinux6_tests.sh
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
Smallest possible fix for #7106
* Replace platform.linux_dependencies with distro.linux_dependencies
* run build.py
* Add minimum version of 1.0.1
* Pin back requests package
* Update changelog
/usr/bin/python no longer exists in RHEL 8. This patch updates
the certbot-auto script to use python3 on nodes running RHEL 8.
Also fixed a bug in the RPM_DIST_VERSION logic which would cause
letsencrypt-auto to fail on servers running CentOS/RHEL 6.
* Fix check permissions logic (#7034)
Fixes#7031
I use the same approach than in `CreateVenv()` and `CompareVersions()`: a new bash function `CheckPathPermissions()` is declared an execute a python script passed to the interpreter through stdin.
This allows:
* to not require the temp_dir that holds a temporary script to be executed
* to reduce at the bare minimum the change to make on the order of bash command to execute (including when the temp_dir is created)
* Fix check permissions logic in certbot-auto by making a temp dir useless
* Update CHANGELOG.md
(cherry picked from commit 71b1b8c2d9)
* Fixup changelog.
Fixes#7012.
Apparently, the previous test we had here doesn't catch the case when certbot-auto prints blank lines. (I don't yet understand why so if someone does, please let me know!)
Regardless, I fixed up the test and verified it fails with the version of letsencrypt-auto in master and then fixed letsencrypt-auto so the test passes.
I ran test farm tests on the changes here and they passed on all instances.
* correct test
* fixes#7012
(cherry picked from commit e15e848474)
This PR attempts to better inform people about the problem identified at https://community.letsencrypt.org/t/certbot-auto-deployment-best-practices/91979/.
I was hesitant to add the flag --no-permissions-check, however, if there's some obscure distro out there (or custom user setup) that has a strange users and groups, I didn't want us to either:
Have to put out a bug fix release
Refuse to fix the problem and let them deal with warnings on every run
* add check_permissions.py
* Update letsencrypt-auto.template.
* build letsencrypt-auto
* Add test_permissions_warnings to auto_test
* Allow uid/gid < 1000.
* Add --no-permissions-check to Certbot.
* Add --no-permissions-check to certbot-auto.
* Add test farm test that letsencrypt-auto is quiet.
As a bonus, this new test will catch problems like the one that the caused
0.33.1 point release.
* Update CHANGELOG about permissions check.
* Update permissions comment.
* Fix symlink handling.
* Use a better default in auto_test.py.
The changelog should still say <version> - master because it will be fixed up automatically by the release script at https://github.com/certbot/certbot/blob/master/tools/_release.sh#L69.
* Protect certbot-auto against non numerical version release in some RPM distributions (#6913)
Fixes#6912
Bash evaluate all condition in a predicate statement, eg. `"$SOMEVAR" = "test" -a "$ANOTHERVAR" = "test2"`, even if it is not necessary, for instance if the first condition is false in the example here.
As a consequence, on non-Fedora distributions, an evaluation of the distribution version could be done on non numeric value, eg. `"6.7" -eq "29"`, making certbot-auto failing in this case.
This PR fixes that, by evaluating the version on RPM distributions only if we are on Fedora. Otherwise, version will be "0".
(cherry picked from commit c2d9ea1f61)
* Update changelog about #6912 fix. (#6914)
(cherry picked from commit 30eafba997)
* cleanup changelog
Dependencies generated by the script introduced with #6839 were not including anymore the fix about enum34 for CentOS 6.
This PR reinserts this fix, and updates the script overrides to ensure that this fix will stay in next dependencies generation.
* Add the environment marker back. Ensure that it will stay by adding an override to dependencies generator.
* Add comments, for future fix
* Update letsencrypt-auto-source/rebuild_dependencies.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update comment
Currently, `tox -e le_auto_jessie` job fails. It breaks in particular the cron pipeline that test everything each night.
The failure occurs while setting up the Jessie Docker container to run the tests for certbot-auto, when `apt-get update` is invoked, with this error:
```
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/main/binary-amd64/Packages 404 Not Found
```
Indeed, if there are `stretch-updates`, `buster-updates` and so on in the repository, there is no `jessie-updates`. I do not know exactly the logic of Debian here, but as `*-updates` folders store stable updates, a distribution moving to LTS support like Jessie has no stable updates anymore. I suppose `jessie-updates have been decommissioned recently, and the official Docker has not been updated yet to use the LTS configuration for repositories.
This PR does that live in the Dockerfile, using official instructions from https://wiki.debian.org/LTS/Using, and fixes this specific job.
An example of a successful job with this modification can be found here: https://travis-ci.com/certbot/certbot/jobs/187864341
Fixes#6698
Fedora maintainers engaged a deprecation path for Python 2.x with Fedora 29. As a first step, python2-virtualenv does not install the virtualenv binary anymore, in favor of python3-virtualenv, and so the installation of Python 3 virtual environments by default.
However, certbot-auto installs python2-virtualenv for all recent RPM distributions, and relies of the execution of virtualenv, and this is failing the process.
Since the plan in the future is to remove Python 2.x from Fedora, this PR follows this logic to fix certbot-auto: started to Fedora 29, certbot-auto will install and execute certbot on Python 3. This implies to detect that we are on Fedora 29+, install python3-virtualenv that will install also Python 3 dependencies and virtualenv binary, then instruct the process to use Python 3. This is in fact similar to EOL distributions shipping with Python 2.6, and for which Python 3.4 from EPEL is installed and used.
Older versions of Fedora continue to use Python 2.x, and their process is untouched. Four scenarios are covered here:
fresh Fedora 28: old process is used, nothing changes
fresh Fedora 29: new process is used, Python 3 is installed, certbot runs on it
update Fedora 29 from 28, already installed certbot-auto without rebootstrapping required: existing venv continue to be used, certbot runs on it
update Fedora 29 from 28, already installed certbot-auto with rebootstrapping required: new process is used, installing python3-virtualenv, python3-devel and python3-rpm-macros, Python 3 is installed, certbot runs on it
* Add a step to handle python3 on fedora29
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update rpm_python3.sh
* Rebuild certbot-auto
* Empty commit to relaunch CI pipeline
* Add changelog
* Update CHANGELOG.md
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update CHANGELOG.md
