Over the weekend, nightly tests on Windows failed for certbot-dns-google: https://dev.azure.com/certbot/web/build.aspx?pcguid=74ef9c03-9faf-405b-9d03-9acf8c43e8d6&builduri=vstfs%3a%2f%2f%2fBuild%2fBuild%2f72
The error occurred inside `oauth2client`'s locking code and the failure seems spurious as it did not reproduce this morning: https://dev.azure.com/certbot/certbot/_build/results?buildId=73
I could not find a relevant changelog entry in `oauth2client` saying they've fixed the problem, but the problematic code no longer exists in `oauth2client>=4.0`. This PR updates our minimum dependency required in an attempt to avoid spurious failures for us in the future. The only downside I am aware of is it'll make it harder for certbot-dns-google to be packaged in Debian Old Stable or Ubuntu 16.04, but I don't expect either of those things to happen anytime soon.
* bump oauth2client dep
* Update dev_constraints.txt.
* Add changelog entry for packagers.
The value of --server will now be respected, except when it is the
default value, in which case it will be changed to the staging server,
preserving Certbot's existing behavior.
When setting up Azure Pipelines, I didn't like having to define codecov_token for each pipeline. This works around it by using a shared variable group.
You can see this working successfully at https://dev.azure.com/certbot/certbot/_build/results?buildId=3.
* Use certbot-common.
* update instructions
This PR defines pipelines that can be run on Azure Pipelines. Currently there are two:
* `.azure-pipelines/main.yml` is the main one, executed on PRs for master, and pushes to master,
* `.azure-pipelines/advanced.yml` add installer testing on top of the main pipeline, and is executed for `test-*` branches, release branches, and nightly run for master.
These two pipelines covers all existing stuff done by AppVeyor currently, and so AppVeyor can be decommissioned once Azure Pipelines is operational.
You can see working pipeline in my fork:
* a PR for `master` (so using main pipeline): https://github.com/adferrand/certbot/pull/65
* a PR for `test-something` (so using advanced pipeline): https://github.com/adferrand/certbot/pull/66
* uploaded coverage from Azure Pipelines: 499aa2cbf2/build
Once this PR is merged, we need to enable Azure Pipelines for Certbot. Instructions are written in `azure-pipelines/INSTALL.md`. This document also references all access rights required to Azure Pipelines onto GitHub to make the CI process work.
Future work for future PRs:
* create a CD pipeline for the releases that will push the installer to GitHub releases
* implement a solution to generate notification on IRC or Mattermost when a nightly build fails
* Define pipelines
* Update locations
* Update nightly
* Use x86
* Update nightly.yml for Azure Pipelines
* Run script
* Use script
* Update install
* Use local installation
* Register warnings
* Fix pywin32 loading
* Clean context
* Enable coverage publication
* Consume codecov token
* Document installation
* Update tool to upload coverage
* Prepare pipeline artifacts
* Update artifact ignore
* Protect against codecov failures
* Add a comment about codecov
* Add a comment on RW access asked by Azure
* Add instructions
* Rename pipeline file
* Update instructions
* Update .azure-pipelines/templates/tests-suite.yml
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update .azure-pipelines/INSTALL.md
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Modified scheduled pipeline
* Add comment
* Remove dynamic version-based installer name
* Remove unnecessary account ID match check.
Right now the Account object calculates an ID using md5. This is
unnecessary and causes problems on FIPS systems that forbid md5. It's
just as good to pick a random series of bytes for the ID, since the ID
gets read out of renewal/foo.conf.
However, if we switched the algorithm right now, we could wind up
breaking forward compatibility / downgradeability, since older versions
would run into this check.
Removing this check now lays the ground to change the ID-calculation
algorithm in the future.
Related to #1948 and
https://github.com/certbot/certbot/pull/1013#issuecomment-149983479.
* Remove test.
* Remove unused import.
The repo description for the [3rd party Icecast plugin](https://github.com/e00E/lets-encrypt-icecast) says that the plugin isn't currently working and the repository hasn't been updated since 2017. Since it seems broken and unmaintained, let's remove it from the list of third party plugins.
I would happily add it again to the list of third party plugins if people fix and maintain it.
The README for the [3rd party heroku plugin](https://github.com/gboudreau/certbot-heroku) says it has been deprecated. Because of this, let's remove it from the list of third party plugins.
Try to primarily fall back to using `platform.linux_distribution()` if `/etc/os-release` isn't available. Only use `distro.linux_distribution()` on Python >= 3.8.
* Try to use platform.linux_distribution() before distro equivalent
* Fix tests for py38
* Added changelog entry
This PR fixes a regression in #7337 (0.38.0) that certbot cannot run with Apache on RHEL 6.
In RHEL 6, `distro.linux_distribution()` returns `RedHatEnterpriseServer`.
In RHEL 6:
```py
>>> import distro
>>> distro.linux_distribution()
(u'RedHatEnterpriseServer', u'6.10', u'Santiago')
>>> import platform
>>> platform.linux_distribution()
('Red Hat Enterprise Linux Server', '6.10', 'Santiago')
```
In RHEL 7:
```py
>>> import distro
>>> distro.linux_distribution()
('Red Hat Enterprise Linux Server', '7.6', 'Maipo')
>>> import platform
>>> platform.linux_distribution()
('Red Hat Enterprise Linux Server', '7.6', 'Maipo')
```
* fix to run with Apache on RHEL 6
* fix docs
Fixes#7368.
When updating the changelog, I replaced the line about running tests on Python 3.8 because I personally think that support for Python 3.8 is the most relevant information for our users/packagers about our changes in this area.
* List support for Python 3.8.
* Update changelog.
Fixes#7152.
* don't check ocsp if cert is expired when getting cert information
* don't check ocsp if the cert is expired in ocsp_revoked
* update tests
* update changelog
* move pytz import to the top of the test file
This PR implements the item "register a scheduled task for certificate renewal" from the list of requirements described in #7365.
This PR adds required instructions in the NSIS installer for Certbot to create a task, named "Certbot Renew Task" in the Windows Scheduler. This task is run twice a day, to execute the command certbot renew and keep the certificates up-to-date.
Uninstalling Certbot will also remove this scheduled task.
* Implementation
* Corrections
* Update template.nsi
* Improve scripts
* Add a random delay of 12 hours
* Synchronize template against default one in pynsist 2.4
* Clean config of scheduled task
* Install only in AllUsers mode
* Add comments
* Remove the logic of single user install
* Get integration tests working on python 3.8
* Run unit tests on py38
* Update coveragercs to use coverage 4.5+ format
* remove line added to tox.ini
* update changelog
* xenial is the new travis default; no need to specify in .travis.yml
Fixes#7212
This PR forbid os.stat and os.fstat, and fix or provide alternatives to avoid its usage in certbot outside of certbot.compat.filesystem.
* Reimplement private key mode propagation
* Remove other os.stat
* Remove last call of os.stat in certbot package
* Forbid stat and fstat
* Implement mode comparison checks
* Add unit tests
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Handle case where multiple ace concerns a given SID in has_min_permissions
* Add a new test scenario
* Add a simple test for has_same_ownership
* Fix name function
* Add a comment explaining an ACE structure
* Move a test in its dedicated class
* Improve a message error
* Calculate has_min_permission result using effective permission rights to be more generic.
* Change an exception message
* Add comments, avoid to skip a test.
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Find OpenSSL version
* Create and update various config files
* Update logic to use new version constraints
* SSL_OPTIONS_HASHES_NEW and SSL_OPTIONS_HASHES_MEDIUM were just being used for testing, and maintaining them is becoming untenable, so remove them.
* if we don't know the openssl version, we can't turn off session tickets
* add unit test for _get_openssl_version
* add unit tests
* placate lint
* Fix docs and tests and clean up code
* use python correctly
* update changelog
* Lint
* make comment a comment
This PR is the first step to create an official distribution channel of Certbot for Windows. It consists essentially in creating a proper Certbot Windows installer.
Usually distributing an application requires, in a way or another, to stabilize the application logic and its dependencies around a given version. On Windows, this usually takes the form of a freezed application, that vendors its dependencies into a single executable.
There are two well-known solutions to create an executable shipping a Python application on Windows: [py2exe](http://www.py2exe.org/) and [pyinstaller](https://www.pyinstaller.org/). However these solutions create self-executable `.EXE` files: you run the `.EXE` file that launches immediately the software.
This is not a end-user solution. Indeed when a Windows user wants to install a piece of software, he expects to find and download an installer. When run the installer would interface with Windows to setup configuration entries in the Registry, update the environment variable, add shortcuts in the Start Menu, and declare a uninstaller entry into the Uninstaller Manager. Quite similarly, this is what you would get from a `.deb` or `.rpm` package.
A solution that builds proper installers is [pynsis](https://pynsist.readthedocs.io/en/latest/). It is a Python project that constructs installers for Python software using [NSIS](https://sourceforge.net/projects/nsis/), the most known free Windows installer builder solution.
This PR uses pynsist to build a Windows installer. The Python script to launch the installer build is `.\windows-installer\construct.py`. Once finished, the installer is located in `.\windows-installer\build\nsis`.
This installer will do the following operations during the installation:
* copy in the install path a full python distribution used exclusively for Certbot
* copy all Python requirements gathered from the `setup.py` of relevant certbot projects
* copy `certbot` and `acme`
* pre-build python binary assets
* register the existence of the application correctly in Windows Registry
* prepare a procedure to uninstall Certbot
* and of course, expose `certbot` executable to the Windows command line, like on Linux, to be able to launch it as any CLI application from Batch or Powershell
This installer support updates: downloading a new version of it and running it on a Windows with existing installation of Certbot will replace it with the new version.
Future capabilities not included in this PR:
* auto-update of Certbot when a new release is available
* online documentation for Windows
* register a scheduled task for certificate renewal
* installer distribution (continuous deployment + distribution channels)
* method to check the downloaded installer is untampered
* Setup config
* Fix shortcut
* Various improvments
* Update windows-installer/construct.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Split into several method
* Change installer name
* Remove DNS plugins for now
* Add a comment about administrator privileges
* Update welcome
* Control python version
* Control bitness
* Update windows-installer/construct.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update windows-installer/construct.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update windows-installer/construct.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
Smallest possible fix for #7106
* Replace platform.linux_dependencies with distro.linux_dependencies
* run build.py
* Add minimum version of 1.0.1
* Pin back requests package
* Update changelog
This PR supersedes #7353.
It fixes the execution of nginx oldest tests when these tests are executed on top of the modifications made in #7337. This execution failure revealed the fact that in some cases, the wrong version of certbot logic was used during integration tests (namely the logic lying in the codebase of the branch built, instead of the logic from the version of certbot declared by certbot-nginx for instance).
I let you appreciate my inline comment for the explanation and the workaround.
Thanks a lot to @bmw who found this python/pytest madness.
You can see the oldest tests succeeding with the logic of #7337 + this PR here: https://travis-ci.com/certbot/certbot/builds/124816254
* Remove certbot root from PYTHONPATH during integration tests
* Add a biiiiig comment.
On Windows you can have several drives (`C:`, `D:`, ...), that is the roughly (really roughly) equivalent of mount points, since each drive is usually associated to a specific physical partition.
So you can have paths like `C:\one\path`, `D:\another\path`.
In parallel, `os.path.relpath(path, start='.')` calculates the relative path between the given `path` and a `start` path (current directory if not provided). In recent versions of Python, `os.path.relpath` will fail if `path` and `start` are not on the same drive, because a relative path between two paths like `C:\one\path`, `D:\another\path` is not possible.
In saw unit tests failing because of this in two locations. This occurs when the certbot codebase that is tested is on a given drive (like `D:`) while the default temporary directory used by `tempfile` is on another drive (most of the time located in `C:` drive).
This PR fixes that.
