ee3b3656ea
Apache test farm tests started failing last night due to a change in pyenv. See https://dev.azure.com/certbot/certbot/_build/results?buildId=3948&view=logs&j=f67c2a39-2c4f-5190-915f-6f32a7a4306f&t=96f0f394-f513-5158-f5e7-a26e55aeadbf&l=26943.
I managed to fix that in d94f20f8b7, however, the OSes the tests were failing on were Debian 9 and Ubuntu 16.04. [Debian 9 reached its end-of-life in July 2020](https://wiki.debian.org/DebianReleases) and [Ubuntu 16.04 reached its end of standard support in April 2021](https://wiki.ubuntu.com/Releases). As shown at the same links, Debian 9 still has support from the LTS team and Ubuntu 16.04 has ESM support. Do we still want to support either of these OSes?
If so, we can use the commit I linked in the first sentence of the last paragraph, but I think supporting the OSes through their standard support is good enough. The Certbot team has enough on their plate and especially when the OSes are so old that we can't even use their packaged version of Python anymore which complicates our tests, I think we can just drop support and move on.
I don't have a strong opinion here though so if someone else does, let me know what you'd like to see or make the PR yourself based on the changes in my linked commit and I'll merge it.
You can see the tests passing with this change at https://dev.azure.com/certbot/certbot/_build/results?buildId=3955&view=results.
* Remove apache tests on old OSes
* remove unused pyenv code
letstest
Simple AWS testfarm scripts for certbot client testing
- Launches EC2 instances with a given list of AMIs for different distros
- Copies certbot repo and puts it on the instances
- Runs certbot tests (bash scripts) on all of these
- Logs execution and success/fail for debugging
Notes
- Some AWS images, e.g. official CentOS and FreeBSD images require acceptance of user terms on the AWS marketplace website. This can't be automated.
- AWS EC2 has a default limit of 20 t2/t1 instances, if more are needed, they need to be requested via online webform.
Installation and configuration
This package is installed in the Certbot development environment that is created by following the instructions at https://certbot.eff.org/docs/contributing.html#running-a-local-copy-of-the-client.
After activating that virtual environment, you can then configure AWS credentials and create a key by running:
>aws configure --profile <profile name>
[interactive: enter secrets for IAM role]
>aws ec2 create-key-pair --profile <profile name> --key-name <key name> --query 'KeyMaterial' --output text > whatever/path/you/want.pem
Note: whatever you pick for <key name> will be shown to other users with AWS access.
When prompted for a default region name, enter: us-east-1.
Usage
To run tests, activate the virtual environment you created above and from this directory run:
>letstest targets/targets.yaml /path/to/your/key.pem <profile name> scripts/<test to run>
You can only run up to two tests at once. The following error is often indicative of there being too many AWS instances running on our account:
NameError: name 'instances' is not defined
If you see this, you can run the following command to shut down all running instances:
aws ec2 terminate-instances --profile <profile name> --instance-ids $(aws ec2 describe-instances --profile <profile name> | grep <key name> | cut -f8)
It will take a minute for these instances to shut down and become available again. Running this will invalidate any in progress tests.
A temporary directory whose name is output by the tests is also created with a log file from each instance of the test and a file named "results" containing the output above. The tests take quite a while to run.
Scripts
Example scripts are in the 'scripts' directory, these are just bash scripts that have a few parameters passed to them at runtime via environment variables. test_apache2.sh is a useful reference.
test_apache2 runs the dev venv and does local tests.
See:
- https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
- https://docs.aws.amazon.com/cli/latest/userguide/cli-ec2-keypairs.html
Main repos: