https://code.forgejo.org/forgejo/runner/pulls/1585 introduced the environment variable `TEST_FEATURES` for switching tests related to particular feature on and off. For example, with `TEST_FEATURES=""`, tests that require Docker would be skipped. Unfortunately, that technique has multiple problems. Environment variables might not be passed on to child processes without it being clearly visible. Also, the usage differs by platform. On top of that, informational messages that are only related to testing were printed by the release binary.
Instead of the environment variable `TEST_FEATURES`, the command line flag `-features` is introduced:
```
$ go test ./... -args -features "docker,lxc"
```
That runs all tests that require the presence of Docker and LXC. It is also the default.
If tests that require Docker should be skipped, run:
```
$ go test ./... -args -features "lxc"
```
To exclude all tests that require the presence of a particular feature, run:
```
$ go test ./... -args -features "-"
```
At the moment, `-features` cannot be controlled when using make targets. That means that `make integration-test` runs all tests that require Docker or LXC.
Supersedes https://code.forgejo.org/forgejo/runner/pulls/1588.
<!--start release-notes-assistant-->
<!--URL:https://code.forgejo.org/forgejo/runner-->
- other
- [PR](https://code.forgejo.org/forgejo/runner/pulls/1591): <!--number 1591 --><!--line 0 --><!--description dGVzdDogdXNlIGZsYWdzIGluc3RlYWQgb2YgZW52IHZhcmlhYmxlcyBmb3Igc2tpcHBpbmcgdGVzdHM=-->test: use flags instead of env variables for skipping tests<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://code.forgejo.org/forgejo/runner/pulls/1591
Reviewed-by: Mathieu Fenniak <mfenniak@noreply.code.forgejo.org>
Tests involving Docker are skipped when `runtime.GOOS` is not `linux`. That is unnecessarily rigid and hinders development. There are other operating systems that can run Docker. It makes it impossible to run integration tests on Linux without Docker. Verifying that all tests that require Docker are skipped without Docker being present is impossible on Linux due to that, too.
Instead of relying on `runtime.GOOS`, developers can control whether tests involving Docker should be run with the help of the environment variable `TEST_FEATURES`.
If the variable is undefined, tests involving Docker will be run, because they are enabled by default:
```
$ make integration-test
```
That is the same as:
```
$ TEST_FEATURES="docker" make integration-test
```
If the variable does not contain the string `docker`, tests involving Docker will be **skipped**:
```
$ TEST_FEATURES="" make integration-test
```
If a test requires Docker, use `testutils.RequireTestFeatures(t, testutils.TestFeatureDocker)` to indicate that.
<!--start release-notes-assistant-->
<!--URL:https://code.forgejo.org/forgejo/runner-->
- other
- [PR](https://code.forgejo.org/forgejo/runner/pulls/1585): <!--number 1585 --><!--line 0 --><!--description dGVzdDogdXNlIGVudmlyb25tZW50IHZhcmlhYmxlIGZvciBza2lwcGluZyBEb2NrZXIgdGVzdHM=-->test: use environment variable for skipping Docker tests<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://code.forgejo.org/forgejo/runner/pulls/1585
Reviewed-by: Mathieu Fenniak <mfenniak@noreply.code.forgejo.org>
<!--start release-notes-assistant-->
<!--URL:https://code.forgejo.org/forgejo/runner-->
- features
- [PR](https://code.forgejo.org/forgejo/runner/pulls/757): <!--number 757 --><!--line 0 --><!--description ZmVhdDogdGhlIG5ldyBgZm9yZ2Vqby1ydW5uZXIgdmFsaWRhdGVgIGNvbW1hbmQgY2FuIGJlIHVzZWQgdG8gdmVyaWZ5IGlmIGFuIGFjdGlvbiBvciBhIHdvcmtmbG93IGlzIGNvbmZvcm1hbnQgd2l0aCB0aGUgZXhwZWN0ZWQgc2NoZW1hLiBgZm9yZ2Vqby1ydW5uZXIgdmFsaWRhdGUgLS1yZXBvc2l0b3J5IGh0dHBzOi8vZXhhbXBsZS5jb20vbXkvcmVwb3NpdG9yeWAgd2lsbCB2YWxpZGF0ZSBhbGwgdGhlIHdvcmtmbG93cyBhbmQgYWN0aW9ucyBhIEdpdCByZXBvc2l0b3J5IGNvbnRhaW5zLiBBbHRlcm5hdGl2ZWx5ICBgZm9yZ2Vqby1ydW5uZXIgdmFsaWRhdGUgLS1wYXRoIG15YWN0aW9uL2FjdGlvbi55bWwgLS1hY3Rpb25gIG9yIGBmb3JnZWpvLXJ1bm5lciB2YWxpZGF0ZSAtLXBhdGggLmZvcmdlam8vd29ya2Zsb3dzL3Rlc3QueW1sIC0td29ya2Zsb3dgIGNhbiBiZSB1c2VkIHRvIHZhbGlkYXRlIGEgc2luZ2xlIGZpbGUuIEl0IGlzIHJlY29tbWVuZGVkIHRvIHVzZSB0aGVzZSBjb21tYW5kcyB0byB2ZXJpZnkgZXhpc3RpbmcgYWN0aW9ucyBhbmQgd29ya2Zsb3dzIHBhc3MgYmVmb3JlIHVwZ3JhZGluZyB0byBbRm9yZ2VqbyBydW5uZXIgdjguMC4wXShodHRwczovL2NvZGUuZm9yZ2Vqby5vcmcvZm9yZ2Vqby9ydW5uZXIvc3JjL2JyYW5jaC9tYWluL1JFTEVBU0UtTk9URVMubWQjOC0wLTApIG9yIGFib3ZlIHRvIG5vdCBkaXNydXB0IGV4aXN0aW5nIHdvcmtmbG93cy4=-->feat: the new `forgejo-runner validate` command can be used to verify if an action or a workflow is conformant with the expected schema. `forgejo-runner validate --repository https://example.com/my/repository` will validate all the workflows and actions a Git repository contains. Alternatively `forgejo-runner validate --path myaction/action.yml --action` or `forgejo-runner validate --path .forgejo/workflows/test.yml --workflow` can be used to validate a single file. It is recommended to use these commands to verify existing actions and workflows pass before upgrading to [Forgejo runner v8.0.0](https://code.forgejo.org/forgejo/runner/src/branch/main/RELEASE-NOTES.md#8-0-0) or above to not disrupt existing workflows.<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://code.forgejo.org/forgejo/runner/pulls/757
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
Co-authored-by: Earl Warren <contact@earl-warren.org>
Co-committed-by: Earl Warren <contact@earl-warren.org>
