mirror of
https://github.com/Mbed-TLS/mbedtls.git
synced 2025-07-28 00:21:48 +03:00
Merge remote-tracking branch 'myfork-public/development' into merge-crypto-development-20191115
First deal with deleted files. * Files deleted by us: keep them deleted. * Files deleted by them, whether modified by us or not: keep our version. ``` git rm $(git status -s | sed -n 's/^DU //p') git reset -- $(git status -s | sed -n 's/^D //p') git checkout -- $(git status -s | sed -n 's/^ D //p') git add -- $(git status -s | sed -n 's/^UD //p') ``` Individual files with conflicts: * `3rdparty/everest/library/Hacl_Curve25519_joined.c`: spurious conflict because git mistakenly identified this file as a rename. Keep our version. * `README.md`: conflict due to their change in a paragraph that doesn't exist in our version. Keep our version of this paragraph. * `docs/architecture/Makefile`: near-identical additions. Adapt the definition of `all_markdown` and include the clean target. * `doxygen/input/docs_mainpage.h`: conflict in the version number. Keep our version number. * `include/mbedtls/config.h`: two delete/modify conflicts. Keep the removed chunks out. * `library/CMakeLists.txt`: discard all their changes as they are not relevant. * `library/Makefile`: * Discard the added chunk about the crypto submodule starting with `INCLUDING_FROM_MBEDTLS:=1`. * delete/modify: keep the removed chunk out. * library build: This is almost delete/modify. Their changes are mostly not applicable. Do keep the `libmbedcrypto.$(DLEXT): | libmbedcrypto.a` order dependency. * `.c.o`: `-o` was added on both sides but in a different place. Change to their place. * `library/error.c`: to be regenerated. * `library/version_features.c`: to be regenerated. * `programs/Makefile`: Most of the changes are not relevant. The one relevant change is in the `clean` target for Windows; adapt it by removing `/S` from our version. * `programs/test/query_config.c`: to be regenerated. * `scripts/config.py`: added in parallel on both sides. Keep our version. * `scripts/footprint.sh`: parallel changes. Keep our version. * `scripts/generate_visualc_files.pl`: one delete/modify conflict. Keep the removed chunks out. * `tests/Makefile`: discard all of their changes. * `tests/scripts/all.sh`: * `pre_initialize_variables` add `append_outcome`: add it. * `pre_initialize_variables` add `ASAN_CFLAGS`: already there, keep our version. * `pre_parse_command_line` add `--no-append-outcome`: add it. * `pre_parse_command_line` add `--outcome-file`: add it. * `pre_print_configuration`: add `MBEDTLS_TEST_OUTCOME_FILE`. * Several changes in SSL-specific components: keep our version without them. * Several changes where `config.pl` was changed to `config.py` and there was an adjacent difference: keep our version. * Changes regarding the inclusion of `MBEDTLS_MEMORY_xxx`: ignore them here, they will be normalized in a subsequent commit. * `component_test_full_cmake_gcc_asan`: add it without the TLS tests. * `component_test_no_use_psa_crypto_full_cmake_asan`: keep the fixed `msg`, discard other changes. * `component_test_memory_buffer_allocator_backtrace`, `component_test_memory_buffer_allocator`: add them without the TLS tests. * `component_test_m32_everest`: added in parallel on both sides. Keep our version. * `tests/scripts/check-names.sh`, `tests/scripts/list-enum-consts.pl`, `tests/scripts/list-identifiers.sh`, ``tests/scripts/list-macros.sh`: discard all of their changes. * `tests/scripts/test-ref-configs.pl`: the change in the conflict is not relevant, so keep our version there. * `visualc/VS2010/*.vcxproj`: to be regenerated. Regenerate files: ``` scripts/generate_visualc_files.pl git add visualc/VS2010/*.vcxproj scripts/generate_errors.pl git add library/error.c scripts/generate_features.pl git add library/version_features.c scripts/generate_query_config.pl git add programs/test/query_config.c ``` Rejected changes in non-conflicting files: * `CMakeLists.txt`: discard their addition which has already been side-ported. * `doxygen/mbedtls.doxyfile`: keep the version number change. Discard the changes related to `../crypto` paths. Keep the following changes after examination: * `.travis.yml`: all of their changes are relevant. * `include/mbedtls/error.h`: do keep their changes. Even though Crypto doesn't use TLS errors, it must not encroach on TLS's allocated numbers. * `tests/scripts/check-test-cases.py`: keep the code dealing with `ssl-opt.sh`. It works correctly when the file is not present.
This commit is contained in:
58
docs/architecture/testing/test-framework.md
Normal file
58
docs/architecture/testing/test-framework.md
Normal file
@ -0,0 +1,58 @@
|
||||
# Mbed TLS test framework
|
||||
|
||||
This document is an overview of the Mbed TLS test framework and test tools.
|
||||
|
||||
This document is incomplete. You can help by expanding it.
|
||||
|
||||
## Unit tests
|
||||
|
||||
See <https://tls.mbed.org/kb/development/test_suites>
|
||||
|
||||
### Unit test descriptions
|
||||
|
||||
Each test case has a description which succinctly describes for a human audience what the test does. The first non-comment line of each paragraph in a `.data` file is the test description. The following rules and guidelines apply:
|
||||
|
||||
* Test descriptions may not contain semicolons, line breaks and other control characters, or non-ASCII characters. <br>
|
||||
Rationale: keep the tools that process test descriptions (`generate_test_code.py`, [outcome file](#outcome-file) tools) simple.
|
||||
* Test descriptions must be unique within a `.data` file. If you can't think of a better description, the convention is to append `#1`, `#2`, etc. <br>
|
||||
Rationale: make it easy to relate a failure log to the test data. Avoid confusion between cases in the [outcome file](#outcome-file).
|
||||
* Test descriptions should be a maximum of **66 characters**. <br>
|
||||
Rationale: 66 characters is what our various tools assume (leaving room for 14 more characters on an 80-column line). Longer descriptions may be truncated or may break a visual alignment. <br>
|
||||
We have a lot of test cases with longer descriptions, but they should be avoided. At least please make sure that the first 66 characters describe the test uniquely.
|
||||
* Make the description descriptive. “foo: x=2, y=4” is more descriptive than “foo #2”. “foo: 0<x<y, both even” is even better if these inequalities and parities are why this particular test data was chosen.
|
||||
* Avoid changing the description of an existing test case without a good reason. This breaks the tracking of failures across CI runs, since this tracking is based on the descriptions.
|
||||
|
||||
`tests/scripts/check-test-cases.py` enforces some rules and warns if some guidelines are violated.
|
||||
|
||||
## TLS tests
|
||||
|
||||
### SSL extension tests
|
||||
|
||||
#### SSL test case descriptions
|
||||
|
||||
Each test case in `ssl-opt.sh` has a description which succinctly describes for a human audience what the test does. The test description is the first parameter to `run_tests`.
|
||||
|
||||
The same rules and guidelines apply as for [unit test descriptions](#unit-test-descriptions). In addition, the description must be written on the same line as `run_test`, in double quotes, for the sake of `check-test-cases.py`.
|
||||
|
||||
## Running tests
|
||||
|
||||
### Outcome file
|
||||
|
||||
#### Generating an outcome file
|
||||
|
||||
Unit tests and `ssl-opt.sh` record the outcome of each test case in a **test outcome file**. This feature is enabled if the environment variable `MBEDTLS_TEST_OUTCOME_FILE` is set. Set it to the path of the desired file.
|
||||
|
||||
If you run `all.sh --outcome-file test-outcome.csv`, this collects the outcome of all the test cases in `test-outcome.csv`.
|
||||
|
||||
#### Outcome file format
|
||||
|
||||
The outcome file is in a CSV format using `;` (semicolon) as the delimiter and no quoting. This means that fields may not contain newlines or semicolons. There is no title line.
|
||||
|
||||
The outcome file has 6 fields:
|
||||
|
||||
* **Platform**: a description of the platform, e.g. `Linux-x86_64` or `Linux-x86_64-gcc7-msan`.
|
||||
* **Configuration**: a unique description of the configuration (`config.h`).
|
||||
* **Test suite**: `test_suite_xxx` or `ssl-opt`.
|
||||
* **Test case**: the description of the test case.
|
||||
* **Result**: one of `PASS`, `SKIP` or `FAIL`.
|
||||
* **Cause**: more information explaining the result.
|
Reference in New Issue
Block a user