lib/libssh2
1
0
Fork 0
mirror of https://github.com/libssh2/libssh2.git synced 2025-11-03 22:13:11 +03:00
Code Activity

tests: workaround for intermittent first test failures (#832)

Browse Source 
Flakiness got continously worse these last days. It didn't seem related
to recent commits. Flakiness also picked up in GitHub CI runs, something
rarely seen before. Manual restart consistently fixed them.

The repeating pattern was the _first_ test (`test_hostkey`) failing,
with `libssh2_session_handshake failed (-13): Failed getting banner`.
Failures came after a lengthy wait, suggesting a timeout.

I then reversed the order of the first two tests, and it turned out that
the _first_ test failed again (`test_hostkey_hash`). Also pointing to a
timeout issue.

Then I added a dummy test to "warm up" whatever needs warming up in the
layers of CI + Docker + ssh server and their interconnects. This helped,
and GitHub CI tests run without failure right for the first time.
AppVeyor CI also improved a little.

This patch adds a new first test called `test_warmup`, that creates a
new libssh2 session, and exits with success even if that attempt failed.

A stop-gap solution at best, and there is no guarantee it will continue
to fix this or similar future issues, but it's also untenable to have
almost every CI run fail for intermittent reasons.

In some [1] cases [2] it's not the first test failing intermittently.
That's a different issue, and this patch doesn't fix it.

[1] #804
[2] https://ci.appveyor.com/project/libssh2org/libssh2/builds/46440828/job/8rej6cq6itg7vc4w#L500
This commit is contained in:
Viktor Szakats
2023-03-10 00:53:55 +01:00
committed by GitHub
parent ae90a35d15
commit 40ac6b230a
3 changed files with 34 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
tests/CMakeLists.txt
View File

@@ -115,6 +115,7 @@ if(CRYPTO_BACKEND STREQUAL "mbedTLS" OR NOT CRYPTO_BACKEND)
endif()
set(TESTS
warmup
hostkey
hostkey_hash
password_auth_succeeds_with_correct_credentials
@@ -165,7 +166,11 @@ target_include_directories(runner PRIVATE "${CMAKE_CURRENT_BINARY_DIR}")
foreach(test ${TESTS})
add_executable(test_${test} test_${test}.c)
target_link_libraries(test_${test} libssh2 runner ${LIBRARIES})
if(TESTS STREQUAL "warmup")
target_link_libraries(test_${test} libssh2 ${LIBRARIES})
else()
target_link_libraries(test_${test} libssh2 runner ${LIBRARIES})
endif()
target_include_directories(test_${test} PRIVATE "${CMAKE_CURRENT_BINARY_DIR}")
list(APPEND TEST_TARGETS test_${test})
add_definitions(-DFIXTURE_WORKDIR="${CMAKE_CURRENT_SOURCE_DIR}")

1
tests/Makefile.am
View File

@@ -12,6 +12,7 @@ check_PROGRAMS += ssh2
endif
INTEGRATION_TESTS = \
test_warmup \
test_agent_forward_succeeds \
test_hostkey \
test_hostkey_hash \

27
tests/test_warmup.c Normal file
View File

@@ -0,0 +1,27 @@
/* Warm-up test. Always return 0.
Workaround for CI/docker/etc flakiness on the first run. */
#include "session_fixture.h"
#include "runner.h"
#include <libssh2.h>
#include <stdio.h>
int main(void)
{
LIBSSH2_SESSION *session = start_session_fixture();
if(session != NULL) {
size_t len = 0;
int type = 0;
const char *hostkey = libssh2_session_hostkey(session, &len, &type);
(void)hostkey;
fprintf(stdout,
"libssh2_session_hostkey returned len, type: %d, %d\n",
(int)len, type);
}
stop_session_fixture();
return 0;
}