TL;DR: Sync test builds between autotools and CMake. Sync sshd
configuration between Docker and non-Docker fixtures. Bump up
sshd_config for recent OpenSSH releases.
This also opens up the path to have non-Docker tests that use a
local sshd process. Though sshd is practically unusable on Windows
CI machines out of the box, so this will need further efforts.
Details:
- cmake: run sshd fixture test just like autotool did already.
- sync tests and their order between autotools and CMake.
It makes `test_aa_warmup` the first test with both.
- cmake: load test lists from `Makefile.am`.
Needed to update the loader to throw away certain lines to keep the
converted output conform CMake syntax. Using regexp might be an
alternative way of doing this, but couldn't make it work.
- cmake: use the official way to configure test environment variables.
Switch to syntax that's extendable.
- cmake: allow to run the same test both under Docker and sshd fixture.
Useful for testing the sshd fixture runner, or how the same test
behaves in each fixture.
- update test fixture to read the username from `USER` envvar instead of
using the Dockfile-specific hardwired one, when running outside Docker.
- rework `ssh2.sh` into `sshd_fixture.sh`, to:
- allow running any tests (not just `test_ssh2`).
- configure Docker tests for running outside Docker.
- fixup `SSHD` path when running on Windows (e.g. in AppVeyor CI).
Fixes: `sshd re-exec requires execution with an absolute path`
- allow overriding `PUBKEY` and `PRIVKEY` envvars.
- allow overriding `ssh_config` via `SSHD_FIXTURE_CONFIG`.
- prepare support for running multiple tests via sshd_fixture.
Add a TAP runner for autotools and extend CMake logic. The TAP runner
loads the test list from `Makefile.am`.
Notice however that on Windows, `sshd_fixture.sh` is very flaky with
GitHub Actions. And consistently broken for subsequent tests in
AppVeyor CI:
'libssh2_session_handshake failed (-43): Failed getting banner'
Another way to try is a single sshd instance serving all tests.
For CMake this would probably mean using an external script.
- ed25519 test keys were identical for auth and host. Regenerate the
auth keypair to make them distinct.
- sync the sshd environment between Docker and sshd_fixture.
- use common via `openssh_server/sshd_config`.
- accept same auth keys.
- offer the same host keys.
- sync TrustedUserCAKeys.
- delete now unused keypairs: `etc/host*`, `etc/user*`.
- bump up startup delay for Windows (randomly, to 5 secs, from 3).
- delete `UsePrivilegeSeparation no` to avoid deprecation warnings.
`command-line line 0: Deprecated option UsePrivilegeSeparation`
- delete `Protocol 2` to avoid deprecation warnings.
It has been the default since OpenSSH 3.0 (2001-11-06).
- delete `StrictModes no` (CI tests work without it, Docker tests
never used it).
- bump `Dockerfile` base image to `testing-slim` (from `bullseye-slim`).
It needed `sshd_config` updates to keep things working with
OpenSSH 9.2 (compared to bullseye's 8.4).
- replace `ChallengeResponseAuthentication` alias with
`KbdInteractiveAuthentication`.
The former is no longer present in default `sshd_config` since
OpenSSH 8.7 (2021-08-20). This broke the `Dockerfile` script.
The new name is documented since OpenSSH 4.9 (2008-03-31)
- add `PubkeyAcceptedKeyTypes +ssh-rsa,ssh-dss,ssh-rsa-cert-v01@openssh.com`
and `HostKeyAlgorithms +ssh-rsa`.
Original-patch-by: Eric van Gyzen (@vangyzen on github)
Fixes#691
There is a new name for `PubkeyAcceptedKeyTypes`:
`PubkeyAcceptedAlgorithms`.
It requires OpenSSH 8.5 (2021-03-03) and breaks some envs so we're
not using it just yet.
- drop `rijndael-cbc@lysator.liu.se` tests and references from config.
This is a draft alias for `aes256-cbc`. No need to test it twice.
Also this alias is no longer recognized by OpenSSH 8.5 (2021-03-03).
- update `mansyntax.sh` and `sshd_fixture.sh` to not rely on `srcdir`.
Hopefully this works with out-of-tree builds.
- fix `test_read_algos.test` to honor CRLF EOLs in their inputs
(necessary when running on Windows.)
- fix `test_read_algos.test` to honor `EXEEXT`. Might be useful when
running tests under cross-builds?
- `test_ssh2.c`:
- use libssh2 API to set blocking mode. This makes it support all
platforms.
- adapt socket open timeout logic from `openssh_fixture.c`.
Sadly this did not help fix flakiness on GHA Windows.
- tests: delete unused C headers and variable initialization.
- delete unused test files: `sshd_fixture.sh.in`, `sshdwrap`,
`etc/sshd_config`.
Ref: cf80f2f4b5
- autotools: delete stray `.c` test sources from `EXTRA_DIST` in tests.
- `tests/.gitignore`: drop two stray tests.
- autotools: fix passing `SSHD` containing space (Windows needs this).
- autotools: sort `EXTRA_DIST` in tests.
- cmake: fix to add `test_ssh2` to `TEST_TARGETS`.
- fix `authorized_key` order in `tests/gen_keys.sh`.
- silence shellcheck warning in `ci/checksrc.sh`.
- set `SSHD` for autotools on GitHub Actions Windows. [skipped]
Auto-detection doesn't work (maybe because sshd is installed via
Git for Windows and we're using MSYS2's shell.)
It enables running sshd fixture (non-Docker) tests in these jobs.
I did not include this in the final patch due to flakiness:
```
Connection to 127.0.0.1:4711 attempt #0 failed: retrying...
Connection to 127.0.0.1:4711 attempt #1 failed: retrying...
Connection to 127.0.0.1:4711 attempt #2 failed: retrying...
Failure establishing SSH session: -43
```
Can be enabled with:
`export SSHD='C:/Program Files/Git/usr/bin/sshd.exe'`
Closes#996
Works like the `FIXTURE_TRACE_ALL` envvar, but enables full trace for
the connection phase only.
Also fix a possible NULL deref with `FIXTURE_TRACE_ALL` and a failed
`libssh2_session_init_ex()`.
Tested in #979
- rename tests to have more succint names and a more useful natural
order.
- rename `simple` and `ssh2` in tests to have the `test_` prefix.
This avoids a name collisions with `ssh2` in examples.
- cmake: drop the `example-` prefix for generated examples.
Bringing their names in sync with other build tools, like autotools.
- move common auth test code into the fixture and simplify tests by
using that.
- move feature guards from CMake to preprocessor for auth tests.
Now it works with all build tools and it's easier to keep it in sync
with the lib itself.
For this we need to include `libssh2_priv.h` in tests, which in turn
needs tweaking on the trick we use to suppress extra MSVS warnings
when building tests and examples.
- move mbedTLS blocklist for crypto tests from CMake to the test
fixture.
- add ed25519 hostkey tests to `test_hostkey` and `test_hostkey_hash`.
- add shell script to regenerate all test keys used for our tests.
- alpha-sort tests.
- rename `signed_*` keys to begin with `key` like the rest of the keys
do.
- whitespace fixes.
Closes#969
- fix skip auth if `userauthlist` is NULL.
Closes#836 (Reported-by: @sudipm-mukherjee on github)
- fix most silenced `checksrc` warnings.
- sync examples/tests code between each other.
(output messages, error handling, declaration order, comments)
- stop including unnecessary headers.
- always deinitialize in case of error.
- drop some redundant variables.
- add error handling where missing.
- show more error codes.
- switch `perror()` to `fprintf()`.
- fix some `printf()`s to be `fprintf()`.
- formatting.
Closes#960
Implement picky warnings with clang in autotools. Extend picky gcc
warnings, sync them between build tools and compilers and greatly
speed up detection in CMake.
- autotools: enable clang compiler warnings with `--enable-debug`.
- autotools: enable more gcc compiler warnings with `--enable-debug`.
- autotools/cmake: sync compiler warning options between gcc and clang.
- sync compiler warning options between autotools and cmake.
- cmake: reduce option-checks to speed up the detection phase.
Bring them down to 3 (from 35). Leaving some checks to keep the
CMake logic alive and for an easy way to add new options.
clang 3.0 (2011-11-29) and gcc 2.95 (1999-07-31) now required.
- autotools logic copied from curl, with these differences:
- delete `-Wimplicit-fallthrough=4` due to a false positive.
- reduce `-Wformat-truncation=2` to `1` due to a false positive.
- simplify MinGW detection for `-Wno-pedantic-ms-format`.
- cmake: show enabled picky compiler options (like autotools).
- cmake: do compile `tests/simple.c` and `tests/ssh2.c`.
- fix new compiler warnings.
- `tests/CMakeLists.txt`: fix indentation.
Original source of autotools logic:
- a8fbdb461c/acinclude.m4
- a8fbdb461c/m4/curl-compilers.m4
Notice that the autotools implementation considers Apple clang as
legacy clang 3.7. CMake detection works more accurately, at the same
time more error-prone and difficult to update due to the sparsely
documented nature of Apple clang option evolution.
Closes#952
- introduce the concept of a project level setup header
`src/libssh2_setup.h`, that is used by `src`, `example` and `tests`
alike. Move there all common platform/compiler configuration from
`src/libssh2_priv.h`, individual sources and `CMakeFiles.txt` files.
Also move there our hand-crafted (= not auto-generated by CMake or
autotools) configuration `win32/libssh2-config.h`.
- `win32` directory is empty now, delete it.
- `Makefile.mk`: adapt to the above. Build-directory is the target
triplet, or any custom name set via `BLD_DIR`.
- sync header path order between build systems:
build/src -> source/src -> source/include
- delete redundant references to `windows.h`, `winsock2.h`,
`ws2tcpip.h`.
- delete unnecessary #includes, update order (`libssh2_setup.h` first,
`winsock2.h` first), simplify where possible.
This makes the code warning-free without `WIN32_LEAN_AND_MEAN`.
At the same time this patch applies this macro globally, to avoid
header bloat.
- example: add missing *nix header guards.
- example: fix misindented `HAVE_UNISTD_H` `#ifdef`s.
- set `WIN32` with all build-tools.
- set `HAVE_SYS_PARAM_H` in the hand-crafted config for MinGW.
To match auto-detection.
- move a source-specific macro to `misc.c` from `libssh2_priv.h`.
See the PR's individual commits for step-by-step updates.
Closes#932
`libssh2.h` required `winsock2.h` for `_WIN32` since
81d53de4dc (2011-06-04).
Apply that to the whole codebase. This makes it unnecessary to detect
`HAVE_WINSOCK2_H` and allows to drop all its uses.
Completes TODO from b66d7317ca
TODO: Straighten out the use a mixture of `HAVE_WINDOWS_H`,
`WIN32`, `_WIN32` to detect Windows.
Fix or silence all C compiler warnings discovered with (or without)
`PICKY_COMPILER=ON` (in CMake). This means all warnings showing up in
CI (gcc, clang, MSVS 2013/2015), in local tests on macOS (clang 14) and
Windows cross-builds using gcc (12) and llvm/clang (14/15).
Also fix the expression `nread -= nread` in `sftp_RW_nonblock.c`.
Cherry-picked from: #846Closes#861
- cmake: always link `ws2_32` on Windows. Also add it to `libssh2.pc`.
Fixes#745
- agent: fix gcc compiler warning:
`src/agent.c:296:35: warning: 'snprintf' output truncated before the last format character [-Wformat-truncation=]`
- autotools: fix `EVP_aes_128_ctr` detection with binutils `ld`
The prerequisite for a successful detection is setting
`LIBS=-lbcrypt` if the chosen openssl-compatible library requires
it, e.g. libressl, or quictls/openssl built with
`-DUSE_BCRYPTGENRANDOM`.
With llvm `lld`, detection works out of the box. With binutils `ld`,
it does not. The reason is `ld`s world-famous pickiness with lib
order.
To fix it, we pass all custom libs before and after the TLS libs.
This ugly hack makes `ld` happy and detection succeed.
- agent: fix Windows-specific warning:
`src/agent.c:318:10: warning: implicit conversion loses integer precision: 'LRESULT' (aka 'long long') to 'int' [-Wshorten-64-to-32]`
- src: fix llvm/clang compiler warning:
`src/libssh2_priv.h:987:28: warning: variadic macros are a C99 feature [-Wvariadic-macros]`
- src: support `inline` with `__GNUC__` (llvm/clang and gcc), fixing:
```
src/libssh2_priv.h:990:8: warning: extension used [-Wlanguage-extension-token]
static inline void
^
```
- blowfish: support `inline` keyword with MSVC.
Also switch to `__inline__` (from `__inline`) for `__GNUC__`:
https://gcc.gnu.org/onlinedocs/gcc/Inline.htmlhttps://clang.llvm.org/docs/UsersManual.html#differences-between-various-standard-modes
- example/test: fix MSVC compiler warnings:
- `example\direct_tcpip.c(209): warning C4244: 'function': conversion from 'unsigned int' to 'u_short', possible loss of data`
- `tests\session_fixture.c(96): warning C4013: 'getcwd' undefined; assuming extern returning int`
- `tests\session_fixture.c(100): warning C4013: 'chdir' undefined; assuming extern returning int`
- delete unused macros:
- `HAVE_SOCKET`
- `HAVE_INET_ADDR`
- `NEED_LIB_NSL`
- `NEED_LIB_SOCKET`
- `HAVE_NTSTATUS_H`
- `HAVE_NTDEF_H`
- build: delete stale zlib/openssl version numbers from path defaults.
- cmake: convert tabs to spaces, add newline at EOFs.
Closes#811
Improve tests added in 7487dcf4b4 by
running `test_read` commands directly. This makes external shell/batch
files unnecessary, and is friendlier with cross-builds and when run
from non-default shells, like MSYS2.
Also extend CRYPT/MAC test error messages with the CRYPT/MAC name.
External runner shell scripts kept for future use.
Closes#814
- in `hostkey.c` check the result of `libssh2_sha256_init()` and
`libssh2_sha512_init()` calls. This avoid the warning that we're
ignoring the return values.
- fix code using `int` (or `SOCKET`) for sockets. Use libssh2's
dedicated `libssh2_socket_t` and `LIBSSH2_INVALID_SOCKET` instead.
- fix compiler warnings due to `STATUS_*` macro redefinitions between
`ntstatus.h` / `winnt.h`. Solve it by manually defining the single
`STATUS` value we need from `ntstatus.h` and stop including the whole
header.
Fixes#733
- improve Windows UWP/WinRT builds by detecting it with code copied
from the curl project. Then excluding problematic libssh2 parts
according to PR by Dmitry Kostjučenko.
Fixes#734
- always use `SecureZeroMemory()` on Windows.
We can tweak this if not found or not inlined by a C compiler which
we otherwise support. Same if it causes issues with UWP apps.
Ref: https://learn.microsoft.com/en-us/previous-versions/windows/desktop/legacy/aa366877(v=vs.85)
Ref: https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-rtlsecurezeromemory
- always enable `LIBSSH2_CLEAR_MEMORY` on Windows. CMake and
curl-for-win builds already did that. Delete `SecureZeroMemory()`
detection from autotools' WinCNG backend logic, that this
setting used to depend on.
TODO: Enable it for all platforms in a separate PR.
TODO: For clearing buffers in WinCNG, call `_libssh2_explicit_zero()`,
insead of a local function or explicit `SecureZeroMemory()`.
- Makefile.inc: move `os400qc3.h` to `HEADERS`. This fixes
compilation on non-unixy platforms. Recent regression.
- `libssh2.rc`: replace copyright with plain ASCII, as in curl.
Ref: curl/curl@1ca62bb
Ref: curl/curl#7765
Ref: curl/curl#7776
- CMake fixes and improvements:
- enable warnings with llvm/clang.
- enable more comprehensive warnings with gcc and llvm/clang.
Logic copied from curl:
233810bb5f/CMakeLists.txt (L131-L148)
- fix `Policy CMP0080` CMake warning by deleting that reference.
- add `ENABLE_WERROR` (default: `OFF`) option. Ported from curl.
- add `PICKY_COMPILER` (default: `ON`) option, as known from curl.
It controls both the newly added picky warnings for llvm/clang and
gcc, and also the pre-existing ones for MSVC.
- `win32/GNUmakefile` fixes and improvements:
- delete `_AMD64_` and add missing `-m64` for x64 builds under test.
- add support for `ARCH=custom`.
It disables hardcoded Intel 64-bit and Intel 32-bit options,
allowing ARM64 builds.
- add support for `LIBSSH2_RCFLAG_EXTRAS`.
To pass custom options to windres, e.g. in ARM64 builds.
- add support for `LIBSSH2_RC`. To override `windres`.
- delete support for Metrowerks C. Last released in 2004.
- `win32/libssh2_config.h`: delete unnecessary socket #includes
`src/libssh2_priv.h` includes `winsock2.h` and `ws2tcpip.h` further
down the line, triggered by `HAVE_WINSOCK2_H`.
`mswsock.h` does not seem to be necessary anymore.
Double-including these (before `windows.h`) caused compiler failures
when building against BoringSSL and warnings with LibreSSL. We could
work this around by passing `-DNOCRYPT`. Deleting the duplicates
fixes these issues.
Timeline:
2013: c910cd382d deleted `mswsock.h` from `src/libssh2_priv.h`
2008: 8c43bc52b1 added `winsock2.h` and `ws2tcpip.h` to `src/libssh2_priv.h`
2005: dc4bb1af96 added the now deleted #includes
- delete or replace `LIBSSH2_WIN32` with `WIN32`.
- replace hand-rolled `HAVE_WINDOWS_H` macro with `WIN32`. Also delete
its detections/definitions.
- delete unused `LIBSSH2_DARWIN` macro.
- delete unused `writev()` Windows implementation
There is no reference to `writev()` since 2007-02-02, commit
9d55db6501.
- fix a bunch of MSVC / llvm/clang / gcc compiler warnings:
- `warning C4100: '...': unreferenced formal parameter`
- using value of undefined PP macro `LIBSSH2DEBUG`
- missing void from function definition
- `if()` block missing in non-debug builds
- unreferenced variable in non-debug builds
- `warning: must specify at least one argument for '...' parameter of variadic macro [-Wgnu-zero-variadic-macro-arguments]`
in `_libssh2_debug()`
- `warning C4295: 'ciphertext' : array is too small to include a terminating null character`
- `warning C4706: assignment within conditional expression`
- `warning C4996: 'inet_addr': Use inet_pton() or InetPton() instead or
define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings`
By suppressning it. Would be best to use inet_pton() as suggested.
On Windows this needs Vista though.
- `warning C4152: nonstandard extension, function/data pointer conversion in expression`
(silenced locally)
- `warning C4068: unknown pragma`
Ref: https://ci.appveyor.com/project/libssh2org/libssh2/builds/46354480/job/j7d0m34qgq8rag5wCloses#808
Two warnings (in tests & examples) in particular would cause problems:
bad format causing invalid data output or a bad chdir due to out of
scope buffer use.
* tests: Remove if-pyramids
* tests: Switch run_command arguments
* tests: Make run_command a vararg function
* tests: Xcode doesn't obey CMake's test working directory
* openssl: move manual AES-CTR cipher into crypto init
* cmake: Move our include dir before all other include paths
This introduces a test suite for libssh2. It runs OpenSSH in a Docker
container because that works well on Windows (via docker-machine) as
well as Linux. Presumably it works on Mac too with docker-machine, but
I've not tested that.
Because the test suite is docker-machine aware, you can also run it
against a cloud provider, for more realistic network testing, by setting
your cloud provider as your active docker machine. The Appveyor CI setup
in this commit does that because Appveyor doesn't support docker
locally.
