- cmake: extend workaround for linking a test with shared libssh2.
One of the tests uses internal libssh2 functions, and with CMake it
compiles `src/misc.c` directly for this. `misc.c` references bcrypt /
blowfish code. This needs a workaround for build configs where libssh2
doesn't export these.
Before this patch, we enabled this workaround for MSVC.
In the patch we extend this to all Windows. There is no CI test for
this, but gcc and llvm/clang + mingw64 builds also need it. This may
well apply to other configurations (it should, as shared libs are not
supposed to export internal functions), so also make it easy to enable
it at a single point.
[ autotools builds force-link this one test against static libssh2. ]
- make `misc.c` not depend on bcrypt.
By moving out our `bcrypt_pbkdf()` wrapper into `bcrypt_pbkdf.c`
itself.
This allows to compile `misc.c` into tests without pulling in bcrypt /
blowfish functions, and simplify the above workaround.
Source code uses `HAVE_BCRYPT_PBKDF`, a leftover from original bcrypt
source. We never define this inside libssh2. Defining it breaks the
build, and this patch doesn't change that.
- make `bcrypt_pbkdf()` static.
While here, make the low-level `bcrypt_pbkdf()` function static to
avoid namespace pollution.
Closes#855
- add timeout to SSH connection wait loop in AppVeyor test prep.
(2 minutes)
- switch to per-step timeout for GitHub CI cmake/ctest runs.
(10 minutes)
ctest timeout (of 450 seconds) didn't seem to make any difference.
- `ctest` shows a the default timeout '10000000' (turns out to be
in seconds), cause infinite waits e.g. in case the necessary server
worker is not available.
CMake CI tests take approx:
- GitHub / Linux : 125 seconds
- AppVeyor / Windows: 300 seconds
New timeouts are: 450 and 900 seconds respectively.
- set timeouts for style-check, fuzz, Linux and Windows GitHub CI
jobs to avoid hanging forever.
Also:
- move `choco install` to before_test to make builds start faster
in `appveyor.yml`.
- fix some yamllint `ON`/`OFF`-confusion issue by quoting these
values in `appveyor.yml`.
- fix indentation in `appveyor.yml`.
- convert to GitHub workflows to LF line-ending.
Ref: https://github.com/libssh2/libssh2/pull/655#issuecomment-1472853493Closes#851
- build x64 first
x64 is the more interesting target. Most type conversion issues are
revealed here. Also more commonly used by now.
- test VS 2013 earlier
- test WinCNG earlier
- delete reference to no longer used VS 2008
After this patch we end up starting with all Shared builds (2015, 2013,
OpenSSL, WinCNG), then continue with Static ones. Shared/Static makes
a minor if any difference in builds/tests compared to different VS
versions of TLS backends.
--
CI run times:
Preparation + build takes:
8 x VS2015 4.5 mins -> total: 36
8 x VS2013 2 mins -> total: 16
Total: 52 mins
with our 30 tests, it increases to:
8 x VS2015 8-10 mins -> total: 72
8 x VS2013 6- 9 mins -> total: 60
Total: 132 mins
Without tests: https://ci.appveyor.com/project/libssh2org/libssh2/builds/46475315
With tests: https://ci.appveyor.com/project/libssh2org/libssh2/builds/46480549
- avoid outputting 4000 log lines by hiding the progress bar.
Reduces log size by 5x.
- decrease timeout (from the default 2700 seconds).
- omit unnecessary output.
Tested as part of #846
- add wolfSSL support.
- reduce size and redundant logic.
- fix a bunch of small issues.
- rework configuration, now with: `CC`, `AR`, `RC`, `TRIPLET`, `CFLAGS`,
`CPPFLAGS`, `LDFLAGS`, `RCFLAGS`, `LIBS`, `LIBSSH2_DLL_SUFFIX`,
`LIBSSH2_LDFLAGS_LIB`, `LIBSSH2_LDFLAGS_BIN` (and more).
- merge examples build into the main Makefile.
- relative dependency paths are now the same for building libssh2 or
examples.
- drop detection for obsolete OpenSSL versions (can be configure via new
`OPENSSL_LIBS`).
- merge dev/dist distribution zip options.
- build libssh2 with `-DHAVE_STRTOLL`.
- tidy-up.
- build examples in static mode by default (use `DYN` to build them in
shared mode).
- drop forced (in non-debug mode) `-O2`.
- drop Win9x support.
- deprecate `ARCH` in favour of custom options and `TRIPLET`.
- drop Windows resources from examples for simplicity
- drop `WITH_ZLIB`. Default `ZLIB_PATH` to enable zlib support.
- drop `LIBSSH2_DLL_A_SUFFIX`, use standard value `.dll` (as in
`libssh2.dll.a`).
- always link `bcrypt` (for LibreSSL and OpenSSL) and `crypt32`
(for wolfSSL).
- unhide executed build commands.
- fix mbedTLS `lib` path
- drop specific options to force static linking. Custom options seems
a better way for this.
- based on similar work made for curl:
a8861b6ccdCloses#842
Before this patch, with debug logging disabled, libssh2 code used a
variadic macro to catch `_libssh2_debug()` calls, and convert them to
no-ops. In certain conditions, it used an empty inline function instead.
Variadic macro is a C99 feature. It means that depending on compiler,
and build settings, it littered the build log with warnings about this.
The new solution uses the trick of passing the variable arg list as a
single argument and pass that down to the debug function with a regular
macro. When disabled, another regular C89-compatible macro converts it
to a no-op.
This makes inlining, C99 variadic macros and maintaining the conditions
for each unnecessary and also makes the codebase compile more
consistently, e.g. with forced C standards and/or picky warnings.
TL;DR: It makes this feature C89-compliant.
Building with wolfSSL or pre-OpenSSL v1.1.1 triggered it.
```
../src/openssl.h:130:5: warning: 'LIBRESSL_VERSION_NUMBER' is not defined, evaluates to 0 [-Wundef]
LIBRESSL_VERSION_NUMBER >= 0x3070000fL
^
```
Regression from 2e2812dde8
wolfSSL supports building with zlib as a dependency, that's the reason
for the ZLIB logic in the patch.
Also add it to `docs/INSTALL_CMAKE.md` and to the help text in
`src/CMakeLists.txt`.
Running tests not actually tested.
Follow-up to 9f217a17f6
Ref: #817
Flakiness got continously worse these last days. It didn't seem related
to recent commits. Flakiness also picked up in GitHub CI runs, something
rarely seen before. Manual restart consistently fixed them.
The repeating pattern was the _first_ test (`test_hostkey`) failing,
with `libssh2_session_handshake failed (-13): Failed getting banner`.
Failures came after a lengthy wait, suggesting a timeout.
I then reversed the order of the first two tests, and it turned out that
the _first_ test failed again (`test_hostkey_hash`). Also pointing to a
timeout issue.
Then I added a dummy test to "warm up" whatever needs warming up in the
layers of CI + Docker + ssh server and their interconnects. This helped,
and GitHub CI tests run without failure right for the first time.
AppVeyor CI also improved a little.
This patch adds a new first test called `test_warmup`, that creates a
new libssh2 session, and exits with success even if that attempt failed.
A stop-gap solution at best, and there is no guarantee it will continue
to fix this or similar future issues, but it's also untenable to have
almost every CI run fail for intermittent reasons.
In some [1] cases [2] it's not the first test failing intermittently.
That's a different issue, and this patch doesn't fix it.
[1] #804
[2] https://ci.appveyor.com/project/libssh2org/libssh2/builds/46440828/job/8rej6cq6itg7vc4w#L500
Turns out `test_keyboard_interactive_auth_info_request.c` requires
`src/libssh2_priv.h`, which in turn requires a correctly set
`HAVE_SNPRINTF`.
Follow-up to 4cdf785cd3.
Added in cf80f2f4b5 (on 2016-08-14),
with the title "Basic dockerised test suite".
It's not clear why a C standard was explicitly set, but a side-effect
of this is that CMake-built binaries diverged from ones built with
autotools or GNU Make (using the same compiler and configuration).
Another issue is that this may introduce ABI incompatibility with
binaries built with a different C standard flag, e.g. the C compiler
default or one used for other components of a final app.
Seems unlikely, but if our tests require this option, we should set it
for the CI builds only?
This restores socket libs to their pre-regression positions.
Without this, `ld` doesn't find `ws2_32` symbols when referenced
from TLS libs.
Regression from 31fb8860db
Should fix these warnings with MSVS 2013 and older:
`agent.c(294): warning C4013: '_libssh2_snprintf' undefined; assuming extern returning int`
Follow-up to 4cdf785cd3.
autotools builds already did auto-detect and set this mingw-specific
macro, but CMake and GNU Make builds did not. This patch fixes that.
Necessary for `src/scp.c`.
Add OpenSSL 3 and versionless DLL names. Also modernize warning messages
and variable names.
Do we need the OpenSSL-Windows-specific check and the related
`RUNTIME_DEPENDENCIES` feature? The list of OpenSSL DLLs was out of date
for 1.5 years without anybody noticing. Keeping it fresh is a chore and
copying around DLL dependencies rarely helps as much as expected. This
check also results in unuseful warnings in certain build scenarios, e.g.
when linking to OpenSSL statically.
Implement wolfSSL support for libssh2 when building with CMake.
Configuration example from curl-for-win:
```
-DCRYPTO_BACKEND=wolfSSL
-DWOLFSSL_LIBRARY=/path-to/wolfssl/lib/libwolfssl.a
-DWOLFSSL_INCLUDE_DIR=/path-to/wolfssl/include
```
Module `cmake/Findwolfssl.cmake` copied from:
e4d920c4b7/cmake/Findwolfssl.cmake
via commit:
296396d373
by Stefan Eissing
I mistakenly pruned some non-Windows logic, also missing the fact that
our local `check_function_exists_may_need_library()` set the `NEED_*`
variables. Oddly, only `src` imported this function, yet also `examples`
and `tests` called it indirectly. The referenced `HAVE_SOCKET` /
`HAVE_INET_ADDR` variables might be coming from an upstream CMake
project? Leaving those there also, just in case.
Regression from 31fb8860db
- cmake: always link `ws2_32` on Windows. Also add it to `libssh2.pc`.
Fixes#745
- agent: fix gcc compiler warning:
`src/agent.c:296:35: warning: 'snprintf' output truncated before the last format character [-Wformat-truncation=]`
- autotools: fix `EVP_aes_128_ctr` detection with binutils `ld`
The prerequisite for a successful detection is setting
`LIBS=-lbcrypt` if the chosen openssl-compatible library requires
it, e.g. libressl, or quictls/openssl built with
`-DUSE_BCRYPTGENRANDOM`.
With llvm `lld`, detection works out of the box. With binutils `ld`,
it does not. The reason is `ld`s world-famous pickiness with lib
order.
To fix it, we pass all custom libs before and after the TLS libs.
This ugly hack makes `ld` happy and detection succeed.
- agent: fix Windows-specific warning:
`src/agent.c:318:10: warning: implicit conversion loses integer precision: 'LRESULT' (aka 'long long') to 'int' [-Wshorten-64-to-32]`
- src: fix llvm/clang compiler warning:
`src/libssh2_priv.h:987:28: warning: variadic macros are a C99 feature [-Wvariadic-macros]`
- src: support `inline` with `__GNUC__` (llvm/clang and gcc), fixing:
```
src/libssh2_priv.h:990:8: warning: extension used [-Wlanguage-extension-token]
static inline void
^
```
- blowfish: support `inline` keyword with MSVC.
Also switch to `__inline__` (from `__inline`) for `__GNUC__`:
https://gcc.gnu.org/onlinedocs/gcc/Inline.htmlhttps://clang.llvm.org/docs/UsersManual.html#differences-between-various-standard-modes
- example/test: fix MSVC compiler warnings:
- `example\direct_tcpip.c(209): warning C4244: 'function': conversion from 'unsigned int' to 'u_short', possible loss of data`
- `tests\session_fixture.c(96): warning C4013: 'getcwd' undefined; assuming extern returning int`
- `tests\session_fixture.c(100): warning C4013: 'chdir' undefined; assuming extern returning int`
- delete unused macros:
- `HAVE_SOCKET`
- `HAVE_INET_ADDR`
- `NEED_LIB_NSL`
- `NEED_LIB_SOCKET`
- `HAVE_NTSTATUS_H`
- `HAVE_NTDEF_H`
- build: delete stale zlib/openssl version numbers from path defaults.
- cmake: convert tabs to spaces, add newline at EOFs.
Closes#811
Improve tests added in 7487dcf4b4 by
running `test_read` commands directly. This makes external shell/batch
files unnecessary, and is friendlier with cross-builds and when run
from non-default shells, like MSYS2.
Also extend CRYPT/MAC test error messages with the CRYPT/MAC name.
External runner shell scripts kept for future use.
Closes#814
- convert `_libssh2_explicit_zero()` to macro. This allows inlining
where supported (e.g. `SecureZeroMemory()`).
- replace `SecureZeroMemory()` (in `wincng.c`) and
`LIBSSH2_CLEAR_MEMORY`-guarded `memset()` (in `os400qc3.c`) with
`_libssh2_explicit_zero()` macro.
- delete `LIBSSH2_CLEAR_MEMORY` guards, which enables secure-zeroing
universally.
- add `LIBSSH2_NO_CLEAR_MEMORY` option to disable secure-zeroing.
- while here, delete double/triple inclusion of `misc.h`.
`libssh2_priv.h` included it already.
Closes#810
Before this patch, the `snprintf()` fallback logic for envs not
supporting this function (i.e. Visual Studio 2013 and older) varied
depending on build tool, and used different techniques in examples,
tests and libssh2 itself.
This patch aims to apply a common logic to libssh2 and examples/tests.
- libssh2: use local `snprintf()` fallback with all build tools.
We already had a local implementation, but only with CMake. Move that
to the library as `_libssh2_snprintf()`, and map `snprintf()` to it
when `HAVE_SNPRINTF` is not set.
Also change the length type from `int` to `size_t`, and fix
formatting.
- set or detect `HAVE_SNPRINTF` in non-CMake builds.
Detect in autotools. Keep existing logic in `win32/libssh2_config.h`.
Always set for OS/400, NetWare and VMS, keeping existing behaviour.
(OS/400 builds use a different local implementation)
- examples/tests: drop the CMake-specific fallback logic and map
`snprintf()` to `_snprintf()` for old MSVC versions, like we did
before with other build tools. This is unsafe, but should be fine for
these uses.
- `win32/libssh2_config.h`: make it easier to read.
Closes#812
- Link `bcrypt` for newer (non-fork) OpenSSL.
- Link `bcrypt` and `ws2_32` when using (non-fork) OpenSSL or LibreSSL,
to allow `Looking for EVP_aes_128_ctr` detecting this feature.
With the feature available, but not found by CMake, build failed with:
`openssl.c:636:21: error: incompatible integer to pointer conversion assigning to 'EVP_CIPHER *' (aka 'struct evp_cipher_st *') from 'int' [-Wint-conversion]`
Closes#809
- in `hostkey.c` check the result of `libssh2_sha256_init()` and
`libssh2_sha512_init()` calls. This avoid the warning that we're
ignoring the return values.
- fix code using `int` (or `SOCKET`) for sockets. Use libssh2's
dedicated `libssh2_socket_t` and `LIBSSH2_INVALID_SOCKET` instead.
- fix compiler warnings due to `STATUS_*` macro redefinitions between
`ntstatus.h` / `winnt.h`. Solve it by manually defining the single
`STATUS` value we need from `ntstatus.h` and stop including the whole
header.
Fixes#733
- improve Windows UWP/WinRT builds by detecting it with code copied
from the curl project. Then excluding problematic libssh2 parts
according to PR by Dmitry Kostjučenko.
Fixes#734
- always use `SecureZeroMemory()` on Windows.
We can tweak this if not found or not inlined by a C compiler which
we otherwise support. Same if it causes issues with UWP apps.
Ref: https://learn.microsoft.com/en-us/previous-versions/windows/desktop/legacy/aa366877(v=vs.85)
Ref: https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-rtlsecurezeromemory
- always enable `LIBSSH2_CLEAR_MEMORY` on Windows. CMake and
curl-for-win builds already did that. Delete `SecureZeroMemory()`
detection from autotools' WinCNG backend logic, that this
setting used to depend on.
TODO: Enable it for all platforms in a separate PR.
TODO: For clearing buffers in WinCNG, call `_libssh2_explicit_zero()`,
insead of a local function or explicit `SecureZeroMemory()`.
- Makefile.inc: move `os400qc3.h` to `HEADERS`. This fixes
compilation on non-unixy platforms. Recent regression.
- `libssh2.rc`: replace copyright with plain ASCII, as in curl.
Ref: curl/curl@1ca62bb
Ref: curl/curl#7765
Ref: curl/curl#7776
- CMake fixes and improvements:
- enable warnings with llvm/clang.
- enable more comprehensive warnings with gcc and llvm/clang.
Logic copied from curl:
233810bb5f/CMakeLists.txt (L131-L148)
- fix `Policy CMP0080` CMake warning by deleting that reference.
- add `ENABLE_WERROR` (default: `OFF`) option. Ported from curl.
- add `PICKY_COMPILER` (default: `ON`) option, as known from curl.
It controls both the newly added picky warnings for llvm/clang and
gcc, and also the pre-existing ones for MSVC.
- `win32/GNUmakefile` fixes and improvements:
- delete `_AMD64_` and add missing `-m64` for x64 builds under test.
- add support for `ARCH=custom`.
It disables hardcoded Intel 64-bit and Intel 32-bit options,
allowing ARM64 builds.
- add support for `LIBSSH2_RCFLAG_EXTRAS`.
To pass custom options to windres, e.g. in ARM64 builds.
- add support for `LIBSSH2_RC`. To override `windres`.
- delete support for Metrowerks C. Last released in 2004.
- `win32/libssh2_config.h`: delete unnecessary socket #includes
`src/libssh2_priv.h` includes `winsock2.h` and `ws2tcpip.h` further
down the line, triggered by `HAVE_WINSOCK2_H`.
`mswsock.h` does not seem to be necessary anymore.
Double-including these (before `windows.h`) caused compiler failures
when building against BoringSSL and warnings with LibreSSL. We could
work this around by passing `-DNOCRYPT`. Deleting the duplicates
fixes these issues.
Timeline:
2013: c910cd382d deleted `mswsock.h` from `src/libssh2_priv.h`
2008: 8c43bc52b1 added `winsock2.h` and `ws2tcpip.h` to `src/libssh2_priv.h`
2005: dc4bb1af96 added the now deleted #includes
- delete or replace `LIBSSH2_WIN32` with `WIN32`.
- replace hand-rolled `HAVE_WINDOWS_H` macro with `WIN32`. Also delete
its detections/definitions.
- delete unused `LIBSSH2_DARWIN` macro.
- delete unused `writev()` Windows implementation
There is no reference to `writev()` since 2007-02-02, commit
9d55db6501.
- fix a bunch of MSVC / llvm/clang / gcc compiler warnings:
- `warning C4100: '...': unreferenced formal parameter`
- using value of undefined PP macro `LIBSSH2DEBUG`
- missing void from function definition
- `if()` block missing in non-debug builds
- unreferenced variable in non-debug builds
- `warning: must specify at least one argument for '...' parameter of variadic macro [-Wgnu-zero-variadic-macro-arguments]`
in `_libssh2_debug()`
- `warning C4295: 'ciphertext' : array is too small to include a terminating null character`
- `warning C4706: assignment within conditional expression`
- `warning C4996: 'inet_addr': Use inet_pton() or InetPton() instead or
define _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings`
By suppressning it. Would be best to use inet_pton() as suggested.
On Windows this needs Vista though.
- `warning C4152: nonstandard extension, function/data pointer conversion in expression`
(silenced locally)
- `warning C4068: unknown pragma`
Ref: https://ci.appveyor.com/project/libssh2org/libssh2/builds/46354480/job/j7d0m34qgq8rag5wCloses#808
Connects to the ssh server then downloads several MB of data. This
tests the data transfer path as well as boundary cases in packet
handling as data is split into smaller SSH blocks.
Two warnings (in tests & examples) in particular would cause problems:
bad format causing invalid data output or a bad chdir due to out of
scope buffer use.
This performs better as an example since it shows more working code, and
in the simplest possible way. It also turns the program into an actually
useful tool out of the box, able to run an arbitrary command (with one
restriction) on a remote machine and return the response, without
needing to touch the source.
Closes#800
