Add macOS CI jobs, both cmake and autotools for all supported crypto
backends (except BoringSSL), with debug, zlib enabled. Without running
tests. It also introduces OpenSSL 1.1 into the CI with a non-MSVC
compiler.
Credits to curl's `macos.yml`, that I used as a base.
Fix these issues uncovered by the new tests:
- openssl: fix warning when built with wolfSSL, or OpenSSL 1.1 and
earlier. CI missed it because apparently the only OpenSSL 1.1 test
we had used MSVC, which did not complain.
```
../src/openssl.c:3852:19: error: variable 'sslError' set but not used [-Werror,-Wunused-but-set-variable]
unsigned long sslError;
^
```
Regression from 097c8f0dae
- pem: add hack to build without MD5 crypto-backend support.
The Homebrew wolfSSL build comes with MD5 support disabled. We can
expect this becoming the norm. FIPS also requires MD5 disabled.
We deleted the same hack from `hostkey.c` a month ago:
ad6aae302a
A better fix would be to guard the MD5 logic with our `LIBSSH2_MD5`
macro.
```
pem.c:214:32: error: use of undeclared identifier 'MD5_DIGEST_LENGTH'; did you mean 'SHA_DIGEST_LENGTH'?
unsigned char secret[2*MD5_DIGEST_LENGTH];
^~~~~~~~~~~~~~~~~
SHA_DIGEST_LENGTH
```
Regression from 386e012292
- `configure.ac`: add crypto libs late.
Fix it by adding crypto libs to `LIBS` at the end of the configuration
process.
Otherwise `configure` links crypto libs while doing feature tests,
which can cause unwanted detections. For example LibreSSL publishes
the function `explicit_bzero()`, which masks the system alternative,
e.g. `memset_s()` on macOS. Then when trying to compile libssh2, its
declaration is missing:
```
bcrypt_pbkdf.c:93:5: error: implicit declaration of function 'explicit_bzero' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
_libssh2_explicit_zero(ciphertext, sizeof(ciphertext));
^
../src/misc.h:50:43: note: expanded from macro '_libssh2_explicit_zero'
^
```
Regression from 4f0f4bff5a
- cmake: fix to list our own include directory before the crypto libs',
when building tests.
Otherwise a global crypto header path, such as `/usr/local/include`,
containing an external `libssh2.h` of a different version, could cause
weird errors:
```
cc -DHAVE_CONFIG_H -DLIBSSH2_LIBGCRYPT \
-I../src -I../../src -I/usr/local/include -I[...]/libssh2/include \
-g -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk \
-mmacosx-version-min=12.6 -MD -MT \
tests/CMakeFiles/test_aa_warmup.dir/test_aa_warmup.c.o \
-MF CMakeFiles/test_aa_warmup.dir/test_aa_warmup.c.o.d \
-o CMakeFiles/test_aa_warmup.dir/test_aa_warmup.c.o -c \
[...]/libssh2/tests/test_aa_warmup.c
```
```
[ 62%] Building C object tests/CMakeFiles/test_aa_warmup.dir/test_aa_warmup.c.o
In file included from /Users/runner/work/libssh2/libssh2/tests/test_aa_warmup.c:4:
In file included from /Users/runner/work/libssh2/libssh2/tests/runner.h:42:
In file included from /Users/runner/work/libssh2/libssh2/tests/session_fixture.h:43:
/Users/runner/work/libssh2/libssh2/tests/../src/libssh2_priv.h:649:5: error: type name requires a specifier or qualifier
LIBSSH2_AUTHAGENT_FUNC((*authagent));
^
/Users/runner/work/libssh2/libssh2/tests/../src/libssh2_priv.h:649:30: error: type specifier missing, defaults to 'int' [-Werror,-Wimplicit-int]
LIBSSH2_AUTHAGENT_FUNC((*authagent));
^
/Users/runner/work/libssh2/libssh2/tests/../src/libssh2_priv.h:650:5: error: type name requires a specifier or qualifier
LIBSSH2_ADD_IDENTITIES_FUNC((*addLocalIdentities));
^
/Users/runner/work/libssh2/libssh2/tests/../src/libssh2_priv.h:650:35: error: type specifier missing, defaults to 'int' [-Werror,-Wimplicit-int]
LIBSSH2_ADD_IDENTITIES_FUNC((*addLocalIdentities));
^
/Users/runner/work/libssh2/libssh2/tests/../src/libssh2_priv.h:651:5: error: type name requires a specifier or qualifier
LIBSSH2_AUTHAGENT_SIGN_FUNC((*agentSignCallback));
^
/Users/runner/work/libssh2/libssh2/tests/../src/libssh2_priv.h:651:35: error: type specifier missing, defaults to 'int' [-Werror,-Wimplicit-int]
LIBSSH2_AUTHAGENT_SIGN_FUNC((*agentSignCallback));
^
6 errors generated.
```
- `tests/session_fixture.h`: delete duplicate `libssh2.h`,
`libssh2_priv.h` already includes it.
Follow-up to a683133dfe
CI logs with these errors:
https://github.com/libssh2/libssh2/actions/runs/4824079094https://github.com/libssh2/libssh2/actions/runs/4824270819
curl's `macos.yml`: da2470de96/.github/workflows/macos.yml
Tidying-up while here:
- tests/session_fixture.h: delete duplicate `libssh2.h`.
`libssh2_priv.h` includes it already.
Follow-up to a683133dfe
- ci.yml: yamllint warnings and formatting.
- ci.yml: msvc section formatting and step-naming sync with macOS.
Follow-up to f4a4c05dc3
- ci.yml: enable `--enable-werror` for msys2 jobs.
Follow-up to 71cae949d5
- appveyor.yml: show OpenSSL versions, link to image content.
Closes#1013
- drop unused or duplicate C headers.
- add missing ones (that worked by chance).
(`string.h`, `stdlib.h`)
- mention the functions that need certain headers.
- move some headers from crypto header to crypto C source.
- reorder headers in some places.
- simplify the #if tree for `sys/select.h` in `libssh2_priv.h`.
- move scp-specific macros next to their header to `scp.c`
Follow-up to 5db836b2a8Closes#999
- add MSVC jobs to GitHub Actions. They are similar to the 'Build-only'
jobs we have on AppVeyor CI, though only the ARM64 Windows one is
identical. Major disadvantage is that we don't run tests here. Major
advantage is they only take a few minutes to complete, compared to
an hour on AppVeyor, so WinCNG build results now appear quicker.
Docker tests might be possible, but my light attempts failed.
Finding ZLIB also failed, so we still miss an MSVC test with it.
Tool versions as of now: Server 2022, VS2022, OpenSSL 1.1.1
- add UWP builds for both ARM64 and x64. This hasn't been CI tested
before.
(We could probably enable UWP on AppVeyor CI as well.
I haven't tried.)
- fix two uncovered UWP issues in tests.
- rename internal macro `LIBSSH2_WINDOWS_APP` to `LIBSSH2_WINDOWS_UWP`.
Follow-up to 2addafb77b
- fold long lines and quote truthy values in `.github/workflows/ci.yml`.
Closes#1010
If no `FIXTURE_WORKDIR` macro or envvar is present to set the cwd,
avoid querying the cwd and then calling chdir with the result.
Ref: 54bef4c5da (patch)
Ref: 10a5cbf945 (individual commit)
Closes#1009
Currently SFTP's init will busy loop waiting for the channel to close,
even if the underlying transport returns EAGAIN. While this works for
sockets, it might not work out if you have a different transport that
needs to do some additional processing on the side.
Integration-patches-by: Viktor Szakats
- simplify `.TH` headers.
- delete empty lines before sections.
- update template with an `AVAILABILITY` section.
Left libssh2 version number in the `.TH` header for entries without an
`AVAILABILITY` section, or where there was a different version number
there.
Due to a typo in the `HAVE_*` macro, this header was never included.
A comment suggests that `socklen_t` is not defined on VMS and defines it
manually. This symbol is usually in `sys/socket.h`, so the typo may have
been the reason for it to be missing.
Closes#1007
- add #included C files to `EXTRA_DIST`.
Regression from 4f0f4bff5a
- fix `tests/sshd_fixture.sh` to not write into the test dir, by using
a pre-assembled `TrustedUserCAKeys` file. Update `Dockerfile` too to
use this.
Regression from a459a25302
Also update `tests/sshd_fixture.sh` to use
`openssh_server/authorized_keys` like `Dockerfile` does. And a few more
cosmetic updates.
Closes#1006
Unless I'm missing something, it looks like `libssh2.h` has been using
`libssh2_int64_t` unconditionally since at least 2010-04-17 when
`libssh2_scp_send64()` landed via commit
be9ee7095e.
This makes it redundant to detect `HAVE_LONGLONG` to fallback to a
32-bit `scpRecv_size` in `libssh2_priv.h`. Then deal with possible
combinations of this flag and `strtoll()` options, which was
error-prone.
Instead, assume in `libssh2_priv.h` that we have `libssh2_int64_t`, and
use it always.
For MSVC, this means `_MSC_VER` `1310` (from year 2003) is now
required. Based on the above, this was already so before this patch.
If there happens to be no 64-bit `strtoll()` detected, fall back to the
32-bit `strtol()` (this should never happen with MSVC, and probably
neither with any other reasonably modern toolchain.)
Also make sure to set `HAVE_STRTOI64` for older, non-CMake, MSVC builds
(e.g. `Makefile.mk` or `NMakefile` ones).
Closes#1002
- openssl: fix potentially missing `ERR_*` constants by including
`openssl/err.h`. This could happen with recent version of Xcode
or when building against OpenSSL built with the `OPENSSL_NO_ENGINE`
option.
Regression from 097c8f0dae (#789)
- channel: fix an issue that would corrupt the data stream when
attempting to initialize the agent in non-blocking mode, as it is
necessary to propagate the `EAGAIN` signal upstream when the transport
returns `EAGAIN`.
Regression from bc4e619e76 (#752)
- packet: the current code does not set the state machine upon reaching
this point which means that if the code is suspended due to the
transport returning an `EAGAIN`, this will re-initialize the structure
every time.
The issue is that this keeps assigning a new channel-id downstream,
which does not match the initial channel-id that is initially
generated, causing a lookup later to fail as there is no matching
channel.
Regression from bc4e619e76 (#752)
Simplify the way we handle `gettimeofday()` fallback for platforms
without native support or without any support. Make it similar to
how we handle `snprintf()`.
In case of no native `gettimeofday()` support and a non-Windows
platform, our local fallback returns zero in `tv_usec` and `tv_sec`,
ending up with a zero `timeout_remaining` in `session.c`, same as
before this patch.
Also:
- drop unused `sys/time.h` headers.
- fix our fallback code to compile with any Windows compilers
(not just MSVC)
- delete unnecessary casts.
Closes#1001
Noticed this when libasan started kicking out errors when sending in
MACs preferences that were not supported yet.
Reported-by: fourierules on github
Fixes#611Closes#1000
TL;DR: Sync test builds between autotools and CMake. Sync sshd
configuration between Docker and non-Docker fixtures. Bump up
sshd_config for recent OpenSSH releases.
This also opens up the path to have non-Docker tests that use a
local sshd process. Though sshd is practically unusable on Windows
CI machines out of the box, so this will need further efforts.
Details:
- cmake: run sshd fixture test just like autotool did already.
- sync tests and their order between autotools and CMake.
It makes `test_aa_warmup` the first test with both.
- cmake: load test lists from `Makefile.am`.
Needed to update the loader to throw away certain lines to keep the
converted output conform CMake syntax. Using regexp might be an
alternative way of doing this, but couldn't make it work.
- cmake: use the official way to configure test environment variables.
Switch to syntax that's extendable.
- cmake: allow to run the same test both under Docker and sshd fixture.
Useful for testing the sshd fixture runner, or how the same test
behaves in each fixture.
- update test fixture to read the username from `USER` envvar instead of
using the Dockfile-specific hardwired one, when running outside Docker.
- rework `ssh2.sh` into `sshd_fixture.sh`, to:
- allow running any tests (not just `test_ssh2`).
- configure Docker tests for running outside Docker.
- fixup `SSHD` path when running on Windows (e.g. in AppVeyor CI).
Fixes: `sshd re-exec requires execution with an absolute path`
- allow overriding `PUBKEY` and `PRIVKEY` envvars.
- allow overriding `ssh_config` via `SSHD_FIXTURE_CONFIG`.
- prepare support for running multiple tests via sshd_fixture.
Add a TAP runner for autotools and extend CMake logic. The TAP runner
loads the test list from `Makefile.am`.
Notice however that on Windows, `sshd_fixture.sh` is very flaky with
GitHub Actions. And consistently broken for subsequent tests in
AppVeyor CI:
'libssh2_session_handshake failed (-43): Failed getting banner'
Another way to try is a single sshd instance serving all tests.
For CMake this would probably mean using an external script.
- ed25519 test keys were identical for auth and host. Regenerate the
auth keypair to make them distinct.
- sync the sshd environment between Docker and sshd_fixture.
- use common via `openssh_server/sshd_config`.
- accept same auth keys.
- offer the same host keys.
- sync TrustedUserCAKeys.
- delete now unused keypairs: `etc/host*`, `etc/user*`.
- bump up startup delay for Windows (randomly, to 5 secs, from 3).
- delete `UsePrivilegeSeparation no` to avoid deprecation warnings.
`command-line line 0: Deprecated option UsePrivilegeSeparation`
- delete `Protocol 2` to avoid deprecation warnings.
It has been the default since OpenSSH 3.0 (2001-11-06).
- delete `StrictModes no` (CI tests work without it, Docker tests
never used it).
- bump `Dockerfile` base image to `testing-slim` (from `bullseye-slim`).
It needed `sshd_config` updates to keep things working with
OpenSSH 9.2 (compared to bullseye's 8.4).
- replace `ChallengeResponseAuthentication` alias with
`KbdInteractiveAuthentication`.
The former is no longer present in default `sshd_config` since
OpenSSH 8.7 (2021-08-20). This broke the `Dockerfile` script.
The new name is documented since OpenSSH 4.9 (2008-03-31)
- add `PubkeyAcceptedKeyTypes +ssh-rsa,ssh-dss,ssh-rsa-cert-v01@openssh.com`
and `HostKeyAlgorithms +ssh-rsa`.
Original-patch-by: Eric van Gyzen (@vangyzen on github)
Fixes#691
There is a new name for `PubkeyAcceptedKeyTypes`:
`PubkeyAcceptedAlgorithms`.
It requires OpenSSH 8.5 (2021-03-03) and breaks some envs so we're
not using it just yet.
- drop `rijndael-cbc@lysator.liu.se` tests and references from config.
This is a draft alias for `aes256-cbc`. No need to test it twice.
Also this alias is no longer recognized by OpenSSH 8.5 (2021-03-03).
- update `mansyntax.sh` and `sshd_fixture.sh` to not rely on `srcdir`.
Hopefully this works with out-of-tree builds.
- fix `test_read_algos.test` to honor CRLF EOLs in their inputs
(necessary when running on Windows.)
- fix `test_read_algos.test` to honor `EXEEXT`. Might be useful when
running tests under cross-builds?
- `test_ssh2.c`:
- use libssh2 API to set blocking mode. This makes it support all
platforms.
- adapt socket open timeout logic from `openssh_fixture.c`.
Sadly this did not help fix flakiness on GHA Windows.
- tests: delete unused C headers and variable initialization.
- delete unused test files: `sshd_fixture.sh.in`, `sshdwrap`,
`etc/sshd_config`.
Ref: cf80f2f4b5
- autotools: delete stray `.c` test sources from `EXTRA_DIST` in tests.
- `tests/.gitignore`: drop two stray tests.
- autotools: fix passing `SSHD` containing space (Windows needs this).
- autotools: sort `EXTRA_DIST` in tests.
- cmake: fix to add `test_ssh2` to `TEST_TARGETS`.
- fix `authorized_key` order in `tests/gen_keys.sh`.
- silence shellcheck warning in `ci/checksrc.sh`.
- set `SSHD` for autotools on GitHub Actions Windows. [skipped]
Auto-detection doesn't work (maybe because sshd is installed via
Git for Windows and we're using MSYS2's shell.)
It enables running sshd fixture (non-Docker) tests in these jobs.
I did not include this in the final patch due to flakiness:
```
Connection to 127.0.0.1:4711 attempt #0 failed: retrying...
Connection to 127.0.0.1:4711 attempt #1 failed: retrying...
Connection to 127.0.0.1:4711 attempt #2 failed: retrying...
Failure establishing SSH session: -43
```
Can be enabled with:
`export SSHD='C:/Program Files/Git/usr/bin/sshd.exe'`
Closes#996
Make the block count customizable in `test_read` via environment
`FIXTURE_XFER_COUNT`.
Set the custom count lower than the default when running on AppVeyor.
The goal is to reduce CI roundtrip times.
Closes#995
This PR contains a series of patches that date back many years and I
believe were discussed on the mailing list, but never merged. We have
been using these in our local copy of libssh2 without issue since 2015,
if not earlier. I believe this is the full set of changes, as we tried
to use comments to mark where our copy of libssh2 differs from the
canonical version.
This also contains changes I made earlier this year, but which were not
discussed on the mailing list, to support certificates and FIDO2 keys
with agent forwarding.
Note that this is not a complete implementation of agent forwarding, as
that is outside the scope of libssh2. Clients still need to provide
their own implementation that parses ssh-agent methods after calling
libssh2_channel_read() and calls the appropriate callback messages in
libssh2. See the man page changes in this PR for more details.
Integration-patches-by: Viktor Szakats
* prefer size_t
* prefer unsigned int over u_int in public function
* add const
* docs, indent, checksrc, debug call, compiler warning fixes
We had Windows Server 2012 R2 (8.1) and Windows Server 2019 (10) before
this patch. After, we also have Windows Server 2016 (10).
The WinCNG flakey tests should have a better chance when running on the
newer OS.
This update does not change the compiler mix.
Also change the test fixture to not use the `--quiet` option with the
`docker pull` commant. This option requires docker v19.03, and
AppVeyor's Visual Studio 2017 image doesn't support it. Log output did
not change without `--quiet`, so it seems safe to delete it. In case
we'd need it, another solution is to retry without `--quiet` if the
command fails. docker's exit status is 125 in that case.
Ref: https://github.com/libssh2/libssh2/issues/804#issuecomment-1515232799
Ref: https://www.appveyor.com/docs/windows-images-software/Closes#994
Keep a single list for mac and crypt algos that we use in both CMake
and autotools. Use the same test names across build tools.
Use the TAP protocol to track individual tests run from a single shell
script.
Also:
- enable the rest of our tests with autotools.
- set `make check` verbose to see errors in case they happen.
- silence stray 'command not found' error when running `mansyntax.sh`
on Windows.
GitHub Actions Windows docker tests disabled due to:
```
Command: docker build --quiet -t libssh2/openssh_server ../tests/openssh_server
Error running command 'docker build --quiet -t libssh2/openssh_server ../tests/openssh_server' (exit 1): Sending build context to Docker daemon 22.02kB
Step 1/42 : FROM debian:bullseye-slim
bullseye-slim: Pulling from library/debian
no matching manifest for windows/amd64 10.0.20348 in the manifest list entries
Failed to build docker image
```
Closes#993
Support for calculating MAC (message authentication code) on encrypted
data instead of plain text data.
This adds support for the following MACs:
- `hmac-sha1-etm@openssh.com`
- `hmac-sha2-256-etm@openssh.com`
- `hmac-sha2-512-etm@openssh.com`
Integration-patches-by: Viktor Szakats
* rebase on master
* fix checksec warnings
* fix compiler warning
* fix indent/whitespace/eol
* rebase/manual merge onto AES-GCM patch #797
* more manual merge of `libssh2_transport_send()` based
on dfandrich/shellfish
Fixes#582Closes#655Closes#987
Can send specific signals to remote process. Allows for slightly
improved remote process management, if the server supports it.
Integration-patches-by: Viktor Szakats
* doc updates
* change `signame_len` to `size_t`
* variable scopes
* fix checksrc warnings
Closes#672Closes#991
Add support for aes256-gcm@openssh.com and aes128-gcm@openssh.com
ciphers, which are the OpenSSH implementations of AES-GCM cryptography.
It is similar to RFC5647 but has changes to the MAC protocol
negotiation. These are implemented for recent versions of OpenSSL only.
The ciphers work differently than most previous ones in two big areas:
the cipher includes its own integrated MAC, and the packet length field
in the SSH frame is left unencrypted. The code changes necessary are
gated by flags in the LIBSSH2_CRYPT_METHOD configuration structure.
These differences mean that both the first and last parts of a block
require special handling during encryption. The first part is where the
packet length field is, which must be kept out of the encryption path
but in the authenticated part (as AAD). The last part is where the
Authentication Tag is found, which is calculated and appended during
encryption or removed and validated on decryption. As encryption/
decryption is performed on each packet in a loop, one block at a time,
flags indicating when the first and last blocks are being processed are
passed down to the encryption layers.
The strict block-by-block encryption that occurs with other protocols is
inappropriate for AES-GCM, since the packet length shifts the first
encrypted byte 4 bytes into the block. Additionally, the final part of
the block must contain the AES-GCM's Authentication Tag, so it must be
presented to the lower encryption layer whole. These requirements mean
added code to consolidate blocks as they are passed down.
When AES-GCM is negotiated as the cipher, its built-in MAC is
automatically used as the SSH MAC so further MAC negotiation is not
necessary. The SSH negotiation is skipped when _libssh2_mac_override()
indicates that such a cipher is in use. The virtual MAC configuration
block mac_method_hmac_aesgcm is then used as the MAC placeholder.
This work was sponsored by Anders Borum.
Integration-patches-by: Viktor Szakats
* fix checksrc errors
* fix openssl.c warning
* fix transport.c warnings
* switch to `LIBSSH2_MIN/MAX()` from `MIN()`/`MAX()`
* fix indent
* fix libgcrypt unused warning
* fix mbedtls unused warning
* fix wincng unused warning
* fix old openssl unused variable warnings
* delete blank lines
* updates to help merging with the ETM patch
Add new guard `LIBSSH2_RSA_SHA1`. Add missing guards for `LIBSSH2_RSA`,
`LIBSSH2_DSA`.
Fix warnings when all options are disabled.
This is still not complete and it's possible to break a build with
certain crypto backends (e.g. mbedTLS) and/or combination of options.
It's not guaranteed that all bits everywhere get disabled by these
settings. Consider this a "best effort".
Add these new options to disable certain crypto elements:
- `LIBSSH2_NO_3DES`
- `LIBSSH2_NO_AES_CTR`
- `LIBSSH2_NO_BLOWFISH`
- `LIBSSH2_NO_CAST`
- `LIBSSH2_NO_ECDSA`
- `LIBSSH2_NO_RC4`
- `LIBSSH2_NO_RSA_SHA1`
- `LIBSSH2_NO_RSA`
The goal is to offer a way to disable legacy/obsolete/insecure ones.
See also: 146a25a06d `LIBSSH2_NO_HMAC_RIPEMD`
See also: 38015f4e46 `LIBSSH2_NO_DSA`
See also: be31457f30 `LIBSSH2_NO_MD5`
Closes#986
This also significantly (by 7-8x in my limited tests with a short
string) speeds up this function. The impact is still minor as this
function is only used in `knownhost.c` in release builds.
Closes#985
Works like the `FIXTURE_TRACE_ALL` envvar, but enables full trace for
the connection phase only.
Also fix a possible NULL deref with `FIXTURE_TRACE_ALL` and a failed
`libssh2_session_init_ex()`.
Tested in #979
`CONFIGURATION` was never passed to the cmake command, so it had
never enabled logging when set to `Debug`.
Also `CONFIGURATION` is ambiguous depending on the "generator" used
by CMake. In case of Visual Studio, this is a build/ctest-time
setting, not a cmake-config parameter.
So set this permanently to `Release` and enable logging via our
dedicated CMake option `ENABLE_DEBUG_LOGGING`.
Tested in #979
