1
0
mirror of https://github.com/libssh2/libssh2.git synced 2026-10-24 03:48:58 +03:00
Commit Graph

4016 Commits

Author SHA1 Message Date
Viktor Szakats
7839760cdc GHA/codeql: switch GHA check to arm64
Identical performance, better efficiency.

Closes #2647
2026-10-02 23:38:33 +02:00
Viktor Szakats
3b3c1cff2e GHA/codeql: switch Linux job to arm64
For slightly shorter runs, better efficiency.

Closes #2646
2026-10-02 22:48:01 +02:00
filtede98
1f977e77f5 agent: add NULL guards to all public libssh2_agent_*() API functions
All public API functions in `src/agent.c` that take a `LIBSSH2_AGENT *`
were missing a NULL-pointer check at entry.

A caller that passes NULL (for example, after `libssh2_agent_init()`
returns NULL due to OOM) immediately dereferences the null pointer and
crashes.  Every other source file in this tree already has the pattern:
`channel.c`, `sftp.c`, `session.c`, `userauth.c`, and `knownhost.c` all
guard their primary handle before touching any field. `agent.c` was the
only outlier.

Add `if(!agent)` guards consistent with the style used in the rest of
the codebase.

Closes #2635
2026-10-02 18:57:37 +02:00
syed abdul khaliq
4f2692fd81 kex: zero-pad x25519 shared secret in mlkem768x25519
In `kex_mlkem768x25519_sha256` the x25519 half of the hybrid shared
secret is written into a fixed 64-byte buffer with `ssh2_bn_to_bin`,
which emits only the significant bytes and drops any leading zeros. When
the x25519 secret has a leading zero byte (about 1 in 256 handshakes,
and it comes from the server's ephemeral key) the write is short, so the
tail of the buffer is hashed while still uninitialized and the value
sits left-shifted instead of filling the fixed 32-byte field. That means
an uninitialized stack read into the exchange hash and a wrong K, which
fails the signature check for mlkem768x25519-sha256, the first KEX we
offer. This zero-fills the 32-byte field and writes the value
right-aligned so it is always the full fixed-width secret. The sibling
`kex_mlkem_nistp` sizes its heap buffer to the bignum length, so it has
no uninitialized read.

Bug possibly related: https://github.com/libssh2/libssh2/pull/1644#issuecomment-4761871304
Follow-up to 3ba252e2a6 #1644

Closes #2633
2026-10-02 18:57:37 +02:00
syed abdul khaliq
01d0c895fa wincng: fix ECC private key blob allocation size
`wcng_privatekey_from_point()` sizes the ECC private-key blob with
`sizeof(BCRYPT_ECCPRIVATE_BLOB)`, but that name is the wide blob-type
string `L"ECCPRIVATEBLOB"`, not a struct, so the `sizeof` measures the
string (30 bytes on Windows) rather than the 8-byte `BCRYPT_ECCKEY_BLOB`
header. The three `memcpy` offsets right below it, and the public-key
sibling `wcng_publickey_from_point()`, already use
`sizeof(BCRYPT_ECCKEY_BLOB)`, so the buffer ends up oversized and its
uninitialized tail is handed to `BCryptImportKeyPair()` with an inflated
`cbInput`. This is on the path that loads an OpenSSH-format ECDSA
private key through
`libssh2_userauth_publickey_frommemory()`/`fromfile()` on the CNG
backend.

Using `sizeof(BCRYPT_ECCKEY_BLOB)` lines the allocation up with the
header the copies actually write from and with the length passed to
`BCryptImportKeyPair()`. Keeping it in step with the public-key path
means the private blob is described to CNG with exactly the bytes it
holds.

Follow-up to 3e72343737 #1315

Closes #2631
2026-10-02 18:57:37 +02:00
Viktor Szakats
a95ed7d84a GHA: move some jobs to ubuntu-slim and arm64
For better efficiency.

Closes #2644
2026-10-02 12:20:45 +02:00
Viktor Szakats
bb63c2f52f GHA: run clang-tidy jobs on arm64 Linux
For 30-100% faster build steps, and more efficiency.

Closes #2643
2026-10-02 12:13:09 +02:00
Viktor Szakats
c7541acd4b tests/cmake: suppress stderr in if expression
Silencing, seen with MSYS2 shell:
```
+ ln -s ../.. libssh2
ln: failed to create symbolic link 'libssh2': Too many levels of symbolic links
+ rm -rf libssh2
```
Ref: https://github.com/libssh2/libssh2/actions/runs/36978012817/job/110745989451#step:15:40

Closes #2642
2026-10-02 11:11:29 +02:00
Viktor Szakats
b0028ca522 GHA: bump dependencies
Closes #2641
2026-10-02 11:11:29 +02:00
Viktor Szakats
b3e141ff4d GHA/distcheck: pass unix-style paths in MSYS2 configurations
Closes #2640
2026-10-02 10:42:04 +02:00
Viktor Szakats
244b422835 GHA: set cache-mode: none where missing
Follow-up to 917f38bfe3 #2601

Closes #2639
2026-10-02 09:11:39 +02:00
Viktor Szakats
5fa500d58e GHA: bump msys2/setup-msys2 to v2.33.0, related improvements
Also:
- enable MSYS2 cache for both curl tests.
- inherit pre-installed MSYS2 in the ARM64 curl job.
- bump integration tests from mingw64 to ucrt64 env.
- switch deprecated mingw64 jobs to ucrt64/clang64.
  (Keep one mingw64 and one mingw32 job.)
- silence mingw64/mingw32 deprecation warning.

Closes #2638
2026-10-02 00:32:05 +02:00
Viktor Szakats
805317f6eb GHA/windows: move Linux cross-builds to arm64 runners
For slightly better performance, and better efficiency.
Saving over 1 minute. The clang-tidy build is 2x fast.

Also:
- GHA/windows: fix `windres` command-line for non-intel machines.

Closes #2637
2026-10-01 23:44:49 +02:00
Viktor Szakats
9bf9048bee GHA: drop actionlint, abandoned upstream
No release in 6 months, no commit in 5 months. Lots of debt and a
matching number of local workarounds.

Ref: https://github.com/rhysd/actionlint
Follow-up to f61703cc98 #1740

Closes #2636
2026-10-01 23:44:49 +02:00
dependabot[bot]
788ee6d83a GHA: bump Actions
- update `docker/build-push-action` from 7.3.0 to 7.4.0
- update `github/codeql-action/analyze` from 4.37.8 to 4.38.2
- update `github/codeql-action/init` from 4.37.8 to 4.38.2

Closes #2634
2026-10-01 23:17:57 +02:00
Viktor Szakats
2e1717456b GHA: omit running tests in clang-tidy Linux jobs
To reduce CI runtimes and total CI turnaround time. clang-tidy jobs were
often the longest running ones before this patch.

Closes #2627
2026-09-14 11:54:32 +02:00
Viktor Szakats
c3de8626c1 CI: use --parallel for ctest
Go with 8-10 paralellism, which seems to be stable. If it introduces
flakiness, it possible to settle on ctest defaults, or low manual value.

It makes the test step take 2x-3x less time. Typically from 1m/1m30s to
~30s.

Also enable in AppVeyor, using the the CMake-default parallel value,
subject for refinement if we ever run tests there again.

Before: https://github.com/libssh2/libssh2/actions/runs/34788327227
After: https://github.com/libssh2/libssh2/actions/runs/34825431130?pr=2625

Closes #2625
2026-09-14 11:22:52 +02:00
Viktor Szakats
d702b04352 GHA: reduce parallelism to -j4 in macOS runners
To make builds finish a little bit faster, and to match the value used
in curl and the cores offered by these machines. (Was: `-j5`)

Closes #2626
2026-09-14 10:15:01 +02:00
Viktor Szakats
2a76dafa9c src: disable more RSA SHA1 code when unused
Also:
- os400qc3: fix an RSA guard.

Closes #2624
2026-09-14 00:58:11 +02:00
Viktor Szakats
738c2ac4d1 mbedtls: fix -Wunused-variable when building with LIBSSH2_NO_RSA
Fixing:
```
src/mbedtls.c:721:24: error: unused variable 'method_buf_len' [-Wunused-variable]
  721 |     size_t keylen = 0, method_buf_len = 0;
      |                        ^~~~~~~~~~~~~~
src/mbedtls.c:717:56: error: unused parameter 'pkey' [-Wunused-parameter]
  717 |                              const mbedtls_pk_context *pkey)
      |                                                        ^
```

Closes #2623
2026-09-14 00:28:54 +02:00
Viktor Szakats
45aa5c5fa4 tests: dump container sshd log for test failures
To help debugging issues.

Also:
- bump sshd LogLevel to DEBUG1, for essential details.
- bump command output buffer in `run_command_varg()`
- drop redundant condition in `run_command_varg()`
- replace `BUFSIZ` with literal sizes in `run_command_varg()`.

After this patch it's showing this in the CI log for the libgcrypt
fallout that previously needed a long time to diagnose:
```
61: ---- sshd log ----
61: No matching private key for certificate: /etc/ssh/ssh_host_mldsa44-ed25519_key-cert.pub
61: Server listening on 0.0.0.0 port 22.
61: Server listening on :: port 22.
61: debug1: sshd-session version OpenSSH_10.5, OpenSSL 3.6.3 9 Jun 2026
61: Connection from 172.17.0.1 port 51878 on 172.17.0.2 port 22 rdomain ""
61: debug1: sshd-auth version OpenSSH_10.5, OpenSSL 3.6.3 9 Jun 2026 [preauth]
61: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,rsa-sha2-512-cert-v01@openssh.com,
rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,
ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com [preauth]
61: debug1: Local version string SSH-2.0-OpenSSH_10.5p1 Debian-1 [preauth]
61: debug1: Remote protocol version 2.0, remote software version libssh2_1.11.2_DEV [preauth]
61: debug1: compat_banner: no match: libssh2_1.11.2_DEV [preauth]
61: debug1: SSH2_MSG_KEXINIT sent [preauth]
61: debug1: SSH2_MSG_KEXINIT received [preauth]
61: debug1: kex: algorithm: (no match) [preauth]
61: Unable to negotiate with 172.17.0.1 port 51878: no matching key exchange method found.
Their offer: diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,
diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,
diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1,ext-info-c,
kex-strict-c-v00@openssh.com [preauth]
[...]
61: ------------------
```

Follow-up to 1d0b86ac84 #2484
Follow-up to 8be5716437 #2174
Follow-up to 51f6259d18 #1720

Closes #2621
2026-09-13 21:20:56 +02:00
Viktor Szakats
418c2a2a67 openssh_fixture: merge Apple container commands with others
Use the full `image pull`, `image tag` commands for docker and podman,
to make them all use the same commands.

Ref:
https://docs.docker.com/reference/cli/docker/image/pull/
https://docs.docker.com/reference/cli/docker/image/tag/

Follow-up to 9bd1f16dc2 #2605

Closes #2622
2026-09-13 18:52:20 +02:00
Viktor Szakats
1061e299f2 openssh_fixture: drop progress from image pull with Apple container
Follow-up to 9bd1f16dc2 #2605

Closes #2620
2026-09-13 18:10:47 +02:00
Viktor Szakats
14fbba968b tidy-up: prefer host key in two words for human-readable text
Closes #2618
2026-09-13 16:12:32 +02:00
Viktor Szakats
967a24b3bb tidy-up: miscellaneous
- wincng: add version comments.
  Follow-up to 7a26697ede #397
- wincng: fix two version support comments.
- tests/keys-generate.sh: add comment, newlines.
- mbedtls: casing in comment.
- wording: use `null-terminator` more.
- typos: `ssh-dss`.
- knownhost: drop obsolete comment.
- include/libssh2.h: clarify comments.
- include/libssh2.h: drop empty lines.
- libssh2_knownhost_writeline.md: fix grammar.
- GHA: reorder Cygwin settings, improve package search URL.
- fixup newlines.

Closes #2617
2026-09-13 16:12:32 +02:00
Viktor Szakats
f00f16e399 RELEASE-NOTES: sync [ci skip] 2026-09-13 15:03:52 +02:00
Viktor Szakats
f1572b2888 GHA/openssh_server: drop docker login Action, limit scope
In favor of direct execution or `docker login` and `docker logout`.

Also to limit the logged-in time to the steps the require it, and to
avoid passing a secret via the command-line.

Closes #2616
2026-09-13 14:50:26 +02:00
Viktor Szakats
520087ace4 spellcheck.sh: exclude test keys to avoid false positives
Follow-up to 5c50ea8a5d #2615
2026-09-13 14:28:02 +02:00
Viktor Szakats
781b6ca8a5 sshd_config: drop accidental ML-DSA change [ci skip]
Follow-up to 5c50ea8a5d #2615
2026-09-13 14:28:02 +02:00
Viktor Szakats
5c50ea8a5d tests: add ML-DSA host keys (not used or supported by libssh2)
Requires OpenSSH 10.3. As of 10.5, to enable in sshd, this patch needs
to be applied:

```diff
--- a/tests/openssh_server/sshd_config
+++ b/tests/openssh_server/sshd_config
@@ -3,9 +3,9 @@
 # SPDX-License-Identifier: BSD-3-Clause

 CASignatureAlgorithms +ssh-rsa
-HostKeyAlgorithms +ssh-rsa,ssh-rsa-cert-v01@openssh.com
+HostKeyAlgorithms +ssh-rsa,ssh-rsa-cert-v01@openssh.com,ssh-mldsa44-ed25519@openssh.com,ssh-mldsa44-ed25519-cert-v01@openssh.com
 # modern name in OpenSSH 8.5+: PubkeyAcceptedAlgorithms
-PubkeyAcceptedKeyTypes +ssh-rsa,ssh-rsa-cert-v01@openssh.com
+PubkeyAcceptedKeyTypes +ssh-rsa,ssh-rsa-cert-v01@openssh.com,ssh-mldsa44-ed25519@openssh.com,ssh-mldsa44-ed25519-cert-v01@openssh.com
 MACs +hmac-sha1,hmac-sha1-96,hmac-sha2-256,hmac-sha2-512,hmac-md5,hmac-md5-96,umac-64@openssh.com,umac-128@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-md5-etm@openssh.com,hmac-md5-96-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com
 Ciphers +3des-cbc,aes128-cbc,aes192-cbc,aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
 # for libgcrypt, WinCNG
```
2026-09-13 13:44:17 +02:00
Viktor Szakats
e3f1853298 Dockerfile: copy ssh_host_*.pub keys to avoid sshd warnings
To overwrite default ones with the ones matching the copied private
keys.

Fixing in sshd log:
```
Public key for /etc/ssh/ssh_host_rsa_key does not match private key
Public key for /etc/ssh/ssh_host_ecdsa_key does not match private key
Public key for /etc/ssh/ssh_host_ed25519_key does not match private key
```

Closes #2614
2026-09-13 13:09:52 +02:00
Viktor Szakats
7bf89e2bb8 Dockerfile: drop repeating source filenames in the destination
Closes #2613
2026-09-13 12:40:48 +02:00
Viktor Szakats
7498e661b4 test_sshd.test: add RSA and ECDSA HostCertificate options
Follow-up to c704efcc7d #2539

Closes #2612
2026-09-13 12:40:48 +02:00
Viktor Szakats
86dca4efea openssh_fixture: drop docker-machine ip logic
Also drop using the `DOCKER_MACHINE_NAME` env which is part of the logic
deleted.

The `docker-machine` command and package no longer exists upstream. It's
missing from distros, its last commit was on 2019-09-02, repo archived
on 2025-07-18.

Refs:
https://github.com/docker-archive-public/docker.machine
https://github.com/docker/roadmap/issues/245
https://docs.docker.com/retired/

Follow-up to 2ded2f95c5 #2609
Follow-up to cf80f2f4b5 #100

Closes #2611
2026-09-13 12:08:11 +02:00
Viktor Szakats
ff182c52f3 GHA: bump cross-platform-actions, FreeBSD 14.5, enable DragonFly BSD
Try DragonFly BSD again, after upstream fix.

Ref: https://github.com/cross-platform-actions/action/releases/tag/v1.6.0

Follow-up to e4f700e934 #1948

Closes #2610
2026-09-13 11:54:32 +02:00
Viktor Szakats
2ded2f95c5 tidy-up: prefer term 'container' over 'docker'
To reflect the support for multiple container engines, and to make
terminology more consistent and implementation-agnostic.

Also:
- add FIXME for `docker-machine ip` command that has been dropped
  upstream.

Follow-up to 9bd1f16dc2 #2605

Closes #2609
2026-09-13 11:54:32 +02:00
Viktor Szakats
9bd1f16dc2 tests: add support for podman, Apple container
- add `FIXTURE_CONTAINER_CMD` to the container command. Supported:
  `podman`, `container`, `docker` (default).
- deprecate `OPENSSH_NO_DOCKER` env. Replaced by setting
  `FIXTURE_CONTAINER_CMD` to an empty string.
- GHA: replace Docker with `podman` in a some Linux jobs.
- openssh_fixture: fixup `inspect` commands to be compatible with both
  docker and podman.
- openssh_fixture: add `inspect` fallback for podman, which does not
  support `HostIp` before 6.1.0.
  Ref: https://github.com/podman-container-tools/podman/issues/29164
- openssh_fixture: add Apple container alternatives for `pull` and
  `tag`.
- openssh_fixture: fix `run` command to suppress progress output when
  using Apple container.
- openssh_fixture: if `OPENSSH_SERVER_PORT` is set, omit querying
  the port number via the container tool.

`podman` seems to be a tiny bit slower than Docker in CI.

Requirements for Apple container:
- Apple container 0.8.0+ (2026-01-02). Tested with 1.4.1.
- `jq` installed in PATH.
- setting the sshd port manually on the host via `OPENSSH_SERVER_PORT`
  (`container` cannot automatically assign one).
- physical macOS machine.

Example to run conainerized tests on macOS:
```sh
brew install container jq
container system start --disable-kernel-install
container system kernel set --recommended
export FIXTURE_CONTAINER_CMD=container
export OPENSSH_SERVER_PORT=<free-port-number>
export OPENSSH_SERVER_IMAGE; OPENSSH_SERVER_IMAGE=ghcr.io/libssh2/ci_tests_openssh_server:$(git rev-parse --short=20 HEAD:tests/openssh_server)
ctest -VV
```

Also tried:
- using Apple container in CI. Does not work because nested
  virtualization is not supported in GHA runners.
- installing WSL on Windows. Failed with cryptic error upon
  `wsl --install --distribution Debian`.

Closes #2605
2026-09-13 10:34:14 +02:00
Viktor Szakats
ca8864db09 GHA: switch a Linux job (with tests) to arm64
Follow-up to 667f7cac77
Follow-up to 7d9d0f8db5 #2606

Closes #2607
2026-09-12 22:50:21 +02:00
Viktor Szakats
667f7cac77 GHA/openssh_server: try with qemu-user
Ref: https://packages.ubuntu.com/resolute/qemu-user

Follow-up to 7d9d0f8db5 #2606
2026-09-12 19:51:28 +02:00
Viktor Szakats
7d9d0f8db5 openssh_server: build container for arm64 also, fix Dockerfile for trixie
To enable testing on arm64 Linux and possibly also on macOS, and
possibly possibly on arm64 Windows.

Also:
- switch to debian:testing to trigger a container rebuild.
- Dockerfile: install `adduser` explicitly. Required for debian:forky.
- Dockerfile: stop installing suggested/recommended packages.

Ref: https://github.com/docker/build-push-action

Closes #2606
2026-09-12 19:24:57 +02:00
Viktor Szakats
b937ded9f0 tests: replace DSA with RSA key for fail test
To use a key type that is:
- supported by all backends.
- can be generated with the most recent `ssh-keygen` version.
- here to stay for the foreseeable future.
  (DSA is scheduled for removal from the libssh2 codebase.)

Cherry-picked from #2598

Closes #2604
2026-09-12 01:36:44 +02:00
Viktor Szakats
5bfa574460 tests: drop WinCNG retry workarounds
Should no longer be necessary after fixing the root cause.
(Though we do not run these tests in CI at the moment.)

Ref: #804
Follow-up to f216bbe3aa #2583
Follow-up to bc120a343b #1023

Closes #2603
2026-09-12 00:47:41 +02:00
Viktor Szakats
917f38bfe3 GHA: use cache-mode
Limit cache access to the minimum required.

Refs:
https://github.blog/changelog/2026-09-10-control-github-actions-cache-access-with-cache-mode/
https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#cache-mode
https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#bypassing-the-default-untrusted-trigger-cache-restriction

Closes #2601
2026-09-12 00:27:39 +02:00
Viktor Szakats
1a1e62c548 GHA: drop redundant shellcheck expression filter
GHA expressions are no longer used and no longer allowed in GHA scripts.

Follow-up to d7cf63bb05 #1609

Closes #2602
2026-09-11 23:36:20 +02:00
Viktor Szakats
3f26c15b2f wincng: fix builds with RSA and DSA disabled
Fixing:
```
x86_64-w64-mingw32-ld: src/CMakeFiles/libssh2_shared.dir/Unity/unity_0_c.c.obj: in function `userauth_hostbased_fromfile':
src/userauth.c:923:(.text+0x34c29): undefined reference to `ssh2_pub_privkey'
x86_64-w64-mingw32-ld: src/CMakeFiles/libssh2_shared.dir/Unity/unity_0_c.c.obj: in function `userauth_publickey':
src/userauth.c:1745:(.text+0x37171): undefined reference to `ssh2_pub_privkey'
collect2: error: ld returned 1 exit status
```

Note: This fixes the build, but leaves `ssh2_pub_privkey()` a no-op, because it's not implemented for ECDSA.

Closes #2600
2026-09-11 20:36:00 +02:00
Viktor Szakats
65a9333204 docs: update list of key types where necessary
Closes #2599
2026-09-11 18:46:57 +02:00
Viktor Szakats
bb8e0957b8 GHA: test WinCNG SCP/SFTP via curl runtests
The WinCNG backend has been missing runtime tests for a while. To
restore some level of testing, this PR make use of curl's regression
test suite (`runtests`) to run SCP/SFTP-specific tests with the
master/PR libssh2 revision at hand. Some tests need to be disabled when
using the built-in Windows OpenSSH server.

The current CI jobs reproduce the long-running WinCNG flaky failure
reported in #804 reliability.

Note, if these CI jobs make libssh2 CI a pain to use, they will be
removed. This has a reasonably fair chance to happen, based on the
experience with curl CI with Windows.

Also:
- test both ECDH and non-ECDH builds.
- test build on x64 and ARM64.
- test with both Windows built-in OpenSSH port and MSYS2's sshd.
- enable debug logging in libssh2 while running the tests.
- for extra stress, run each of the 56-57 tests 6 times.
  This may be lowered in a future commit to avoid flakiness and reduce
  CI load and job times.

Caveats:
- as seen in curl CI, runtests may result in hangs due to broken pipes
  and error 2304 for undiscovered reasons.
  Discussion: https://github.com/curl/curl/discussions/14854
- MSBuild has an undisablable "smart" feature that sometimes marks a
  successful run a failure.
  Refs:
  a477789c43
  https://github.com/curl/curl/pull/18301
- these tests are different in scope than libssh2's own.
- tests sometime fail flaky with:
  ```
  Failure Event: -43 - Failed getting banner
  ```
  Ref: https://github.com/libssh2/libssh2/actions/runs/34134974899/job/101783681252?pr=2590
- running on modern runner machines, the Win10-introduced DH algo is
  always enabled.

Ref: https://github.com/libssh2/libssh2/discussions/2049
Ref: #2583

Closes #2590
2026-09-08 20:42:06 +02:00
lv2dev
f216bbe3aa wincng: normalize fixed-width DH values before mpint encoding
- normalize fixed-width WinCNG bignums before SSH `mpint` serialization
- apply the normalization to DH public values and DH/ECDH shared secrets
- zero the removed tail bytes and add common bignum conversion tests

WinCNG returns DH public values and raw DH/ECDH shared secrets in
fixed-width buffers. The WinCNG bignum retained that buffer width as its
numeric length. When a value started with `0x00`, libssh2 included the
redundant byte in the exchange hash even though RFC 4251 requires a
canonical `mpint` representation.

The peer hashes the normalized integer, so the exchange hashes differ
and a valid server host-key signature is rejected. This appeared
intermittently as `LIBSSH2_ERROR_KEY_EXCHANGE_FAILURE` because it
depends on the generated value.

The helper introduced here moves significant bytes to the start of the
existing allocation, explicitly clears the removed tail, and updates
only the logical length. It does not add an allocation or a new failure
path.

Reported-by: Dan Fandrich
Fixes #804
Ref: #2590 (tests)

Closes #2583
2026-09-08 20:19:44 +02:00
Viktor Szakats
a951e06640 test_simple: make test_ssh2_dh_validate() portable
Omit direct backend-specific API use. Drop one test using negative
input, that's impossible with the internal crypto API. Using
backend-specific code in tests is highly undesired. It's also
discouraged to special-case any, in this case WinCNG had to be because
it does not implement the DH validation function.

Also:
- mbedtls: fix an internal API type difference with a cast.
- unfold a line.

Follow-up to fcf830788d #2239

Closes #2595
2026-09-07 16:44:32 +02:00
Viktor Szakats
f0ac38ab93 libgcrypt: fix ssh2_bn_set_word() return value
It was returning a non-NULL address on success, which translated to
permanent failure as mapped to this internal crypto API. Fix to match it
with the rest of crypto backends and the expectation of callers.

Reported by Copilot
Bug: https://github.com/libssh2/libssh2/pull/2595#discussion_r3950301182
Follow-up to 95c824d5d9 #1354
Follow-up to c9d40afa14

Closes #2596
2026-09-07 16:26:02 +02:00