src: silence compiler warnings 3 (change types)

Apply type changes to avoid casts and warnings. In most cases this
means changing to a larger type, usually `size_t` or `ssize_t`.

Change signedness in a few places.

Also introduce new variables to avoid reusing them for multiple
purposes, to avoid casts and warnings.

- add FIXME for public `libssh2_sftp_readdir_ex()` return type.

- fix `_libssh2_mbedtls_rsa_sha2_verify()` to verify if `sig_len`
  is large enough.

- fix `_libssh2_dh_key_pair()` in `wincng.c` to return error if
  `group_order` input is negative.

  Maybe we should also reject zero?

- bump `_libssh2_random()` size type `int` -> `size_t`. Add checks
  for WinCNG and OpenSSL to return error if requested more than they
  support (`ULONG_MAX`, `INT_MAX` respectively).

- change `_libssh2_ntohu32()` return value `unsigned int` -> `uint32_t`.

- fix `_libssh2_mbedtls_bignum_random()` to check for a negative `top`
  input.

- size down `_libssh2_wincng_key_sha_verify()` `hashlen` to match
  Windows'.

- fix `session_disconnect()` to limit length of `lang_len`
  (to 256 bytes).

- fix bad syntax in an `assert()`.

- add a few `const` to casts.

- `while(1)` -> `for(;;)`.

- add casts that didn't fit into #876.

- update `docs/HACKING-CRYPTO` with new sizes.

May need review for OS400QC3: /cc @monnerat @jonrumsey

See warning details in the PR's individual commits.

Cherry-picked from #846
Closes #879

src/crypto.h

@@ -85,8 +85,8 @@ int _libssh2_rsa_new_private(libssh2_rsa_ctx ** rsa,
                              unsigned const char *passphrase);
 int _libssh2_rsa_sha1_verify(libssh2_rsa_ctx * rsa,
                              const unsigned char *sig,
-                             unsigned long sig_len,
-                             const unsigned char *m, unsigned long m_len);
+                             size_t sig_len,
+                             const unsigned char *m, size_t m_len);
 int _libssh2_rsa_sha1_sign(LIBSSH2_SESSION * session,
                            libssh2_rsa_ctx * rsactx,
                            const unsigned char *hash,
@@ -103,8 +103,8 @@ int _libssh2_rsa_sha2_sign(LIBSSH2_SESSION * session,
 int _libssh2_rsa_sha2_verify(libssh2_rsa_ctx * rsa,
                              size_t hash_len,
                              const unsigned char *sig,
-                             unsigned long sig_len,
-                             const unsigned char *m, unsigned long m_len);
+                             size_t sig_len,
+                             const unsigned char *m, size_t m_len);
 #endif
 int _libssh2_rsa_new_private_frommemory(libssh2_rsa_ctx ** rsa,
                                         LIBSSH2_SESSION * session,
@@ -130,7 +130,7 @@ int _libssh2_dsa_new_private(libssh2_dsa_ctx ** dsa,
                              unsigned const char *passphrase);
 int _libssh2_dsa_sha1_verify(libssh2_dsa_ctx * dsactx,
                              const unsigned char *sig,
-                             const unsigned char *m, unsigned long m_len);
+                             const unsigned char *m, size_t m_len);
 int _libssh2_dsa_sha1_sign(libssh2_dsa_ctx * dsactx,
                            const unsigned char *hash,
                            unsigned long hash_len, unsigned char *sig);
@@ -246,7 +246,7 @@ int
 _libssh2_ed25519_new_public(libssh2_ed25519_ctx **ed_ctx,
                             LIBSSH2_SESSION *session,
                             const unsigned char *raw_pub_key,
-                            const uint8_t key_len);
+                            const size_t key_len);
 
 int
 _libssh2_ed25519_sign(libssh2_ed25519_ctx *ctx, LIBSSH2_SESSION *session,