src: silence compiler warnings 3 (change types)

Apply type changes to avoid casts and warnings. In most cases this
means changing to a larger type, usually `size_t` or `ssize_t`.

Change signedness in a few places.

Also introduce new variables to avoid reusing them for multiple
purposes, to avoid casts and warnings.

- add FIXME for public `libssh2_sftp_readdir_ex()` return type.

- fix `_libssh2_mbedtls_rsa_sha2_verify()` to verify if `sig_len`
  is large enough.

- fix `_libssh2_dh_key_pair()` in `wincng.c` to return error if
  `group_order` input is negative.

  Maybe we should also reject zero?

- bump `_libssh2_random()` size type `int` -> `size_t`. Add checks
  for WinCNG and OpenSSL to return error if requested more than they
  support (`ULONG_MAX`, `INT_MAX` respectively).

- change `_libssh2_ntohu32()` return value `unsigned int` -> `uint32_t`.

- fix `_libssh2_mbedtls_bignum_random()` to check for a negative `top`
  input.

- size down `_libssh2_wincng_key_sha_verify()` `hashlen` to match
  Windows'.

- fix `session_disconnect()` to limit length of `lang_len`
  (to 256 bytes).

- fix bad syntax in an `assert()`.

- add a few `const` to casts.

- `while(1)` -> `for(;;)`.

- add casts that didn't fit into #876.

- update `docs/HACKING-CRYPTO` with new sizes.

May need review for OS400QC3: /cc @monnerat @jonrumsey

See warning details in the PR's individual commits.

Cherry-picked from #846
Closes #879

src/libssh2_priv.h

@@ -410,7 +410,7 @@ struct _LIBSSH2_CHANNEL
     /* Amount of bytes to be refunded to receive window (but not yet sent) */
     uint32_t adjust_queue;
     /* Data immediately available for reading */
-    uint32_t read_avail;
+    size_t read_avail;
 
     LIBSSH2_SESSION *session;
 
@@ -567,7 +567,7 @@ struct transportpacket
     /* ------------- for outgoing data --------------- */
     unsigned char outbuf[MAX_SSH_PACKET_LEN]; /* area for the outgoing data */
 
-    int ototal_num;         /* size of outbuf in number of bytes */
+    ssize_t ototal_num;     /* size of outbuf in number of bytes */
     const unsigned char *odata; /* original pointer to the data */
     size_t olen;            /* original size of the data we stored in
                                outbuf */
@@ -850,8 +850,8 @@ struct _LIBSSH2_SESSION
     LIBSSH2_CHANNEL *sftpInit_channel;
     unsigned char sftpInit_buffer[9];   /* sftp_header(5){excludes request_id}
                                            + version_id(4) */
-    int sftpInit_sent; /* number of bytes from the buffer that have been
-                          sent */
+    size_t sftpInit_sent; /* number of bytes from the buffer that have been
+                             sent */
 
     /* State variables used in libssh2_scp_recv() / libssh_scp_recv2() */
     libssh2_nonblocking_states scpRecv_state;