Make OS/400 implementation work again (#953)

* os400: support QADRT development files in a non-standard directory

This enables the possibility to compile libssh2 even if the ascii
runtime development files are not installed system-wide.

* userauth_kbd_packet: fix a pointer target type mismatch.

A temporary variable matching the parameter type is used before copying
to the real target and checking for overflow (that should not occur!).

* os400qc3: move and fix big number procedures

A bug added by a previous code style cleaning is fixed.
_libssh2_random() now checks and return the success status.

* os400qc3: fix cipher definition block lengths

They were wrongly set to the key size.

* Diffie-Hellman min/max modulus sizes are dependent of crypto-backend

In particular, os400qc3 limits the maximum group size to 2048-bits.
Move definitions of these parameters to crypto backend header files.

* kex: return an error if Diffie-Hellman key pair generation fails

* os400: add an ascii assert.h header file

* os400qc3: implement RSA SHA2 256/512

src/openssl.h

@@ -419,6 +419,14 @@ libssh2_curve_type;
 #define _libssh2_bn_bits(bn) BN_num_bits(bn)
 #define _libssh2_bn_free(bn) BN_clear_free(bn)
 
+/* Default generate and safe prime sizes for
+   diffie-hellman-group-exchange-sha1 */
+#define LIBSSH2_DH_GEX_MINGROUP     2048
+#define LIBSSH2_DH_GEX_OPTGROUP     4096
+#define LIBSSH2_DH_GEX_MAXGROUP     8192
+
+#define LIBSSH2_DH_MAX_MODULUS_BITS 16384
+
 #define _libssh2_dh_ctx BIGNUM *
 #define libssh2_dh_init(dhctx) _libssh2_dh_init(dhctx)
 #define libssh2_dh_key_pair(dhctx, public, g, p, group_order, bnctx) \