Implement picky warnings with clang in autotools. Extend picky gcc
warnings, sync them between build tools and compilers and greatly
speed up detection in CMake.
- autotools: enable clang compiler warnings with `--enable-debug`.
- autotools: enable more gcc compiler warnings with `--enable-debug`.
- autotools/cmake: sync compiler warning options between gcc and clang.
- sync compiler warning options between autotools and cmake.
- cmake: reduce option-checks to speed up the detection phase.
Bring them down to 3 (from 35). Leaving some checks to keep the
CMake logic alive and for an easy way to add new options.
clang 3.0 (2011-11-29) and gcc 2.95 (1999-07-31) now required.
- autotools logic copied from curl, with these differences:
- delete `-Wimplicit-fallthrough=4` due to a false positive.
- reduce `-Wformat-truncation=2` to `1` due to a false positive.
- simplify MinGW detection for `-Wno-pedantic-ms-format`.
- cmake: show enabled picky compiler options (like autotools).
- cmake: do compile `tests/simple.c` and `tests/ssh2.c`.
- fix new compiler warnings.
- `tests/CMakeLists.txt`: fix indentation.
Original source of autotools logic:
- a8fbdb461c/acinclude.m4
- a8fbdb461c/m4/curl-compilers.m4
Notice that the autotools implementation considers Apple clang as
legacy clang 3.7. CMake detection works more accurately, at the same
time more error-prone and difficult to update due to the sparsely
documented nature of Apple clang option evolution.
Closes#952
Before this patch, with debug logging disabled, libssh2 code used a
variadic macro to catch `_libssh2_debug()` calls, and convert them to
no-ops. In certain conditions, it used an empty inline function instead.
Variadic macro is a C99 feature. It means that depending on compiler,
and build settings, it littered the build log with warnings about this.
The new solution uses the trick of passing the variable arg list as a
single argument and pass that down to the debug function with a regular
macro. When disabled, another regular C89-compatible macro converts it
to a no-op.
This makes inlining, C99 variadic macros and maintaining the conditions
for each unnecessary and also makes the codebase compile more
consistently, e.g. with forced C standards and/or picky warnings.
TL;DR: It makes this feature C89-compliant.
Use checksrc.pl from the curl project, with (for now)
suppressed long line warnings and indentation set to
4 spaces. Fixes are whitespace for the most part.
Warning count went down from 2704 to 12.
Also fix codespell typos, two non-ANSI C89 comments
and a stray tab in include/libssh2.h.
Ref: https://github.com/libssh2/libssh2/pull/235
Deflate may return Z_OK even when not all data has been compressed
if the output buffer becomes full.
In practice this is very unlikely to happen because the output buffer
size is always some KBs larger than the size of the data passed for
compression from the upper layers and I think that zlib never expands
the data so much, even on the worst cases.
Anyway, this patch plays on the safe side checking that the output
buffer is not exhausted.
Signed-off-by: Salvador <sfandino@yahoo.com>
The old algorithm was O(N^2), causing lots and lots of reallocations
when highly compressed data was transferred.
This patch implements a simpler one that just doubles the buffer size
everytime it is exhausted. It results in O(N) complexity.
Also a smaller inflate ratio is used to calculate the initial size (x4).
Signed-off-by: Salvador <sfandino@yahoo.com>
Data may remain in zlib internal buffers when inflate() returns Z_OK
and avail_out == 0. In that case, inflate has to be called again.
Also, once all the data has been inflated, it returns Z_BUF_ERROR to
signal that the input buffer has been exhausted.
Until now, the way to detect that a packet payload had been completely
decompressed was to check that no data remained on the input buffer
but that didn't account for the case where data remained on the internal
zlib buffers.
That resulted in packets not being completely decompressed and the
missing data reappearing on the next packet, though the bug was masked
by the buffer allocation algorithm most of the time and only manifested
when transferring highly compressible data.
This patch fixes the zlib usage.
Signed-off-by: Salvador <sfandino@yahoo.com>
Add a "use_in_auth" flag to the LIBSSH2_COMP_METHOD struct and a
separate "zlib@openssh.com" method, along with checking session->state
for LIBSSH2_STATE_AUTHENTICATED. Appears to work on the OpenSSH servers
I've tried against, and it should work as before with normal zlib
compression.
When using libssh2 to perform an SFTP file transfer from the "JSCAPE MFT
Server" (http://www.jscape.com) the transfer failed. The default JSCAPE
configuration is to enforce zlib compression on SSH2 sessions so the
session was compressed. The relevant part of the debug trace contained:
[libssh2] 1.052750 Transport: unhandled zlib error -5
[libssh2] 1.052750 Failure Event: -29 - decompression failure
The trace comes from comp_method_zlib_decomp() in comp.c. The "unhandled
zlib error -5" is the status returned from the zlib function
inflate(). The -5 status corresponds to "Z_BUF_ERROR".
The inflate() function takes a pointer to a z_stream structure and
"inflates" (decompresses) as much as it can. The relevant fields of the
z_stream structure are:
next_in - pointer to the input buffer containing compressed data
avail_in - the number of bytes available at next_in
next_out - pointer to the output buffer to be filled with uncompressed
data
avail_out - how much space available at next_out
To decompress data you set up a z_stream struct with the relevant fields
filled in and pass it to inflate(). On return the fields will have been
updated so next_in and avail_in show how much compressed data is yet to
be processed and next_out and avail_out show how much space is left in
the output buffer.
If the supplied output buffer is too small then on return there will be
compressed data yet to be processed (avail_in != 0) and inflate() will
return Z_OK. In this case the output buffer must be grown, avail_out
updated and inflate() called again.
If the supplied output buffer was big enough then on return the
compressed data will have been exhausted (avail_in == 0) and inflate()
will return Z_OK, so the data has all been uncompressed.
There is a corner case where inflate() makes no progress. That is, there
may be unprocessed compressed data and space available in the output
buffer and yet the function does nothing. In this case inflate() will
return Z_BUF_ERROR. From the zlib documentation and the source code it
is not clear under what circumstances this happens. It could be that it
needs to write multiple bytes (all in one go) from its internal state to
the output buffer before processing the next chunk of input but but
can't because there is not enough space (though my guesses as to the
cause are not really relevant). Recovery from Z_BUF_ERROR is pretty
simple - just grow the output buffer, update avail_out and call
inflate() again.
The comp_method_zlib_decomp() function does not handle the case when
inflate() returns Z_BUF_ERROR. It treats it as a non-recoverable error
and basically aborts the session.
Fixes#240
In the transport functions we avoid a strcmp() now and just check a
boolean instead.
The compress/decompress function's return code is now acknowledged and
used as actual return code in case of failures.
In KEXINIT messages, the client and server agree on, among other
things, whether to use compression. This method agreement occurs
in src/kex.c's kex_agree_methods() function. However, if
compression is enabled (either client->server, server->client, or
both), then the compression layer is initialized in
kex_agree_methods() -- before NEWKEYS has been received.
Instead, the initialization of the compression layer should
happen after NEWKEYS has been received. This looks to occur
insrc/kex.c's diffie_hellman_sha1(), which even has the comment:
/* The first key exchange has been performed,
switch to active crypt/comp/mac mode */
There, after NEWKEYS is received, the cipher and mac algorithms
are initialized, and that is where the compression should be
initialized as well.
The current implementation fails if server->client compression is
enabled because most server implementations follow OpenSSH's
lead, where compression is initialized after NEWKEYS. Since the
server initializes compression after NEWKEYS, but libssh2
initializes compression after KEXINIT (i.e. before NEWKEYS), they
are out of sync.
Reported in bug report #180
We reserve ^libssh2_ for public symbols and we use _libssh2 as
prefix for internal ones. I fixed the intendation of all these
edits with emacs afterwards, which then changed it slightly more
than just _libssh2_error() expressions but I didn't see any
obvious problems.
libssh2_error() no longer allocates a string and only accepts a const
error string. I also made a lot of functions use the construct of
return libssh2_error(...) instead of having one call to
libssh2_error() and then a separate return call. In several of those
cases I then also changed the former -1 return code to a more
detailed one - something that I think will not change behaviors
anywhere but it's worth keeping an eye open for any such.
the recent commits converted the tabs to 4 spaces, which matched the
initial indent size. Other commits converted the tabs to 8 spaces, this
didn't match.
All the code has been converted to 4 space indents. No changes to line
lengths or actual code was performed. This is in preperation to my up
coming non-blocking work so my commits should only be code changes and
line lengths in the code I am working on.
