libssh2/docs/TODO.md

Things TODO

• Fix -Wsign-conversion warnings in src.

• Fix the numerous malloc+copy operations for sending data, see "Buffering Improvements" below for details.

• make sure the windowing code adapts better to slow situations so that it does not then use as much memory as today. Possibly by an app-controllable "Window mode"?

• Decrease the number of mallocs. Everywhere. It gets easier once the buffering improvements have been done.

• Use SO_NOSIGPIPE for macOS/BSD systems where MSG_NOSIGNAL does not exist/work.

• Extend the test suite to actually test lots of aspects of libssh2.

• Update public API to drop casts added to fix compiler warnings.

• Expose error messages sent by the server.

• select() is troublesome with libssh2 when using multiple channels over the same session. See "New Transport API" below for more details.

• for obsolete/weak/insecure algorithms: either stop enabling them by default at build-time, or delete support for them completely.

At next SONAME bump

• stop using #defined macros as part of the official API. The macros should either be turned into real functions or discarded from the API.

• delete or deprecate libssh2_session_callback_set().

• bump length arguments in callback functions to size_t/ssize_t.

• remove the following functions from the API/ABI:

  libssh2_base64_decode()
  libssh2_session_flag() (needs replacement for LIBSSH2_FLAG_COMPRESS support)
  libssh2_channel_handle_extended_data()
  libssh2_channel_receive_window_adjust()
  libssh2_poll()
  libssh2_poll_channel_read()
  libssh2_scp_send()
  libssh2_scp_send_ex()
  libssh2_session_startup() (libssh2_session_handshake() is the replacement)
  libssh2_banner_set() (libssh2_session_banner_set() is the replacement)
  

• Rename a few functions:

  libssh2_hostkey_hash() => libssh2_session_hostkey_hash()
  libssh2_banner_set() => libssh2_session_banner_set()
  libssh2_sign_sk() => libssh2_userauth_sign_sk()
  

• change int to libssh2_socket_t in the public API for sockets.

• Use size_t for string lengths in all functions.

• Add a comment field to struct libssh2_knownhost.

• remove the existing libssh2_knownhost_add() function and rename libssh2_knownhost_addc to become the new libssh2_knownhost_add instead.

• remove the existing libssh2_scp_send_ex() function and rename libssh2_scp_send64 to become the new libssh2_scp_send instead.

• remove the existing libssh2_knownhost_check() function and rename libssh2_knownhost_checkp() to become the new libssh2_knownhost_check instead.

Buffering Improvements

• transport_write

  If this function gets called with a total packet size that is larger than 32K, it should create more than one SSH packet so that it keeps the largest one below 32K.

• sftp_write

  Should not copy/allocate anything for the data, only create a header chunk and pass on the payload data to channel_write "pointed to".

New Transport API

THE PROBLEM

The problem in a nutshell is that when an application opens up multiple channels over a single session, those are all using the same socket. If the application is then using select() to wait for traffic (like any sensible app does) and wants to act on the data when select() tells there is something to for example read, what does an application do?

With our current API, you have to loop over all the channels and read from them to see if they have data. This effectively makes blocking reads impossible. If the app has many channels in a setup like this, it even becomes slow. (The original API had the libssh2_poll_channel_read() and libssh2_poll() to somewhat overcome this hurdle, but they too have pretty much the same problems plus a few others.)

Traffic in the other direction is similarly limited: the app has to try sending to all channels, even though some of them may well not accept any data at that point.

A SOLUTION

I suggest we introduce two new helper functions:

• libssh2_transport_read()

  Read "a bunch" of data from the given socket and returns information to the app about what channels that are now readable (i.e. they do not block when read from). The function can be called over and over and it repeatedly returns info about what channels that are readable at that moment.

• libssh2_transport_write()

  Returns information about what channels that are writable, in the sense that they have windows set from the remote side that allows data to get sent. Writing to one of those channels does not block. Of course, the underlying socket may only accept a certain amount of data, so at the first short return, nothing more should be attempted to get sent until select() (or equivalent) has been used on the master socket again.

I have not yet figured out a sensible API for how these functions should return that info, but if we agree on the general principles I guess we can work that out.

VOLUNTARY

I wanted to mention that these two helper functions would not be mandatory in any way. They would be there for those who want them, and existing programs can remain using the old functions only if they prefer to.

New SFTP API

PURPOSE

Provide API functions that explicitly tells at once that a (full) SFTP file transfer is wanted, to allow libssh2 to leverage on that knowledge to speed up things internally. It can for example do read ahead, buffer writes (merge small writes into larger chunks), better tune the SSH window and more. This sort of API is already provided for SCP transfers.

API

New functions:

LIBSSH2_SFTP_HANDLE *libssh2_sftp_send(SFTP_SESSION *sftp,
                                       libssh2_uint64_t filesize,
                                       char *remote_path,
                                       size_t remote_path_len,
                                       long mode);

Tell libssh2 that a local file with a given size is about to get sent to the SFTP server.

LIBSSH2_SFTP_HANDLE *libssh2_sftp_recv();

Tell libssh2 that a remote file is requested to get downloaded from the SFTP server.

Only the setup of the file transfer is different from an application's point of view. Depending on direction of the transfer(s), the following already existing functions should then be used until the transfer is complete:

• libssh2_sftp_read()
• libssh2_sftp_write()

HOW TO USE

1. Setup the transfer using one of the two new functions.
2. Loop through the reading or writing of data.
3. Cleanup the transfer