lib/libssh
1
0
Fork 0
mirror of https://git.libssh.org/projects/libssh.git synced 2025-07-28 01:41:48 +03:00
Code Activity

Add tutorial for the sftp aio API

Browse Source 
Signed-off-by: Eshan Kelkar <eshankelkar@galorithm.com>
Reviewed-by: Sahana Prasad <sahana@redhat.com>
Reviewed-by: Jakub Jelen <jjelen@redhat.com>
This commit is contained in:
Eshan Kelkar
2023-09-04 15:58:02 +05:30
committed by Sahana Prasad
parent 12f28a519b
commit d1960cb9a2
2 changed files with 577 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
doc/introduction.dox
View File

@ -46,6 +46,8 @@ Table of contents:
@subpage libssh_tutor_pkcs11
@subpage libssh_tutor_sftp_aio
@subpage libssh_tutor_todo
*/

575
doc/sftp_aio.dox Normal file
View File

@ -0,0 +1,575 @@
/**
@page libssh_tutor_sftp_aio Chapter 10: The SFTP asynchronous I/O
@section sftp_aio_api The SFTP asynchronous I/O
NOTE : Please read @ref libssh_tutor_sftp before reading this page. The
synchronous sftp_read() and sftp_write() have been described there.
SFTP AIO stands for "SFTP Asynchronous Input/Output". This API contains
functions which perform async read/write operations on remote files.
File transfers performed using the asynchronous sftp aio API can be
significantly faster than the file transfers performed using the synchronous
sftp read/write API (see sftp_read() and sftp_write()).
The sftp aio API functions are divided into two categories :
- sftp_aio_begin_*() [see sftp_aio_begin_read(), sftp_aio_begin_write()]:
These functions send a request for an i/o operation to the server and
provide the caller an sftp aio handle corresponding to the sent request.
- sftp_aio_wait_*() [see sftp_aio_wait_read(), sftp_aio_wait_write()]:
These functions wait for the server response corresponding to a previously
issued request. Which request ? the request corresponding to the sftp aio
handle supplied by the caller to these functions.
Conceptually, you can think of the sftp aio handle as a request identifier.
Technically, the sftp_aio_begin_*() functions dynamically allocate memory to
store information about the i/o request they send and provide the caller a
handle to this memory, we call this handle an sftp aio handle.
sftp_aio_wait_*() functions use the information stored in that memory (handled
by the caller supplied sftp aio handle) to identify a request, and then they
wait for that request's response. These functions also release the memory
handled by the caller supplied sftp aio handle (except when they return
SSH_AGAIN).
sftp_aio_free() can also be used to release the memory handled by an sftp aio
handle but unlike the sftp_aio_wait_*() functions, it doesn't wait for a
response. This should be used to release the memory corresponding to an sftp
aio handle when some failure occurs. An example has been provided at the
end of this page to show the usage of sftp_aio_free().
To begin with, this tutorial will provide basic examples that describe the
usage of sftp aio API to perform a single read/write operation.
The later sections describe the usage of the sftp aio API to obtain faster file
transfers as compared to the transfers performed using the synchronous sftp
read/write API.
On encountering an error, the sftp aio API functions set the sftp and ssh
errors just like any other libssh sftp API function. These errors can be
obtained using sftp_get_error(), ssh_get_error() and ssh_get_error_code().
The code examples provided on this page ignore error handling for the sake of
brevity.
@subsection sftp_aio_read Using the sftp aio API for reading (a basic example)
For performing an async read operation on a sftp file (see sftp_open()),
the first step is to call sftp_aio_begin_read() to send a read request to the
server. The caller is provided an sftp aio handle corresponding to the sent
read request.
The second step is to pass a pointer to this aio handle to
sftp_aio_wait_read(), this function waits for the server response which
indicates the success/failure of the read request. On success, the response
indicates EOF or contains the data read from the sftp file.
The following code example shows how a read operation can be performed
on an sftp file using the sftp aio API.
@code
ssize_t read_chunk(sftp_file file, void *buf, size_t to_read)
{
ssize_t bytes_read;
int rc;
// Variable to store an sftp aio handle
sftp_aio aio = NULL;
// Send a read request to the sftp server
rc = sftp_aio_begin_read(file, to_read, &aio);
if (rc == SSH_ERROR) {
// handle error
}
// Wait for the response of the read request corresponding to the
// sftp aio handle stored in the aio variable.
bytes_read = sftp_aio_wait_read(&aio, buf, to_read);
if (bytes_read == SSH_ERROR) {
// handle error
}
return bytes_read;
}
@endcode
@subsection sftp_aio_write Using the sftp aio API for writing (a basic example)
For performing an async write operation on a sftp file (see sftp_open()),
the first step is to call sftp_aio_begin_write() to send a write request to
the server. The caller is provided an sftp aio handle corresponding to the
sent write request.
The second step is to pass a pointer to this aio handle to
sftp_aio_wait_write(), this function waits for the server response which
indicates the success/failure of the write request.
The following code example shows how a write operation can be performed on an
sftp file using the sftp aio API.
@code
ssize_t write_chunk(sftp_file file, void *buf, size_t to_write)
{
ssize_t bytes_written;
int rc;
// Variable to store an sftp aio handle
sftp_aio aio = NULL;
// Send a write request to the sftp server
rc = sftp_aio_begin_write(file, buf, to_write, &aio);
if (rc == SSH_ERROR) {
// handle error
}
// Wait for the response of the write request corresponding to
// the sftp aio handle stored in the aio variable.
bytes_written = sftp_aio_wait_write(&aio);
if (bytes_written == SSH_ERROR) {
// handle error
}
return bytes_written;
}
@endcode
@subsection sftp_aio_actual_use Using the sftp aio API to speed up a transfer
The above examples were provided to introduce the sftp aio API.
This is not how the sftp aio API is intended to be used, because the
above usage offers no advantage over the synchronous sftp read/write API
which does the same thing i.e issue a request and then immediately wait for
its response.
The facility that the sftp aio API provides is that the user can do
anything between issuing a request and getting the corresponding response.
Any number of operations can be performed after calling sftp_aio_begin_*()
[which issues a request] and before calling sftp_aio_wait_*() [which waits
for a response]
The code can leverage this feature by calling sftp_aio_begin_*() multiple times
to issue multiple requests before calling sftp_aio_wait_*() to wait for the
response of an earlier issued request. This approach will keep a certain number
of requests outstanding at the client side.
After issuing those requests, while the client code does something else (for
example waiting for an outstanding request's response, processing an obtained
response, issuing another request or any other operation the client wants
to perform), at the same time :
- Some of those outstanding requests may be travelling over the
network towards the server.
- Some of the outstanding requests may have reached the server and may
be queued for processing at the server side.
- Some of the outstanding requests may have been processed and the
corresponding responses may be travelling over the network towards the
client.
- Some of the responses corresponding to the outstanding requests may
have already reached the client side.
Clearly in this case, operations that the client performs and operations
involved in transfer/processing of a outstanding request can occur in
parallel. Also, operations involved in transfer/processing of two or more
outstanding requests may also occur in parallel (for example when one request
travels to the server, another request's response may be incoming towards the
client). Such kind of parallelism makes the overall transfer faster as compared
to a transfer performed using the synchronous sftp read/write API.
When the synchronous sftp read/write API is used to perform a transfer,
a strict sequence is followed:
- The client issues a single read/write request.
- Then waits for its response.
- On obtaining the response, the client processes it.
- After the processing ends, the client issues the next read/write request.
A file transfer performed in this manner would be slower than the case where
multiple read/write requests are kept outstanding at the client side. Because
here at any given time, operations related to transfer/processing of only one
request/response pair occurs. This is in contrast to the multiple outstanding
requests scenario where operations related to transfer/processing of multiple
request/response pairs may occur at the same time.
Although it's true that keeping multiple requests outstanding can speed up a
transfer, those outstanding requests come at a cost of increased memory
consumption both at the client side and the server side. Hence care must be
taken to use a reasonable limit for the number of requests kept outstanding.
The further sections provide code examples to show how uploads/downloads
can be performed using the sftp aio API and the concept of outstanding requests
discussed in this section. In those code examples, error handling has been
ignored and at some places pseudo code has been used for the sake of brevity.
The complete code for performing uploads/downloads using the sftp aio API,
can be found at https://gitlab.com/libssh/libssh-mirror/-/tree/master.
- libssh benchmarks for uploads performed using the sftp aio API [See
tests/benchmarks/bench_sftp.c]
- libssh benchmarks for downloads performed using the sftp aio API. [See
tests/benchmarks/bench_sftp.c]
- libssh sftp ft API code for performing a local to remote transfer (upload).
[See src/sftp_ft.c]
- libssh sftp ft API code for performing a remote to local transfer
(download). [See src/sftp_ft.c]
@subsection sftp_aio_download_example Performing a download using the sftp aio API
Terminologies used in the following code snippets :
- file : The sftp file handle of the remote file to download data
from. (See sftp_open())
- file_size : the size of the sftp file to download. This size can be obtained
by statting the remote file to download (e.g by using sftp_stat())
- We will need to maintain a queue which will be used to store the sftp aio
handles corresponding to the outstanding requests.
First, we issue the read requests while ensuring that their count
doesn't exceed a particular limit decided by us, and the number of bytes
requested don't exceed the size of the file to download.
@code
sftp_aio aio = NULL;
// Using a chunk size of 16 KB
size_t chunk_size = 16 * 1024;
// Max number of requests to keep outstanding at a time
size_t in_flight_requests = 5;
// Number of bytes for which requests have been sent
size_t bytes_requested = 0;
// Number of bytes which have been downloaded
size_t bytes_downloaded = 0;
// Buffer to use for the download
char *buffer = NULL;
buffer = malloc(chunk_size);
if (buffer == NULL) {
// handle error
}
... // Code to open the remote file (to download) using sftp_open().
... // Code to stat the remote file's file size.
... // Code to open the local file in which downloaded data is to be stored.
... // Code to initialize the queue which will be used to store sftp aio
// handles.
for (i = 0;
i < in_flight_requests && bytes_requested < file_size;
++i) {
to_read = file_size - bytes_requested;
if (to_read > chunk_size) {
to_read = chunk_size;
}
// Issue a read request
rc = sftp_aio_begin_read(file, to_read, &aio);
if (rc == SSH_ERROR) {
// handle error
}
bytes_requested += to_read;
// Pseudo code
ENQUEUE aio in the queue;
}
@endcode
At this point, at max in_flight_requests number of requests may be
outstanding. Now we wait for the response corresponding to the earliest
issued outstanding request.
On getting that response, we issue another read request if there are
still some bytes in the sftp file (to download) for which we haven't sent the
read request. (This happens when bytes_requested < file_size)
This issuing of another read request (under a condition) is done to
keep the number of outstanding requests equal to the value of the
in_flight_requests variable.
This process has to be repeated for every remaining outstanding request.
@code
while (the queue is not empty) {
// Pseudo code
aio = DEQUEUE an sftp aio handle from the queue of sftp aio handles;
// Wait for the response of the request corresponding to the aio
bytes_read = sftp_aio_wait_read(&aio, buffer, chunk_size);
if (bytes_read == SSH_ERROR) {
//handle error
}
bytes_downloaded += bytes_read;
if (bytes_read != chunk_size && bytes_downloaded != file_size) {
// A short read encountered on the remote file before reaching EOF,
// handle it.
}
// Pseudo code
WRITE bytes_read bytes from the buffer into the local file
in which downloaded data is to be stored ;
if (bytes_requested == file_size) {
// no need to issue more read requests
continue;
}
// else issue a read request
to_read = file_size - bytes_requested;
if (to_read > chunk_size) {
to_read = chunk_size;
}
rc = sftp_aio_begin_read(file, to_read, &aio);
if (rc == SSH_ERROR) {
// handle error
}
bytes_requested += to_read;
// Pseudo code
ENQUEUE aio in the queue;
}
free(buffer);
... // Code to destroy the queue which was used to store the sftp aio
// handles.
@endcode
After exiting the while (the queue is not empty) loop, the download
would've been complete (assuming no error occurs).
@subsection sftp_aio_upload_example Performing an upload using the sftp aio API
Terminologies used in the following code snippets :
- file : The sftp file handle of the remote file in which uploaded data
is to be stored. (See sftp_open())
- file_size : The size of the local file to upload. This size can be
obtained by statting the local file to upload (e.g by using stat())
- We will need maintain a queue which will be used to store the sftp aio
handles corresponding to the outstanding requests.
First, we issue the write requests while ensuring that their count
doesn't exceed a particular limit decided by us, and the number of bytes
requested to write don't exceed the size of the file to upload.
@code
sftp_aio aio = NULL;
// Using a chunk size of 16 KB
size_t chunk_size = 16 * 1024;
// Max number of requests to keep outstanding at a time
size_t in_flight_requests = 5;
// Number of bytes for which write requests have been sent
size_t bytes_requested = 0;
// Buffer to use for the upload
char *buffer = NULL;
buffer = malloc(chunk_size);
if (buffer == NULL) {
// handle error
}
... // Code to open the local file (to upload) [e.g using open(), fopen()].
... // Code to stat the local file's file size [e.g using stat()].
... // Code to open the remote file in which uploaded data will be stored [see
// sftp_open()].
... // Code to initialize the queue which will be used to store sftp aio
// handles.
for (i = 0;
i < in_flight_requests && bytes_requested < file_size;
++i) {
to_write = file_size - bytes_requested;
if (to_write > chunk_size) {
to_write = chunk_size;
}
// Pseudo code
READ to_write bytes from the local file (to upload) into the buffer;
rc = sftp_aio_begin_write(file, buffer, to_write, &aio);
if (rc == SSH_ERROR) {
// handle error
}
bytes_requested += to_write;
// Pseudo code
ENQUEUE aio in the queue;
}
@endcode
At this point, at max in_flight_requests number of requests may be
outstanding. Now we wait for the response corresponding to the earliest
issued outstanding request.
On getting that response, we issue another write request if there are
still some bytes in the local file (to upload) for which we haven't sent
the write request. (This happens when bytes_requested < file_size)
This issuing of another write request (under a condition) is done to
keep the number of outstanding requests equal to the value of the
in_flight_requests variable.
This process has to be repeated for every remaining outstanding request.
@code
while (the queue is not empty) {
// Pseudo code
aio = DEQUEUE an sftp aio handle from the queue of sftp aio handles;
// Wait for the response of the request corresponding to the aio
bytes_written = sftp_aio_wait_write(&aio);
if (bytes_written == SSH_ERROR) {
// handle error
}
// sftp_aio_wait_write() won't report a short write, so no need
// to check for a short write here.
if (bytes_requested == file_size) {
// no need to issue more write requests
continue;
}
// else issue a write request
to_write = file_size - bytes_requested;
if (to_write > chunk_size) {
to_write = chunk_size;
}
// Pseudo code
READ to_write bytes from the local file (to upload) into a buffer;
rc = sftp_aio_begin_write(file, buffer, to_write, &aio);
if (rc == SSH_ERROR) {
// handle error
}
bytes_requested += to_write;
// Pseudo code
ENQUEUE aio in the queue;
}
free(buffer);
... // Code to destroy the queue which was used to store the sftp aio
// handles.
@endcode
After exiting the while (the queue is not empty) loop, the upload
would've been complete (assuming no error occurs).
@subsection sftp_aio_free Example showing the usage of sftp_aio_free()
The purpose of sftp_aio_free() was discussed at the beginning of this page,
the following code example shows how it can be used during cleanup.
@code
void print_sftp_error(sftp_session sftp)
{
if (sftp == NULL) {
return;
}
fprintf(stderr, "sftp error : %d\n", sftp_get_error(sftp));
fprintf(stderr, "ssh error : %s\n", ssh_get_error(sftp->session));
}
// Returns 0 on success, -1 on error
int write_strings(sftp_file file)
{
const char * strings[] = {
"This is the first string",
"This is the second string",
"This is the third string",
"This is the fourth string"
};
size_t string_count = sizeof(strings) / sizeof(strings[0]);
size_t i;
sftp_session sftp = NULL;
sftp_aio aio = NULL;
int rc;
if (file == NULL) {
return -1;
}
... // Code to initialize the queue which will be used to store sftp aio
// handles
sftp = file->sftp;
for (i = 0; i < string_count; ++i) {
rc = sftp_aio_begin_write(file,
strings[i],
strlen(strings[i]),
&aio);
if (rc == SSH_ERROR) {
print_sftp_error(sftp);
goto err;
}
// Pseudo code
ENQUEUE aio in the queue of sftp aio handles
}
for (i = 0; i < string_count; ++i) {
// Pseudo code
aio = DEQUEUE an sftp aio handle from the queue of sftp aio handles;
rc = sftp_aio_wait_write(&aio);
if (rc == SSH_ERROR) {
print_sftp_error(sftp);
goto err;
}
}
... // Code to destroy the queue in which sftp aio handles were
// stored
return 0;
err:
while (queue is not empty) {
// Pseudo code
aio = DEQUEUE an sftp aio handle from the queue of sftp aio handles;
sftp_aio_free(aio);
}
... // Code to destroy the queue in which sftp aio handles were
// stored.
return -1;
}
@endcode
*/