The S3 API specification states that x-amz-copy-source should be
urlencoded. libmarias3 wasn't doing this but Amazon's S3 implementation
appeared to be forgiving enough that it didn't matter. When using
compatible APIs such as MinIO this requirement became stricter.
The code now urlencodes the source path for ms3_copy / ms3_move.
In addition this patch adds support for MinIO in the test suite by
adding environment variables for port and disabling https.
LibXML has been removed in favour of a modified version of xml.c
(modified to work with AWS responses and our code base usage).
Also fixes:
* AC_MSG_ERROR usage (fixes#83)
* Some places where we weren't using ms3_c alloc functions
* ms3_get doing an ms3_list with an empty key (fixes#82)
"Buckets created in Regions launched after March 20, 2019 are not
reachable via the https://bucket.s3.amazonaws.com naming scheme."
This patch enhances the one Markus wrote to just completely remove the
domain based access method.
If libcurl was built with OpenSSL < 1.1.0 then there is a potential
thread safety issue which can cause libmarias3 to crash.
This fix detects which version of OpenSSL (if any) that libcurl was
built with and adds thread locking callbacks to OpenSSL if found unsafe.
Fixesmariadb-corporation/libmarias3#70
Google Cloud has some weird behaviours we need to support:
* Auth keys are longer than the fixed 20 characters of AWS
* Header responses are lower case
* DELETE returns a 404 on file not found
Fixesmariadb-corporation/libmarias3#71
Add the warning -Wdeclaration-after-statement and fix everything it
finds.
Add MS3_OPT_FORCE_PROTOCOL_VERSION to ms3_set_option() with a test to
force protocol version when the autodetection does the wrong thing.
* The error response may not be the XML we expect which would cause a
NULL node to be dereferenced
* Setting the server error message with a NULL message would cause a
double-free
* Fix a few potential pointer arithmatic issues
* Fix race condition on time generation
* Added TSAN to ci-scripts
* Fix minor issues found in cppcheck
* Stop buffer overrun if the buffer chunk size is set smaller than packet
* Fix ms3_get() returning random data if a CURL request completely fails
Fixesmariadb-corporation/libmarias3#55
Use a pointer which is pushed straight to network instead of the
callback which does memcpy to a buffer. In tests this provides up to 20%
performance improvement.
This function acts like ms3_list() but only looks in the prefix up to
the first slash and includes directory names.
Also remove the 1MB lower limit on chunk size
If an error occurs in ms3_get the error response can be returned as
data. Which if not freed by the application can lead to a leak.
We now free this data after the error message has been extracted.
Also bump version.
* Autoswitch to bucket path instead of bucket domain access method (for IP urls)
* Fixed issue with SSL disabled verification
* Fixed minor leak when base_domain is set
* Add ``S3NOVERIFY`` env var to tests which will disable SSL verification when set to ``1``
* Will use V1 by default for non-Amazon S3 servers
* V1 or V2 can be forced using ms3_set_option()
* Updated tests to test for both
* Set version and ABI version correctly
* ms3_server_error() gets the last server error or Curl error
* Remove the deprecated ms3_init(), it could not work with
ms3_server_error() and no one uses it
* Test cleanups
* Version bump for semver
* ms3_get() was calling realloc every 16KB which caused it to be CPU
heavy on large files. We now use 1MB buffers instead.
* Make longlist test use a different prefix so that it doesn't generate
a random bad count if other tests are running
CURL will automatically reuse connections if the CURL object is reused.
This commit adds functionality so that a thread can have a CURL object
that can be reused without dramatically altering the API.
It add two new functions:
* ms3_library_init() - to initialize curl for multithreaded operations
* ms3_thread_init() - to initialize a connection object for a thread
This patch also fixes tests that are affecting each other.
