Minio can return during deletion sometimes, like missing quorum,
but mc can block in that case. The reason is that Remove() in
client-s3 doesn't actively listen for errors reported by minio-go
RemoveObjects() API because that would also block this latter.
Tested by transferring 29GiB data, around 246569 files
over 100Gig network.
- master branch
```
time mc mirror /root myminio/testbucket/
real 2m10s
```
- after changes
```
time mc mirror /root myminio/testbucket/
real 1m2s
```
By default runtime.NumCPU() workers are started.
Additionally this PR also allows for creating
prefixes on S3 compatible object storage backends
in the following form
```
mc mb s3/harshavardhana/object/prefix/
```
Standard headers like Content-Disposition, Content-Encoding,
Cache-Control, Content-Language were made part of user metadata
causing it to fail while mirroring, if the source has these headers
set.
Fixes#2358
This requires --dangerous flag in addition to --recursive and --force
Change namespace identification check to be more robust to user
input errors like mc rm --r --force play//
This PR brings in changes for
- PutObjectStream with no content-length is heavily
optimized so `mc pipe` will not run out of memory
for large uploads.
- Fixes a bug in Signature v2 for HTTP requests.
- Support us-gov-west-1 region as well.
- Auto region handling is inside minio-go now.
The probe package is no longer available in the Minio server
repo. This change just moves probe from the vendor directory to its
own dir under `pkg`.
This is a maintenance fix.
This PR brings following changes
- Brings streaming support for mc uploads.
- Brings support for copying objects upto 5TiB
as a consequence of ComposeObject implementation
in minio-go.
* client: Avoid returning metadata with Get()
client-s3.Get() calls Stat() method because Get() is supposed to return
object metadata. This PR makes a change in client interface so metadata
is returned in Stat() method. This is beneficial for us to avoid calling
HEAD method in S3 when run `mc cat`
* cat: Ignore Stat() returned error
In case of cache gateway, HEAD objects returns 404 even though
objects are gettable. This PR ignores Stat() error since its
purpose in case of cat is to check if the download object size
is equal to what the server announced.
This patch adds UTCNow() function which returns current UTC time.
This is equivalent of UTCNow() == time.Now().UTC()
Refer https://github.com/minio/minio/pull/3931
TLSConfig should be allocated regardless of the
URL scheme so that the re-direction works properly.
Without this change if a server is configured with
self signed certs redirection would fail with
```
x509: certificate signed by unknown authority
```
Allocate TLSConfig always with default values.
Simply use
```
mc config host add s3-accel https://s3-accelerate.amazonaws.com <access_id> <secret_id>
```
This is all is needed, if you have buckets which has s3 acceleration
enabled it just works transparently.
Fixes#1912
