Mirror code depends on rm code which naturally prints delete messages,
this PR will make mirror uses its own delete code since it is simple
enough to do.
Also add size information to remove message
S3 lists 1000 objects in batches, but once this batch is being
processed to be copied over there is no way to lock across the
entire batch for mirroring. Due to this we introduce a scenario
when objects might have been deleted in parallel from another
application.
In such a scenario simply ignore such objects and proceed to
copy other files.
Fixes#2537
Tested by transferring 29GiB data, around 246569 files
over 100Gig network.
- master branch
```
time mc mirror /root myminio/testbucket/
real 2m10s
```
- after changes
```
time mc mirror /root myminio/testbucket/
real 1m2s
```
By default runtime.NumCPU() workers are started.
Currently even if the command supports encrypted objects,
the syntax verification would fail on them in certain
cases. Since syntax validation of source/targets doesn't
use the parsed encrypt key pairs. This PR fixes this
behavior.
The event.Path had few prefixes appended in it (only in darwin).
Instead of taking the trimmed directory path, It takes the full path, say '/private/...".
This disturbs the target URL. Which indeed inserts it as a whole '/private/{target_path}',
So trimming the prefixes in the event.path solved it.
Solves:#2443
If we have any listing problem, we should stop comparing difference
between source & target lists because it is already so hard to know
what mc should do next. So, let's inform users and let them deal
with the problem.
mc used to show a warning messages during mirroring when it
detects a need to overwrite or remove a remote object but no
flag is specified. This commit efeaf2ee72
changed this behavior but this commit will restablish
the old behavior.
This PR removes --parallel flag and implements a dynamic way of adding
threads to accelerate mirror/cp command.
After the calculation of objects that need to be copied/removed, mc
sends tasks (upload/remove) to a parallel manager which monitors the
progress of tasks and decide to allocate more threads to it to
accelerate the overall work.
This is very useful when we copy/mirror many relatively small files.
Added exclude options to exclude unwanted system files.
Exclude option excludes the source files/objects that match
the passed shell file name pattern
Fixes#1903
The probe package is no longer available in the Minio server
repo. This change just moves probe from the vendor directory to its
own dir under `pkg`.
This is a maintenance fix.
This patch adds UTCNow() function which returns current UTC time.
This is equivalent of UTCNow() == time.Now().UTC()
Refer https://github.com/minio/minio/pull/3931
mc can quit when mirror terminates but progress bar is not able
to show all made progress. This commit makes mc waits until all updates
are sent to the status channel.
It is easier to use {{.HelpName}} in help text template to print
the mc binary name plus the whole command hierarchy (binary name +
command + subcommand, etc..)
