An error, such as, `Bad Request` when copying an object without supplied
keys is not reflected in the exit code.
This commit simplifies a bit the mirror code. It makes mirror before and
after starting to send the status to mj.status.
Previously OlderThan and NewerThan was configured for day
which has been changed to Days, Hours and minutes.
Also, changed the `flag` in `find` command to be `newer-than` and `older-than` to be consistent with other commands.
A bad synchornization between startMirror() and its caller mirror() happens when preparing the difference of objects between source & target returns an error: mirror() quits when it receives an error from startMirror() without give a chance for that latter to properly stop the parallel mirror manager.
This PR will:
1. Quit mirroring when listing returns an error: this means calculating diff needs to stop and no further action needs to be taken since it can be dangerous especially when --remove flag is passed.
2. Properly stop parallel mirroring manager before quitting.
Mirror code depends on rm code which naturally prints delete messages,
this PR will make mirror uses its own delete code since it is simple
enough to do.
Also add size information to remove message
S3 lists 1000 objects in batches, but once this batch is being
processed to be copied over there is no way to lock across the
entire batch for mirroring. Due to this we introduce a scenario
when objects might have been deleted in parallel from another
application.
In such a scenario simply ignore such objects and proceed to
copy other files.
Fixes#2537
Tested by transferring 29GiB data, around 246569 files
over 100Gig network.
- master branch
```
time mc mirror /root myminio/testbucket/
real 2m10s
```
- after changes
```
time mc mirror /root myminio/testbucket/
real 1m2s
```
By default runtime.NumCPU() workers are started.
Currently even if the command supports encrypted objects,
the syntax verification would fail on them in certain
cases. Since syntax validation of source/targets doesn't
use the parsed encrypt key pairs. This PR fixes this
behavior.
The event.Path had few prefixes appended in it (only in darwin).
Instead of taking the trimmed directory path, It takes the full path, say '/private/...".
This disturbs the target URL. Which indeed inserts it as a whole '/private/{target_path}',
So trimming the prefixes in the event.path solved it.
Solves:#2443
If we have any listing problem, we should stop comparing difference
between source & target lists because it is already so hard to know
what mc should do next. So, let's inform users and let them deal
with the problem.
mc used to show a warning messages during mirroring when it
detects a need to overwrite or remove a remote object but no
flag is specified. This commit efeaf2ee72
changed this behavior but this commit will restablish
the old behavior.
This PR removes --parallel flag and implements a dynamic way of adding
threads to accelerate mirror/cp command.
After the calculation of objects that need to be copied/removed, mc
sends tasks (upload/remove) to a parallel manager which monitors the
progress of tasks and decide to allocate more threads to it to
accelerate the overall work.
This is very useful when we copy/mirror many relatively small files.
Added exclude options to exclude unwanted system files.
Exclude option excludes the source files/objects that match
the passed shell file name pattern
Fixes#1903
The probe package is no longer available in the Minio server
repo. This change just moves probe from the vendor directory to its
own dir under `pkg`.
This is a maintenance fix.
This patch adds UTCNow() function which returns current UTC time.
This is equivalent of UTCNow() == time.Now().UTC()
Refer https://github.com/minio/minio/pull/3931
