1
0
mirror of https://github.com/minio/mc.git synced 2025-11-12 01:02:26 +03:00
Commit Graph

7 Commits

Author SHA1 Message Date
ebozduman
83a9d8b91d Adds missing codebase MinIO and min.io changes (#2745) 2019-04-16 22:31:54 -07:00
Harshavardhana
12627c35e4 Fix parallel upload performance issues (#2521)
Tested by transferring 29GiB data, around 246569 files
over 100Gig network.

- master branch
```
time mc mirror /root myminio/testbucket/
real	2m10s
```

- after changes
```
time mc mirror /root myminio/testbucket/
real	1m2s
```

By default runtime.NumCPU() workers are started.
2018-08-30 22:31:08 +05:30
Anis Elleuch
757fff7764 Simplify parallel manager implementation (#2413)
Each 5 seconds, calculate current bandwidth and add new workers as
bandwidth gets increased. Stop adding workers if it is not useful
for increasing bandwidth.
2018-04-06 18:46:35 -07:00
Harshavardhana
766e04d79d fix crashes on 32bit systems use LoadUint32() instead (#2389) 2018-02-12 10:25:06 -08:00
Harshavardhana
3b4626ea15 Bump max parallel workers to 32 (#2360) 2018-01-18 19:46:31 -08:00
Harshavardhana
13259eea21 mirror with --watch should copy and exit for s3 endpoints (#2345)
Current code wouldn't exit unless we press CTRL+C
even if the endpoints do not support --watch, this PR
fixes this behavior.
2018-01-08 10:56:56 -08:00
A. Elleuch
5ddf9728f8 Parallelize upload/remove in cp/mirror commands (#2289)
This PR removes --parallel flag and implements a dynamic way of adding
threads to accelerate mirror/cp command.

After the calculation of objects that need to be copied/removed, mc
sends tasks (upload/remove) to a parallel manager which monitors the
progress of tasks and decide to allocate more threads to it to
accelerate the overall work.

This is very useful when we copy/mirror many relatively small files.
2017-12-07 16:20:34 -08:00