database/mariadb-columnstore-engine
1
0
Fork 0
mirror of https://github.com/mariadb-corporation/mariadb-columnstore-engine.git synced 2025-07-30 19:23:07 +03:00
Code Activity

Revert "MCOL-4576: remove S3 options from cpimport. (#2307)"

Browse Source 
This reverts commit 14c4840d53.
This commit is contained in:
Roman Nozdrin
2022-03-23 12:33:23 +00:00
parent 879387af34
commit 2b4946f53a
3 changed files with 17 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
writeengine/splitter/we_filereadthread.cpp
View File

@ -481,15 +481,6 @@ void WEFileReadThread::openInFile()
use ms3 lib to d/l data into mem
use boost::iostreams to wrap the mem in a stream interface
point infile's stream buffer to it.
MCOL-4576: The options to setup S3 with cpimport have been removed and this
code is unreachable. However we may need to resurrect it at some point in some form.
Performance issues with extremely large data files as well as the fact files larger
than system memory will cause an OOM error. Multipart downloads/uploads need to be
implemented or more likely a different streaming solution developed with external API tools
MCOL-4576 work around is to use 3rd party CLI tools and pipe data file from S3 bucket
into cpimport stdin. 3rd party tooling for large object downloads will be more efficient.
*/
if (fSdh.getDebugLvl())