database/mariadb-columnstore-engine
1
0
Fork 0
mirror of https://github.com/mariadb-corporation/mariadb-columnstore-engine.git synced 2025-12-15 12:09:09 +03:00
Code Activity

Merging the prefix-ownership feature.

Browse Source 
Squashed commit of the following:

commit 4a4c3dab2e6acf942bbdfd4d760c000bc9cbfc6a
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Tue Jul 30 10:50:08 2019 -0500

    Standardized a couple status msgs.

commit 1b76f7e6411424c9633dcd4ebe7f61e9fce2f0ac
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Tue Jul 30 09:19:31 2019 -0500

    Fixed the thread-joining-itself problem.

commit 4fdb79e87496eab64c4c5af72321bc57423297ba
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Mon Jul 29 17:15:05 2019 -0500

    Checkpointing.  Need to release ownership differently, realized
    I have 1 thread trying to join itself.

commit 04d0183735e9697d76a2472c6135d90755ca61b5
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Mon Jul 29 16:12:33 2019 -0500

    Checkpointing a whole lot of fixes.  test000 works here.

commit 72e9c998c62b095cad1cf33f885f6c7697bde214
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Fri Jul 26 16:29:15 2019 -0500

    Checkpointing.  Started debugging.  Several small fixes.  Doesn't work yet.

commit ab728e1481debec94d676e697954b1d164302a0c
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Fri Jul 26 13:15:47 2019 -0500

    Checkpointing.  Got everything to build.

commit a2c6d07cdc12c45530c1d5cf4205d3aee8738d80
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Fri Jul 26 12:59:26 2019 -0500

    Checkpointing.  Got the library to build.

commit 9f6bf19a64f512e17e6139b0fc04850cdcdb3b3a
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Fri Jul 26 12:24:46 2019 -0500

    Checkpointing.  Still WIP.  Feature touches everything.

commit a79ca8dc88a99f812432d5dca34ed54474df1933
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Thu Jul 25 16:38:20 2019 -0500

    Checkpointing more changes.

commit a9e81af3e4e00f8a3d30b3796a2c3aa94c04f7c0
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Thu Jul 25 15:07:44 2019 -0500

    Checkpointing changes to the other classes that need to be aware
    of separately managed prefixes.

commit d85dfaa401b49a7bb714701649dec303eb7c068c
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Wed Jul 24 14:24:23 2019 -0500

    Added the new class to CMakeLists.

commit 66d6d550b13be94ada107311574378bd848951cd
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Wed Jul 24 14:23:49 2019 -0500

    Checkpointing.  Got the new class to build except for a
    to-be-implemented fcn in Cache.

commit e1b62dba7f05b37b9f12681a53d6632c6ce66d54
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Wed Jul 24 14:23:09 2019 -0500

    Added some documentation to the object_size param.

commit e671cf37c49ed084fbdec1bac50fbaa5ad7c43f9
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Wed Jul 24 10:34:54 2019 -0500

    Checkpointing a new class to manage ownership of prefixes.

commit e5f234ff4c05b5157d37fa17c44d7f626f5e4eb3
Author: Patrick LeBlanc <patrick.leblanc@mariadb.com>
Date:   Tue Jul 23 15:36:48 2019 -0500

    Fixed some copy/paste typos in Metadatafile config err msgs.
This commit is contained in:
Patrick LeBlanc
2019-07-30 10:51:05 -05:00
parent 5f43f55ea5
commit 1c5979576e
21 changed files with 1640 additions and 900 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
src/Synchronizer.h
View File

@@ -21,7 +21,6 @@ namespace storagemanager
class Cache; // break circular dependency in header files
class IOCoordinator;
/* TODO: Need to think about how errors are handled / propagated */
class Synchronizer : public boost::noncopyable
{
public:
@@ -30,12 +29,16 @@ class Synchronizer : public boost::noncopyable
// these take keys as parameters, not full path names, ex, pass in '12345' not
// 'cache/12345'.
void newJournalEntry(const std::string &key, size_t len);
void newJournalEntries(const std::vector<std::pair<std::string, size_t> > &keys);
void newObjects(const std::vector<std::string> &keys);
void deletedObjects(const std::vector<std::string> &keys);
void flushObject(const std::string &key);
void forceFlush();
void newJournalEntry(const boost::filesystem::path &firstDir, const std::string &key, size_t len);
void newJournalEntries(const boost::filesystem::path &firstDir, const std::vector<std::pair<std::string, size_t> > &keys);
void newObjects(const boost::filesystem::path &firstDir, const std::vector<std::string> &keys);
void deletedObjects(const boost::filesystem::path &firstDir, const std::vector<std::string> &keys);
void flushObject(const boost::filesystem::path &firstDir, const std::string &key);
void forceFlush(); // ideally, make a version of this that takes a firstDir parameter
void newPrefix(const boost::filesystem::path &p);
void dropPrefix(const boost::filesystem::path &p);
// for testing primarily
boost::filesystem::path getJournalPath();
@@ -43,7 +46,7 @@ class Synchronizer : public boost::noncopyable
private:
Synchronizer();
void _newJournalEntry(const std::string &key, size_t len);
void _newJournalEntry(const boost::filesystem::path &firstDir, const std::string &key, size_t len);
void process(std::list<std::string>::iterator key);
void synchronize(const std::string &sourceFile, std::list<std::string>::iterator &it);
void synchronizeDelete(const std::string &sourceFile, std::list<std::string>::iterator &it);
@@ -66,8 +69,8 @@ class Synchronizer : public boost::noncopyable
struct Job : public ThreadPool::Job
{
Job(Synchronizer *s, std::list<std::string>::iterator i) : sync(s), it(i) { }
void operator()() { sync->process(it); }
Job(Synchronizer *s, std::list<std::string>::iterator i);
void operator()();
Synchronizer *sync;
std::list<std::string>::iterator it;
};
@@ -88,7 +91,11 @@ class Synchronizer : public boost::noncopyable
boost::thread syncThread;
const boost::chrono::seconds syncInterval = boost::chrono::seconds(10);
void periodicSync();
size_t uncommittedJournalSize, journalSizeThreshold;
std::map<boost::filesystem::path, size_t> uncommittedJournalSize;
size_t journalSizeThreshold;
bool blockNewJobs;
void syncNow(const boost::filesystem::path &prefix); // a synchronous version of forceFlush()
SMLogging *logger;
Cache *cache;