Mcol 4841 dev6 Handle large joins without OOM (#2155)

* MCOL-4846 dev-6 Handle large join results
Use a loop to shrink the number of results reported per message to something manageable.

* MCOL-4841 small changes requested by review

* Add EXTRA threads to prioritythreadpool
prioritythreadpool is configured at startup with a fixed number of threads available. This is to prevent thread thrashing. Since most of the time, BPP job steps are short lived, and a rescheduling mechanism exist if no threads are available, this works to keep cpu wastage to a minimum.

However, if a query or queries consume all the threads in prioritythreadpool and then block (due to the consumer not consuming fast enough) we can run out of threads and no work will be done until some threads unblock. A new mechanism allows for EXTRA threads to be generated for the duration of the blocking action. These threads can act on new queries. When all blocking is completed, these threads will be released when idle.

* MCOL-4841 dev6 Reconcile with changes in develop-6

* MCOL-4841 Some format corrections

* MCOL-4841 dev clean up some things based on review

* MCOL-4841 dev 6 ExeMgr Crashes after large join
This commit fixes up memory accounting issues in ExeMgr

* MCOL-4841 remove LDI change
Opened MCOL-4968 to address the issue

* MCOL-4841 Add fMaxBPPSendQueue to ResourceManager
This causes the setting to be loaded at run time (requires restart to accept a change) BPPSendthread gets this in it's ctor
Also rolled back changes to TupleHashJoinStep::smallRunnerFcn() that used a local variable to count locally allocated memory, then added it into the global counter at function's end. Not counting the memory globally caused conversion to UM only join way later than it should. This resulted in MCOL-4971.

* MCOL-4841 make blockedThreads and extraThreads atomic
Also restore previous scope of locks in bppsendthread. There is some small chance the new scope could be incorrect, and the performance boost is negligible. Better safe than sorry.

utils/threadpool/prioritythreadpool.cpp

@@ -28,6 +28,7 @@ using namespace std;
 
 #include "messageobj.h"
 #include "messagelog.h"
+#include "threadnaming.h"
 using namespace logging;
 
 #include "prioritythreadpool.h"
@@ -40,7 +41,8 @@ namespace threadpool
 
 PriorityThreadPool::PriorityThreadPool(uint targetWeightPerRun, uint highThreads,
                                        uint midThreads, uint lowThreads, uint ID) :
-    _stop(false), weightPerRun(targetWeightPerRun), id(ID)
+    _stop(false), weightPerRun(targetWeightPerRun), id(ID), 
+    blockedThreads(0), extraThreads(0), stopExtra(true)
 {
     boost::thread* newThread;
     for (uint32_t i = 0; i < highThreads; i++)
@@ -99,7 +101,22 @@ void PriorityThreadPool::addJob(const Job& job, bool useLock)
         newThread->detach();
         threadCounts[LOW]++;
     }
-
+    
+    // If some threads have blocked (because of output queue full)
+    // Temporarily add some extra worker threads to make up for the blocked threads.
+    if (blockedThreads > extraThreads)
+    {
+        stopExtra = false;
+        newThread = threads.create_thread(ThreadHelper(this, EXTRA));
+        newThread->detach();
+        extraThreads++;
+    }
+    else if (blockedThreads == 0)
+    {
+        // Release the temporary threads -- some threads have become unblocked.
+        stopExtra = true;
+    }
+    
     if (job.priority > 66)
         jobQueues[HIGH].push_back(job);
     else if (job.priority > 33)
@@ -127,7 +144,7 @@ void PriorityThreadPool::removeJobs(uint32_t id)
 
 PriorityThreadPool::Priority PriorityThreadPool::pickAQueue(Priority preference)
 {
-    if (!jobQueues[preference].empty())
+    if (preference != EXTRA && !jobQueues[preference].empty())
         return preference;
     else if (!jobQueues[HIGH].empty())
         return HIGH;
@@ -139,6 +156,10 @@ PriorityThreadPool::Priority PriorityThreadPool::pickAQueue(Priority preference)
 
 void PriorityThreadPool::threadFcn(const Priority preferredQueue) throw()
 {
+    if (preferredQueue == EXTRA)
+        utils::setThreadName("Extra");
+    else
+        utils::setThreadName("Idle");
     Priority queue = LOW;
     uint32_t weight, i = 0;
     vector<Job> runList;
@@ -158,6 +179,14 @@ void PriorityThreadPool::threadFcn(const Priority preferredQueue) throw()
 
             if (jobQueues[queue].empty())
             {
+                // If this is an EXTRA thread due toother threads blocking, and all blockers are unblocked, 
+                // we don't want this one any more.
+                if (preferredQueue == EXTRA && stopExtra)
+                {
+                    extraThreads--;
+                    return;
+                }
+                
                 newJob.wait(lk);
                 continue;
             }
@@ -194,7 +223,11 @@ void PriorityThreadPool::threadFcn(const Priority preferredQueue) throw()
                 if (reschedule[i])
                     rescheduleCount++;
             }
-
+            if (preferredQueue == EXTRA)
+                utils::setThreadName("Extra (used)");
+            else
+                utils::setThreadName("Idle");
+            
             // no real work was done, prevent intensive busy waiting
             if (rescheduleCount == runList.size())
                 usleep(1000);
@@ -216,6 +249,7 @@ void PriorityThreadPool::threadFcn(const Priority preferredQueue) throw()
             }
 
             runList.clear();
+            
         }
     }
     catch (std::exception& ex)