Introduce Suspendable Requests to the Event MPM.

Using this basic framework, you can return SUSPENDED from an HTTP Handler,
and then register a callback that is invoked by the MPM at a later time.

This initial version only supports _timers_ as callbacks, but in the future I
would like to add things like wait for socket activity, on a socket specified by
the handler.

Once in a callback, It is then the responsibility of the callback fucntion 
to finish the HTTP Request handling, but this alows you to do cool things like 
a fully async proxy, COMET support, or even rate limiting.

To prove I'm not insane, I've inlcuded an example module, mod_dialup.

You can configure it like this:
<Location "/docs">
  ModemStandard "V.32"
</Location>

And for static files inside that path, you will be rate limited to V.32 speeds, 
aka 9.6 kilobits/second.

Does anyone besides Rüdiger read commit emails :-) ?

I know there are likely huge problems with this, but I would like to see how far
we can push the Event MPM, figure out what to do better, if there is anything, 
and then really dive into the 3.0 development before ApacheCon.

* server/mpm/experimental/event/fdqueue.h:
    (timer_event_t): New structure to hold timer events and callback functions.
    
* server/mpm/experimental/event/fdqueue.c
    (ap_queue_empty): Modify to also look at Timer Ring.

    (ap_queue_init): Initialize Timer Ring.

    (ap_queue_push_timer): New function, pushes a timer event into the queue.

    (ap_queue_pop_something): Renamed function, returns a timer event or
        a socket/pool for a worker thread to run.


* server/mpm/experimental/event/event.c
    (process_socket): If the connection is in SUSPENDED state, don't force it
        into linger mode yet, the callback will have to take care of that.

    (push_timer2worker): New shortcut function, pushes timer event into queue
        for a worker to run.

    (timer_free_ring): New global data structure to recycle memory used by 
        timer events.

    (timer_ring): New global data structure to hold active timer events.

    (g_timer_ring_mtx): Thread mutex to protect timer event data structures.

    (ap_mpm_register_timed_callback): New Function, registers a callback to be
        invoked by the MPM at a later time.

    (listener_thread): Calculate our wakeup time based on the upcoming Event 
        Queue, and after pollset_poll runs, push any Timers that have passed
        onto worker threads to run.
    
    (worker_thread): Call new queue pop method, and if the Timer Event is 
        non-null, invoke the callback.  Once the callback is done, push the
        structure onto the timer_free_ring, to be recycled.

    (child_main): Initialize new mutex and ring structures.


* server/config.c
    (ap_invoke_handler): Allow SUSPENDED aa valid return code from handlers.


* modules/http/http_core.c
    (ap_process_http_async_connection): Don't close the connection when in 
        SUSPENDED state.


* modules/http/http_request.c
    (ap_process_request_after_handler): New function, body pulled from the old,
        ap_process_async_request.  Split to let handlers invoke this so they 
        don't need to know all of the details of finishing a request.

    (ap_process_async_request): If the handler returns SUSPENDED, don't do
        anything but return.


* include/ap_mmn.h: Bump MMN.


* include/ap_mpm.h
    (ap_mpm_register_timed_callback): New function.


* include/httpd.h:
    (SUSPENDED): New return code for handlers.
    (request_rec::invoke_mtx): New mutex to protect callback invokcations
        from being run before the original handler finishes running.
    (conn_state_e): Add a suspended state.


* include/http_request.h
    (ap_process_request_after_handler): New function to make it easier for 
        handlers to finish the HTTP Request.


* modules/test/config.m4: Add mod_dialup to build.


* modules/test/mod_dialup.c: New rate limiting module, requires the Event MPM 
    to work.




git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@697357 13f79535-47bb-0310-9956-ffa450edef68

modules/http/http_request.c

@@ -213,13 +213,39 @@ static void check_pipeline(conn_rec *c)
 }
 
 
-void ap_process_async_request(request_rec *r)
+void ap_process_request_after_handler(request_rec *r)
 {
-    int access_status;
     apr_bucket_brigade *bb;
     apr_bucket *b;
     conn_rec *c = r->connection;
 
+    /* Send an EOR bucket through the output filter chain.  When
+     * this bucket is destroyed, the request will be logged and
+     * its pool will be freed
+     */
+    bb = apr_brigade_create(r->connection->pool, r->connection->bucket_alloc);
+    b = ap_bucket_eor_create(r->connection->bucket_alloc, r);
+    APR_BRIGADE_INSERT_HEAD(bb, b);
+    
+    ap_pass_brigade(r->connection->output_filters, bb);
+    
+    /* From here onward, it is no longer safe to reference r
+     * or r->pool, because r->pool may have been destroyed
+     * already by the EOR bucket's cleanup function.
+     */
+    
+    c->cs->state = CONN_STATE_WRITE_COMPLETION;
+    check_pipeline(c);
+    if (ap_extended_status) {
+        ap_time_process_request(c->sbh, STOP_PREQUEST);
+    }
+}
+
+void ap_process_async_request(request_rec *r)
+{
+    conn_rec *c = r->connection;
+    int access_status;
+
     /* Give quick handlers a shot at serving the request on the fast
      * path, bypassing all of the other Apache hooks.
      *
@@ -234,8 +260,12 @@ void ap_process_async_request(request_rec *r)
      * Use this hook with extreme care and only if you know what you are
      * doing.
      */
-    if (ap_extended_status)
+    if (ap_extended_status) {
         ap_time_process_request(r->connection->sbh, START_PREQUEST);
+    }
+
+    apr_thread_mutex_create(&r->invoke_mtx, APR_THREAD_MUTEX_DEFAULT, r->pool);
+    apr_thread_mutex_lock(r->invoke_mtx);
     access_status = ap_run_quick_handler(r, 0);  /* Not a look-up request */
     if (access_status == DECLINED) {
         access_status = ap_process_request_internal(r);
@@ -244,6 +274,16 @@ void ap_process_async_request(request_rec *r)
         }
     }
 
+    if (access_status == SUSPENDED) {
+        if (ap_extended_status) {
+            ap_time_process_request(c->sbh, STOP_PREQUEST);
+        }
+        c->cs->state = CONN_STATE_SUSPENDED;
+        apr_thread_mutex_unlock(r->invoke_mtx);
+        return;
+    }
+    apr_thread_mutex_unlock(r->invoke_mtx);
+
     if (access_status == DONE) {
         /* e.g., something not in storage like TRACE */
         access_status = OK;
@@ -257,24 +297,7 @@ void ap_process_async_request(request_rec *r)
         ap_die(access_status, r);
     }
 
-    /* Send an EOR bucket through the output filter chain.  When
-     * this bucket is destroyed, the request will be logged and
-     * its pool will be freed
-     */
-    bb = apr_brigade_create(r->connection->pool, r->connection->bucket_alloc);
-    b = ap_bucket_eor_create(r->connection->bucket_alloc, r);
-    APR_BRIGADE_INSERT_HEAD(bb, b);
-    ap_pass_brigade(r->connection->output_filters, bb);
-
-    /* From here onward, it is no longer safe to reference r
-     * or r->pool, because r->pool may have been destroyed
-     * already by the EOR bucket's cleanup function.
-     */
-
-    c->cs->state = CONN_STATE_WRITE_COMPLETION;
-    check_pipeline(c);
-    if (ap_extended_status)
-        ap_time_process_request(c->sbh, STOP_PREQUEST);
+    return ap_process_request_after_handler(r);
 }
 
 void ap_process_request(request_rec *r)