database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-11-07 19:06:32 +03:00
Code Activity

Background worker processes

Browse Source 
Background workers are postmaster subprocesses that run arbitrary
user-specified code.  They can request shared memory access as well as
backend database connections; or they can just use plain libpq frontend
database connections.

Modules listed in shared_preload_libraries can register background
workers in their _PG_init() function; this is early enough that it's not
necessary to provide an extra GUC option, because the necessary extra
resources can be allocated early on.  Modules can install more than one
bgworker, if necessary.

Care is taken that these extra processes do not interfere with other
postmaster tasks: only one such process is started on each ServerLoop
iteration.  This means a large number of them could be waiting to be
started up and postmaster is still able to quickly service external
connection requests.  Also, shutdown sequence should not be impacted by
a worker process that's reasonably well behaved (i.e. promptly responds
to termination signals.)

The current implementation lets worker processes specify their start
time, i.e. at what point in the server startup process they are to be
started: right after postmaster start (in which case they mustn't ask
for shared memory access), when consistent state has been reached
(useful during recovery in a HOT standby server), or when recovery has
terminated (i.e. when normal backends are allowed).

In case of a bgworker crash, actions to take depend on registration
data: if shared memory was requested, then all other connections are
taken down (as well as other bgworkers), just like it were a regular
backend crashing.  The bgworker itself is restarted, too, within a
configurable timeframe (which can be configured to be never).

More features to add to this framework can be imagined without much
effort, and have been discussed, but this seems good enough as a useful
unit already.

An elementary sample module is supplied.

Author: Álvaro Herrera

This patch is loosely based on prior patches submitted by KaiGai Kohei,
and unsubmitted code by Simon Riggs.

Reviewed by: KaiGai Kohei, Markus Wanner, Andres Freund,
Heikki Linnakangas, Simon Riggs, Amit Kapila
This commit is contained in:
Alvaro Herrera
2012-12-06 14:57:52 -03:00
parent e31d524867
commit da07a1e856
16 changed files with 1690 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
src/backend/storage/lmgr/proc.c
View File

@@ -140,7 +140,9 @@ ProcGlobalSemas(void)
* So, now we grab enough semaphores to support the desired max number
* of backends immediately at initialization --- if the sysadmin has set
* MaxConnections or autovacuum_max_workers higher than his kernel will
* support, he'll find out sooner rather than later.
* support, he'll find out sooner rather than later. (The number of
* background worker processes registered by loadable modules is also taken
* into consideration.)
*
* Another reason for creating semaphores here is that the semaphore
* implementation typically requires us to create semaphores in the
@@ -171,6 +173,7 @@ InitProcGlobal(void)
ProcGlobal->spins_per_delay = DEFAULT_SPINS_PER_DELAY;
ProcGlobal->freeProcs = NULL;
ProcGlobal->autovacFreeProcs = NULL;
ProcGlobal->bgworkerFreeProcs = NULL;
ProcGlobal->startupProc = NULL;
ProcGlobal->startupProcPid = 0;
ProcGlobal->startupBufferPinWaitBufId = -1;
@@ -179,10 +182,11 @@ InitProcGlobal(void)
/*
* Create and initialize all the PGPROC structures we'll need. There are
* four separate consumers: (1) normal backends, (2) autovacuum workers
* and the autovacuum launcher, (3) auxiliary processes, and (4) prepared
* transactions. Each PGPROC structure is dedicated to exactly one of
* these purposes, and they do not move between groups.
* five separate consumers: (1) normal backends, (2) autovacuum workers
* and the autovacuum launcher, (3) background workers, (4) auxiliary
* processes, and (5) prepared transactions. Each PGPROC structure is
* dedicated to exactly one of these purposes, and they do not move between
* groups.
*/
procs = (PGPROC *) ShmemAlloc(TotalProcs * sizeof(PGPROC));
ProcGlobal->allProcs = procs;
@@ -223,12 +227,12 @@ InitProcGlobal(void)
procs[i].pgprocno = i;
/*
* Newly created PGPROCs for normal backends or for autovacuum must be
* queued up on the appropriate free list. Because there can only
* ever be a small, fixed number of auxiliary processes, no free list
* is used in that case; InitAuxiliaryProcess() instead uses a linear
* search. PGPROCs for prepared transactions are added to a free list
* by TwoPhaseShmemInit().
* Newly created PGPROCs for normal backends, autovacuum and bgworkers
* must be queued up on the appropriate free list. Because there can
* only ever be a small, fixed number of auxiliary processes, no free
* list is used in that case; InitAuxiliaryProcess() instead uses a
* linear search. PGPROCs for prepared transactions are added to a
* free list by TwoPhaseShmemInit().
*/
if (i < MaxConnections)
{
@@ -236,12 +240,18 @@ InitProcGlobal(void)
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->freeProcs;
ProcGlobal->freeProcs = &procs[i];
}
else if (i < MaxBackends)
else if (i < MaxConnections + autovacuum_max_workers + 1)
{
/* PGPROC for AV launcher/worker, add to autovacFreeProcs list */
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->autovacFreeProcs;
ProcGlobal->autovacFreeProcs = &procs[i];
}
else if (i < MaxBackends)
{
/* PGPROC for bgworker, add to bgworkerFreeProcs list */
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->bgworkerFreeProcs;
ProcGlobal->bgworkerFreeProcs = &procs[i];
}
/* Initialize myProcLocks[] shared memory queues. */
for (j = 0; j < NUM_LOCK_PARTITIONS; j++)
@@ -299,6 +309,8 @@ InitProcess(void)
if (IsAnyAutoVacuumProcess())
MyProc = procglobal->autovacFreeProcs;
else if (IsBackgroundWorker)
MyProc = procglobal->bgworkerFreeProcs;
else
MyProc = procglobal->freeProcs;
@@ -306,6 +318,8 @@ InitProcess(void)
{
if (IsAnyAutoVacuumProcess())
procglobal->autovacFreeProcs = (PGPROC *) MyProc->links.next;
else if (IsBackgroundWorker)
procglobal->bgworkerFreeProcs = (PGPROC *) MyProc->links.next;
else
procglobal->freeProcs = (PGPROC *) MyProc->links.next;
SpinLockRelease(ProcStructLock);
@@ -782,6 +796,11 @@ ProcKill(int code, Datum arg)
MyProc->links.next = (SHM_QUEUE *) procglobal->autovacFreeProcs;
procglobal->autovacFreeProcs = MyProc;
}
else if (IsBackgroundWorker)
{
MyProc->links.next = (SHM_QUEUE *) procglobal->bgworkerFreeProcs;
procglobal->bgworkerFreeProcs = MyProc;
}
else
{
MyProc->links.next = (SHM_QUEUE *) procglobal->freeProcs;