database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-11-24 00:23:06 +03:00
Code Activity

Add function to log the memory contexts of specified backend process.

Browse Source 
Commit 3e98c0bafb added pg_backend_memory_contexts view to display
the memory contexts of the backend process. However its target process
is limited to the backend that is accessing to the view. So this is
not so convenient when investigating the local memory bloat of other
backend process. To improve this situation, this commit adds
pg_log_backend_memory_contexts() function that requests to log
the memory contexts of the specified backend process.

This information can be also collected by calling
MemoryContextStats(TopMemoryContext) via a debugger. But
this technique cannot be used in some environments because no debugger
is available there. So, pg_log_backend_memory_contexts() allows us to
see the memory contexts of specified backend more easily.

Only superusers are allowed to request to log the memory contexts
because allowing any users to issue this request at an unbounded rate
would cause lots of log messages and which can lead to denial of service.

On receipt of the request, at the next CHECK_FOR_INTERRUPTS(),
the target backend logs its memory contexts at LOG_SERVER_ONLY level,
so that these memory contexts will appear in the server log but not
be sent to the client. It logs one message per memory context.
Because if it buffers all memory contexts into StringInfo to log them
as one message, which may require the buffer to be enlarged very much
and lead to OOM error since there can be a large number of memory
contexts in a backend.

When a backend process is consuming huge memory, logging all its
memory contexts might overrun available disk space. To prevent this,
now this patch limits the number of child contexts to log per parent
to 100. As with MemoryContextStats(), it supposes that practical cases
where the log gets long will typically be huge numbers of siblings
under the same parent context; while the additional debugging value
from seeing details about individual siblings beyond 100 will not be large.

There was another proposed patch to add the function to return
the memory contexts of specified backend as the result sets,
instead of logging them, in the discussion. However that patch is
not included in this commit because it had several issues to address.

Thanks to Tatsuhito Kasahara, Andres Freund, Tom Lane, Tomas Vondra,
Michael Paquier, Kyotaro Horiguchi and Zhihong Yu for the discussion.

Bump catalog version.

Author: Atsushi Torikoshi
Reviewed-by: Kyotaro Horiguchi, Zhihong Yu, Fujii Masao
Discussion: https://postgr.es/m/0271f440ac77f2a4180e0e56ebd944d1@oss.nttdata.com
This commit is contained in:
Fujii Masao
2021-04-06 13:44:15 +09:00
parent 5a71964a83
commit 43620e3286
17 changed files with 319 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

178
src/backend/utils/mmgr/mcxt.c
View File

@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "storage/procsignal.h"
#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -55,9 +59,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
MemoryContextCounters *totals);
MemoryContextCounters *totals,
bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
const char *stats_string);
const char *stats_string,
bool print_to_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -499,28 +505,52 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
MemoryContextStatsDetail(context, 100);
MemoryContextStatsDetail(context, 100, true);
}
/*
* MemoryContextStatsDetail
*
* Entry point for use if you want to vary the number of child contexts shown.
*
* If print_to_stderr is true, print statistics about the memory contexts
* with fprintf(stderr), otherwise use ereport().
*/
void
MemoryContextStatsDetail(MemoryContext context, int max_children)
MemoryContextStatsDetail(MemoryContext context, int max_children,
bool print_to_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, print_to_stderr);
fprintf(stderr,
"Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
grand_totals.totalspace, grand_totals.nblocks,
grand_totals.freespace, grand_totals.freechunks,
grand_totals.totalspace - grand_totals.freespace);
if (print_to_stderr)
fprintf(stderr,
"Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
grand_totals.totalspace, grand_totals.nblocks,
grand_totals.freespace, grand_totals.freechunks,
grand_totals.totalspace - grand_totals.freespace);
else
/*
* Use LOG_SERVER_ONLY to prevent the memory contexts from being sent
* to the connected client.
*
* We don't buffer the information about all memory contexts in a
* backend into StringInfo and log it as one message. Otherwise which
* may require the buffer to be enlarged very much and lead to OOM
* error since there can be a large number of memory contexts in a
* backend. Instead, we log one message per memory context.
*/
ereport(LOG_SERVER_ONLY,
(errhidestmt(true),
errhidecontext(true),
errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
grand_totals.totalspace, grand_totals.nblocks,
grand_totals.freespace, grand_totals.freechunks,
grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +563,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
MemoryContextCounters *totals)
MemoryContextCounters *totals,
bool print_to_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +576,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
totals);
totals, print_to_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +591,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
totals);
totals,
print_to_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
&local_totals);
&local_totals,
print_to_stderr);
}
/* Deal with excess children */
@@ -572,18 +605,33 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
if (print)
{
int i;
if (print_to_stderr)
{
int i;
for (i = 0; i <= level; i++)
fprintf(stderr, " ");
fprintf(stderr,
"%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
ichild - max_children,
local_totals.totalspace,
local_totals.nblocks,
local_totals.freespace,
local_totals.freechunks,
local_totals.totalspace - local_totals.freespace);
for (i = 0; i <= level; i++)
fprintf(stderr, " ");
fprintf(stderr,
"%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
ichild - max_children,
local_totals.totalspace,
local_totals.nblocks,
local_totals.freespace,
local_totals.freechunks,
local_totals.totalspace - local_totals.freespace);
}
else
ereport(LOG_SERVER_ONLY,
(errhidestmt(true),
errhidecontext(true),
errmsg_internal("level: %d; %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
level,
ichild - max_children,
local_totals.totalspace,
local_totals.nblocks,
local_totals.freespace,
local_totals.freechunks,
local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,11 +653,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
const char *stats_string)
const char *stats_string,
bool print_to_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
char truncated_ident[110];
int i;
/*
@@ -623,9 +673,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
for (i = 0; i < level; i++)
fprintf(stderr, " ");
fprintf(stderr, "%s: %s", name, stats_string);
truncated_ident[0] = '\0';
if (ident)
{
/*
@@ -637,24 +686,41 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
strcpy(truncated_ident, ": ");
i = strlen(truncated_ident);
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
fprintf(stderr, ": ");
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
fputc(c, stderr);
truncated_ident[i++] = c;
}
truncated_ident[i] = '\0';
if (truncated)
fprintf(stderr, "...");
strcat(truncated_ident, "...");
}
fputc('\n', stderr);
if (print_to_stderr)
{
for (i = 0; i < level; i++)
fprintf(stderr, " ");
fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
}
else
ereport(LOG_SERVER_ONLY,
(errhidestmt(true),
errhidecontext(true),
errmsg_internal("level: %d; %s: %s%s",
level, name, stats_string, truncated_ident)));
}
/*
@@ -946,6 +1012,52 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
return ret;
}
/*
* HandleLogMemoryContextInterrupt
* Handle receipt of an interrupt indicating logging of memory
* contexts.
*
* All the actual work is deferred to ProcessLogMemoryContextInterrupt(),
* because we cannot safely emit a log message inside the signal handler.
*/
void
HandleLogMemoryContextInterrupt(void)
{
InterruptPending = true;
LogMemoryContextPending = true;
/* latch will be set by procsignal_sigusr1_handler */
}
/*
* ProcessLogMemoryContextInterrupt
* Perform logging of memory contexts of this backend process.
*
* Any backend that participates in ProcSignal signaling must arrange
* to call this function if we see LogMemoryContextPending set.
* It is called from CHECK_FOR_INTERRUPTS(), which is enough because
* the target process for logging of memory contexts is a backend.
*/
void
ProcessLogMemoryContextInterrupt(void)
{
LogMemoryContextPending = false;
ereport(LOG,
(errmsg("logging memory contexts of PID %d", MyProcPid)));
/*
* When a backend process is consuming huge memory, logging all its memory
* contexts might overrun available disk space. To prevent this, we limit
* the number of child contexts to log per parent to 100.
*
* As with MemoryContextStats(), we suppose that practical cases where the
* dump gets long will typically be huge numbers of siblings under the
* same parent context; while the additional debugging value from seeing
* details about individual siblings beyond 100 will not be large.
*/
MemoryContextStatsDetail(TopMemoryContext, 100, false);
}
void *
palloc(Size size)
{