database/postgres
database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-04-25 21:42:33 +03:00
Code

Add simple VACUUM progress reporting.

Browse Source 
There's a lot more that could be done here yet - in particular, this
reports only very coarse-grained information about the index vacuuming
phase - but even as it stands, the new pg_stat_progress_vacuum can
tell you quite a bit about what a long-running vacuum is actually
doing.

Amit Langote and Robert Haas, based on earlier work by Vinayak Pokale
and Rahila Syed.
This commit is contained in:
Robert Haas 2016-03-15 13:31:18 -04:00
parent 0e9b89986b
commit c16dc1aca5
8 changed files with 394 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

207
doc/src/sgml/monitoring.sgml
View File

@ -507,6 +507,12 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
yet included in <structname>pg_stat_user_functions</>).</entry>
</row>
<row>
<entry><structname>pg_stat_progress_vacuum</><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry>
<entry>One row for each backend (including autovacuum worker processes) running
<command>VACUUM</>, showing current progress.
See <xref linkend='vacuum-progress-reporting'>.</entry>
</row>
</tbody>
</tgroup>
</table>
@ -2490,6 +2496,207 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
</para>
</sect1>
<sect1 id="progress-reporting">
<title>Progress Reporting</title>
<para>
<productname>PostgreSQL</> has the ability to report the progress of
certain commands during command execution. Currently, the only command
which supports progress reporting is <command>VACUUM</>. This may be
expanded in the future.
</para>
<sect2 id="vacuum-progress-reporting">
<title>VACUUM Progress Reporting</title>
<para>
Whenever <command>VACUUM</> is running, the
<structname>pg_stat_progress_vacuum</structname> view will contain
one row for each backend (including autovacuum worker processes) that is
currently vacuuming. The tables below describe the information
that will be reported and provide information about how to interpret it.
Progress reporting is not currently supported for <command>VACUUM FULL</>
and backends running <command>VACUUM FULL</> will not be listed in this
view.
</para>
<table id="pg-stat-progress-vacuum" xreflabel="pg_stat_progress_vacuum">
<title><structname>pg_stat_progress_vacuum</structname> View</title>
<tgroup cols="3">
<thead>
<row>
<entry>Column</entry>
<entry>Type</entry>
<entry>Description</entry>
</row>
</thead>
<tbody>
<row>
<entry><structfield>pid</></entry>
<entry><type>integer</></entry>
<entry>Process ID of backend.</entry>
</row>
<row>
<entry><structfield>datid</></entry>
<entry><type>oid</></entry>
<entry>OID of the database to which this backend is connected.</entry>
</row>
<row>
<entry><structfield>datname</></entry>
<entry><type>name</></entry>
<entry>Name of the database to which this backend is connected.</entry>
</row>
<row>
<entry><structfield>relid</></entry>
<entry><type>oid</></entry>
<entry>OID of the table being vacuumed.</entry>
</row>
<row>
<entry><structfield>phase</></entry>
<entry><type>text</></entry>
<entry>
Current processing phase of vacuum. See <xref linkend='vacuum-phases'>.
</entry>
</row>
<row>
<entry><structfield>heap_blks_total</></entry>
<entry><type>bigint</></entry>
<entry>
Total number of heap blocks in the table. This number is reported
as of the beginning of the scan; blocks added later will not be (and
need not be) visited by this <command>VACUUM</>.
</entry>
</row>
<row>
<entry><structfield>heap_blks_scanned</></entry>
<entry><type>bigint</></entry>
<entry>
Number of heap blocks scanned. Because the
<link linkend="storage-vm">visibility map</> is used to optimize scans,
some blocks will be skipped without inspection; skipped blocks are
included this total, so that this number will eventually become
equal to <structfield>heap_blks_total</> when the vacuum is complete.
This counter only advances when the phase is <literal>scanning heap</>.
</entry>
</row>
<row>
<entry><structfield>heap_blks_vacuumed</></entry>
<entry><type>bigint</></entry>
<entry>
Number of heap blocks vacuumed. Unless the table has no indexes, this
counter only advances when the phase is <literal>vacuuming heap</>.
Blocks that contain no dead tuples are skipped, so the counter may
sometimes skip forward in large increments.
</entry>
</row>
<row>
<entry><structfield>index_vacuum_count</></entry>
<entry><type>bigint</></entry>
<entry>
Number of completed index vacuums cycles.
</entry>
</row>
<row>
<entry><structfield>max_dead_tuples</></entry>
<entry><type>bigint</></entry>
<entry>
Number of dead tuples that we can store before needing to perform
an index vacuum cycle, based on
<xref linkend="guc-maintenance-work-mem">.
</entry>
</row>
<row>
<entry><structfield>num_dead_tuples</></entry>
<entry><type>bigint</></entry>
<entry>
Number of dead tuples collected since the last index vacuum cycle.
</entry>
</row>
</tbody>
</tgroup>
</table>
<table id="vacuum-phases" xreflabel="VACUUM phases">
<title>VACUUM phases</title>
<tgroup cols="2">
<thead>
<row>
<entry>Phase</entry>
<entry>Description</entry>
</row>
</thead>
<tbody>
<row>
<entry><literal>initializing</literal></entry>
<entry>
<command>VACUUM</> is preparing to begin scanning the heap. This
phase is expected to be very brief.
</entry>
</row>
<row>
<entry><literal>scanning heap</literal></entry>
<entry>
<command>VACUUM</> is currently scanning the heap. It will prune and
defragment each page if required, and possibly perform freezing
activity. The <structfield>heap_blks_scanned</> column can be used
to monitor the progress of the scan.
</entry>
</row>
<row>
<entry><literal>vacuuming indexes</literal></entry>
<entry>
<command>VACUUM</> is currently vacuuming the indexes. If a table has
any indexes, this will happen at least once per vacuum, after the heap
has been completely scanned. It may happen multiple times per vacuum
if <xref linkend="guc-maintenance-work-mem"> is insufficient to
store the number of dead tuples found.
</entry>
</row>
<row>
<entry><literal>vacuuming heap</literal></entry>
<entry>
<command>VACUUM</> is currently vacuuming the heap. Vacuuming the heap
is distinct from scanning the heap, and occurs after each instance of
vacuuming indexes. If <structfield>heap_blks_scanned</> is less than
<structfield>heap_blks_total</>, the system will return to scanning
the heap after this phase is completed; otherwise, it will begin
cleaning up indexes after this phase is completed.
</entry>
</row>
<row>
<entry><literal>cleaning up indexes</literal></entry>
<entry>
<command>VACUUM</> is currently cleaning up indexes. This occurs after
the heap has been completely scanned and all vacuuming of the indexes
and the heap has been completed.
</entry>
</row>
<row>
<entry><literal>truncating heap</literal></entry>
<entry>
<command>VACUUM</> is currently truncating the heap so as to return
empty pages at the end of the relation to the operating system. This
occurs after cleaning up indexes.
</entry>
</row>
<row>
<entry><literal>performing final cleanup</literal></entry>
<entry>
<command>VACUUM</> is performing final cleanup. During this phase,
<command>VACUUM</> will vacuum the free space map, update statistics
in <literal>pg_class</>, and report statistics to the statistics
collector. When this phase is completed, <command>VACUUM</> will end.
</entry>
</row>
</tbody>
</tgroup>
</table>
</sect2>
</sect1>
<sect1 id="dynamic-trace">
<title>Dynamic Tracing</title>

18
src/backend/catalog/system_views.sql
View File

@ -796,6 +796,24 @@ CREATE VIEW pg_stat_bgwriter AS
pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
CREATE VIEW pg_stat_progress_vacuum AS
SELECT
S.pid AS pid, S.datid AS datid, D.datname AS datname,
S.relid AS relid,
CASE S.param1 WHEN 0 THEN 'initializing'
WHEN 1 THEN 'scanning heap'
WHEN 2 THEN 'vacuuming indexes'
WHEN 3 THEN 'vacuuming heap'
WHEN 4 THEN 'cleaning up indexes'
WHEN 5 THEN 'truncating heap'
WHEN 6 THEN 'performing final cleanup'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
FROM pg_stat_get_progress_info('VACUUM') AS S
JOIN pg_database D ON S.datid = D.oid;
CREATE VIEW pg_user_mappings AS
SELECT
U.oid AS umid,

78
src/backend/commands/vacuumlazy.c
View File

@ -48,6 +48,7 @@
#include "catalog/catalog.h"
#include "catalog/storage.h"
#include "commands/dbcommands.h"
#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
#include "pgstat.h"
@ -272,6 +273,10 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params,
if (should_attempt_truncation(vacrelstats))
lazy_truncate_heap(onerel, vacrelstats);
/* Report that we are now doing final cleanup */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_FINAL_CLEANUP);
/* Vacuum the Free Space Map */
FreeSpaceMapVacuum(onerel);
@ -457,6 +462,12 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
bool skipping_blocks;
xl_heap_freeze_tuple *frozen;
StringInfoData buf;
const int initprog_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
PROGRESS_VACUUM_MAX_DEAD_TUPLES
};
int64 initprog_val[3];
pg_rusage_init(&ru0);
@ -481,6 +492,12 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
lazy_space_alloc(vacrelstats, nblocks);
frozen = palloc(sizeof(xl_heap_freeze_tuple) * MaxHeapTuplesPerPage);
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = nblocks;
initprog_val[2] = vacrelstats->max_dead_tuples;
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/*
* Except when aggressive is set, we want to skip pages that are
* all-visible according to the visibility map, but only when we can skip
@ -572,6 +589,8 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
#define FORCE_CHECK_PAGE() \
(blkno == nblocks - 1 && should_attempt_truncation(vacrelstats))
pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
if (blkno == next_unskippable_block)
{
/* Time to advance next_unskippable_block */
@ -652,6 +671,12 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
if ((vacrelstats->max_dead_tuples - vacrelstats->num_dead_tuples) < MaxHeapTuplesPerPage &&
vacrelstats->num_dead_tuples > 0)
{
const int hvp_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_NUM_INDEX_VACUUMS
};
int64 hvp_val[2];
/*
* Before beginning index vacuuming, we release any pin we may
* hold on the visibility map page. This isn't necessary for
@ -667,11 +692,26 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
/* Log cleanup info before we touch indexes */
vacuum_log_cleanup_info(onerel, vacrelstats);
/* Report that we are now vacuuming indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
/* Remove index entries */
for (i = 0; i < nindexes; i++)
lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
/*
* Report that we are now vacuuming the heap. We also increase
* the number of index scans here; note that by using
* pgstat_progress_update_multi_param we can update both
* parameters atomically.
*/
hvp_val[0] = PROGRESS_VACUUM_PHASE_VACUUM_HEAP;
hvp_val[1] = vacrelstats->num_index_scans + 1;
pgstat_progress_update_multi_param(2, hvp_index, hvp_val);
/* Remove tuples from heap */
lazy_vacuum_heap(onerel, vacrelstats);
@ -682,6 +722,10 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
*/
vacrelstats->num_dead_tuples = 0;
vacrelstats->num_index_scans++;
/* Report that we are once again scanning the heap */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_SCAN_HEAP);
}
/*
@ -1182,6 +1226,10 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
RecordPageWithFreeSpace(onerel, blkno, freespace);
}
/* report that everything is scanned and vacuumed */
pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_VACUUMED, blkno);
pfree(frozen);
/* save stats for use later */
@ -1208,19 +1256,41 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
/* XXX put a threshold on min number of tuples here? */
if (vacrelstats->num_dead_tuples > 0)
{
const int hvp_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_NUM_INDEX_VACUUMS
};
int64 hvp_val[2];
/* Log cleanup info before we touch indexes */
vacuum_log_cleanup_info(onerel, vacrelstats);
/* Report that we are now vacuuming indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
/* Remove index entries */
for (i = 0; i < nindexes; i++)
lazy_vacuum_index(Irel[i],
&indstats[i],
vacrelstats);
/* Report that we are now vacuuming the heap */
hvp_val[0] = PROGRESS_VACUUM_PHASE_VACUUM_HEAP;
hvp_val[1] = vacrelstats->num_index_scans + 1;
pgstat_progress_update_multi_param(2, hvp_index, hvp_val);
/* Remove tuples from heap */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_HEAP);
lazy_vacuum_heap(onerel, vacrelstats);
vacrelstats->num_index_scans++;
}
/* report we're now in the cleanup phase */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
/* Do post-vacuum cleanup and statistics update for each index */
for (i = 0; i < nindexes; i++)
lazy_cleanup_index(Irel[i], indstats[i], vacrelstats);
@ -1350,6 +1420,8 @@ lazy_vacuum_page(Relation onerel, BlockNumber blkno, Buffer buffer,
TransactionId visibility_cutoff_xid;
bool all_frozen;
pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_VACUUMED, blkno);
START_CRIT_SECTION();
for (; tupindex < vacrelstats->num_dead_tuples; tupindex++)
@ -1607,6 +1679,10 @@ lazy_truncate_heap(Relation onerel, LVRelStats *vacrelstats)
pg_rusage_init(&ru0);
/* Report that we are now truncating */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_TRUNCATE);
/*
* Loop until no more truncating can be done.
*/
@ -1887,6 +1963,8 @@ lazy_record_dead_tuple(LVRelStats *vacrelstats,
{
vacrelstats->dead_tuples[vacrelstats->num_dead_tuples] = *itemptr;
vacrelstats->num_dead_tuples++;
pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
vacrelstats->num_dead_tuples);
}
}

29
src/backend/postmaster/pgstat.c
View File

@ -2902,6 +2902,35 @@ pgstat_progress_update_param(int index, int64 val)
pgstat_increment_changecount_after(beentry);
}
/*-----------
* pgstat_progress_update_params() -
*
* Automatically update multiple members in st_progress_param[] of own backend
* entry.
*-----------
*/
void
pgstat_progress_update_multi_param(int nparam, const int *index,
const int64 *val)
{
volatile PgBackendStatus *beentry = MyBEEntry;
int i;
if (!beentry || !pgstat_track_activities || nparam == 0)
return;
pgstat_increment_changecount_before(beentry);
for (i = 0; i < nparam; ++i)
{
Assert(index[i] >= 0 && index[i] < PGSTAT_NUM_PROGRESS_PARAM);
beentry->st_progress_param[index[i]] = val[i];
}
pgstat_increment_changecount_after(beentry);
}
/*-----------
* pgstat_progress_end_command() -
*

2
src/include/catalog/catversion.h
View File

@ -53,6 +53,6 @@
*/
/* yyyymmddN */
#define CATALOG_VERSION_NO 201603111
#define CATALOG_VERSION_NO 201603151
#endif

37
src/include/commands/progress.h Normal file
View File

@ -0,0 +1,37 @@
/*-------------------------------------------------------------------------
*
* progress.h
* Constants used with the progress reporting facilities defined in
* pgstat.h. These are possibly interesting to extensions, so we
* expose them via this header file. Note that if you update these
* constants, you probably also need to update the views based on them
* in system_views.sql.
*
* Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
* src/include/commands/progress.h
*
*-------------------------------------------------------------------------
*/
#ifndef PROGRESS_H
#define PROGRESS_H
/* Progress parameters for (lazy) vacuum */
#define PROGRESS_VACUUM_PHASE 0
#define PROGRESS_VACUUM_TOTAL_HEAP_BLKS 1
#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX 2
#define PROGRESS_VACUUM_PHASE_VACUUM_HEAP 3
#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
#endif

2
src/include/pgstat.h
View File

@ -978,6 +978,8 @@ extern const char *pgstat_get_crashed_backend_activity(int pid, char *buffer,
extern void pgstat_progress_start_command(ProgressCommandType cmdtype,
Oid relid);
extern void pgstat_progress_update_param(int index, int64 val);
extern void pgstat_progress_update_multi_param(int nparam, const int *index,
const int64 *val);
extern void pgstat_progress_end_command(void);
extern PgStat_TableStatus *find_tabstat_entry(Oid rel_id);

22
src/test/regress/expected/rules.out
View File

@ -1747,6 +1747,28 @@ pg_stat_database_conflicts| SELECT d.oid AS datid,
pg_stat_get_db_conflict_bufferpin(d.oid) AS confl_bufferpin,
pg_stat_get_db_conflict_startup_deadlock(d.oid) AS confl_deadlock
FROM pg_database d;
pg_stat_progress_vacuum| SELECT s.pid,
s.datid,
d.datname,
s.relid,
CASE s.param1
WHEN 0 THEN 'initializing'::text
WHEN 1 THEN 'scanning heap'::text
WHEN 2 THEN 'vacuuming indexes'::text
WHEN 3 THEN 'vacuuming heap'::text
WHEN 4 THEN 'cleaning up indexes'::text
WHEN 5 THEN 'truncating heap'::text
WHEN 6 THEN 'performing final cleanup'::text
ELSE NULL::text
END AS phase,
s.param2 AS heap_blks_total,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
s.param7 AS num_dead_tuples
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,