database/postgres
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-11-24 00:23:06 +03:00
Code Activity

Introduce a new smgr bulk loading facility.

Browse Source 
The new facility makes it easier to optimize bulk loading, as the
logic for buffering, WAL-logging, and syncing the relation only needs
to be implemented once. It's also less error-prone: We have had a
number of bugs in how a relation is fsync'd - or not - at the end of a
bulk loading operation. By centralizing that logic to one place, we
only need to write it correctly once.

The new facility is faster for small relations: Instead of of calling
smgrimmedsync(), we register the fsync to happen at next checkpoint,
which avoids the fsync latency. That can make a big difference if you
are e.g. restoring a schema-only dump with lots of relations.

It is also slightly more efficient with large relations, as the WAL
logging is performed multiple pages at a time. That avoids some WAL
header overhead. The sorted GiST index build did that already, this
moves the buffering to the new facility.

The changes to pageinspect GiST test needs an explanation: Before this
patch, the sorted GiST index build set the LSN on every page to the
special GistBuildLSN value, not the LSN of the WAL record, even though
they were WAL-logged. There was no particular need for it, it just
happened naturally when we wrote out the pages before WAL-logging
them. Now we WAL-log the pages first, like in B-tree build, so the
pages are stamped with the record's real LSN. When the build is not
WAL-logged, we still use GistBuildLSN. To make the test output
predictable, use an unlogged index.

Reviewed-by: Andres Freund
Discussion: https://www.postgresql.org/message-id/30e8f366-58b3-b239-c521-422122dd5150%40iki.fi
This commit is contained in:
Heikki Linnakangas
2024-02-23 16:10:51 +02:00
parent e612384fc7
commit 8af2565248
17 changed files with 551 additions and 354 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
src/backend/catalog/storage.c
View File

@@ -28,6 +28,7 @@
#include "catalog/storage.h"
#include "catalog/storage_xlog.h"
#include "miscadmin.h"
#include "storage/bulk_write.h"
#include "storage/freespace.h"
#include "storage/smgr.h"
#include "utils/hsearch.h"
@@ -451,14 +452,11 @@ void
RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
ForkNumber forkNum, char relpersistence)
{
PGIOAlignedBlock buf;
Page page;
bool use_wal;
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
page = (Page) buf.data;
BulkWriteState *bulkstate;
/*
* The init fork for an unlogged relation in many respects has to be
@@ -477,16 +475,21 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
bulkstate = smgr_bulk_start_smgr(dst, forkNum, use_wal);
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
{
BulkWriteBuffer buf;
/* If we got a cancel signal during the copy of the data, quit */
CHECK_FOR_INTERRUPTS();
smgrread(src, forkNum, blkno, buf.data);
buf = smgr_bulk_get_buf(bulkstate);
smgrread(src, forkNum, blkno, (Page) buf);
if (!PageIsVerifiedExtended(page, blkno,
if (!PageIsVerifiedExtended((Page) buf, blkno,
PIV_LOG_WARNING | PIV_REPORT_STAT))
{
/*
@@ -507,34 +510,13 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
}
/*
* WAL-log the copied page. Unfortunately we don't know what kind of a
* page this is, so we have to log the full page including any unused
* space.
* Queue the page for WAL-logging and writing out. Unfortunately we
* don't know what kind of a page this is, so we have to log the full
* page including any unused space.
*/
if (use_wal)
log_newpage(&dst->smgr_rlocator.locator, forkNum, blkno, page, false);
PageSetChecksumInplace(page, blkno);
/*
* Now write the page. We say skipFsync = true because there's no
* need for smgr to schedule an fsync for this write; we'll do it
* ourselves below.
*/
smgrextend(dst, forkNum, blkno, buf.data, true);
smgr_bulk_write(bulkstate, blkno, buf, false);
}
/*
* When we WAL-logged rel pages, we must nonetheless fsync them. The
* reason is that since we're copying outside shared buffers, a CHECKPOINT
* occurring during the copy has no way to flush the previously written
* data to disk (indeed it won't know the new rel even exists). A crash
* later on would replay WAL from the checkpoint, therefore it wouldn't
* replay our earlier WAL entries. If we do not fsync those pages here,
* they might still not be on disk when the crash occurs.
*/
if (use_wal || copying_initfork)
smgrimmedsync(dst, forkNum);
smgr_bulk_finish(bulkstate);
}
/*