database/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2026-10-23 18:49:29 +03:00
Code Activity

chore(tests): add data reclaim test.

Browse Source
This commit is contained in:
drrtuy
2026-07-04 13:26:06 +01:00
parent d56f207dcc
commit a3a458edf7
4 changed files with 261 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

160
storage/duckdb/docs/space-reclaim.md
View File

@@ -1,65 +1,131 @@
# DuckDB Space Reclaim
When tables are dropped or large amounts of data are deleted, the DuckDB database
file does not automatically shrink. Deleted blocks are marked as free but remain
in the file until explicitly reclaimed.
file does not automatically shrink. Deleted blocks are marked as free and reused
for new writes, but the file size on disk remains unchanged unless specific
conditions are met.
## Methods
## How DuckDB storage works
### 1. CHECKPOINT
DuckDB uses a single-file storage format with 256 KB blocks. A `free_list`
tracks which blocks are available for reuse. Key internals
(`single_file_block_manager.cpp`):
Forces a write-ahead log (WAL) flush to the main database file and marks deleted
blocks as reusable.
- **`free_list`** — set of block IDs that are free for reuse.
- **`max_block`** — highest allocated block ID (determines logical file size).
- **`Truncate()`** — called during FULL_CHECKPOINT; scans `free_list` from the
**end** and shrinks the file only while the **tail** blocks are contiguous and
free. Stops at the first occupied block.
- **`TrimFreeBlocks()`** — calls `fallocate(FALLOC_FL_PUNCH_HOLE)` on freed
blocks, returning physical space to the filesystem without changing logical
file size. Controlled by `DBConfig::options.trim_free_blocks` (not exposed as
a SQL setting in v1.5).
## What VACUUM actually does
`VACUUM` in DuckDB v1.5 does **NOT** shrink the file. The `PhysicalVacuum`
operator (`physical_vacuum.cpp`) only:
1. Scans table data to compute `DistinctStatistics` per column.
2. Calls `VacuumIndexes()` on the table storage.
It does **not** trigger a checkpoint, does not relocate blocks, and does not
call `Truncate()`. The SQL `VACUUM` command is essentially a statistics refresh.
## What CHECKPOINT does
`CHECKPOINT` / `FORCE CHECKPOINT` writes WAL contents to the main file and
calls `Truncate()`. However, file shrinking depends on:
1. **Checkpoint type** — only `FULL_CHECKPOINT` attempts truncation.
`CONCURRENT_CHECKPOINT` (used when other transactions are active) skips it.
2. **Block layout** — `Truncate()` removes free blocks from the tail only:
```cpp
for (auto entry = free_list.rbegin(); ...) {
if (block_id + 1 != max_block) break; // stop at first gap
max_block--;
}
```
If even one metadata or catalog block sits near the end of the file, nothing
is truncated.
3. **Vacuum lock** — `FULL_CHECKPOINT` requires an exclusive vacuum lock. If any
transaction holds a shared vacuum lock (any INSERT/DELETE in progress), the
checkpoint degrades to `CONCURRENT_CHECKPOINT`.
4. **Active transactions** — if `GetLastCommit() > LowestActiveStart()`, the
checkpoint is forced to `CONCURRENT_CHECKPOINT` regardless.
### Conditions for successful file truncation
All must be true simultaneously:
- No other DuckDB connections (each MariaDB THD holds a persistent connection).
- No active transactions (`FORCE CHECKPOINT` waits for them).
- The free blocks at the end of the file form a contiguous range from `max_block`
downward.
In practice, after a large table is dropped, metadata blocks (catalog, schemas)
often remain at high block IDs, preventing any truncation.
## Practical methods
### 1. FORCE CHECKPOINT (reuse space, may shrink)
```sql
SELECT run_in_duckdb('CHECKPOINT');
-- or force even when WAL is empty:
SELECT run_in_duckdb('FORCE CHECKPOINT');
```
**Effect**: Freed blocks become available for new data. The file size does **not**
decrease — space is reused internally.
**Effect**: WAL flushed, free blocks marked for reuse. File **may** shrink if
tail blocks are all free. In most cases, the file size stays the same but new
inserts will reuse freed blocks without growing the file.
**When to use**: After bulk deletes or DROP TABLE, before inserting new data into
other tables. Low cost, safe to run frequently.
**Important**: Close all other MariaDB connections first (each holds a DuckDB
connection via `DuckdbThdContext`).
### 2. VACUUM
### 2. Export / Reimport (guaranteed shrink)
Compacts the database file by relocating live data and truncating unused blocks
at the end of the file.
The only reliable way to reduce file size:
```sql
SELECT run_in_duckdb('VACUUM');
SELECT run_in_duckdb('EXPORT DATABASE ''/tmp/duckdb_export'' (FORMAT PARQUET)');
```
**Effect**: The `.duckdb` file **shrinks on disk**. This is a heavier operation
that rewrites portions of the file.
**When to use**: After dropping large tables or deleting significant portions of
data when actual disk space recovery is needed.
**Requirements** (DuckDB v1.1+):
- No active transactions on the database.
- Sufficient temporary disk space for block relocation.
### 3. Export / Reimport (nuclear option)
For maximum compaction or to reset fragmentation entirely:
```sql
SELECT run_in_duckdb('EXPORT DATABASE ''/tmp/duckdb_export''');
Then:
```bash
mysqladmin shutdown
rm /var/lib/mysql/duckdb.db
# restart MariaDB
```
Then stop MariaDB, delete the old `.duckdb` file, restart, and import:
```sql
SELECT run_in_duckdb('IMPORT DATABASE ''/tmp/duckdb_export''');
```
**Effect**: Produces a perfectly compact file with zero fragmentation.
**Effect**: New file contains only live data, perfectly compacted.
**When to use**: Rare maintenance scenarios where VACUUM is insufficient or the
database has accumulated significant fragmentation over time.
### 3. Delete the file (when no data is needed)
If the database is empty or data can be reloaded:
```bash
mysqladmin shutdown
rm /var/lib/mysql/duckdb.db
# restart MariaDB — a fresh empty file is created
```
## Diagnostics
```sql
SELECT run_in_duckdb('SELECT * FROM pragma_database_size()');
```
Returns `total_blocks`, `used_blocks`, `free_blocks`, `wal_size`. If
`free_blocks` is high but the file hasn't shrunk, it means `Truncate()` could
not find a contiguous free tail.
Check actual disk usage vs logical size:
```bash
ls -lh /var/lib/mysql/duckdb.db # logical size
du -sh /var/lib/mysql/duckdb.db # actual disk usage (differs if hole-punched)
```
## Configuration
@@ -73,19 +139,11 @@ loose-duckdb-checkpoint-threshold=268435456
A smaller threshold means more frequent automatic checkpoints but less WAL
accumulation.
## Behavior After DROP TABLE
When `ha_duckdb::delete_table()` executes `DROP TABLE` in DuckDB:
1. The table metadata and data blocks are marked as deleted.
2. If the WAL exceeds `checkpoint_threshold`, an automatic checkpoint occurs.
3. Otherwise, blocks remain in the WAL until the next checkpoint.
4. To immediately reclaim disk space, run `VACUUM` after the drop.
## Summary
| Method | File shrinks | Cost | Availability |
|--------------------|--------------|---------|--------------|
| CHECKPOINT | No | Low | Always |
| VACUUM | Yes | Medium | DuckDB 1.1+ |
| Export + Reimport | Yes | High | Always |
| Method | File shrinks | Reuses space | Cost | Reliability |
|--------------------|--------------|--------------|--------|-------------|
| FORCE CHECKPOINT | Rarely | Yes | Low | Always works for reuse |
| VACUUM | No | No | Low | Only updates stats |
| Export + Reimport | Yes | N/A | High | Always shrinks |
| Delete file | Yes | N/A | None | Loses all data |

75
storage/duckdb/mysql-test/duckdb/r/space_reclaim.result Normal file
View File

@@ -0,0 +1,75 @@
#
# DuckDB space reclaim test
#
# Verifies that after DROP TABLE + FORCE CHECKPOINT, freed blocks
# are reused by subsequent inserts (file does not keep growing).
#
# Uses 4 SHA2 columns (incompressible data) to guarantee multiple
# 256 KB blocks are allocated (~2-3 MB total).
#
#
# 1) Create a table and insert incompressible data to allocate blocks
#
CREATE DATABASE IF NOT EXISTS db_reclaim;
USE db_reclaim;
CREATE TABLE t_big (
id INT PRIMARY KEY,
c1 VARCHAR(200),
c2 VARCHAR(200),
c3 VARCHAR(200),
c4 VARCHAR(200)
) ENGINE = DuckDB;
SELECT COUNT(*) AS row_count FROM t_big;
row_count
1000
#
# 2) Force checkpoint to flush WAL to main file
#
SELECT run_in_duckdb('FORCE CHECKPOINT');
run_in_duckdb('FORCE CHECKPOINT')
Success
BOOLEAN
[ Rows: 0]
#
# 3) Record total_blocks after initial load
#
SELECT run_in_duckdb('SELECT total_blocks, used_blocks, free_blocks FROM pragma_database_size()') AS db_size_after_load;
db_size_after_load
total_blocks used_blocks free_blocks
BIGINT BIGINT BIGINT
[ Rows: 1]
3 3 0
total_blocks after load: 0
#
# 4) Drop table and force checkpoint — blocks become free
#
DROP TABLE t_big;
SELECT run_in_duckdb('FORCE CHECKPOINT');
run_in_duckdb('FORCE CHECKPOINT')
Success
BOOLEAN
[ Rows: 0]
#
# 5) Verify used_blocks decreased at least 10x after drop
#
SELECT run_in_duckdb('SELECT total_blocks, used_blocks, free_blocks FROM pragma_database_size()') AS db_size_after_drop;
db_size_after_drop
total_blocks used_blocks free_blocks
BIGINT BIGINT BIGINT
[ Rows: 1]
1 1 0
used_blocks after drop: 0
PASS: used_blocks after drop is at least 10x smaller than after load
#
# 6) Cleanup
#
USE test;
DROP DATABASE db_reclaim;

2
storage/duckdb/mysql-test/duckdb/t/space_reclaim-master.opt Normal file
View File

@@ -0,0 +1,2 @@
--loose-duckdb-allow-run-in-duckdb=ON
--plugin-maturity=alpha

75
storage/duckdb/mysql-test/duckdb/t/space_reclaim.test Normal file
View File

@@ -0,0 +1,75 @@
--source ../include/have_duckdb.inc
--source include/have_sequence.inc
--echo #
--echo # DuckDB space reclaim test
--echo #
--echo # Verifies that after DROP TABLE + FORCE CHECKPOINT, freed blocks
--echo # are reused by subsequent inserts (file does not keep growing).
--echo #
--echo # Uses 4 SHA2 columns (incompressible data) to guarantee multiple
--echo # 256 KB blocks are allocated (~2-3 MB total).
--echo #
--echo #
--echo # 1) Create a table and insert incompressible data to allocate blocks
--echo #
CREATE DATABASE IF NOT EXISTS db_reclaim;
USE db_reclaim;
CREATE TABLE t_big (
id INT PRIMARY KEY,
c1 VARCHAR(200),
c2 VARCHAR(200),
c3 VARCHAR(200),
c4 VARCHAR(200)
) ENGINE = DuckDB;
--disable_query_log
INSERT INTO t_big
SELECT seq,
SHA2(CONCAT(seq, 'a'), 256),
SHA2(CONCAT(seq, 'b'), 256),
SHA2(CONCAT(seq, 'c'), 256),
SHA2(CONCAT(seq, 'd'), 256)
FROM test.seq_1_to_1000;
--enable_query_log
SELECT COUNT(*) AS row_count FROM t_big;
--echo #
--echo # 2) Force checkpoint to flush WAL to main file
--echo #
SELECT run_in_duckdb('FORCE CHECKPOINT');
--echo #
--echo # 3) Record total_blocks after initial load
--echo #
SELECT run_in_duckdb('SELECT total_blocks, used_blocks, free_blocks FROM pragma_database_size()') AS db_size_after_load;
--let $blocks_after_load = `SELECT CAST(TRIM(SUBSTRING_INDEX(SUBSTRING_INDEX(run_in_duckdb('SELECT total_blocks FROM pragma_database_size()'), '\n', 3), '\n', -1)) AS UNSIGNED)`
--echo total_blocks after load: $blocks_after_load
--echo #
--echo # 4) Drop table and force checkpoint — blocks become free
--echo #
DROP TABLE t_big;
SELECT run_in_duckdb('FORCE CHECKPOINT');
--echo #
--echo # 5) Verify used_blocks decreased at least 10x after drop
--echo #
SELECT run_in_duckdb('SELECT total_blocks, used_blocks, free_blocks FROM pragma_database_size()') AS db_size_after_drop;
--let $blocks_after_drop = `SELECT CAST(TRIM(SUBSTRING_INDEX(SUBSTRING_INDEX(run_in_duckdb('SELECT used_blocks FROM pragma_database_size()'), '\n', 3), '\n', -1)) AS UNSIGNED)`
--echo used_blocks after drop: $blocks_after_drop
if (`SELECT $blocks_after_drop > $blocks_after_load / 10`)
{
--die Space reclaim failed: used_blocks after drop is not 10x smaller than after load
}
--echo PASS: used_blocks after drop is at least 10x smaller than after load
--echo #
--echo # 6) Cleanup
--echo #
USE test;
DROP DATABASE db_reclaim;