diff --git a/storage/duckdb/docs/space-reclaim.md b/storage/duckdb/docs/space-reclaim.md index e65a051ccc3..640a05db46a 100644 --- a/storage/duckdb/docs/space-reclaim.md +++ b/storage/duckdb/docs/space-reclaim.md @@ -1,65 +1,131 @@ # DuckDB Space Reclaim When tables are dropped or large amounts of data are deleted, the DuckDB database -file does not automatically shrink. Deleted blocks are marked as free but remain -in the file until explicitly reclaimed. +file does not automatically shrink. Deleted blocks are marked as free and reused +for new writes, but the file size on disk remains unchanged unless specific +conditions are met. -## Methods +## How DuckDB storage works -### 1. CHECKPOINT +DuckDB uses a single-file storage format with 256 KB blocks. A `free_list` +tracks which blocks are available for reuse. Key internals +(`single_file_block_manager.cpp`): -Forces a write-ahead log (WAL) flush to the main database file and marks deleted -blocks as reusable. +- **`free_list`** — set of block IDs that are free for reuse. +- **`max_block`** — highest allocated block ID (determines logical file size). +- **`Truncate()`** — called during FULL_CHECKPOINT; scans `free_list` from the + **end** and shrinks the file only while the **tail** blocks are contiguous and + free. Stops at the first occupied block. +- **`TrimFreeBlocks()`** — calls `fallocate(FALLOC_FL_PUNCH_HOLE)` on freed + blocks, returning physical space to the filesystem without changing logical + file size. Controlled by `DBConfig::options.trim_free_blocks` (not exposed as + a SQL setting in v1.5). + +## What VACUUM actually does + +`VACUUM` in DuckDB v1.5 does **NOT** shrink the file. The `PhysicalVacuum` +operator (`physical_vacuum.cpp`) only: + +1. Scans table data to compute `DistinctStatistics` per column. +2. Calls `VacuumIndexes()` on the table storage. + +It does **not** trigger a checkpoint, does not relocate blocks, and does not +call `Truncate()`. The SQL `VACUUM` command is essentially a statistics refresh. + +## What CHECKPOINT does + +`CHECKPOINT` / `FORCE CHECKPOINT` writes WAL contents to the main file and +calls `Truncate()`. However, file shrinking depends on: + +1. **Checkpoint type** — only `FULL_CHECKPOINT` attempts truncation. + `CONCURRENT_CHECKPOINT` (used when other transactions are active) skips it. +2. **Block layout** — `Truncate()` removes free blocks from the tail only: + ```cpp + for (auto entry = free_list.rbegin(); ...) { + if (block_id + 1 != max_block) break; // stop at first gap + max_block--; + } + ``` + If even one metadata or catalog block sits near the end of the file, nothing + is truncated. +3. **Vacuum lock** — `FULL_CHECKPOINT` requires an exclusive vacuum lock. If any + transaction holds a shared vacuum lock (any INSERT/DELETE in progress), the + checkpoint degrades to `CONCURRENT_CHECKPOINT`. +4. **Active transactions** — if `GetLastCommit() > LowestActiveStart()`, the + checkpoint is forced to `CONCURRENT_CHECKPOINT` regardless. + +### Conditions for successful file truncation + +All must be true simultaneously: +- No other DuckDB connections (each MariaDB THD holds a persistent connection). +- No active transactions (`FORCE CHECKPOINT` waits for them). +- The free blocks at the end of the file form a contiguous range from `max_block` + downward. + +In practice, after a large table is dropped, metadata blocks (catalog, schemas) +often remain at high block IDs, preventing any truncation. + +## Practical methods + +### 1. FORCE CHECKPOINT (reuse space, may shrink) ```sql -SELECT run_in_duckdb('CHECKPOINT'); --- or force even when WAL is empty: SELECT run_in_duckdb('FORCE CHECKPOINT'); ``` -**Effect**: Freed blocks become available for new data. The file size does **not** -decrease — space is reused internally. +**Effect**: WAL flushed, free blocks marked for reuse. File **may** shrink if +tail blocks are all free. In most cases, the file size stays the same but new +inserts will reuse freed blocks without growing the file. -**When to use**: After bulk deletes or DROP TABLE, before inserting new data into -other tables. Low cost, safe to run frequently. +**Important**: Close all other MariaDB connections first (each holds a DuckDB +connection via `DuckdbThdContext`). -### 2. VACUUM +### 2. Export / Reimport (guaranteed shrink) -Compacts the database file by relocating live data and truncating unused blocks -at the end of the file. +The only reliable way to reduce file size: ```sql -SELECT run_in_duckdb('VACUUM'); +SELECT run_in_duckdb('EXPORT DATABASE ''/tmp/duckdb_export'' (FORMAT PARQUET)'); ``` -**Effect**: The `.duckdb` file **shrinks on disk**. This is a heavier operation -that rewrites portions of the file. - -**When to use**: After dropping large tables or deleting significant portions of -data when actual disk space recovery is needed. - -**Requirements** (DuckDB v1.1+): -- No active transactions on the database. -- Sufficient temporary disk space for block relocation. - -### 3. Export / Reimport (nuclear option) - -For maximum compaction or to reset fragmentation entirely: - -```sql -SELECT run_in_duckdb('EXPORT DATABASE ''/tmp/duckdb_export'''); +Then: +```bash +mysqladmin shutdown +rm /var/lib/mysql/duckdb.db +# restart MariaDB ``` -Then stop MariaDB, delete the old `.duckdb` file, restart, and import: - ```sql SELECT run_in_duckdb('IMPORT DATABASE ''/tmp/duckdb_export'''); ``` -**Effect**: Produces a perfectly compact file with zero fragmentation. +**Effect**: New file contains only live data, perfectly compacted. -**When to use**: Rare maintenance scenarios where VACUUM is insufficient or the -database has accumulated significant fragmentation over time. +### 3. Delete the file (when no data is needed) + +If the database is empty or data can be reloaded: + +```bash +mysqladmin shutdown +rm /var/lib/mysql/duckdb.db +# restart MariaDB — a fresh empty file is created +``` + +## Diagnostics + +```sql +SELECT run_in_duckdb('SELECT * FROM pragma_database_size()'); +``` + +Returns `total_blocks`, `used_blocks`, `free_blocks`, `wal_size`. If +`free_blocks` is high but the file hasn't shrunk, it means `Truncate()` could +not find a contiguous free tail. + +Check actual disk usage vs logical size: +```bash +ls -lh /var/lib/mysql/duckdb.db # logical size +du -sh /var/lib/mysql/duckdb.db # actual disk usage (differs if hole-punched) +``` ## Configuration @@ -73,19 +139,11 @@ loose-duckdb-checkpoint-threshold=268435456 A smaller threshold means more frequent automatic checkpoints but less WAL accumulation. -## Behavior After DROP TABLE - -When `ha_duckdb::delete_table()` executes `DROP TABLE` in DuckDB: - -1. The table metadata and data blocks are marked as deleted. -2. If the WAL exceeds `checkpoint_threshold`, an automatic checkpoint occurs. -3. Otherwise, blocks remain in the WAL until the next checkpoint. -4. To immediately reclaim disk space, run `VACUUM` after the drop. - ## Summary -| Method | File shrinks | Cost | Availability | -|--------------------|--------------|---------|--------------| -| CHECKPOINT | No | Low | Always | -| VACUUM | Yes | Medium | DuckDB 1.1+ | -| Export + Reimport | Yes | High | Always | +| Method | File shrinks | Reuses space | Cost | Reliability | +|--------------------|--------------|--------------|--------|-------------| +| FORCE CHECKPOINT | Rarely | Yes | Low | Always works for reuse | +| VACUUM | No | No | Low | Only updates stats | +| Export + Reimport | Yes | N/A | High | Always shrinks | +| Delete file | Yes | N/A | None | Loses all data | diff --git a/storage/duckdb/mysql-test/duckdb/r/space_reclaim.result b/storage/duckdb/mysql-test/duckdb/r/space_reclaim.result new file mode 100644 index 00000000000..4fe76334406 --- /dev/null +++ b/storage/duckdb/mysql-test/duckdb/r/space_reclaim.result @@ -0,0 +1,75 @@ +# +# DuckDB space reclaim test +# +# Verifies that after DROP TABLE + FORCE CHECKPOINT, freed blocks +# are reused by subsequent inserts (file does not keep growing). +# +# Uses 4 SHA2 columns (incompressible data) to guarantee multiple +# 256 KB blocks are allocated (~2-3 MB total). +# +# +# 1) Create a table and insert incompressible data to allocate blocks +# +CREATE DATABASE IF NOT EXISTS db_reclaim; +USE db_reclaim; +CREATE TABLE t_big ( +id INT PRIMARY KEY, +c1 VARCHAR(200), +c2 VARCHAR(200), +c3 VARCHAR(200), +c4 VARCHAR(200) +) ENGINE = DuckDB; +SELECT COUNT(*) AS row_count FROM t_big; +row_count +1000 +# +# 2) Force checkpoint to flush WAL to main file +# +SELECT run_in_duckdb('FORCE CHECKPOINT'); +run_in_duckdb('FORCE CHECKPOINT') +Success +BOOLEAN +[ Rows: 0] + + +# +# 3) Record total_blocks after initial load +# +SELECT run_in_duckdb('SELECT total_blocks, used_blocks, free_blocks FROM pragma_database_size()') AS db_size_after_load; +db_size_after_load +total_blocks used_blocks free_blocks +BIGINT BIGINT BIGINT +[ Rows: 1] +3 3 0 + + +total_blocks after load: 0 +# +# 4) Drop table and force checkpoint — blocks become free +# +DROP TABLE t_big; +SELECT run_in_duckdb('FORCE CHECKPOINT'); +run_in_duckdb('FORCE CHECKPOINT') +Success +BOOLEAN +[ Rows: 0] + + +# +# 5) Verify used_blocks decreased at least 10x after drop +# +SELECT run_in_duckdb('SELECT total_blocks, used_blocks, free_blocks FROM pragma_database_size()') AS db_size_after_drop; +db_size_after_drop +total_blocks used_blocks free_blocks +BIGINT BIGINT BIGINT +[ Rows: 1] +1 1 0 + + +used_blocks after drop: 0 +PASS: used_blocks after drop is at least 10x smaller than after load +# +# 6) Cleanup +# +USE test; +DROP DATABASE db_reclaim; diff --git a/storage/duckdb/mysql-test/duckdb/t/space_reclaim-master.opt b/storage/duckdb/mysql-test/duckdb/t/space_reclaim-master.opt new file mode 100644 index 00000000000..999f8fa992d --- /dev/null +++ b/storage/duckdb/mysql-test/duckdb/t/space_reclaim-master.opt @@ -0,0 +1,2 @@ +--loose-duckdb-allow-run-in-duckdb=ON +--plugin-maturity=alpha diff --git a/storage/duckdb/mysql-test/duckdb/t/space_reclaim.test b/storage/duckdb/mysql-test/duckdb/t/space_reclaim.test new file mode 100644 index 00000000000..cb090afafa6 --- /dev/null +++ b/storage/duckdb/mysql-test/duckdb/t/space_reclaim.test @@ -0,0 +1,75 @@ +--source ../include/have_duckdb.inc +--source include/have_sequence.inc + +--echo # +--echo # DuckDB space reclaim test +--echo # +--echo # Verifies that after DROP TABLE + FORCE CHECKPOINT, freed blocks +--echo # are reused by subsequent inserts (file does not keep growing). +--echo # +--echo # Uses 4 SHA2 columns (incompressible data) to guarantee multiple +--echo # 256 KB blocks are allocated (~2-3 MB total). +--echo # + +--echo # +--echo # 1) Create a table and insert incompressible data to allocate blocks +--echo # +CREATE DATABASE IF NOT EXISTS db_reclaim; +USE db_reclaim; + +CREATE TABLE t_big ( + id INT PRIMARY KEY, + c1 VARCHAR(200), + c2 VARCHAR(200), + c3 VARCHAR(200), + c4 VARCHAR(200) +) ENGINE = DuckDB; + +--disable_query_log +INSERT INTO t_big +SELECT seq, + SHA2(CONCAT(seq, 'a'), 256), + SHA2(CONCAT(seq, 'b'), 256), + SHA2(CONCAT(seq, 'c'), 256), + SHA2(CONCAT(seq, 'd'), 256) +FROM test.seq_1_to_1000; +--enable_query_log + +SELECT COUNT(*) AS row_count FROM t_big; + +--echo # +--echo # 2) Force checkpoint to flush WAL to main file +--echo # +SELECT run_in_duckdb('FORCE CHECKPOINT'); + +--echo # +--echo # 3) Record total_blocks after initial load +--echo # +SELECT run_in_duckdb('SELECT total_blocks, used_blocks, free_blocks FROM pragma_database_size()') AS db_size_after_load; +--let $blocks_after_load = `SELECT CAST(TRIM(SUBSTRING_INDEX(SUBSTRING_INDEX(run_in_duckdb('SELECT total_blocks FROM pragma_database_size()'), '\n', 3), '\n', -1)) AS UNSIGNED)` +--echo total_blocks after load: $blocks_after_load + +--echo # +--echo # 4) Drop table and force checkpoint — blocks become free +--echo # +DROP TABLE t_big; +SELECT run_in_duckdb('FORCE CHECKPOINT'); + +--echo # +--echo # 5) Verify used_blocks decreased at least 10x after drop +--echo # +SELECT run_in_duckdb('SELECT total_blocks, used_blocks, free_blocks FROM pragma_database_size()') AS db_size_after_drop; +--let $blocks_after_drop = `SELECT CAST(TRIM(SUBSTRING_INDEX(SUBSTRING_INDEX(run_in_duckdb('SELECT used_blocks FROM pragma_database_size()'), '\n', 3), '\n', -1)) AS UNSIGNED)` +--echo used_blocks after drop: $blocks_after_drop + +if (`SELECT $blocks_after_drop > $blocks_after_load / 10`) +{ + --die Space reclaim failed: used_blocks after drop is not 10x smaller than after load +} +--echo PASS: used_blocks after drop is at least 10x smaller than after load + +--echo # +--echo # 6) Cleanup +--echo # +USE test; +DROP DATABASE db_reclaim;