1
0
mirror of https://github.com/facebook/rocksdb.git synced 2026-10-23 08:09:02 +03:00
Files
rocksdb/tools
Jay Huh 990827e916 db_bench: tolerate remote IO errors by skipping ops and continuing (#14971)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14971

Follow-up to https://github.com/facebook/rocksdb/pull/14942. That PR made db_bench read ops end the run with an INVALID result (`SetFatal` -> exit 1) on a tolerable remote IO error and left write ops on unconditional `SetFatal`. In practice that still failed the remote benchmark TW jobs (exit 1) on transient WS infra errors, and did not cover write ops or `DB::Open`.

Change the toleration to skip-and-continue so a transient remote IO error no longer fails the run:
- `HandleBenchmarkIOError` now just logs and returns on a tolerable error (was: `SetFatal`); the caller skips the failed op and keeps going. Non-tolerable, data-loss, and local-DB errors still `abort()` (core dump for debugging).
- Applied uniformly to all worker op error paths: `Get` / `MultiGet`, put / merge, delete / delete-range, `DoDelete`, `RandomWithVerify` put/delete-many, and the `BGScan` iterator (stops the scan and continues).
- `DB::Open`: on a tolerable remote IO error, exit cleanly (0) -- the benchmark cannot run, but it is not a db_bench / RocksDB failure.

Gated by `--tolerate_io_errors_for_remote_dbs` plus a remote `--env_uri` / `--fs_uri` backend, so local runs and non-IO / data-loss errors are unaffected. The result is best-effort (skipped ops are not counted); the goal is to keep remote benchmark jobs green through transient WS infrastructure errors instead of failing or crashing.

db_stress is intentionally NOT changed: it is a correctness (crash-recovery) test and must keep failing on real errors.

Differential Revision: D112749712

fbshipit-source-id: a8ccd46ef66e6c3c97a1baffd35d27830879d428
2026-07-21 08:15:35 -07:00
..
2024-10-14 03:01:20 -07:00
2022-10-13 09:00:37 -07:00
2024-10-14 03:01:20 -07:00
2017-08-03 15:56:46 -07:00
2026-07-01 19:04:40 -07:00
2017-08-03 15:56:46 -07:00
2023-01-27 13:14:19 -08:00
2023-01-27 13:14:19 -08:00
2024-10-14 03:01:20 -07:00