From b94cd46a831c767b6eb92544429e863ad2db3258 Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Tue, 3 Jan 2023 12:17:04 -0600 Subject: [PATCH] Adds a recommendation to limit prefix contents to 10k objects. (#674) Per @klauspost we need to add info to the docs about MinIO's recommendation to limit prefixes to no more than 10k objects. This updates the glossary entry, object management doc, and admin concepts to add that recommendation. Closes #666 --- source/administration/concepts.rst | 10 ++++++++++ source/administration/object-management.rst | 14 ++++++++++++-- source/glossary.rst | 8 ++++++++ 3 files changed, 30 insertions(+), 2 deletions(-) diff --git a/source/administration/concepts.rst b/source/administration/concepts.rst index 8e14e0fe..4fdded2a 100644 --- a/source/administration/concepts.rst +++ b/source/administration/concepts.rst @@ -103,6 +103,16 @@ MinIO automatically generates two folders in the ``articles`` bucket based on th 2022-01-02-MinIO-Advanced-Deployment-comments.json 2022-01-04-MinIO-Interview.md +MinIO itself does not limit the number of objects that any specific prefix can contain. +However, hardware and network conditions may show performance impacts with large prefixes. + +- Deployments with modest or budget-focused hardware should architect their workloads to target 10,000 objects per prefix as a baseline. + Increase this target based on benchmarking and monitoring of real world workloads up to what the hardware can meaningfully handle. +- Deployments with high-performance or enterprise-grade :ref:`hardware ` can typically handle prefixes with millions of objects or more. + +|SUBNET| Enterprise accounts can utilize yearly architecture reviews as part of the deployment and maintenance strategy to ensure long-term performance and success of your MinIO-dependent projects. + +For a deeper discussion on the benefits of limiting prefix contents, see the article on :s3-docs:`optimizing S3 performance `. How can I backup and restore objects on MinIO? ---------------------------------------------- diff --git a/source/administration/object-management.rst b/source/administration/object-management.rst index c6f83d7e..e0b050b8 100644 --- a/source/administration/object-management.rst +++ b/source/administration/object-management.rst @@ -45,8 +45,18 @@ following: 2020-01-02-MinIO-Advanced-Deployment-comments.json 2020-01-04-MinIO-Interview.md -MinIO supports multiple levels of nested directories and objects to support -even the most dynamic object storage workloads. +MinIO supports multiple levels of nested directories and objects using a :term:`prefix` structure to support even the most dynamic object storage workloads. + +MinIO itself does not limit the number of objects that any specific prefix can contain. +However, hardware and network conditions may show performance impacts with large prefixes. + +- Deployments with modest or budget-focused hardware should architect their workloads to target 10,000 objects per prefix as a baseline. + Increase this target based on benchmarking and monitoring of real world workloads up to what the hardware can meaningfully handle. +- Deployments with high-performance or enterprise-grade :ref:`hardware ` can typically handle prefixes with millions of objects or more. + +|SUBNET| Enterprise accounts can utilize yearly architecture reviews as part of the deployment and maintenance strategy to ensure long-term performance and success of your MinIO-dependent projects. + +For a deeper discussion on the benefits of limiting prefix contents, see the article on :s3-docs:`optimizing S3 performance `. Object Versioning ----------------- diff --git a/source/glossary.rst b/source/glossary.rst index 9c3732bb..4bf6d11d 100644 --- a/source/glossary.rst +++ b/source/glossary.rst @@ -193,6 +193,14 @@ Glossary Use a delimiter character, typically a `/` to add layers to the hierarchy. While prefixed objects may resemble a directory structure in some file systems, prefixes are not directories. + MinIO itself does not limit the number of objects that any specific prefix can contain. + However, hardware and network conditions may show performance impacts with large prefixes. + + - Deployments with modest or budget-focused hardware should architect their workloads to target 10,000 objects per prefix as a baseline. + Increase this target based on benchmarking and monitoring of real world workloads up to what the hardware can meaningfully handle. + - Deployments with high-performance or enterprise-grade :ref:`hardware ` can typically handle prefixes with millions of objects or more. + +|SUBNET| Enterprise accounts can utilize yearly architecture reviews as part of the deployment and maintenance strategy to ensure long-term performance and success of your MinIO-dependent projects. RAID Initialism for "Redundant Array of Independent Disks". The technology merges multiple separate physical disks into a single storage unit or array.