docs/source/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.rst

Deploy MinIO: Multi-Node Multi-Drive

minio

Table of Contents

The procedures on this page cover deploying MinIO in a Multi-Node Multi-Drive (MNMD) or "Distributed" configuration. deployments provide enterprise-grade performance, availability, and scalability and are the recommended topology for all production workloads.

deployments support erasure coding <minio-ec-parity> configurations which tolerate the loss of up to half the nodes or drives in the deployment while continuing to serve read operations. Use the MinIO Erasure Code Calculator when planning and designing your MinIO deployment to explore the effect of erasure code settings on your intended topology.

Prerequisites

Networking and Firewalls

Each node should have full bidirectional network access to every other node in the deployment. For containerized or orchestrated infrastructures, this may require specific configuration of networking and routing components such as ingress or load balancers. Certain operating systems may also require setting firewall rules. For example, the following command explicitly opens the default MinIO server API port 9000 for servers running firewalld :

firewall-cmd --permanent --zone=public --add-port=9000/tcp
firewall-cmd --reload

All MinIO servers in the deployment must use the same listen port.

If you set a static MinIO Console <minio-console> port (e.g. :9001) you must also grant access to that port to ensure connectivity from external clients.

MinIO strongly recomends using a load balancer to manage connectivity to the cluster. The Load Balancer should use a "Least Connections" algorithm for routing requests to the MinIO deployment, since any MinIO node in the deployment can receive, route, or process client requests.

The following load balancers are known to work well with MinIO:

• NGINX
• HAProxy

Configuring firewalls or load balancers to support MinIO is out of scope for this procedure. The integrations-nginx-proxy reference provides a baseline configuration for using NGINX as a reverse proxy with basic load balancing configured.

Sequential Hostnames

MinIO requires using expansion notation {x...y} to denote a sequential series of MinIO hosts when creating a server pool. MinIO supports using either a sequential series of hostnames or IP addresses to represent each minio server process in the deployment.

This procedure assumes use of sequential hostnames due to the lower overhead of management, especially in larger distributed clusters.

Create the necessary DNS hostname mappings prior to starting this procedure. For example, the following hostnames would support a 4-node distributed deployment:

• minio-01.example.com
• minio-02.example.com
• minio-03.example.com
• minio-04.example.com

You can specify the entire range of hostnames using the expansion notation minio-0{1...4}.example.com.

Non-Sequential Hostnames or IP Addresses

MinIO does not support non-sequential hostnames or IP addresses for distributed deployments. You can instead use /etc/hosts on each node to set a simple DNS scheme that supports expansion notation. For example:

# /etc/hosts

198.0.2.10    minio-01.example.net
198.51.100.3  minio-02.example.net
198.0.2.43    minio-03.example.net
198.51.100.12 minio-04.example.net

The above hosts configuration supports expansion notation of minio-0{1...4}.example.net, mapping the sequential hostnames to the desired IP addresses.

Storage Requirements

Memory Requirements

RELEASE.2024-01-28T22-35-53Z

MinIO pre-allocates 2GiB of system memory at startup.

MinIO recommends a minimum of 32GiB of memory per host. See minio-hardware-checklist-memory for more guidance on memory allocation in MinIO.

Time Synchronization

Multi-node systems must maintain synchronized time and date to maintain stable internode operations and interactions. Make sure all nodes sync to the same time server regularly. Operating systems vary for methods used to synchronize time and date, such as with ntp, timedatectl, or timesyncd.

Check the documentation for your operating system for how to set up and maintain accurate and identical system clock times across nodes.

Considerations

Erasure Coding Parity

MinIO erasure coding <minio-erasure-coding> is a data redundancy and availability feature that allows MinIO deployments to automatically reconstruct objects on-the-fly despite the loss of multiple drives or nodes in the cluster.

MinIO defaults to EC:4, or 4 parity blocks per erasure set <minio-ec-erasure-set>. You can set a custom parity level by setting the appropriate MinIO Storage Class environment variable <minio-server-envvar-storage-class>. Consider using the MinIO Erasure Code Calculator for guidance in selecting the appropriate erasure code parity level for your cluster.

Important

While you can change erasure parity settings at any time, objects written with a given parity do not update to the new parity settings. MinIO only applies the changed parity to newly written objects. Existing objects retain the parity value in place at the time of their creation.

Capacity-Based Planning

MinIO recommends planning storage capacity sufficient to store at least 2 years of data before reaching 70% usage. Performing server pool expansion <expand-minio-distributed> more frequently or on a "just-in-time" basis generally indicates an architecture or planning issue.

For example, consider an application suite expected to produce at least 100 TiB of data per year and a 3 year target before expansion. By ensuring the deployment has ~500TiB of usable storage up front, the cluster can safely meet the 70% threshold with additional buffer for growth in data storage output per year.

Since MinIO erasure coding <minio-erasure-coding> requires some storage for parity, the total raw storage must exceed the planned usable capacity. Consider using the MinIO Erasure Code Calculator for guidance in planning capacity around specific erasure code settings.

Recommended Operating Systems

linux

This tutorial assumes all hosts running MinIO use a recommended Linux operating system <minio-installation-platform-support> such as RHEL8+ or Ubuntu 18.04+.

macos

This tutorial assumes all hosts running MinIO use a non-EOL macOS version (10.14+).

Windows

This tutorial assumes all hosts running MinIO use a non-EOL Windows distribution.

Support for running distributed MinIO deployments on Windows is experimental.

Pre-Existing Data

When starting a new MinIO server in a distributed environment, the storage devices must not have existing data.

Once you start the MinIO server, all interactions with the data must be done through the S3 API. Use the MinIO Client <minio-client>, the MinIO Console <minio-console>, or one of the MinIO Software Development Kits <minio-drivers> to work with the buckets and objects.

Warning

Modifying files on the backend drives can result in data corruption or data loss.

Deploy Distributed MinIO

The following procedure creates a new distributed MinIO deployment consisting of a single Server Pool <minio-intro-server-pool>.

All commands provided below use example values. Replace these values with those appropriate for your deployment.

Review the deploy-minio-distributed-prereqs before starting this procedure.

1) Install the MinIO Binary on Each Node

linux

macos

2) Create the systemd Service File

3) Create the Service Environment File

Create an environment file at /etc/default/minio. The MinIO service uses this file as the source of all environment variables <minio-server-environment-variables> used by MinIO and the minio.service file.

The following examples assumes that:

• The deployment has a single server pool consisting of four MinIO server hosts with sequential hostnames.

  minio1.example.com   minio3.example.com
  minio2.example.com   minio4.example.com

• All hosts have four locally-attached drives with sequential mount-points:

  /mnt/disk1/minio   /mnt/disk3/minio
  /mnt/disk2/minio   /mnt/disk4/minio

• The deployment has a load balancer running at https://minio.example.net that manages connections across all four MinIO hosts.

Modify the example to reflect your deployment topology:

# Set the hosts and volumes MinIO uses at startup
# The command uses MinIO expansion notation {x...y} to denote a
# sequential series. 
# 
# The following example covers four MinIO hosts
# with 4 drives each at the specified hostname and drive locations.
# The command includes the port that each MinIO server listens on
# (default 9000)

MINIO_VOLUMES="https://minio{1...4}.example.net:9000/mnt/disk{1...4}/minio"

# Set all MinIO server options
#
# The following explicitly sets the MinIO Console listen address to
# port 9001 on all network interfaces. The default behavior is dynamic
# port selection.

MINIO_OPTS="--console-address :9001"

# Set the root username. This user has unrestricted permissions to
# perform S3 and administrative API operations on any resource in the
# deployment.
#
# Defer to your organizations requirements for superadmin user name.

MINIO_ROOT_USER=minioadmin

# Set the root password
#
# Use a long, random, unique string that meets your organizations
# requirements for passwords.

MINIO_ROOT_PASSWORD=minio-secret-key-CHANGE-ME

You may specify other environment variables <minio-server-environment-variables> or server commandline options as required by your deployment. All MinIO nodes in the deployment should include the same environment variables with the same values for each variable.

4) Add TLS/SSL Certificates

5) Run the MinIO Server Process

Issue the following commands on each node in the deployment to start the MinIO service:

6) Open the MinIO Console

7) Next Steps

• Create an alias <minio-mc-alias> for accessing the deployment using mc.
• Create users and policies to control access to the deployment <minio-authentication-and-identity-management>.