From 05e2452c36c21c682604478311cee4bdd2f2fd6e Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Fri, 20 Jan 2023 15:20:10 -0600 Subject: [PATCH] Updates for minio RELEASE.2023-01-18T04-36-38Z (#711) minio [RELEASE.2023-01-18T04-36-38Z](https://github.com/minio/minio/releases/tag/RELEASE.2023-01-18T04-36-38Z) affects the docs: - Decommissioning of multiple pools [PR #16416](https://github.com/minio/minio/pull/16416) - Automatic SUBNET license update [PR #16391](https://github.com/minio/minio/pull/16416) There is no doc issue to track this update. --- .../decommission-server-pool.rst | 234 +++++++++++++++--- source/operations/troubleshooting.rst | 12 + .../minio-mc/mc-license-register.rst | 19 +- .../reference/minio-mc/mc-license-update.rst | 8 +- 4 files changed, 235 insertions(+), 38 deletions(-) diff --git a/source/operations/install-deploy-manage/decommission-server-pool.rst b/source/operations/install-deploy-manage/decommission-server-pool.rst index 260134d5..86a40481 100644 --- a/source/operations/install-deploy-manage/decommission-server-pool.rst +++ b/source/operations/install-deploy-manage/decommission-server-pool.rst @@ -1,8 +1,8 @@ .. _minio-decommissioning: -========================== -Decommission a Server Pool -========================== +========================= +Decommission Server Pools +========================= .. default-domain:: minio @@ -10,43 +10,36 @@ Decommission a Server Pool :local: :depth: 1 -Starting with :minio-release:`RELEASE.2022-01-25T19-56-04Z`, MinIO supports -decommissioning and removing a :ref:`server pool ` -from a deployment. Decommissioning is designed for removing an older server pool -whose hardware is no longer sufficient or performant compared to the pools in -the deployment. MinIO automatically migrates data from the decommissioned pool -to the remaining pools in the deployment based on the ratio of free space -available in each pool. +MinIO supports decommissioning and removing :ref:`server pools ` from a deployment with two or more pools. +To decommission, there must be at least one remaining pool with sufficient available space to receive the objects from the decommissioned pools. -During the decommissioning process, MinIO routes read operations (e.g. ``GET``, -``LIST``, ``HEAD``) normally. MinIO routes write operations (e.g. ``PUT``, -versioned ``DELETE``) to the remaining "active" pools in the deployment. +Starting with ``RELEASE.2023-01-18T04-36-38Z``, MinIO supports queueing :ref:`multiple pools ` in a single decommission command. +Each listed pool immediately enters a read-only status, but draining occurs one pool at a time. + +Decommissioning is designed for removing an older server pool whose hardware is no longer sufficient or performant compared to the pools in the deployment. +MinIO automatically migrates data from the decommissioned pools to the remaining pools in the deployment based on the ratio of free space available in each pool. + +During the decommissioning process, MinIO routes read operations (e.g. ``GET``, ``LIST``, ``HEAD``) normally. +MinIO routes write operations (e.g. ``PUT``, versioned ``DELETE``) to the remaining "active" pools in the deployment. Versioned objects maintain their ordering throughout the migration process. -The procedure on this page decommissions and removes a server pool from -a :ref:`distributed ` MinIO deployment with -*at least* two server pools. +The procedures on this page decommission and remove one or more server pools from a :ref:`distributed ` MinIO deployment with *at least* two server pools. .. admonition:: Decommissioning is Permanent :class: important - Once MinIO begins decommissioning a pool, it marks that pool as *permanently* - inactive ("draining"). Cancelling or otherwise interrupting the - decommissioning procedure does **not** restore the pool to an active - state. + Once MinIO begins decommissioning a pool, it marks that pool as *permanently* inactive ("draining"). + Cancelling or otherwise interrupting the decommissioning procedure does **not** restore the pool to an active state. + Use extra caution when decommissioning multiple pools. - Decommissioning is a major administrative operation that requires care - in planning and execution, and is not a trivial or 'daily' task. + Decommissioning is a major administrative operation that requires care in planning and execution, and is not a trivial or 'daily' task. - `MinIO SUBNET `__ users can - `log in `__ and create a new issue related to - decommissioning. Coordination with MinIO Engineering via SUBNET can ensure - successful decommissioning, including performance testing and health - diagnostics. + `MinIO SUBNET `__ users can `log in `__ and create a new issue related to decommissioning. + Coordination with MinIO Engineering via SUBNET can ensure successful decommissioning, including performance testing and health diagnostics. + + Community users can seek support on the `MinIO Community Slack `__. + Community Support is best-effort only and has no SLAs around responsiveness. - Community users can seek support on the `MinIO Community Slack - `__. Community Support is best-effort only and has - no SLAs around responsiveness. .. _minio-decommissioning-prereqs: @@ -354,3 +347,184 @@ to restart the MinIO service: Once the deployment is online, use :mc:`mc admin info` to confirm the uptime of all remaining servers in the deployment. + +.. _minio-decommission-multiple-pools: + +Decommission Multiple Server Pools +---------------------------------- + +.. versionchanged:: RELEASE.2023-01-18T04-36-38Z + +You can start the decommission process for multiple server pools when issuing a decommission command. + +After entering the command: + +- MinIO immediately stops write access to all pools to be decommissioned. +- Decommissioning happens one pool at a time. +- Each pool completes the decommission draining process before MinIO begins draining the next pool. + +To decommission multiple server pools from one command, add the full description of each server pool to decommission as a comma-separated list. + +All other considerations about decommissioning apply when performing the process on multiple servers. + +- Decommissioning is permanent. +- Once you mark the pools as decommissioned, you **cannot** restore them. +- Confirm you select the intended pools. + +1) Review the MinIO Deployment Topology +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The :mc:`mc admin decommission` command returns a list of all pools in the MinIO deployment: + +.. code-block:: shell + :class: copyable + + mc admin decommission status myminio + +The command returns output similar to the following: + +.. code-block:: shell + + ┌─────┬────────────────────────────────────────────────────────────────┬──────────────────────────────────┬────────┐ + │ ID │ Pools │ Capacity │ Status │ + │ 1st │ https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio │ 10 TiB (used) / 10 TiB (total) │ Active │ + │ 2nd │ https://minio-{05...08}.example.com:9000/mnt/disk{1...4}/minio │ 95 TiB (used) / 100 TiB (total) │ Active │ + │ 3rd │ https://minio-{09...12}.example.com:9000/mnt/disk{1...4}/minio │ 40 TiB (used) / 500 TiB (total) │ Active │ + │ 4th │ https://minio-{13...16}.example.com:9000/mnt/disk{1...4}/minio │ 0 TiB (used) / 500 TiB (total) │ Active │ + └─────┴────────────────────────────────────────────────────────────────┴──────────────────────────────────┴────────┘ + +The example deployment above has three pools. +Each pool has four servers with four drives each. + +Identify the target pool for decommissioning and review the current capacity. +The remaining pools in the deployment *must* have sufficient total capacity to migrate all object stored in the decommissioned pool. + +In the example above, the deployment has 1110TiB total storage with 145TiB used. + +- The first pool (``minio-{01...04}``) is the first decommissioning target, as it was provisioned when the MinIO deployment was created and is completely full. +- The second pool (``minio-{05...08}``) is the second decommissioning target, as it was also provisioned when the MinIO deployment was created and is nearly full. +- The fourth pool (``minio-{13...16}``) is a newly added pool with new hardware from a completed server expansion. + +The third and fourth pools can absorb all objects stored on the first pool without significantly impacting total available storage. + +.. important:: + + Complete any server expansion to add new storage resources _before_ beginning a decommission process. + +2) Start the Decommissioning Process +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. admonition:: Decommissioning is Permanent + :class: warning + + Once MinIO begins decommissioning the pools, it marks those pools as *permanently* inactive ("draining"). + Cancelling or otherwise interrupting the decommissioning procedure does **not** restore the pools to an active state. + + Review and validate that you are decommissioning the correct pools *before* running the following command. + +Use the :mc-cmd:`mc admin decommission start` command to begin decommissioning the target pool. +Specify the :ref:`alias ` of the deployment and a comma-separated list of the full description of each pool to decommission, including all hosts, disks, and file paths. + +.. code-block:: shell + :class: copyable + + mc admin decommission start myminio/ https://minio-{01...04}.example.net:9000/mnt/disk{1...4}/minio,https://minio-{05...08}.example.net:9000/mnt/disk{1...4}/minio + +The example command begins decommissioning the two listed matching server pools on the ``myminio`` deployment. + +During the decommissioning process, MinIO continues routing read operations (``GET``, ``LIST``, ``HEAD``) operations to the pools for those objects not yet migrated. +MinIO routes all new write operations (``PUT``) to the remaining pools in the deployment not scheduled for decommissioning. + +Draining of decommissioned pools happens one pool at a time, completing the decommission of each pool in sequence. +Draining does _not_ happen concurrently for all decommissioning pools. + +Load balancers, reverse proxy, or other network control components which manage connections to the deployment do not need to modify their configurations at this time. + +3) Monitor the Decommissioning Process +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :mc-cmd:`mc admin decommission status` command to monitor the decommissioning process. + +.. code-block:: shell + :class: copyable + + mc admin decommission status myminio + +The command returns output similar to the following: + +.. code-block:: shell + + ┌─────┬────────────────────────────────────────────────────────────────┬──────────────────────────────────┬──────────┐ + │ ID │ Pools │ Capacity │ Status │ + │ 1st │ https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio │ 10 TiB (used) / 10 TiB (total) │ Draining │ + │ 2nd │ https://minio-{05...08}.example.com:9000/mnt/disk{1...4}/minio │ 95 TiB (used) / 100 TiB (total) │ Pending │ + │ 3rd │ https://minio-{09...12}.example.com:9000/mnt/disk{1...4}/minio │ 40 TiB (used) / 500 TiB (total) │ Active │ + │ 4th │ https://minio-{13...16}.example.com:9000/mnt/disk{1...4}/minio │ 0 TiB (used) / 500 TiB (total) │ Active │ + └─────┴────────────────────────────────────────────────────────────────┴──────────────────────────────────┴──────────┘ + +You can retrieve more detailed information by specifying the description of the server pool to the command: + +.. code-block:: shell + :class: copyable + + mc admin decommission status myminio https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio + +The command returns output similar to the following: + +.. code-block:: shell + + Decommissioning rate at 100MiB/sec [1TiB/10TiB] + Started: 30 minutes ago + +:mc-cmd:`mc admin decommission status` marks the :guilabel:`Status` as :guilabel:`Complete` once decommissioning is completed. +You can move on to the next step once MinIO completes decommissioning for all pools. + +If :guilabel:`Status` reads as failed, you can re-run the :mc-cmd:`mc admin decommission start` command to resume the process. +For persistent failures, use :mc:`mc admin console` or review the ``systemd`` logs (e.g. ``journalctl -u minio``) to identify more specific errors. + +4) Remove the Decommissioned Pools from the Deployment Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once decommissioning completes, you can safely remove the pools from the deployment configuration. +Modify the startup command for each remaining MinIO server in the deployment and remove the decommissioned pool. + +The ``.deb`` or ``.rpm`` packages install a `systemd `__ service file to ``/etc/systemd/system/minio.service``. +For binary installations, this procedure assumes the file was created manually as per the :ref:`deploy-minio-distributed` procedure. + +The ``minio.service`` file uses an environment file located at ``/etc/default/minio`` for sourcing configuration settings, including the startup. +Specifically, the ``MINIO_VOLUMES`` variable sets the startup command: + +.. code-block:: shell + :class: copyable + + cat /etc/default/minio | grep "MINIO_VOLUMES" + +The command returns output similar to the following: + +.. code-block:: shell + + MINIO_VOLUMES="https://minio-{1...4}.example.net:9000/mnt/disk{1...4}/minio https://minio-{5...8}.example.net:9000/mnt/disk{1...4}/minio https://minio-{9...12}.example.net:9000/mnt/disk{1...4}/minio" + +Edit the environment file and remove the decommissioned pools from the ``MINIO_VOLUMES`` value. + +5) Update Network Control Plane +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Update any load balancers, reverse proxies, or other network control planes to remove the decommissioned server pools from the connection configuration for the MinIO deployment. + +Specific instructions for configuring network control plane components is out of scope for this procedure. + +6) Restart the MinIO Deployment +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Issue the following commands on each node **simultaneously** in the deployment to restart the MinIO service: + +.. include:: /includes/linux/common-installation.rst + :start-after: start-install-minio-restart-service-desc + :end-before: end-install-minio-restart-service-desc + +.. include:: /includes/common-installation.rst + :start-after: start-nondisruptive-upgrade-desc + :end-before: end-nondisruptive-upgrade-desc + +Once the deployment is online, use :mc:`mc admin info` to confirm the uptime of all remaining servers in the deployment. diff --git a/source/operations/troubleshooting.rst b/source/operations/troubleshooting.rst index f273ca93..7017441c 100644 --- a/source/operations/troubleshooting.rst +++ b/source/operations/troubleshooting.rst @@ -79,6 +79,18 @@ Registering Your MinIO Deployment with SUBNET For clusters without direct Internet access, refer to the instructions in the :ref:`airgap example ` of the :mc:`mc license register` documentation. +.. _minio-subnet-license-file-download: + +Download License File +~~~~~~~~~~~~~~~~~~~~~ + +Download the license file from SUBNET on a machine with access to the Internet. + +#. Log in to |SUBNET| +#. Go to the :guilabel:`Deployments` tab +#. Select the deployment from the list +#. Select the :guilabel:`License` number near the top of the page to display the :guilabel:`Cluster License Key` +#. Select :guilabel:`Download` SUBNET Issues ------------- diff --git a/source/reference/minio-mc/mc-license-register.rst b/source/reference/minio-mc/mc-license-register.rst index dcefc2c5..118576df 100644 --- a/source/reference/minio-mc/mc-license-register.rst +++ b/source/reference/minio-mc/mc-license-register.rst @@ -11,11 +11,6 @@ .. mc:: mc support register .. mc:: mc license register -.. note:: - - .. versionchanged:: RELEASE.2022-07-15T09-20-55Z - - ``mc license register`` replaces the ``mc support register`` command. Description ----------- @@ -72,7 +67,6 @@ Register a MinIO deployment at alias ``minio3`` on SUBNET that does not have dir #. Copy the API token that displays #. Back in the terminal, paste or enter the API token to complete the registration process - Syntax ------ @@ -110,10 +104,21 @@ Parameters If the deployment is airgapped, but the local device where you are using the :ref:`minio client ` has network access, you do not need to use the ``--airgap`` flag. - Global Flags ~~~~~~~~~~~~ .. include:: /includes/common-minio-mc.rst :start-after: start-minio-mc-globals :end-before: end-minio-mc-globals + +Behavior +-------- + +Automatic License Updates +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. versionadded:: RELEASE.2023-01-18T04-36-38Z + +Once registered for |SUBNET|, MinIO automatically checks for and updates the license every month. + +In airgapped or other environments where the server does not have direct access to the internet, use :mc-cmd:`mc license update` with the path to the file to update the registration. \ No newline at end of file diff --git a/source/reference/minio-mc/mc-license-update.rst b/source/reference/minio-mc/mc-license-update.rst index 23b8501f..a226935e 100644 --- a/source/reference/minio-mc/mc-license-update.rst +++ b/source/reference/minio-mc/mc-license-update.rst @@ -15,6 +15,12 @@ Description Use the :mc-cmd:`mc license update` command to replace a license key for a deployment. +.. versionchanged:: RELEASE.2023-01-18T04-36-38Z + + For deployments registered for |SUBNET|, MinIO automatically checks for and updates the license every month. + +If not passed with the command, MinIO checks the license file on SUBNET and automatically updates it. + Examples -------- @@ -51,7 +57,7 @@ Parameters The path (relative to the current working directory) and file name of the key to use to update the deployment's license. - If not passed with the command, MinIO checks the license file on SUBNET and automatically updates it. + See the :ref:`troubleshooting page ` for instructions on downloading the license file. .. mc-cmd:: --airgap :optional: