minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-07-28 19:42:10 +03:00
Code Activity

DOCS-677: Update site replication recovery procedure (#694)

Browse Source 
Closes #677 

Updates the site replication recovery tutorial with slight tweaks to the multi-site expansion tutorial to account for the latest changes to MinIO/MC

Co-authored-by: Poorna <poornas@users.noreply.github.com>
This commit is contained in:
Ravind Kumar
2023-01-06 18:03:59 -05:00
committed by GitHub
parent c2dacd93f2
commit dbbfce1aeb
2 changed files with 84 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files

90
source/operations/data-recovery/recover-after-site-failure.rst
View File

@ -24,29 +24,6 @@ Site recovery depends on the replication option you use for the site.
* - :mc:`mc mirror` * - :mc:`mc mirror`
- Data restoration of objects only from a healthy remote location with no versioning - Data restoration of objects only from a healthy remote location with no versioning
Site Replication
----------------
:ref:`Site replication <minio-site-replication-overview>` keeps two or more MinIO deployments in sync with IAM policies, buckets, bucket configurations, objects, and object metadata.
If a peer site fails, such as due to a major disaster or long power outage, you can use the remaining healthy site(s) to restore the :ref:`replicable data <minio-site-replication-what-replicates>`.
The following procedure can restore data in scenarios where :ref:`site replication <minio-site-replication-overview>` was active prior to the site loss.
1. Remove the failed site from the MinIO site replication configuration using the :mc-cmd:`mc admin replicate remove` command.
For example, the following command removes a failed site with :ref:`alias <alias>` ``siteB`` from a site replication configuration that includes healthy site with alias ``siteA``:
.. code-block:: shell
:class: copyable
mc admin replicate remove siteA siteB --force
2. Deploy a new MinIO site using the same ``root`` credentials as used on other deployments in the site replication configuration
You can use the original hardware from the failed site, if still available and functional, but you must first wipe any remaining data before creating the new site.
Ensure you have fully remediated any issues that resulted in the original failure state prior to reusing the hardware.
3. Configure the new site with the same Identity Provider (IDp) as the other site(s)
4. :ref:`Expand the existing site replication <minio-expand-site-replication>` by adding the newly deployed site
Site replication healing automatically adds IAM settings, buckets, bucket configurations, and objects from the existing site(s) to the new site with no further action required. Site replication healing automatically adds IAM settings, buckets, bucket configurations, and objects from the existing site(s) to the new site with no further action required.
You cannot configure site replication if any bucket replication rules remain in place on other healthy sites. You cannot configure site replication if any bucket replication rules remain in place on other healthy sites.
@ -54,6 +31,73 @@ Bucket replication is mutually exclusive with site replication.
If you are switching from using bucket replication to using site replication, you must first remove all bucket replication rules from the healthy site prior to setting up site replication. If you are switching from using bucket replication to using site replication, you must first remove all bucket replication rules from the healthy site prior to setting up site replication.
Restore an Unhealthy Peer to Site Replication
---------------------------------------------
.. important::
The :minio-release:`RELEASE.2023-01-02T09-40-09Z` MinIO server release includes important fixes for removing a downed site in replication configurations containing three or more peer sites.
For deployments configured for site replication, plan to :ref:`test and upgrade <minio-upgrade>` all peer sites to the specified release.
In the event of a site failure, you can update the remaining healthy sites to the specified version and use this procedure.
:ref:`Site replication <minio-site-replication-overview>` keeps two or more MinIO deployments in sync with IAM policies, buckets, bucket configurations, objects, and object metadata.
If a peer site fails, such as due to a major disaster or long power outage, you can use the remaining healthy site(s) to restore the :ref:`replicable data <minio-site-replication-what-replicates>`.
The following procedure can restore data in scenarios where :ref:`site replication <minio-site-replication-overview>` was active prior to the site loss.
This procedure assumes a *total loss* of one or more peer sites versus replication lag or delays due to latency or transient deployment downtime.
1. Remove the failed site from the MinIO site replication configuration using the :mc-cmd:`mc admin replicate remove` command with the ``--force`` option.
The following command force-removes an unhealthy peer site from the replication configuration:
.. code-block:: shell
:class: copyable
mc admin replicate remove HEALTHY_PEER UNHEALTHY_PEER --force
- Replace ``HEALTHY_PEER`` with the :ref:`alias <alias>` of any healthy peer in the replication configuration
- Replace ``UNHEALTHY_PEER`` with the alias of the unhealthy peer site
All healthy peers in the site replication configuration update to remove the unhealthy peer automatically.
You can use the :mc-cmd:`mc admin replicate info` command to verify the new site replication configuration.
2. Deploy a new MinIO site following the :ref:`site replication requirements <minio-expand-site-replication>`.
Do not upload any data or otherwise configure the deployment beyond the stated requirements.
Validate that the new MinIO deployment functions normally and has bidirectional connectivity to the other peer sites.
.. warning::
The :mc-cmd:`mc admin replicate remove --force` command only operates on the online or healthy nodes in the site replication configuration.
The removed offline MinIO deployment retains its original replication configuration, such that if the deployment resumes normal operations it would continue replication operations to its configured peer sites.
If you plan to re-use the hardware for the site replication configuration, you **must** completely wipe the drives for the deployment before re-initializing MinIO and adding the site back to the replication configuration.
3. :ref:`Add the replacement peer site <minio-expand-site-replication>` to the replication configuration.
Use the :mc-cmd:`mc admin replicate add` command to update the replication configuration with the new site:
.. code-block:: shell
:class: copyable
mc admin replicate add HEALTHY_PEER NEW_PEER
- Replace ``HEALTHY_PEER`` with the :ref:`alias <alias>` of any healthy peer in the replication configuration
- Replace ``NEW_PEER`` with the alias of the new peer
All healthy peers in the site replication configuration update for the new peer automatically.
You can use the :mc-cmd:`mc admin replicate info` command to verify the new site replication configuration.
4. Validate the replication status.
Use the following commands to track the replication status:
- :mc-cmd:`mc admin replicate status` - provides overall status and progress of replication
- :mc-cmd:`mc replicate status` - provides bucket-level and global replication status
Active Bucket Replication Resynchronization Active Bucket Replication Resynchronization
------------------------------------------- -------------------------------------------

38
source/operations/install-deploy-manage/multi-site-replication.rst
View File

@ -346,12 +346,10 @@ The new site must meet the following requirements:
.. tab-item:: Command Line .. tab-item:: Command Line
:sync: cli :sync: cli
#. Deploy three or more separate MinIO sites, using the same external IDP #. Deploy the new MinIO peer site(s) following the stated requirements.
Only one site can have any buckets or objects on it.
The other sites must be empty.
#. Configure an alias for each site #. Configure an alias for the new site
.. include:: /includes/common-replication.rst .. include:: /includes/common-replication.rst
:start-after: start-mc-admin-replicate-load-balancing :start-after: start-mc-admin-replicate-load-balancing
@ -359,34 +357,28 @@ The new site must meet the following requirements:
To check the existing aliases, use :mc:`mc alias list`. To check the existing aliases, use :mc:`mc alias list`.
For example, for three MinIO sites, you might create aliases ``minio1``, ``minio2``, and ``minio3``. Use :mc:`mc alias set` to define the hostname or IP of the load balancer managing connections to the new site(s).
Use :mc:`mc alias set` to define the hostname or IP of the load balancer managing connections to the site.
.. code-block:: shell .. code-block:: shell
mc alias set minio1 https://minio1.example.com:9000 adminuser adminpassword mc alias set minio4 https://minio4.example.com:9000 adminuser adminpassword
mc alias set minio2 https://minio2.example.com:9000 adminuser adminpassword
mc alias set minio3 https://minio3.example.com:9000 adminuser adminpassword
or define environment variables or define environment variables
.. code-block:: shell .. code-block:: shell
export MC_HOST_minio1=https://adminuser:adminpassword@minio1.example.com export MC_HOST_minio4=https://adminuser:adminpassword@minio4.example.com
export MC_HOST_minio2=https://adminuser:adminpassword@minio2.example.com
export MC_HOST_minio3=https://adminuser:adminpassword@minio3.example.com
#. Add site replication configuration #. Add site replication configuration
List all existing replicated sites first, then list the new site(s) to add. Use the :mc-cmd:`mc admin replicate add` command to expand the site replication configuration with the new peer site.
In this example, ``minio1``, ``minio2``, and ``minio3`` are already configured for replication. Specify the alias of any existing healthy peer site as the first parameter and the alias of the new site as the second parameter.
The command adds ``minio4`` and ``minio5`` as new sites to add to the replication.
``minio4`` and ``minio5`` must be empty. For example, the following command adds the new peer site ``minio4`` to an existing site replication configuration on ``minio1``.
.. code-block:: shell .. code-block:: shell
mc admin replicate add minio1 minio2 minio3 minio4 minio5 mc admin replicate add minio1 minio4
#. Query the site replication configuration to verify #. Query the site replication configuration to verify
@ -494,11 +486,15 @@ You can re-add the site at a later date, but you must first completely wipe buck
.. code-block:: shell .. code-block:: shell
mc admin replicate remove <ALIAS> --all --force mc admin replicate remove ALIAS PEER_TO_REMOVE --force
The ``-all`` flag removes the site as a peer from all participating sites. - Replace ``ALIAS`` with the :ref:`alias <ref>` of any peer site in the replication configuration.
The ``--force`` flag is required to removes the site from the site replication configuration. - Replace ``PEER_TO_REMOVE`` with the alias of the peer site to remove.
All healthy peers in the site replication configuration update to remove the specified peer automatically.
MinIO requires the ``--force`` flag to remove the peer from the site replication configuration.
.. _minio-site-replication-status-tutorial: .. _minio-site-replication-status-tutorial: