mirror of
https://github.com/minio/docs.git
synced 2025-07-28 19:42:10 +03:00
DOCS-677: Update site replication recovery procedure (#694)
Closes #677 Updates the site replication recovery tutorial with slight tweaks to the multi-site expansion tutorial to account for the latest changes to MinIO/MC Co-authored-by: Poorna <poornas@users.noreply.github.com>
This commit is contained in:
Ravind Kumar
GitHub
@ -24,29 +24,6 @@ Site recovery depends on the replication option you use for the site.
|
||||
* - :mc:`mc mirror`
|
||||
- Data restoration of objects only from a healthy remote location with no versioning
|
||||
|
||||
Site Replication
|
||||
----------------
|
||||
|
||||
:ref:`Site replication <minio-site-replication-overview>` keeps two or more MinIO deployments in sync with IAM policies, buckets, bucket configurations, objects, and object metadata.
|
||||
If a peer site fails, such as due to a major disaster or long power outage, you can use the remaining healthy site(s) to restore the :ref:`replicable data <minio-site-replication-what-replicates>`.
|
||||
|
||||
The following procedure can restore data in scenarios where :ref:`site replication <minio-site-replication-overview>` was active prior to the site loss.
|
||||
|
||||
1. Remove the failed site from the MinIO site replication configuration using the :mc-cmd:`mc admin replicate remove` command.
|
||||
For example, the following command removes a failed site with :ref:`alias <alias>` ``siteB`` from a site replication configuration that includes healthy site with alias ``siteA``:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin replicate remove siteA siteB --force
|
||||
|
||||
2. Deploy a new MinIO site using the same ``root`` credentials as used on other deployments in the site replication configuration
|
||||
|
||||
You can use the original hardware from the failed site, if still available and functional, but you must first wipe any remaining data before creating the new site.
|
||||
Ensure you have fully remediated any issues that resulted in the original failure state prior to reusing the hardware.
|
||||
3. Configure the new site with the same Identity Provider (IDp) as the other site(s)
|
||||
4. :ref:`Expand the existing site replication <minio-expand-site-replication>` by adding the newly deployed site
|
||||
|
||||
Site replication healing automatically adds IAM settings, buckets, bucket configurations, and objects from the existing site(s) to the new site with no further action required.
|
||||
|
||||
You cannot configure site replication if any bucket replication rules remain in place on other healthy sites.
|
||||
@ -54,6 +31,73 @@ Bucket replication is mutually exclusive with site replication.
|
||||
|
||||
If you are switching from using bucket replication to using site replication, you must first remove all bucket replication rules from the healthy site prior to setting up site replication.
|
||||
|
||||
Restore an Unhealthy Peer to Site Replication
|
||||
---------------------------------------------
|
||||
|
||||
.. important::
|
||||
|
||||
The :minio-release:`RELEASE.2023-01-02T09-40-09Z` MinIO server release includes important fixes for removing a downed site in replication configurations containing three or more peer sites.
|
||||
|
||||
For deployments configured for site replication, plan to :ref:`test and upgrade <minio-upgrade>` all peer sites to the specified release.
|
||||
In the event of a site failure, you can update the remaining healthy sites to the specified version and use this procedure.
|
||||
|
||||
:ref:`Site replication <minio-site-replication-overview>` keeps two or more MinIO deployments in sync with IAM policies, buckets, bucket configurations, objects, and object metadata.
|
||||
If a peer site fails, such as due to a major disaster or long power outage, you can use the remaining healthy site(s) to restore the :ref:`replicable data <minio-site-replication-what-replicates>`.
|
||||
|
||||
The following procedure can restore data in scenarios where :ref:`site replication <minio-site-replication-overview>` was active prior to the site loss.
|
||||
This procedure assumes a *total loss* of one or more peer sites versus replication lag or delays due to latency or transient deployment downtime.
|
||||
|
||||
1. Remove the failed site from the MinIO site replication configuration using the :mc-cmd:`mc admin replicate remove` command with the ``--force`` option.
|
||||
|
||||
The following command force-removes an unhealthy peer site from the replication configuration:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin replicate remove HEALTHY_PEER UNHEALTHY_PEER --force
|
||||
|
||||
- Replace ``HEALTHY_PEER`` with the :ref:`alias <alias>` of any healthy peer in the replication configuration
|
||||
|
||||
- Replace ``UNHEALTHY_PEER`` with the alias of the unhealthy peer site
|
||||
|
||||
All healthy peers in the site replication configuration update to remove the unhealthy peer automatically.
|
||||
You can use the :mc-cmd:`mc admin replicate info` command to verify the new site replication configuration.
|
||||
|
||||
2. Deploy a new MinIO site following the :ref:`site replication requirements <minio-expand-site-replication>`.
|
||||
|
||||
Do not upload any data or otherwise configure the deployment beyond the stated requirements.
|
||||
Validate that the new MinIO deployment functions normally and has bidirectional connectivity to the other peer sites.
|
||||
|
||||
.. warning::
|
||||
|
||||
The :mc-cmd:`mc admin replicate remove --force` command only operates on the online or healthy nodes in the site replication configuration.
|
||||
The removed offline MinIO deployment retains its original replication configuration, such that if the deployment resumes normal operations it would continue replication operations to its configured peer sites.
|
||||
|
||||
If you plan to re-use the hardware for the site replication configuration, you **must** completely wipe the drives for the deployment before re-initializing MinIO and adding the site back to the replication configuration.
|
||||
|
||||
3. :ref:`Add the replacement peer site <minio-expand-site-replication>` to the replication configuration.
|
||||
|
||||
Use the :mc-cmd:`mc admin replicate add` command to update the replication configuration with the new site:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin replicate add HEALTHY_PEER NEW_PEER
|
||||
|
||||
- Replace ``HEALTHY_PEER`` with the :ref:`alias <alias>` of any healthy peer in the replication configuration
|
||||
|
||||
- Replace ``NEW_PEER`` with the alias of the new peer
|
||||
|
||||
All healthy peers in the site replication configuration update for the new peer automatically.
|
||||
You can use the :mc-cmd:`mc admin replicate info` command to verify the new site replication configuration.
|
||||
|
||||
4. Validate the replication status.
|
||||
|
||||
Use the following commands to track the replication status:
|
||||
|
||||
- :mc-cmd:`mc admin replicate status` - provides overall status and progress of replication
|
||||
- :mc-cmd:`mc replicate status` - provides bucket-level and global replication status
|
||||
|
||||
Active Bucket Replication Resynchronization
|
||||
-------------------------------------------
|
||||
|
||||
|
||||
@ -346,12 +346,10 @@ The new site must meet the following requirements:
|
||||
.. tab-item:: Command Line
|
||||
:sync: cli
|
||||
|
||||
#. Deploy three or more separate MinIO sites, using the same external IDP
|
||||
#. Deploy the new MinIO peer site(s) following the stated requirements.
|
||||
|
||||
Only one site can have any buckets or objects on it.
|
||||
The other sites must be empty.
|
||||
|
||||
#. Configure an alias for each site
|
||||
#. Configure an alias for the new site
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-mc-admin-replicate-load-balancing
|
||||
@ -359,34 +357,28 @@ The new site must meet the following requirements:
|
||||
|
||||
To check the existing aliases, use :mc:`mc alias list`.
|
||||
|
||||
For example, for three MinIO sites, you might create aliases ``minio1``, ``minio2``, and ``minio3``.
|
||||
|
||||
Use :mc:`mc alias set` to define the hostname or IP of the load balancer managing connections to the site.
|
||||
Use :mc:`mc alias set` to define the hostname or IP of the load balancer managing connections to the new site(s).
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mc alias set minio1 https://minio1.example.com:9000 adminuser adminpassword
|
||||
mc alias set minio2 https://minio2.example.com:9000 adminuser adminpassword
|
||||
mc alias set minio3 https://minio3.example.com:9000 adminuser adminpassword
|
||||
mc alias set minio4 https://minio4.example.com:9000 adminuser adminpassword
|
||||
|
||||
or define environment variables
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
export MC_HOST_minio1=https://adminuser:adminpassword@minio1.example.com
|
||||
export MC_HOST_minio2=https://adminuser:adminpassword@minio2.example.com
|
||||
export MC_HOST_minio3=https://adminuser:adminpassword@minio3.example.com
|
||||
export MC_HOST_minio4=https://adminuser:adminpassword@minio4.example.com
|
||||
|
||||
#. Add site replication configuration
|
||||
|
||||
List all existing replicated sites first, then list the new site(s) to add.
|
||||
In this example, ``minio1``, ``minio2``, and ``minio3`` are already configured for replication.
|
||||
The command adds ``minio4`` and ``minio5`` as new sites to add to the replication.
|
||||
``minio4`` and ``minio5`` must be empty.
|
||||
Use the :mc-cmd:`mc admin replicate add` command to expand the site replication configuration with the new peer site.
|
||||
Specify the alias of any existing healthy peer site as the first parameter and the alias of the new site as the second parameter.
|
||||
|
||||
For example, the following command adds the new peer site ``minio4`` to an existing site replication configuration on ``minio1``.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mc admin replicate add minio1 minio2 minio3 minio4 minio5
|
||||
mc admin replicate add minio1 minio4
|
||||
|
||||
#. Query the site replication configuration to verify
|
||||
|
||||
@ -494,11 +486,15 @@ You can re-add the site at a later date, but you must first completely wipe buck
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mc admin replicate remove <ALIAS> --all --force
|
||||
mc admin replicate remove ALIAS PEER_TO_REMOVE --force
|
||||
|
||||
The ``-all`` flag removes the site as a peer from all participating sites.
|
||||
- Replace ``ALIAS`` with the :ref:`alias <ref>` of any peer site in the replication configuration.
|
||||
|
||||
The ``--force`` flag is required to removes the site from the site replication configuration.
|
||||
- Replace ``PEER_TO_REMOVE`` with the alias of the peer site to remove.
|
||||
|
||||
All healthy peers in the site replication configuration update to remove the specified peer automatically.
|
||||
|
||||
MinIO requires the ``--force`` flag to remove the peer from the site replication configuration.
|
||||
|
||||
.. _minio-site-replication-status-tutorial:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user