mirror of
https://github.com/minio/docs.git
synced 2025-07-31 18:04:52 +03:00
Erasure Coding First Pass
Eco fixes more fixes
This commit is contained in:
ravindk89
Ravind Kumar
@ -1,18 +0,0 @@
|
||||
.. _minio-bitrot-protection:
|
||||
|
||||
=================
|
||||
Bitrot Protection
|
||||
=================
|
||||
|
||||
Silent data corruption or bitrot is a serious problem faced by disk drives
|
||||
resulting in data getting corrupted without the user’s knowledge. The reasons
|
||||
are manifold (ageing drives, current spikes, bugs in disk firmware, phantom
|
||||
writes, misdirected reads/writes, driver errors, accidental overwrites) but the
|
||||
result is the same - compromised data.
|
||||
|
||||
MinIO’s optimized implementation of the HighwayHash algorithm ensures that it
|
||||
will never read corrupted data - it captures and heals corrupted objects on the
|
||||
fly. Integrity is ensured from end to end by computing a hash on READ and
|
||||
verifying it on WRITE from the application, across the network and to the
|
||||
memory/drive. The implementation is designed for speed and can achieve hashing
|
||||
speeds over 10 GB/sec on a single core on Intel CPUs.
|
||||
@ -1,67 +0,0 @@
|
||||
=====================
|
||||
Deployment Topologies
|
||||
=====================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
MinIO supports three deployment topologies:
|
||||
|
||||
.. list-table::
|
||||
:widths: 30 70
|
||||
:header-rows: 1
|
||||
|
||||
* - Deployment Type
|
||||
- Description
|
||||
|
||||
* - :ref:`Standalone <minio-deployment-standalone>`
|
||||
- A single MinIO server.
|
||||
|
||||
Standalone deployments are ideal for local development and evaluation.
|
||||
|
||||
* - :ref:`Distributed <minio-deployment-distributed>`
|
||||
- Multiple MinIO servers allow for horizontal scaling of storage while
|
||||
allowing applications to treat the deployment as a single MinIO
|
||||
instance.
|
||||
|
||||
Distributed deployments are ideal for production environments.
|
||||
|
||||
* - :ref:`Active-Active <minio-deployment-active-active>`
|
||||
- Multiple distributed deployments with intra-deployment
|
||||
replication to synchronize :ref:`objects <objects>` across
|
||||
deployments.
|
||||
|
||||
Active-Active Distributed deployments are ideal for production
|
||||
environments with globally distributed applications, where applications
|
||||
prefer routing to the geographically-nearest MinIO instance.
|
||||
|
||||
.. _minio-deployment-standalone:
|
||||
|
||||
Standalone Deployment
|
||||
---------------------
|
||||
|
||||
TBD:
|
||||
- Add a diagram of a standalone deployment
|
||||
- List the drawbacks (if any)
|
||||
- Link to deployment tutorials (kubernetes, bare-metal)
|
||||
|
||||
.. _minio-deployment-distributed:
|
||||
.. _minio-zones:
|
||||
|
||||
Distributed Deployment
|
||||
----------------------
|
||||
|
||||
TBD:
|
||||
- Add a diagram of a distributed deployment
|
||||
- List the drawbacks (if any)
|
||||
- Link to deployment tutorials (kubernetes, bare-metal)
|
||||
- Discuss horizontal expansion / zones
|
||||
|
||||
.. _minio-deployment-active-active:
|
||||
|
||||
Active-Active
|
||||
-------------
|
||||
|
||||
TBD:
|
||||
- Add a diagram of a distributed deployment
|
||||
- List the drawbacks (if any)
|
||||
- Link to deployment tutorials (kubernetes, bare-metal)
|
||||
@ -1,19 +0,0 @@
|
||||
.. _minio-erasure-coding:
|
||||
|
||||
==============
|
||||
Erasure Coding
|
||||
==============
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
MinIO protects data with per-object, inline erasure coding, which is written in
|
||||
assembly code to deliver the highest performance possible. MinIO uses
|
||||
Reed-Solomon code to stripe objects into `n/2` data and ``n/2`` parity blocks -
|
||||
although these can be configured to any desired redundancy level.
|
||||
|
||||
This means that in a 12 drive setup, an object is sharded across as 6 data and 6
|
||||
parity blocks. Even if you lose as many as 5 ((n/2)–1) drives, be it parity or
|
||||
data, you can still reconstruct the data reliably from the remaining drives.
|
||||
MinIO's implementation ensures that objects can be read or new objects are
|
||||
written even if multiple devices are lost or unavailable. Finally, MinIO's
|
||||
erasure code is at the object level and can heal one object at a time.
|
||||
@ -46,6 +46,78 @@ following:
|
||||
2020-01-02-blog-comments.json
|
||||
2020-01-02-blog-comments.json
|
||||
|
||||
Deployment Architecture
|
||||
-----------------------
|
||||
|
||||
The following diagram describes the individual components in a MinIO
|
||||
deployment:
|
||||
|
||||
<DIAGRAM ErasureSet -> ServerSet -> Cluster >
|
||||
|
||||
:ref:`Erasure Set <minio-ec-erasure-set>`
|
||||
A set of disks that supports MinIO :ref:`Erasure Coding
|
||||
<minio-erasure-coding>`. Erasure Coding provides high availability,
|
||||
reliability, and redundancy of data stored on a MinIO deployment.
|
||||
|
||||
MinIO divides objects into chunks and evenly distributes them among each
|
||||
drive in the Erasure Set. MinIO can continue seamlessly serving read and
|
||||
write requests despite the loss of any single drive. At the highest
|
||||
redundancy levels, MinIO can serve read requests with minimal performance
|
||||
impact despite the loss of up to half (``N/2``) of the total drives in the
|
||||
deployment.
|
||||
|
||||
.. _minio-intro-server-set:
|
||||
|
||||
:ref:`Server Set <minio-intro-server-set>`
|
||||
A set of MinIO :mc-cmd:`minio server` nodes which pool their drives and
|
||||
resources for supporting object storage/retrieval requests. The
|
||||
:mc-cmd:`~minio server HOSTNAME` argument passed to the
|
||||
:mc-cmd:`minio server` command represents a Server Set:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
minio server https://minio{1...4}.example.net/mnt/disk{1...4}
|
||||
|
||||
| Server Set |
|
||||
|
||||
The above example describes a single Server Set with
|
||||
4 :mc:`minio server` nodes and 4 drives each for a total of 16 drives.
|
||||
MinIO requires starting each :mc:`minio server` in the set with the same
|
||||
startup command to enable awareness of all set peers.
|
||||
|
||||
See :mc-cmd:`minio server` for complete syntax and usage.
|
||||
|
||||
MinIO calculates the size and number of Erasure Sets in the Server Set based
|
||||
on the total number of drives in the set *and* the number of :mc:`minio`
|
||||
servers in the set. See :ref:`minio-ec-erasure-set` for more information.
|
||||
|
||||
.. _minio-intro-cluster:
|
||||
|
||||
:ref:`Cluster <minio-intro-cluster>`
|
||||
The whole MinIO deployment consisting of one or more Server Sets. Each
|
||||
:mc-cmd:`~minio server HOSTNAME` argument passed to the
|
||||
:mc-cmd:`minio server` command represents one Server Set:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
minio server https://minio{1...4}.example.net/mnt/disk{1...4} \
|
||||
https://minio{5...8}.example.net/mnt/disk{1...4}
|
||||
|
||||
| Server Set |
|
||||
|
||||
The above example describes two Server Sets, each consisting of 4
|
||||
:mc:`minio server` nodes with 4 drives each for a total of 32 drives.
|
||||
|
||||
Server Set expansion is a function of Horizontal Scaling, where each new set
|
||||
expands the cluster storage and compute resources. Server Set expansion
|
||||
is not intended to support migrating existing sets to newer hardware.
|
||||
|
||||
MinIO Standalone clusters consist of a single Server Set with a single
|
||||
:mc:`minio server` node. Standalone clusters are best suited for initial
|
||||
development and evaluation. MinIO strongly recommends production
|
||||
clusters consist of a *minimum* of 4 :mc:`minio server` nodes in a
|
||||
Server Set.
|
||||
|
||||
Deploying MinIO
|
||||
---------------
|
||||
|
||||
@ -57,10 +129,3 @@ or containerized environments, install and run the :mc:`minio server` on
|
||||
each host in the MinIO deployment. See :ref:`minio-baremetal` for more
|
||||
information.
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:titlesonly:
|
||||
|
||||
/introduction/deployment-topologies.rst
|
||||
/introduction/erasure-coding.rst
|
||||
/introduction/bitrot-protection.rst
|
||||
Reference in New Issue
Block a user