mirror of
https://github.com/minio/docs.git
synced 2025-08-08 01:43:18 +03:00
Docs Multiplatform Slice
This commit is contained in:
320
source/administration/bucket-replication.rst
Normal file
320
source/administration/bucket-replication.rst
Normal file
@@ -0,0 +1,320 @@
|
||||
.. _minio-bucket-replication:
|
||||
|
||||
==================
|
||||
Bucket Replication
|
||||
==================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO supports server-side and client-side replication of objects between source
|
||||
and destination buckets.
|
||||
|
||||
:ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`
|
||||
Configure per-bucket rules for automatically synchronizing objects between
|
||||
buckets within the same MinIO cluster *or* between two independent MinIO
|
||||
Clusters. MinIO applies rules as part of object write operations
|
||||
(e.g. ``PUT``) and automatically synchronizes new objects *and* object
|
||||
mutations, such as new object versions or changes to object metadata.
|
||||
|
||||
MinIO server-side bucket replication only supports MinIO clusters for the
|
||||
remote replication target.
|
||||
|
||||
Client-side Bucket Replication
|
||||
Use The command process to synchronize objects between buckets
|
||||
within the same S3-compatible cluster *or* between two independent
|
||||
S3-compatible clusters. Client-side replication using :mc-cmd:`mc mirror`
|
||||
supports MinIO-to-S3 and similar replication configurations.
|
||||
|
||||
.. admonition:: Bucket vs Site Replication
|
||||
:class: note
|
||||
|
||||
Bucket Replication is distinct from and mutually exclusive with :ref:`site replication <minio-site-replication-overview>`.
|
||||
|
||||
- Bucket Replication synchronizes data at the bucket level, such as bucket prefix paths and objects.
|
||||
|
||||
You can configure bucket replication at any time, and the remote MinIO deployments may have pre-existing data on the replication target buckets.
|
||||
|
||||
- Site Replication extends bucket replication to include :ref:`IAM <minio-authentication-and-identity-management>`, security tokens, service accounts, and bucket-level configurations.
|
||||
|
||||
Site replication is typically configured when initially deploying the MinIO peer sites.
|
||||
Only one site can hold any bucket or objects at the time of initial configuration.
|
||||
|
||||
|
||||
.. _minio-bucket-replication-serverside:
|
||||
|
||||
Server-Side Bucket Replication
|
||||
------------------------------
|
||||
|
||||
MinIO server-side bucket replication is an automatic bucket-level configuration
|
||||
that synchronizes objects between a source and destination bucket. MinIO
|
||||
server-side replication *requires* the source and destination bucket be two separate MinIO
|
||||
clusters.
|
||||
|
||||
For each write operation to the bucket, MinIO checks all configured replication
|
||||
rules for the bucket and applies the matching rule with highest configured
|
||||
priority. MinIO synchronizes new objects *and* object mutations, such as
|
||||
new object versions or changes to object metadata. This includes
|
||||
metadata operations such as enabling or modifying object locking or
|
||||
retention settings.
|
||||
|
||||
MinIO server-side bucket replication is functionally similar to Amazon S3
|
||||
replication while adding the following MinIO-only features:
|
||||
|
||||
- Source and destination bucket names can match, supporting site-to-site
|
||||
use cases such as Splunk or Veeam BC/DR.
|
||||
|
||||
- Simplified implementation than S3 bucket replication configuration, removing
|
||||
the need to configure settings like AccessControlTranslation, Metrics, and
|
||||
SourceSelectionCriteria.
|
||||
|
||||
- Active-Active (Two-Way) replication of objects between source and destination
|
||||
buckets.
|
||||
|
||||
- Multi-Site replication of objects between three or more MinIO deployments
|
||||
(New in :minio-release:`RELEASE.2021-09-23T04-46-24Z`).
|
||||
|
||||
.. _minio-replication-behavior-resync:
|
||||
|
||||
Resynchronization (Disaster Recovery)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Resynchronization primarily supports recovery after partial or total loss of the
|
||||
data on a MinIO deployment using a healthy deployment in the replica
|
||||
configuration. Use the :mc-cmd:`mc replicate resync` command completely
|
||||
resynchronize the remote target (:mc-cmd:`mc admin bucket remote`) using the
|
||||
specified source bucket.
|
||||
|
||||
The resynchronization process checks all objects in the source bucket against
|
||||
all configured replication rules that include :ref:`existing object replication
|
||||
<minio-replication-behavior-existing-objects>`. For each object which matches a
|
||||
rule, the resynchronization process places the object into the replication
|
||||
:ref:`queue <minio-replication-process>` regardless of the object's current
|
||||
:ref:`replication status <minio-replication-process>`.
|
||||
|
||||
MinIO skips synchronizing those objects whose remote copy exactly match the
|
||||
source, including object metadata. MinIO otherwise does not prioritize or modify
|
||||
the queue with regards to the existing contents of the target.
|
||||
|
||||
:mc-cmd:`mc replicate resync` operates at the bucket level and does
|
||||
*not* support prefix-level granularity. Initiating resynchronization on a large
|
||||
bucket may result in a significant increase in replication-related load
|
||||
and traffic. Use this command with caution and only when necessary.
|
||||
|
||||
For buckets with :ref:`object transition (Tiering)
|
||||
<minio-lifecycle-management-tiering>` configured, replication resynchronization
|
||||
restores objects in a non-transitioned state with no associated transition
|
||||
metadata. Any data previously transitioned to the remote storage is therefore
|
||||
permanently disconnected from the remote MinIO deployment. For tiering
|
||||
configurations which specify an explicit human-readable prefix as part of the
|
||||
remote configuration, you can safely purge the transitioned data in that prefix
|
||||
to avoid costs associated to the "lost" data.
|
||||
|
||||
.. _minio-replication-behavior-delete:
|
||||
|
||||
Replication of Delete Operations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports replicating delete operations, where MinIO synchronizes
|
||||
deleting specific object versions *and* new
|
||||
:s3-docs:`delete markers <delete-marker-replication.html>`. Delete operation
|
||||
replication uses the same :ref:`replication process <minio-replication-process>`
|
||||
as all other replication operations.
|
||||
|
||||
MinIO requires explicitly enabling versioned deletes and delete marker
|
||||
replication . Use the :mc-cmd:`mc replicate add --replicate` field to
|
||||
specify both or either ``delete`` and ``delete-marker`` to enable versioned
|
||||
deletes and delete marker replication respectively. To enable both, specify both
|
||||
strings using a comma separator ``delete,delete-marker``.
|
||||
|
||||
For delete marker replication, MinIO begins the replication process after
|
||||
a delete operation creates the delete marker. MinIO uses the
|
||||
``X-Minio-Replication-DeleteMarker-Status`` metadata field for tracking
|
||||
delete marker replication status. In
|
||||
:ref:`active-active <minio-bucket-replication-serverside-twoway>`
|
||||
replication configurations, MinIO may produce duplicate delete markers if
|
||||
both clusters concurrently create a delete marker for an object *or*
|
||||
if one or both clusters were down before the replication event synchronized.
|
||||
|
||||
For replicating the deletion of a specific object version, MinIO marks the
|
||||
object version as ``PENDING`` until replication completes. Once the remote
|
||||
target deletes that object version, MinIO deletes the object on the source.
|
||||
While this process ensures near-synchronized version deletion, it may result
|
||||
in listing operations returning the object version after the initial
|
||||
delete operation. MinIO uses the ``X-Minio-Replication-Delete-Status`` for
|
||||
tracking delete version replication status.
|
||||
|
||||
MinIO only replicates explicit client-driven delete operations. MinIO does *not*
|
||||
replicate objects deleted from the application of
|
||||
:ref:`lifecycle management expiration rules
|
||||
<minio-lifecycle-management-expiration>`. For :ref:`active-active
|
||||
<minio-bucket-replication-serverside-twoway>` configurations, set the same
|
||||
expiration rules on *all* of of the replication buckets to ensure consistent
|
||||
application of object expiration.
|
||||
|
||||
.. admonition:: MinIO Trims Empty Object Prefixes on Source and Remote Bucket
|
||||
:class: note, dropdown
|
||||
|
||||
If a delete operation removes the last object in a bucket prefix, MinIO
|
||||
recursively removes each empty part of the prefix up to the bucket root.
|
||||
MinIO only applies the recursive removal to prefixes created *implicitly* as
|
||||
part of object write operations - that is, the prefix was not created using
|
||||
an explicit directory creation command such as :mc:`mc mb`.
|
||||
|
||||
If a replication rule enables replication delete operations, the replication
|
||||
process *also* applies the implicit prefix trimming behavior on the
|
||||
destination MinIO cluster.
|
||||
|
||||
For example, consider a bucket ``photos`` with the following object prefixes:
|
||||
|
||||
- ``photos/2021/january/myphoto.jpg``
|
||||
- ``photos/2021/february/myotherphoto.jpg``
|
||||
- ``photos/NYE21/NewYears.jpg``
|
||||
|
||||
``photos/NYE21`` is the *only* prefix explicitly created using :mc:`mc mb`.
|
||||
All other prefixes were *implicitly* created as part of writing the object
|
||||
located at that prefix.
|
||||
|
||||
- A command removes ``myphoto.jpg``. MinIO automatically trims the empty
|
||||
``/janaury`` prefix.
|
||||
|
||||
- A command then removes the ``myotherphoto.jpg``. MinIO automatically
|
||||
trims the ``/february`` prefix *and* the now-empty ``/2021`` prefix.
|
||||
|
||||
- A command removes the ``NewYears.jpg`` object. MinIO leaves the
|
||||
``/NYE21`` prefix remains in place since it was *explicitly* created.
|
||||
|
||||
.. _minio-replication-behavior-existing-objects:
|
||||
|
||||
Replication of Existing Objects
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO by default does not enable existing object replication. Objects
|
||||
created before replication was configured *or* while replication is
|
||||
disabled are not synchronized to the target deployment.
|
||||
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
|
||||
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
|
||||
:minio-git:`RELEASE.2021-06-07T21-40-51Z
|
||||
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports enabling
|
||||
replication of existing objects in a bucket.
|
||||
|
||||
Enabling existing object replication marks all objects or object prefixes that
|
||||
satisfy the replication rules as eligible for synchronization to the source
|
||||
cluster, *even if* those objects were created prior to configuring or enabling
|
||||
replication. You can enable existing object replication while configuring
|
||||
or modifying a replication rule:
|
||||
|
||||
- For new replication rules, include ``"existing-objects"`` to the list of
|
||||
replication features specified to :mc-cmd:`mc replicate add --replicate`.
|
||||
|
||||
- For existing replication rules, add ``"existing-objects"`` to the list of
|
||||
existing replication features using
|
||||
:mc-cmd:`mc replicate edit --replicate`. You must specify *all* desired
|
||||
replication features when editing the replication rule.
|
||||
|
||||
Enabling existing object replication does not increase the priority of objects
|
||||
pending replication. MinIO uses the same core
|
||||
:ref:`replication scanner and queue system <minio-replication-process>` for
|
||||
detecting and synchronizing objects regardless of the enabled replication
|
||||
feature. The time required to fully synchronize a bucket depends on a number of
|
||||
factors, including but not limited to the current cluster replication load,
|
||||
overall cluster load, and the size of the namespace (all objects in the bucket).
|
||||
|
||||
If versioning was not previously enabled when configuring bucket replication,
|
||||
existing objects have a ``versionid = null``. These objects do replicate.
|
||||
|
||||
MinIO existing object replication
|
||||
implements functionality similar to
|
||||
`AWS: Replicating existing objects between S3 buckets
|
||||
<https://aws.amazon.com/blogs/storage/replicating-existing-objects-between-s3-buckets/>`__
|
||||
without the overhead of contacting technical support.
|
||||
|
||||
Synchronous vs Asynchronous Replication
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports specifying either asynchronous (default) or synchronous
|
||||
replication for a given remote target.
|
||||
|
||||
With the default asynchronous replication, MinIO completes the originating
|
||||
``PUT`` operation *before* placing the object into a :ref:`replication queue
|
||||
<minio-replication-process>`. The originating client may therefore see a
|
||||
successful ``PUT`` operation *before* the object is replicated. While
|
||||
this may result in stale or missing objects on the remote, it mitigates
|
||||
the risk of slow write operations due to replication load.
|
||||
|
||||
With synchronous replication, MinIO attempts to replicate the object *prior* to
|
||||
completing the originating ``PUT`` operation. MinIO returns a successful ``PUT``
|
||||
operation whether or not the replication attempts succeeds. While this may
|
||||
result in more reliable synchronization between the source and remote target,
|
||||
it may also increase the time of each write operation due to replication load.
|
||||
|
||||
You must explicitly enable synchronous replication when configuring the remote
|
||||
target target using the :mc-cmd:`mc admin bucket remote add` command with the
|
||||
:mc-cmd:`~mc admin bucket remote add` flag.
|
||||
|
||||
Replication Internals
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This section documents internal replication behavior and is not critical to
|
||||
using or implementing replication. This documentation is provided strictly
|
||||
for learning and educational purposes.
|
||||
|
||||
.. _minio-replication-process:
|
||||
|
||||
Replication Process
|
||||
+++++++++++++++++++
|
||||
|
||||
MinIO uses a replication queuing system with multiple concurrent replication
|
||||
workers operating on that queue. MinIO continuously works to replicate and
|
||||
remove objects from the queue while scanning for new unreplicated objects to
|
||||
add to the queue.
|
||||
|
||||
MinIO sets the ``X-Amz-Replication-Status`` metadata field according to the
|
||||
replication state of the object:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:width: 100%
|
||||
:widths: 30 70
|
||||
|
||||
* - Replication State
|
||||
- Description
|
||||
|
||||
* - ``PENDING``
|
||||
- The object has not yet been replicated. MinIO applies this state
|
||||
if the object meets one of the configured replication rules on the
|
||||
bucket. MinIO continuously scans for ``PENDING`` objects not yet in the
|
||||
replication queue and adds them to the queue as space is available.
|
||||
|
||||
For multi-site replication, objects remain
|
||||
in the ``PENDING`` state until replicated to *all* configured
|
||||
remotes for that bucket or bucket prefix.
|
||||
|
||||
* - ``COMPLETED``
|
||||
- The object has successfully replicated to the remote cluster.
|
||||
|
||||
* - ``FAILED``
|
||||
- The object failed to replicate to the remote cluster.
|
||||
|
||||
MinIO continuously scans for ``FAILED`` objects not yet in the
|
||||
replication queue and adds them to the queue as space is available.
|
||||
|
||||
* - ``REPLICA``
|
||||
- The object is itself a replica from a remote source.
|
||||
|
||||
The replication process generally has one of the following flows:
|
||||
|
||||
- ``PENDING -> COMPLETED``
|
||||
- ``PENDING -> FAILED -> COMPLETED``
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:titlesonly:
|
||||
|
||||
/administration/bucket-replication/enable-server-side-one-way-bucket-replication
|
||||
/administration/bucket-replication/enable-server-side-two-way-bucket-replication
|
||||
/administration/bucket-replication/enable-server-side-multi-site-bucket-replication
|
||||
/administration/bucket-replication/server-side-replication-resynchronize-remote
|
@@ -0,0 +1,230 @@
|
||||
.. _minio-bucket-replication-serverside-multi:
|
||||
|
||||
================================================
|
||||
Enable Multi-Site Server-Side Bucket Replication
|
||||
================================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
|
||||
The procedure on this page configures automatic server-side bucket replication between multiple MinIO deployments. Multi-Site Active-Active replication builds on the :ref:`minio-bucket-replication-serverside-twoway` procedure with additional considerations required to ensure predictable replication behavior across all sites.
|
||||
|
||||
.. image:: /images/replication/active-active-multi-replication.svg
|
||||
:width: 600px
|
||||
:alt: Active-Active Replication synchronizes data between multiple remote deployments.
|
||||
:align: center
|
||||
|
||||
- To configure replication between arbitrary S3-compatible services, use :mc-cmd:`mc mirror`.
|
||||
|
||||
- To configure one-way "active-active" replication between two MinIO deployments, see :ref:`minio-bucket-replication-serverside-twoway`.
|
||||
|
||||
- To configure one-way "active-passive" replication between MinIO deployments, see :ref:`minio-bucket-replication-serverside-oneway`.
|
||||
|
||||
Multi-Site Active-Active replication configurations can span multiple racks, datacenters, or geographic locations. Complexity of configuring and maintaining multi-site configurations generally increase with the number of sites and size of each site. Enterprises looking to implement multi-site replication should consider leveraging `MinIO SUBNET <https://min.io/pricing?ref=docs>`__ support to access the expertise, planning, and engineering resources required for addressing that use case.
|
||||
|
||||
.. seealso::
|
||||
|
||||
- Use the :mc-cmd:`mc replicate edit` command to modify an existing replication rule.
|
||||
|
||||
- Use the :mc-cmd:`mc replicate edit` command with the :mc-cmd:`--state "disable" <mc replicate edit --state>` flag to disable an existing replication rule.
|
||||
|
||||
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule.
|
||||
|
||||
.. _minio-bucket-replication-serverside-multi-requirements:
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
Install and Configure ``mc`` with Access to Both Clusters.
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This procedure uses :mc:`mc` for performing operations on both the source and destination MinIO cluster. Install :mc:`mc` on a machine with network access to both source and destination deployments. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for instructions on downloading and installing ``mc``.
|
||||
|
||||
Use the :mc:`mc alias` command to create an alias for both MinIO deployments. Alias creation requires specifying an access key for a user on the cluster. This user **must** have permission to create and manage users and policies on the cluster. Specifically, ensure the user has *at minimum*:
|
||||
|
||||
- :policy-action:`admin:CreateUser`
|
||||
- :policy-action:`admin:ListUsers`
|
||||
- :policy-action:`admin:GetUser`
|
||||
- :policy-action:`admin:CreatePolicy`
|
||||
- :policy-action:`admin:GetPolicy`
|
||||
- :policy-action:`admin:AttachUserOrGroupPolicy`
|
||||
|
||||
.. _minio-bucket-replication-serverside-multi-permissions:
|
||||
|
||||
Required Permissions
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-required-permissions
|
||||
:end-before: end-replication-required-permissions
|
||||
|
||||
Replication Requires Matching Object Encryption Settings
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-encrypted-objects
|
||||
:end-before: end-replication-encrypted-objects
|
||||
|
||||
Replication Requires MinIO Deployments
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-minio-only
|
||||
:end-before: end-replication-minio-only
|
||||
|
||||
Replication Requires Versioning
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-requires-versioning
|
||||
:end-before: end-replication-requires-versioning
|
||||
|
||||
Replication Requires Matching Object Locking State
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-requires-object-locking
|
||||
:end-before: end-replication-requires-object-locking
|
||||
|
||||
Considerations
|
||||
--------------
|
||||
|
||||
Use Consistent Replication Settings
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports customizing the replication configuration to enable or disable the following replication behaviors:
|
||||
|
||||
- Replication of delete operations
|
||||
- Replication of delete markers
|
||||
- Replication of existing objects
|
||||
- Replication of metadata-only changes
|
||||
|
||||
When configuring replication rules for a bucket, ensure that all MinIO deployments participating in multi-site replication use the *same* replication behaviors to ensure consistent and predictable synchronization of objects.
|
||||
|
||||
Replication of Existing Objects
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports automatically replicating existing objects in a bucket.
|
||||
|
||||
MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag. This procedure includes the required flags for enabling replication of existing objects.
|
||||
|
||||
Replication of Delete Operations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports replicating delete operations onto the target bucket. Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects:
|
||||
|
||||
- For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
|
||||
|
||||
- For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
|
||||
|
||||
MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`. This procedure includes the required flags for enabling replication of delete operations and delete markers.
|
||||
|
||||
MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`. Configure matching expiration rules for the bucket on all replication sites to ensure consistent application of object expiration.
|
||||
|
||||
Procedure
|
||||
---------
|
||||
|
||||
This procedure requires repeating steps for each MinIO deployment participating in the multi-site replication configuration. Depending on the number of deployments, this procedure may require significant time and care in implementation. MinIO recommends reading through the procedure *before* attempting to implement the documented steps.
|
||||
|
||||
This procedure uses the placeholder ``ALIAS`` to reference the :ref:`alias <alias>` each MinIO deployment being configured for replication. Replace these values with the appropriate alias for each MinIO deployment.
|
||||
|
||||
This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-multi-permissions>`.
|
||||
|
||||
1) Create the Replication Remote Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin bucket remote add` command to create a replication target for the each deployment. MinIO supports *one* remote target per destination bucket. You cannot create multiple remote targets for the same destination bucket.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin bucket remote add ALIAS/BUCKET \
|
||||
https://ReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/BUCKET \
|
||||
--service "replication"
|
||||
|
||||
- Replace ``BUCKET`` with the name of the bucket on the ``ALIAS`` deployment to use as the replication source. Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication.
|
||||
|
||||
- Replace ``HOSTNAME`` with the URL of the remote MinIO deployment.
|
||||
|
||||
- Replace ``BUCKET`` with the name of the bucket on the remote deployment to use as the replication destination.
|
||||
|
||||
The command returns an ARN similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
Role ARN = 'arn:minio:replication::<UUID>:BUCKET'
|
||||
|
||||
Copy the ARN string for use in the next step, noting the MinIO deployment on which it was created.
|
||||
|
||||
Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration. For example, a multi-site replication configuration consisting of MinIO deployments ``Alpha``, ``Baker``, and ``Charlie`` would require repeating this step on each deployment for each remote. Specifically:
|
||||
|
||||
- The ``Alpha`` deployment would perform this step once for
|
||||
``Baker`` and once for ``Charlie``.
|
||||
|
||||
- The ``Baker`` deployment would perform this step once for ``Alpha`` and
|
||||
once for ``Charlie``.
|
||||
|
||||
- The ``Charlie`` deployment would perform this step once for ``Baker`` and
|
||||
once for ``Alpha``.
|
||||
|
||||
2) Create a New Bucket Replication Rule
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc replicate add` command to add the new server-side
|
||||
replication rule to the each MinIO deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc replicate add ALIAS/BUCKET \
|
||||
--remote-bucket 'arn:minio:replication::<UUID>:BUCKET' \
|
||||
--replicate "delete,delete-marker,existing-objects"
|
||||
|
||||
- Replace ``BUCKET`` with the name of the bucket on the ``ALIAS`` deployment to use as the replication source. Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication.
|
||||
|
||||
- Replace the ``--remote-bucket`` value with the ARN returned in the previous step. Ensure you specify the ARN created on the ``ALIAS`` deployment. You can use :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the deployment.
|
||||
|
||||
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables the following replication features:
|
||||
|
||||
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
|
||||
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
|
||||
|
||||
See :mc-cmd:`mc replicate add --replicate` for more complete documentation. Omit these fields to disable replication of delete operations or replication of existing objects respectively.
|
||||
|
||||
Specify any other supported optional arguments for :mc-cmd:`mc replicate add`.
|
||||
|
||||
Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration. For example, a multi-site replication configuration consisting of MinIO deployments ``Alpha``, ``Baker``, and ``Charlie`` would require repeating this step on each deployment for each remote. Specifically:
|
||||
|
||||
- The ``Alpha`` deployment would perform this step once for
|
||||
``Baker`` and once for ``Charlie``.
|
||||
|
||||
- The ``Baker`` deployment would perform this step once for ``Alpha`` and
|
||||
once for ``Charlie``.
|
||||
|
||||
- The ``Charlie`` deployment would perform this step once for ``Baker`` and
|
||||
once for ``Alpha``.
|
||||
|
||||
3) Validate the Replication Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use :mc-cmd:`mc cp` to copy a new object the bucket on any of the deployments:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/foo.txt ALIAS/BUCKET
|
||||
|
||||
Use :mc-cmd:`mc ls` to verify the object exists on each remote deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc ls REMOTE/BUCKET
|
||||
|
||||
Repeat this test on each of the deployments by copying a new unique file and checking the other deployments for that file.
|
||||
|
||||
You can also use :mc-cmd:`mc stat` to check the file to check the current :ref:`replication stage <minio-replication-process>` of the object.
|
@@ -0,0 +1,186 @@
|
||||
.. _minio-bucket-replication-serverside-oneway:
|
||||
|
||||
=============================================
|
||||
Enable One-Way Server-Side Bucket Replication
|
||||
=============================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
|
||||
The procedure on this page creates a new bucket replication rule for one-way synchronization of objects between MinIO buckets.
|
||||
|
||||
.. image:: /images/replication/active-passive-oneway-replication.svg
|
||||
:width: 450px
|
||||
:alt: Active-Passive Replication synchronizes data from a source MinIO cluster to a remote MinIO cluster.
|
||||
:align: center
|
||||
|
||||
- To configure replication between arbitrary S3-compatible services, use :mc-cmd:`mc mirror`.
|
||||
|
||||
- To configure two-way "active-active" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-twoway`.
|
||||
|
||||
- To configure multi-site "active-active" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-multi`
|
||||
|
||||
.. seealso::
|
||||
|
||||
- Use the :mc-cmd:`mc replicate edit` command to modify an existing replication rule.
|
||||
|
||||
- Use the :mc-cmd:`mc replicate edit` command with the :mc-cmd:`--state "disable" <mc replicate edit --state>` flag to disable an existing replication rule.
|
||||
|
||||
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule.
|
||||
|
||||
.. _minio-bucket-replication-serverside-oneway-requirements:
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
.. _minio-bucket-replication-serverside-oneway-permissions:
|
||||
|
||||
Required Permissions
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-required-permissions
|
||||
:end-before: end-replication-required-permissions
|
||||
|
||||
Replication Requires Matching Object Encryption Settings
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-encrypted-objects
|
||||
:end-before: end-replication-encrypted-objects
|
||||
|
||||
Replication Requires MinIO Deployments
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-minio-only
|
||||
:end-before: end-replication-minio-only
|
||||
|
||||
Replication Requires Versioning
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-requires-versioning
|
||||
:end-before: end-replication-requires-versioning
|
||||
|
||||
Replication Requires Matching Object Locking State
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-requires-object-locking
|
||||
:end-before: end-replication-requires-object-locking
|
||||
|
||||
Considerations
|
||||
--------------
|
||||
|
||||
Replication of Existing Objects
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports automatically replicating existing objects in a bucket.
|
||||
|
||||
MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag. This procedure includes the required flags for enabling replication of existing objects.
|
||||
|
||||
Replication of Delete Operations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports replicating S3 ``DELETE`` operations onto the target bucket. Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects:
|
||||
|
||||
- For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
|
||||
|
||||
- For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
|
||||
|
||||
MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`. This procedure includes the required flags for enabling replication of delete operations and delete markers.
|
||||
|
||||
MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`.
|
||||
|
||||
See :ref:`minio-replication-behavior-delete` for more complete documentation.
|
||||
|
||||
Multi-Site Replication
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports configuring multiple remote targets per bucket or bucket prefix. For example, you can configure a bucket to replicate data to two or more remote MinIO deployments, where one deployment is a 1:1 copy (replication of all operations including deletions) and another is a full historical record (replication of only non-destructive write operations).
|
||||
|
||||
This procedure documents one-way replication to a single remote MinIO deployment. You can repeat this tutorial for multiple remote targets for a single bucket.
|
||||
|
||||
Procedure
|
||||
---------
|
||||
|
||||
This procedure uses the :ref:`aliases <alias>` ``SOURCE`` and ``REMOTE`` to reference each MinIO deployment being configured for replication. Replace these values with the appropriate alias for your target MinIO deployments.
|
||||
|
||||
This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-oneway-permissions>`.
|
||||
|
||||
1) Create the Replication Remote Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin bucket remote add` command to create a replication target for the destination cluster. MinIO supports *one* remote target per destination bucket. You cannot create multiple remote targets for the same destination bucket.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin bucket remote add SOURCE/BUCKET \
|
||||
https://ReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/BUCKET \
|
||||
--service "replication"
|
||||
[--sync]
|
||||
|
||||
- Replace ``BUCKET`` with the name of the bucket on the ``SOURCE`` deployment to use as the replication source. Replace ``SOURCE`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication.
|
||||
|
||||
- Replace ``HOSTNAME`` with the URL of the ``REMOTE`` cluster.
|
||||
|
||||
- Replace ``BUCKET`` with the name of the bucket on the ``REMOTE`` deployment to use as the replication destination.
|
||||
|
||||
- Include the :mc-cmd:`~mc admin bucket remote add --sync` option to enable synchronous replication. Omit the option to use the default of asynchronous replication. See the reference documentation for :mc-cmd:`mc admin bucket remote add` for more information on synchronous vs asynchronous replication before using this parameter.
|
||||
|
||||
The command returns an ARN similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
Role ARN = 'arn:minio:replication::<UUID>:BUCKET'
|
||||
|
||||
Copy the ARN string for use in the next step.
|
||||
|
||||
2) Create a New Bucket Replication Rule
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc replicate add` command to add the new server-side
|
||||
replication rule to the source MinIO cluster.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc replicate add SOURCE/BUCKET \
|
||||
--remote-bucket 'arn:minio:replication::<UUID>:BUCKET' \
|
||||
--replicate "delete,delete-marker,existing-objects"
|
||||
|
||||
- Replace ``BUCKET`` with the name of the bucket on the ``SOURCE`` deployment to use as the replication source. Replace ``SOURCE`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication. The name *must* match the bucket specified when creating the remote target in the previous step.
|
||||
|
||||
- Replace the ``--remote-bucket`` value with the ARN returned in the previous step. Ensure you specify the ARN created on the ``SOURCE`` deployment. You can use :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the deployment.
|
||||
|
||||
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables the following replication features:
|
||||
|
||||
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
|
||||
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
|
||||
|
||||
See :mc-cmd:`mc replicate add --replicate` for more complete documentation. Omit these fields to disable replication of delete operations or replication of existing objects respectively.
|
||||
|
||||
Specify any other supported optional arguments for :mc-cmd:`mc replicate add`.
|
||||
|
||||
3) Validate the Replication Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use :mc-cmd:`mc cp` to copy a new object to the source bucket.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/foo.txt SOURCE/BUCKET
|
||||
|
||||
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc ls TARGET/BUCKET
|
@@ -0,0 +1,235 @@
|
||||
.. _minio-bucket-replication-serverside-twoway:
|
||||
|
||||
=============================================
|
||||
Enable Two-Way Server-Side Bucket Replication
|
||||
=============================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
|
||||
The procedure on this page creates a new bucket replication rule for two-way "active-active" synchronization of objects between MinIO buckets.
|
||||
|
||||
.. image:: /images/replication/active-active-twoway-replication.svg
|
||||
:width: 600px
|
||||
:alt: Active-Active Replication synchronizes data between two remote clusters.
|
||||
:align: center
|
||||
|
||||
- To configure replication between arbitrary S3-compatible services, use :mc-cmd:`mc mirror`.
|
||||
|
||||
- To configure one-way "active-passive" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-oneway`.
|
||||
|
||||
- To configure multi-site "active-active" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-multi`.
|
||||
|
||||
This tutorial covers configuring Active-Active replication between two MinIO clusters. For a tutorial on multi-site replication between three or more MinIO clusters, see :ref:`minio-bucket-replication-serverside-multi`.
|
||||
|
||||
.. seealso::
|
||||
|
||||
- Use the :mc-cmd:`mc replicate edit` command to modify an existing
|
||||
replication rule.
|
||||
|
||||
- Use the :mc-cmd:`mc replicate edit` command with the :mc-cmd:`--state "disable" <mc replicate edit --state>` flag to disable an existing replication rule.
|
||||
|
||||
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule.
|
||||
|
||||
.. _minio-bucket-replication-serverside-twoway-requirements:
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
Install and Configure ``mc`` with Access to Both Clusters.
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This procedure uses :mc:`mc` for performing operations on both the source and destination MinIO cluster. Install :mc:`mc` on a machine with network access to both source and destination clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for instructions on downloading and installing ``mc``.
|
||||
|
||||
Use the :mc:`mc alias` command to create an alias for both MinIO clusters. Alias creation requires specifying an access key for a user on the cluster. This user **must** have permission to create and manage users and policies on the cluster. Specifically, ensure the user has *at minimum*:
|
||||
|
||||
- :policy-action:`admin:CreateUser`
|
||||
- :policy-action:`admin:ListUsers`
|
||||
- :policy-action:`admin:GetUser`
|
||||
- :policy-action:`admin:CreatePolicy`
|
||||
- :policy-action:`admin:GetPolicy`
|
||||
- :policy-action:`admin:AttachUserOrGroupPolicy`
|
||||
|
||||
.. _minio-bucket-replication-serverside-twoway-permissions:
|
||||
|
||||
Required Permissions
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-required-permissions
|
||||
:end-before: end-replication-required-permissions
|
||||
|
||||
Replication Requires Matching Object Encryption Settings
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-encrypted-objects
|
||||
:end-before: end-replication-encrypted-objects
|
||||
|
||||
Replication Requires MinIO Deployments
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-minio-only
|
||||
:end-before: end-replication-minio-only
|
||||
|
||||
Replication Requires Versioning
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-requires-versioning
|
||||
:end-before: end-replication-requires-versioning
|
||||
|
||||
Replication Requires Matching Object Locking State
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-requires-object-locking
|
||||
:end-before: end-replication-requires-object-locking
|
||||
|
||||
Considerations
|
||||
--------------
|
||||
|
||||
Use Consistent Replication Settings
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports customizing the replication configuration to enable or disable
|
||||
the following replication behaviors:
|
||||
|
||||
- Replication of delete operations
|
||||
- Replication of delete markers
|
||||
- Replication of existing objects
|
||||
- Replication of metadata-only changes
|
||||
|
||||
When configuring replication rules for a bucket, ensure that both MinIO deployments participating in active-active replication use the *same* replication behaviors to ensure consistent and predictable synchronization of objects.
|
||||
|
||||
Replication of Existing Objects
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports automatically replicating existing objects in a bucket.
|
||||
|
||||
MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag. This procedure includes the required flags for enabling replication of existing objects.
|
||||
|
||||
Replication of Delete Operations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports replicating delete operations onto the target bucket. Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects:
|
||||
|
||||
- For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
|
||||
|
||||
- For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
|
||||
|
||||
MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`. This procedure includes the required flags for enabling replication of delete operations and delete markers.
|
||||
|
||||
MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`. Configure matching expiration rules on both the source and destination bucket to ensure consistent application of object expiration.
|
||||
|
||||
See :ref:`minio-replication-behavior-delete` for more complete documentation.
|
||||
|
||||
Multi-Site Replication
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports configuring multiple remote targets per bucket or bucket prefix. This enables configuring multi-site active-active replication between MinIO deployments.
|
||||
|
||||
This procedure covers active-active replication between *two* MinIO sites. You can repeat this procedure for each "pair" of MinIO deployments in the replication mesh. For a dedicated tutorial, see :ref:`minio-bucket-replication-serverside-multi`.
|
||||
|
||||
Procedure
|
||||
---------
|
||||
|
||||
This procedure uses the :ref:`aliases <alias>` ``ALPHA`` and ``BAKER`` to reference each MinIO deployment being configured for replication. Replace these values with the appropriate alias for your target MinIO deployments.
|
||||
|
||||
This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-twoway-permissions>`.
|
||||
|
||||
1) Create the Replication Remote Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin bucket remote add` command to create a replication target for the each deployment. MinIO supports *one* remote target per destination bucket. You cannot create multiple remote targets for the same destination bucket.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin bucket remote add ALPHA/BUCKET \
|
||||
https://ReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/BUCKET \
|
||||
--service "replication"
|
||||
|
||||
- Replace ``BUCKET`` with the name of the bucket on the ``ALPHA`` deployment to use as the replication source. Replace ``ALPHA`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication.
|
||||
|
||||
- Replace ``HOSTNAME`` with the URL of the ``BAKER`` deployment.
|
||||
|
||||
- Replace ``BUCKET`` with the name of the bucket on the ``REMOTE`` deployment to use as the replication destination.
|
||||
|
||||
The command returns an ARN similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
Role ARN = 'arn:minio:replication::<UUID>:BUCKET'
|
||||
|
||||
Copy the ARN string for use in the next step, noting the MinIO deployment on which it was created.
|
||||
|
||||
Repeat this step on the second MinIO deployment, replacing the ``ALPHA`` alias with the ``BAKER`` alias and the ``HOSTNAME`` with the URL of the ``ALPHA`` deployment.
|
||||
|
||||
You should have two ARNs at the conclusion of this step - one created on ``ALPHA/BUCKET`` pointing at ``BAKER/BUCKET``, and one created on ``BAKER/BUCKET`` pointing at ``ALPHA/BUCKET``. Use the :mc-cmd:`mc admin bucket remote ls` command to verify the created replication remote targets before proceeding.
|
||||
|
||||
2) Create a New Bucket Replication Rule
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc replicate add` command to add the new server-side
|
||||
replication rule to the each MinIO deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc replicate add ALPHA/BUCKET \
|
||||
--remote-bucket 'arn:minio:replication::<UUID>:BUCKET' \
|
||||
--replicate "delete,delete-marker,existing-objects"
|
||||
|
||||
- Replace ``BUCKET`` with the name of the bucket on the ``ALPHA`` deployment to use as the replication source. Replace ``ALPHA`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication. The name *must* match the bucket specified when creating the remote target in the previous step.
|
||||
|
||||
- Replace the ``--remote-bucket`` value with the ARN returned in the previous step. Ensure you specify the ARN created on the ``ALPHA`` deployment. You can use :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the deployment.
|
||||
|
||||
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables the following replication features:
|
||||
|
||||
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
|
||||
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
|
||||
|
||||
See :mc-cmd:`mc replicate add --replicate` for more complete documentation. Omit these fields to disable replication of delete operations or replication of existing objects respectively.
|
||||
|
||||
Specify any other supported optional arguments for :mc-cmd:`mc replicate add`.
|
||||
|
||||
Repeat this step on the second MinIO deployment, replacing the ``ALPHA`` alias with the ``BAKER`` alias and the ``HOSTNAME`` with the URL of the ``ALPHA`` deployment.
|
||||
|
||||
You should have two replication rules configured at the conclusion of this step - one created on ``ALPHA/BUCKET`` and one created on ``BAKER/BUCKET``. Use the :mc-cmd:`mc replicate ls` command to verify the created replication rules.
|
||||
|
||||
3) Validate the Replication Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use :mc-cmd:`mc cp` to copy a new object to the ``ALPHA/BUCKET`` bucket.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/foo.txt ALPHA/BUCKET
|
||||
|
||||
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc ls BAKER/BUCKET
|
||||
|
||||
Repeat this test by copying a new object to the ``Baker`` source bucket.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/otherfoo.txt BAKER/BUCKET
|
||||
|
||||
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc ls ALPHA/BUCKET
|
@@ -0,0 +1,182 @@
|
||||
.. _minio-bucket-replication-resynchronize:
|
||||
|
||||
|
||||
========================================
|
||||
Resynchronize Bucket from Remote Replica
|
||||
========================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
The procedure on this page resynchronizes the contents of a MinIO bucket using a healthy replication remote. Resynchronization supports recovery after partial or total loss of data on a MinIO deployment in a replica configuration.
|
||||
|
||||
For example, consider a MinIO active-active replication configuration similar to the following:
|
||||
|
||||
.. image:: /images/replication/active-active-twoway-replication.svg
|
||||
:width: 600px
|
||||
:alt: Active-Active Replication synchronizes data between two remote deployments.
|
||||
:align: center
|
||||
|
||||
Resynchronization allows using the healthy data on one of the participating MinIO deployments as the source for rebuilding the other deployment.
|
||||
|
||||
Resynchronization is a per-bucket process. You must repeat resynchronization for each bucket on the remote which suffered partial or total data loss.
|
||||
|
||||
.. admonition:: Professional Support during BC/DR Operations
|
||||
:class: important
|
||||
|
||||
`MinIO SUBNET <https://min.io/pricing?jmp=docs>`__ users can `log in <https://subnet.min.io/>`__ and create a new issue related to resynchronization. Coordination with MinIO Engineering via SUBNET can ensure successful resynchronization and restoration of normal operations, including performance testing and health diagnostics.
|
||||
|
||||
Community users can seek support on the `MinIO Community Slack <https://minio.slack.com>`__. Community Support is best-effort only and has no SLAs around responsiveness.
|
||||
|
||||
.. _minio-bucket-replication-serverside-resynchronize-requirements:
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
MinIO Deployments Must Be Online
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Resynchronization requires both the source and target deployments be online and able to accept read and write operations. The source *must* have complete network connectivity to the remote.
|
||||
|
||||
The remote deployment may be "unhealthy" in that it has suffered partial or total data loss. Resynchronization addresses the data loss as long as both source and destination maintain connectivity.
|
||||
|
||||
Resynchronization Requires Existing Replication Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Resynchronization requires the healthy source deployment have an existing replication configuration for the unhealthy target bucket. Additionally, resynchronization only applies to those replication rules created with the :ref:`existing object replication <minio-replication-behavior-existing-objects>` option.
|
||||
|
||||
- Use :mc-cmd:`mc admin bucket remote ls` to review the configured remote targets on the healthy source bucket.
|
||||
- Use :mc-cmd:`mc replicate ls` to review the configured replication rules on the healthy source bucket.
|
||||
|
||||
Replication Requires Matching Object Encryption Settings
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-encrypted-objects
|
||||
:end-before: end-replication-encrypted-objects
|
||||
|
||||
Replication Requires MinIO Deployments
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-minio-only
|
||||
:end-before: end-replication-minio-only
|
||||
|
||||
Replication Requires Versioning
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-requires-versioning
|
||||
:end-before: end-replication-requires-versioning
|
||||
|
||||
Replication Requires Matching Object Locking State
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-replication.rst
|
||||
:start-after: start-replication-requires-object-locking
|
||||
:end-before: end-replication-requires-object-locking
|
||||
|
||||
Considerations
|
||||
--------------
|
||||
|
||||
Resynchronization Requires Time
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Resynchronization is a background processes that continually checks objects in the source MinIO bucket and copies them to the remote as-needed. The time required for replication to complete may vary depending on the number and size of objects, the throughput to the remote MinIO deployment, and the load on the source MinIO deployment. Total time for completion is generally not predictable due to these variables.
|
||||
|
||||
MinIO recommends configuring load balancers or proxies to direct traffic only to the healthy cluster until synchronization completes. The following commands can provide insight into the resynchronization status:
|
||||
|
||||
- :mc-cmd:`mc replicate resync status` on the source to track the resynchronization progress.
|
||||
|
||||
- :mc-cmd:`mc replicate status` on the source and remote to track normal replication data.
|
||||
|
||||
- Run ``mc ls -r --versions ALIAS/BUCKET | wc -l`` against both source and remote to validate the total number of objects and object versions on each.
|
||||
|
||||
Resynchronize Objects after Data Loss
|
||||
-------------------------------------
|
||||
|
||||
This procedure uses an existing :ref:`MinIO replication configuration <minio-bucket-replication-serverside>` to restore missing data to one of the MinIO deployments participating in that configuration. Specifically, a healthy MinIO deployment (the ``SOURCE``) synchronizes it's existing data to the unhealthy MinIO deployment (the ``TARGET``).
|
||||
|
||||
This procedure assumes an existing :ref:`alias <alias>` for the ``SOURCE`` that has the :ref:`necessary permissions <minio-bucket-replication-serverside-twoway-permissions>` for configuring replication.
|
||||
|
||||
You can repeat this procedure for each bucket that requires resynchronization. You can have no more than one replication job running per bucket.
|
||||
|
||||
1) List the Configured Replication Targets on the Healthy Source
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Run the :mc-cmd:`mc admin bucket remote ls` command to list the configured remote targets on the healthy ``SOURCE`` deployment for the ``BUCKET`` that requires resynchronization.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin bucket remote ls SOURCE/BUCKET
|
||||
|
||||
- Replace ``SOURCE`` with the :ref:`alias <alias>` of the source MinIO deployment.
|
||||
|
||||
- Replace ``BUCKET`` with the name of the bucket to use as the source for resynchronization.
|
||||
|
||||
The output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
Remote URL Source->Target ARN SYNC PROXY
|
||||
https://minio-2.example.net:9000 data ->data arn:minio:replication::UUID:BUCKET proxy
|
||||
|
||||
|
||||
Identify the ARN associated to the ``BUCKET`` on the unhealthy ``TARGET`` MinIO deployment for use in the next step.
|
||||
|
||||
2) Start the Resynchronization Procedure
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Run the :mc-cmd:`mc replicate resync start` command to begin the resynchronization process:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc replicate resync start --remote-bucket "arn:minio:replication::UUID:BUCKET" SOURCE/BUCKET
|
||||
|
||||
- Replace the ``--remote-bucket`` value with the ARN of the unhealthy ``BUCKET`` on the ``TARGET`` MinIO deployment.
|
||||
|
||||
- Replaced ``SOURCE`` with the :ref:`alias <alias>` of the source MinIO deployment.
|
||||
|
||||
- Replace the ``BUCKET`` with the name of the bucket on the healthy ``SOURCE`` MinIO
|
||||
deployment.
|
||||
|
||||
The command returns a resynchronization job ID indicating that the process has begun.
|
||||
|
||||
3) Monitor Resynchronization
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc replicate resync status` command on the source deployment to track the received replication data:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc replicate resync status ALIAS/BUCKET
|
||||
|
||||
The output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mc replicate resync status /data
|
||||
Resync status summary:
|
||||
● arn:minio:replication::6593d572-4dc3-4bb9-8d90-7f79cc612f01:data
|
||||
Status: Ongoing
|
||||
Replication Status | Size (Bytes) | Count
|
||||
Replicated | 2.3 GiB | 18
|
||||
Failed | 0 B | 0
|
||||
|
||||
The :guilabel:`Status` updates to ``Completed`` once the resynchronization
|
||||
process completes.
|
||||
|
||||
4) Next Steps
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
- If the ``TARGET`` bucket damage extends to replication rules, you must recreate those rules to match the previous replication configuration. See :ref:`minio-bucket-replication-serverside-twoway` for additional guidance.
|
||||
|
||||
- Perform basic validation that all buckets in the replication configuration show similar results for commands such as :mc-cmd:`mc ls` and :mc-cmd:`mc stat`.
|
||||
|
||||
- After restoring any replication rules and verifying replication between sites, you can configure the reverse proxy, load balancer, or other network control plane managing connections to resume sending traffic to the resynchronized deployment.
|
143
source/administration/identity-access-management.rst
Normal file
143
source/administration/identity-access-management.rst
Normal file
@@ -0,0 +1,143 @@
|
||||
.. _minio-authentication-and-identity-management:
|
||||
|
||||
==============================
|
||||
Identity and Access Management
|
||||
==============================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
MinIO requires the client perform both authentication and authorization for each new operation.
|
||||
|
||||
Authentication
|
||||
The process of verifying the identity of a connecting client.
|
||||
MinIO requires clients authenticate using :s3-api:`AWS Signature Version 4 protocol <sig-v4-authenticating-requests.html>` with support for the deprecated Signature Version 2 protocol.
|
||||
Specifically, clients must present a valid access key and secret key to access any S3 or MinIO administrative API, such as ``PUT``, ``GET``, and ``DELETE`` operations.
|
||||
|
||||
Authorization
|
||||
The process of restricting the actions and resources the authenticated client can perform on the deployment.
|
||||
MinIO uses Policy-Based Access Control (PBAC), where each policy describes one or more rules that outline the permissions of a user or group of users.
|
||||
MinIO supports S3-specific:ref:`actions <minio-policy-actions>` and :ref:`conditions <minio-policy-conditions>` when creating policies.
|
||||
By default, MinIO *denies* access to actions or resources not explicitly referenced in a user's assigned or inherited policies.
|
||||
|
||||
Identity Management
|
||||
-------------------
|
||||
|
||||
MinIO supports both internal and external identity management:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
:width: 100%
|
||||
|
||||
* - IDentity Provider (IDP)
|
||||
- Description
|
||||
|
||||
* - :ref:`MinIO Internal IDP <minio-internal-idp>`
|
||||
- Provides built-in identity management functionality.
|
||||
|
||||
* - :ref:`OpenID <minio-external-identity-management-openid>`
|
||||
- Supports managing identities through an OpenID Connect (OIDC) compatible
|
||||
service.
|
||||
|
||||
* - :ref:`Active Directory / LDAP
|
||||
<minio-external-identity-management-ad-ldap>`
|
||||
- Supports managing identities through an Active Directory or LDAP service.
|
||||
|
||||
Once authenticated, MinIO either allows or rejects the client request depending
|
||||
on whether or not the authenticated identity is *authorized* to perform the
|
||||
operation on the specified resource.
|
||||
|
||||
Enabling external identity management disables the MinIO internal IDP, with the exception of the creating :ref:`service accounts <minio-idp-service-account>`.
|
||||
|
||||
.. _minio-access-management:
|
||||
|
||||
Access Management
|
||||
-----------------
|
||||
|
||||
MinIO uses Policy-Based Access Control (PBAC) to define the authorized actions
|
||||
and resources to which an authenticated user has access. Each policy describes
|
||||
one or more :ref:`actions <minio-policy-actions>` and :ref:`conditions
|
||||
<minio-policy-conditions>` that outline the permissions of a
|
||||
:ref:`user <minio-users>` or :ref:`group <minio-groups>` of
|
||||
users.
|
||||
|
||||
MinIO manages the creation and storage of policies. The process for
|
||||
assigning a policy to a user or group depends on the configured
|
||||
:ref:`IDentity Provider (IDP) <minio-authentication-and-identity-management>`.
|
||||
|
||||
MinIO deployments using the :ref:`MinIO Internal IDP <minio-internal-idp>`
|
||||
require explicitly associating a user to a policy or policies using the
|
||||
:mc-cmd:`mc admin policy set` command. A user can also inherit the policies
|
||||
attached to the :ref:`groups <minio-groups>` in which they have membership.
|
||||
|
||||
By default, MinIO *denies* access to actions or resources not explicitly allowed
|
||||
by an attached or inherited policy. A user with no explicitly assigned or
|
||||
inherited policies cannot perform any S3 or MinIO administrative API operations.
|
||||
|
||||
For MinIO deployments using an External IDP, policy assignment depends on the
|
||||
choice of IDP:
|
||||
|
||||
.. list-table::
|
||||
:stub-columns: 1
|
||||
:widths: 30 70
|
||||
:width: 100%
|
||||
|
||||
* - :ref:`OpenID Connect (OIDC) <minio-external-identity-management-openid>`
|
||||
- MinIO checks for a JSON Web Token (JWT) claim (``policy`` by default)
|
||||
containing the name of the policy or policies to attach to the
|
||||
authenticated user. If the policies do not exist, the user cannot
|
||||
perform any action on the MinIO deployment.
|
||||
|
||||
MinIO does not support assigning OIDC user identities to
|
||||
:ref:`groups <minio-groups>`. The IDP administrator must instead
|
||||
assign all necessary policies to the user's policy claim.
|
||||
|
||||
See :ref:`Access Control for Externally Managed Identities
|
||||
<minio-external-identity-management-openid-access-control>` for
|
||||
more information.
|
||||
|
||||
* - :ref:`Active Directory / LDAP (AD/LDAP)
|
||||
<minio-external-identity-management-ad-ldap>`
|
||||
- MinIO checks for a policy whose name matches the Distinguished Name (DN)
|
||||
of the authenticated AD/LDAP user.
|
||||
|
||||
MinIO also supports querying for the authenticated AD/LDAP user's
|
||||
group memberships. MinIO assigns any policy whose name matches the
|
||||
DN for each returned group.
|
||||
|
||||
If no policies match either the user DN *or* any of the user's group DNs,
|
||||
the user cannot perform any action on the MinIO deployment.
|
||||
|
||||
See :ref:`Access Control for Externally Managed Identities
|
||||
<minio-external-identity-management-ad-ldap-access-control>` for more
|
||||
information.
|
||||
|
||||
MinIO PBAC is built for compatibility with AWS IAM policy syntax, structure, and
|
||||
behavior. The MinIO documentation makes a best-effort to cover IAM-specific
|
||||
behavior and functionality. Consider deferring to the :iam-docs:`IAM
|
||||
documentation <>` for more complete documentation on IAM, IAM policies, or IAM
|
||||
JSON syntax.
|
||||
|
||||
.. admonition:: ``Deny`` overrides ``Allow``
|
||||
:class: note
|
||||
|
||||
MinIO follows AWS IAM policy evaluation rules where a ``Deny`` rule overrides
|
||||
``Allow`` rule on the same action/resource. For example, if a user has an
|
||||
explicitly assigned policy with an ``Allow`` rule for an action/resource
|
||||
while one of its groups has an assigned policy with a ``Deny`` rule for that
|
||||
action/resource, MinIO would apply only the ``Deny`` rule.
|
||||
|
||||
For more information on IAM policy evaluation logic, see the IAM documentation on :iam-docs:`Determining Whether a Request is Allowed or Denied Within an Account <reference_policies_evaluation-logic.html#policy-eval-denyallow>`.
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
:hidden:
|
||||
|
||||
/administration/identity-access-management/minio-identity-management
|
||||
/administration/identity-access-management/oidc-access-management
|
||||
/administration/identity-access-management/ad-ldap-access-management
|
||||
/administration/identity-access-management/policy-based-access-control
|
@@ -0,0 +1,103 @@
|
||||
.. _minio-external-identity-management-ad-ldap:
|
||||
|
||||
=========================================
|
||||
Active Directory / LDAP Access Management
|
||||
=========================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
MinIO supports using an Active Directory or LDAP (AD/LDAP) service for external
|
||||
management of user identities.
|
||||
|
||||
For identities managed by the external AD/LDAP provider, MinIO uses the user's Distinguished Name and attempts to map it against an existing :ref:`policy <minio-policy>`.
|
||||
|
||||
If the AD/LDAP configuration includes the necessary settings to query the user's AD/LDAP group membership, MinIO *also* uses those group Distinguished Names and attempts to map each against an existing :ref:`policy <minio-policy>`.
|
||||
|
||||
MinIO by default denies access to all actions or resources not explicitly allowed by a user's assigned or inherited :ref:`policies <minio-policy>`.
|
||||
Users managed by an AD/LDAP provider must specify the necessary policies as part of the user profile data.
|
||||
If no policies match either the user DN or group DNs, MinIO blocks all access to actions and resources on the deployment.
|
||||
|
||||
The specific AD/LDAP queries MinIO issues to authenticate the user and retrieve it's group membership are configured as part of :ref:`deploying the cluster with Active Directory / LDAP identity management <minio-external-iam-ad-ldap>`.
|
||||
This page covers creation of MinIO policies to match the possible returned Distinguished Names.
|
||||
|
||||
Authentication and Authorization Flow
|
||||
-------------------------------------
|
||||
|
||||
The login flow for an application using Active Directory / LDAP
|
||||
credentials is as follows:
|
||||
|
||||
1. Specify the AD/LDAP credentials to the MinIO Security Token Service (STS)
|
||||
:ref:`minio-sts-assumerolewithldapidentity` API endpoint.
|
||||
|
||||
2. MinIO verifies the provided credentials against the AD/LDAP server.
|
||||
|
||||
3. MinIO checks for any :ref:`policy <minio-policy>` whose name matches the
|
||||
user Distinguished Name (DN) and assigns that policy to the authenticated
|
||||
user.
|
||||
|
||||
If configured to perform group queries, MinIO also queries for a list of
|
||||
AD/LDAP groups in which the user has membership. MinIO checks for any policy
|
||||
whose name matches a returned group DN and assigns that
|
||||
policy to the authenticated user.
|
||||
|
||||
4. MinIO returns temporary credentials in the STS API response in the form of an
|
||||
access key, secret key, and session token. The credentials have permissions
|
||||
matching those policies whose name matches either the authenticated user DN
|
||||
*or* a group DN.
|
||||
|
||||
MinIO provides an example Go application
|
||||
:minio-git:`ldap.go <minio/blob/master/docs/sts/ldap.go>` that handles the
|
||||
full login flow.
|
||||
|
||||
AD/LDAP users can alternatively create :ref:`service accounts <minio-idp-service-account>` associated to their AD/LDAP user Distinguished Name. Service accounts are long-lived credentials which inherit their privileges from the parent user. The parent user can further restrict those privileges while creating the service account. Use either of the following methods to create a new service account
|
||||
|
||||
- Log into the :ref:`MinIO Console <minio-console>` using the AD/LDAP-managed user credentials. From the :guilabel:`Identity` section of the left navigation, select :guilabel:`Service Accounts` followed by the :guilabel:`Create service account +` button.
|
||||
|
||||
- Use the :mc-cmd:`mc admin user svcacct add` command to create the service account. Specify the user Distinguished Name as the username to which to associate the service account.
|
||||
|
||||
|
||||
Mapping Policies to User DN
|
||||
---------------------------
|
||||
|
||||
Consider the following policy assignments:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mc admin policy set --consoleAdmin user='cn=sisko,cn=users,dc=example,dc=com'
|
||||
mc admin policy set --readwrite,diagnostics user='cn=dax,cn=users,dc=example,dc=com'
|
||||
|
||||
- MinIO would assign an authenticated user with DN matching
|
||||
``cn=sisko,cn=users,dc=example,dc=com`` the :userpolicy:`consoleAdmin`
|
||||
policy, granting complete access to the MinIO server.
|
||||
|
||||
- MinIO would assign an authenticated user with DN matching
|
||||
``cn=dax,cn=users,dc=example,dc=com`` both the :userpolicy:`readwrite` and
|
||||
:userpolicy:`diagnostics` policies, granting general read/write access to the
|
||||
MinIO server *and* access to diagnostic administrative operations.
|
||||
|
||||
- MinIO would assign no policies to an authenticated user with DN matching
|
||||
``cn=quark,cn=users,dc=example,dc=com`` and deny all access to API operations.
|
||||
|
||||
Mapping Policies to Group DN
|
||||
----------------------------
|
||||
|
||||
Consider the following policy assignments:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mc admin policy set --consoleAdmin group='cn=ops,cn=groups,dc=example,dc=com'
|
||||
mc admin policy set --diagnostics group='cn=engineering,cn=groups,dc=example,dc=com'
|
||||
|
||||
- MinIO would assign any authenticating user with membership in the
|
||||
``cn=ops,cn=groups,dc=example,dc=com`` AD/LDAP group the
|
||||
:userpolicy:`consoleAdmin` policy, granting complete access to the MinIO
|
||||
server.
|
||||
|
||||
- MinIO would assign any authenticating user with membership in the
|
||||
``cn=engineering,cn=groups,dc=example,dc=com`` AD/LDAP group the
|
||||
:userpolicy:`diagnostics` policy, granting access to diagnostic administrative
|
||||
operations.
|
@@ -0,0 +1,60 @@
|
||||
.. _minio-groups:
|
||||
|
||||
================
|
||||
Group Management
|
||||
================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
A *group* is a collection of :ref:`users <minio-users>`. Each group
|
||||
can have one or more assigned :ref:`policies <minio-policy>`
|
||||
that explicitly list the actions and resources to which group members are
|
||||
allowed or denied access.
|
||||
|
||||
For example, consider the following groups. Each group is assigned a
|
||||
:ref:`built-in policy <minio-policy-built-in>` or supported
|
||||
:ref:`policy action <minio-policy-actions>`. Each group also has one or
|
||||
more assigned users. Each user's total set of permissions consists of their
|
||||
explicitly assigned permission *and* the inherited permissions from each of
|
||||
their assigned groups. MinIO by default *denies* access to any resource or
|
||||
operation not explicitly allowed by a user's assigned or inherited policies.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 40 40
|
||||
:width: 100%
|
||||
|
||||
* - Group
|
||||
- Policy
|
||||
- Members
|
||||
|
||||
* - ``Operations``
|
||||
- | :userpolicy:`readwrite` on ``finance`` bucket
|
||||
| :userpolicy:`readonly` on ``audit`` bucket
|
||||
|
||||
- ``john.doe``, ``jane.doe``
|
||||
|
||||
* - ``Auditing``
|
||||
- | :userpolicy:`readonly` on ``audit`` bucket
|
||||
- ``jen.doe``, ``joe.doe``
|
||||
|
||||
* - ``Admin``
|
||||
- :policy-action:`admin:*`
|
||||
- ``greg.doe``, ``jen.doe``
|
||||
|
||||
Groups provide a simplified method for managing shared permissions among
|
||||
users with common access patterns and workloads. Client's *cannot* authenticate
|
||||
to a MinIO deployment using a group as an identity.
|
||||
|
||||
|
||||
The :mc-cmd:`mc admin group` command supports the creation and management of
|
||||
groups on the MinIO deployment. See the command reference for examples of
|
||||
usage.
|
||||
|
@@ -0,0 +1,56 @@
|
||||
.. _minio-internal-idp:
|
||||
|
||||
=========================
|
||||
MinIO Identity Management
|
||||
=========================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
MinIO includes a built-in IDentity Provider (IDP) that provides core identity
|
||||
management functionality. The MinIO IDP supports creating an arbitrary number of
|
||||
long-lived users on the deployment for supporting client authentication.
|
||||
|
||||
Each user consists of a unique access key (username) and corresponding secret
|
||||
key (password). Clients must authenticate their identity by specifying both
|
||||
a valid access key (username) and the corresponding secret key (password) of
|
||||
an existing MinIO user.
|
||||
|
||||
Administrators use the :mc-cmd:`mc admin user` command to create and manage
|
||||
MinIO users. The :minio-git:`MinIO Console <console>` provides a graphical
|
||||
interface for creating users.
|
||||
|
||||
MinIO also supports creating :ref:`service accounts
|
||||
<minio-idp-service-account>`. Service accounts are child identities of an
|
||||
authenticated parent user and inherit their permissions from the parent.
|
||||
|
||||
MinIO by default denies access to all actions or resources not explicitly
|
||||
allowed by a user's assigned or inherited :ref:`policies <minio-policy>`. You
|
||||
must either explicitly assign a :ref:`policy <minio-policy>` describing the
|
||||
user's authorized actions and resources *or* assign the user to :ref:`groups
|
||||
<minio-groups>` which have associated policies. See
|
||||
:ref:`minio-access-management` for more information.
|
||||
|
||||
.. admonition:: External Identity Management
|
||||
:class: dropdown, note
|
||||
|
||||
MinIO supports external management of identities using either an
|
||||
OpenID Connect (OIDC) or Active Directory/LDAP IDentity Provider (IDP).
|
||||
For more information, see:
|
||||
|
||||
- :ref:`minio-external-identity-management-openid`
|
||||
- :ref:`minio-external-identity-management-ad-ldap`
|
||||
|
||||
Enabling external identity management disables the MinIO internal IDP, with
|
||||
the exception of creating :ref:`service accounts
|
||||
<minio-idp-service-account>`.
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
:hidden:
|
||||
|
||||
/administration/identity-access-management/minio-user-management.rst
|
||||
/administration/identity-access-management/minio-group-management.rst
|
@@ -0,0 +1,173 @@
|
||||
.. _minio-users:
|
||||
|
||||
===============
|
||||
User Management
|
||||
===============
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
A MinIO user consists of a unique access key (username) and corresponding secret
|
||||
key (password). Clients must authenticate their identity by specifying both
|
||||
a valid access key (username) and the corresponding secret key (password) of
|
||||
an existing MinIO user.
|
||||
|
||||
Each user can have one or more assigned :ref:`policies <minio-policy>` that
|
||||
explicitly list the actions and resources to which that user has access.
|
||||
Users can also inherit policies from the :ref:`groups <minio-groups>` in which
|
||||
they have membership.
|
||||
|
||||
MinIO by default denies access to all actions or resources not explicitly
|
||||
allowed by a user's assigned or inherited :ref:`policies <minio-policy>`. You
|
||||
must either explicitly assign a :ref:`policy <minio-policy>` describing the
|
||||
user's authorized actions and resources *or* assign the user to :ref:`groups
|
||||
<minio-groups>` which have associated policies. See
|
||||
:ref:`minio-access-management` for more information.
|
||||
|
||||
This page documents user management for the MinIO internal IDentity Provider
|
||||
(IDP). MinIO also external management of identities using either an
|
||||
OpenID Connect (OIDC) or Active Directory/LDAP IDentity Provider (IDP).
|
||||
For more information, see:
|
||||
|
||||
- :ref:`minio-external-identity-management-openid`
|
||||
- :ref:`minio-external-identity-management-ad-ldap`
|
||||
|
||||
Enabling external identity management disables the MinIO internal IDP, with
|
||||
the exception of creating :ref:`service accounts
|
||||
<minio-idp-service-account>`.
|
||||
|
||||
.. _minio-idp-service-account:
|
||||
|
||||
Service Accounts
|
||||
----------------
|
||||
|
||||
MinIO service accounts are child identities of an authenticated MinIO user,
|
||||
including :ref:`externally managed identities
|
||||
<minio-authentication-and-identity-management>`. Each service account inherits
|
||||
its privileges based on the :ref:`policies <minio-policy>` attached to it's
|
||||
parent user *or* those groups in which the parent user has membership. Service
|
||||
accounts also support an optional inline policy which further restricts access
|
||||
to a subset of actions and resources available to the parent user.
|
||||
|
||||
A MinIO user can generate any number of service accounts. This allows
|
||||
application owners to generate arbitrary service accounts for their applications
|
||||
without requiring action from the MinIO administrators. Since the generated
|
||||
service accounts have the same or fewer permissions as the parents,
|
||||
administrators can focus on managing the top-level parent users without
|
||||
micro-managing generated service accounts.
|
||||
|
||||
You can create service accounts using either the :ref:`MinIO Console <minio-console>` *or* by using the :mc-cmd:`mc admin user svcacct add` command.
|
||||
|
||||
.. admonition:: Service Accounts are for Programmatic Access
|
||||
:class: dropdown, note
|
||||
|
||||
Service Accounts support programmatic access by applications. You cannot
|
||||
use a Service Account to log into the MinIO Console.
|
||||
|
||||
.. _minio-users-root:
|
||||
|
||||
MinIO ``root`` User
|
||||
-------------------
|
||||
|
||||
MinIO deployments have a ``root`` user with access to all actions and resources
|
||||
on the deployment, regardless of the configured :ref:`identity manager
|
||||
<minio-authentication-and-identity-management>`. When a :mc:`minio` server first
|
||||
starts, it sets the ``root`` user credentials by checking the value of the
|
||||
following environment variables:
|
||||
|
||||
- :envvar:`MINIO_ROOT_USER`
|
||||
- :envvar:`MINIO_ROOT_PASSWORD`
|
||||
|
||||
Rotating the root user credentials requires updating either or both variables
|
||||
for all MinIO servers in the deployment. Specify *long, unique, and random*
|
||||
strings for root credentials. Exercise all possible precautions in storing the
|
||||
access key and secret key, such that only known and trusted individuals who
|
||||
*require* superuser access to the deployment can retrieve the ``root``
|
||||
credentials.
|
||||
|
||||
- MinIO *strongly discourages* using the ``root`` user for regular client access
|
||||
regardless of the environment (development, staging, or production).
|
||||
|
||||
- MinIO *strongly recommends* creating users such that each client has access to
|
||||
the minimal set of actions and resources required to perform their assigned
|
||||
workloads.
|
||||
|
||||
If these variables are unset, :mc:`minio` defaults to ``minioadmin`` and
|
||||
``minioadmin`` as the access key and secret key respectively. MinIO *strongly
|
||||
discourages* use of the default credentials regardless of deployment
|
||||
environment.
|
||||
|
||||
.. admonition:: Deprecation of Legacy Root User Environment Variables
|
||||
:class: dropdown, important
|
||||
|
||||
MinIO :minio-release:`RELEASE.2021-04-22T15-44-28Z` and later deprecates the
|
||||
following variables used for setting or updating root user
|
||||
credentials:
|
||||
|
||||
- :envvar:`MINIO_ACCESS_KEY` to the new access key.
|
||||
- :envvar:`MINIO_SECRET_KEY` to the new secret key.
|
||||
- :envvar:`MINIO_ACCESS_KEY_OLD` to the old access key.
|
||||
- :envvar:`MINIO_SECRET_KEY_OLD` to the old secret key.
|
||||
|
||||
User Management
|
||||
---------------
|
||||
|
||||
Create a User
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin user add` command to create a new user on the
|
||||
MinIO deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin user add ALIAS ACCESSKEY SECRETKEY
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc admin user add TARGET>` with the
|
||||
:mc-cmd:`alias <mc alias>` of the MinIO deployment.
|
||||
|
||||
- Replace :mc-cmd:`ACCESSKEY <mc admin user add ACCESSKEY>` with the
|
||||
access key for the user. MinIO allows retrieving the access key after
|
||||
user creation through the :mc-cmd:`mc admin user info` command.
|
||||
|
||||
- Replace :mc-cmd:`SECRETKEY <mc admin user add SECRETKEY>` with the
|
||||
secret key for the user. MinIO *does not* provide any method for retrieving
|
||||
the secret key once set.
|
||||
|
||||
Specify a unique, random, and long string for both the ``ACCESSKEY`` and
|
||||
``SECRETKEY``. Your organization may have specific internal or regulatory
|
||||
requirements around generating values for use with access or secret keys.
|
||||
|
||||
After creating the user, use :mc-cmd:`mc admin policy set` to associate a
|
||||
:ref:`MinIO Policy Based Access Control <minio-policy>` to the new user.
|
||||
The following command assigns the built-in :userpolicy:`readwrite` policy:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin policy set ALIAS readwrite user=USERNAME
|
||||
|
||||
Replace ``USERNAME`` with the ``ACCESSKEY`` created in the previous step.
|
||||
|
||||
Delete a User
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin user remove` command to remove a user on a
|
||||
MinIO deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin user remove ALIAS USERNAME
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc admin user remove TARGET>` with the
|
||||
:mc-cmd:`alias <mc alias>` of the MinIO deployment.
|
||||
|
||||
- Replace :mc-cmd:`USERNAME <mc admin user remove USERNAME>` with the name of
|
||||
the user to remove.
|
@@ -0,0 +1,82 @@
|
||||
.. _minio-external-identity-management-openid:
|
||||
.. _minio-external-identity-management-openid-access-control:
|
||||
|
||||
================================
|
||||
OpenID Connect Access Management
|
||||
================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
MinIO supports using an OpenID Connect (OIDC) compatible IDentity Provider (IDP)
|
||||
such as Okta, KeyCloak, Dex, Google, or Facebook for external management of user
|
||||
identities.
|
||||
|
||||
For identities managed by the external OpenID Connect (OIDC) compatible provider, MinIO uses the `JSON Web Token claim <https://datatracker.ietf.org/doc/html/rfc7519#section-4>`__ returned as part of the OIDC authentication flow to identify the :ref:`policies <minio-policy>` to assign to the authenticated user.
|
||||
|
||||
MinIO by default denies access to all actions or resources not explicitly allowed by a user's assigned or inherited :ref:`policies <minio-policy>`.
|
||||
Users managed by an OIDC provider must specify the necessary policies as part of the JWT claim. If the user JWT claim has no matching MinIO policies, that user has no permissions to access any action or resource on the MinIO deployment.
|
||||
|
||||
The specific claim which MinIO looks for is configured as part of :ref:`deploying the cluster with OIDC identity management <minio-external-iam-oidc>`. This page focuses on creating MinIO policies to match the configured OIDC claims.
|
||||
|
||||
Authentication and Authorization Flow
|
||||
-------------------------------------
|
||||
|
||||
The login flow for an application using :abbr:`OIDC (OpenID Connect)`
|
||||
credentials is as follows:
|
||||
|
||||
1. Authenticate to the configured :abbr:`OIDC (OpenID Connect)`
|
||||
provider and retrieve a
|
||||
`JSON Web Token (JWT) <https://jwt.io/introduction>`__.
|
||||
|
||||
MinIO only supports the
|
||||
`OpenID Authorization Code Flow
|
||||
<https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowAuth>`__.
|
||||
Authentication using Implicit Flow is not supported.
|
||||
|
||||
2. Specify the :abbr:`JWT (JSON Web Token)` to the MinIO Security Token Service
|
||||
(STS) :ref:`minio-sts-assumerolewithwebidentity` API endpoint.
|
||||
|
||||
MinIO verifies the :abbr:`JWT (JSON Web Token)` against the
|
||||
configured OIDC provider.
|
||||
|
||||
If the JWT is valid, MinIO checks for a :ref:`claim
|
||||
<minio-external-identity-management-openid-access-control>` specifying a list
|
||||
of one or more :ref:`policies <minio-policy>` to assign to the
|
||||
authenticated user. MinIO defaults to checking the ``policy`` claim.
|
||||
|
||||
3. MinIO returns temporary credentials in the STS API response in the form of an
|
||||
access key, secret key, and session token. The credentials have
|
||||
permissions matching those policies specified in the JWT claim.
|
||||
|
||||
4. Applications use the temporary credentials returned by the STS endpoint to
|
||||
perform authenticated S3 operations on MinIO.
|
||||
|
||||
MinIO provides an example Go application
|
||||
:minio-git:`web-identity.go <minio/blob/master/docs/sts/web-identity.go>` that
|
||||
handles the full login flow.
|
||||
|
||||
OIDC users can alternatively create :ref:`service accounts <minio-idp-service-account>` associated to their AD/LDAP user. Service accounts are long-lived credentials which inherit their privileges from the parent user. The parent user can further restrict those privileges while creating the service account. To create a new service account, log into the :ref:`MinIO Console <minio-console>` using the OIDC-managed user credentials. From the :guilabel:`Identity` section of the left navigation, select :guilabel:`Service Accounts` followed by the :guilabel:`Create service account +` button.
|
||||
|
||||
Identifying the JWT Claim Value
|
||||
-------------------------------
|
||||
|
||||
MinIO uses the JWT token returned as part of the OIDC authentication flow to identify the specific policies to assign to the authenticated user.
|
||||
|
||||
You can use a `JWT Debugging tool <https://jwt.io/>`__ to decode the returned JWT token and validate that the user attributes include the required claims.
|
||||
|
||||
.. todo - example JWT claim
|
||||
|
||||
See `RFC 7519: JWT Claim <https://datatracker.ietf.org/doc/html/rfc7519#section-4>`__ for more information on JWT claims.
|
||||
|
||||
Defer to the documentation for your preferred OIDC provider for instructions on configuring user claims.
|
||||
|
||||
Creating Policies to Match Claims
|
||||
---------------------------------
|
||||
|
||||
Use either the MinIO Console *or* the :mc-cmd:`mc admin policy` command to create policies that match one or more claim values:
|
||||
|
||||
.. todo - instructions
|
@@ -0,0 +1,778 @@
|
||||
.. _minio-policy:
|
||||
|
||||
=================
|
||||
Access Management
|
||||
=================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
MinIO uses Policy-Based Access Control (PBAC) to define the authorized actions
|
||||
and resources to which an authenticated user has access. Each policy describes
|
||||
one or more :ref:`actions <minio-policy-actions>` and :ref:`conditions
|
||||
<minio-policy-conditions>` that outline the permissions of a
|
||||
:ref:`user <minio-users>` or :ref:`group <minio-groups>` of
|
||||
users.
|
||||
|
||||
MinIO PBAC is built for compatibility with AWS IAM policy syntax, structure, and
|
||||
behavior. The MinIO documentation makes a best-effort to cover IAM-specific
|
||||
behavior and functionality. Consider deferring to the :iam-docs:`IAM
|
||||
documentation <>` for more complete documentation on AWS IAM-specific topics.
|
||||
|
||||
The :mc-cmd:`mc admin policy` command supports creation and management of
|
||||
policies on the MinIO deployment. See the command reference for examples of
|
||||
usage.
|
||||
|
||||
.. _minio-policy-built-in:
|
||||
|
||||
Built-In Policies
|
||||
-----------------
|
||||
|
||||
MinIO provides the following built-in policies for assigning to
|
||||
:ref:`users <minio-users>` or :ref:`groups <minio-groups>`:
|
||||
|
||||
.. userpolicy:: consoleAdmin
|
||||
|
||||
Grants complete access to all S3 and administrative API operations against
|
||||
all resources on the MinIO deployment. Equivalent to the following set of
|
||||
actions:
|
||||
|
||||
- :policy-action:`s3:*`
|
||||
- :policy-action:`admin:*`
|
||||
|
||||
.. userpolicy:: readonly
|
||||
|
||||
Grants read-only permissions on any object on the MinIO deployment. The GET
|
||||
action *must* apply to a specific object without requiring any listing.
|
||||
Equivalent to the following set of actions:
|
||||
|
||||
- :policy-action:`s3:GetBucketLocation`
|
||||
- :policy-action:`s3:GetObject`
|
||||
|
||||
For example, this policy specifically supports GET operations on objects at a
|
||||
specific path (e.g. ``GET play/mybucket/object.file``), such as:
|
||||
|
||||
- :mc-cmd:`mc cp`
|
||||
- :mc-cmd:`mc stat`
|
||||
- :mc-cmd:`mc head`
|
||||
- :mc-cmd:`mc cat`
|
||||
|
||||
The exclusion of listing permissions is intentional, as typical use cases
|
||||
do not intend for a "read-only" role to have complete discoverability
|
||||
(listing all buckets and objects) on the object storage resource.
|
||||
|
||||
.. userpolicy:: readwrite
|
||||
|
||||
Grants read and write permissions for all buckets and objects on the
|
||||
MinIO server. Equivalent to :policy-action:`s3:*`.
|
||||
|
||||
.. userpolicy:: diagnostics
|
||||
|
||||
Grants permission to perform diagnostic actions on the MinIO deployment.
|
||||
Specifically includes the following actions:
|
||||
|
||||
- :policy-action:`admin:ServerTrace`
|
||||
- :policy-action:`admin:Profiling`
|
||||
- :policy-action:`admin:ConsoleLog`
|
||||
- :policy-action:`admin:ServerInfo`
|
||||
- :policy-action:`admin:TopLocksInfo`
|
||||
- :policy-action:`admin:OBDInfo`
|
||||
- :policy-action:`admin:BandwidthMonitor`
|
||||
- :policy-action:`admin:Prometheus`
|
||||
|
||||
.. userpolicy:: writeonly
|
||||
|
||||
Grants write-only permissions to any namespace (bucket and path to object)
|
||||
the MinIO deployment. The PUT action *must* apply to a specific object
|
||||
location without requiring any listing.
|
||||
Equivalent to the :policy-action:`s3:PutObject` action.
|
||||
|
||||
Use :mc-cmd:`mc admin policy set` to associate a policy to a
|
||||
user or group on a MinIO deployment.
|
||||
|
||||
For example, consider the following table of users. Each user is assigned
|
||||
a :ref:`built-in policy <minio-policy-built-in>` or
|
||||
a supported :ref:`action <minio-policy-actions>`. The table
|
||||
describes a subset of operations a client could perform if authenticated
|
||||
as that user:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 40 40
|
||||
:width: 100%
|
||||
|
||||
* - User
|
||||
- Policy
|
||||
- Operations
|
||||
|
||||
* - ``Operations``
|
||||
- | :userpolicy:`readwrite` on ``finance`` bucket
|
||||
| :userpolicy:`readonly` on ``audit`` bucket
|
||||
|
||||
- | ``PUT`` and ``GET`` on ``finance`` bucket.
|
||||
| ``PUT`` on ``audit`` bucket
|
||||
|
||||
* - ``Auditing``
|
||||
- | :userpolicy:`readonly` on ``audit`` bucket
|
||||
- ``GET`` on ``audit`` bucket
|
||||
|
||||
* - ``Admin``
|
||||
- :policy-action:`admin:*`
|
||||
- All :mc-cmd:`mc admin` commands.
|
||||
|
||||
Each user can access only those resources and operations which are *explicitly*
|
||||
granted by the built-in role. MinIO denies access to any other resource or
|
||||
action by default.
|
||||
|
||||
.. admonition:: ``Deny`` overrides ``Allow``
|
||||
:class: note
|
||||
|
||||
MinIO follows the IAM policy evaluation rules where a ``Deny`` rule overrides
|
||||
``Allow`` rule on the same action/resource. For example, if a user has an
|
||||
explicitly assigned policy with an ``Allow`` rule for an action/resource
|
||||
while one of its groups has an assigned policy with a ``Deny`` rule for that
|
||||
action/resource, MinIO would apply only the ``Deny`` rule.
|
||||
|
||||
For more information on IAM policy evaluation logic, see the IAM
|
||||
documentation on
|
||||
:iam-docs:`Determining Whether a Request is Allowed or Denied Within an Account
|
||||
<reference_policies_evaluation-logic.html#policy-eval-denyallow>`.
|
||||
|
||||
.. _minio-policy-document:
|
||||
|
||||
Policy Document Structure
|
||||
-------------------------
|
||||
|
||||
MinIO policy documents use the same schema as
|
||||
:aws-docs:`AWS IAM Policy <IAM/latest/UserGuide/access.html>` documents.
|
||||
|
||||
The following sample document provides a template for creating custom
|
||||
policies for use with a MinIO deployment. For more complete documentation on IAM
|
||||
policy elements, see the :aws-docs:`IAM JSON Policy Elements Reference
|
||||
<IAM/latest/UserGuide/reference_policies_elements.html>`.
|
||||
|
||||
.. code-block:: javascript
|
||||
:class: copyable
|
||||
|
||||
{
|
||||
"Version" : "2012-10-17",
|
||||
"Statement" : [
|
||||
{
|
||||
"Effect" : "Allow",
|
||||
"Action" : [ "s3:<ActionName>", ... ],
|
||||
"Resource" : "arn:aws:s3:::*",
|
||||
"Condition" : { ... }
|
||||
},
|
||||
{
|
||||
"Effect" : "Deny",
|
||||
"Action" : [ "s3:<ActionName>", ... ],
|
||||
"Resource" : "arn:aws:s3:::*",
|
||||
"Condition" : { ... }
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
- For the ``Statement.Action`` array, specify one or more
|
||||
:ref:`supported S3 API operations <minio-policy-actions>`. MinIO deployments
|
||||
supports a subset of AWS S3 API operations.
|
||||
|
||||
- For the ``Statement.Resource`` key, you can replace the ``*`` with
|
||||
the specific bucket to which the policy statement should apply.
|
||||
Using ``*`` applies the statement to all resources on the MinIO deployment.
|
||||
|
||||
- For the ``Statement.Condition`` key, you can specify one or more
|
||||
:ref:`supported Conditions <minio-policy-conditions>`. MinIO
|
||||
deployments supports a subset of AWS S3 conditions.
|
||||
|
||||
.. _minio-policy-actions:
|
||||
|
||||
Supported S3 Policy Actions
|
||||
---------------------------
|
||||
|
||||
MinIO policy documents support a subset of IAM
|
||||
:iam-docs:`S3 Action keys <list_amazons3.html#amazons3-actions-as-permissions>`.
|
||||
|
||||
The following actions control access to common S3 operations. The remaining
|
||||
subsections document actions for more advanced S3 operations:
|
||||
|
||||
.. policy-action:: s3:*
|
||||
|
||||
Selector for *all* MinIO S3 operations. Applying this action to a given
|
||||
resource allows the user to perform *any* S3 operation against that
|
||||
resource.
|
||||
|
||||
.. policy-action:: s3:CreateBucket
|
||||
|
||||
Controls access to the :s3-api:`CreateBucket <API_CreateBucket.html>` S3 API
|
||||
operation.
|
||||
|
||||
.. policy-action:: s3:DeleteBucket
|
||||
|
||||
Controls access to the :s3-api:`DeleteBucket <API_DeleteBucket.html>` S3 API
|
||||
operation.
|
||||
|
||||
.. policy-action:: s3:ForceDeleteBucket
|
||||
|
||||
Controls access to the :s3-api:`DeleteBucket <API_DeleteBucket.html>`
|
||||
S3 API operation for operations with the ``x-minio-force-delete`` flag.
|
||||
Required for removing non-empty buckets.
|
||||
|
||||
.. policy-action:: s3:GetBucketLocation
|
||||
|
||||
Controls access to the :s3-api:`GetBucketLocation
|
||||
<API_GetBucketLocation.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:ListAllMyBuckets
|
||||
|
||||
Controls access to the :s3-api:`ListBuckets <API_ListBuckets.html>`
|
||||
S3 API operation.
|
||||
|
||||
.. policy-action:: s3:DeleteObject
|
||||
|
||||
Controls access to the :s3-api:`DeleteObject <API_DeleteObject.html>` S3 API
|
||||
operation.
|
||||
|
||||
.. policy-action:: s3:GetObject
|
||||
|
||||
Controls access to the :s3-api:`GetObject <API_GetObject.html>` S3 API
|
||||
operation.
|
||||
|
||||
.. policy-action:: s3:ListBucket
|
||||
|
||||
Controls access to the :s3-api:`ListObjectsV2 <API_ListObjectsV2.html>` S3 API
|
||||
operation.
|
||||
|
||||
.. policy-action:: s3:PutObject
|
||||
|
||||
Controls access to the :s3-api:`PutObject <API_PutObject.html>` S3 API
|
||||
operation.
|
||||
|
||||
.. policy-action:: s3:PutObjectTagging
|
||||
|
||||
Controls access to the :s3-api:`PutObjectTagging <API_PutObjectTagging.html>`
|
||||
S3 API operation.
|
||||
|
||||
.. policy-action:: s3:GetObjectTagging
|
||||
|
||||
Controls access to the :s3-api:`GetObjectTagging <API_GetObjectTagging.html>`
|
||||
S3 API operation.
|
||||
|
||||
Bucket Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. policy-action:: s3:GetBucketPolicy
|
||||
|
||||
Controls access to the :s3-api:`GetBucketPolicy <API_GetBucketPolicy.html>`
|
||||
S3 API operation.
|
||||
|
||||
.. policy-action:: s3:PutBucketPolicy
|
||||
|
||||
Controls access to the :s3-api:`PutBucketPolicy <API_PutBucketPolicy.html>`
|
||||
S3 API operation.
|
||||
|
||||
.. policy-action:: s3:DeleteBucketPolicy
|
||||
|
||||
Controls access to the :s3-api:`DeleteBucketPolicy
|
||||
<API_DeleteBucketPolicy.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:GetBucketTagging
|
||||
|
||||
Controls access to the :s3-api:`GetBucketTagging <API_GetBucketTagging.html>`
|
||||
S3 API operation.
|
||||
|
||||
.. policy-action:: s3:PutBucketTagging
|
||||
|
||||
Controls access to the :s3-api:`PutBucketTagging <API_PutBucketTagging.html>`
|
||||
S3 API operation.
|
||||
|
||||
Multipart Upload
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
.. policy-action:: s3:AbortMultipartUpload
|
||||
|
||||
Controls access to the :s3-api:`AbortMultipartUpload
|
||||
<API_AbortMultipartUpload.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:ListMultipartUploadParts
|
||||
|
||||
Controls access to the :s3-api:`ListParts <API_ListParts.html>` S3 API
|
||||
operation.
|
||||
|
||||
.. policy-action:: s3:ListBucketMultipartUploads
|
||||
|
||||
Controls access to the :s3-api:`ListMultipartUploads
|
||||
<API_ListMultipartUploads.html>` S3 API operation.
|
||||
|
||||
Versioning and Retention
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. policy-action:: s3:PutBucketVersioning
|
||||
|
||||
Controls access to the :s3-api:`PutBucketVersioning
|
||||
<API_PutBucketVersioning.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:GetBucketVersioning
|
||||
|
||||
Controls access to the :s3-api:`GetBucketVersioning
|
||||
<API_GetBucketVersioning.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:DeleteObjectVersion
|
||||
|
||||
Controls access to the :s3-api:`DeleteObjectVersion
|
||||
<API_DeleteObjectVersion.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:DeleteObjectVersionTagging
|
||||
|
||||
Controls access to the :s3-api:`DeleteObjectVersionTagging
|
||||
<API_DeleteObjectVersionTagging.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:GetObjectVersion
|
||||
|
||||
Controls access to the :s3-api:`GetObjectVersion
|
||||
<API_GetObjectVersion.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:BypassGovernanceRetention
|
||||
|
||||
Controls access to the following S3 API operations on objects
|
||||
locked under :mc-cmd:`GOVERNANCE <mc retention set MODE>`
|
||||
retention mode:
|
||||
|
||||
- ``PutObjectRetention``
|
||||
- ``PutObject``
|
||||
- ``DeleteObject``
|
||||
|
||||
See the S3 documentation on :s3-docs:`s3:BypassGovernanceRetention
|
||||
<object-lock-managing.html#object-lock-managing-bypass>` for more
|
||||
information.
|
||||
|
||||
.. policy-action:: s3:PutObjectRetention
|
||||
|
||||
Controls access to the :s3-api:`PutObjectRetention
|
||||
<API_PutObjectRetention.html>` S3 API operation.
|
||||
|
||||
Required for any ``PutObject`` operation that specifies
|
||||
:ref:`retention metadata <minio-object-locking>`.
|
||||
|
||||
.. policy-action:: s3:GetObjectRetention
|
||||
|
||||
Controls access to the :s3-api:`GetObjectRetention
|
||||
<API_GetObjectRetention.html>` S3 API operation.
|
||||
|
||||
Required for including :ref:`object locking metadata <minio-object-locking>`
|
||||
as part of the response to a ``GetObject`` or ``HeadObject`` operation.
|
||||
|
||||
.. policy-action:: s3:GetObjectLegalHold
|
||||
|
||||
Controls access to the :s3-api:`GetObjectLegalHold
|
||||
<API_GetObjectLegalHold.html>` S3 API operation.
|
||||
|
||||
Required for including :ref:`object locking metadata <minio-object-locking>`
|
||||
as part of the response to a ``GetObject`` or ``HeadObject`` operation.
|
||||
|
||||
.. policy-action:: s3:PutObjectLegalHold
|
||||
|
||||
Controls access to the :s3-api:`PutObjectLegalHold
|
||||
<API_PutObjectLegalHold.html>` S3 API operation.
|
||||
|
||||
Required for any ``PutObject`` operation that specifies
|
||||
:ref:`legal hold metadata <minio-object-locking>`.
|
||||
|
||||
.. policy-action:: s3:GetBucketObjectLockConfiguration
|
||||
|
||||
Controls access to the :s3-api:`GetObjectLockConfiguration
|
||||
<API_GetObjectLockConfiguration.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:PutBucketObjectLockConfiguration
|
||||
|
||||
Controls access to the :s3-api:`PutObjectLockConfiguration
|
||||
<API_PutObjectLockConfiguration.html>` S3 API operation.
|
||||
|
||||
Bucket Notifications
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. policy-action:: s3:GetBucketNotification
|
||||
|
||||
Controls access to the :s3-api:`GetBucketNotification
|
||||
<API_GetBucketNotification.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:PutBucketNotification
|
||||
|
||||
Controls access to the :s3-api:`PutBucketNotification
|
||||
<API_PutBucketNotification.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:ListenNotification
|
||||
|
||||
MinIO Extension for controlling API operations related to MinIO Bucket
|
||||
Notifications.
|
||||
|
||||
This action is **not** intended for use with other S3-compatible services.
|
||||
|
||||
.. policy-action:: s3:ListenBucketNotification
|
||||
|
||||
MinIO Extension for controlling API operations related to MinIO Bucket
|
||||
Notifications.
|
||||
|
||||
This action is **not** intended for use with other S3-compatible services.
|
||||
|
||||
Object Lifecycle Management
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. policy-action:: s3:PutLifecycleConfiguration
|
||||
|
||||
Controls access to the :s3-api:`PutLifecycleConfiguration
|
||||
<API_PutBucketLifecycleConfiguration.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:GetLifecycleConfiguration
|
||||
|
||||
Controls access to the :s3-api:`GetLifecycleConfiguration
|
||||
<API_GetBucketLifecycleConfiguration.html>` S3 API operation.
|
||||
|
||||
Object Encryption
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. policy-action:: s3:PutEncryptionConfiguration
|
||||
|
||||
Controls access to the :s3-api:`PutEncryptionConfiguration
|
||||
<API_PutBucketEncryption.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:GetEncryptionConfiguration
|
||||
|
||||
Controls access to the :s3-api:`GetEncryptionConfiguration
|
||||
<API_GetBucketEncryption.html>` S3 API operation.
|
||||
|
||||
Bucket Replication
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. policy-action:: s3:GetReplicationConfiguration
|
||||
|
||||
Controls access to the :s3-api:`GetBucketReplication
|
||||
<API_GetBucketReplication.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:PutReplicationConfiguration
|
||||
|
||||
Controls access to the :s3-api:`PutBucketReplication
|
||||
<PutBucketReplication.html>` S3 API operation.
|
||||
|
||||
.. policy-action:: s3:ReplicateObject
|
||||
|
||||
MinIO Extension for controlling API operations related to
|
||||
:ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`.
|
||||
|
||||
Required for server-side replication.
|
||||
|
||||
.. policy-action:: s3:ReplicateDelete
|
||||
|
||||
MinIO Extension for controlling API operations related to
|
||||
:ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`.
|
||||
|
||||
Required for synchronizing delete operations as part of server-side
|
||||
replication.
|
||||
|
||||
.. policy-action:: s3:ReplicateTags
|
||||
|
||||
MinIO Extension for controlling API operations related to
|
||||
:ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`.
|
||||
|
||||
Required for server-side replication.
|
||||
|
||||
.. policy-action:: s3:GetObjectVersionForReplication
|
||||
|
||||
MinIO Extension for controlling API operations related to
|
||||
:ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`.
|
||||
|
||||
Required for server-side replication.
|
||||
|
||||
.. _minio-policy-conditions:
|
||||
|
||||
Supported S3 Policy Condition Keys
|
||||
----------------------------------
|
||||
|
||||
MinIO policy documents support IAM
|
||||
:iam-docs:`conditional statements <reference_policies_elements_condition.html>`.
|
||||
|
||||
Each condition element consists of
|
||||
:iam-docs:`operators <reference_policies_elements_condition_operators.html>`
|
||||
and condition keys. MinIO supports a subset of IAM condition keys. For complete
|
||||
information on any listed condition key, see the
|
||||
:iam-docs:`IAM Condition Element Documentation
|
||||
<reference_policies_elements_condition.html>`
|
||||
|
||||
MinIO supports the following condition keys for all supported
|
||||
:ref:`actions <minio-policy-actions>`:
|
||||
|
||||
- ``aws:Referer``
|
||||
- ``aws:SourceIp``
|
||||
- ``aws:UserAgent``
|
||||
- ``aws:SecureTransport``
|
||||
- ``aws:CurrentTime``
|
||||
- ``aws:EpochTime``
|
||||
- ``aws:PrincipalType``
|
||||
- ``aws:userid``
|
||||
- ``aws:username``
|
||||
- ``x-amz-content-sha256``
|
||||
|
||||
The following table lists additional supported condition keys for specific
|
||||
actions:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
:width: 100%
|
||||
|
||||
* - Action Key
|
||||
- Condition Keys
|
||||
|
||||
* - :policy-action:`s3:GetObject`
|
||||
- | ``x-amz-server-side-encryption``
|
||||
| ``x-amz-server-side-encryption-customer-algorithm``
|
||||
|
||||
* - :policy-action:`s3:ListBucket`
|
||||
- | ``prefix``
|
||||
| ``delimiter``
|
||||
| ``max-keys``
|
||||
|
||||
* - :policy-action:`s3:PutObject`
|
||||
- | ``x-amz-copy-source``
|
||||
| ``x-amz-server-side-encryption``
|
||||
| ``x-amz-server-side-encryption-customer-algorithm``
|
||||
| ``x-amz-metadata-directive``
|
||||
| ``x-amz-storage-class``
|
||||
| ``object-lock-retain-until-date``
|
||||
| ``object-lock-mode``
|
||||
| ``object-lock-legal-hold``
|
||||
|
||||
* - :policy-action:`s3:PutObjectRetention`
|
||||
- | ``x-amz-object-lock-remaining-retention-days``
|
||||
| ``x-amz-object-lock-retain-until-date``
|
||||
| ``x-amz-object-lock-mode``
|
||||
|
||||
* - :policy-action:`s3:PutObjectLegalHold`
|
||||
- ``object-lock-legal-hold``
|
||||
|
||||
* - :policy-action:`s3:BypassGovernanceRetention`
|
||||
- | ``object-lock-remaining-retention-days``
|
||||
| ``object-lock-retain-until-date``
|
||||
| ``object-lock-mode``
|
||||
| ``object-lock-legal-hold``
|
||||
|
||||
* - :policy-action:`s3:GetObjectVersion`
|
||||
- ``versionid``
|
||||
|
||||
|
||||
* - :policy-action:`s3:DeleteObjectVersion`
|
||||
- ``versionid``
|
||||
|
||||
.. _minio-policy-mc-admin-actions:
|
||||
|
||||
``mc admin`` Policy Action Keys
|
||||
-------------------------------
|
||||
|
||||
MinIO supports the following actions for use with defining policies
|
||||
for :mc-cmd:`mc admin` operations. These actions are *only* valid for
|
||||
MinIO deployments and are *not* intended for use with other S3-compatible
|
||||
services:
|
||||
|
||||
.. policy-action:: admin:*
|
||||
|
||||
Selector for all admin action keys.
|
||||
|
||||
.. policy-action:: admin:Heal
|
||||
|
||||
Allows heal command
|
||||
|
||||
.. policy-action:: admin:StorageInfo
|
||||
|
||||
Allows listing server info
|
||||
|
||||
.. policy-action:: admin:DataUsageInfo
|
||||
|
||||
Allows listing data usage info
|
||||
|
||||
.. policy-action:: admin:TopLocksInfo
|
||||
|
||||
Allows listing top locks
|
||||
|
||||
.. policy-action:: admin:Profiling
|
||||
|
||||
Allows profiling
|
||||
|
||||
.. policy-action:: admin:ServerTrace
|
||||
|
||||
Allows listing server trace
|
||||
|
||||
.. policy-action:: admin:ConsoleLog
|
||||
|
||||
Allows listing console logs on terminal
|
||||
|
||||
.. policy-action:: admin:KMSCreateKey
|
||||
|
||||
Allows creating a new KMS master key
|
||||
|
||||
.. policy-action:: admin:KMSKeyStatus
|
||||
|
||||
Allows getting KMS key status
|
||||
|
||||
.. policy-action:: admin:ServerInfo
|
||||
|
||||
Allows listing server info
|
||||
|
||||
.. policy-action:: admin:OBDInfo
|
||||
|
||||
Allows obtaining cluster on-board diagnostics
|
||||
|
||||
.. policy-action:: admin:ServerUpdate
|
||||
|
||||
Allows MinIO binary update
|
||||
|
||||
.. policy-action:: admin:ServiceRestart
|
||||
|
||||
Allows restart of MinIO service.
|
||||
|
||||
.. policy-action:: admin:ServiceStop
|
||||
|
||||
Allows stopping MinIO service.
|
||||
|
||||
.. policy-action:: admin:ConfigUpdate
|
||||
|
||||
Allows MinIO config management
|
||||
|
||||
.. policy-action:: admin:CreateUser
|
||||
|
||||
Allows creating MinIO user
|
||||
|
||||
.. policy-action:: admin:DeleteUser
|
||||
|
||||
Allows deleting MinIO user
|
||||
|
||||
.. policy-action:: admin:ListUsers
|
||||
|
||||
Allows list users permission
|
||||
|
||||
.. policy-action:: admin:EnableUser
|
||||
|
||||
Allows enable user permission
|
||||
|
||||
.. policy-action:: admin:DisableUser
|
||||
|
||||
Allows disable user permission
|
||||
|
||||
.. policy-action:: admin:GetUser
|
||||
|
||||
Allows GET permission on user info
|
||||
|
||||
.. policy-action:: admin:AddUserToGroup
|
||||
|
||||
Allows adding user to group permission
|
||||
|
||||
.. policy-action:: admin:RemoveUserFromGroup
|
||||
|
||||
Allows removing user to group permission
|
||||
|
||||
.. policy-action:: admin:GetGroup
|
||||
|
||||
Allows getting group info
|
||||
|
||||
.. policy-action:: admin:ListGroups
|
||||
|
||||
Allows list groups permission
|
||||
|
||||
.. policy-action:: admin:EnableGroup
|
||||
|
||||
Allows enable group permission
|
||||
|
||||
.. policy-action:: admin:DisableGroup
|
||||
|
||||
Allows disable group permission
|
||||
|
||||
.. policy-action:: admin:CreatePolicy
|
||||
|
||||
Allows create policy permission
|
||||
|
||||
.. policy-action:: admin:DeletePolicy
|
||||
|
||||
Allows delete policy permission
|
||||
|
||||
.. policy-action:: admin:GetPolicy
|
||||
|
||||
Allows get policy permission
|
||||
|
||||
.. policy-action:: admin:AttachUserOrGroupPolicy
|
||||
|
||||
Allows attaching a policy to a user/group
|
||||
|
||||
.. policy-action:: admin:ListUserPolicies
|
||||
|
||||
Allows listing user policies
|
||||
|
||||
.. policy-action:: admin:CreateServiceAccount
|
||||
|
||||
Allows creating MinIO Service Account
|
||||
|
||||
.. policy-action:: admin:UpdateServiceAccount
|
||||
|
||||
Allows updating MinIO Service Account
|
||||
|
||||
.. policy-action:: admin:RemoveServiceAccount
|
||||
|
||||
Allows deleting MinIO Service Account
|
||||
|
||||
.. policy-action:: admin:ListServiceAccounts
|
||||
|
||||
Allows listing MinIO Service Account
|
||||
|
||||
.. policy-action:: admin:SetBucketQuota
|
||||
|
||||
Allows setting bucket quota
|
||||
|
||||
.. policy-action:: admin:GetBucketQuota
|
||||
|
||||
Allows getting bucket quota
|
||||
|
||||
.. policy-action:: admin:SetBucketTarget
|
||||
|
||||
Allows setting bucket target
|
||||
|
||||
.. policy-action:: admin:GetBucketTarget
|
||||
|
||||
Allows getting bucket targets
|
||||
|
||||
.. policy-action:: admin:SetTier
|
||||
|
||||
Allows creating and modifying remote storage tiers using the
|
||||
:mc-cmd:`mc admin tier` command.
|
||||
|
||||
.. policy-action:: admin:ListTier
|
||||
|
||||
Allows listing configured remote storage tiers using the
|
||||
:mc-cmd:`mc admin tier` command.
|
||||
|
||||
.. policy-action:: admin:BandwidthMonitor
|
||||
|
||||
Allows retrieving metrics related to current bandwidth consumption.
|
||||
|
||||
.. policy-action:: admin:Prometheus
|
||||
|
||||
Allows access to MinIO :ref:`metrics <minio-metrics-and-alerts-endpoints>`.
|
||||
Only required if MinIO requires authentication for scraping metrics.
|
||||
|
||||
``mc admin`` Policy Condition Keys
|
||||
----------------------------------
|
||||
|
||||
MinIO supports the following conditions for use with defining policies for
|
||||
:mc-cmd:`mc admin` :ref:`actions <minio-policy-mc-admin-actions>`.
|
||||
|
||||
- ``aws:Referer``
|
||||
- ``aws:SourceIp``
|
||||
- ``aws:UserAgent``
|
||||
- ``aws:SecureTransport``
|
||||
- ``aws:CurrentTime``
|
||||
- ``aws:EpochTime``
|
||||
|
||||
For complete information on any listed condition key, see the :iam-docs:`IAM
|
||||
Condition Element Documentation <reference_policies_elements_condition.html>`
|
525
source/administration/minio-console.rst
Normal file
525
source/administration/minio-console.rst
Normal file
@@ -0,0 +1,525 @@
|
||||
.. _minio-console:
|
||||
|
||||
=============
|
||||
MinIO Console
|
||||
=============
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
|
||||
The MinIO Console is a rich graphical user interface that provides similar
|
||||
functionality to the :mc:`mc` command line tool.
|
||||
|
||||
.. image:: /images/minio-console/minio-console.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Landing Page provides a view of Buckets on the deployment
|
||||
:align: center
|
||||
|
||||
You can use the MinIO Console for administration tasks like Identity and
|
||||
Access Management, Metrics and Log Monitoring, or Server Configuration.
|
||||
|
||||
The MinIO Console is embedded as part of the MinIO Server binary starting
|
||||
with :minio-release:`RELEASE.2021-07-08T01-15-01Z`. You can also deploy a
|
||||
standalone MinIO Console using the instructions in the
|
||||
:minio-git:`github repository <console>`.
|
||||
|
||||
You can explore the Console using https://play.min.io:9443. Log in with
|
||||
the following credentials:
|
||||
|
||||
- Username: ``Q3AM3UQ867SPQQA43P2F``
|
||||
- Password: ``zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG``
|
||||
|
||||
The Play Console connects to the MinIO Play deployment at https://play.min.io.
|
||||
You can also access this deployment using :mc:`mc` and using the ``play``
|
||||
alias.
|
||||
|
||||
This page documents the high level configuration settings and features of the
|
||||
MinIO Console.
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
The MinIO Console inherits the majority of its configuration settings from the
|
||||
MinIO Server. The following environment variables enable specific behavior in
|
||||
the MinIO Console:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
:width: 100%
|
||||
|
||||
* - Environment Variable
|
||||
- Description
|
||||
|
||||
* - :envvar:`MINIO_PROMETHEUS_URL`
|
||||
- The URL for a Prometheus server configured to scrape metrics from the
|
||||
MinIO deployment. The MinIO Console uses this server for populating the
|
||||
metrics dashboard.
|
||||
|
||||
See :ref:`minio-metrics-collect-using-prometheus` for a tutorial on
|
||||
configuring Prometheus to collect metrics from MinIO.
|
||||
|
||||
* - :envvar:`MINIO_SERVER_URL`
|
||||
- The URL hostname the MinIO Console uses for connecting to the MinIO
|
||||
Server. The hostname *must* be resolveable and reachable for the
|
||||
Console to function correctly.
|
||||
|
||||
The MinIO Console connects to the MinIO Server using an IP
|
||||
address by default. For example, when the MinIO Server starts up,
|
||||
the server logs include a line
|
||||
``API: https://<IP ADDRESS 1> https://<IP ADDRESS 2>``.
|
||||
The MinIO Console defaults to connecting using ``<IP ADDRESS 1>``.
|
||||
|
||||
The MinIO Console may require setting this variable in the following
|
||||
scenarios:
|
||||
|
||||
- The MinIO server TLS certificates do not include the local IP address
|
||||
as a :rfc:`Subject Alternative Name <5280#section-4.2.1.6>` (SAN).
|
||||
Specify a hostname contained in the TLS certificate to allow the MinIO
|
||||
Console to validate the TLS connection.
|
||||
|
||||
- The MinIO server's local IP address is not reachable by the MinIO
|
||||
Console. Specify a resolveable hostname for the MinIO Server.
|
||||
|
||||
- A load balancer or reverse proxy controls traffic to the MinIO server,
|
||||
such that the MinIO Console cannot reach the server without going
|
||||
through the load balancer/proxy. Specify the load balancer/proxy
|
||||
URL for the MinIO server.
|
||||
|
||||
* - :envvar:`MINIO_BROWSER_REDIRECT_URL`
|
||||
- The externally resolvable hostname for the MinIO Console used by the
|
||||
configured :ref:`external identity manager
|
||||
<minio-authentication-and-identity-management>` for returning the
|
||||
authentication response.
|
||||
|
||||
This variable is typically necessary when using a reverse proxy,
|
||||
load balancer, or similar system to expose the MinIO Console to the
|
||||
public internet. Specify an externally reachable hostname that resolves
|
||||
to the MinIO Console.
|
||||
|
||||
Static vs Dynamic Port Assignment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO by default selects a random port for the MinIO Console on each server
|
||||
startup. Browser clients accessing the MinIO Server are automatically
|
||||
redirected to the MinIO Console on its dynamically selected port.
|
||||
This behavior emulates the legacy web browser behavior while reducing the
|
||||
the risk of a port collision on systems which were running MinIO *before* the
|
||||
embedded Console update.
|
||||
|
||||
You can select an explicit static port by passing the
|
||||
:mc-cmd:`minio server --console-address` commandline option when starting
|
||||
each MinIO Server in the deployment.
|
||||
|
||||
For example, the following command starts a distributed MinIO deployment using
|
||||
a static port assignment of ``9001`` for the MinIO Console. This deployment
|
||||
would respond to S3 API operations on the default MinIO server port ``:9000``
|
||||
and browser access on the MinIO Console port ``:9001``.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
minio server https://minio-{1...4}.example.net/mnt/disk-{1...4} \
|
||||
--console-address ":9001"
|
||||
|
||||
Deployments behind network routing components which require static ports for
|
||||
routing rules may require setting a static MinIO Console port. For example,
|
||||
load balancers, reverse proxies, or Kubernetes ingress may by default block
|
||||
or exhibit unexpected behavior with the the dynamic redirection behavior.
|
||||
|
||||
.. _minio-console-admin-buckets:
|
||||
|
||||
Buckets
|
||||
-------
|
||||
|
||||
.. image:: /images/minio-console/console-object-browser.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Object Browser
|
||||
:align: center
|
||||
|
||||
The Console :guilabel:`Object Browser` section displays all buckets and objects to which the authenticated user has :ref:`access <minio-policy>`.
|
||||
|
||||
Use the :guilabel:`Search` bar to search for specific buckets or objects.
|
||||
Select the row for the bucket or object to browse.
|
||||
|
||||
Select :guilabel:`Create Bucket` to create a new bucket on the deployment.
|
||||
|
||||
Each bucket has :guilabel:`Manage` and :guilabel:`Browse` buttons.
|
||||
|
||||
- Select :guilabel:`Manage` to open the management interface for the bucket:
|
||||
|
||||
The :guilabel:`Summary` view displays a summary of the bucket's configuration.
|
||||
|
||||
The :guilabel:`Events` view supports configuring :ref:`notification events <minio-bucket-notifications>` using a configured notification target.
|
||||
|
||||
The :guilabel:`Replication` view supports creating and managing :ref:`Server Side Bucket Replication Rules <minio-bucket-replication-serverside>`.
|
||||
|
||||
The :guilabel:`Lifecycle` view supports creating and managing :ref:`Object Lifecycle Management Rules <minio-lifecycle-management>` for the bucket.
|
||||
|
||||
The :guilabel:`Access Audit` view displays all :ref:`policies <minio-policy>` and :ref:`users <minio-users>` with access to that bucket.
|
||||
|
||||
The :guilabel:`Access Rules` view supports creating and managing anonymous bucket policies to attach to the bucket or bucket prefix.
|
||||
Anonymous rules allow clients to access the bucket or prefix without explicitly authenticating with user credentials.
|
||||
|
||||
- Select :guilabel:`Browse` to view the contents of the bucket.
|
||||
You can view and download individual objects, upload new objects, or use the :guilabel:`Rewind` function to view only those :ref:`versions <minio-bucket-versioning>` of an object which existed at the selected timestamp.
|
||||
|
||||
Identity
|
||||
--------
|
||||
|
||||
The :guilabel:`Identity` section provides a management interface for :ref:`MinIO-Managed users <minio-users>`.
|
||||
|
||||
The section contains the following subsections.
|
||||
Some subsections may not be visible if the authenticated user does not have the :ref:`required administrative permissions <minio-policy-mc-admin-actions>`.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Users
|
||||
|
||||
.. image:: /images/minio-console/console-users.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Manage Users
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Users` section displays all MinIO-managed :ref:`users <minio-users>` on the deployment.
|
||||
|
||||
This section is not visible for deployments using an external identity manager such as Active Directory or an OIDC-compatible provider.
|
||||
|
||||
- Select :guilabel:`Create User` to create a new MinIO-managed user.
|
||||
|
||||
You can assign :ref:`groups <minio-groups>` and :ref:`policies <minio-policy>` to the user during creation.
|
||||
|
||||
- Select a user's row to view details for that user.
|
||||
|
||||
You can view and modify the user's assigned :ref:`groups <minio-groups>` and :ref:`policies <minio-policy>`.
|
||||
|
||||
You can also view and manage any :ref:`Service Accounts <minio-idp-service-account>` associated to the user.
|
||||
|
||||
.. tab-item:: Groups
|
||||
|
||||
.. image:: /images/minio-console/console-groups.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Manage Groups
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Groups` section displays all :ref:`groups <minio-groups>` on the MinIO deployment.
|
||||
|
||||
This section is not visible for deployments using an external identity manager such as Active Directory or an OIDC-compatible provider.
|
||||
|
||||
- Select :guilabel:`Create Group` to create a new MinIO Group.
|
||||
|
||||
You can assign new users to the group during creation.
|
||||
|
||||
You can assign policies to the group after creation.
|
||||
|
||||
- Select the group row to open the details for that group.
|
||||
|
||||
You can modify the group membership from the :guilabel:`Members` view.
|
||||
|
||||
You can modify the group's assigned policies from the :guilabel:`Policies` view.
|
||||
|
||||
Changing a user's group membership modifies the policies that user inherits. See :ref:`minio-access-management` for more information.
|
||||
|
||||
.. tab-item:: Service Accounts
|
||||
|
||||
.. image:: /images/minio-console/console-service-accounts.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Service Accounts
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Accounts` section displays all :ref:`minio-idp-service-account` associated to the authenticated user.
|
||||
|
||||
Service accounts support providing applications authentication credentials which inherit permissions from the "parent" user.
|
||||
|
||||
For deployments using an external identity manager such as Active Directory or an OIDC-compatible provider, service accounts provide a way for users to create long-lived credentials.
|
||||
|
||||
- You can select the service account row to view its custom policy, if one exists.
|
||||
|
||||
You can create or modify the policy from this screen.
|
||||
Service account policies cannot exceed the permissions granted to the parent user.
|
||||
|
||||
- You can create a new service account by selecting the :guilabel:`Create service account` button.
|
||||
|
||||
The Console auto-generates an access key and password for the account.
|
||||
You can override these values as necessary.
|
||||
|
||||
You can set a custom policy for the service account that further restricts the permissions granted to the account.
|
||||
|
||||
The Console only displays the service account credentials *once*. You cannot
|
||||
change or retrieve the credentials later. To rotate credentials for an
|
||||
application, create a new service account and delete the old one once the
|
||||
application updates to using the new credentials.
|
||||
|
||||
Access
|
||||
------
|
||||
|
||||
.. image:: /images/minio-console/console-iam.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Manage IAM Policies
|
||||
:align: center
|
||||
|
||||
The :guilabel:`IAM Policies` section displays all :ref:`policies <minio-policy>` on the MinIO deployment.
|
||||
|
||||
This tab or its contents may not be visible if the authenticated user does not have the :ref:`required administrative permissions <minio-policy-mc-admin-actions>`.
|
||||
|
||||
- Select :guilabel:`+ Create Policy` to create a new MinIO Policy.
|
||||
|
||||
- Select the policy row to manage the policy details.
|
||||
|
||||
The :guilabel:`Summary` view displays a summary of the policy.
|
||||
|
||||
The :guilabel:`Users` view displays all users assigned to the policy.
|
||||
|
||||
The :guilabel:`Groups` view displays all groups assigned to the policy.
|
||||
|
||||
The :guilabel:`Raw Policy` view displays the raw JSON policy.
|
||||
|
||||
Use the :guilabel:`Identity: Users` and :guilabel:`Identity: Groups` views to assign a created policy to users and groups, respectively.
|
||||
|
||||
Monitoring
|
||||
----------
|
||||
|
||||
The :guilabel:`Monitoring` section provides an interface for monitoring the MinIO deployment.
|
||||
|
||||
The section contains the following subsections,
|
||||
Some subsections may not be visible if the authenticated user does not have the :ref:`required administrative permissions <minio-policy-mc-admin-actions>`.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Metrics
|
||||
|
||||
.. image:: /images/minio-console/console-metrics.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Metrics displaying detailed data using Prometheus
|
||||
:align: center
|
||||
|
||||
The Console :guilabel:`Dashboard` section displays metrics for the MinIO deployment.
|
||||
|
||||
The Console depends on a :ref:`configured Prometheus service <minio-metrics-collect-using-prometheus>` to generate the detailed metrics shown above.
|
||||
|
||||
The default metrics view provides a high-level overview of the deployment status, including the uptime and availability of individual servers and drives.
|
||||
|
||||
.. image:: /images/minio-console/console-metrics-simple.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Metrics displaying simplified data
|
||||
:align: center
|
||||
|
||||
This view requires configuring a Prometheus service to scrape the deployment metrics.
|
||||
See :ref:`minio-metrics-collect-using-prometheus` for complete instructions.
|
||||
|
||||
.. tab-item:: Logs
|
||||
|
||||
.. image:: /images/minio-console/console-logs.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Logs displaying a list of server logs
|
||||
:align: center
|
||||
|
||||
The Console :guilabel:`Logs` section displays :ref:`server logs <minio-logging>` generated by the MinIO Deployment.
|
||||
|
||||
- Use the :guilabel:`Nodes` dropdown to filter logs to a subset of server nodes in the MinIO deployment.
|
||||
|
||||
- Use the :guilabel:`Log Types` dropdown to filter logs to a subset of log types.
|
||||
|
||||
- Use the :guilabel:`Filter` to apply text filters to the log results
|
||||
|
||||
Select the :guilabel:`Start Logs` button to begin collecting logs using the selected filters and settings.
|
||||
|
||||
.. tab-item:: Audit
|
||||
|
||||
The Audit Log section provides an interface for viewing :ref:`audit logs <minio-logging>` collected by a configured PostgreSQL service.
|
||||
|
||||
The Audit Logging feature is configured and enabled automatically for MinIO deployments created using the :docs-k8s:`MinIO Kubernetes Operator <>`.
|
||||
|
||||
.. tab-item:: Trace
|
||||
|
||||
.. image:: /images/minio-console/console-trace.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Trace
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Trace` section provides HTTP trace functionality for a bucket or buckets on the deployment.
|
||||
This section provides similar functionality to :mc:`mc admin trace`.
|
||||
|
||||
You can modify the trace to show only specific trace calls.
|
||||
The default is to show only :guilabel:`S3` related HTTP traces.
|
||||
|
||||
Select :guilabel:`Filters` to open additional filters to apply to trace output, such as restricting the :guilabel:`Path` on which the trace applies to a specific bucket or bucket prefix.
|
||||
|
||||
.. tab-item:: Watch
|
||||
|
||||
.. image:: /images/minio-console/console-watch.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Watch
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Watch` section displays S3 events as they occur on the selected bucket.
|
||||
This section provides similar functionality to :mc:`mc watch`.
|
||||
|
||||
.. tab-item:: Drives
|
||||
|
||||
.. image:: /images/minio-console/console-drives.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Drive Health Status
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Drives` section displays the healing status for a bucket.
|
||||
MinIO automatically heals objects and drives when it detects problems, such as drive-level corruption or a replacement drive.
|
||||
|
||||
.. important::
|
||||
|
||||
MinIO does not recommend performing manual healing unless explicitly directed by support.
|
||||
|
||||
Support
|
||||
-------
|
||||
|
||||
The :guilabel:`Support` section provides an interface for generating health and performance reports.
|
||||
You can also register your deployment with |subnet| to allow upload of health reports directly through the MinIO Console.
|
||||
|
||||
This section contains the following subsections.
|
||||
Some subsections may not be visible if the authenticated user does not have the :ref:`required administrative permissions <minio-policy-mc-admin-actions>`.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Register
|
||||
|
||||
.. image:: /images/minio-console/console-register.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console - SUBNET Registration login form
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Register` section provides a login form to connect your MinIO deployment to |subnet|.
|
||||
|
||||
After registration, you can upload your deployment health reports directly to SUBNET for reference by MinIO Engineering.
|
||||
.. tab-item:: Health
|
||||
|
||||
.. image:: /images/minio-console/console-health.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console - Health Diagnostics
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Health` section provides an interface for running a health diagnostic for the MinIO Deployment.
|
||||
|
||||
The resulting health report is intended for use by MinIO Engineering via |subnet| and may contain internal or private data points such as hostnames.
|
||||
Exercise caution before sending a health report to a third party or posting the health report in a public forum.
|
||||
|
||||
.. tab-item:: Performance
|
||||
|
||||
.. image:: /images/minio-console/console-performance.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console - Performance Tests
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Performance` section provides an interface for running a performance test of the deployment.
|
||||
The resulting test can provide a general guideline of deployment performance under S3 ``GET`` and ``PUT`` requests.
|
||||
|
||||
For more complete performance testing, consider using a combination of load-testing using your staging application environments and the MinIO :minio-git:`WARP <warp>` tool.
|
||||
|
||||
.. tab-item:: Profile
|
||||
|
||||
.. image:: /images/minio-console/console-profile.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console - Profile Tests
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Profile` section provides an interface for running system profiling of the deployment.
|
||||
The results can provide insight into the MinIO server process running on a given node.
|
||||
|
||||
The resulting report is intended for use by MinIO Engineering via |subnet|.
|
||||
Independent or third-party use of these profiles for diagnostics and remediation is done at your own risk.
|
||||
|
||||
.. tab-item:: Inspect
|
||||
|
||||
.. image:: /images/minio-console/console-inspect.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console - Inspect an Object
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Inspect` section provides an interface for capturing the erasure-coded metadata associated to an object or objects.
|
||||
MinIO Engineering may request this output as part of diagnostics in |subnet|.
|
||||
|
||||
|
||||
The resulting object may be read using MinIO's :minio-git:`debugging tool <minio/tree/master/docs/debugging#decoding-metadata>`.
|
||||
Independent or third-party use of the output for diagnostics or remediation is done at your own risk.
|
||||
You can optionally encrypt the object such that it can only be read if the generated encryption key is included as part of the debugging toolchain.
|
||||
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
The :guilabel:`License` section displays information on the licensing status of the MinIO deployment.
|
||||
|
||||
For deployments not registered via |subnet|, the Console displays a table comparison of MinIO License and Support plans:
|
||||
|
||||
.. image:: /images/minio-console/console-license.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console - License Plans
|
||||
:align: center
|
||||
|
||||
MinIO is Open Source software under the :minio-git:`GNU AGPLv3 license <mc/blob/master/LICENSE>`.
|
||||
Applications using MinIO should follow local laws and regulations around licensing to ensure compliance with the AGPLv3 license, which may include open sourcing the application stack.
|
||||
|
||||
Proprietary application stacks can register for either the SUBNET :guilabel:`Standard` or :guilabel:`Enterprise` License and Support plan to use MinIO under a commercial license.
|
||||
|
||||
Settings
|
||||
--------
|
||||
|
||||
The :guilabel:`Configuration` section displays information on MinIO server configuration settings.
|
||||
|
||||
This section contains the following subsections.
|
||||
Some subsections may not be visible if the authenticated user does not have the :ref:`required administrative permissions <minio-policy-mc-admin-actions>`.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Configuration
|
||||
|
||||
.. image:: /images/minio-console/console-settings-configuration.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Settings - Configuration View
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Configuration` subsection provides an interface for viewing and retrieving :ref:`configuration settings <minio-server-configuration-settings>` for all MinIO Servers in the deployment.
|
||||
|
||||
The interface functionality mimics that of using :mc-cmd:`mc admin config get` or :mc-cmd:`mc admin config set`
|
||||
|
||||
Some configuration settings may require restarting the MinIO deployment to apply changes.
|
||||
|
||||
.. tab-item:: Notifications
|
||||
|
||||
.. image:: /images/minio-console/console-settings-notifications.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Settings - Notifications View
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Notifications` subsection provides an interface for adding and managing :ref:`bucket notification targets <minio-bucket-notifications>`.
|
||||
|
||||
Select the :guilabel:`Add Notification Target +` button to add a new target to the deployment.
|
||||
|
||||
You can select an existing notification target from the list to view its details.
|
||||
|
||||
.. tab-item:: Tiers
|
||||
|
||||
.. image:: /images/minio-console/console-settings-tiers.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Settings - Tiering
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Tiers` subsection provides an interface for adding and managing :ref:`remote tiers <minio-lifecycle-management-tiering>` to support lifecycle management transition rules.
|
||||
|
||||
Select the :guilabel:`Create Tier +` button to add a new tier to the deployment.
|
||||
|
||||
You can select an existing tier from the list to view its details.
|
||||
|
||||
.. tab-item:: Site Replication
|
||||
|
||||
.. image:: /images/minio-console/console-settings-site-replication.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Settings - Site Replication
|
||||
:align: center
|
||||
|
||||
The :guilabel:`Site Replication` subsection provides an interface for adding and managing the site replication configuration for the deployment.
|
||||
|
||||
Configuring site replication requires that only a single site have existing buckets or objects (if any).
|
56
source/administration/monitoring.rst
Normal file
56
source/administration/monitoring.rst
Normal file
@@ -0,0 +1,56 @@
|
||||
===================================
|
||||
Monitoring Bucket and Object Events
|
||||
===================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
Bucket Notifications
|
||||
--------------------
|
||||
|
||||
MinIO bucket notifications allow administrators to send notifications to supported external services on certain object or bucket events.
|
||||
MinIO supports bucket and object-level S3 events similar to the
|
||||
:s3-docs:`Amazon S3 Event Notifications <NotificationHowTo.html>`.
|
||||
|
||||
MinIO supports publishing bucket or object events to the following supported
|
||||
targets on certain supported events.
|
||||
|
||||
- :ref:`minio-bucket-notifications-publish-amqp`
|
||||
- :ref:`minio-bucket-notifications-publish-mqtt`
|
||||
- :ref:`minio-bucket-notifications-publish-nats`
|
||||
- :ref:`minio-bucket-notifications-publish-nsq`
|
||||
- :ref:`minio-bucket-notifications-publish-elasticsearch`
|
||||
- :ref:`minio-bucket-notifications-publish-kafka`
|
||||
- :ref:`minio-bucket-notifications-publish-mysql`
|
||||
- :ref:`minio-bucket-notifications-publish-postgresql`
|
||||
- :ref:`minio-bucket-notifications-publish-redis`
|
||||
- :ref:`minio-bucket-notifications-publish-webhook`
|
||||
|
||||
See :ref:`minio-bucket-notifications`
|
||||
for more complete documentation on MinIO Bucket Notifications.
|
||||
|
||||
Deployment Metrics
|
||||
------------------
|
||||
|
||||
MinIO provides a Prometheus-compatible endpoint for supporting time-series querying of metrics.
|
||||
|
||||
MinIO deployments :ref:`configured to enable Prometheus scraping <minio-metrics-and-alerts-endpoints>` provide a detailed metrics view through the MinIO Console.
|
||||
|
||||
Server Logs
|
||||
-----------
|
||||
|
||||
MinIO provides the following interfaces for remotely reading server logs:
|
||||
|
||||
- The MinIO Console provides a server log viewer at <details>
|
||||
- The :mc:`mc admin console` command returns the specified server's console output.
|
||||
- MinIO supports pushing server logs to an HTTP webhook for further ingestion.
|
||||
See :ref:`minio-logging-publish-server-logs` for more information.
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
:hidden:
|
||||
|
||||
/administration/monitoring/bucket-notifications
|
184
source/administration/monitoring/bucket-notifications.rst
Normal file
184
source/administration/monitoring/bucket-notifications.rst
Normal file
@@ -0,0 +1,184 @@
|
||||
.. _minio-bucket-notifications:
|
||||
|
||||
====================
|
||||
Bucket Notifications
|
||||
====================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
MinIO bucket notifications allow administrators to send notifications to supported external services on certain object or bucket events.
|
||||
MinIO supports bucket and object-level S3 events similar to the
|
||||
:s3-docs:`Amazon S3 Event Notifications <NotificationHowTo.html>`.
|
||||
|
||||
Supported Notification Targets
|
||||
------------------------------
|
||||
|
||||
MinIO supports publishing event notifications to the following targets:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
:width: 100%
|
||||
|
||||
* - Target
|
||||
- Description
|
||||
|
||||
* - :guilabel:`AMQP` (RabbitMQ)
|
||||
- Publish notifications to an AMQP service such as
|
||||
`RabbitMQ <https://www.rabbitmq.com>`__.
|
||||
|
||||
See :ref:`minio-bucket-notifications-publish-amqp` for a tutorial.
|
||||
|
||||
* - :guilabel:`MQTT`
|
||||
- Publish notifications to an `MQTT <https://www.mqtt.org/>`__ service.
|
||||
|
||||
See :ref:`minio-bucket-notifications-publish-mqtt` for a tutorial.
|
||||
|
||||
* - :guilabel:`NATS`
|
||||
- Publish notifications to a `NATS <https://nats.io/>`__ service.
|
||||
|
||||
See :ref:`minio-bucket-notifications-publish-nats` for a tutorial.
|
||||
|
||||
* - :guilabel:`NSQ`
|
||||
- Publish notifications to a `NSQ <https://nsq.io/>`__ service.
|
||||
|
||||
See :ref:`minio-bucket-notifications-publish-nsq` for a tutorial
|
||||
|
||||
* - :guilabel:`Elasticsearch`
|
||||
- Publish notifications to a `Elasticsearch <https://www.elastic.co/>`__ service.
|
||||
|
||||
See :ref:`minio-bucket-notifications-publish-elasticsearch` for a tutorial.
|
||||
|
||||
* - :guilabel:`Kafka`
|
||||
- Publish notifications to a `Kafka <https://kafka.apache.org/>`__ service.
|
||||
|
||||
See :ref:`minio-bucket-notifications-publish-kafka` for a tutorial.
|
||||
|
||||
* - :guilabel:`MySQL`
|
||||
- Publish notifications to a `MySQL <https://www.mysql.com/>`__ service.
|
||||
|
||||
See :ref:`minio-bucket-notifications-publish-mysql` for a tutorial.
|
||||
|
||||
* - :guilabel:`PostgreSQL`
|
||||
- Publish notifications to a `PostgreSQL <https://www.postgresql.org/>`__ service.
|
||||
|
||||
See :ref:`minio-bucket-notifications-publish-postgresql` for a tutorial.
|
||||
|
||||
* - :guilabel:`Redis`
|
||||
- Publish notifications to a `Redis <https://redis.io/>`__ service.
|
||||
|
||||
See :ref:`minio-bucket-notifications-publish-redis` for a tutorial.
|
||||
|
||||
* - :guilabel:`webhook`
|
||||
- Publish notifications to a `Webhook
|
||||
<https://en.wikipedia.org/wiki/Webhook>`__ service.
|
||||
|
||||
See :ref:`minio-bucket-notifications-publish-webhook` for a tutorial.
|
||||
|
||||
|
||||
.. _minio-bucket-notifications-event-types:
|
||||
|
||||
Supported S3 Event Types
|
||||
------------------------
|
||||
|
||||
MinIO bucket notifications are compatible with :s3-docs:`Amazon S3 Event Notifications <NotificationHowTo.html>`.
|
||||
This section lists all supported events.
|
||||
|
||||
Object Events
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports triggering notifications on the following S3 object events:
|
||||
|
||||
.. data:: s3:ObjectAccessed:Get
|
||||
.. data:: s3:ObjectAccessed:GetLegalHold
|
||||
.. data:: s3:ObjectAccessed:GetRetention
|
||||
.. data:: s3:ObjectAccessed:Head
|
||||
.. data:: s3:ObjectCreated:CompleteMultipartUpload
|
||||
.. data:: s3:ObjectCreated:Copy
|
||||
.. data:: s3:ObjectCreated:DeleteTagging
|
||||
.. data:: s3:ObjectCreated:Post
|
||||
.. data:: s3:ObjectCreated:Put
|
||||
.. data:: s3:ObjectCreated:PutLegalHold
|
||||
.. data:: s3:ObjectCreated:PutRetention
|
||||
.. data:: s3:ObjectCreated:PutTagging
|
||||
.. data:: s3:ObjectRemoved:Delete
|
||||
.. data:: s3:ObjectRemoved:DeleteMarkerCreated
|
||||
|
||||
Specify the wildcard ``*`` character to select all events related to a prefix:
|
||||
|
||||
.. data:: s3:ObjectAccessed:*
|
||||
|
||||
Selects all ``s3:ObjectAccessed``\ -prefixed events.
|
||||
|
||||
.. data:: s3:ObjectCreated:*
|
||||
|
||||
Selects all ``s3:ObjectCreated``\ -prefixed events.
|
||||
|
||||
.. data:: s3:ObjectRemoved:*
|
||||
|
||||
Selects all ``s3:ObjectRemoved``\ -prefixed events.
|
||||
|
||||
Replication Events
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports triggering notifications on the following S3 replication events:
|
||||
|
||||
.. data:: s3:Replication:OperationCompletedReplication
|
||||
.. data:: s3:Replication:OperationFailedReplication
|
||||
.. data:: s3:Replication:OperationMissedThreshold
|
||||
.. data:: s3:Replication:OperationNotTracked
|
||||
.. data:: s3:Replication:OperationReplicatedAfterThreshold
|
||||
|
||||
Specify the wildcard ``*`` character to select all ``s3:Replication`` events:
|
||||
|
||||
.. data:: s3:Replication:*
|
||||
|
||||
ILM Transition Events
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports triggering notifications on the following S3 ILM transition events:
|
||||
|
||||
.. data:: s3:ObjectRestore:Post
|
||||
.. data:: s3:ObjectRestore:Completed
|
||||
.. data:: s3:ObjectTransition:Failed
|
||||
.. data:: s3:ObjectTransition:Complete
|
||||
|
||||
Specify the wildcard ``*`` character to select all events related to a prefix:
|
||||
|
||||
.. data:: s3:ObjectTransition:*
|
||||
|
||||
Selects all ``s3:ObjectTransition``\ -prefixed events.
|
||||
|
||||
.. data:: s3:ObjectRestore:*
|
||||
|
||||
Selects all ``s3:ObjectRestore``\ -prefixed events.
|
||||
|
||||
Global Events
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports triggering notifications on the following global events.
|
||||
You can only listen to these events through the :legacy:`ListenNotification <golang-client-api-reference.html#ListenNotification>` API:
|
||||
|
||||
.. data:: s3:BucketCreated
|
||||
.. data:: s3:BucketRemoved
|
||||
|
||||
.. todo
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
:hidden:
|
||||
|
||||
/administration/monitoring/publish-events-to-amqp
|
||||
/administration/monitoring/publish-events-to-mqtt
|
||||
/administration/monitoring/publish-events-to-nats
|
||||
/administration/monitoring/publish-events-to-nsq
|
||||
/administration/monitoring/publish-events-to-elasticsearch
|
||||
/administration/monitoring/publish-events-to-kafka
|
||||
/administration/monitoring/publish-events-to-mysql
|
||||
/administration/monitoring/publish-events-to-postgresql
|
||||
/administration/monitoring/publish-events-to-redis
|
||||
/administration/monitoring/publish-events-to-webhook
|
362
source/administration/monitoring/publish-events-to-amqp.rst
Normal file
362
source/administration/monitoring/publish-events-to-amqp.rst
Normal file
@@ -0,0 +1,362 @@
|
||||
.. _minio-bucket-notifications-publish-amqp:
|
||||
|
||||
=================================
|
||||
Publish Events to AMQP (RabbitMQ)
|
||||
=================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. |ARN| replace:: ``arn:minio:sqs::primary:amqp``
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO supports publishing :ref:`bucket notification
|
||||
<minio-bucket-notifications>` events to a `AMQP 0-9-1 <https://www.amqp.org/>`__
|
||||
service endpoint such as `RabbitMQ <https://www.rabbitmq.com>`__.
|
||||
|
||||
MinIO relies on the :github:`streadway/amqp` project for AMQP connectivity. The
|
||||
project is primarily tested against `RabbitMQ <https://www.rabbitmq.com/>`__
|
||||
deployments, though other `AMQP 0-9-1-compatible <https://www.amqp.org/>`__
|
||||
services *may* also work. The procedures on this page assume a RabbitMQ
|
||||
deployment using the AMQP 0-9-1 protocol as the service endpoint.
|
||||
|
||||
Add an AMQP Endpoint to a MinIO Deployment
|
||||
------------------------------------------
|
||||
|
||||
The following procedure adds a new AMQP service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
AMQP 0-9-1 Service Endpoint
|
||||
+++++++++++++++++++++++++++
|
||||
|
||||
MinIO relies on the :github:`streadway/amqp` project for AMQP connectivity. The
|
||||
project is primarily tested against `RabbitMQ <https://www.rabbitmq.com/>`__
|
||||
deployments, though other `AMQP 0-9-1-compatible <https://www.amqp.org/>`__
|
||||
services *may* also work. This procedure assumes a RabbitMQ deployment
|
||||
using the 0-9-1 protocol as the service endpoint.
|
||||
|
||||
If the AMQP service requires authentication, you *must* provide an appropriate
|
||||
username and password during the configuration process to grant MinIO access
|
||||
to the service.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
1) Add the AMQP Endpoint to MinIO
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure a new AMQP service endpoint using either environment variables
|
||||
*or* by setting runtime configuration settings.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Environment Variables
|
||||
|
||||
MinIO supports specifying the AMQP service endpoint and associated
|
||||
configuration settings using
|
||||
:ref:`environment variables
|
||||
<minio-server-envvar-bucket-notification-amqp>`. The
|
||||
:mc:`minio server` process applies the specified settings on its
|
||||
next startup.
|
||||
|
||||
The following example code sets *all* environment variables
|
||||
related to configuring an AMQP service endpoint. The minimum
|
||||
*required* variables are
|
||||
:envvar:`MINIO_NOTIFY_AMQP_ENABLE` and :envvar:`MINIO_NOTIFY_AMQP_URL`:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
set MINIO_NOTIFY_AMQP_ENABLE_<IDENTIFIER>="on"
|
||||
set MINIO_NOTIFY_AMQP_URL_<IDENTIFIER>="<ENDPOINT>"
|
||||
set MINIO_NOTIFY_AMQP_EXCHANGE_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_EXCHANGE_TYPE_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_ROUTING_KEY_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_MANDATORY_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_DURABLE_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_NO_WAIT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_INTERNAL_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_AUTO_DELETED_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_DELIVERY_MODE_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_QUEUE_DIR_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_QUEUE_LIMIT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_AMQP_COMMENT_<IDENTIFIER>="<string>"
|
||||
|
||||
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
|
||||
AMQP service endpoint. Use the same ``<IDENTIFIER>`` value for all
|
||||
environment variables related to the new AMQP service endpoint.
|
||||
The following examples assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``<IDENTIFIER>`` matches an existing AMQP service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_amqp <mc admin config get>` to
|
||||
review the currently configured AMQP endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the URL of the AMQP service endpoint.
|
||||
For example:
|
||||
|
||||
``amqp://user:password@hostname:port``
|
||||
|
||||
See :ref:`AMQP Service for Bucket Notifications
|
||||
<minio-server-envvar-bucket-notification-amqp>` for complete documentation
|
||||
on each environment variable.
|
||||
|
||||
.. tab-item:: Configuration Settings
|
||||
|
||||
MinIO supports adding or updating AMQP endpoints on a running
|
||||
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
|
||||
and the :mc-conf:`notify_amqp` configuration key. You must restart the
|
||||
:mc:`minio server` process to apply any new or updated configuration
|
||||
settings.
|
||||
|
||||
The following example code sets *all* settings related to configuring an
|
||||
AMQP service endpoint. The minimum *required* setting is
|
||||
:mc-conf:`notify_amqp url <notify_amqp.url>`:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_amqp:IDENTIFIER \
|
||||
url="ENDPOINT" \
|
||||
exchange="<string>" \
|
||||
exchange_type="<string>" \
|
||||
routing_key="<string>" \
|
||||
mandatory="<string>" \
|
||||
durable="<string>" \
|
||||
no_wait="<string>" \
|
||||
internal="<string>" \
|
||||
auto_deleted="<string>" \
|
||||
delivery_mode="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
- Replace ``IDENTIFIER`` with a unique descriptive string for the
|
||||
AMQP service endpoint. The following examples in this procedure
|
||||
assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``IDENTIFIER`` matches an existing AMQP service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_amqp <mc admin config get>` to
|
||||
review the currently configured AMQP endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``ENDPOINT`` with the URL of the AMQP service endpoint.
|
||||
For example:
|
||||
|
||||
``amqp://user:password@hostname:port``
|
||||
|
||||
See :ref:`AMQP Bucket Notification Configuration Settings
|
||||
<minio-server-config-bucket-notification-amqp>` for complete
|
||||
documentation on each setting.
|
||||
|
||||
2) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured AMQP
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:amqp
|
||||
|
||||
You must specify the ARN resource when configuring bucket notifications with
|
||||
the associated AMQP deployment as a target.
|
||||
|
||||
.. include:: /includes/common-bucket-notifications.rst
|
||||
:start-after: start-bucket-notification-find-arn
|
||||
:end-before: end-bucket-notification-find-arn
|
||||
|
||||
3) Configure Bucket Notifications using the AMQP Endpoint as a Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc event add` command to add a new bucket notification
|
||||
event with the configured AMQP service as a target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event add ALIAS/BUCKET arn:minio:sqs::primary:amqp \
|
||||
--event EVENTS
|
||||
|
||||
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
|
||||
- Replace ``BUCKET`` with the name of the bucket in which to configure the
|
||||
event.
|
||||
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
|
||||
<mc-event-supported-events>` for which MinIO triggers notifications.
|
||||
|
||||
Use :mc-cmd:`mc event list` to view all configured bucket events for
|
||||
a given notification target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event list ALIAS/BUCKET arn:minio:sqs::primary:amqp
|
||||
|
||||
4) Validate the Configured Events
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on the bucket for which you configured the new event and
|
||||
check the AMQP service for the notification data. The action required
|
||||
depends on which :mc-cmd:`events <mc event add --event>` were specified
|
||||
when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
||||
|
||||
|
||||
|
||||
Update an AMQP Endpoint in a MinIO Deployment
|
||||
---------------------------------------------
|
||||
|
||||
The following procedure updates an existing AMQP service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
AMQP 0-9-1 Service Endpoint
|
||||
+++++++++++++++++++++++++++
|
||||
|
||||
MinIO relies on the :github:`streadway/amqp` project for AMQP connectivity. The
|
||||
project is primarily tested against `RabbitMQ <https://www.rabbitmq.com/>`__
|
||||
deployments, though other `AMQP 0-9-1-compatible <https://www.amqp.org/>`__
|
||||
services *may* also work. This procedure *assumes* a RabbitMQ deployment
|
||||
as the service endpoint.
|
||||
|
||||
If the AMQP service requires authentication, you *must* provide an appropriate
|
||||
username and password during the configuration process to grant MinIO access
|
||||
to the service.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
|
||||
1) List Configured AMQP Endpoints In The Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config get` command to list the currently
|
||||
configured AMQP service endpoints in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config get ALIAS/ notify_amqp
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
The command output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
notify_amqp:primary delivery_mode="0" exchange_type="" no_wait="off" queue_dir="" queue_limit="0" url="amqp://user:password@hostname:port" auto_deleted="off" durable="off" exchange="" internal="off" mandatory="off" routing_key=""
|
||||
notify_amqp:secondary delivery_mode="0" exchange_type="" no_wait="off" queue_dir="" queue_limit="0" url="amqp://user:password@hostname:port" auto_deleted="off" durable="off" exchange="" internal="off" mandatory="off" routing_key=""
|
||||
|
||||
The :mc-conf:`notify_amqp` key is the top-level configuration key for an
|
||||
:ref:`minio-server-config-bucket-notification-amqp`. The
|
||||
:mc-conf:`url <notify_amqp.url>` key specifies the AMQP service endpoint
|
||||
for the given `notify_amqp` key. The ``notify_amqp:<IDENTIFIER>`` suffix
|
||||
describes the unique identifier for that AMQP service endpoint.
|
||||
|
||||
Note the identifier for the AMQP service endpoint you want to update for
|
||||
the next step.
|
||||
|
||||
2) Update the AMQP Endpoint
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config set` command to set the new configuration
|
||||
for the AMQP service endpoint:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_amqp:<IDENTIFIER> \
|
||||
url="amqp://user:password@hostname:port" \
|
||||
exchange="<string>" \
|
||||
exchange_type="<string>" \
|
||||
routing_key="<string>" \
|
||||
mandatory="<string>" \
|
||||
durable="<string>" \
|
||||
no_wait="<string>" \
|
||||
internal="<string>" \
|
||||
auto_deleted="<string>" \
|
||||
delivery_mode="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
The :mc-conf:`notify_amqp url <notify_amqp.url>` configuration setting is the
|
||||
*minimum* required for an AMQP service endpoint. All other configuration
|
||||
settings are *optional*. See :ref:`minio-server-config-bucket-notification-amqp`
|
||||
for a complete list of AMQP configuration settings.
|
||||
|
||||
3) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured AMQP
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:amqp
|
||||
|
||||
4) Validate the Changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on a bucket which has an event configuration using the updated
|
||||
AMQP service endpoint and check the AMQP service for the notification data. The
|
||||
action required depends on which :mc-cmd:`events <mc event add --event>` were
|
||||
specified when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
@@ -0,0 +1,344 @@
|
||||
.. _minio-bucket-notifications-publish-elasticsearch:
|
||||
|
||||
===============================
|
||||
Publish Events to Elasticsearch
|
||||
===============================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. |ARN| replace:: ``arn:minio:sqs::primary:elasticsearch``
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO supports publishing :ref:`bucket notification
|
||||
<minio-bucket-notifications>` events to an
|
||||
`Elasticsearch <https://www.elastic.co/>`__ service endpoint.
|
||||
|
||||
MinIO relies on the :github:`olivere/elastic` v7 project for Elastic
|
||||
connectivity. The ``elastic/v7`` library specifically targets Elasticsearch
|
||||
v7.0 and is *not compatible with earlier Elasticsearch versions*.
|
||||
|
||||
Add a Elasticsearch Endpoint to a MinIO Deployment
|
||||
--------------------------------------------------
|
||||
|
||||
The following procedure adds a new Elasticsearch service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Elasticsearch v7.0 and later
|
||||
++++++++++++++++++++++++++++
|
||||
|
||||
MinIO relies on the :github:`olivere/elastic` v7 project for Elastic
|
||||
connectivity. The ``elastic/v7`` library specifically targets Elasticsearch
|
||||
v7.0 and is *not compatible with earlier Elasticsearch versions*.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
1) Add the Elasticsearch Endpoint to MinIO
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure a new Elasticsearch service endpoint using either environment variables
|
||||
*or* by setting runtime configuration settings.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Environment Variables
|
||||
|
||||
MinIO supports specifying the Elasticsearch service endpoint and associated
|
||||
configuration settings using
|
||||
:ref:`environment variables
|
||||
<minio-server-envvar-bucket-notification-elasticsearch>`. The
|
||||
:mc:`minio server` process applies the specified settings on its
|
||||
next startup.
|
||||
|
||||
The following example code sets *all* environment variables
|
||||
related to configuring an Elasticsearch service endpoint. The minimum
|
||||
*required* variables are:
|
||||
|
||||
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_ENABLE`
|
||||
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_URL`
|
||||
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_INDEX`
|
||||
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_FORMAT`
|
||||
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
set MINIO_NOTIFY_ELASTICSEARCH_ENABLE_<IDENTIFIER>="on"
|
||||
set MINIO_NOTIFY_ELASTICSEARCH_URL_<IDENTIFIER>="<ENDPOINT>"
|
||||
set MINIO_NOTIFY_ELASTICSEARCH_INDEX_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_ELASTICSEARCH_FORMAT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_ELASTICSEARCH_USERNAME_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_ELASTICSEARCH_PASSWORD_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_ELASTICSEARCH_QUEUE_DIR_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_ELASTICSEARCH_QUEUE_LIMIT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_ELASTICSEARCH_COMMENT_<IDENTIFIER>="<string>"
|
||||
|
||||
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
|
||||
TARGET service endpoint. Use the same ``<IDENTIFIER>`` value for all
|
||||
environment variables related to the new target service endpoint.
|
||||
The following examples assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``<IDENTIFIER>`` matches an existing Elasticsearch
|
||||
service endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_elasticsearch <mc admin config get>`
|
||||
to review the currently configured Elasticsearch endpoints on the MinIO
|
||||
deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the URL of the Elasticsearch service endpoint.
|
||||
For example:
|
||||
|
||||
See :ref:`Elasticsearch Service for Bucket Notifications
|
||||
<minio-server-envvar-bucket-notification-elasticsearch>` for complete
|
||||
documentation on each environment variable.
|
||||
|
||||
.. tab-item:: Configuration Settings
|
||||
|
||||
MinIO supports adding or updating Elasticsearch endpoints on a running
|
||||
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
|
||||
and the :mc-conf:`notify_elasticsearch` configuration key. You must restart the
|
||||
:mc:`minio server` process to apply any new or updated configuration
|
||||
settings.
|
||||
|
||||
The following example code sets *all* settings related to configuring an
|
||||
Elasticsearch service endpoint. The minimum *required* settings are:
|
||||
|
||||
- :mc-conf:`url <notify_elasticsearch.url>`
|
||||
- :mc-conf:`index <notify_elasticsearch.index>`
|
||||
- :mc-conf:`format <notify_elasticsearch.format>`
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_elasticsearch:IDENTIFIER \
|
||||
url="ENDPOINT" \
|
||||
index="<string>" \
|
||||
format="<string>" \
|
||||
username="<string>" \
|
||||
password="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
|
||||
- Replace ``IDENTIFIER`` with a unique descriptive string for the
|
||||
Elasticsearch service endpoint. The following examples in this procedure
|
||||
assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``IDENTIFIER`` matches an existing Elasticsearch service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_elasticsearch <mc admin config get>`
|
||||
to review the currently configured Elasticsearch endpoints on the MinIO
|
||||
deployment.
|
||||
|
||||
- Replace ``ENDPOINT`` with the URL of the Elasticsearch service endpoint.
|
||||
For example:
|
||||
|
||||
``https://user:password@hostname:port``
|
||||
|
||||
See :ref:`Elasticsearch Bucket Notification Configuration Settings
|
||||
<minio-server-config-bucket-notification-elasticsearch>` for complete
|
||||
documentation on each setting.
|
||||
|
||||
2) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured Elasticsearch
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:elasticsearch
|
||||
|
||||
You must specify the ARN resource when configuring bucket notifications with
|
||||
the associated Elasticsearch deployment as a target.
|
||||
|
||||
.. include:: /includes/common-bucket-notifications.rst
|
||||
:start-after: start-bucket-notification-find-arn
|
||||
:end-before: end-bucket-notification-find-arn
|
||||
|
||||
|
||||
3) Configure Bucket Notifications using the Elasticsearch Endpoint as a Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc event add` command to add a new bucket notification
|
||||
event with the configured Elasticsearch service as a target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event add ALIAS/BUCKET arn:minio:sqs::primary:elasticsearch \
|
||||
--event EVENTS
|
||||
|
||||
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
|
||||
- Replace ``BUCKET`` with the name of the bucket in which to configure the
|
||||
event.
|
||||
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
|
||||
<mc-event-supported-events>` for which MinIO triggers notifications.
|
||||
|
||||
Use :mc-cmd:`mc event list` to view all configured bucket events for
|
||||
a given notification target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event list ALIAS/BUCKET arn:minio:sqs::primary:elasticsearch
|
||||
|
||||
4) Validate the Configured Events
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on the bucket for which you configured the new event and
|
||||
check the Elasticsearch service for the notification data. The action required
|
||||
depends on which :mc-cmd:`events <mc event add --event>` were specified
|
||||
when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
||||
|
||||
Update an Elasticsearch Endpoint in a MinIO Deployment
|
||||
------------------------------------------------------
|
||||
|
||||
The following procedure updates an existing Elasticsearch service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Elasticsearch v7.0 and later
|
||||
++++++++++++++++++++++++++++
|
||||
|
||||
MinIO relies on the :github:`olivere/elastic` v7 project for Elastic
|
||||
connectivity. The ``elastic/v7`` library specifically targets Elasticsearch
|
||||
v7.0 and is *not compatible with earlier Elasticsearch versions*.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
|
||||
1) List Configured Elasticsearch Endpoints In The Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config get` command to list the currently
|
||||
configured Elasticsearch service endpoints in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config get ALIAS/ notify_elasticsearch
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
The command output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
notify_elasticsearch:primary queue_dir="" queue_limit="0" url="https://user:password@hostname:port" format="namespace" index=""
|
||||
notify_elasticsearch:secondary queue_dir="" queue_limit="0" url="https://user:password@hostname:port" format="namespace" index=""
|
||||
|
||||
The :mc-conf:`notify_elasticsearch` key is the top-level configuration key for
|
||||
an :ref:`minio-server-config-bucket-notification-elasticsearch`. The
|
||||
:mc-conf:`url <notify_elasticsearch.url>` key specifies the Elasticsearch
|
||||
service endpoint for the given `notify_elasticsearch` key. The
|
||||
``notify_elasticsearch:<IDENTIFIER>`` suffix describes the unique identifier for
|
||||
that Elasticsearch service endpoint.
|
||||
|
||||
Note the identifier for the Elasticsearch service endpoint you want to update for
|
||||
the next step.
|
||||
|
||||
2) Update the Elasticsearch Endpoint
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config set` command to set the new configuration
|
||||
for the Elasticsearch service endpoint:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_elasticsearch:<IDENTIFIER> \
|
||||
url="https://user:password@hostname:port" \
|
||||
index="<string>" \
|
||||
format="<string>" \
|
||||
username="<string>" \
|
||||
password="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
The :mc-conf:`notify_elasticsearch url <notify_elasticsearch.url>` configuration
|
||||
setting is the *minimum* required for an Elasticsearch service endpoint. All
|
||||
other configuration settings are *optional*. See
|
||||
:ref:`minio-server-config-bucket-notification-elasticsearch` for a complete list
|
||||
of Elasticsearch configuration settings.
|
||||
|
||||
3) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured
|
||||
Elasticsearch target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:elasticsearch
|
||||
|
||||
4) Validate the Changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on a bucket which has an event configuration using the updated
|
||||
Elasticsearch service endpoint and check the Elasticsearch service for the
|
||||
notification data. The action required depends on which
|
||||
:mc-cmd:`events <mc event add --event>` were specified when configuring the bucket
|
||||
notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
356
source/administration/monitoring/publish-events-to-kafka.rst
Normal file
356
source/administration/monitoring/publish-events-to-kafka.rst
Normal file
@@ -0,0 +1,356 @@
|
||||
.. _minio-bucket-notifications-publish-kafka:
|
||||
|
||||
=======================
|
||||
Publish Events to Kafka
|
||||
=======================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. |ARN| replace:: ``arn:minio:sqs::primary:kafka``
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO supports publishing :ref:`bucket notification
|
||||
<minio-bucket-notifications>` events to a `Kafka <https://kafka.apache.org/>`__
|
||||
service endpoint.
|
||||
|
||||
MinIO relies on the :github:`Shopify/sarama` project for Kafka connectivity
|
||||
and shares that project's Kafka support. See the
|
||||
``sarama`` :github:`Compatibility and API stability
|
||||
<Shopify/sarama/#compatibility-and-api-stability>` section for more details.
|
||||
|
||||
Add a Kafka Endpoint to a MinIO Deployment
|
||||
------------------------------------------
|
||||
|
||||
The following procedure adds a new Kafka service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Kafka Minimum Versions and Supported Versions
|
||||
+++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
MinIO relies on the :github:`Shopify/sarama` project for Kafka connectivity
|
||||
and shares that project's Kafka support. See the
|
||||
``sarama`` :github:`Compatibility and API stability
|
||||
<Shopify/sarama/#compatibility-and-api-stability>` section for more details.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
1) Add the Kafka Endpoint to MinIO
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure a new Kafka service endpoint using either environment variables
|
||||
*or* by setting runtime configuration settings.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Environment Variables
|
||||
|
||||
MinIO supports specifying the Kafka service endpoint and associated
|
||||
configuration settings using
|
||||
:ref:`environment variables
|
||||
<minio-server-envvar-bucket-notification-kafka>`. The
|
||||
:mc:`minio server` process applies the specified settings on its
|
||||
next startup.
|
||||
|
||||
The following example code sets *all* environment variables
|
||||
related to configuring a Kafka service endpoint. The minimum
|
||||
*required* variables are
|
||||
:envvar:`MINIO_NOTIFY_KAFKA_ENABLE` and
|
||||
:envvar:`MINIO_NOTIFY_KAFKA_BROKERS`:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
set MINIO_NOTIFY_KAFKA_ENABLE_<IDENTIFIER>="on"
|
||||
set MINIO_NOTIFY_KAFKA_BROKERS_<IDENTIFIER>="<ENDPOINT>"
|
||||
set MINIO_NOTIFY_KAFKA_TOPIC_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_SASL_USERNAME_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_SASL_PASSWORD_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_SASL_MECHANISM_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_TLS_CLIENT_AUTH_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_SASL_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_TLS_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_TLS_SKIP_VERIFY_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_CLIENT_TLS_CERT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_CLIENT_TLS_KEY_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_QUEUE_DIR_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_QUEUE_LIMIT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_VERSION_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_KAFKA_COMMENT_<IDENTIFIER>="<string>"
|
||||
|
||||
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
|
||||
Kafka service endpoint. Use the same ``<IDENTIFIER>`` value for all
|
||||
environment variables related to the new target service endpoint.
|
||||
The following examples assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``<IDENTIFIER>`` matches an existing Kafka service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_kafka <mc admin config get>` to
|
||||
review the currently configured Kafka endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with a comma-separated list of Kafka brokers.
|
||||
For example:
|
||||
|
||||
``"kafka1.example.com:2021,kafka2.example.com:2021"``
|
||||
|
||||
See :ref:`Kafka Service for Bucket Notifications
|
||||
<minio-server-envvar-bucket-notification-kafka>` for complete documentation
|
||||
on each environment variable.
|
||||
|
||||
.. tab-item:: Configuration Settings
|
||||
|
||||
MinIO supports adding or updating Kafka endpoints on a running
|
||||
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
|
||||
and the :mc-conf:`notify_kafka` configuration key. You must restart the
|
||||
:mc:`minio server` process to apply any new or updated configuration
|
||||
settings.
|
||||
|
||||
The following example code sets *all* settings related to configuring an
|
||||
Kafka service endpoint. The minimum *required* setting is
|
||||
:mc-conf:`notify_kafka brokers <notify_kafka.brokers>`:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_kafka:IDENTIFIER \
|
||||
brokers="<ENDPOINT>" \
|
||||
topic="<string>" \
|
||||
sasl_username="<string>" \
|
||||
sasl_password="<string>" \
|
||||
sasl_mechanism="<string>" \
|
||||
tls_client_auth="<string>" \
|
||||
tls="<string>" \
|
||||
tls_skip_verify="<string>" \
|
||||
client_tls_cert="<string>" \
|
||||
client_tls_key="<string>" \
|
||||
version="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
- Replace ``IDENTIFIER`` with a unique descriptive string for the
|
||||
Kafka service endpoint. The following examples in this procedure
|
||||
assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``IDENTIFIER`` matches an existing Kafka service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_kafka <mc admin config get>` to
|
||||
review the currently configured Kafka endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``ENDPOINT`` with a comma separated list of Kafka brokers.
|
||||
For example:
|
||||
|
||||
``"kafka1.example.com:2021,kafka2.example.com:2021"``
|
||||
|
||||
See :ref:`Kafka Bucket Notification Configuration Settings
|
||||
<minio-server-config-bucket-notification-kafka>` for complete
|
||||
documentation on each setting.
|
||||
|
||||
2) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured
|
||||
Kafka target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:kafka
|
||||
|
||||
You must specify the ARN resource when configuring bucket notifications with
|
||||
the associated Kafka deployment as a target.
|
||||
|
||||
.. include:: /includes/common-bucket-notifications.rst
|
||||
:start-after: start-bucket-notification-find-arn
|
||||
:end-before: end-bucket-notification-find-arn
|
||||
|
||||
|
||||
3) Configure Bucket Notifications using the Kafka Endpoint as a Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc event add` command to add a new bucket notification
|
||||
event with the configured Kafka service as a target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event add ALIAS/BUCKET arn:minio:sqs::primary:kafka \
|
||||
--event EVENTS
|
||||
|
||||
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
|
||||
- Replace ``BUCKET`` with the name of the bucket in which to configure the
|
||||
event.
|
||||
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
|
||||
<mc-event-supported-events>` for which MinIO triggers notifications.
|
||||
|
||||
Use :mc-cmd:`mc event list` to view all configured bucket events for
|
||||
a given notification target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event list ALIAS/BUCKET arn:minio:sqs::primary:kafka
|
||||
|
||||
4) Validate the Configured Events
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on the bucket for which you configured the new event and
|
||||
check the Kafka service for the notification data. The action required
|
||||
depends on which :mc-cmd:`events <mc event add --event>` were specified
|
||||
when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
||||
|
||||
Update a Kafka Endpoint in a MinIO Deployment
|
||||
---------------------------------------------
|
||||
|
||||
The following procedure updates an existing Kafka service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Kafka Minimum Versions and Supported Versions
|
||||
+++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
MinIO relies on the :github:`Shopify/sarama` project for Kafka connectivity
|
||||
and shares that project's Kafka support. See the
|
||||
``sarama`` :github:`Compatibility and API stability
|
||||
<Shopify/sarama/#compatibility-and-api-stability>` section for more details.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
|
||||
1) List Configured Kafka Endpoints In The Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config get` command to list the currently
|
||||
configured Kafka service endpoints in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config get ALIAS/ notify_kafka
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
The command output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
notify_kafka:primary tls_skip_verify="off" queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" brokers="" topic="" client_tls_cert="" client_tls_key="" version=""
|
||||
notify_kafka:secondary tls_skip_verify="off" queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" brokers="" topic="" client_tls_cert="" client_tls_key="" version=""
|
||||
|
||||
The :mc-conf:`notify_kafka` key is the top-level configuration key for an
|
||||
:ref:`minio-server-config-bucket-notification-kafka`. The
|
||||
:mc-conf:`brokers <notify_kafka.brokers>` key specifies the Kafka service
|
||||
endpoint for the given `notify_kafka` key. The ``notify_kafka:<IDENTIFIER>``
|
||||
suffix describes the unique identifier for that Kafka service endpoint.
|
||||
|
||||
Note the identifier for the Kafka service endpoint you want to update for
|
||||
the next step.
|
||||
|
||||
2) Update the Kafka Endpoint
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config set` command to set the new configuration
|
||||
for the Kafka service endpoint:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_kafka:<IDENTIFIER> \
|
||||
brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200" \
|
||||
topic="<string>" \
|
||||
sasl_username="<string>" \
|
||||
sasl_password="<string>" \
|
||||
sasl_mechanism="<string>" \
|
||||
tls_client_auth="<string>" \
|
||||
tls="<string>" \
|
||||
tls_skip_verify="<string>" \
|
||||
client_tls_cert="<string>" \
|
||||
client_tls_key="<string>" \
|
||||
version="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
The :mc-conf:`notify_kafka brokers <notify_kafka.brokers>` configuration setting
|
||||
is the *minimum* required for a Kafka service endpoint. All other configuration
|
||||
settings are *optional*. See
|
||||
:ref:`minio-server-config-bucket-notification-kafka` for a complete list of
|
||||
Kafka configuration settings.
|
||||
|
||||
3) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured
|
||||
Kafka target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:kafka
|
||||
|
||||
4) Validate the Changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on a bucket which has an event configuration using the updated
|
||||
Kafka service endpoint and check the Kafka service for the notification data.
|
||||
The action required depends on which :mc-cmd:`events <mc event add --event>` were
|
||||
specified when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
369
source/administration/monitoring/publish-events-to-mqtt.rst
Normal file
369
source/administration/monitoring/publish-events-to-mqtt.rst
Normal file
@@ -0,0 +1,369 @@
|
||||
.. _minio-bucket-notifications-publish-mqtt:
|
||||
|
||||
======================
|
||||
Publish Events to MQTT
|
||||
======================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. |ARN| replace:: ``arn:minio:sqs::primary:mqtt``
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO supports publishing :ref:`bucket notification
|
||||
<minio-bucket-notifications>` events to `MQTT <https://www.mqtt.org/>`__
|
||||
server/broker endpoint.
|
||||
|
||||
Add an MQTT Endpoint to a MinIO Deployment
|
||||
------------------------------------------
|
||||
|
||||
The following procedure adds a new MQTT service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
MQTT 3.1 or 3.1.1 Server/Broker
|
||||
+++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure assumes an existing MQTT 3.1 or 3.1.1 server/broker to which the
|
||||
MinIO deployment has connectivity. See the
|
||||
`mqtt.org software listing <https://mqtt.org/software/>`__ for a list of
|
||||
MQTT-compatible server/brokers.
|
||||
|
||||
If the MQTT service requires authentication, you *must* provide an appropriate
|
||||
username and password during the configuration process to grant MinIO access
|
||||
to the service.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
1) Add the MQTT Endpoint to MinIO
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure a new MQTT service endpoint using either environment variables
|
||||
*or* by setting runtime configuration settings.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Environment Variables
|
||||
|
||||
MinIO supports specifying the MQTT service endpoint and associated
|
||||
configuration settings using
|
||||
:ref:`environment variables
|
||||
<minio-server-envvar-bucket-notification-mqtt>`. The
|
||||
:mc:`minio server` process applies the specified settings on its
|
||||
next startup.
|
||||
|
||||
The following example code sets *all* environment variables
|
||||
related to configuring an MQTT service endpoint. The minimum *required*
|
||||
variables are:
|
||||
|
||||
- :envvar:`MINIO_NOTIFY_MQTT_ENABLE`
|
||||
- :envvar:`MINIO_NOTIFY_MQTT_BROKER`
|
||||
- :envvar:`MINIO_NOTIFY_MQTT_TOPIC`
|
||||
- :envvar:`MINIO_NOTIFY_MQTT_USERNAME` *Required if the MQTT server/broker enforces authentication/authorization*
|
||||
- :envvar:`MINIO_NOTIFY_MQTT_PASSWORD` *Required if the MQTT server/broker enforces authentication/authorization*
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
set MINIO_NOTIFY_MQTT_ENABLE_<IDENTIFIER>="on"
|
||||
set MINIO_NOTIFY_MQTT_BROKER_<IDENTIFIER>="ENDPOINT"
|
||||
set MINIO_NOTIFY_MQTT_TOPIC_<IDENTIFIER>="TOPIC"
|
||||
set MINIO_NOTIFY_MQTT_USERNAME_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MQTT_PASSWORD_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MQTT_QOS_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MQTT_KEEP_ALIVE_INTERVAL_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MQTT_RECONNECT_INTERVAL_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MQTT_QUEUE_DIR_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MQTT_QUEUE_LIMIT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MQTT_COMMENT_<IDENTIFIER>="<string>"
|
||||
|
||||
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
|
||||
MQTT service endpoint. Use the same ``<IDENTIFIER>`` value for all
|
||||
environment variables related to the new MQTT service endpoint.
|
||||
The following examples assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``<IDENTIFIER>`` matches an existing MQTT service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_mqtt <mc admin config get>` to
|
||||
review the currently configured MQTT endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the URL of the MQTT service endpoint.
|
||||
For example:
|
||||
|
||||
``tcp://hostname:port``
|
||||
|
||||
- Replace ``TOPIC`` with the MQTT topic to which MinIO associates
|
||||
events published to the server/broker.
|
||||
|
||||
See :ref:`MQTT Service for Bucket Notifications
|
||||
<minio-server-envvar-bucket-notification-mqtt>` for complete documentation
|
||||
on each environment variable.
|
||||
|
||||
.. tab-item:: Configuration Settings
|
||||
|
||||
MinIO supports adding or updating MQTT endpoints on a running
|
||||
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
|
||||
and the :mc-conf:`notify_mqtt` configuration key. You must restart the
|
||||
:mc:`minio server` process to apply any new or updated configuration
|
||||
settings.
|
||||
|
||||
The following example code sets *all* settings related to configuring an
|
||||
MQTT service endpoint. The following configuration settings are the
|
||||
*minimum* required for an MQTT server/broker endpoint:
|
||||
|
||||
- :mc-conf:`~notify_mqtt.broker`
|
||||
- :mc-conf:`~notify_mqtt.topic`
|
||||
- :mc-conf:`~notify_mqtt.username` *Required if the MQTT server/broker enforces authentication/authorization*
|
||||
- :mc-conf:`~notify_mqtt.password` *Required if the MQTT server/broker enforces authentication/authorization*
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_mqtt:IDENTIFIER \
|
||||
broker="ENDPOINT" \
|
||||
topic="TOPIC" \
|
||||
username="username" \
|
||||
password="password" \
|
||||
qos="<integer>" \
|
||||
keep_alive_interval="60s|m|h|d"
|
||||
reconnect_interval="60s|m|h|d"
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
- Replace ``IDENTIFIER`` with a unique descriptive string for the
|
||||
MQTT service endpoint. The following examples in this procedure
|
||||
assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``IDENTIFIER`` matches an existing MQTT service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_mqtt <mc admin config get>` to
|
||||
review the currently configured MQTT endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``ENDPOINT`` with the URL of the MQTT service endpoint.
|
||||
For example:
|
||||
|
||||
``tcp://hostname:port``
|
||||
|
||||
- Replace ``TOPIC`` with the MQTT topic to which MinIO associates
|
||||
events published to the server/broker.
|
||||
|
||||
See :ref:`MQTT Bucket Notification Configuration Settings
|
||||
<minio-server-config-bucket-notification-mqtt>` for complete
|
||||
documentation on each setting.
|
||||
|
||||
2) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured MQTT
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:mqtt
|
||||
|
||||
You must specify the ARN resource when configuring bucket notifications with
|
||||
the associated MQTT deployment as a target.
|
||||
|
||||
.. include:: /includes/common-bucket-notifications.rst
|
||||
:start-after: start-bucket-notification-find-arn
|
||||
:end-before: end-bucket-notification-find-arn
|
||||
|
||||
1) Configure Bucket Notifications using the MQTT Endpoint as a Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc event add` command to add a new bucket notification
|
||||
event with the configured MQTT service as a target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event add ALIAS/BUCKET arn:minio:sqs::primary:mqtt \
|
||||
--event EVENTS
|
||||
|
||||
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
|
||||
- Replace ``BUCKET`` with the name of the bucket in which to configure the
|
||||
event.
|
||||
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
|
||||
<mc-event-supported-events>` for which MinIO triggers notifications.
|
||||
|
||||
Use :mc-cmd:`mc event list` to view all configured bucket events for
|
||||
a given notification target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event list ALIAS/BUCKET arn:minio:sqs::primary:MQTT
|
||||
|
||||
4) Validate the Configured Events
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on the bucket for which you configured the new event and
|
||||
check the MQTT service for the notification data. The action required
|
||||
depends on which :mc-cmd:`events <mc event add --event>` were specified
|
||||
when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
||||
|
||||
Update an MQTT Endpoint in a MinIO Deployment
|
||||
---------------------------------------------
|
||||
|
||||
The following procedure updates an existing MQTT service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
MQTT 3.1 or 3.1.1 Server/Broker Endpoint
|
||||
++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure assumes an existing MQTT 3.1 or 3.1.1 server/broker to which the
|
||||
MinIO deployment has connectivity. See the
|
||||
`mqtt.org software listing <https://mqtt.org/software/>`__ for a list of
|
||||
MQTT-compatible server/brokers.
|
||||
|
||||
If the MQTT service requires authentication, you *must* provide an appropriate
|
||||
username and password during the configuration process to grant MinIO access
|
||||
to the service.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
|
||||
1) List Configured MQTT Endpoints In The Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config get` command to list the currently
|
||||
configured MQTT service endpoints in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config get ALIAS/ notify_mqtt
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
The command output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
notify_mqtt:primary broker="tcp://mqtt-primary.example.net:port" password="" queue_dir="" queue_limit="0" reconnect_interval="0s" keep_alive_interval="0s" qos="0" topic="" username=""
|
||||
notify_mqtt:secondary broker="tcp://mqtt-primary.example.net:port" password="" queue_dir="" queue_limit="0" reconnect_interval="0s" keep_alive_interval="0s" qos="0" topic="" username=""
|
||||
|
||||
The :mc-conf:`notify_mqtt` key is the top-level configuration key for an
|
||||
:ref:`minio-server-config-bucket-notification-mqtt`. The
|
||||
:mc-conf:`broker <notify_mqtt.broker>` key specifies the MQTT server/broker endpoint
|
||||
for the given `notify_mqtt` key. The ``notify_mqtt:<IDENTIFIER>`` suffix
|
||||
describes the unique identifier for that MQTT service endpoint.
|
||||
|
||||
Note the identifier for the MQTT service endpoint you want to update for
|
||||
the next step.
|
||||
|
||||
2) Update the MQTT Endpoint
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config set` command to set the new configuration
|
||||
for the MQTT service endpoint:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_mqtt:<IDENTIFIER> \
|
||||
url="MQTT://user:password@hostname:port" \
|
||||
exchange="<string>" \
|
||||
exchange_type="<string>" \
|
||||
routing_key="<string>" \
|
||||
mandatory="<string>" \
|
||||
durable="<string>" \
|
||||
no_wait="<string>" \
|
||||
internal="<string>" \
|
||||
auto_deleted="<string>" \
|
||||
delivery_mode="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
The following configuration settings are the *minimum* required for an
|
||||
MQTT server/broker endpoint:
|
||||
|
||||
- :mc-conf:`~notify_mqtt.broker`
|
||||
- :mc-conf:`~notify_mqtt.topic`
|
||||
- :mc-conf:`~notify_mqtt.username` *Required if the MQTT server/broker enforces authentication/authorization*
|
||||
- :mc-conf:`~notify_mqtt.password` *Required if the MQTT server/broker enforces authentication/authorization*
|
||||
|
||||
All other configuration settings are *optional*. See
|
||||
:ref:`minio-server-config-bucket-notification-mqtt` for a complete list of MQTT
|
||||
configuration settings.
|
||||
|
||||
3) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured MQTT
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:mqtt
|
||||
|
||||
3) Validate the Changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on a bucket which has an event configuration using the updated
|
||||
MQTT service endpoint and check the MQTT service for the notification data. The
|
||||
action required depends on which :mc-cmd:`events <mc event add --event>` were
|
||||
specified when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
342
source/administration/monitoring/publish-events-to-mysql.rst
Normal file
342
source/administration/monitoring/publish-events-to-mysql.rst
Normal file
@@ -0,0 +1,342 @@
|
||||
.. _minio-bucket-notifications-publish-mysql:
|
||||
|
||||
=======================
|
||||
Publish Events to MySQL
|
||||
=======================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. |ARN| replace:: ``arn:minio:sqs::primary:mysql``
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO supports publishing :ref:`bucket notification
|
||||
<minio-bucket-notifications>` events to a My`MySQL <https://www.mysql.com/>`__
|
||||
service endpoint. MinIO supports MySQL 5.7.8 and later *only*.
|
||||
|
||||
Add a MySQL Endpoint to a MinIO Deployment
|
||||
------------------------------------------
|
||||
|
||||
The following procedure adds a new MySQL service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
MySQL 5.7.8 and later
|
||||
+++++++++++++++++++++
|
||||
|
||||
MinIO relies on features introduced with MySQL 5.7.8.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
1) Add the MySQL Endpoint to MinIO
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure a new MySQL service endpoint using either environment variables
|
||||
*or* by setting runtime configuration settings.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Environment Variables
|
||||
|
||||
MinIO supports specifying the MySQL service endpoint and associated
|
||||
configuration settings using
|
||||
:ref:`environment variables
|
||||
<minio-server-envvar-bucket-notification-mysql>`. The
|
||||
:mc:`minio server` process applies the specified settings on its
|
||||
next startup.
|
||||
|
||||
The following example code sets *all* environment variables
|
||||
related to configuring a MySQL service endpoint. The minimum
|
||||
*required* variables are:
|
||||
|
||||
- :envvar:`MINIO_NOTIFY_MYSQL_ENABLE`
|
||||
- :envvar:`MINIO_NOTIFY_MYSQL_DSN_STRING`
|
||||
- :envvar:`MINIO_NOTIFY_MYSQL_TABLE`
|
||||
- :envvar:`MINIO_NOTIFY_MYSQL_FORMAT`
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
set MINIO_NOTIFY_MYSQL_DSN_STRING_<IDENTIFIER>="on"
|
||||
set MINIO_NOTIFY_MYSQL_TABLE_<IDENTIFIER>="<ENDPOINT>"
|
||||
set MINIO_NOTIFY_MYSQL_FORMAT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MYSQL_QUEUE_DIR_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MYSQL_QUEUE_LIMIT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_MYSQL_COMMENT_<IDENTIFIER>="<string>"
|
||||
|
||||
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
|
||||
MySQL service endpoint. Use the same ``<IDENTIFIER>`` value for all
|
||||
environment variables related to the new target service endpoint.
|
||||
The following examples assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``<IDENTIFIER>`` matches an existing MySQL service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_mysql <mc admin config get>` to
|
||||
review the currently configured MySQL endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the DSN of the MySQL service endpoint.
|
||||
MinIO expects the following format:
|
||||
|
||||
``<user>:<password>@tcp(<host>:<port>)/<database>``
|
||||
|
||||
For example:
|
||||
|
||||
``"username:password@tcp(mysql.example.com:3306)/miniodb"``
|
||||
|
||||
|
||||
See :ref:`MySQL Service for Bucket Notifications
|
||||
<minio-server-envvar-bucket-notification-mysql>` for complete documentation
|
||||
on each environment variable.
|
||||
|
||||
.. tab-item:: Configuration Settings
|
||||
|
||||
MinIO supports adding or updating MySQL endpoints on a running
|
||||
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
|
||||
and the :mc-conf:`notify_mysql` configuration key. You must restart the
|
||||
:mc:`minio server` process to apply any new or updated configuration
|
||||
settings.
|
||||
|
||||
The following example code sets *all* settings related to configuring an
|
||||
MySQL service endpoint. The minimum *required* settings are:
|
||||
|
||||
- :mc-conf:`notify_mysql dsn_string <notify_mysql.dsn_string>`
|
||||
- :mc-conf:`notify_mysql table <notify_mysql.table>`
|
||||
- :mc-conf:`notify_mysql format <notify_mysql.format>`
|
||||
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_mysql:IDENTIFIER \
|
||||
dsn_string="<ENDPOINT>" \
|
||||
table="<string>" \
|
||||
format="<string>" \
|
||||
max_open_connections="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
- Replace ``IDENTIFIER`` with a unique descriptive string for the
|
||||
MySQL service endpoint. The following examples in this procedure
|
||||
assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``IDENTIFIER`` matches an existing MySQL service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_mysql <mc admin config get>` to
|
||||
review the currently configured MySQL endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the DSN of the MySQL service endpoint.
|
||||
MinIO expects the following format:
|
||||
|
||||
``<user>:<password>@tcp(<host>:<port>)/<database>``
|
||||
|
||||
For example:
|
||||
|
||||
``"username:password@tcp(mysql.example.com:3306)/miniodb"``
|
||||
|
||||
See :ref:`MySQL Bucket Notification Configuration Settings
|
||||
<minio-server-config-bucket-notification-mysql>` for complete
|
||||
documentation on each setting.
|
||||
|
||||
2) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured MySQL
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:mysql
|
||||
|
||||
You must specify the ARN resource when configuring bucket notifications with
|
||||
the associated MySQL deployment as a target.
|
||||
|
||||
.. include:: /includes/common-bucket-notifications.rst
|
||||
:start-after: start-bucket-notification-find-arn
|
||||
:end-before: end-bucket-notification-find-arn
|
||||
|
||||
3) Configure Bucket Notifications using the MySQL Endpoint as a Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc event add` command to add a new bucket notification
|
||||
event with the configured MySQL service as a target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event add ALIAS/BUCKET arn:minio:sqs::primary:mysql \
|
||||
--event EVENTS
|
||||
|
||||
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
|
||||
- Replace ``BUCKET`` with the name of the bucket in which to configure the
|
||||
event.
|
||||
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
|
||||
<mc-event-supported-events>` for which MinIO triggers notifications.
|
||||
|
||||
Use :mc-cmd:`mc event list` to view all configured bucket events for
|
||||
a given notification target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event list ALIAS/BUCKET arn:minio:sqs::primary:mysql
|
||||
|
||||
4) Validate the Configured Events
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on the bucket for which you configured the new event and
|
||||
check the MySQL service for the notification data. The action required
|
||||
depends on which :mc-cmd:`events <mc event add --event>` were specified
|
||||
when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
||||
|
||||
Update a MySQL Endpoint in a MinIO Deployment
|
||||
---------------------------------------------
|
||||
|
||||
The following procedure updates an existing MySQL service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
MySQL 5.7.8 and later
|
||||
+++++++++++++++++++++
|
||||
|
||||
MinIO relies on features introduced with MySQL 5.7.8.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
|
||||
1) List Configured MySQL Endpoints In The Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config get` command to list the currently
|
||||
configured MySQL service endpoints in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config get ALIAS/ notify_mysql
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
The command output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
notify_mysql:primary format="namespace" table="minio_images" dsn_string="user:pass@tcp(mysql.example.com:3306)/miniodb"
|
||||
notify_mysql:secondary format="namespace" table="minio_images" dsn_string="user:pass@tcp(mysql.example.com:3306)/miniodb"
|
||||
|
||||
The :mc-conf:`notify_mysql` key is the top-level configuration key for an
|
||||
:ref:`minio-server-config-bucket-notification-mysql`. The
|
||||
:mc-conf:`dsn_string <notify_mysql.dsn_string>` key specifies the MySQL service
|
||||
endpoint for the given `notify_mysql` key. The ``notify_mysql:<IDENTIFIER>``
|
||||
suffix describes the unique identifier for that MySQL service endpoint.
|
||||
|
||||
Note the identifier for the MySQL service endpoint you want to update for
|
||||
the next step.
|
||||
|
||||
2) Update the MySQL Endpoint
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config set` command to set the new configuration
|
||||
for the MySQL service endpoint:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_mysql:IDENTIFIER \
|
||||
dsn_string="<ENDPOINT>" \
|
||||
table="<string>" \
|
||||
format="<string>" \
|
||||
max_open_connections="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
The following configuration settings are the *minimum required* for a MySQL
|
||||
service endpoint:
|
||||
|
||||
- :mc-conf:`notify_mysql dsn_string <notify_mysql.dsn_string>`
|
||||
- :mc-conf:`notify_mysql table <notify_mysql.table>`
|
||||
- :mc-conf:`notify_mysql format <notify_mysql.format>`
|
||||
|
||||
All other configuration settings are *optional*. See
|
||||
:ref:`minio-server-config-bucket-notification-mysql` for a complete list of
|
||||
MySQL configuration settings.
|
||||
|
||||
3) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured MySQL
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:mysql
|
||||
|
||||
4) Validate the Changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on a bucket which has an event configuration using the updated
|
||||
MySQL service endpoint and check the MySQL service for the notification data. The
|
||||
action required depends on which :mc-cmd:`events <mc event add --event>` were
|
||||
specified when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
342
source/administration/monitoring/publish-events-to-nats.rst
Normal file
342
source/administration/monitoring/publish-events-to-nats.rst
Normal file
@@ -0,0 +1,342 @@
|
||||
.. _minio-bucket-notifications-publish-nats:
|
||||
|
||||
======================
|
||||
Publish Events to NATS
|
||||
======================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. |ARN| replace:: ``arn:minio:sqs::primary:nats``
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO supports publishing :ref:`bucket notification
|
||||
<minio-bucket-notifications>` events to a
|
||||
`NATS <https://nats.io/>`__ service endpoint.
|
||||
|
||||
Add a NATS Endpoint to a MinIO Deployment
|
||||
-----------------------------------------
|
||||
|
||||
The following procedure adds a new NATS service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
1) Add the NATS Endpoint to MinIO
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure a new NATS service endpoint using either environment variables
|
||||
*or* by setting runtime configuration settings.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Environment Variables
|
||||
|
||||
MinIO supports specifying the NATS service endpoint and associated
|
||||
configuration settings using
|
||||
:ref:`environment variables
|
||||
<minio-server-envvar-bucket-notification-nats>`. The
|
||||
:mc:`minio server` process applies the specified settings on its
|
||||
next startup.
|
||||
|
||||
The following example code sets *all* environment variables
|
||||
related to configuring an NATS service endpoint. The minimum
|
||||
*required* variables are
|
||||
:envvar:`MINIO_NOTIFY_NATS_ADDRESS` and
|
||||
:envvar:`MINIO_NOTIFY_NATS_SUBJECT`:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
set MINIO_NOTIFY_NATS_ENABLE_<IDENTIFIER>="on"
|
||||
set MINIO_NOTIFY_NATS_ADDRESS_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_SUBJECT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_USERNAME_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_PASSWORD_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_TOKEN_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_TLS_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_TLS_SKIP_VERIFY_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_PING_INTERVAL_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_STREAMING_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_STREAMING_ASYNC_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_STREAMING_MAX_PUB_ACKS_IN_FLIGHT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_STREAMING_CLUSTER_ID_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_CERT_AUTHORITY_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_CLIENT_CERT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_CLIENT_KEY_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_QUEUE_DIR_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_QUEUE_LIMIT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NATS_COMMENT_<IDENTIFIER>="<string>"
|
||||
|
||||
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
|
||||
NATS service endpoint. Use the same ``<IDENTIFIER>`` value for all
|
||||
environment variables related to the new target service endpoint.
|
||||
The following examples assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``<IDENTIFIER>`` matches an existing NATS service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_nats <mc admin config get>` to
|
||||
review the currently configured NATS endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the URL of the NATS service endpoint.
|
||||
For example: ``htpps://nats-endpoint.example.com:4222``
|
||||
|
||||
See :ref:`NATS Service for Bucket Notifications
|
||||
<minio-server-envvar-bucket-notification-nats>` for complete documentation
|
||||
on each environment variable.
|
||||
|
||||
.. tab-item:: Configuration Settings
|
||||
|
||||
MinIO supports adding or updating NATS endpoints on a running
|
||||
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
|
||||
and the :mc-conf:`notify_nats` configuration key. You must restart the
|
||||
:mc:`minio server` process to apply any new or updated configuration
|
||||
settings.
|
||||
|
||||
The following example code sets *all* settings related to configuring an
|
||||
NATS service endpoint. The minimum *required* setting are
|
||||
:mc-conf:`notify_nats address <notify_nats.address>` and
|
||||
:mc-conf:`notify_nats subject <notify_nats.subject>`:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_nats:IDENTIFIER \
|
||||
address="HOSTNAME" \
|
||||
subject="<string>" \
|
||||
username="<string>" \
|
||||
password="<string>" \
|
||||
token="<string>" \
|
||||
tls="<string>" \
|
||||
tls_skip_verify="<string>" \
|
||||
ping_interval="<string>" \
|
||||
streaming="<string>" \
|
||||
streaming_async="<string>" \
|
||||
streaming_max_pub_acks_in_flight="<string>" \
|
||||
streaming_cluster_id="<string>" \
|
||||
cert_authority="<string>" \
|
||||
client_cert="<string>" \
|
||||
client_key="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
|
||||
- Replace ``IDENTIFIER`` with a unique descriptive string for the
|
||||
NATS service endpoint. The following examples in this procedure
|
||||
assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``IDENTIFIER`` matches an existing NATS service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_nats <mc admin config get>` to
|
||||
review the currently configured NATS endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``ENDPOINT`` with the URL of the NATS service endpoint.
|
||||
For example: ``htpps://nats-endpoint.example.com:4222``.
|
||||
|
||||
See :ref:`NATS Bucket Notification Configuration Settings
|
||||
<minio-server-config-bucket-notification-nats>` for complete
|
||||
documentation on each setting.
|
||||
|
||||
2) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured NATS
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:nats
|
||||
|
||||
You must specify the ARN resource when configuring bucket notifications with
|
||||
the associated NATS deployment as a target.
|
||||
|
||||
.. include:: /includes/common-bucket-notifications.rst
|
||||
:start-after: start-bucket-notification-find-arn
|
||||
:end-before: end-bucket-notification-find-arn
|
||||
|
||||
3) Configure Bucket Notifications using the NATS Endpoint as a Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc event add` command to add a new bucket notification
|
||||
event with the configured NATS service as a target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event add ALIAS/BUCKET arn:minio:sqs::primary:nats \
|
||||
--event EVENTS
|
||||
|
||||
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
|
||||
- Replace ``BUCKET`` with the name of the bucket in which to configure the
|
||||
event.
|
||||
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
|
||||
<mc-event-supported-events>` for which MinIO triggers notifications.
|
||||
|
||||
Use :mc-cmd:`mc event list` to view all configured bucket events for
|
||||
a given notification target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event list ALIAS/BUCKET arn:minio:sqs::primary:nats
|
||||
|
||||
4) Validate the Configured Events
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on the bucket for which you configured the new event and
|
||||
check the NATS service for the notification data. The action required
|
||||
depends on which :mc-cmd:`events <mc event add --event>` were specified
|
||||
when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
||||
|
||||
Update an NATS Endpoint in a MinIO Deployment
|
||||
---------------------------------------------
|
||||
|
||||
The following procedure updates an existing NATS service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
|
||||
1) List Configured NATS Endpoints In The Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config get` command to list the currently
|
||||
configured NATS service endpoints in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config get ALIAS/ notify_nats
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
The command output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
notify_nats:primary password="yoursecret" streaming_max_pub_acks_in_flight="10" subject="" address="nats-endpoint.example.com:4222" token="" username="yourusername" ping_interval="0" queue_limit="0" tls="off" tls_skip_verify="off" streaming_async="on" queue_dir="" streaming_cluster_id="test-cluster" streaming_enable="on"
|
||||
notify_nats:secondary password="yoursecret" streaming_max_pub_acks_in_flight="10" subject="" address="nats-endpoint.example.com:4222" token="" username="yourusername" ping_interval="0" queue_limit="0" tls="off" tls_skip_verify="off" streaming_async="on" queue_dir="" streaming_cluster_id="test-cluster" streaming_enable="on"
|
||||
|
||||
The :mc-conf:`notify_nats` key is the top-level configuration key for an
|
||||
:ref:`minio-server-config-bucket-notification-nats`. The
|
||||
:mc-conf:`address <notify_nats.address>` key specifies the NATS service endpoint
|
||||
for the given ``notify_nats`` key. The ``notify_nats:<IDENTIFIER>`` suffix
|
||||
describes the unique identifier for that NATS service endpoint.
|
||||
|
||||
Note the identifier for the NATS service endpoint you want to update for
|
||||
the next step.
|
||||
|
||||
2) Update the NATS Endpoint
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config set` command to set the new configuration
|
||||
for the NATS service endpoint:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_nats:IDENTIFIER \
|
||||
address="HOSTNAME" \
|
||||
subject="<string>" \
|
||||
username="<string>" \
|
||||
password="<string>" \
|
||||
token="<string>" \
|
||||
tls="<string>" \
|
||||
tls_skip_verify="<string>" \
|
||||
ping_interval="<string>" \
|
||||
streaming="<string>" \
|
||||
streaming_async="<string>" \
|
||||
streaming_max_pub_acks_in_flight="<string>" \
|
||||
streaming_cluster_id="<string>" \
|
||||
cert_authority="<string>" \
|
||||
client_cert="<string>" \
|
||||
client_key="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
The :mc-conf:`notify_nats address <notify_nats.address>` configuration setting
|
||||
is the *minimum* required for an NATS service endpoint. All other configuration
|
||||
settings are *optional*. See :ref:`minio-server-config-bucket-notification-nats`
|
||||
for a complete list of NATS configuration settings.
|
||||
|
||||
3) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured NATS
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:nats
|
||||
|
||||
3) Validate the Changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on a bucket which has an event configuration using the updated
|
||||
NATS service endpoint and check the NATS service for the notification data. The
|
||||
action required depends on which :mc-cmd:`events <mc event add --event>` were
|
||||
specified when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
288
source/administration/monitoring/publish-events-to-nsq.rst
Normal file
288
source/administration/monitoring/publish-events-to-nsq.rst
Normal file
@@ -0,0 +1,288 @@
|
||||
.. _minio-bucket-notifications-publish-nsq:
|
||||
|
||||
=====================
|
||||
Publish Events to NSQ
|
||||
=====================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. |ARN| replace:: ``arn:minio:sqs::primary:nsq``
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO supports publishing :ref:`bucket notification
|
||||
<minio-bucket-notifications>` events to `NSQ <https://nsq.io/>`__
|
||||
service endpoint.
|
||||
|
||||
Add a NSQ Endpoint to a MinIO Deployment
|
||||
----------------------------------------
|
||||
|
||||
The following procedure adds a new NSQ service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
1) Add the NSQ Endpoint to MinIO
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure a new NSQ service endpoint using either environment variables
|
||||
*or* by setting runtime configuration settings.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Environment Variables
|
||||
|
||||
MinIO supports specifying the NSQ service endpoint and associated
|
||||
configuration settings using
|
||||
:ref:`environment variables
|
||||
<minio-server-envvar-bucket-notification-nsq>`. The
|
||||
:mc:`minio server` process applies the specified settings on its
|
||||
next startup.
|
||||
|
||||
The following example code sets *all* environment variables
|
||||
related to configuring an NSQ service endpoint. The minimum
|
||||
*required* variables are
|
||||
:envvar:`MINIO_NOTIFY_NSQ_NSQD_ADDRESS` and
|
||||
:envvar:`MINIO_NOTIFY_NSQ_TOPIC`:
|
||||
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
set MINIO_NOTIFY_NSQ_ENABLE_<IDENTIFIER>="on"
|
||||
set MINIO_NOTIFY_NSQ_NSQD_ADDRESS_<IDENTIFIER>="<ENDPOINT>"
|
||||
set MINIO_NOTIFY_NSQ_TOPIC_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NSQ_TLS_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NSQ_TLS_SKIP_VERIFY_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NSQ_QUEUE_DIR_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NSQ_QUEUE_LIMIT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_NSQ_COMMENT_<IDENTIFIER>="<string>"
|
||||
|
||||
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
|
||||
TARGET service endpoint. Use the same ``<IDENTIFIER>`` value for all
|
||||
environment variables related to the new target service endpoint.
|
||||
The following examples assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``<IDENTIFIER>`` matches an existing NSQ service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_nsq <mc admin config get>` to
|
||||
review the currently configured NSQ endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the URL of the NSQ service endpoint.
|
||||
For example, ``https://nsq-service.example.com:4150``.
|
||||
|
||||
See :ref:`NSQ Service for Bucket Notifications
|
||||
<minio-server-envvar-bucket-notification-nsq>` for complete documentation
|
||||
on each environment variable.
|
||||
|
||||
.. tab-item:: Configuration Settings
|
||||
|
||||
MinIO supports adding or updating NSQ endpoints on a running
|
||||
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
|
||||
and the :mc-conf:`notify_nsq` configuration key. You must restart the
|
||||
:mc:`minio server` process to apply any new or updated configuration
|
||||
settings.
|
||||
|
||||
The following example code sets *all* settings related to configuring an
|
||||
NSQ service endpoint. The minimum *required* setting is
|
||||
:mc-conf:`notify_nsq nsqd_address <notify_nsq.nsqd_address>` and
|
||||
:mc-conf:`notify_nsq topic <notify_nsq.topic>`:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_nsq:IDENTIFIER \
|
||||
nsqd_address="ENDPOINT" \
|
||||
topic="<string>" \
|
||||
tls="<string>" \
|
||||
tls_skip_verify="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
|
||||
- Replace ``IDENTIFIER`` with a unique descriptive string for the
|
||||
NSQ service endpoint. The following examples in this procedure
|
||||
assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``IDENTIFIER`` matches an existing NSQ service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_nsq <mc admin config get>` to
|
||||
review the currently configured NSQ endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``ENDPOINT`` with the URL of the NSQ service endpoint.
|
||||
For example:
|
||||
|
||||
``NSQ://user:password@hostname:port``
|
||||
|
||||
See :ref:`NSQ Bucket Notification Configuration Settings
|
||||
<minio-server-config-bucket-notification-nsq>` for complete
|
||||
documentation on each setting.
|
||||
|
||||
2) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured NSQ
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: |ARN|
|
||||
|
||||
You must specify the ARN resource when configuring bucket notifications with the associated NSQ deployment as a target.
|
||||
|
||||
.. include:: /includes/common-bucket-notifications.rst
|
||||
:start-after: start-bucket-notification-find-arn
|
||||
:end-before: end-bucket-notification-find-arn
|
||||
|
||||
3) Configure Bucket Notifications using the NSQ Endpoint as a Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc event add` command to add a new bucket notification event with the configured NSQ service as a target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event add ALIAS/BUCKET arn:minio:sqs::primary:nsq \
|
||||
--event EVENTS
|
||||
|
||||
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
|
||||
- Replace ``BUCKET`` with the name of the bucket in which to configure the event.
|
||||
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
|
||||
<mc-event-supported-events>` for which MinIO triggers notifications.
|
||||
|
||||
Use :mc-cmd:`mc event list` to view all configured bucket events for a given notification target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event list ALIAS/BUCKET arn:minio:sqs::primary:nsq
|
||||
|
||||
4) Validate the Configured Events
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on the bucket for which you configured the new event and check the NSQ service for the notification data.
|
||||
The action required depends on which :mc-cmd:`events <mc event add --event>` were specified when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the ``s3:ObjectCreated:Put`` event, you can use the :mc-cmd:`mc cp` command to create a new object in the bucket and trigger a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
||||
|
||||
Update an NSQ Endpoint in a MinIO Deployment
|
||||
--------------------------------------------
|
||||
|
||||
The following procedure updates an existing NSQ service endpoint for supporting :ref:`bucket notifications <minio-bucket-notifications>` in a MinIO deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
|
||||
1) List Configured NSQ Endpoints In The Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config get` command to list the currently configured NSQ service endpoints in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config get ALIAS/ notify_nsq
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
The command output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
notify_nsq:primary nsqd_address="https://nsq.example.com" queue_dir="" queue_limit="0" tls="off" tls_skip_verify="off" topic=""
|
||||
notify_nsq:secondary nsqd_address="https://nsq.example.com" queue_dir="" queue_limit="0" tls="off" tls_skip_verify="off" topic=""
|
||||
|
||||
The :mc-conf:`notify_nsq` key is the top-level configuration key for an :ref:`minio-server-config-bucket-notification-nsq`.
|
||||
The :mc-conf:`nsqd_address <notify_nsq.nsqd_address>` key specifies the NSQ service endpoint for the given `notify_nsq` key.
|
||||
The ``notify_nsq:<IDENTIFIER>`` suffix describes the unique identifier for that NSQ service endpoint.
|
||||
|
||||
Note the identifier for the NSQ service endpoint you want to update for the next step.
|
||||
|
||||
2) Update the NSQ Endpoint
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config set` command to set the new configuration for the NSQ service endpoint:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_nsq:<IDENTIFIER> \
|
||||
nsqd_address="NSQ://user:password@hostname:port" \
|
||||
topic="<string>" \
|
||||
tls="<string>" \
|
||||
tls_skip_verify="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
The :mc-conf:`notify_nsq nsqd_address <notify_nsq.nsqd_address>` configuration setting is the *minimum* required for an NSQ service endpoint.
|
||||
All other configuration settings are *optional*.
|
||||
See :ref:`minio-server-config-bucket-notification-nsq` for a complete list of NSQ configuration settings.
|
||||
|
||||
3) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured NSQ target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:NSQ
|
||||
|
||||
4) Validate the Changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on a bucket which has an event configuration using the updated NSQ service endpoint and check the NSQ service for the notification data.
|
||||
The action required depends on which :mc-cmd:`events <mc event add --event>` were specified when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the ``s3:ObjectCreated:Put`` event, you can use the :mc-cmd:`mc cp` command to create a new object in the bucket and trigger a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
@@ -0,0 +1,346 @@
|
||||
.. _minio-bucket-notifications-publish-postgresql:
|
||||
|
||||
============================
|
||||
Publish Events to PostgreSQL
|
||||
============================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. |ARN| replace:: ``arn:minio:sqs::primary:postgresql``
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
.. |postgresql-uri-reference| replace:: `PostgreSQL Connection String <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING>`__
|
||||
|
||||
MinIO supports publishing :ref:`bucket notification
|
||||
<minio-bucket-notifications>` events to
|
||||
`PostgreSQL <https://www.postgresql.org/>`__. MinIO supports
|
||||
PostgreSQL 9.5 and later *only*.
|
||||
|
||||
Add a PostgreSQL Endpoint to a MinIO Deployment
|
||||
-----------------------------------------------
|
||||
|
||||
The following procedure adds a new PostgreSQL service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
PostgreSQL 9.5 and later
|
||||
++++++++++++++++++++++++
|
||||
|
||||
MinIO relies on features introduced with PostgreSQL 9.5.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
1) Add the PostgreSQL Endpoint to MinIO
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure a new PostgreSQL service endpoint using either environment
|
||||
variables *or* by setting runtime configuration settings.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Environment Variables
|
||||
|
||||
MinIO supports specifying the PostgreSQL service endpoint and associated
|
||||
configuration settings using
|
||||
:ref:`environment variables
|
||||
<minio-server-envvar-bucket-notification-postgresql>`. The
|
||||
:mc:`minio server` process applies the specified settings on its
|
||||
next startup.
|
||||
|
||||
The following example code sets *all* environment variables
|
||||
related to configuring a PostgreSQL service endpoint. The minimum
|
||||
*required* variables are:
|
||||
|
||||
- :envvar:`MINIO_NOTIFY_POSTGRESQL_CONNECTION_STRING`
|
||||
- :envvar:`MINIO_NOTIFY_POSTGRESQL_TABLE`
|
||||
- :envvar:`MINIO_NOTIFY_POSTGRESQL_FORMAT`
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
set MINIO_NOTIFY_CONNECTION_STRING_<IDENTIFIER>="on"
|
||||
set MINIO_NOTIFY_TABLE_<IDENTIFIER>="ENDPOINT"
|
||||
set MINIO_NOTIFY_FORMAT_<IDENTIFIER>=""
|
||||
set MINIO_NOTIFY_MAX_OPEN_CONNECTIONS_<IDENTIFIER>=""
|
||||
set MINIO_NOTIFY_QUEUE_DIR_<IDENTIFIER>=""
|
||||
set MINIO_NOTIFY_QUEUE_LIMIT_<IDENTIFIER>=""
|
||||
set MINIO_NOTIFY_COMMENT_<IDENTIFIER>=""
|
||||
|
||||
|
||||
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
|
||||
PostgreSQL service endpoint. Use the same ``<IDENTIFIER>`` value for all
|
||||
environment variables related to the new target service endpoint.
|
||||
The following examples assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``<IDENTIFIER>`` matches an existing PostgreSQL service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_postgresql <mc admin config get>` to
|
||||
review the currently configured PostgreSQL endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the |postgresql-uri-reference|
|
||||
for PostgreSQL service endpoint. MinIO supports ``key=value`` format for
|
||||
the connection string. For example:
|
||||
|
||||
``"host=https://postgresql.example.com port=5432 ..."``
|
||||
|
||||
For more complete documentation on supported PostgreSQL connection
|
||||
string parameters, see |postgresql-uri-reference|.
|
||||
|
||||
See :ref:`PostgreSQL Service for Bucket Notifications
|
||||
<minio-server-envvar-bucket-notification-postgresql>` for complete
|
||||
documentation on each environment variable.
|
||||
|
||||
.. tab-item:: Configuration Settings
|
||||
|
||||
MinIO supports adding or updating PostgreSQL endpoints on a running
|
||||
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
|
||||
and the :mc-conf:`notify_postgresql` configuration key. You must restart the
|
||||
:mc:`minio server` process to apply any new or updated configuration
|
||||
settings.
|
||||
|
||||
The following example code sets *all* settings related to configuring an
|
||||
PostgreSQL service endpoint. The minimum *required* setting are:
|
||||
|
||||
- :mc-conf:`notify_postgresql connection_string
|
||||
<notify_postgresql.connection_string>`
|
||||
- :mc-conf:`notify_postgresql table <notify_postgresql.table>`
|
||||
- :mc-conf:`notify_postgresql format <notify_postgresql.format>`
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_postgresql:IDENTIFIER \
|
||||
connection_string="ENDPOINT" \
|
||||
table="<string>" \
|
||||
format="<string>" \
|
||||
max_open_connections="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
- Replace ``IDENTIFIER`` with a unique descriptive string for the
|
||||
PostgreSQL service endpoint. The following examples in this procedure
|
||||
assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``IDENTIFIER`` matches an existing PostgreSQL service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_postgresql <mc admin config get>` to
|
||||
review the currently configured PostgreSQL endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the `PostgreSQL URI connection string
|
||||
<https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING>`__
|
||||
of the PostgreSQL service endpoint. MinIO supports ``key=value`` format
|
||||
for the PostgreSQL connection string. For example:
|
||||
|
||||
``"host=https://postgresql.example.com port=5432 ..."``
|
||||
|
||||
For more complete documentation on supported PostgreSQL connection
|
||||
string parameters, see |postgresql-uri-reference|.
|
||||
|
||||
See :ref:`PostgreSQL Bucket Notification Configuration Settings
|
||||
<minio-server-config-bucket-notification-postgresql>` for complete
|
||||
documentation on each setting.
|
||||
|
||||
2) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured PostgreSQL
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:postgresql
|
||||
|
||||
You must specify the ARN resource when configuring bucket notifications with
|
||||
the associated PostgreSQL deployment as a target.
|
||||
|
||||
.. include:: /includes/common-bucket-notifications.rst
|
||||
:start-after: start-bucket-notification-find-arn
|
||||
:end-before: end-bucket-notification-find-arn
|
||||
|
||||
3) Configure Bucket Notifications using the PostgreSQL Endpoint as a Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc event add` command to add a new bucket notification
|
||||
event with the configured PostgreSQL service as a target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event add ALIAS/BUCKET arn:minio:sqs::primary:postgresql \
|
||||
--event EVENTS
|
||||
|
||||
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
|
||||
- Replace ``BUCKET`` with the name of the bucket in which to configure the ßevent.
|
||||
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
|
||||
<mc-event-supported-events>` for which MinIO triggers notifications.
|
||||
|
||||
Use :mc-cmd:`mc event list` to view all configured bucket events for
|
||||
a given notification target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event list ALIAS/BUCKET arn:minio:sqs::primary:postgresql
|
||||
|
||||
4) Validate the Configured Events
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on the bucket for which you configured the new event and
|
||||
check the PostgreSQL service for the notification data. The action required
|
||||
depends on which :mc-cmd:`events <mc event add --event>` were specified
|
||||
when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
||||
|
||||
Update a PostgreSQL Endpoint in a MinIO Deployment
|
||||
---------------------------------------------------
|
||||
|
||||
The following procedure updates an existing PostgreSQL service endpoint for
|
||||
supporting :ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
PostgreSQL 9.5 and later
|
||||
++++++++++++++++++++++++
|
||||
|
||||
MinIO relies on features introduced with PostgreSQL 9.5.
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
|
||||
1) List Configured PostgreSQL Endpoints In The Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config get` command to list the currently
|
||||
configured PostgreSQL service endpoints in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config get ALIAS/ notify_postgresql
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
The command output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
notify_postgresql:primary queue_dir="" connection_string="postgresql://" queue_limit="0" table="" format="namespace"
|
||||
notify_postgresql:secondary queue_dir="" connection_string="" queue_limit="0" table="" format="namespace"
|
||||
|
||||
The :mc-conf:`notify_postgresql` key is the top-level configuration key for an
|
||||
:ref:`minio-server-config-bucket-notification-postgresql`. The
|
||||
:mc-conf:`connection_string <notify_postgresql.connection_string>` key specifies
|
||||
the PostgreSQL service endpoint for the given `notify_postgresql` key. The
|
||||
``notify_postgresql:<IDENTIFIER>`` suffix describes the unique identifier for
|
||||
that PostgreSQL service endpoint.
|
||||
|
||||
Note the identifier for the PostgreSQL service endpoint you want to update for
|
||||
the next step.
|
||||
|
||||
2) Update the PostgreSQL Endpoint
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config set` command to set the new configuration
|
||||
for the PostgreSQL service endpoint:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_postgresql:IDENTIFIER \
|
||||
connection_string="ENDPOINT" \
|
||||
table="<string>" \
|
||||
format="<string>" \
|
||||
max_open_connections="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
The following configuration settings are the *minimum* required for a
|
||||
PostgreSQL service endpoint:
|
||||
|
||||
- :mc-conf:`notify_postgresql connection_string
|
||||
<notify_postgresql.connection_string>`
|
||||
- :mc-conf:`notify_postgresql table <notify_postgresql.table>`
|
||||
- :mc-conf:`notify_postgresql format <notify_postgresql.format>`
|
||||
|
||||
All other configuration settings are *optional*. See
|
||||
:ref:`minio-server-config-bucket-notification-postgresql` for a complete list of
|
||||
PostgreSQL configuration settings.
|
||||
|
||||
3) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured PostgreSQL
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:postgresql
|
||||
|
||||
4) Validate the Changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on a bucket which has an event configuration using the updated
|
||||
PostgreSQL service endpoint and check the PostgreSQL service for the notification data. The
|
||||
action required depends on which :mc-cmd:`events <mc event add --event>` were
|
||||
specified when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
315
source/administration/monitoring/publish-events-to-redis.rst
Normal file
315
source/administration/monitoring/publish-events-to-redis.rst
Normal file
@@ -0,0 +1,315 @@
|
||||
.. _minio-bucket-notifications-publish-redis:
|
||||
|
||||
=======================
|
||||
Publish Events to Redis
|
||||
=======================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. |ARN| replace:: ``arn:minio:sqs::primary:redis``
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO supports publishing :ref:`bucket notification
|
||||
<minio-bucket-notifications>` events to a `Redis <https://redis.io/>`__
|
||||
service endpoint.
|
||||
|
||||
Add a Redis Endpoint to a MinIO Deployment
|
||||
-------------------------------------------
|
||||
|
||||
The following procedure adds a new Redis service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
1) Add the Redis Endpoint to MinIO
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure a new Redis service endpoint using either environment variables
|
||||
*or* by setting runtime configuration settings.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Environment Variables
|
||||
|
||||
MinIO supports specifying the Redis service endpoint and associated
|
||||
configuration settings using
|
||||
:ref:`environment variables
|
||||
<minio-server-envvar-bucket-notification-redis>`. The
|
||||
:mc:`minio server` process applies the specified settings on its
|
||||
next startup.
|
||||
|
||||
The following example code sets *all* environment variables
|
||||
related to configuring an Redis service endpoint. The minimum
|
||||
*required* variables are:
|
||||
|
||||
- :envvar:`MINIO_NOTIFY_REDIS_ENABLE`
|
||||
- :envvar:`MINIO_NOTIFY_REDIS_ADDRESS`
|
||||
- :envvar:`MINIO_NOTIFY_REDIS_KEY`
|
||||
- :envvar:`MINIO_NOTIFY_REDIS_FORMAT`
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
set MINIO_NOTIFY_REDIS_ENABLE_<IDENTIFIER>="on"
|
||||
set MINIO_NOTIFY_REDIS_ADDRESS_<IDENTIFIER>="<ENDPOINT>"
|
||||
set MINIO_NOTIFY_REDIS_KEY_<IDENTIFIER>="<STRING>"
|
||||
set MINIO_NOTIFY_REDIS_FORMAT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_REDIS_PASSWORD_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_REDIS_QUEUE_DIR_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_REDIS_QUEUE_LIMIT_<IDENTIFIER>="<string>"
|
||||
set MINIO_NOTIFY_REDIS_COMMENT_<IDENTIFIER>="<string>"
|
||||
|
||||
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
|
||||
TARGET service endpoint. Use the same ``<IDENTIFIER>`` value for all
|
||||
environment variables related to the new target service endpoint.
|
||||
The following examples assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``<IDENTIFIER>`` matches an existing Redis service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_redis <mc admin config get>` to
|
||||
review the currently configured Redis endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the URL of the Redis service endpoint.
|
||||
For example: ``https://redis.example.com:6369``
|
||||
|
||||
|
||||
See :ref:`Redis Service for Bucket Notifications
|
||||
<minio-server-envvar-bucket-notification-redis>` for complete documentation
|
||||
on each environment variable.
|
||||
|
||||
.. tab-item:: Configuration Settings
|
||||
|
||||
MinIO supports adding or updating Redis endpoints on a running
|
||||
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
|
||||
and the :mc-conf:`notify_redis` configuration key. You must restart the
|
||||
:mc:`minio server` process to apply any new or updated configuration
|
||||
settings.
|
||||
|
||||
The following example code sets *all* settings related to configuring an
|
||||
Redis service endpoint. The minimum *required* settings are:
|
||||
|
||||
- :mc-conf:`notify_redis address <notify_redis.address>`
|
||||
- :mc-conf:`notify_redis key <notify_redis.key>`
|
||||
- :mc-conf:`notify_redis format <notify_redis.format>`
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_redis:IDENTIFIER \
|
||||
address="ENDPOINT" \
|
||||
key="<string>" \
|
||||
format="<string>" \
|
||||
password="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
- Replace ``IDENTIFIER`` with a unique descriptive string for the
|
||||
Redis service endpoint. The following examples in this procedure
|
||||
assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``IDENTIFIER`` matches an existing Redis service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_redis <mc admin config get>` to
|
||||
review the currently configured Redis endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``ENDPOINT`` with the URL of the Redis service endpoint.
|
||||
For example: ``https://redis.example.com:6369``
|
||||
|
||||
See :ref:`Redis Bucket Notification Configuration Settings
|
||||
<minio-server-config-bucket-notification-redis>` for complete
|
||||
documentation on each setting.
|
||||
|
||||
2) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured Redis
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:redis
|
||||
|
||||
You must specify the ARN resource when configuring bucket notifications with
|
||||
the associated Redis deployment as a target.
|
||||
|
||||
.. include:: /includes/common-bucket-notifications.rst
|
||||
:start-after: start-bucket-notification-find-arn
|
||||
:end-before: end-bucket-notification-find-arn
|
||||
|
||||
3) Configure Bucket Notifications using the Redis Endpoint as a Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc event add` command to add a new bucket notification
|
||||
event with the configured Redis service as a target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event add ALIAS/BUCKET arn:minio:sqs::primary:redis \
|
||||
--event EVENTS
|
||||
|
||||
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
|
||||
- Replace ``BUCKET`` with the name of the bucket in which to configure the
|
||||
event.
|
||||
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
|
||||
<mc-event-supported-events>` for which MinIO triggers notifications.
|
||||
|
||||
Use :mc-cmd:`mc event list` to view all configured bucket events for
|
||||
a given notification target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event list ALIAS/BUCKET arn:minio:sqs::primary:redis
|
||||
|
||||
4) Validate the Configured Events
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on the bucket for which you configured the new event and
|
||||
check the Redis service for the notification data. The action required
|
||||
depends on which :mc-cmd:`events <mc event add --event>` were specified
|
||||
when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
||||
|
||||
Update an Redis Endpoint in a MinIO Deployment
|
||||
----------------------------------------------
|
||||
|
||||
The following procedure updates an existing Redis service endpoint for
|
||||
supporting :ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
|
||||
1) List Configured Redis Endpoints In The Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config get` command to list the currently
|
||||
configured Redis service endpoints in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config get ALIAS/ notify_redis
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
The command output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
notify_redis:primary address="https://redis.example.com:6369" format="namespace" key="minioevent" password="" queue_dir="" queue_limit="0"
|
||||
notify_redis:secondary address="https://redis.example.com:6369" format="namespace" key="minioevent" password="" queue_dir="" queue_limit="0"
|
||||
|
||||
The :mc-conf:`notify_redis` key is the top-level configuration key for an
|
||||
:ref:`minio-server-config-bucket-notification-redis`. The
|
||||
:mc-conf:`address <notify_redis.address>` key specifies the Redis service endpoint
|
||||
for the given `notify_redis` key. The ``notify_redis:<IDENTIFIER>`` suffix
|
||||
describes the unique identifier for that Redis service endpoint.
|
||||
|
||||
Note the identifier for the Redis service endpoint you want to update for
|
||||
the next step.
|
||||
|
||||
2) Update the Redis Endpoint
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config set` command to set the new configuration
|
||||
for the Redis service endpoint:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_redis:IDENTIFIER \
|
||||
address="ENDPOINT" \
|
||||
key="<string>" \
|
||||
format="<string>" \
|
||||
password="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
The :mc-conf:`notify_redis address <notify_redis.address>` configuration setting
|
||||
is the *minimum* required for an Redis service endpoint. All other configuration
|
||||
settings are *optional*. See
|
||||
:ref:`minio-server-config-bucket-notification-redis` for a complete list of
|
||||
Redis configuration settings.
|
||||
|
||||
3) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured Redis
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:redis
|
||||
|
||||
4) Validate the Changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on a bucket which has an event configuration using the updated
|
||||
Redis service endpoint and check the Redis service for the notification data. The
|
||||
action required depends on which :mc-cmd:`events <mc event add --event>` were
|
||||
specified when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
313
source/administration/monitoring/publish-events-to-webhook.rst
Normal file
313
source/administration/monitoring/publish-events-to-webhook.rst
Normal file
@@ -0,0 +1,313 @@
|
||||
.. _minio-bucket-notifications-publish-webhook:
|
||||
|
||||
=========================
|
||||
Publish Events to Webhook
|
||||
=========================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. |ARN| replace:: ``arn:minio:sqs::primary:webhook``
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO supports publishing :ref:`bucket notification
|
||||
<minio-bucket-notifications>` events to a
|
||||
`Webhook <https://en.wikipedia.org/wiki/Webhook>`__ service endpoint.
|
||||
|
||||
Add a Webhook Endpoint to a MinIO Deployment
|
||||
--------------------------------------------
|
||||
|
||||
The following procedure adds a new Webhook service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
1) Add the Webhook Endpoint to MinIO
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure a new Webhook service endpoint using either environment
|
||||
variables *or* by setting runtime configuration settings.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Environment Variables
|
||||
|
||||
MinIO supports specifying the Webhook service endpoint and associated
|
||||
configuration settings using
|
||||
:ref:`environment variables
|
||||
<minio-server-envvar-bucket-notification-webhook>`. The
|
||||
:mc:`minio server` process applies the specified settings on its
|
||||
next startup.
|
||||
|
||||
The following example code sets *all* environment variables
|
||||
related to configuring an Webhook service endpoint. The minimum
|
||||
*required* variables are
|
||||
:envvar:`MINIO_NOTIFY_WEBHOOK_ENABLE` and
|
||||
:envvar:`MINIO_NOTIFY_WEBHOOK_ENDPOINT`:
|
||||
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
set MINIO_NOTIFY_WEBHOOK_ENABLE_<IDENTIFIER>_="on"
|
||||
set MINIO_NOTIFY_WEBHOOK_ENDPOINT_<IDENTIFIER>_="ENDPOINT"
|
||||
set MINIO_NOTIFY_WEBHOOK_AUTH_TOKEN_<IDENTIFIER>_="<string>"
|
||||
set MINIO_NOTIFY_WEBHOOK_QUEUE_DIR_<IDENTIFIER>_="<string>"
|
||||
set MINIO_NOTIFY_WEBHOOK_QUEUE_LIMIT_<IDENTIFIER>_="<string>"
|
||||
set MINIO_NOTIFY_WEBHOOK_CLIENT_CERT_<IDENTIFIER>_="<string>"
|
||||
set MINIO_NOTIFY_WEBHOOK_CLIENT_KEY_<IDENTIFIER>_="<string>"
|
||||
set MINIO_NOTIFY_WEBHOOK_COMMENT_<IDENTIFIER>_="<string>"
|
||||
|
||||
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
|
||||
Webhook service endpoint. Use the same ``<IDENTIFIER>`` value for all
|
||||
environment variables related to the new target service endpoint.
|
||||
The following examples assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``<IDENTIFIER>`` matches an existing Webhook service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_webhook <mc admin config get>` to
|
||||
review the currently configured Webhook endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``<ENDPOINT>`` with the URL of the Webhook service endpoint.
|
||||
For example:
|
||||
|
||||
``https://webhook.example.com``
|
||||
|
||||
See :ref:`Webhook Service for Bucket Notifications
|
||||
<minio-server-envvar-bucket-notification-webhook>` for complete documentation
|
||||
on each environment variable.
|
||||
|
||||
.. tab-item:: Configuration Settings
|
||||
|
||||
MinIO supports adding or updating Webhook endpoints on a running
|
||||
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
|
||||
and the :mc-conf:`notify_webhook` configuration key. You must restart the
|
||||
:mc:`minio server` process to apply any new or updated configuration
|
||||
settings.
|
||||
|
||||
The following example code sets *all* settings related to configuring an
|
||||
Webhook service endpoint. The minimum *required* setting is
|
||||
:mc-conf:`notify_webhook endpoint <notify_webhook.endpoint>`:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_webhook:IDENTIFIER \
|
||||
endpoint="<ENDPOINT>" \
|
||||
auth_token="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
client_cert="<string>" \
|
||||
client_key="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
- Replace ``IDENTIFIER`` with a unique descriptive string for the
|
||||
Webhook service endpoint. The following examples in this procedure
|
||||
assume an identifier of ``PRIMARY``.
|
||||
|
||||
If the specified ``IDENTIFIER`` matches an existing Webhook service
|
||||
endpoint on the MinIO deployment, the new settings *override*
|
||||
any existing settings for that endpoint. Use
|
||||
:mc-cmd:`mc admin config get notify_webhook <mc admin config get>` to
|
||||
review the currently configured Webhook endpoints on the MinIO deployment.
|
||||
|
||||
- Replace ``ENDPOINT`` with the URL of the Webhook service endpoint.
|
||||
For example:
|
||||
|
||||
``https://webhook.example.com``
|
||||
|
||||
See :ref:`Webhook Bucket Notification Configuration Settings
|
||||
<minio-server-config-bucket-notification-webhook>` for complete
|
||||
documentation on each setting.
|
||||
|
||||
2) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured
|
||||
Webhook target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:webhook
|
||||
|
||||
You must specify the ARN resource when configuring bucket notifications with
|
||||
the associated Webhook deployment as a target.
|
||||
|
||||
.. include:: /includes/common-bucket-notifications.rst
|
||||
:start-after: start-bucket-notification-find-arn
|
||||
:end-before: end-bucket-notification-find-arn
|
||||
|
||||
3) Configure Bucket Notifications using the Webhook Endpoint as a Target
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc event add` command to add a new bucket notification
|
||||
event with the configured Webhook service as a target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event add ALIAS/BUCKET arn:minio:sqs::primary:webhook \
|
||||
--event EVENTS
|
||||
|
||||
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
|
||||
- Replace ``BUCKET`` with the name of the bucket in which to configure the
|
||||
event.
|
||||
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
|
||||
<mc-event-supported-events>` for which MinIO triggers notifications.
|
||||
|
||||
Use :mc-cmd:`mc event list` to view all configured bucket events for
|
||||
a given notification target:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc event list ALIAS/BUCKET arn:minio:sqs::primary:webhook
|
||||
|
||||
4) Validate the Configured Events
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on the bucket for which you configured the new event and
|
||||
check the Webhook service for the notification data. The action required
|
||||
depends on which :mc-cmd:`events <mc event add --event>` were specified
|
||||
when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
||||
|
||||
Update an Webhook Endpoint in a MinIO Deployment
|
||||
------------------------------------------------
|
||||
|
||||
The following procedure updates an existing Webhook service endpoint for supporting
|
||||
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
|
||||
deployment.
|
||||
|
||||
Prerequisites
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
MinIO ``mc`` Command Line Tool
|
||||
++++++++++++++++++++++++++++++
|
||||
|
||||
This procedure uses the :mc:`mc` command line tool for certain actions.
|
||||
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
|
||||
|
||||
|
||||
1) List Configured Webhook Endpoints In The Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config get` command to list the currently
|
||||
configured Webhook service endpoints in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config get ALIAS/ notify_webhook
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
The command output resembles the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
notify_webhook:primary endpoint="https://webhook.example.com" auth_token="" queue_limit="0" queue_dir="" client_cert="" client_key=""
|
||||
notify_webhook:secondary endpoint="https://webhook.example.com" auth_token="" queue_limit="0" queue_dir="" client_cert="" client_key=""
|
||||
|
||||
The :mc-conf:`notify_webhook` key is the top-level configuration key for an
|
||||
:ref:`minio-server-config-bucket-notification-webhook`. The
|
||||
:mc-conf:`endpoint <notify_webhook.endpoint>` key specifies the Webhook service
|
||||
endpoint for the given `notify_webhook` key. The ``notify_webhook:<IDENTIFIER>``
|
||||
suffix describes the unique identifier for that Webhook service endpoint.
|
||||
|
||||
Note the identifier for the Webhook service endpoint you want to update for
|
||||
the next step.
|
||||
|
||||
2) Update the Webhook Endpoint
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin config set` command to set the new configuration
|
||||
for the Webhook service endpoint:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin config set ALIAS/ notify_webhook:IDENTIFIER \
|
||||
endpoint="<ENDPOINT>" \
|
||||
auth_token="<string>" \
|
||||
queue_dir="<string>" \
|
||||
queue_limit="<string>" \
|
||||
client_cert="<string>" \
|
||||
client_key="<string>" \
|
||||
comment="<string>"
|
||||
|
||||
The :mc-conf:`notify_webhook endpoint <notify_webhook.endpoint>` configuration
|
||||
setting is the *minimum* required for an Webhook service endpoint. All other
|
||||
configuration settings are *optional*. See
|
||||
:ref:`minio-server-config-bucket-notification-webhook` for a complete list of
|
||||
Webhook configuration settings.
|
||||
|
||||
3) Restart the MinIO Deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
The :mc:`minio server` process prints a line on startup for each configured Webhook
|
||||
target similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
SQS ARNs: arn:minio:sqs::primary:webhook
|
||||
|
||||
4) Validate the Changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Perform an action on a bucket which has an event configuration using the updated
|
||||
Webhook service endpoint and check the Webhook service for the notification
|
||||
data. The action required depends on which :mc-cmd:`events <mc event add --event>`
|
||||
were specified when configuring the bucket notification.
|
||||
|
||||
For example, if the bucket notification configuration includes the
|
||||
``s3:ObjectCreated:Put`` event, you can use the
|
||||
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
|
||||
a notification.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/new-object.txt ALIAS/BUCKET
|
102
source/administration/object-management.rst
Normal file
102
source/administration/object-management.rst
Normal file
@@ -0,0 +1,102 @@
|
||||
=================
|
||||
Object Management
|
||||
=================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
.. _objects:
|
||||
|
||||
An :ref:`object <objects>` is binary data, sometimes referred to as a Binary
|
||||
Large OBject (BLOB). Blobs can be images, audio files, spreadsheets, or even
|
||||
binary executable code. Object Storage platforms like MinIO provide dedicated
|
||||
tools and capabilities for storing, retrieving, and searching for blobs.
|
||||
|
||||
.. _buckets:
|
||||
|
||||
MinIO Object Storage uses :ref:`buckets <buckets>` to organize objects.
|
||||
A bucket is similar to a folder or directory in a filesystem, where each
|
||||
bucket can hold an arbitrary number of objects. MinIO buckets provide the
|
||||
same functionality as AWS S3 buckets.
|
||||
|
||||
For example, consider an application that hosts a web blog. The application
|
||||
needs to store a variety of blobs, including rich multimedia like videos and
|
||||
images. The structure of objects on the MinIO server might look similar to the
|
||||
following:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
/ #root
|
||||
/images/
|
||||
2020-01-02-MinIO-Diagram.png
|
||||
2020-01-03-MinIO-Advanced-Deployment.png
|
||||
MinIO-Logo.png
|
||||
/videos/
|
||||
2020-01-04-MinIO-Interview.mp4
|
||||
/articles/
|
||||
/john.doe/
|
||||
2020-01-02-MinIO-Object-Storage.md
|
||||
2020-01-02-MinIO-Object-Storage-comments.json
|
||||
/jane.doe/
|
||||
2020-01-03-MinIO-Advanced-Deployment.png
|
||||
2020-01-02-MinIO-Advanced-Deployment-comments.json
|
||||
2020-01-04-MinIO-Interview.md
|
||||
|
||||
MinIO supports multiple levels of nested directories and objects to support
|
||||
even the most dynamic object storage workloads.
|
||||
|
||||
Object Versioning
|
||||
-----------------
|
||||
|
||||
MinIO supports keeping multiple "versions" of an object in a single bucket.
|
||||
Write operations which would normally overwrite an existing object instead result in the creation of a new versioned object.
|
||||
|
||||
.. image:: /images/retention/minio-versioning-multiple-versions.svg
|
||||
:alt: Object with Multiple Versions
|
||||
:align: center
|
||||
|
||||
MinIO versioning protects from unintended overwrites and deletions while providing support for "undoing" a write operation.
|
||||
|
||||
See :ref:`minio-bucket-versioning` for more complete documentation.
|
||||
|
||||
Object Retention
|
||||
----------------
|
||||
|
||||
MinIO Object Locking ("Object Retention") enforces Write-Once Read-Many (WORM) immutability to protect :ref:`versioned objects <minio-bucket-versioning>` from deletion.
|
||||
MinIO supports both :ref:`duration based object retention <minio-object-locking-retention-modes>` and :ref:`indefinite Legal Hold retention <minio-object-locking-legalhold>`.
|
||||
|
||||
.. image:: /images/retention/minio-object-locking.svg
|
||||
:alt: 30 Day Locked Objects
|
||||
:align: center
|
||||
:width: 600px
|
||||
|
||||
See :ref:`minio-object-locking` for more complete documentation.
|
||||
|
||||
MinIO Object Locking provides key data retention compliance and meets SEC17a-4(f), FINRA 4511(C), and CFTC 1.31(c)-(d) requirements as per `Cohasset Associates <https://min.io/cohasset?ref=docs>`__.
|
||||
|
||||
Object Lifecycle Management
|
||||
---------------------------
|
||||
|
||||
MinIO Object Lifecycle Management allows creating rules for time or date
|
||||
based automatic transition or expiry of objects. For object transition,
|
||||
MinIO automatically moves the object to a configured remote storage
|
||||
tier. For object expiry, MinIO automatically deletes the object.
|
||||
|
||||
MinIO lifecycle management is built for behavior and syntax compatibility with
|
||||
:s3-docs:`AWS S3 Lifecycle Management <object-lifecycle-mgmt.html>`. For
|
||||
example, you can export S3 lifecycle management rules and import them into
|
||||
MinIO or vice-versa. MinIO uses JSON to describe lifecycle management rules,
|
||||
and conversion to or from XML may be required.
|
||||
|
||||
See :ref:`minio-lifecycle-management` for more complete documentation.
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
:hidden:
|
||||
|
||||
/administration/object-management/object-versioning
|
||||
/administration/object-management/object-retention
|
||||
/administration/object-management/object-lifecycle-management
|
@@ -0,0 +1,112 @@
|
||||
.. _minio-lifecycle-management-create-expiry-rule:
|
||||
|
||||
===========================
|
||||
Automatic Object Expiration
|
||||
===========================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
Each procedure on this page creates a new object lifecycle management rule that
|
||||
expires objects on a MinIO bucket. This procedure supports use cases like
|
||||
removing "old" objects after a certain time period or calendar date.
|
||||
|
||||
.. todo: diagram
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
Install and Configure ``mc``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This procedure uses :mc:`mc` for performing operations on the MinIO cluster.
|
||||
Install :mc:`mc` on a machine with network access to both source and destination
|
||||
clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
|
||||
instructions on downloading and installing ``mc``.
|
||||
|
||||
Use the :mc:`mc alias` command to create an alias for the source MinIO cluster
|
||||
and the destination S3-compatible service. Alias creation requires specifying an
|
||||
access key for a user on the source and destination clusters. The specified
|
||||
users must have :ref:`permissions
|
||||
<minio-lifecycle-management-create-expiry-rule-permissions>` for configuring
|
||||
and applying expiry operations.
|
||||
|
||||
.. _minio-lifecycle-management-create-expiry-rule-permissions:
|
||||
|
||||
Required Permissions
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO requires the following permissions scoped to the bucket or buckets
|
||||
for which you are creating lifecycle management rules.
|
||||
|
||||
- :policy-action:`s3:PutLifecycleConfiguration`
|
||||
- :policy-action:`s3:GetLifecycleConfiguration`
|
||||
|
||||
MinIO also requires the following administrative permissions on the cluster
|
||||
in which you are creating remote tiers for object transition lifecycle
|
||||
management rules:
|
||||
|
||||
- :policy-action:`admin:SetTier`
|
||||
- :policy-action:`admin:ListTier`
|
||||
|
||||
For example, the following policy provides permission for configuring object
|
||||
transition lifecycle management rules on any bucket in the cluster:.
|
||||
|
||||
.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json
|
||||
:language: json
|
||||
:class: copyable
|
||||
|
||||
Expire Objects after Number of Days
|
||||
-----------------------------------
|
||||
|
||||
Use :mc-cmd:`mc ilm add` with :mc-cmd:`~mc ilm add --expiry-days` to
|
||||
expire bucket contents a number of days after object creation:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc ilm add ALIAS/PATH --expiry-days "DAYS"
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc ilm add ALIAS>` with the
|
||||
:mc:`alias <mc alias>` of the S3-compatible host.
|
||||
|
||||
- Replace :mc-cmd:`PATH <mc ilm add ALIAS>` with the path to the bucket on the
|
||||
S3-compatible host.
|
||||
|
||||
- Replace :mc-cmd:`DAYS <mc ilm add --expiry-days>` with the number of days after
|
||||
which to expire the object. For example, specify ``30`` to expire the
|
||||
object 30 days after creation.
|
||||
|
||||
Expire Versioned Objects
|
||||
------------------------
|
||||
|
||||
Use :mc-cmd:`mc ilm add` to expiring noncurrent object versions and object
|
||||
delete markers:
|
||||
|
||||
- To expire noncurrent object versions after a specific duration in days,
|
||||
include :mc-cmd:`~mc ilm add --noncurrentversion-expiration-days`.
|
||||
|
||||
- To expire delete markers for objects with no remaining versions,
|
||||
include :mc-cmd:`~mc ilm add --expired-object-delete-marker`.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc ilm add ALIAS/PATH \
|
||||
--noncurrentversion-expiration-days NONCURRENT_DAYS \
|
||||
--expired-object-delete-marker
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc ilm add ALIAS>` with the
|
||||
:mc:`alias <mc alias>` of the S3-compatible host.
|
||||
|
||||
- Replace :mc-cmd:`PATH <mc ilm add ALIAS>` with the path to the bucket on the
|
||||
S3-compatible host.
|
||||
|
||||
- Replace :mc-cmd:`NONCURRENT_DAYS
|
||||
<mc ilm add --noncurrentversion-expiration-days>` with the number of days after
|
||||
which to expire noncurrent object versions. For example, specify ``30d`` to
|
||||
expire a version after it has been noncurrent for at least 30 days.
|
||||
|
@@ -0,0 +1,144 @@
|
||||
.. _minio-lifecycle-management:
|
||||
|
||||
===========================
|
||||
Object Lifecycle Management
|
||||
===========================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
MinIO Object Lifecycle Management allows creating rules for time or date
|
||||
based automatic transition or expiry of objects. For object transition,
|
||||
MinIO automatically moves the object to a configured remote storage
|
||||
tier. For object expiry, MinIO automatically deletes the object.
|
||||
|
||||
MinIO lifecycle management is built for behavior and syntax compatibility with
|
||||
:s3-docs:`AWS S3 Lifecycle Management <object-lifecycle-mgmt.html>`. For
|
||||
example, you can export S3 lifecycle management rules and import them into
|
||||
MinIO or vice-versa. MinIO uses JSON to describe lifecycle management rules,
|
||||
and conversion to or from XML may be required.
|
||||
|
||||
.. _minio-lifecycle-management-tiering:
|
||||
|
||||
Object Transition ("Tiering")
|
||||
-----------------------------
|
||||
|
||||
MinIO supports creating object transition lifecycle management rules, where
|
||||
MinIO can automatically move an object to a remote storage "tier". MinIO
|
||||
supports any S3-compatible service as a remote tier *in addition to* the
|
||||
following public cloud storage services:
|
||||
|
||||
- :ref:`Amazon S3 <minio-lifecycle-management-transition-to-s3>`
|
||||
- :ref:`Google Cloud Storage <minio-lifecycle-management-transition-to-gcs>`
|
||||
- :ref:`Microsoft Azure Blob Storage
|
||||
<minio-lifecycle-management-transition-to-azure>`
|
||||
|
||||
MinIO object transition supports use cases like moving aged data from MinIO
|
||||
clusters in private or public cloud infrastructure to low-cost private or public
|
||||
cloud storage solutions. MinIO manages retrieving tiered objects on-the-fly
|
||||
without any additional application-side logic.
|
||||
|
||||
Use the :mc-cmd:`mc admin tier` command to create a remote target for tiering
|
||||
data to a supported Cloud Service Provider object storage. You can then use the
|
||||
:mc-cmd:`mc ilm add --transition-days` command to transition objects to the
|
||||
remote tier after a specified number of calendar days.
|
||||
|
||||
Exclusive Access to Remote Data
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-transition-bucket-access-desc
|
||||
:end-before: end-transition-bucket-access-desc
|
||||
|
||||
Availability of Remote Data
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-transition-data-loss-desc
|
||||
:end-before: end-transition-data-loss-desc
|
||||
|
||||
Versioned Buckets
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO adopts :s3-docs:`S3 behavior
|
||||
<intro-lifecycle-rules.html#intro-lifecycle-rules-actions>` for transition rules
|
||||
on :ref:`versioned buckets <minio-bucket-versioning>`. Specifically, MinIO by
|
||||
default applies the transition operation to the *current* object version.
|
||||
|
||||
To transition noncurrent object versions, specify the
|
||||
:mc-cmd:`~mc ilm add --noncurrentversion-transition-days` and
|
||||
:mc-cmd:`~mc ilm add --noncurrentversion-transition-storage-class` options
|
||||
when creating the transition rule.
|
||||
|
||||
.. _minio-lifecycle-management-expiration:
|
||||
|
||||
Object Expiration
|
||||
-----------------
|
||||
|
||||
MinIO lifecycle management supports expiring objects on a bucket. Object
|
||||
"expiration" involves performing a ``DELETE`` operation on the object. For
|
||||
example, you can create a lifecycle management rule to expire any object
|
||||
older than 365 days.
|
||||
|
||||
.. todo: Diagram of MinIO Expiration
|
||||
|
||||
Use :mc-cmd:`mc ilm add --expiry-days` to expire objects after a specified
|
||||
number of calendar days.
|
||||
|
||||
For buckets with :ref:`replication <minio-bucket-replication>` configured, MinIO
|
||||
does not replicate objects deleted by a lifecycle management expiration rule.
|
||||
See :ref:`minio-replication-behavior-delete` for more information.
|
||||
|
||||
Versioned Buckets
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO adopts :s3-docs:`S3 behavior
|
||||
<intro-lifecycle-rules.html#intro-lifecycle-rules-actions>` for expiration rules
|
||||
on :ref:`versioned buckets <minio-bucket-versioning>`. MinIO has two
|
||||
specific default behaviors for versioned buckets:
|
||||
|
||||
- MinIO applies the expiration option to only the *current* object version by
|
||||
creating a ``DeleteMarker`` as is normal with versioned delete.
|
||||
|
||||
To expire noncurrent object versions, specify the
|
||||
:mc-cmd:`~mc ilm add --noncurrentversion-expiration-days` option
|
||||
when creating the expiration rule.
|
||||
|
||||
- MinIO does not expire ``DeleteMarkers`` *even if* no other versions of
|
||||
that object exist.
|
||||
|
||||
To expire delete markers when there are no remaining versions for that
|
||||
object, specify the :mc-cmd:`~mc ilm add --expired-object-delete-marker`
|
||||
option when creating the expiration rule.
|
||||
|
||||
.. _minio-lifecycle-management-scanner:
|
||||
|
||||
Lifecycle Management Object Scanner
|
||||
-----------------------------------
|
||||
|
||||
MinIO uses a built-in scanner to actively check objects against all
|
||||
configured lifecycle management rules. The scanner is a low-priority process
|
||||
that yields to high IO workloads to prevent performance spikes triggered
|
||||
by rule timing. The scanner may therefore not detect an object as eligible
|
||||
for a configured transition or expiration lifecycle rule until *after*
|
||||
the lifecycle rule period has passed.
|
||||
|
||||
Delayed application of lifecycle management rules is typically associated to
|
||||
limited node resources and cluster size. Scanner speed tends to slow as
|
||||
clusters grow as more time is required to visit all buckets and objects.
|
||||
This can be exacerbated if the cluster hardware is undersized for regular
|
||||
workloads, as the scanner will yield to high cluster load to avoid performance
|
||||
loss. Consider regularly checking cluster metrics, capacity, and resource
|
||||
usage to ensure the cluster hardware is scaling alongside cluster and workload
|
||||
growth.
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
/administration/object-management/transition-objects-to-s3.rst
|
||||
/administration/object-management/transition-objects-to-gcs.rst
|
||||
/administration/object-management/transition-objects-to-azure.rst
|
||||
/administration/object-management/create-lifecycle-management-expiration-rule.rst
|
433
source/administration/object-management/object-retention.rst
Normal file
433
source/administration/object-management/object-retention.rst
Normal file
@@ -0,0 +1,433 @@
|
||||
.. _minio-object-locking:
|
||||
.. _minio-object-retention:
|
||||
|
||||
====================
|
||||
MinIO Object Locking
|
||||
====================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
MinIO Object Locking ("Object Retention") enforces Write-Once Read-Many (WORM)
|
||||
immutability to protect :ref:`versioned objects <minio-bucket-versioning>` from
|
||||
deletion. MinIO supports both
|
||||
:ref:`duration based object retention <minio-object-locking-retention-modes>`
|
||||
and
|
||||
:ref:`indefinite Legal Hold retention <minio-object-locking-legalhold>`.
|
||||
|
||||
MinIO Object Locking provides key data retention compliance and meets
|
||||
SEC17a-4(f), FINRA 4511(C), and CFTC 1.31(c)-(d) requirements as per
|
||||
`Cohasset Associates <https://min.io/cohasset?ref-docs>`__.
|
||||
|
||||
.. card-carousel:: 1
|
||||
|
||||
.. card:: Bucket Without Locking
|
||||
|
||||
.. image:: /images/retention/minio-versioning-delete-object.svg
|
||||
:alt: Deleting an Object
|
||||
:align: center
|
||||
:width: 600px
|
||||
|
||||
MinIO versioning preserves the full history of object mutations.
|
||||
However, applications can explicitly delete specific object versions.
|
||||
|
||||
.. card:: Bucket With Locking
|
||||
|
||||
.. image:: /images/retention/minio-object-locking.svg
|
||||
:alt: 30 Day Locked Objects
|
||||
:align: center
|
||||
:width: 600px
|
||||
|
||||
Applying a default 30 Day WORM lock to objects in the bucket ensures
|
||||
a minimum period of retention and protection for all object versions.
|
||||
|
||||
.. card:: Delete Operations in Locked Bucket
|
||||
|
||||
.. image:: /images/retention/minio-object-locking-delete.svg
|
||||
:alt: Delete Operation in Locked Bucket
|
||||
:align: center
|
||||
:width: 600px
|
||||
|
||||
Delete operations follow normal behavior in
|
||||
:ref:`versioned buckets <minio-bucket-versioning-delete>`, where MinIO
|
||||
creates a ``DeleteMarker`` for the object. However, non-Delete Marker
|
||||
versions of the object remain under the retention rules and are protected
|
||||
from any specific deletion or overwrite attempts.
|
||||
|
||||
.. card:: Versioned Delete Operations in Locked Bucket
|
||||
|
||||
.. image:: /images/retention/minio-object-locking-delete-version.svg
|
||||
:alt: Versioned Delete Operation in a Locked Bucket
|
||||
:align: center
|
||||
:width: 600px
|
||||
|
||||
MinIO blocks any attempt to delete a specific object version held under
|
||||
WORM lock. The earliest possible time after which a client may delete
|
||||
the version is when the lock expires.
|
||||
|
||||
MinIO object locking is
|
||||
:s3-docs:`feature and API compatible with AWS S3 <object-lock.html>`.
|
||||
This page summarizes Object Locking / Retention concepts as implemented by
|
||||
MinIO. See the AWS S3 documentation on
|
||||
:s3-docs:`How S3 Object Lock works <object-lock.html>` for additional
|
||||
resources.
|
||||
|
||||
You can only enable object locking during bucket creation as per
|
||||
:s3-docs:`S3 behavior <object-lock-overview.html#object-lock-bucket-config>`.
|
||||
You cannot enable object locking on a bucket created without locking
|
||||
enabled. You can then configure object retention rules at any time.
|
||||
Object locking requires :ref:`versioning <minio-bucket-versioning>` and
|
||||
enables the feature implicitly.
|
||||
|
||||
.. _minio-bucket-locking-interactions-versioning:
|
||||
|
||||
Interaction with Versioning
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Objects held under WORM locked are immutable until the lock expires or is
|
||||
explicitly lifted. Locking is per-object version, where each version is
|
||||
independently immutable.
|
||||
|
||||
If an application performs an unversioned delete operation on a locked object,
|
||||
the operation produces a :ref:`delete marker <minio-bucket-versioning-delete>`.
|
||||
Attempts to explicitly delete any WORM-locked object fail with an error.
|
||||
Delete Markers are *not* eligible for protection under WORM locking.
|
||||
See the S3 documentation on
|
||||
:s3-docs:`Managing delete markers and object lifecycles
|
||||
<object-lock-managing.html#object-lock-managing-lifecycle>` for more
|
||||
information.
|
||||
|
||||
For example, consider the following bucket with
|
||||
:ref:`minio-object-locking-governance` locking enabled by default:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ mc ls --versions play/locking-guide
|
||||
|
||||
[DATETIME] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
|
||||
[DATETIME] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
|
||||
[DATETIME] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv
|
||||
|
||||
Attempting to perform a delete on a *specific version* of ``data.csv`` fails
|
||||
due to the object locking settings:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ mc rm --version-id 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 play/data.csv
|
||||
|
||||
Removing `play/locking-guide/data.csv` (versionId=62429eb1-9cb7-4dc5-b507-9cc23d0cc691).
|
||||
mc: <ERROR> Failed to remove `play/locking-guide/data.csv`.
|
||||
Object, 'data.csv (Version ID=62429eb1-9cb7-4dc5-b507-9cc23d0cc691)' is
|
||||
WORM protected and cannot be overwritten
|
||||
|
||||
Attempting to perform an unversioned delete on ``data.csv`` succeeds and creates
|
||||
a new ``DeleteMarker`` for the object:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ mc rm play/locking-guide/data.csv
|
||||
|
||||
[DATETIME] 0B acce329f-ad32-46d9-8649-5fe8bf4ec6e0 v4 DEL data.csv
|
||||
[DATETIME] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
|
||||
[DATETIME] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
|
||||
[DATETIME] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv
|
||||
|
||||
Interaction with Lifecycle Management
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO :ref:`object expiration <minio-lifecycle-management-expiration>`
|
||||
respects any active object lock and retention settings for objects covered by
|
||||
the expiration rule.
|
||||
|
||||
- For expiration rules operating on only the *current* object version,
|
||||
MinIO creates a Delete Marker for the locked object.
|
||||
|
||||
- For expiration rules operating on *non-current object versions*,
|
||||
MinIO can only expire the non-current versions *after* the retention period
|
||||
has passed *or* has been explicitly lifted (e.g. Legal Holds).
|
||||
|
||||
For example, consider the following bucket with
|
||||
:ref:`minio-object-locking-governance` locking enabled by default for 45 days:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ mc ls --versions play/locking-guide
|
||||
|
||||
[7D] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
|
||||
[30D] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
|
||||
[60D] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv
|
||||
|
||||
Creating an expiration rule for *current* objects older than 7 days results in
|
||||
a Delete Marker for the object:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ mc ls --versions play/locking-guide
|
||||
|
||||
[0D] 0B acce329f-ad32-46d9-8649-5fe8bf4ec6e0 v4 DEL data.csv
|
||||
[7D] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
|
||||
[30D] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
|
||||
[60D] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv
|
||||
|
||||
However, an expiration rule for *non-current* objects older than 7 days would
|
||||
only take effect *after* the configured WORM lock expires. Since the bucket
|
||||
has a 45 day ``GOVERNANCE`` retention set, only the ``v1`` version of
|
||||
``data.csv`` is unlocked and therefore eligible for deletion.
|
||||
|
||||
Tutorials
|
||||
---------
|
||||
|
||||
Create Bucket with Object Locking Enabled
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must enable object locking during bucket creation as per S3 behavior.
|
||||
You can create a bucket with object locking enabled using the MinIO Console,
|
||||
the MinIO :mc:`mc` CLI, or using an S3-compatible SDK.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: MinIO Console
|
||||
:sync: console
|
||||
|
||||
Select the :guilabel:`Buckets` section of the MinIO Console to access
|
||||
bucket creation and management functions. Select the bucket row from the
|
||||
list of buckets. You can use the :octicon:`search` :guilabel:`Search` bar
|
||||
to filter the list.
|
||||
|
||||
.. image:: /images/minio-console/console-bucket.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Bucket Management
|
||||
:align: center
|
||||
|
||||
Click the :guilabel:`Create Bucket` button to open the bucket creation
|
||||
modal. Toggle the :guilabel:`Object Locking` selector to enable object
|
||||
locking on the bucket.
|
||||
|
||||
.. image:: /images/minio-console/console-bucket-create-bucket-with-locking.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Bucket Management
|
||||
:align: center
|
||||
|
||||
.. tab-item:: MinIO CLI
|
||||
:sync: cli
|
||||
|
||||
Use the :mc-cmd:`mc mb` command with the :mc-cmd:`~mc mb --with-lock`
|
||||
option to create a bucket with object locking enabled:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc mb --with-lock ALIAS/BUCKET
|
||||
|
||||
- Replace ``ALIAS`` with the :mc:`alias <mc alias>` of a configured
|
||||
MinIO deployment.
|
||||
|
||||
- Replace ``BUCKET`` with the
|
||||
:mc:`name <mc version ALIAS>` of the bucket to create.
|
||||
|
||||
Configure Bucket-Default Object Retention
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can configure object locking rules ("object retention") using the
|
||||
MinIO Console, the MinIO :mc:`mc` CLI, or using an S3-compatible SDK.
|
||||
|
||||
MinIO supports setting both bucket-default *and* per-object retention rules.
|
||||
The following examples set bucket-default retention. For per-object retention
|
||||
settings, defer to the documentation for the ``PUT`` operation used by your
|
||||
preferred SDK.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: MinIO Console
|
||||
:sync: console
|
||||
|
||||
Select the :guilabel:`Buckets` section of the MinIO Console to access bucket creation and management functions. You can use the :octicon:`search` :guilabel:`Search` bar to filter the list.
|
||||
|
||||
.. image:: /images/minio-console/console-bucket.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Bucket Management
|
||||
:align: center
|
||||
|
||||
Each bucket row has a :guilabel:`Manage` button that opens the management view for that bucket.
|
||||
|
||||
.. image:: /images/minio-console/console-bucket-manage.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Bucket Management
|
||||
:align: center
|
||||
|
||||
From the :guilabel:`Retention` section, select :guilabel:`Enabled`.
|
||||
This section is only visible for buckets created with object locking enabled.
|
||||
|
||||
From the :guilabel:`Set Retention Configuration` modal, set the desired bucket default retention settings.
|
||||
|
||||
- For :guilabel:`Retention Mode`, select either :ref:`COMPLIANCE <minio-object-locking-compliance>` or :ref:`GOVERNANCE <minio-object-locking-governance>`.
|
||||
|
||||
- For :guilabel:`Duration`, select the retention duration units of :guilabel:`Days` or :guilabel:`Years`.
|
||||
|
||||
- For :guilabel:`Retention Validity`, set the duration of time for which MinIO holds objects under the specified retention mode for the bucket.
|
||||
|
||||
.. tab-item:: MinIO CLI
|
||||
:sync: cli
|
||||
|
||||
Use the :mc-cmd:`mc retention set` command with the
|
||||
:mc-cmd:`--recursive <mc retention set --recursive>` and
|
||||
:mc-cmd:`--default <mc retention set --default>` options to set the
|
||||
default retention mode for a bucket:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc retention set --recursive --default MODE DURATION ALIAS/BUCKET
|
||||
|
||||
- Replace :mc-cmd:`MODE <mc retention set MODE>` with either either
|
||||
:ref:`COMPLIANCE <minio-object-locking-compliance>` or
|
||||
:ref:`GOVERNANCE <minio-object-locking-governance>`.
|
||||
|
||||
- Replace :mc-cmd:`DURATION <mc retention set VALIDITY>` with the
|
||||
duration for which the object lock remains in effect.
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc retention set ALIAS>` with the
|
||||
:mc:`alias <mc alias>` of a configured MinIO deployment.
|
||||
|
||||
- Replace :mc-cmd:`BUCKET <mc retention set ALIAS>` with the
|
||||
name of the bucket on which to set the default retention rule.
|
||||
|
||||
Enable Legal Hold Retention
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can enable or disable indefinite Legal Hold retention for an object using the MinIO Console, the MinIO :mc:`mc` CLI, or using an S3-compatible SDK.
|
||||
|
||||
You can place a legal hold on an object already held under a :ref:`COMPLIANCE <minio-object-locking-compliance>` or :ref:`GOVERNANCE <minio-object-locking-governance>` lock.
|
||||
The object remains WORM locked under the legal hold even when the retention lock expires.
|
||||
You or another user with the necessary permissions must explicitly lift the legal hold to remove the WORM lock.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: MinIO Console
|
||||
:sync: console
|
||||
|
||||
Select the :guilabel:`Buckets` section of the MinIO Console to access bucket creation and management functions.
|
||||
You can use the :octicon:`search` :guilabel:`Search` bar to filter the list.
|
||||
|
||||
.. image:: /images/minio-console/console-bucket.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Bucket Management
|
||||
:align: center
|
||||
|
||||
Each bucket row has a :guilabel:`Manage` button that opens the management view for that bucket.
|
||||
|
||||
.. image:: /images/minio-console/console-object-browser.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Bucket Object Browser
|
||||
:align: center
|
||||
|
||||
Browse to the object and select it to open the object details view.
|
||||
Select the :guilabel:`Legal Hold` button to toggle the Legal Hold status of the object.
|
||||
|
||||
.. tab-item:: MinIO CLI
|
||||
:sync: cli
|
||||
|
||||
Use the :guilabel:`mc legalhold set` command to toggle the legal hold status on an object.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc legalhold set ALIAS/PATH
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc legalhold set ALIAS>` with the
|
||||
:mc:`alias <mc alias>` of a configured MinIO deployment.
|
||||
|
||||
- Replace :mc-cmd:`PATH <mc legalhold set ALIAS>` with the
|
||||
path to the object for which to enable the legal hold.
|
||||
|
||||
.. _minio-object-locking-retention-modes:
|
||||
|
||||
Object Retention Modes
|
||||
----------------------
|
||||
|
||||
MinIO implements the following
|
||||
:s3-docs:`S3 Object Locking Modes <object-lock-overview.html>`:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 40 60
|
||||
:width: 100%
|
||||
|
||||
* - Mode
|
||||
- Summary
|
||||
|
||||
* - :ref:`minio-object-locking-governance`
|
||||
- Prevents any operation that would mutate or modify the object or its
|
||||
locking settings by non-privileged users.
|
||||
|
||||
Users with the :policy-action:`s3:BypassGovernanceRetention` permission
|
||||
on the bucket or object can modify the object or its locking settings.
|
||||
|
||||
MinIO lifts the lock automatically after the configured retention rule
|
||||
duration has passed.
|
||||
|
||||
* - :ref:`minio-object-locking-compliance`
|
||||
- Prevents any operation that would mutate or modify the object or its
|
||||
locking settings.
|
||||
|
||||
No MinIO user can modify the object or its settings, including the
|
||||
:ref:`MinIO root <minio-users-root>` user.
|
||||
|
||||
MinIO lifts the lock automatically after the configured retention rule
|
||||
duration has passed.
|
||||
|
||||
.. _minio-object-locking-governance:
|
||||
|
||||
GOVERNANCE Mode
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
An object under ``GOVERNANCE`` lock is protected from write operations by
|
||||
non-privileged users.
|
||||
|
||||
``GOVERNANCE`` locked objects enforce managed-immutability for locked objects,
|
||||
where users with the :policy-action:`s3:BypassGovernanceRetention` action can
|
||||
modify the locked object, change the retention duration, or lift the lock
|
||||
entirely. Bypassing ``GOVERNANCE`` retention also requires setting the
|
||||
``x-amz-bypass-governance-retention:true`` header as part of the request.
|
||||
|
||||
The MinIO ``GOVERNANCE`` lock is functionally identical to the
|
||||
:s3-docs:`S3 GOVERNANCE mode
|
||||
<object-lock-overview.html#object-lock-retention-modes>`.
|
||||
|
||||
.. _minio-object-locking-compliance:
|
||||
|
||||
COMPLIANCE Mode
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
An object under ``COMPLIANCE`` lock is protected from write operations by *all*
|
||||
users, including the :ref:`MinIO root <minio-users-root>` user.
|
||||
|
||||
``COMPLIANCE`` locked objects enforce complete immutability for locked objects.
|
||||
You cannot change or remove the lock before the configured retention
|
||||
duration has passed.
|
||||
|
||||
The MinIO ``COMPLIANCE`` lock is functionally identical to the
|
||||
:s3-docs:`S3 GOVERNANCE mode
|
||||
<object-lock-overview.html#object-lock-retention-modes>`.
|
||||
|
||||
.. _minio-object-locking-legalhold:
|
||||
|
||||
Legal Hold
|
||||
----------
|
||||
|
||||
An object under Legal Hold is protected from write operations by *all*
|
||||
users, including the :ref:`MinIO root <minio-users-root>` user.
|
||||
|
||||
Legal Holds are indefinite and enforce complete immutability for locked objects.
|
||||
Only privileged users with the :policy-action:`s3:PutObjectLegalHold` permission can set or lift the Legal Hold.
|
||||
|
||||
Legal holds are complementary to both :ref:`minio-object-locking-governance` and :ref:`minio-object-locking-compliance` retention settings.
|
||||
An object held under both legal hold *and* a ``GOVERNANCE/COMPLIANCE`` retention rule remains WORM locked until the legal hold is lifted *and* the rule expires.
|
||||
|
||||
For ``GOVERNANCE`` locked objects, the legal hold prevents mutating the object *even if* the user has the necessary privileges to bypass retention.
|
306
source/administration/object-management/object-versioning.rst
Normal file
306
source/administration/object-management/object-versioning.rst
Normal file
@@ -0,0 +1,306 @@
|
||||
.. _minio-bucket-versioning:
|
||||
|
||||
=================
|
||||
Bucket Versioning
|
||||
=================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
MinIO supports keeping multiple "versions" of an object in a single bucket.
|
||||
Write operations which would normally overwrite an existing object instead
|
||||
result in the creation of a new versioned object. MinIO versioning protects from
|
||||
unintended overwrites and deletions while providing support for "undoing" a
|
||||
write operation. Bucket versioning is a prerequisite for configuring
|
||||
:ref:`object locking and retention rules <minio-object-locking>`.
|
||||
|
||||
For versioned buckets, any write operation that mutates an object results in a
|
||||
new version of that object with a unique version ID. MinIO marks the "latest"
|
||||
version of the object that clients retrieve by default. Clients can then
|
||||
explicitly choose to list, retrieve, or remove a specific object version.
|
||||
|
||||
.. card-carousel:: 1
|
||||
|
||||
.. card:: Object with Single Version
|
||||
|
||||
.. image:: /images/retention/minio-versioning-single-version.svg
|
||||
:alt: Object with single version
|
||||
:align: center
|
||||
|
||||
MinIO adds a unique version ID to each object as part of write operations.
|
||||
|
||||
.. card:: Object with Multiple Versions
|
||||
|
||||
.. image:: /images/retention/minio-versioning-multiple-versions.svg
|
||||
:alt: Object with Multiple Versions
|
||||
:align: center
|
||||
|
||||
MinIO retains all versions of an object and marks the most recent
|
||||
version as the "latest".
|
||||
|
||||
.. card:: Retrieving the Latest Object Version
|
||||
|
||||
.. image:: /images/retention/minio-versioning-retrieve-latest-version.svg
|
||||
:alt: Object with Multiple Versions
|
||||
:align: center
|
||||
|
||||
.. card:: Retrieving a Specific Object Version
|
||||
|
||||
.. image:: /images/retention/minio-versioning-retrieve-single-version.svg
|
||||
:alt: Object with Multiple Versions
|
||||
:align: center
|
||||
|
||||
:ref:`Deleting <minio-bucket-versioning-delete>` an object results in a special
|
||||
``DeleteMarker`` tombstone that marks an object as deleted while retaining
|
||||
all previous versions of that object.
|
||||
|
||||
Versioning is Per-Namespace
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO uses the full namespace (the bucket and path to an object) for each object
|
||||
as part of determining object uniqueness. For example, all of the following
|
||||
namespaces are "unique" objects, where mutations of each object result in
|
||||
the creation of new object versions *at that namespace*:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
databucket/object.blob
|
||||
databucket/blobs/object.blob
|
||||
blobbucket/object.blob
|
||||
blobbucket/blobs/object.blob
|
||||
|
||||
While ``object.blob`` might be the same binary across all namespaces,
|
||||
MinIO only enforces versioning with a specific namespace and therefore
|
||||
considers each ``object.blob`` above as distinct and unique.
|
||||
|
||||
Versioning and Storage Capacity
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO does not perform incremental or differential-type versioning. For
|
||||
mutation-heavy workloads, this may result in substantial disk usage by
|
||||
older or aged object versions.
|
||||
|
||||
For example, consider a 1GB object containing log data. An application
|
||||
appends 100MB of data to the log and uploads to MinIO. MinIO would then contain
|
||||
both the 1GB and 1.1GB versions of the object. If the application repeated
|
||||
this process every day for 10 days, the bucket would eventually contain more
|
||||
than 14GB of data associated to a single object.
|
||||
|
||||
MinIO supports configuring configuring :ref:`object lifecycle management rules
|
||||
<minio-lifecycle-management>` to automatically expire or transition aged
|
||||
object versions and free up storage capacity. For example, you can configure
|
||||
a rule to automatically expire object versions 90 days after they become
|
||||
non-current (i.e. no longer the "latest" version of that object). See
|
||||
:ref:`MinIO Object Expiration <minio-lifecycle-management-expiration>` for
|
||||
more information.
|
||||
|
||||
You can alternatively perform manual removal of object versions using the
|
||||
following commands:
|
||||
|
||||
- :mc-cmd:`mc rm --versions` - Removes all versions of an object.
|
||||
- :mc-cmd:`mc rm --versions --older-than <mc rm --older-than>` -
|
||||
Removes all versions of an object older than the specified calendar date.
|
||||
|
||||
.. _minio-bucket-versioning-id:
|
||||
|
||||
Version ID Generation
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO generates a unique and immutable identifier for each versioned object as
|
||||
part of write operations. Each object version ID consists of a 128-bit
|
||||
fixed-size :rfc:`UUIDv4 <4122#section-4.4>`. UUID generation is sufficiently
|
||||
random to ensure high likelihood of uniqueness for any environment, are
|
||||
computationally difficult to guess, and do not require centralized registration
|
||||
process and authority to guarantee uniqueness.
|
||||
|
||||
.. image:: /images/retention/minio-versioning-multiple-versions.svg
|
||||
:alt: Object with Multiple Versions
|
||||
:width: 600px
|
||||
:align: center
|
||||
|
||||
MinIO does not support client-managed version ID allocation. All version ID
|
||||
generation is handled by the MinIO server process.
|
||||
|
||||
For objects created while versioning is disabled or suspended, MinIO
|
||||
uses a ``null`` version ID. You can access or remove these objects by specifying
|
||||
``null`` as the version ID as part of S3 operations.
|
||||
|
||||
.. _minio-bucket-versioning-delete:
|
||||
|
||||
Versioned Delete Operations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Performing a ``DELETE`` operation on a versioned object creates a
|
||||
0-byte ``DeleteMarker`` as the latest version of that object. Clients performing
|
||||
``GET`` operations on that object do not return any results, as MinIO does not
|
||||
return the ``DeleteMarker`` back as part of the response. Similarly, performing
|
||||
a ``LIST`` operation by default returns only objects which are *not* a
|
||||
``DeleteMarker``.
|
||||
|
||||
To permanently delete an object version, perform the ``DELETE`` operation and
|
||||
specify the version ID of the object to delete. Versioned delete operations
|
||||
are **irreversible**.
|
||||
|
||||
.. card-carousel:: 1
|
||||
|
||||
.. card:: Deleting an Object
|
||||
|
||||
.. image:: /images/retention/minio-versioning-delete-object.svg
|
||||
:alt: Deleting an Object
|
||||
:align: center
|
||||
|
||||
Performing a ``DELETE`` operation on a versioned object produces a
|
||||
``DeleteMarker`` for that object.
|
||||
|
||||
.. card:: Reading a Deleted Object
|
||||
|
||||
.. image:: /images/retention/minio-versioning-retrieve-deleted-object.svg
|
||||
:alt: Object with Multiple Versions
|
||||
:align: center
|
||||
|
||||
Clients by default retrieve the "latest" object version. MinIO returns
|
||||
a ``404``-like response if the latest version is a ``DeleteMarker``.
|
||||
|
||||
.. card:: Retrieve Previous Version of Deleted Object
|
||||
|
||||
.. image:: /images/retention/minio-versioning-retrieve-version-before-delete.svg
|
||||
:alt: Retrieve Version of Deleted Object
|
||||
:align: center
|
||||
|
||||
Clients can retrieve any previous version of the object by specifying the
|
||||
version ID, even if the "Latest" version is a ``DeleteMarker``.
|
||||
|
||||
.. card:: Delete a Specific Object Version
|
||||
|
||||
.. image:: /images/retention/minio-versioning-delete-specific-version.svg
|
||||
:alt: Retrieve Version of Deleted Object
|
||||
:align: center
|
||||
|
||||
Clients can delete a specific object version by specifying the version ID
|
||||
as part of the ``DELETE`` operation. Deleting a specific version is
|
||||
**permanent** and does not result in the creation of a ``DeleteMarker``.
|
||||
|
||||
The following :mc:`mc` commands operate on ``DeleteMarkers`` or versioned
|
||||
objects:
|
||||
|
||||
- Use :mc-cmd:`mc ls --versions` to view all versions of an object,
|
||||
including delete markers.
|
||||
|
||||
- Use :mc-cmd:`mc cp --version-id=UUID ... <mc cp --version-id>` to
|
||||
retrieve the version of the "deleted" object with matching ``UUID``.
|
||||
|
||||
- Use :mc-cmd:`mc rm --version-id=UUID ... <mc rm --version-id>` to delete
|
||||
the version of the object with matching ``UUID``.
|
||||
|
||||
- Use :mc-cmd:`mc rm --versions` to delete *all* versions of an object.
|
||||
|
||||
Tutorials
|
||||
---------
|
||||
|
||||
Enable Bucket Versioning
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can enable versioning using the MinIO Console, the MinIO :mc:`mc` CLI, or
|
||||
using an S3-compatible SDK. Versioning is a bucket-scoped feature. You cannot
|
||||
enable versioning on only a prefix or subset of objects in a bucket.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: MinIO Console
|
||||
|
||||
Select the :guilabel:`Buckets` section of the MinIO Console to access bucket creation and management functions. You can use the :octicon:`search` :guilabel:`Search` bar to filter the list.
|
||||
|
||||
.. image:: /images/minio-console/console-bucket.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Bucket Management
|
||||
:align: center
|
||||
|
||||
Each bucket row has a :guilabel:`Manage` button that opens the management view for that bucket.
|
||||
|
||||
.. image:: /images/minio-console/console-bucket-manage.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Bucket Management
|
||||
:align: center
|
||||
|
||||
Toggle the :guilabel:`Versioning` field to enable versioning on the bucket.
|
||||
|
||||
The MinIO Console also supports enabling versioning as part of bucket
|
||||
creation. See :ref:`minio-console-admin-buckets` for more information on
|
||||
bucket management using the MinIO Console.
|
||||
|
||||
.. tab-item:: MinIO CLI
|
||||
|
||||
Use the :mc-cmd:`mc version enable` command to enable versioning on an
|
||||
existing bucket:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc version ALIAS/BUCKET
|
||||
|
||||
- Replace ``ALIAS`` with the :mc:`alias <mc alias>` of a configured
|
||||
MinIO deployment.
|
||||
|
||||
- Replace ``BUCKET`` with the
|
||||
:mc:`target bucket <mc version ALIAS>` on which to enable
|
||||
versioning.
|
||||
|
||||
Objects created prior to enabling versioning have a
|
||||
``null`` :ref:`version ID <minio-bucket-versioning-id>`.
|
||||
|
||||
Suspend Bucket Versioning
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can suspend bucket versioning at any time using the MinIO Console, the
|
||||
MinIO :mc:`mc` CLI, or using an S3-compatible SDK.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: MinIO Console
|
||||
|
||||
Select the :guilabel:`Buckets` section of the MinIO Console to access bucket creation and management functions. You can use the :octicon:`search` :guilabel:`Search` bar to filter the list.
|
||||
|
||||
.. image:: /images/minio-console/console-bucket.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Bucket Management
|
||||
:align: center
|
||||
|
||||
Each bucket row has a :guilabel:`Manage` button that opens the management view for that bucket.
|
||||
|
||||
.. image:: /images/minio-console/console-bucket-manage.png
|
||||
:width: 600px
|
||||
:alt: MinIO Console Bucket Management
|
||||
:align: center
|
||||
|
||||
Select the :guilabel:`Versioning` field and follow the instructions to suspend versioning in the bucket.
|
||||
|
||||
See :ref:`minio-console-admin-buckets` for more information on bucket
|
||||
management using the MinIO Console.
|
||||
|
||||
.. tab-item:: MinIO CLI
|
||||
|
||||
Use the :mc-cmd:`mc version suspend` command to enable versioning on an
|
||||
existing bucket:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc version suspend ALIAS/BUCKET
|
||||
|
||||
- Replace ``ALIAS`` with the :mc:`alias <mc alias>` of a configured
|
||||
MinIO deployment.
|
||||
|
||||
- Replace ``BUCKET`` with the
|
||||
:mc:`target bucket <mc version ALIAS>` on which to disable
|
||||
versioning.
|
||||
|
||||
Objects created while versioning is suspended are assigned a ``null`` :ref:`version ID <minio-bucket-versioning-id>`.
|
||||
Any mutations to an object while versioning is suspended result in overwriting that ``null`` versioned object.
|
||||
MinIO does not remove or otherwise alter existing versioned objects as part of suspending versioning.
|
||||
Clients can continue interacting with any existing object versions in the bucket.
|
@@ -0,0 +1,213 @@
|
||||
.. _minio-lifecycle-management-transition-to-azure:
|
||||
|
||||
======================================
|
||||
Transition Objects from MinIO to Azure
|
||||
======================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
The procedure on this page creates a new object lifecycle management rule that
|
||||
transition objects from a MinIO bucket to a remote storage tier on the
|
||||
:abbr:`Azure (Microsoft Azure)` storage backend. This procedure supports use
|
||||
cases like moving aged data to low-cost public cloud storage solutions after a
|
||||
certain time period or calendar date.
|
||||
|
||||
.. todo: diagram
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
Install and Configure ``mc``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This procedure uses :mc:`mc` for performing operations on the MinIO cluster.
|
||||
Install :mc:`mc` on a machine with network access to both source and destination
|
||||
clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
|
||||
instructions on downloading and installing ``mc``.
|
||||
|
||||
Use the :mc:`mc alias` command to create an alias for the source MinIO cluster.
|
||||
Alias creation requires specifying an access key for a user on the source and
|
||||
destination clusters. The specified users must have :ref:`permissions
|
||||
<minio-lifecycle-management-transition-to-azure-permissions>` for configuring
|
||||
and applying transition operations.
|
||||
|
||||
.. _minio-lifecycle-management-transition-to-azure-permissions:
|
||||
|
||||
Required MinIO Permissions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO requires the following permissions scoped to the bucket or buckets
|
||||
for which you are creating lifecycle management rules.
|
||||
|
||||
- :policy-action:`s3:PutLifecycleConfiguration`
|
||||
- :policy-action:`s3:GetLifecycleConfiguration`
|
||||
|
||||
MinIO also requires the following administrative permissions on the cluster
|
||||
in which you are creating remote tiers for object transition lifecycle
|
||||
management rules:
|
||||
|
||||
- :policy-action:`admin:SetTier`
|
||||
- :policy-action:`admin:ListTier`
|
||||
|
||||
For example, the following policy provides permission for configuring object
|
||||
transition lifecycle management rules on any bucket in the cluster:.
|
||||
|
||||
.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json
|
||||
:language: json
|
||||
:class: copyable
|
||||
|
||||
.. _minio-lifecycle-management-transition-to-azure-permissions-remote:
|
||||
|
||||
Required Azure Permissions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Object transition lifecycle management rules require additional permissions on
|
||||
the remote storage tier. Specifically, MinIO requires the
|
||||
:abbr:`Azure (Microsoft Azure)` credentials provide read, write, list, and
|
||||
delete permissions for the remote bucket.
|
||||
|
||||
Refer to the `Azure RBAC
|
||||
<https://docs.microsoft.com/en-us/azure/role-based-access-control/>`__
|
||||
documentation for more complete guidance on configuring the required
|
||||
permissions.
|
||||
|
||||
Considerations
|
||||
--------------
|
||||
|
||||
Exclusive Access to Remote Data
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-transition-bucket-access-desc
|
||||
:end-before: end-transition-bucket-access-desc
|
||||
|
||||
.. important::
|
||||
|
||||
MinIO does *not* support changing the account name associated to an Azure
|
||||
remote tier. Azure storage backends are tied to the account, such that
|
||||
changing the account would change the storage backend and prevent access
|
||||
to any objects transitioned to the original account/backend.
|
||||
|
||||
Please contact `MinIO Support <https://min.io/pricing?ref=docs>`__ if you need
|
||||
situation-specific guidance around configuring Azure remote tiers.
|
||||
|
||||
Availability of Remote Data
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-transition-data-loss-desc
|
||||
:end-before: end-transition-data-loss-desc
|
||||
|
||||
Procedure
|
||||
---------
|
||||
|
||||
1) Configure User Accounts and Policies for Lifecycle Management
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. |permissions| replace:: :ref:`permissions <minio-lifecycle-management-transition-to-azure-permissions>`
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-create-transition-user-desc
|
||||
:end-before: end-create-transition-user-desc
|
||||
|
||||
2) Configure the Remote Storage Tier
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin tier add` command to add a new remote storage tier:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin tier add --azure TARGET TIER_NAME \
|
||||
--endpoint https://HOSTNAME
|
||||
--bucket BUCKET \
|
||||
--prefix PREFIX
|
||||
--account-name ACCOUNT \
|
||||
--account-key KEY \
|
||||
--region REGION
|
||||
|
||||
|
||||
The example above uses the following arguments:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
:width: 100%
|
||||
|
||||
* - Argument
|
||||
- Description
|
||||
|
||||
* - :mc-cmd:`TARGET <mc admin tier add TARGET>`
|
||||
- The :mc:`alias <mc alias>` of the MinIO deployment on which to configure
|
||||
the remote tier.
|
||||
|
||||
* - :mc-cmd:`TIER_NAME <mc admin tier add TIER_NAME>`
|
||||
- The name to associate with the new :abbr:`Azure (Microsoft Azure)` blob
|
||||
remote storage tier. Specify the name in all-caps, e.g. ``AZURE_TIER``.
|
||||
This value is required in the next step.
|
||||
|
||||
* - :mc-cmd:`HOSTNAME <mc admin tier add --endpoint>`
|
||||
- The URL endpoint for the :abbr:`Azure (Microsoft Azure)` storage
|
||||
backend.
|
||||
|
||||
* - :mc-cmd:`BUCKET <mc admin tier add --bucket>`
|
||||
- The name of the bucket on the :abbr:`Azure (Microsoft Azure)` storage
|
||||
backend to which MinIO transitions objects.
|
||||
|
||||
* - :mc-cmd:`PREFIX <mc admin tier add --prefix>`
|
||||
- The optional bucket prefix within which MinIO transitions objects.
|
||||
|
||||
MinIO stores all transitioned objects in the specified ``BUCKET`` under a
|
||||
unique per-deployment prefix value. Omit this argument to use only that
|
||||
value for isolating and organizing data within the remote storage.
|
||||
|
||||
MinIO recommends specifying this optional prefix for remote storage tiers
|
||||
which contain other data, including transitioned objects from other MinIO
|
||||
deployments. This prefix should provide a clear reference back to the
|
||||
source MinIO deployment to faciliate ease of operations related to
|
||||
diagnostics, maintenance, or disaster recovery.
|
||||
|
||||
* - :mc-cmd:`ACCOUNT <mc admin tier add --account-name>`
|
||||
- The account name MinIO uses to access the bucket. The account name
|
||||
*must* correspond to an :abbr:`Azure (Microsoft Azure)` user with the
|
||||
required :ref:`permissions
|
||||
<minio-lifecycle-management-transition-to-azure-permissions-remote>`.
|
||||
|
||||
You cannot change this account name after creating the tier.
|
||||
|
||||
* - :mc-cmd:`KEY <mc admin tier add --account-key>`
|
||||
- The corresponding key for the specified ``ACCOUNT``.
|
||||
|
||||
* - :mc-cmd:`REGION <mc admin tier add --region>`
|
||||
- The :abbr:`Azure (Microsoft Azure)` blob storage region of the specified
|
||||
``BUCKET``. You can safely omit this option if the ``HOSTNAME`` includes
|
||||
the region.
|
||||
|
||||
3) Create and Apply the Transition Rule
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-create-transition-rule-desc
|
||||
:end-before: end-create-transition-rule-desc
|
||||
|
||||
|
||||
4) Verify the Transition Rule
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc ilm ls` command to review the configured transition
|
||||
rules:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc ilm ls ALIAS/PATH --transition
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc ilm ls ALIAS>` with the :mc:`alias <mc alias>`
|
||||
of the MinIO deployment.
|
||||
|
||||
- Replace :mc-cmd:`PATH <mc ilm ls ALIAS>` with the name of the bucket for
|
||||
which to retrieve the configured lifecycle management rules.
|
@@ -0,0 +1,207 @@
|
||||
.. _minio-lifecycle-management-transition-to-gcs:
|
||||
|
||||
====================================
|
||||
Transition Objects from MinIO to GCS
|
||||
====================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
The procedure on this page creates a new object lifecycle management rule that
|
||||
transition objects from a MinIO bucket to a remote storage tier on the Google
|
||||
Cloud Storage backend. This procedure supports use cases like moving aged data
|
||||
to low-cost public cloud storage solutions after a certain time period or
|
||||
calendar date.
|
||||
|
||||
.. todo: diagram
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
Install and Configure ``mc``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This procedure uses :mc:`mc` for performing operations on the MinIO cluster.
|
||||
Install :mc:`mc` on a machine with network access to both source and destination
|
||||
clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
|
||||
instructions on downloading and installing ``mc``.
|
||||
|
||||
Use the :mc:`mc alias` command to create an alias for the source MinIO cluster.
|
||||
Alias creation requires specifying an access key for a user on the source and
|
||||
destination clusters. The specified users must have :ref:`permissions
|
||||
<minio-lifecycle-management-transition-to-gcs-permissions>` for configuring and
|
||||
applying transition operations.
|
||||
|
||||
.. _minio-lifecycle-management-transition-to-gcs-permissions:
|
||||
|
||||
Required MinIO Permissions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO requires the following permissions scoped to the bucket or buckets
|
||||
for which you are creating lifecycle management rules.
|
||||
|
||||
- :policy-action:`s3:PutLifecycleConfiguration`
|
||||
- :policy-action:`s3:GetLifecycleConfiguration`
|
||||
|
||||
MinIO also requires the following administrative permissions on the cluster
|
||||
in which you are creating remote tiers for object transition lifecycle
|
||||
management rules:
|
||||
|
||||
- :policy-action:`admin:SetTier`
|
||||
- :policy-action:`admin:ListTier`
|
||||
|
||||
For example, the following policy provides permission for configuring object
|
||||
transition lifecycle management rules on any bucket in the cluster:.
|
||||
|
||||
.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json
|
||||
:language: json
|
||||
:class: copyable
|
||||
|
||||
.. _minio-lifecycle-management-transition-to-gcs-permissions-remote:
|
||||
|
||||
Required GCS Permissions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Object transition lifecycle management rules require additional permissions on
|
||||
the remote storage tier. Specifically, MinIO requires the
|
||||
:abbr:`GCS (Google Cloud Storage)` credentials provide read, write, list, and
|
||||
delete permissions for the remote bucket.
|
||||
|
||||
Refer to the `GCS IAM permissions
|
||||
<https://cloud.google.com/storage/docs/access-control/iam-permissions>`__
|
||||
documentation for more complete guidance on configuring the required
|
||||
permissions.
|
||||
|
||||
Considerations
|
||||
--------------
|
||||
|
||||
Lifecycle Management Object Scanner
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO uses a scanner process to check objects against all configured
|
||||
lifecycle management rules. Slow scanning due to high IO workloads or
|
||||
limited system resources may delay application of lifecycle management
|
||||
rules. See :ref:`minio-lifecycle-management-scanner` for more information.
|
||||
|
||||
Exclusive Access to Remote Data
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-transition-bucket-access-desc
|
||||
:end-before: end-transition-bucket-access-desc
|
||||
|
||||
Availability of Remote Data
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-transition-data-loss-desc
|
||||
:end-before: end-transition-data-loss-desc
|
||||
|
||||
Procedure
|
||||
---------
|
||||
|
||||
1) Configure User Accounts and Policies for Lifecycle Management
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. |permissions| replace:: :ref:`permissions <minio-lifecycle-management-transition-to-gcs-permissions>`
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-create-transition-user-desc
|
||||
:end-before: end-create-transition-user-desc
|
||||
|
||||
2) Configure the Remote Storage Tier
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin tier add` command to add a new Google Cloud Storage
|
||||
service as the remote storage tier:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin tier add --gcs TARGET TIER_NAME \
|
||||
--endpoint https://HOSTNAME \
|
||||
--bucket BUCKET \
|
||||
--prefix PREFIX \
|
||||
--credentials-file CREDENTIALS \
|
||||
--region REGION
|
||||
|
||||
The example above uses the following arguments:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
:width: 100%
|
||||
|
||||
* - Argument
|
||||
- Description
|
||||
|
||||
* - :mc-cmd:`TARGET <mc admin tier add TARGET>`
|
||||
- The :mc:`alias <mc alias>` of the MinIO deployment on which to configure
|
||||
the :abbr:`GCS (Google Cloud Storage)` remote tier.
|
||||
|
||||
* - :mc-cmd:`TIER_NAME <mc admin tier add TIER_NAME>`
|
||||
- The name to associate with the new :abbr:`GCS (Google Cloud Storage)`
|
||||
remote storage tier. Specify the name in all-caps, e.g. ``GCS_TIER``.
|
||||
This value is required in the next step.
|
||||
|
||||
* - :mc-cmd:`HOSTNAME <mc admin tier add --endpoint>`
|
||||
- The URL endpoint for the :abbr:`GCS (Google Cloud Storage)` storage
|
||||
backend.
|
||||
|
||||
* - :mc-cmd:`BUCKET <mc admin tier add --bucket>`
|
||||
- The name of the bucket on the :abbr:`GCS (Google Cloud Storage)` storage
|
||||
backend to which MinIO transitions objects.
|
||||
|
||||
* - :mc-cmd:`PREFIX <mc admin tier add --prefix>`
|
||||
- The optional bucket prefix within which MinIO transitions objects.
|
||||
|
||||
MinIO stores all transitioned objects in the specified ``BUCKET`` under a
|
||||
unique per-deployment prefix value. Omit this argument to use only that
|
||||
value for isolating and organizing data within the remote storage.
|
||||
|
||||
MinIO recommends specifying this optional prefix for remote storage tiers
|
||||
which contain other data, including transitioned objects from other MinIO
|
||||
deployments. This prefix should provide a clear reference back to the
|
||||
source MinIO deployment to faciliate ease of operations related to
|
||||
diagnostics, maintenance, or disaster recovery.
|
||||
|
||||
* - :mc-cmd:`CREDENTIALS <mc admin tier add --credentials-file>`
|
||||
- The `credential file
|
||||
<https://cloud.google.com/docs/authentication/getting-started>`__ for a
|
||||
user on the remote GCS tier. The specified user credentials *must*
|
||||
correspond to a GCS user with the required
|
||||
:ref:`permissions
|
||||
<minio-lifecycle-management-transition-to-gcs-permissions-remote>`.
|
||||
|
||||
* - :mc-cmd:`REGION <mc admin tier add --region>`
|
||||
- The :abbr:`GCS (Google Cloud Storage)` region of the specified
|
||||
``BUCKET``. You can safely omit this
|
||||
option if the ``HOSTNAME`` includes the region.
|
||||
|
||||
3) Create and Apply the Transition Rule
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-create-transition-rule-desc
|
||||
:end-before: end-create-transition-rule-desc
|
||||
|
||||
|
||||
4) Verify the Transition Rule
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc ilm ls` command to review the configured transition
|
||||
rules:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc ilm ls ALIAS/PATH --transition
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc ilm ls ALIAS>` with the :mc:`alias <mc alias>`
|
||||
of the MinIO deployment.
|
||||
|
||||
- Replace :mc-cmd:`PATH <mc ilm ls ALIAS>` with the name of the bucket for
|
||||
which to retrieve the configured lifecycle management rules.
|
@@ -0,0 +1,225 @@
|
||||
.. _minio-lifecycle-management-transition-to-s3:
|
||||
|
||||
===================================
|
||||
Transition Objects from MinIO to S3
|
||||
===================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
The procedure on this page creates a new object lifecycle management rule that
|
||||
transition objects from a MinIO bucket to a remote storage tier on the Amazon
|
||||
Web Services S3 storage backend *or* an S3-compatible service. This procedure
|
||||
supports use cases such as tiering objects to low-cost or archival storage after
|
||||
a certain time period or calendar date.
|
||||
|
||||
.. todo: diagram
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
Install and Configure ``mc``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This procedure uses :mc:`mc` for performing operations on the MinIO cluster.
|
||||
Install :mc:`mc` on a machine with network access to both source and destination
|
||||
clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
|
||||
instructions on downloading and installing ``mc``.
|
||||
|
||||
Use the :mc:`mc alias` command to create an alias for the source MinIO cluster.
|
||||
Alias creation requires specifying an access key for a user on the source and
|
||||
destination clusters. The specified users must have :ref:`permissions
|
||||
<minio-lifecycle-management-transition-to-s3-permissions>` for configuring and
|
||||
applying transition operations.
|
||||
|
||||
.. _minio-lifecycle-management-transition-to-s3-permissions:
|
||||
|
||||
Required MinIO Permissions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO requires the following permissions scoped to the bucket or buckets
|
||||
for which you are creating lifecycle management rules.
|
||||
|
||||
- :policy-action:`s3:PutLifecycleConfiguration`
|
||||
- :policy-action:`s3:GetLifecycleConfiguration`
|
||||
|
||||
MinIO also requires the following administrative permissions on the cluster
|
||||
in which you are creating remote tiers for object transition lifecycle
|
||||
management rules:
|
||||
|
||||
- :policy-action:`admin:SetTier`
|
||||
- :policy-action:`admin:ListTier`
|
||||
|
||||
For example, the following policy provides permission for configuring object
|
||||
transition lifecycle management rules on any bucket in the cluster:.
|
||||
|
||||
.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json
|
||||
:language: json
|
||||
:class: copyable
|
||||
|
||||
.. _minio-lifecycle-management-transition-to-s3-permissions-remote:
|
||||
|
||||
Required S3 Permissions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Object transition lifecycle management rules require additional permissions
|
||||
on the remote storage tier. Specifically, MinIO requires the remote
|
||||
tier credentials provide read, write, list, and delete permissions for the
|
||||
remote bucket.
|
||||
|
||||
For example, the following policy provides the necessary permission
|
||||
for transitioning objects into and out of the remote tier:
|
||||
|
||||
.. literalinclude:: /extra/examples/LifecycleManagementUser.json
|
||||
:language: json
|
||||
:class: copyable
|
||||
|
||||
Modify the ``Resource`` for the bucket into which MinIO tiers objects.
|
||||
|
||||
Refer to the :aws-docs:`Amazon S3 Permissions
|
||||
<service-authorization/latest/reference/list_amazons3.html#amazons3-actions-as-permissions>`
|
||||
documentation for more complete guidance on configuring the required
|
||||
permissions.
|
||||
|
||||
Considerations
|
||||
--------------
|
||||
|
||||
Lifecycle Management Object Scanner
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO uses a scanner process to check objects against all configured
|
||||
lifecycle management rules. Slow scanning due to high IO workloads or
|
||||
limited system resources may delay application of lifecycle management
|
||||
rules. See :ref:`minio-lifecycle-management-scanner` for more information.
|
||||
|
||||
Exclusive Access to Remote Data
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-transition-bucket-access-desc
|
||||
:end-before: end-transition-bucket-access-desc
|
||||
|
||||
Availability of Remote Data
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-transition-data-loss-desc
|
||||
:end-before: end-transition-data-loss-desc
|
||||
|
||||
Procedure
|
||||
---------
|
||||
|
||||
1) Configure User Accounts and Policies for Lifecycle Management
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. |permissions| replace:: :ref:`permissions <minio-lifecycle-management-transition-to-s3-permissions>`
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-create-transition-user-desc
|
||||
:end-before: end-create-transition-user-desc
|
||||
|
||||
2) Configure the Remote Storage Tier
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc admin tier add` command to add an Amazon S3 service as the
|
||||
new remote storage tier:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin tier add --s3 TARGET TIER_NAME \
|
||||
--endpoint https://HOSTNAME \
|
||||
--bucket BUCKET \
|
||||
--prefix PREFIX
|
||||
--access-key ACCESS_KEY \
|
||||
--secret-key SECRET_KEY \
|
||||
--region REGION \
|
||||
--storage-class STORAGE_CLASS
|
||||
|
||||
The example above uses the following arguments:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
:width: 100%
|
||||
|
||||
* - Argument
|
||||
- Description
|
||||
|
||||
* - :mc-cmd:`TARGET <mc admin tier add TARGET>`
|
||||
- The :mc:`alias <mc alias>` of the MinIO deployment on which to configure
|
||||
the S3 remote tier.
|
||||
|
||||
* - :mc-cmd:`TIER_NAME <mc admin tier add TIER_NAME>`
|
||||
- The name to associate with the new S3 remote storage tier. Specify the
|
||||
name in all-caps, e.g. ``S3_TIER``. This value is required in the next
|
||||
step.
|
||||
|
||||
* - :mc-cmd:`HOSTNAME <mc admin tier add --endpoint>`
|
||||
- The URL endpoint for the S3 storage backend.
|
||||
|
||||
* - :mc-cmd:`BUCKET <mc admin tier add --bucket>`
|
||||
- The name of the bucket on the S3 storage backend to which MinIO
|
||||
transitions objects.
|
||||
|
||||
* - :mc-cmd:`PREFIX <mc admin tier add --prefix>`
|
||||
- The optional bucket prefix within which MinIO transitions objects.
|
||||
|
||||
MinIO stores all transitioned objects in the specified ``BUCKET`` under a
|
||||
unique per-deployment prefix value. Omit this argument to use only that
|
||||
value for isolating and organizing data within the remote storage.
|
||||
|
||||
MinIO recommends specifying this optional prefix for remote storage tiers
|
||||
which contain other data, including transitioned objects from other MinIO
|
||||
deployments. This prefix should provide a clear reference back to the
|
||||
source MinIO deployment to faciliate ease of operations related to
|
||||
diagnostics, maintenance, or disaster recovery.
|
||||
|
||||
* - :mc-cmd:`ACCESS_KEY <mc admin tier add --access-key>`
|
||||
- The S3 access key MinIO uses to access the bucket. The
|
||||
access key *must* correspond to an IAM user with the
|
||||
required
|
||||
:ref:`permissions
|
||||
<minio-lifecycle-management-transition-to-s3-permissions-remote>`.
|
||||
|
||||
* - :mc-cmd:`SECRET_KEY <mc admin tier add --secret-key>`
|
||||
- The corresponding secret key for the specified ``ACCESS_KEY``.
|
||||
|
||||
* - :mc-cmd:`REGION <mc admin tier add --region>`
|
||||
- The AWS S3 region of the specified ``BUCKET``. You can safely omit this
|
||||
option if the ``HOSTNAME`` includes the region.
|
||||
|
||||
* - :mc-cmd:`STORAGE_CLASS <mc admin tier add --storage-class>`
|
||||
- The S3 storage class to which MinIO transitions objects. Specify
|
||||
one of the following supported storage classes:
|
||||
|
||||
- ``STANDARD``
|
||||
- ``REDUCED``
|
||||
|
||||
3) Create and Apply the Transition Rule
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-tiering.rst
|
||||
:start-after: start-create-transition-rule-desc
|
||||
:end-before: end-create-transition-rule-desc
|
||||
|
||||
|
||||
4) Verify the Transition Rule
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc ilm ls` command to review the configured transition
|
||||
rules:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc ilm ls ALIAS/PATH --transition
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc ilm ls ALIAS>` with the :mc:`alias <mc alias>`
|
||||
of the MinIO deployment.
|
||||
|
||||
- Replace :mc-cmd:`PATH <mc ilm ls ALIAS>` with the name of the bucket for
|
||||
which to retrieve the configured lifecycle management rules.
|
195
source/administration/server-side-encryption.rst
Normal file
195
source/administration/server-side-encryption.rst
Normal file
@@ -0,0 +1,195 @@
|
||||
.. _minio-sse:
|
||||
.. _minio-encryption-overview:
|
||||
|
||||
=================================
|
||||
Server-Side Encryption of Objects
|
||||
=================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
.. |EK| replace:: :abbr:`EK (External Key)`
|
||||
.. |SSE| replace:: :abbr:`SSE (Server-Side Encryption)`
|
||||
.. |KMS| replace:: :abbr:`KMS (Key Management System)`
|
||||
|
||||
MinIO Server-Side Encryption (SSE) protects objects as part of write operations,
|
||||
allowing clients to take advantage of server processing power to secure objects
|
||||
at the storage layer (encryption-at-rest). SSE also provides key functionality
|
||||
to regulatory and compliance requirements around secure locking and erasure.
|
||||
|
||||
MinIO SSE uses the :minio-git:`MinIO Key Encryption Service (KES) <kes>` and an
|
||||
external Key Management Service (KMS) for performing secured cryptographic
|
||||
operations at scale. MinIO also supports client-managed key management, where
|
||||
the application takes full responsibility for creating and managing encryption
|
||||
keys for use with MinIO SSE.
|
||||
|
||||
MinIO SSE is feature and API compatible with
|
||||
:s3-docs:`AWS Server-Side Encryption <server-side-encryption.html>` and
|
||||
supports the following encryption strategies:
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: SSE-KMS *Recommended*
|
||||
:sync: sse-kms
|
||||
|
||||
MinIO supports enabling automatic SSE-KMS encryption of all objects
|
||||
written to a bucket using a specific External Key (EK) stored on the
|
||||
external |KMS|. Clients can override the bucket-default |EK| by specifying
|
||||
an explicit key as part of the write operation.
|
||||
|
||||
For buckets without automatic SSE-KMS encryption, clients can specify
|
||||
an |EK| as part of the write operation instead.
|
||||
|
||||
SSE-KMS provides more granular and customizable encryption compared to
|
||||
SSE-S3 and SSE-C and is recommended over the other supported encryption
|
||||
methods.
|
||||
|
||||
For a tutorial on enabling SSE-KMS in a local (non-production) MinIO
|
||||
Deployment, see :ref:`minio-encryption-sse-kms-quickstart`. For
|
||||
production MinIO deployments, use one of the following guides:
|
||||
|
||||
- :ref:`AWS SecretsManager <minio-sse-aws>`
|
||||
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
|
||||
- :ref:`Azure Key Vault <minio-sse-azure>`
|
||||
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
|
||||
|
||||
.. tab-item:: SSE-S3
|
||||
:sync: sse-s3
|
||||
|
||||
MinIO supports enabling automatic SSE-S3 encryption of all objects
|
||||
written to a bucket using an |EK| stored on the external |KMS|. MinIO
|
||||
SSE-S3 supports *one* |EK| for the entire deployment.
|
||||
|
||||
For buckets without automatic SSE-S3 encryption, clients can request
|
||||
SSE encryption as part of the write operation instead.
|
||||
|
||||
For a tutorial on enabling SSE-s3 in a local (non-production) MinIO
|
||||
Deployment, see :ref:`minio-encryption-sse-s3-quickstart`. For
|
||||
production MinIO deployments, use one of the following guides:
|
||||
|
||||
- :ref:`AWS SecretsManager <minio-sse-aws>`
|
||||
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
|
||||
- :ref:`Azure Key Vault <minio-sse-azure>`
|
||||
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
|
||||
|
||||
.. tab-item:: SSE-C
|
||||
:sync: sse-c
|
||||
|
||||
Clients specify an |EK| as part of the write operation for an object.
|
||||
MinIO uses the specified |EK| to perform SSE-S3.
|
||||
|
||||
SSE-C does not support bucket-default encryption settings and requires
|
||||
clients perform all key management operations.
|
||||
|
||||
MinIO SSE requires enabling :ref:`minio-tls`.
|
||||
|
||||
.. _minio-encryption-sse-secure-erasure-locking:
|
||||
|
||||
Secure Erasure and Locking
|
||||
--------------------------
|
||||
|
||||
MinIO requires access to the Encryption Key (EK) *and* external Key Management
|
||||
System (KMS) used as part of SSE operations to decrypt an object. You can use
|
||||
this dependency to securely erase and lock objects from access by disabling
|
||||
access to the EK or KMS used for encryption.
|
||||
|
||||
General strategies include, but are not limited to:
|
||||
|
||||
- Seal the |KMS| such that it cannot be accessed by MinIO server anymore. This
|
||||
locks all SSE-KMS or SSE-S3 encrypted objects protected by any |EK| stored on
|
||||
the KMS. The encrypted objects remain unreadable as long as the KMS remains
|
||||
sealed.
|
||||
|
||||
- Seal/Unmount an |EK|. This locks all SSE-KMS or SSE-S3 encrypted objects
|
||||
protected by that EK. The encrypted objects remain unreadable as long
|
||||
as the CMK(s) remains sealed.
|
||||
|
||||
- Delete an |EK|. This renders all SSE-KMS or SSE-S3 encrypted objects protected
|
||||
by that EK as permanently unreadable. The combination of deleting an EK and
|
||||
deleting the data may fulfill regulatory requirements around secure deletion
|
||||
of data.
|
||||
|
||||
Deleting an |EK| is typically irreversible. Exercise extreme caution
|
||||
before intentionally deleting a master key.
|
||||
|
||||
For more information, see:
|
||||
|
||||
- :ref:`SSE-KMS Secure Erasure and Locking
|
||||
<minio-encryption-sse-kms-erasure-locking>`
|
||||
|
||||
- :ref:`SSE-S3 Secure Erasure and Locking
|
||||
<minio-encryption-sse-s3-erasure-locking>`
|
||||
|
||||
- :ref:`SSE-C Secure Erasure and Locking
|
||||
<minio-encryption-sse-c-erasure-locking>`
|
||||
|
||||
Encryption Internals
|
||||
--------------------
|
||||
|
||||
.. note::
|
||||
|
||||
The following section describes MinIO internal logic and functionality.
|
||||
This information is purely educational and is not necessary for
|
||||
configuring or implementing any MinIO feature.
|
||||
|
||||
.. _minio-encryption-sse-content-encryption:
|
||||
|
||||
Content Encryption
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The MinIO server uses an authenticated encryption scheme
|
||||
(:ref:`AEAD <minio-encryption-sse-primitives>`) to en/decrypt and authenticate
|
||||
the object content. The AEAD is combined with some state to build a
|
||||
**Secure Channel**. A Secure Channel is a cryptographic construction that
|
||||
ensures confidentiality and integrity of the processed data. In particular, the
|
||||
Secure Channel splits the plaintext content into fixed size chunks and
|
||||
en/decrypts each chunk separately using an unique key-nonce combination.
|
||||
|
||||
The following text diagram illustrates Secure Channel Construction of an
|
||||
encrypted object:
|
||||
|
||||
The Secure Channel splits the object content into chunks of a fixed size of
|
||||
``65536`` bytes. The last chunk may be smaller to avoid adding additional
|
||||
overhead and is treated specially to prevent truncation attacks. The nonce
|
||||
value is ``96`` bits long and generated randomly per object / multi-part part.
|
||||
The Secure Channel supports plaintexts up to ``65536 * 2^32 = 256 TiB``.
|
||||
|
||||
For S3 multi-part operations, each object part is en/decrypted with the Secure
|
||||
Channel Construction scheme shown above. For each part, MinIO generates a secret
|
||||
key derived from the Object Encryption Key (OEK) and the part number using a
|
||||
pseudo-random function (:ref:`PRF <minio-encryption-sse-primitives>`), such that
|
||||
``key = PRF(OEK, part_id)``.
|
||||
|
||||
.. _minio-encryption-sse-primitives:
|
||||
|
||||
Cryptographic Primitives
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The MinIO server uses the following cryptographic primitive implementations:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 40 60
|
||||
:width: 100%
|
||||
|
||||
* -
|
||||
- Primitives
|
||||
|
||||
* - Pseudo-Random Functions (PRF)
|
||||
- HMAC-SHA-256
|
||||
|
||||
* - :ref:`AEAD <minio-encryption-sse-content-encryption>`
|
||||
- ``ChaCha20Poly-1305`` by default.
|
||||
|
||||
``AES-256-GCM`` for x86-64 CPUs with the AES-NI extension.
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
:hidden:
|
||||
|
||||
/administration/server-side-encryption/server-side-encryption-sse-kms
|
||||
/administration/server-side-encryption/server-side-encryption-sse-s3
|
||||
/administration/server-side-encryption/server-side-encryption-sse-c
|
@@ -0,0 +1,174 @@
|
||||
.. _minio-encryption-sse-c:
|
||||
|
||||
=======================================================
|
||||
Server-Side Encryption with Client-Managed Keys (SSE-C)
|
||||
=======================================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
.. |EK| replace:: :abbr:`EK (External Key)`
|
||||
.. |DEK| replace:: :abbr:`DEK (Data Encryption Key)`
|
||||
.. |KEK| replace:: :abbr:`KEK (Key Encryption Key)`
|
||||
.. |OEK| replace:: :abbr:`OEK (Object Encryption Key)`
|
||||
.. |SSE| replace:: :abbr:`SSE (Server-Side Encryption)`
|
||||
.. |KMS| replace:: :abbr:`KMS (Key Management Service)`
|
||||
.. |KES| replace:: :abbr:`KES (Key Encryption Service)`
|
||||
|
||||
MinIO Server-Side Encryption (SSE) protects objects as part of write operations,
|
||||
allowing clients to take advantage of server processing power to secure objects
|
||||
at the storage layer (encryption-at-rest). SSE also provides key functionality
|
||||
to regulatory and compliance requirements around secure locking and erasure.
|
||||
|
||||
The procedure on this page configures and enables Server-Side Encryption
|
||||
with Client-Managed Keys (SSE-C). MinIO SSE-C supports client-driven
|
||||
encryption of objects *before* writing the object to disk. Clients must
|
||||
specify the correct key to decrypt objects for read operations.
|
||||
|
||||
MinIO SSE-C is functionally compatible with Amazon
|
||||
:s3-docs:`Server-Side Encryption with Customer-Provided Keys
|
||||
<ServerSideEncryptionCustomerKeys.html>`.
|
||||
|
||||
.. _minio-encryption-sse-c-erasure-locking:
|
||||
|
||||
Secure Erasure and Locking
|
||||
--------------------------
|
||||
|
||||
SSE-C protects objects using an |EK| specified by the client as part
|
||||
of the write operation. Assuming the client-side key management
|
||||
supports disabling or deleting these keys:
|
||||
|
||||
- Disabling the |EK| temporarily locks any objects encrypted using that
|
||||
|EK| by rendering them unreadable. You can later enable the |EK| to
|
||||
resume normal read operations on those objects.
|
||||
|
||||
- Deleting the |EK| renders all objects encrypted by that |EK|
|
||||
*permanently* unreadable. If the client-side KMS does not support
|
||||
backups of the |EK|, this process is *irreversible*.
|
||||
|
||||
The scope of a single |EK| depends on the number of write operations
|
||||
which specified that |EK| when requesting SSE-C encryption.
|
||||
|
||||
Considerations
|
||||
--------------
|
||||
|
||||
SSE-C is Incompatible with Bucket Replication
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
SSE-C encrypted objects are not compatible with MinIO
|
||||
:ref:`bucket replication <minio-bucket-replication>`. Use
|
||||
:ref:`SSE-KMS <minio-encryption-sse-kms>` or
|
||||
:ref:`SSE-S3 <minio-encryption-sse-s3>` to ensure encrypted
|
||||
objects are compatible with bucket replication.
|
||||
|
||||
SSE-C Overrides SSE-S3 and SSE-KMS
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Encrypting an object using SSE-C prevents MinIO from applying
|
||||
:ref:`SSE-KMS <minio-encryption-sse-kms>` or
|
||||
:ref:`SSE-S3 <minio-encryption-sse-s3>` encryption to that object.
|
||||
|
||||
Quickstart
|
||||
----------
|
||||
|
||||
MinIO SSE-C requires the client to perform all key creation and storage
|
||||
operations.
|
||||
|
||||
This procedure uses :mc:`mc` for performing operations on the source MinIO
|
||||
deployment. Install :mc:`mc` on a machine with network access to the source
|
||||
deployment. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
|
||||
instructions on downloading and installing ``mc``.
|
||||
|
||||
The SSE-C key *must* be a 256-bit base64-encoded string. The client
|
||||
application is responsible for generation and storage of the encryption key.
|
||||
MinIO does *not* store SSE-C encryption keys and cannot decrypt SSE-C
|
||||
encrypted objects without the client-managed key.
|
||||
|
||||
1) Generate the Encryption Key
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Generate the 256-bit base64-encoded string for use as the encryption key.
|
||||
|
||||
The following example generates a string that meets the encryption key
|
||||
requirements. The resulting string is appropriate for non-production
|
||||
environments:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
cat /dev/urandom | head -c 32 | base64 -
|
||||
|
||||
Defer to your organizations requirements for generating cryptographically
|
||||
secure encryption keys.
|
||||
|
||||
Copy the encryption key for use in the next step.
|
||||
|
||||
2) Encrypt an Object using SSE-C
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports the following AWS S3 headers for specifying SSE-C encryption:
|
||||
|
||||
- ``X-Amz-Server-Side-Encryption-Customer-Algorithm`` set to ``AES256``.
|
||||
|
||||
- ``X-Amz-Server-Side-Encryption-Customer-Key`` set to the encryption key value.
|
||||
|
||||
- ``X-Amz-Server-Side-Encryption-Customer-Key-MD5`` to the 128-bit MD5 digest of
|
||||
the encryption key.
|
||||
|
||||
The MinIO :mc:`mc` commandline tool S3-compatible SDKs include specific syntax
|
||||
for setting headers. Certain :mc:`mc` commands like :mc:`mc cp` include specific
|
||||
arguments for enabling SSE-S3 encryption:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp ~/data/mydata.json ALIAS/BUCKET/mydata.json \
|
||||
--encrypt-key "ALIAS/BUCKET/=c2VjcmV0ZW5jcnlwdGlvbmtleWNoYW5nZW1lMTIzNAo="
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc encrypt set ALIAS>` with the
|
||||
:mc:`alias <mc alias>` of the MinIO deployment on which you want to write
|
||||
the SSE-C encrypted object.
|
||||
|
||||
- Replace :mc-cmd:`BUCKET <mc encrypt set ALIAS>` with the full path to the
|
||||
bucket or bucket prefix to which you want to write the SSE-C encrypted object.
|
||||
|
||||
3) Copy an SSE-C Encrypted Object
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO supports the following AWS S3 headers for copying an SSE-C encrypted
|
||||
object to another S3-compatible service:
|
||||
|
||||
- ``X-Amz-Copy-Source-Server-Side-Encryption-Algorithm`` set to ``AES256``
|
||||
|
||||
- ``X-Amz-Copy-Source-Server-Side-Encryption-Key`` set to the encryption key
|
||||
value. The copy operation will fail if the specified key does not match
|
||||
the key used to SSE-C encrypt the object.
|
||||
|
||||
- ``X-Amz-Copy-Source-Server-Side-Encryption-Key-MD5`` set to the 128-bit MD5
|
||||
digest of the encryption key.
|
||||
|
||||
The MinIO :mc:`mc` commandline tool S3-compatible SDKs include specific syntax
|
||||
for setting headers. Certain :mc:`mc` commands like :mc:`mc cp` include specific
|
||||
arguments for enabling SSE-S3 encryption:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp SOURCE/BUCKET/mydata.json TARGET/BUCKET/mydata.json \
|
||||
--encrypt-key "SOURCE/BUCKET/=c2VjcmV0ZW5jcnlwdGlvbmtleWNoYW5nZW1lMTIzNAo=" \
|
||||
--encrypt-key "TARGET/BUCKET/=c2VjcmV0ZW5jcnlwdGlvbmtleWNoYW5nZW1lMTIzNAo="
|
||||
|
||||
- Replace :mc-cmd:`SOURCE/BUCKET <mc encrypt set ALIAS>` with the
|
||||
:mc:`alias <mc alias>` of the MinIO deployment from which you are reading the
|
||||
encrypted object and the full path to the
|
||||
bucket or bucket prefix from which you want to read the SSE-C encrypted
|
||||
object.
|
||||
|
||||
- Replace :mc-cmd:`TARGET/BUCKET <mc encrypt set ALIAS>` with the
|
||||
:mc:`alias <mc alias>` of the MinIO deployment from which you are writing the
|
||||
encrypted object and the full path to the
|
||||
bucket or bucket prefix to which you want to write the SSE-C encrypted
|
||||
object.
|
@@ -0,0 +1,323 @@
|
||||
.. _minio-encryption-sse-kms:
|
||||
|
||||
=====================================================
|
||||
Server-Side Encryption with Per-Bucket Keys (SSE-KMS)
|
||||
=====================================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
.. |EK| replace:: :abbr:`EK (External Key)`
|
||||
.. |DEK| replace:: :abbr:`DEK (Data Encryption Key)`
|
||||
.. |KEK| replace:: :abbr:`KEK (Key Encryption Key)`
|
||||
.. |OEK| replace:: :abbr:`OEK (Object Encryption Key)`
|
||||
.. |SSE| replace:: :abbr:`SSE (Server-Side Encryption)`
|
||||
.. |KMS| replace:: :abbr:`KMS (Key Management Service)`
|
||||
.. |KES| replace:: :abbr:`KES (Key Encryption Service)`
|
||||
|
||||
MinIO Server-Side Encryption (SSE) protects objects as part of write operations,
|
||||
allowing clients to take advantage of server processing power to secure objects
|
||||
at the storage layer (encryption-at-rest). SSE also provides key functionality
|
||||
to regulatory and compliance requirements around secure locking and erasure.
|
||||
|
||||
MinIO SSE uses the :minio-git:`MinIO Key Encryption Service (KES) <kes>` and an
|
||||
external Key Management Service (KMS) for performing secured cryptographic
|
||||
operations at scale. MinIO also supports client-managed key management, where
|
||||
the application takes full responsibility for creating and managing encryption
|
||||
keys for use with MinIO SSE.
|
||||
|
||||
MinIO SSE-KMS en/decrypts objects using an External Key (EK) managed by a Key
|
||||
Management System (KMS). Each bucket and object can have a separate |EK|,
|
||||
supporting more granular cryptographic operations in the deployment. MinIO can
|
||||
only decrypt an object if it can access both the KMS *and* the |EK| used to
|
||||
encrypt that object.
|
||||
|
||||
You can enable bucket-default SSE-KMS encryption using the
|
||||
:mc-cmd:`mc encrypt set` command:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc encrypt set sse-kms EXTERNALKEY play/mybucket
|
||||
|
||||
- Replace ``EXTERNALKEY`` with the name of the |EK| to use for encrypting
|
||||
objects in the bucket.
|
||||
|
||||
- Replace ``play/mybucket`` with the :mc-cmd:`alias <mc alias>` and bucket
|
||||
on which you want to enable automatic SSE-KMS encryption.
|
||||
|
||||
MinIO SSE-KMS is functionally compatible with AWS S3
|
||||
:s3-docs:`Server-Side Encryption with KMS keys stored in AWS
|
||||
<UsingKMSEncryption.html>` while expanding support to include the
|
||||
following KMS providers:
|
||||
|
||||
- :ref:`AWS SecretsManager <minio-sse-aws>`
|
||||
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
|
||||
- :ref:`Azure Key Vault <minio-sse-azure>`
|
||||
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
|
||||
- Thales CipherTrust (formerly Gemalto KeySecure)
|
||||
|
||||
.. _minio-encryption-sse-kms-quickstart:
|
||||
|
||||
Quickstart
|
||||
----------
|
||||
|
||||
The following procedure uses the ``play`` MinIO |KES| sandbox for
|
||||
supporting |SSE| with SSE-KMS in evaluation and early development environments.
|
||||
|
||||
For extended development or production environments, use one of the following
|
||||
supported external Key Management Services (KMS):
|
||||
|
||||
- :ref:`AWS SecretsManager <minio-sse-aws>`
|
||||
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
|
||||
- :ref:`Azure Key Vault <minio-sse-azure>`
|
||||
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
|
||||
|
||||
.. include:: /includes/common-minio-kes.rst
|
||||
:start-after: start-kes-play-sandbox-warning
|
||||
:end-before: end-kes-play-sandbox-warning
|
||||
|
||||
This procedure requires the following components:
|
||||
|
||||
- Install :mc:`mc` on a machine with network access to the source
|
||||
deployment. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
|
||||
instructions on downloading and installing ``mc``.
|
||||
|
||||
|
||||
- Install :minio-git:`MinIO Key Encryption Service (KES) <kes>` on a machine
|
||||
with internet access. See the ``kes``
|
||||
:minio-git:`Getting Started <kes/wiki/Getting-Started>` guide for
|
||||
instructions on downloading, installing, and configuring KES.
|
||||
|
||||
1) Create an Encryption Key for SSE-KMS Encryption
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :minio-git:`kes <kes>` commandline tool to create a new External Key
|
||||
(EK) for use with SSE-KMS Encryption.
|
||||
|
||||
Issue the following command to retrieve the root
|
||||
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>` for the KES
|
||||
server:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
curl -sSL --tlsv1.2 \
|
||||
-O 'https://raw.githubusercontent.com/minio/kes/master/root.key' \
|
||||
-O 'https://raw.githubusercontent.com/minio/kes/master/root.cert'
|
||||
|
||||
Set the following environment variables in the terminal or shell:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
export KES_CLIENT_KEY=root.key
|
||||
export KES_CLIENT_CERT=root.cert
|
||||
|
||||
.. list-table::
|
||||
:stub-columns: 1
|
||||
:widths: 40 60
|
||||
:width: 100%
|
||||
|
||||
* - ``KES_CLIENT_KEY``
|
||||
- The private key for an :minio-git:`identity
|
||||
<kes/wiki/Configuration#policy-configuration>` on the KES server.
|
||||
The identity must grant access to at minimum the ``/v1/create``,
|
||||
``/v1/generate``, and ``/v1/list`` :minio-git:`API endpoints
|
||||
<kes/wiki/Server-API#api-overview>`. This step uses the root
|
||||
identity for the MinIO ``play`` KES sandbox, which provides access
|
||||
to all operations on the KES server.
|
||||
|
||||
* - ``KES_CLIENT_CERT``
|
||||
- The corresponding certificate for the :minio-git:`identity
|
||||
<kes/wiki/Configuration#policy-configuration>` on the KES server.
|
||||
This step uses the root identity for the MinIO ``play`` KES
|
||||
sandbox, which provides access to all operations on the KES server.
|
||||
|
||||
Issue the following command to create a new |EK| through
|
||||
KES.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
kes key create my-minio-sse-kms-key
|
||||
|
||||
This tutorial uses the example ``my-minio-sse-kms-key`` name for ease of
|
||||
reference. Specify a unique key name to prevent collision
|
||||
with existing keys.
|
||||
|
||||
2) Configure MinIO for SSE-KMS Object Encryption
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Specify the following environment variables in the shell or terminal on each
|
||||
MinIO server host in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
export MINIO_KMS_KES_ENDPOINT=https://play.min.io:7373
|
||||
export MINIO_KMS_KES_KEY_FILE=root.key
|
||||
export MINIO_KMS_KES_CERT_FILE=root.cert
|
||||
export MINIO_KMS_KES_KEY_NAME=my-minio-sse-kms-key
|
||||
|
||||
.. list-table::
|
||||
:stub-columns: 1
|
||||
:widths: 30 80
|
||||
|
||||
* - :envvar:`MINIO_KMS_KES_ENDPOINT`
|
||||
- The endpoint for the MinIO ``Play`` KES service.
|
||||
|
||||
* - :envvar:`MINIO_KMS_KES_KEY_FILE`
|
||||
- The private key file corresponding to an
|
||||
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>`
|
||||
on the KES service. The identity must grant permission to
|
||||
create, generate, and decrypt keys. Specify the same
|
||||
identity key file as the ``KES_KEY_FILE`` environment variable
|
||||
in the previous step.
|
||||
|
||||
* - :envvar:`MINIO_KMS_KES_CERT_FILE`
|
||||
- The public certificate file corresponding to an
|
||||
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>`
|
||||
on the KES service. The identity must grant permission to
|
||||
create, generate, and decrypt keys. Specify the same
|
||||
identity certificate as the ``KES_CERT_FILE`` environment
|
||||
variable in the previous step.
|
||||
|
||||
* - :envvar:`MINIO_KMS_KES_KEY_NAME`
|
||||
- The name of the External Key (EK) to use for
|
||||
performing SSE encryption operations. KES retrieves the |EK| from
|
||||
the configured Key Management Service (KMS). Specify the name of the
|
||||
key created in the previous step.
|
||||
|
||||
|
||||
3) Restart the MinIO Deployment to Enable SSE-KMS
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
4) Configure Automatic Bucket Encryption
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :mc-cmd:`mc encrypt set` command to enable automatic SSE-KMS protection
|
||||
of all objects written to a specific bucket.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc encrypt set sse-kms my-minio-sse-kms-key ALIAS/BUCKET
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc encrypt set ALIAS>` with the
|
||||
:mc:`alias <mc alias>` of the MinIO deployment on which you enabled SSE-KMS.
|
||||
|
||||
- Replace :mc-cmd:`BUCKET <mc encrypt set ALIAS>` with the full path to the
|
||||
bucket or bucket prefix on which you want to enable automatic SSE-KMS.
|
||||
|
||||
Objects written to the specified bucket are automatically encrypted using
|
||||
the specified |EK|
|
||||
|
||||
Repeat this step for each bucket on which you want to enable automatic
|
||||
SSE-KMS encryption. You can generate additional keys per bucket or bucket
|
||||
prefix, such that the scope of each |EK| is limited to a subset of objects.
|
||||
|
||||
|
||||
.. _minio-encryption-sse-kms-erasure-locking:
|
||||
|
||||
Secure Erasure and Locking
|
||||
--------------------------
|
||||
|
||||
SSE-KMS protects objects using an |EK| specified either as part of the
|
||||
bucket automatic encryption settings *or* as part of the write operation.
|
||||
MinIO therefore *requires* access to that |EK| for decrypting that object.
|
||||
|
||||
- Disabling the |EK| temporarily locks objects encrypted with that |EK| by
|
||||
rendering them unreadable. You can later enable the |EK| to resume
|
||||
normal read operations on those objects.
|
||||
|
||||
- Deleting the |EK| renders all objects encrypted by that |EK|
|
||||
*permanently* unreadable. If the KMS does not have or support backups
|
||||
of the |EK|, this process is *irreversible*.
|
||||
|
||||
The scope of a single |EK| depends on:
|
||||
|
||||
- Which buckets specified that |EK| for automatic SSE-KMS encryption,
|
||||
*and*
|
||||
- Which write operations specified that |EK| when requesting SSE-KMS
|
||||
encryption.
|
||||
|
||||
For example, consider a MinIO deployment using one |EK| per bucket.
|
||||
Disabling a single |EK| renders all objects in the associated bucket
|
||||
unreadable without affecting other buckets. If the deployment instead used
|
||||
one |EK| for all objects and buckets, disabling that |EK| renders all
|
||||
objects in the deployment unreadable.
|
||||
|
||||
.. _minio-encryption-sse-kms-encryption-process:
|
||||
|
||||
Encryption Process
|
||||
------------------
|
||||
|
||||
.. note::
|
||||
|
||||
This section describes MinIO internal logic and functionality.
|
||||
This information is purely educational and is not a prerequisite for
|
||||
configuring or implementing any MinIO feature.
|
||||
|
||||
SSE-KMS uses an External Key (EK) managed by the configured Key Management
|
||||
System (KMS) for performing cryptographic operations and protecting objects.
|
||||
The table below describes each stage of the encryption process:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
|
||||
* - Stage
|
||||
- Description
|
||||
|
||||
* - SSE-Enabled Write Operation
|
||||
- MinIO receives a write operation requesting SSE-KMS encryption.
|
||||
The write operation *must* have an associated External Key (EK) to use
|
||||
for encrypting the object.
|
||||
|
||||
- For write operations in buckets with automatic SSE-KMS enabled,
|
||||
MinIO uses the bucket |EK|. If the write operation includes an
|
||||
explicit EK, MinIO uses that *instead* of the bucket EK.
|
||||
|
||||
- For write operations in buckets *without* automatic SSE-KMS enabled,
|
||||
MinIO uses the |EK| specified to the write operation.
|
||||
|
||||
* - Generate the Data Encryption Key (DEK)
|
||||
- .. include:: /includes/common-minio-sse.rst
|
||||
:start-after: start-sse-dek
|
||||
:end-before: end-sse-dek
|
||||
|
||||
* - Generate the Key Encryption Key (KEK)
|
||||
- .. include:: /includes/common-minio-sse.rst
|
||||
:start-after: start-sse-kek
|
||||
:end-before: end-sse-kek
|
||||
|
||||
* - Generate the Object Encryption Key (OEK)
|
||||
- .. include:: /includes/common-minio-sse.rst
|
||||
:start-after: start-sse-oek
|
||||
:end-before: end-sse-oek
|
||||
|
||||
* - Encrypt the Object
|
||||
- MinIO uses the |OEK| to encrypt the object *prior* to storing the
|
||||
object to disk. MinIO then encrypts the |OEK| with the |KEK|.
|
||||
|
||||
MinIO stores the encrypted representation of the |OEK| and |DEK| as part
|
||||
of the metadata.
|
||||
|
||||
For read operations, MinIO decrypts the object by retrieving the |EK| to
|
||||
decrypt the |DEK|. MinIO then regenerates the |KEK|, decrypts the |OEK|, and
|
||||
decrypts the object.
|
@@ -0,0 +1,296 @@
|
||||
.. _minio-encryption-sse-s3:
|
||||
|
||||
==================================================
|
||||
Server-Side Encryption Per-Deployment Key (SSE-S3)
|
||||
==================================================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
.. |EK| replace:: :abbr:`EK (External Key)`
|
||||
.. |DEK| replace:: :abbr:`DEK (Data Encryption Key)`
|
||||
.. |KEK| replace:: :abbr:`KEK (Key Encryption Key)`
|
||||
.. |OEK| replace:: :abbr:`OEK (Object Encryption Key)`
|
||||
.. |SSE| replace:: :abbr:`SSE (Server-Side Encryption)`
|
||||
.. |KMS| replace:: :abbr:`KMS (Key Management Service)`
|
||||
.. |KES| replace:: :abbr:`KES (Key Encryption Service)`
|
||||
|
||||
MinIO Server-Side Encryption (SSE) protects objects as part of write operations,
|
||||
allowing clients to take advantage of server processing power to secure objects
|
||||
at the storage layer (encryption-at-rest). SSE also provides key functionality
|
||||
to regulatory and compliance requirements around secure locking and erasure.
|
||||
|
||||
MinIO SSE uses the :minio-git:`MinIO Key Encryption Service (KES) <kes>` and an
|
||||
external Key Management Service (KMS) for performing secured cryptographic
|
||||
operations at scale. MinIO also supports client-managed key management, where
|
||||
the application takes full responsibility for creating and managing encryption
|
||||
keys for use with MinIO SSE.
|
||||
|
||||
MinIO SSE-S3 en/decrypts objects using an External Key (EK) managed by a
|
||||
Key Management System (KMS). You must specify the |EK| using the
|
||||
:envvar:`MINIO_KMS_KES_KEY_NAME` environment variable when starting up the
|
||||
MinIO server. MinIO uses the same EK for *all* SSE-S3 cryptographic operations.
|
||||
|
||||
You can enable bucket-default SSE-S3 encryption using the
|
||||
:mc-cmd:`mc encrypt set` command:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc encrypt set sse-s3 play/mybucket
|
||||
|
||||
- Replace ``play/mybucket`` with the :mc-cmd:`alias <mc alias>` and bucket
|
||||
on which you want to enable automatic SSE-KMS encryption.
|
||||
|
||||
MinIO SSE-S3 is functionally compatible with AWS S3
|
||||
:s3-docs:`Server-Side Encryption with Amazon S3-Managed Keys
|
||||
<UsingServerSideEncryption.html>` while expanding support to include the
|
||||
following KMS providers:
|
||||
|
||||
- :ref:`AWS SecretsManager <minio-sse-aws>`
|
||||
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
|
||||
- :ref:`Azure Key Vault <minio-sse-azure>`
|
||||
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
|
||||
- Thales CipherTrust (formerly Gemalto KeySecure)
|
||||
|
||||
.. _minio-encryption-sse-s3-quickstart:
|
||||
|
||||
Quickstart
|
||||
----------
|
||||
|
||||
The following procedure uses the ``play`` MinIO |KES| sandbox for
|
||||
supporting |SSE| with SSE-S3 in evaluation and early development environments.
|
||||
|
||||
For extended development or production environments, use one of the following
|
||||
supported external Key Management Services (KMS):
|
||||
|
||||
- :ref:`AWS SecretsManager <minio-sse-aws>`
|
||||
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
|
||||
- :ref:`Azure Key Vault <minio-sse-azure>`
|
||||
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
|
||||
- Thales CipherTrust (formerly Gemalto KeySecure)
|
||||
|
||||
.. include:: /includes/common-minio-kes.rst
|
||||
:start-after: start-kes-play-sandbox-warning
|
||||
:end-before: end-kes-play-sandbox-warning
|
||||
|
||||
This procedure requires the following components:
|
||||
|
||||
- Install :mc:`mc` on a machine with network access to the source
|
||||
deployment. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
|
||||
instructions on downloading and installing ``mc``.
|
||||
|
||||
|
||||
- Install :minio-git:`MinIO Key Encryption Service (KES) <kes>` on a machine
|
||||
with internet access. See the ``kes``
|
||||
:minio-git:`Getting Started <kes/wiki/Getting-Started>` guide for
|
||||
instructions on downloading, installing, and configuring KES.
|
||||
|
||||
|
||||
1) Create an Encryption Key for SSE-S3 Encryption
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the :minio-git:`kes <kes>` commandline tool to create a new External Key
|
||||
(EK) for use with SSE-S3 Encryption.
|
||||
|
||||
Issue the following command to retrieve the root
|
||||
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>` for the KES
|
||||
server:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
curl -sSL --tlsv1.2 \
|
||||
-O 'https://raw.githubusercontent.com/minio/kes/master/root.key' \
|
||||
-O 'https://raw.githubusercontent.com/minio/kes/master/root.cert'
|
||||
|
||||
Set the following environment variables in the terminal or shell:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
export KES_CLIENT_KEY=root.key
|
||||
export KES_CLIENT_CERT=root.cert
|
||||
|
||||
.. list-table::
|
||||
:stub-columns: 1
|
||||
:widths: 40 60
|
||||
:width: 100%
|
||||
|
||||
* - ``KES_CLIENT_KEY``
|
||||
- The private key for an :minio-git:`identity
|
||||
<kes/wiki/Configuration#policy-configuration>` on the KES server.
|
||||
The identity must grant access to at minimum the ``/v1/create``,
|
||||
``/v1/generate``, and ``/v1/list`` :minio-git:`API endpoints
|
||||
<kes/wiki/Server-API#api-overview>`. This step uses the root
|
||||
identity for the MinIO ``play`` KES sandbox, which provides access
|
||||
to all operations on the KES server.
|
||||
|
||||
* - ``KES_CLIENT_CERT``
|
||||
- The corresponding certificate for the :minio-git:`identity
|
||||
<kes/wiki/Configuration#policy-configuration>` on the KES server.
|
||||
This step uses the root identity for the MinIO ``play`` KES
|
||||
sandbox, which provides access to all operations on the KES server.
|
||||
|
||||
Issue the following command to create a new |EK| through
|
||||
KES:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
kes key create my-minio-sse-s3-key
|
||||
|
||||
This tutorial uses the example ``my-minio-sse-s3-key`` name for ease of
|
||||
reference. Specify a unique key name to prevent collision with existing keys.
|
||||
|
||||
2) Configure MinIO for SSE-S3 Object Encryption
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Specify the following environment variables in the shell or terminal on each
|
||||
MinIO server host in the deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
export MINIO_KMS_KES_ENDPOINT=https://play.min.io:7373
|
||||
export MINIO_KMS_KES_KEY_FILE=root.key
|
||||
export MINIO_KMS_KES_CERT_FILE=root.cert
|
||||
export MINIO_KMS_KES_KEY_NAME=my-minio-sse-s3-key
|
||||
|
||||
.. list-table::
|
||||
:stub-columns: 1
|
||||
:widths: 30 80
|
||||
|
||||
* - :envvar:`MINIO_KMS_KES_ENDPOINT`
|
||||
- The endpoint for the MinIO ``Play`` KES service.
|
||||
|
||||
* - :envvar:`MINIO_KMS_KES_KEY_FILE`
|
||||
- The private key file corresponding to an
|
||||
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>`
|
||||
on the KES service. The identity must grant permission to
|
||||
create, generate, and decrypt keys. Specify the same
|
||||
identity key file as the ``KES_KEY_FILE`` environment variable
|
||||
in the previous step.
|
||||
|
||||
* - :envvar:`MINIO_KMS_KES_CERT_FILE`
|
||||
- The public certificate file corresponding to an
|
||||
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>`
|
||||
on the KES service. The identity must grant permission to
|
||||
create, generate, and decrypt keys. Specify the same
|
||||
identity certificate as the ``KES_CERT_FILE`` environment
|
||||
variable in the previous step.
|
||||
|
||||
* - :envvar:`MINIO_KMS_KES_KEY_NAME`
|
||||
- The name of the External Key (EK) to use for
|
||||
performing SSE encryption operations. KES retrieves the |EK| from
|
||||
the configured Key Management System (KMS). Specify the name of the
|
||||
key created in the previous step.
|
||||
|
||||
3) Restart the MinIO Deployment to Enable SSE-S3
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must restart the MinIO deployment to apply the configuration changes.
|
||||
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc admin service restart ALIAS
|
||||
|
||||
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
|
||||
restart.
|
||||
|
||||
4) Configure Automatic Bucket Encryption
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
*Optional*
|
||||
|
||||
You can skip this step if you intend to use only client-driven SSE-S3.
|
||||
|
||||
Use the :mc-cmd:`mc encrypt set` command to enable automatic SSE-S3 protection
|
||||
of all objects written to a specific bucket.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc encrypt set sse-s3 ALIAS/BUCKET
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc encrypt set ALIAS>` with the
|
||||
:mc:`alias <mc alias>` of the MinIO deployment on which you enabled SSE-S3.
|
||||
|
||||
- Replace :mc-cmd:`BUCKET <mc encrypt set ALIAS>` with the full path to the
|
||||
bucket or bucket prefix on which you want to enable automatic SSE-S3.
|
||||
|
||||
.. _minio-encryption-sse-s3-erasure-locking:
|
||||
|
||||
Secure Erasure and Locking
|
||||
--------------------------
|
||||
|
||||
SSE-S3 protects objects using an |EK| specified at server startup
|
||||
using the :envvar:`MINIO_KMS_KES_KEY_NAME` environment variable. MinIO
|
||||
therefore *requires* access to that |EK| for decrypting that object.
|
||||
|
||||
- Disabling the |EK| temporarily locks SSE-S3-encrypted objects in the
|
||||
deployment by rendering them unreadable. You can later enable the |EK|
|
||||
to resume normal read operations.
|
||||
|
||||
- Deleting the |EK| renders all SSE-S3-encrypted objects in the deployment
|
||||
*permanently* unreadable. If the KMS does not have or support backups
|
||||
of the |EK|, this process is *irreversible*.
|
||||
|
||||
The scope of the |EK| depends on:
|
||||
|
||||
- Which buckets specified automatic SSE-S3 encryption, *and*
|
||||
- Which write operations requested SSE-S3 encryption.
|
||||
|
||||
.. _minio-encryption-sse-s3-encryption-process:
|
||||
|
||||
Encryption Process
|
||||
------------------
|
||||
|
||||
.. note::
|
||||
|
||||
The following section describes MinIO internal logic and functionality.
|
||||
This information is purely educational and is not necessary for
|
||||
configuring or implementing any MinIO feature.
|
||||
|
||||
SSE-S3 uses an External Key (EK) managed by the configured Key Management
|
||||
System (KMS) for performing cryptographic operations and protecting objects.
|
||||
The table below describes each stage of the encryption process:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
|
||||
* - Stage
|
||||
- Description
|
||||
|
||||
* - SSE-Enabled Write Operation
|
||||
- MinIO receives a write operation requesting SSE-S3 encryption.
|
||||
MinIO uses the key name specified to
|
||||
:envvar:`MINIO_KMS_KES_KEY_NAME` as the External Key (EK).
|
||||
|
||||
* - Generate the Data Encryption Key (DEK)
|
||||
- .. include:: /includes/common-minio-sse.rst
|
||||
:start-after: start-sse-dek
|
||||
:end-before: end-sse-dek
|
||||
|
||||
* - Generate the Key Encryption Key (KEK)
|
||||
- .. include:: /includes/common-minio-sse.rst
|
||||
:start-after: start-sse-kek
|
||||
:end-before: end-sse-kek
|
||||
|
||||
* - Generate the Object Encryption Key (OEK)
|
||||
- .. include:: /includes/common-minio-sse.rst
|
||||
:start-after: start-sse-oek
|
||||
:end-before: end-sse-oek
|
||||
|
||||
* - Encrypt the Object
|
||||
- MinIO uses the |OEK| to encrypt the object *prior* to storing the
|
||||
object to disk. MinIO then encrypts the |OEK| with the |KEK|.
|
||||
|
||||
MinIO stores the encrypted representation of the |OEK| and |DEK| as part
|
||||
of the metadata.
|
Reference in New Issue
Block a user