1
0
mirror of https://github.com/minio/docs.git synced 2025-08-08 01:43:18 +03:00

Docs Multiplatform Slice

This commit is contained in:
Ravind Kumar
2022-05-06 16:44:42 -04:00
parent df33ddee6a
commit b99c20a16f
134 changed files with 3689 additions and 2200 deletions

View File

@@ -0,0 +1,320 @@
.. _minio-bucket-replication:
==================
Bucket Replication
==================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports server-side and client-side replication of objects between source
and destination buckets.
:ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`
Configure per-bucket rules for automatically synchronizing objects between
buckets within the same MinIO cluster *or* between two independent MinIO
Clusters. MinIO applies rules as part of object write operations
(e.g. ``PUT``) and automatically synchronizes new objects *and* object
mutations, such as new object versions or changes to object metadata.
MinIO server-side bucket replication only supports MinIO clusters for the
remote replication target.
Client-side Bucket Replication
Use The command process to synchronize objects between buckets
within the same S3-compatible cluster *or* between two independent
S3-compatible clusters. Client-side replication using :mc-cmd:`mc mirror`
supports MinIO-to-S3 and similar replication configurations.
.. admonition:: Bucket vs Site Replication
:class: note
Bucket Replication is distinct from and mutually exclusive with :ref:`site replication <minio-site-replication-overview>`.
- Bucket Replication synchronizes data at the bucket level, such as bucket prefix paths and objects.
You can configure bucket replication at any time, and the remote MinIO deployments may have pre-existing data on the replication target buckets.
- Site Replication extends bucket replication to include :ref:`IAM <minio-authentication-and-identity-management>`, security tokens, service accounts, and bucket-level configurations.
Site replication is typically configured when initially deploying the MinIO peer sites.
Only one site can hold any bucket or objects at the time of initial configuration.
.. _minio-bucket-replication-serverside:
Server-Side Bucket Replication
------------------------------
MinIO server-side bucket replication is an automatic bucket-level configuration
that synchronizes objects between a source and destination bucket. MinIO
server-side replication *requires* the source and destination bucket be two separate MinIO
clusters.
For each write operation to the bucket, MinIO checks all configured replication
rules for the bucket and applies the matching rule with highest configured
priority. MinIO synchronizes new objects *and* object mutations, such as
new object versions or changes to object metadata. This includes
metadata operations such as enabling or modifying object locking or
retention settings.
MinIO server-side bucket replication is functionally similar to Amazon S3
replication while adding the following MinIO-only features:
- Source and destination bucket names can match, supporting site-to-site
use cases such as Splunk or Veeam BC/DR.
- Simplified implementation than S3 bucket replication configuration, removing
the need to configure settings like AccessControlTranslation, Metrics, and
SourceSelectionCriteria.
- Active-Active (Two-Way) replication of objects between source and destination
buckets.
- Multi-Site replication of objects between three or more MinIO deployments
(New in :minio-release:`RELEASE.2021-09-23T04-46-24Z`).
.. _minio-replication-behavior-resync:
Resynchronization (Disaster Recovery)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Resynchronization primarily supports recovery after partial or total loss of the
data on a MinIO deployment using a healthy deployment in the replica
configuration. Use the :mc-cmd:`mc replicate resync` command completely
resynchronize the remote target (:mc-cmd:`mc admin bucket remote`) using the
specified source bucket.
The resynchronization process checks all objects in the source bucket against
all configured replication rules that include :ref:`existing object replication
<minio-replication-behavior-existing-objects>`. For each object which matches a
rule, the resynchronization process places the object into the replication
:ref:`queue <minio-replication-process>` regardless of the object's current
:ref:`replication status <minio-replication-process>`.
MinIO skips synchronizing those objects whose remote copy exactly match the
source, including object metadata. MinIO otherwise does not prioritize or modify
the queue with regards to the existing contents of the target.
:mc-cmd:`mc replicate resync` operates at the bucket level and does
*not* support prefix-level granularity. Initiating resynchronization on a large
bucket may result in a significant increase in replication-related load
and traffic. Use this command with caution and only when necessary.
For buckets with :ref:`object transition (Tiering)
<minio-lifecycle-management-tiering>` configured, replication resynchronization
restores objects in a non-transitioned state with no associated transition
metadata. Any data previously transitioned to the remote storage is therefore
permanently disconnected from the remote MinIO deployment. For tiering
configurations which specify an explicit human-readable prefix as part of the
remote configuration, you can safely purge the transitioned data in that prefix
to avoid costs associated to the "lost" data.
.. _minio-replication-behavior-delete:
Replication of Delete Operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating delete operations, where MinIO synchronizes
deleting specific object versions *and* new
:s3-docs:`delete markers <delete-marker-replication.html>`. Delete operation
replication uses the same :ref:`replication process <minio-replication-process>`
as all other replication operations.
MinIO requires explicitly enabling versioned deletes and delete marker
replication . Use the :mc-cmd:`mc replicate add --replicate` field to
specify both or either ``delete`` and ``delete-marker`` to enable versioned
deletes and delete marker replication respectively. To enable both, specify both
strings using a comma separator ``delete,delete-marker``.
For delete marker replication, MinIO begins the replication process after
a delete operation creates the delete marker. MinIO uses the
``X-Minio-Replication-DeleteMarker-Status`` metadata field for tracking
delete marker replication status. In
:ref:`active-active <minio-bucket-replication-serverside-twoway>`
replication configurations, MinIO may produce duplicate delete markers if
both clusters concurrently create a delete marker for an object *or*
if one or both clusters were down before the replication event synchronized.
For replicating the deletion of a specific object version, MinIO marks the
object version as ``PENDING`` until replication completes. Once the remote
target deletes that object version, MinIO deletes the object on the source.
While this process ensures near-synchronized version deletion, it may result
in listing operations returning the object version after the initial
delete operation. MinIO uses the ``X-Minio-Replication-Delete-Status`` for
tracking delete version replication status.
MinIO only replicates explicit client-driven delete operations. MinIO does *not*
replicate objects deleted from the application of
:ref:`lifecycle management expiration rules
<minio-lifecycle-management-expiration>`. For :ref:`active-active
<minio-bucket-replication-serverside-twoway>` configurations, set the same
expiration rules on *all* of of the replication buckets to ensure consistent
application of object expiration.
.. admonition:: MinIO Trims Empty Object Prefixes on Source and Remote Bucket
:class: note, dropdown
If a delete operation removes the last object in a bucket prefix, MinIO
recursively removes each empty part of the prefix up to the bucket root.
MinIO only applies the recursive removal to prefixes created *implicitly* as
part of object write operations - that is, the prefix was not created using
an explicit directory creation command such as :mc:`mc mb`.
If a replication rule enables replication delete operations, the replication
process *also* applies the implicit prefix trimming behavior on the
destination MinIO cluster.
For example, consider a bucket ``photos`` with the following object prefixes:
- ``photos/2021/january/myphoto.jpg``
- ``photos/2021/february/myotherphoto.jpg``
- ``photos/NYE21/NewYears.jpg``
``photos/NYE21`` is the *only* prefix explicitly created using :mc:`mc mb`.
All other prefixes were *implicitly* created as part of writing the object
located at that prefix.
- A command removes ``myphoto.jpg``. MinIO automatically trims the empty
``/janaury`` prefix.
- A command then removes the ``myotherphoto.jpg``. MinIO automatically
trims the ``/february`` prefix *and* the now-empty ``/2021`` prefix.
- A command removes the ``NewYears.jpg`` object. MinIO leaves the
``/NYE21`` prefix remains in place since it was *explicitly* created.
.. _minio-replication-behavior-existing-objects:
Replication of Existing Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO by default does not enable existing object replication. Objects
created before replication was configured *or* while replication is
disabled are not synchronized to the target deployment.
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
:minio-git:`RELEASE.2021-06-07T21-40-51Z
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports enabling
replication of existing objects in a bucket.
Enabling existing object replication marks all objects or object prefixes that
satisfy the replication rules as eligible for synchronization to the source
cluster, *even if* those objects were created prior to configuring or enabling
replication. You can enable existing object replication while configuring
or modifying a replication rule:
- For new replication rules, include ``"existing-objects"`` to the list of
replication features specified to :mc-cmd:`mc replicate add --replicate`.
- For existing replication rules, add ``"existing-objects"`` to the list of
existing replication features using
:mc-cmd:`mc replicate edit --replicate`. You must specify *all* desired
replication features when editing the replication rule.
Enabling existing object replication does not increase the priority of objects
pending replication. MinIO uses the same core
:ref:`replication scanner and queue system <minio-replication-process>` for
detecting and synchronizing objects regardless of the enabled replication
feature. The time required to fully synchronize a bucket depends on a number of
factors, including but not limited to the current cluster replication load,
overall cluster load, and the size of the namespace (all objects in the bucket).
If versioning was not previously enabled when configuring bucket replication,
existing objects have a ``versionid = null``. These objects do replicate.
MinIO existing object replication
implements functionality similar to
`AWS: Replicating existing objects between S3 buckets
<https://aws.amazon.com/blogs/storage/replicating-existing-objects-between-s3-buckets/>`__
without the overhead of contacting technical support.
Synchronous vs Asynchronous Replication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports specifying either asynchronous (default) or synchronous
replication for a given remote target.
With the default asynchronous replication, MinIO completes the originating
``PUT`` operation *before* placing the object into a :ref:`replication queue
<minio-replication-process>`. The originating client may therefore see a
successful ``PUT`` operation *before* the object is replicated. While
this may result in stale or missing objects on the remote, it mitigates
the risk of slow write operations due to replication load.
With synchronous replication, MinIO attempts to replicate the object *prior* to
completing the originating ``PUT`` operation. MinIO returns a successful ``PUT``
operation whether or not the replication attempts succeeds. While this may
result in more reliable synchronization between the source and remote target,
it may also increase the time of each write operation due to replication load.
You must explicitly enable synchronous replication when configuring the remote
target target using the :mc-cmd:`mc admin bucket remote add` command with the
:mc-cmd:`~mc admin bucket remote add` flag.
Replication Internals
~~~~~~~~~~~~~~~~~~~~~
This section documents internal replication behavior and is not critical to
using or implementing replication. This documentation is provided strictly
for learning and educational purposes.
.. _minio-replication-process:
Replication Process
+++++++++++++++++++
MinIO uses a replication queuing system with multiple concurrent replication
workers operating on that queue. MinIO continuously works to replicate and
remove objects from the queue while scanning for new unreplicated objects to
add to the queue.
MinIO sets the ``X-Amz-Replication-Status`` metadata field according to the
replication state of the object:
.. list-table::
:header-rows: 1
:width: 100%
:widths: 30 70
* - Replication State
- Description
* - ``PENDING``
- The object has not yet been replicated. MinIO applies this state
if the object meets one of the configured replication rules on the
bucket. MinIO continuously scans for ``PENDING`` objects not yet in the
replication queue and adds them to the queue as space is available.
For multi-site replication, objects remain
in the ``PENDING`` state until replicated to *all* configured
remotes for that bucket or bucket prefix.
* - ``COMPLETED``
- The object has successfully replicated to the remote cluster.
* - ``FAILED``
- The object failed to replicate to the remote cluster.
MinIO continuously scans for ``FAILED`` objects not yet in the
replication queue and adds them to the queue as space is available.
* - ``REPLICA``
- The object is itself a replica from a remote source.
The replication process generally has one of the following flows:
- ``PENDING -> COMPLETED``
- ``PENDING -> FAILED -> COMPLETED``
.. toctree::
:hidden:
:titlesonly:
/administration/bucket-replication/enable-server-side-one-way-bucket-replication
/administration/bucket-replication/enable-server-side-two-way-bucket-replication
/administration/bucket-replication/enable-server-side-multi-site-bucket-replication
/administration/bucket-replication/server-side-replication-resynchronize-remote

View File

@@ -0,0 +1,230 @@
.. _minio-bucket-replication-serverside-multi:
================================================
Enable Multi-Site Server-Side Bucket Replication
================================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
The procedure on this page configures automatic server-side bucket replication between multiple MinIO deployments. Multi-Site Active-Active replication builds on the :ref:`minio-bucket-replication-serverside-twoway` procedure with additional considerations required to ensure predictable replication behavior across all sites.
.. image:: /images/replication/active-active-multi-replication.svg
:width: 600px
:alt: Active-Active Replication synchronizes data between multiple remote deployments.
:align: center
- To configure replication between arbitrary S3-compatible services, use :mc-cmd:`mc mirror`.
- To configure one-way "active-active" replication between two MinIO deployments, see :ref:`minio-bucket-replication-serverside-twoway`.
- To configure one-way "active-passive" replication between MinIO deployments, see :ref:`minio-bucket-replication-serverside-oneway`.
Multi-Site Active-Active replication configurations can span multiple racks, datacenters, or geographic locations. Complexity of configuring and maintaining multi-site configurations generally increase with the number of sites and size of each site. Enterprises looking to implement multi-site replication should consider leveraging `MinIO SUBNET <https://min.io/pricing?ref=docs>`__ support to access the expertise, planning, and engineering resources required for addressing that use case.
.. seealso::
- Use the :mc-cmd:`mc replicate edit` command to modify an existing replication rule.
- Use the :mc-cmd:`mc replicate edit` command with the :mc-cmd:`--state "disable" <mc replicate edit --state>` flag to disable an existing replication rule.
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule.
.. _minio-bucket-replication-serverside-multi-requirements:
Requirements
------------
Install and Configure ``mc`` with Access to Both Clusters.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses :mc:`mc` for performing operations on both the source and destination MinIO cluster. Install :mc:`mc` on a machine with network access to both source and destination deployments. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for instructions on downloading and installing ``mc``.
Use the :mc:`mc alias` command to create an alias for both MinIO deployments. Alias creation requires specifying an access key for a user on the cluster. This user **must** have permission to create and manage users and policies on the cluster. Specifically, ensure the user has *at minimum*:
- :policy-action:`admin:CreateUser`
- :policy-action:`admin:ListUsers`
- :policy-action:`admin:GetUser`
- :policy-action:`admin:CreatePolicy`
- :policy-action:`admin:GetPolicy`
- :policy-action:`admin:AttachUserOrGroupPolicy`
.. _minio-bucket-replication-serverside-multi-permissions:
Required Permissions
~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-required-permissions
:end-before: end-replication-required-permissions
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations
--------------
Use Consistent Replication Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports customizing the replication configuration to enable or disable the following replication behaviors:
- Replication of delete operations
- Replication of delete markers
- Replication of existing objects
- Replication of metadata-only changes
When configuring replication rules for a bucket, ensure that all MinIO deployments participating in multi-site replication use the *same* replication behaviors to ensure consistent and predictable synchronization of objects.
Replication of Existing Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports automatically replicating existing objects in a bucket.
MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag. This procedure includes the required flags for enabling replication of existing objects.
Replication of Delete Operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating delete operations onto the target bucket. Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects:
- For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
- For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`. This procedure includes the required flags for enabling replication of delete operations and delete markers.
MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`. Configure matching expiration rules for the bucket on all replication sites to ensure consistent application of object expiration.
Procedure
---------
This procedure requires repeating steps for each MinIO deployment participating in the multi-site replication configuration. Depending on the number of deployments, this procedure may require significant time and care in implementation. MinIO recommends reading through the procedure *before* attempting to implement the documented steps.
This procedure uses the placeholder ``ALIAS`` to reference the :ref:`alias <alias>` each MinIO deployment being configured for replication. Replace these values with the appropriate alias for each MinIO deployment.
This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-multi-permissions>`.
1) Create the Replication Remote Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin bucket remote add` command to create a replication target for the each deployment. MinIO supports *one* remote target per destination bucket. You cannot create multiple remote targets for the same destination bucket.
.. code-block:: shell
:class: copyable
mc admin bucket remote add ALIAS/BUCKET \
https://ReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/BUCKET \
--service "replication"
- Replace ``BUCKET`` with the name of the bucket on the ``ALIAS`` deployment to use as the replication source. Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication.
- Replace ``HOSTNAME`` with the URL of the remote MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket on the remote deployment to use as the replication destination.
The command returns an ARN similar to the following:
.. code-block:: shell
Role ARN = 'arn:minio:replication::<UUID>:BUCKET'
Copy the ARN string for use in the next step, noting the MinIO deployment on which it was created.
Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration. For example, a multi-site replication configuration consisting of MinIO deployments ``Alpha``, ``Baker``, and ``Charlie`` would require repeating this step on each deployment for each remote. Specifically:
- The ``Alpha`` deployment would perform this step once for
``Baker`` and once for ``Charlie``.
- The ``Baker`` deployment would perform this step once for ``Alpha`` and
once for ``Charlie``.
- The ``Charlie`` deployment would perform this step once for ``Baker`` and
once for ``Alpha``.
2) Create a New Bucket Replication Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc replicate add` command to add the new server-side
replication rule to the each MinIO deployment.
.. code-block:: shell
:class: copyable
mc replicate add ALIAS/BUCKET \
--remote-bucket 'arn:minio:replication::<UUID>:BUCKET' \
--replicate "delete,delete-marker,existing-objects"
- Replace ``BUCKET`` with the name of the bucket on the ``ALIAS`` deployment to use as the replication source. Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication.
- Replace the ``--remote-bucket`` value with the ARN returned in the previous step. Ensure you specify the ARN created on the ``ALIAS`` deployment. You can use :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the deployment.
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables the following replication features:
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
See :mc-cmd:`mc replicate add --replicate` for more complete documentation. Omit these fields to disable replication of delete operations or replication of existing objects respectively.
Specify any other supported optional arguments for :mc-cmd:`mc replicate add`.
Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration. For example, a multi-site replication configuration consisting of MinIO deployments ``Alpha``, ``Baker``, and ``Charlie`` would require repeating this step on each deployment for each remote. Specifically:
- The ``Alpha`` deployment would perform this step once for
``Baker`` and once for ``Charlie``.
- The ``Baker`` deployment would perform this step once for ``Alpha`` and
once for ``Charlie``.
- The ``Charlie`` deployment would perform this step once for ``Baker`` and
once for ``Alpha``.
3) Validate the Replication Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use :mc-cmd:`mc cp` to copy a new object the bucket on any of the deployments:
.. code-block:: shell
:class: copyable
mc cp ~/foo.txt ALIAS/BUCKET
Use :mc-cmd:`mc ls` to verify the object exists on each remote deployment:
.. code-block:: shell
:class: copyable
mc ls REMOTE/BUCKET
Repeat this test on each of the deployments by copying a new unique file and checking the other deployments for that file.
You can also use :mc-cmd:`mc stat` to check the file to check the current :ref:`replication stage <minio-replication-process>` of the object.

View File

@@ -0,0 +1,186 @@
.. _minio-bucket-replication-serverside-oneway:
=============================================
Enable One-Way Server-Side Bucket Replication
=============================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
The procedure on this page creates a new bucket replication rule for one-way synchronization of objects between MinIO buckets.
.. image:: /images/replication/active-passive-oneway-replication.svg
:width: 450px
:alt: Active-Passive Replication synchronizes data from a source MinIO cluster to a remote MinIO cluster.
:align: center
- To configure replication between arbitrary S3-compatible services, use :mc-cmd:`mc mirror`.
- To configure two-way "active-active" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-twoway`.
- To configure multi-site "active-active" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-multi`
.. seealso::
- Use the :mc-cmd:`mc replicate edit` command to modify an existing replication rule.
- Use the :mc-cmd:`mc replicate edit` command with the :mc-cmd:`--state "disable" <mc replicate edit --state>` flag to disable an existing replication rule.
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule.
.. _minio-bucket-replication-serverside-oneway-requirements:
Requirements
------------
.. _minio-bucket-replication-serverside-oneway-permissions:
Required Permissions
~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-required-permissions
:end-before: end-replication-required-permissions
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations
--------------
Replication of Existing Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports automatically replicating existing objects in a bucket.
MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag. This procedure includes the required flags for enabling replication of existing objects.
Replication of Delete Operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating S3 ``DELETE`` operations onto the target bucket. Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects:
- For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
- For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`. This procedure includes the required flags for enabling replication of delete operations and delete markers.
MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`.
See :ref:`minio-replication-behavior-delete` for more complete documentation.
Multi-Site Replication
~~~~~~~~~~~~~~~~~~~~~~
MinIO supports configuring multiple remote targets per bucket or bucket prefix. For example, you can configure a bucket to replicate data to two or more remote MinIO deployments, where one deployment is a 1:1 copy (replication of all operations including deletions) and another is a full historical record (replication of only non-destructive write operations).
This procedure documents one-way replication to a single remote MinIO deployment. You can repeat this tutorial for multiple remote targets for a single bucket.
Procedure
---------
This procedure uses the :ref:`aliases <alias>` ``SOURCE`` and ``REMOTE`` to reference each MinIO deployment being configured for replication. Replace these values with the appropriate alias for your target MinIO deployments.
This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-oneway-permissions>`.
1) Create the Replication Remote Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin bucket remote add` command to create a replication target for the destination cluster. MinIO supports *one* remote target per destination bucket. You cannot create multiple remote targets for the same destination bucket.
.. code-block:: shell
:class: copyable
mc admin bucket remote add SOURCE/BUCKET \
https://ReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/BUCKET \
--service "replication"
[--sync]
- Replace ``BUCKET`` with the name of the bucket on the ``SOURCE`` deployment to use as the replication source. Replace ``SOURCE`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication.
- Replace ``HOSTNAME`` with the URL of the ``REMOTE`` cluster.
- Replace ``BUCKET`` with the name of the bucket on the ``REMOTE`` deployment to use as the replication destination.
- Include the :mc-cmd:`~mc admin bucket remote add --sync` option to enable synchronous replication. Omit the option to use the default of asynchronous replication. See the reference documentation for :mc-cmd:`mc admin bucket remote add` for more information on synchronous vs asynchronous replication before using this parameter.
The command returns an ARN similar to the following:
.. code-block:: shell
Role ARN = 'arn:minio:replication::<UUID>:BUCKET'
Copy the ARN string for use in the next step.
2) Create a New Bucket Replication Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc replicate add` command to add the new server-side
replication rule to the source MinIO cluster.
.. code-block:: shell
:class: copyable
mc replicate add SOURCE/BUCKET \
--remote-bucket 'arn:minio:replication::<UUID>:BUCKET' \
--replicate "delete,delete-marker,existing-objects"
- Replace ``BUCKET`` with the name of the bucket on the ``SOURCE`` deployment to use as the replication source. Replace ``SOURCE`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication. The name *must* match the bucket specified when creating the remote target in the previous step.
- Replace the ``--remote-bucket`` value with the ARN returned in the previous step. Ensure you specify the ARN created on the ``SOURCE`` deployment. You can use :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the deployment.
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables the following replication features:
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
See :mc-cmd:`mc replicate add --replicate` for more complete documentation. Omit these fields to disable replication of delete operations or replication of existing objects respectively.
Specify any other supported optional arguments for :mc-cmd:`mc replicate add`.
3) Validate the Replication Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use :mc-cmd:`mc cp` to copy a new object to the source bucket.
.. code-block:: shell
:class: copyable
mc cp ~/foo.txt SOURCE/BUCKET
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
.. code-block:: shell
:class: copyable
mc ls TARGET/BUCKET

View File

@@ -0,0 +1,235 @@
.. _minio-bucket-replication-serverside-twoway:
=============================================
Enable Two-Way Server-Side Bucket Replication
=============================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
The procedure on this page creates a new bucket replication rule for two-way "active-active" synchronization of objects between MinIO buckets.
.. image:: /images/replication/active-active-twoway-replication.svg
:width: 600px
:alt: Active-Active Replication synchronizes data between two remote clusters.
:align: center
- To configure replication between arbitrary S3-compatible services, use :mc-cmd:`mc mirror`.
- To configure one-way "active-passive" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-oneway`.
- To configure multi-site "active-active" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-multi`.
This tutorial covers configuring Active-Active replication between two MinIO clusters. For a tutorial on multi-site replication between three or more MinIO clusters, see :ref:`minio-bucket-replication-serverside-multi`.
.. seealso::
- Use the :mc-cmd:`mc replicate edit` command to modify an existing
replication rule.
- Use the :mc-cmd:`mc replicate edit` command with the :mc-cmd:`--state "disable" <mc replicate edit --state>` flag to disable an existing replication rule.
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule.
.. _minio-bucket-replication-serverside-twoway-requirements:
Requirements
------------
Install and Configure ``mc`` with Access to Both Clusters.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses :mc:`mc` for performing operations on both the source and destination MinIO cluster. Install :mc:`mc` on a machine with network access to both source and destination clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for instructions on downloading and installing ``mc``.
Use the :mc:`mc alias` command to create an alias for both MinIO clusters. Alias creation requires specifying an access key for a user on the cluster. This user **must** have permission to create and manage users and policies on the cluster. Specifically, ensure the user has *at minimum*:
- :policy-action:`admin:CreateUser`
- :policy-action:`admin:ListUsers`
- :policy-action:`admin:GetUser`
- :policy-action:`admin:CreatePolicy`
- :policy-action:`admin:GetPolicy`
- :policy-action:`admin:AttachUserOrGroupPolicy`
.. _minio-bucket-replication-serverside-twoway-permissions:
Required Permissions
~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-required-permissions
:end-before: end-replication-required-permissions
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations
--------------
Use Consistent Replication Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports customizing the replication configuration to enable or disable
the following replication behaviors:
- Replication of delete operations
- Replication of delete markers
- Replication of existing objects
- Replication of metadata-only changes
When configuring replication rules for a bucket, ensure that both MinIO deployments participating in active-active replication use the *same* replication behaviors to ensure consistent and predictable synchronization of objects.
Replication of Existing Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports automatically replicating existing objects in a bucket.
MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag. This procedure includes the required flags for enabling replication of existing objects.
Replication of Delete Operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating delete operations onto the target bucket. Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects:
- For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
- For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`. This procedure includes the required flags for enabling replication of delete operations and delete markers.
MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`. Configure matching expiration rules on both the source and destination bucket to ensure consistent application of object expiration.
See :ref:`minio-replication-behavior-delete` for more complete documentation.
Multi-Site Replication
~~~~~~~~~~~~~~~~~~~~~~
MinIO supports configuring multiple remote targets per bucket or bucket prefix. This enables configuring multi-site active-active replication between MinIO deployments.
This procedure covers active-active replication between *two* MinIO sites. You can repeat this procedure for each "pair" of MinIO deployments in the replication mesh. For a dedicated tutorial, see :ref:`minio-bucket-replication-serverside-multi`.
Procedure
---------
This procedure uses the :ref:`aliases <alias>` ``ALPHA`` and ``BAKER`` to reference each MinIO deployment being configured for replication. Replace these values with the appropriate alias for your target MinIO deployments.
This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-twoway-permissions>`.
1) Create the Replication Remote Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin bucket remote add` command to create a replication target for the each deployment. MinIO supports *one* remote target per destination bucket. You cannot create multiple remote targets for the same destination bucket.
.. code-block:: shell
:class: copyable
mc admin bucket remote add ALPHA/BUCKET \
https://ReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/BUCKET \
--service "replication"
- Replace ``BUCKET`` with the name of the bucket on the ``ALPHA`` deployment to use as the replication source. Replace ``ALPHA`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication.
- Replace ``HOSTNAME`` with the URL of the ``BAKER`` deployment.
- Replace ``BUCKET`` with the name of the bucket on the ``REMOTE`` deployment to use as the replication destination.
The command returns an ARN similar to the following:
.. code-block:: shell
Role ARN = 'arn:minio:replication::<UUID>:BUCKET'
Copy the ARN string for use in the next step, noting the MinIO deployment on which it was created.
Repeat this step on the second MinIO deployment, replacing the ``ALPHA`` alias with the ``BAKER`` alias and the ``HOSTNAME`` with the URL of the ``ALPHA`` deployment.
You should have two ARNs at the conclusion of this step - one created on ``ALPHA/BUCKET`` pointing at ``BAKER/BUCKET``, and one created on ``BAKER/BUCKET`` pointing at ``ALPHA/BUCKET``. Use the :mc-cmd:`mc admin bucket remote ls` command to verify the created replication remote targets before proceeding.
2) Create a New Bucket Replication Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc replicate add` command to add the new server-side
replication rule to the each MinIO deployment.
.. code-block:: shell
:class: copyable
mc replicate add ALPHA/BUCKET \
--remote-bucket 'arn:minio:replication::<UUID>:BUCKET' \
--replicate "delete,delete-marker,existing-objects"
- Replace ``BUCKET`` with the name of the bucket on the ``ALPHA`` deployment to use as the replication source. Replace ``ALPHA`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication. The name *must* match the bucket specified when creating the remote target in the previous step.
- Replace the ``--remote-bucket`` value with the ARN returned in the previous step. Ensure you specify the ARN created on the ``ALPHA`` deployment. You can use :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the deployment.
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables the following replication features:
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
See :mc-cmd:`mc replicate add --replicate` for more complete documentation. Omit these fields to disable replication of delete operations or replication of existing objects respectively.
Specify any other supported optional arguments for :mc-cmd:`mc replicate add`.
Repeat this step on the second MinIO deployment, replacing the ``ALPHA`` alias with the ``BAKER`` alias and the ``HOSTNAME`` with the URL of the ``ALPHA`` deployment.
You should have two replication rules configured at the conclusion of this step - one created on ``ALPHA/BUCKET`` and one created on ``BAKER/BUCKET``. Use the :mc-cmd:`mc replicate ls` command to verify the created replication rules.
3) Validate the Replication Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use :mc-cmd:`mc cp` to copy a new object to the ``ALPHA/BUCKET`` bucket.
.. code-block:: shell
:class: copyable
mc cp ~/foo.txt ALPHA/BUCKET
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
.. code-block:: shell
:class: copyable
mc ls BAKER/BUCKET
Repeat this test by copying a new object to the ``Baker`` source bucket.
.. code-block:: shell
:class: copyable
mc cp ~/otherfoo.txt BAKER/BUCKET
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
.. code-block:: shell
:class: copyable
mc ls ALPHA/BUCKET

View File

@@ -0,0 +1,182 @@
.. _minio-bucket-replication-resynchronize:
========================================
Resynchronize Bucket from Remote Replica
========================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
The procedure on this page resynchronizes the contents of a MinIO bucket using a healthy replication remote. Resynchronization supports recovery after partial or total loss of data on a MinIO deployment in a replica configuration.
For example, consider a MinIO active-active replication configuration similar to the following:
.. image:: /images/replication/active-active-twoway-replication.svg
:width: 600px
:alt: Active-Active Replication synchronizes data between two remote deployments.
:align: center
Resynchronization allows using the healthy data on one of the participating MinIO deployments as the source for rebuilding the other deployment.
Resynchronization is a per-bucket process. You must repeat resynchronization for each bucket on the remote which suffered partial or total data loss.
.. admonition:: Professional Support during BC/DR Operations
:class: important
`MinIO SUBNET <https://min.io/pricing?jmp=docs>`__ users can `log in <https://subnet.min.io/>`__ and create a new issue related to resynchronization. Coordination with MinIO Engineering via SUBNET can ensure successful resynchronization and restoration of normal operations, including performance testing and health diagnostics.
Community users can seek support on the `MinIO Community Slack <https://minio.slack.com>`__. Community Support is best-effort only and has no SLAs around responsiveness.
.. _minio-bucket-replication-serverside-resynchronize-requirements:
Requirements
------------
MinIO Deployments Must Be Online
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Resynchronization requires both the source and target deployments be online and able to accept read and write operations. The source *must* have complete network connectivity to the remote.
The remote deployment may be "unhealthy" in that it has suffered partial or total data loss. Resynchronization addresses the data loss as long as both source and destination maintain connectivity.
Resynchronization Requires Existing Replication Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Resynchronization requires the healthy source deployment have an existing replication configuration for the unhealthy target bucket. Additionally, resynchronization only applies to those replication rules created with the :ref:`existing object replication <minio-replication-behavior-existing-objects>` option.
- Use :mc-cmd:`mc admin bucket remote ls` to review the configured remote targets on the healthy source bucket.
- Use :mc-cmd:`mc replicate ls` to review the configured replication rules on the healthy source bucket.
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations
--------------
Resynchronization Requires Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Resynchronization is a background processes that continually checks objects in the source MinIO bucket and copies them to the remote as-needed. The time required for replication to complete may vary depending on the number and size of objects, the throughput to the remote MinIO deployment, and the load on the source MinIO deployment. Total time for completion is generally not predictable due to these variables.
MinIO recommends configuring load balancers or proxies to direct traffic only to the healthy cluster until synchronization completes. The following commands can provide insight into the resynchronization status:
- :mc-cmd:`mc replicate resync status` on the source to track the resynchronization progress.
- :mc-cmd:`mc replicate status` on the source and remote to track normal replication data.
- Run ``mc ls -r --versions ALIAS/BUCKET | wc -l`` against both source and remote to validate the total number of objects and object versions on each.
Resynchronize Objects after Data Loss
-------------------------------------
This procedure uses an existing :ref:`MinIO replication configuration <minio-bucket-replication-serverside>` to restore missing data to one of the MinIO deployments participating in that configuration. Specifically, a healthy MinIO deployment (the ``SOURCE``) synchronizes it's existing data to the unhealthy MinIO deployment (the ``TARGET``).
This procedure assumes an existing :ref:`alias <alias>` for the ``SOURCE`` that has the :ref:`necessary permissions <minio-bucket-replication-serverside-twoway-permissions>` for configuring replication.
You can repeat this procedure for each bucket that requires resynchronization. You can have no more than one replication job running per bucket.
1) List the Configured Replication Targets on the Healthy Source
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run the :mc-cmd:`mc admin bucket remote ls` command to list the configured remote targets on the healthy ``SOURCE`` deployment for the ``BUCKET`` that requires resynchronization.
.. code-block:: shell
:class: copyable
mc admin bucket remote ls SOURCE/BUCKET
- Replace ``SOURCE`` with the :ref:`alias <alias>` of the source MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket to use as the source for resynchronization.
The output resembles the following:
.. code-block:: shell
Remote URL Source->Target ARN SYNC PROXY
https://minio-2.example.net:9000 data ->data arn:minio:replication::UUID:BUCKET proxy
Identify the ARN associated to the ``BUCKET`` on the unhealthy ``TARGET`` MinIO deployment for use in the next step.
2) Start the Resynchronization Procedure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run the :mc-cmd:`mc replicate resync start` command to begin the resynchronization process:
.. code-block:: shell
:class: copyable
mc replicate resync start --remote-bucket "arn:minio:replication::UUID:BUCKET" SOURCE/BUCKET
- Replace the ``--remote-bucket`` value with the ARN of the unhealthy ``BUCKET`` on the ``TARGET`` MinIO deployment.
- Replaced ``SOURCE`` with the :ref:`alias <alias>` of the source MinIO deployment.
- Replace the ``BUCKET`` with the name of the bucket on the healthy ``SOURCE`` MinIO
deployment.
The command returns a resynchronization job ID indicating that the process has begun.
3) Monitor Resynchronization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc replicate resync status` command on the source deployment to track the received replication data:
.. code-block:: shell
:class: copyable
mc replicate resync status ALIAS/BUCKET
The output resembles the following:
.. code-block:: shell
mc replicate resync status /data
Resync status summary:
● arn:minio:replication::6593d572-4dc3-4bb9-8d90-7f79cc612f01:data
Status: Ongoing
Replication Status | Size (Bytes) | Count
Replicated | 2.3 GiB | 18
Failed | 0 B | 0
The :guilabel:`Status` updates to ``Completed`` once the resynchronization
process completes.
4) Next Steps
~~~~~~~~~~~~~
- If the ``TARGET`` bucket damage extends to replication rules, you must recreate those rules to match the previous replication configuration. See :ref:`minio-bucket-replication-serverside-twoway` for additional guidance.
- Perform basic validation that all buckets in the replication configuration show similar results for commands such as :mc-cmd:`mc ls` and :mc-cmd:`mc stat`.
- After restoring any replication rules and verifying replication between sites, you can configure the reverse proxy, load balancer, or other network control plane managing connections to resume sending traffic to the resynchronized deployment.

View File

@@ -0,0 +1,143 @@
.. _minio-authentication-and-identity-management:
==============================
Identity and Access Management
==============================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
MinIO requires the client perform both authentication and authorization for each new operation.
Authentication
The process of verifying the identity of a connecting client.
MinIO requires clients authenticate using :s3-api:`AWS Signature Version 4 protocol <sig-v4-authenticating-requests.html>` with support for the deprecated Signature Version 2 protocol.
Specifically, clients must present a valid access key and secret key to access any S3 or MinIO administrative API, such as ``PUT``, ``GET``, and ``DELETE`` operations.
Authorization
The process of restricting the actions and resources the authenticated client can perform on the deployment.
MinIO uses Policy-Based Access Control (PBAC), where each policy describes one or more rules that outline the permissions of a user or group of users.
MinIO supports S3-specific:ref:`actions <minio-policy-actions>` and :ref:`conditions <minio-policy-conditions>` when creating policies.
By default, MinIO *denies* access to actions or resources not explicitly referenced in a user's assigned or inherited policies.
Identity Management
-------------------
MinIO supports both internal and external identity management:
.. list-table::
:header-rows: 1
:widths: 30 70
:width: 100%
* - IDentity Provider (IDP)
- Description
* - :ref:`MinIO Internal IDP <minio-internal-idp>`
- Provides built-in identity management functionality.
* - :ref:`OpenID <minio-external-identity-management-openid>`
- Supports managing identities through an OpenID Connect (OIDC) compatible
service.
* - :ref:`Active Directory / LDAP
<minio-external-identity-management-ad-ldap>`
- Supports managing identities through an Active Directory or LDAP service.
Once authenticated, MinIO either allows or rejects the client request depending
on whether or not the authenticated identity is *authorized* to perform the
operation on the specified resource.
Enabling external identity management disables the MinIO internal IDP, with the exception of the creating :ref:`service accounts <minio-idp-service-account>`.
.. _minio-access-management:
Access Management
-----------------
MinIO uses Policy-Based Access Control (PBAC) to define the authorized actions
and resources to which an authenticated user has access. Each policy describes
one or more :ref:`actions <minio-policy-actions>` and :ref:`conditions
<minio-policy-conditions>` that outline the permissions of a
:ref:`user <minio-users>` or :ref:`group <minio-groups>` of
users.
MinIO manages the creation and storage of policies. The process for
assigning a policy to a user or group depends on the configured
:ref:`IDentity Provider (IDP) <minio-authentication-and-identity-management>`.
MinIO deployments using the :ref:`MinIO Internal IDP <minio-internal-idp>`
require explicitly associating a user to a policy or policies using the
:mc-cmd:`mc admin policy set` command. A user can also inherit the policies
attached to the :ref:`groups <minio-groups>` in which they have membership.
By default, MinIO *denies* access to actions or resources not explicitly allowed
by an attached or inherited policy. A user with no explicitly assigned or
inherited policies cannot perform any S3 or MinIO administrative API operations.
For MinIO deployments using an External IDP, policy assignment depends on the
choice of IDP:
.. list-table::
:stub-columns: 1
:widths: 30 70
:width: 100%
* - :ref:`OpenID Connect (OIDC) <minio-external-identity-management-openid>`
- MinIO checks for a JSON Web Token (JWT) claim (``policy`` by default)
containing the name of the policy or policies to attach to the
authenticated user. If the policies do not exist, the user cannot
perform any action on the MinIO deployment.
MinIO does not support assigning OIDC user identities to
:ref:`groups <minio-groups>`. The IDP administrator must instead
assign all necessary policies to the user's policy claim.
See :ref:`Access Control for Externally Managed Identities
<minio-external-identity-management-openid-access-control>` for
more information.
* - :ref:`Active Directory / LDAP (AD/LDAP)
<minio-external-identity-management-ad-ldap>`
- MinIO checks for a policy whose name matches the Distinguished Name (DN)
of the authenticated AD/LDAP user.
MinIO also supports querying for the authenticated AD/LDAP user's
group memberships. MinIO assigns any policy whose name matches the
DN for each returned group.
If no policies match either the user DN *or* any of the user's group DNs,
the user cannot perform any action on the MinIO deployment.
See :ref:`Access Control for Externally Managed Identities
<minio-external-identity-management-ad-ldap-access-control>` for more
information.
MinIO PBAC is built for compatibility with AWS IAM policy syntax, structure, and
behavior. The MinIO documentation makes a best-effort to cover IAM-specific
behavior and functionality. Consider deferring to the :iam-docs:`IAM
documentation <>` for more complete documentation on IAM, IAM policies, or IAM
JSON syntax.
.. admonition:: ``Deny`` overrides ``Allow``
:class: note
MinIO follows AWS IAM policy evaluation rules where a ``Deny`` rule overrides
``Allow`` rule on the same action/resource. For example, if a user has an
explicitly assigned policy with an ``Allow`` rule for an action/resource
while one of its groups has an assigned policy with a ``Deny`` rule for that
action/resource, MinIO would apply only the ``Deny`` rule.
For more information on IAM policy evaluation logic, see the IAM documentation on :iam-docs:`Determining Whether a Request is Allowed or Denied Within an Account <reference_policies_evaluation-logic.html#policy-eval-denyallow>`.
.. toctree::
:titlesonly:
:hidden:
/administration/identity-access-management/minio-identity-management
/administration/identity-access-management/oidc-access-management
/administration/identity-access-management/ad-ldap-access-management
/administration/identity-access-management/policy-based-access-control

View File

@@ -0,0 +1,103 @@
.. _minio-external-identity-management-ad-ldap:
=========================================
Active Directory / LDAP Access Management
=========================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
MinIO supports using an Active Directory or LDAP (AD/LDAP) service for external
management of user identities.
For identities managed by the external AD/LDAP provider, MinIO uses the user's Distinguished Name and attempts to map it against an existing :ref:`policy <minio-policy>`.
If the AD/LDAP configuration includes the necessary settings to query the user's AD/LDAP group membership, MinIO *also* uses those group Distinguished Names and attempts to map each against an existing :ref:`policy <minio-policy>`.
MinIO by default denies access to all actions or resources not explicitly allowed by a user's assigned or inherited :ref:`policies <minio-policy>`.
Users managed by an AD/LDAP provider must specify the necessary policies as part of the user profile data.
If no policies match either the user DN or group DNs, MinIO blocks all access to actions and resources on the deployment.
The specific AD/LDAP queries MinIO issues to authenticate the user and retrieve it's group membership are configured as part of :ref:`deploying the cluster with Active Directory / LDAP identity management <minio-external-iam-ad-ldap>`.
This page covers creation of MinIO policies to match the possible returned Distinguished Names.
Authentication and Authorization Flow
-------------------------------------
The login flow for an application using Active Directory / LDAP
credentials is as follows:
1. Specify the AD/LDAP credentials to the MinIO Security Token Service (STS)
:ref:`minio-sts-assumerolewithldapidentity` API endpoint.
2. MinIO verifies the provided credentials against the AD/LDAP server.
3. MinIO checks for any :ref:`policy <minio-policy>` whose name matches the
user Distinguished Name (DN) and assigns that policy to the authenticated
user.
If configured to perform group queries, MinIO also queries for a list of
AD/LDAP groups in which the user has membership. MinIO checks for any policy
whose name matches a returned group DN and assigns that
policy to the authenticated user.
4. MinIO returns temporary credentials in the STS API response in the form of an
access key, secret key, and session token. The credentials have permissions
matching those policies whose name matches either the authenticated user DN
*or* a group DN.
MinIO provides an example Go application
:minio-git:`ldap.go <minio/blob/master/docs/sts/ldap.go>` that handles the
full login flow.
AD/LDAP users can alternatively create :ref:`service accounts <minio-idp-service-account>` associated to their AD/LDAP user Distinguished Name. Service accounts are long-lived credentials which inherit their privileges from the parent user. The parent user can further restrict those privileges while creating the service account. Use either of the following methods to create a new service account
- Log into the :ref:`MinIO Console <minio-console>` using the AD/LDAP-managed user credentials. From the :guilabel:`Identity` section of the left navigation, select :guilabel:`Service Accounts` followed by the :guilabel:`Create service account +` button.
- Use the :mc-cmd:`mc admin user svcacct add` command to create the service account. Specify the user Distinguished Name as the username to which to associate the service account.
Mapping Policies to User DN
---------------------------
Consider the following policy assignments:
.. code-block:: shell
mc admin policy set --consoleAdmin user='cn=sisko,cn=users,dc=example,dc=com'
mc admin policy set --readwrite,diagnostics user='cn=dax,cn=users,dc=example,dc=com'
- MinIO would assign an authenticated user with DN matching
``cn=sisko,cn=users,dc=example,dc=com`` the :userpolicy:`consoleAdmin`
policy, granting complete access to the MinIO server.
- MinIO would assign an authenticated user with DN matching
``cn=dax,cn=users,dc=example,dc=com`` both the :userpolicy:`readwrite` and
:userpolicy:`diagnostics` policies, granting general read/write access to the
MinIO server *and* access to diagnostic administrative operations.
- MinIO would assign no policies to an authenticated user with DN matching
``cn=quark,cn=users,dc=example,dc=com`` and deny all access to API operations.
Mapping Policies to Group DN
----------------------------
Consider the following policy assignments:
.. code-block:: shell
mc admin policy set --consoleAdmin group='cn=ops,cn=groups,dc=example,dc=com'
mc admin policy set --diagnostics group='cn=engineering,cn=groups,dc=example,dc=com'
- MinIO would assign any authenticating user with membership in the
``cn=ops,cn=groups,dc=example,dc=com`` AD/LDAP group the
:userpolicy:`consoleAdmin` policy, granting complete access to the MinIO
server.
- MinIO would assign any authenticating user with membership in the
``cn=engineering,cn=groups,dc=example,dc=com`` AD/LDAP group the
:userpolicy:`diagnostics` policy, granting access to diagnostic administrative
operations.

View File

@@ -0,0 +1,60 @@
.. _minio-groups:
================
Group Management
================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
Overview
--------
A *group* is a collection of :ref:`users <minio-users>`. Each group
can have one or more assigned :ref:`policies <minio-policy>`
that explicitly list the actions and resources to which group members are
allowed or denied access.
For example, consider the following groups. Each group is assigned a
:ref:`built-in policy <minio-policy-built-in>` or supported
:ref:`policy action <minio-policy-actions>`. Each group also has one or
more assigned users. Each user's total set of permissions consists of their
explicitly assigned permission *and* the inherited permissions from each of
their assigned groups. MinIO by default *denies* access to any resource or
operation not explicitly allowed by a user's assigned or inherited policies.
.. list-table::
:header-rows: 1
:widths: 20 40 40
:width: 100%
* - Group
- Policy
- Members
* - ``Operations``
- | :userpolicy:`readwrite` on ``finance`` bucket
| :userpolicy:`readonly` on ``audit`` bucket
- ``john.doe``, ``jane.doe``
* - ``Auditing``
- | :userpolicy:`readonly` on ``audit`` bucket
- ``jen.doe``, ``joe.doe``
* - ``Admin``
- :policy-action:`admin:*`
- ``greg.doe``, ``jen.doe``
Groups provide a simplified method for managing shared permissions among
users with common access patterns and workloads. Client's *cannot* authenticate
to a MinIO deployment using a group as an identity.
The :mc-cmd:`mc admin group` command supports the creation and management of
groups on the MinIO deployment. See the command reference for examples of
usage.

View File

@@ -0,0 +1,56 @@
.. _minio-internal-idp:
=========================
MinIO Identity Management
=========================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
MinIO includes a built-in IDentity Provider (IDP) that provides core identity
management functionality. The MinIO IDP supports creating an arbitrary number of
long-lived users on the deployment for supporting client authentication.
Each user consists of a unique access key (username) and corresponding secret
key (password). Clients must authenticate their identity by specifying both
a valid access key (username) and the corresponding secret key (password) of
an existing MinIO user.
Administrators use the :mc-cmd:`mc admin user` command to create and manage
MinIO users. The :minio-git:`MinIO Console <console>` provides a graphical
interface for creating users.
MinIO also supports creating :ref:`service accounts
<minio-idp-service-account>`. Service accounts are child identities of an
authenticated parent user and inherit their permissions from the parent.
MinIO by default denies access to all actions or resources not explicitly
allowed by a user's assigned or inherited :ref:`policies <minio-policy>`. You
must either explicitly assign a :ref:`policy <minio-policy>` describing the
user's authorized actions and resources *or* assign the user to :ref:`groups
<minio-groups>` which have associated policies. See
:ref:`minio-access-management` for more information.
.. admonition:: External Identity Management
:class: dropdown, note
MinIO supports external management of identities using either an
OpenID Connect (OIDC) or Active Directory/LDAP IDentity Provider (IDP).
For more information, see:
- :ref:`minio-external-identity-management-openid`
- :ref:`minio-external-identity-management-ad-ldap`
Enabling external identity management disables the MinIO internal IDP, with
the exception of creating :ref:`service accounts
<minio-idp-service-account>`.
.. toctree::
:titlesonly:
:hidden:
/administration/identity-access-management/minio-user-management.rst
/administration/identity-access-management/minio-group-management.rst

View File

@@ -0,0 +1,173 @@
.. _minio-users:
===============
User Management
===============
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
Overview
--------
A MinIO user consists of a unique access key (username) and corresponding secret
key (password). Clients must authenticate their identity by specifying both
a valid access key (username) and the corresponding secret key (password) of
an existing MinIO user.
Each user can have one or more assigned :ref:`policies <minio-policy>` that
explicitly list the actions and resources to which that user has access.
Users can also inherit policies from the :ref:`groups <minio-groups>` in which
they have membership.
MinIO by default denies access to all actions or resources not explicitly
allowed by a user's assigned or inherited :ref:`policies <minio-policy>`. You
must either explicitly assign a :ref:`policy <minio-policy>` describing the
user's authorized actions and resources *or* assign the user to :ref:`groups
<minio-groups>` which have associated policies. See
:ref:`minio-access-management` for more information.
This page documents user management for the MinIO internal IDentity Provider
(IDP). MinIO also external management of identities using either an
OpenID Connect (OIDC) or Active Directory/LDAP IDentity Provider (IDP).
For more information, see:
- :ref:`minio-external-identity-management-openid`
- :ref:`minio-external-identity-management-ad-ldap`
Enabling external identity management disables the MinIO internal IDP, with
the exception of creating :ref:`service accounts
<minio-idp-service-account>`.
.. _minio-idp-service-account:
Service Accounts
----------------
MinIO service accounts are child identities of an authenticated MinIO user,
including :ref:`externally managed identities
<minio-authentication-and-identity-management>`. Each service account inherits
its privileges based on the :ref:`policies <minio-policy>` attached to it's
parent user *or* those groups in which the parent user has membership. Service
accounts also support an optional inline policy which further restricts access
to a subset of actions and resources available to the parent user.
A MinIO user can generate any number of service accounts. This allows
application owners to generate arbitrary service accounts for their applications
without requiring action from the MinIO administrators. Since the generated
service accounts have the same or fewer permissions as the parents,
administrators can focus on managing the top-level parent users without
micro-managing generated service accounts.
You can create service accounts using either the :ref:`MinIO Console <minio-console>` *or* by using the :mc-cmd:`mc admin user svcacct add` command.
.. admonition:: Service Accounts are for Programmatic Access
:class: dropdown, note
Service Accounts support programmatic access by applications. You cannot
use a Service Account to log into the MinIO Console.
.. _minio-users-root:
MinIO ``root`` User
-------------------
MinIO deployments have a ``root`` user with access to all actions and resources
on the deployment, regardless of the configured :ref:`identity manager
<minio-authentication-and-identity-management>`. When a :mc:`minio` server first
starts, it sets the ``root`` user credentials by checking the value of the
following environment variables:
- :envvar:`MINIO_ROOT_USER`
- :envvar:`MINIO_ROOT_PASSWORD`
Rotating the root user credentials requires updating either or both variables
for all MinIO servers in the deployment. Specify *long, unique, and random*
strings for root credentials. Exercise all possible precautions in storing the
access key and secret key, such that only known and trusted individuals who
*require* superuser access to the deployment can retrieve the ``root``
credentials.
- MinIO *strongly discourages* using the ``root`` user for regular client access
regardless of the environment (development, staging, or production).
- MinIO *strongly recommends* creating users such that each client has access to
the minimal set of actions and resources required to perform their assigned
workloads.
If these variables are unset, :mc:`minio` defaults to ``minioadmin`` and
``minioadmin`` as the access key and secret key respectively. MinIO *strongly
discourages* use of the default credentials regardless of deployment
environment.
.. admonition:: Deprecation of Legacy Root User Environment Variables
:class: dropdown, important
MinIO :minio-release:`RELEASE.2021-04-22T15-44-28Z` and later deprecates the
following variables used for setting or updating root user
credentials:
- :envvar:`MINIO_ACCESS_KEY` to the new access key.
- :envvar:`MINIO_SECRET_KEY` to the new secret key.
- :envvar:`MINIO_ACCESS_KEY_OLD` to the old access key.
- :envvar:`MINIO_SECRET_KEY_OLD` to the old secret key.
User Management
---------------
Create a User
~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin user add` command to create a new user on the
MinIO deployment:
.. code-block:: shell
:class: copyable
mc admin user add ALIAS ACCESSKEY SECRETKEY
- Replace :mc-cmd:`ALIAS <mc admin user add TARGET>` with the
:mc-cmd:`alias <mc alias>` of the MinIO deployment.
- Replace :mc-cmd:`ACCESSKEY <mc admin user add ACCESSKEY>` with the
access key for the user. MinIO allows retrieving the access key after
user creation through the :mc-cmd:`mc admin user info` command.
- Replace :mc-cmd:`SECRETKEY <mc admin user add SECRETKEY>` with the
secret key for the user. MinIO *does not* provide any method for retrieving
the secret key once set.
Specify a unique, random, and long string for both the ``ACCESSKEY`` and
``SECRETKEY``. Your organization may have specific internal or regulatory
requirements around generating values for use with access or secret keys.
After creating the user, use :mc-cmd:`mc admin policy set` to associate a
:ref:`MinIO Policy Based Access Control <minio-policy>` to the new user.
The following command assigns the built-in :userpolicy:`readwrite` policy:
.. code-block:: shell
:class: copyable
mc admin policy set ALIAS readwrite user=USERNAME
Replace ``USERNAME`` with the ``ACCESSKEY`` created in the previous step.
Delete a User
~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin user remove` command to remove a user on a
MinIO deployment:
.. code-block:: shell
:class: copyable
mc admin user remove ALIAS USERNAME
- Replace :mc-cmd:`ALIAS <mc admin user remove TARGET>` with the
:mc-cmd:`alias <mc alias>` of the MinIO deployment.
- Replace :mc-cmd:`USERNAME <mc admin user remove USERNAME>` with the name of
the user to remove.

View File

@@ -0,0 +1,82 @@
.. _minio-external-identity-management-openid:
.. _minio-external-identity-management-openid-access-control:
================================
OpenID Connect Access Management
================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
MinIO supports using an OpenID Connect (OIDC) compatible IDentity Provider (IDP)
such as Okta, KeyCloak, Dex, Google, or Facebook for external management of user
identities.
For identities managed by the external OpenID Connect (OIDC) compatible provider, MinIO uses the `JSON Web Token claim <https://datatracker.ietf.org/doc/html/rfc7519#section-4>`__ returned as part of the OIDC authentication flow to identify the :ref:`policies <minio-policy>` to assign to the authenticated user.
MinIO by default denies access to all actions or resources not explicitly allowed by a user's assigned or inherited :ref:`policies <minio-policy>`.
Users managed by an OIDC provider must specify the necessary policies as part of the JWT claim. If the user JWT claim has no matching MinIO policies, that user has no permissions to access any action or resource on the MinIO deployment.
The specific claim which MinIO looks for is configured as part of :ref:`deploying the cluster with OIDC identity management <minio-external-iam-oidc>`. This page focuses on creating MinIO policies to match the configured OIDC claims.
Authentication and Authorization Flow
-------------------------------------
The login flow for an application using :abbr:`OIDC (OpenID Connect)`
credentials is as follows:
1. Authenticate to the configured :abbr:`OIDC (OpenID Connect)`
provider and retrieve a
`JSON Web Token (JWT) <https://jwt.io/introduction>`__.
MinIO only supports the
`OpenID Authorization Code Flow
<https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowAuth>`__.
Authentication using Implicit Flow is not supported.
2. Specify the :abbr:`JWT (JSON Web Token)` to the MinIO Security Token Service
(STS) :ref:`minio-sts-assumerolewithwebidentity` API endpoint.
MinIO verifies the :abbr:`JWT (JSON Web Token)` against the
configured OIDC provider.
If the JWT is valid, MinIO checks for a :ref:`claim
<minio-external-identity-management-openid-access-control>` specifying a list
of one or more :ref:`policies <minio-policy>` to assign to the
authenticated user. MinIO defaults to checking the ``policy`` claim.
3. MinIO returns temporary credentials in the STS API response in the form of an
access key, secret key, and session token. The credentials have
permissions matching those policies specified in the JWT claim.
4. Applications use the temporary credentials returned by the STS endpoint to
perform authenticated S3 operations on MinIO.
MinIO provides an example Go application
:minio-git:`web-identity.go <minio/blob/master/docs/sts/web-identity.go>` that
handles the full login flow.
OIDC users can alternatively create :ref:`service accounts <minio-idp-service-account>` associated to their AD/LDAP user. Service accounts are long-lived credentials which inherit their privileges from the parent user. The parent user can further restrict those privileges while creating the service account. To create a new service account, log into the :ref:`MinIO Console <minio-console>` using the OIDC-managed user credentials. From the :guilabel:`Identity` section of the left navigation, select :guilabel:`Service Accounts` followed by the :guilabel:`Create service account +` button.
Identifying the JWT Claim Value
-------------------------------
MinIO uses the JWT token returned as part of the OIDC authentication flow to identify the specific policies to assign to the authenticated user.
You can use a `JWT Debugging tool <https://jwt.io/>`__ to decode the returned JWT token and validate that the user attributes include the required claims.
.. todo - example JWT claim
See `RFC 7519: JWT Claim <https://datatracker.ietf.org/doc/html/rfc7519#section-4>`__ for more information on JWT claims.
Defer to the documentation for your preferred OIDC provider for instructions on configuring user claims.
Creating Policies to Match Claims
---------------------------------
Use either the MinIO Console *or* the :mc-cmd:`mc admin policy` command to create policies that match one or more claim values:
.. todo - instructions

View File

@@ -0,0 +1,778 @@
.. _minio-policy:
=================
Access Management
=================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
Overview
--------
MinIO uses Policy-Based Access Control (PBAC) to define the authorized actions
and resources to which an authenticated user has access. Each policy describes
one or more :ref:`actions <minio-policy-actions>` and :ref:`conditions
<minio-policy-conditions>` that outline the permissions of a
:ref:`user <minio-users>` or :ref:`group <minio-groups>` of
users.
MinIO PBAC is built for compatibility with AWS IAM policy syntax, structure, and
behavior. The MinIO documentation makes a best-effort to cover IAM-specific
behavior and functionality. Consider deferring to the :iam-docs:`IAM
documentation <>` for more complete documentation on AWS IAM-specific topics.
The :mc-cmd:`mc admin policy` command supports creation and management of
policies on the MinIO deployment. See the command reference for examples of
usage.
.. _minio-policy-built-in:
Built-In Policies
-----------------
MinIO provides the following built-in policies for assigning to
:ref:`users <minio-users>` or :ref:`groups <minio-groups>`:
.. userpolicy:: consoleAdmin
Grants complete access to all S3 and administrative API operations against
all resources on the MinIO deployment. Equivalent to the following set of
actions:
- :policy-action:`s3:*`
- :policy-action:`admin:*`
.. userpolicy:: readonly
Grants read-only permissions on any object on the MinIO deployment. The GET
action *must* apply to a specific object without requiring any listing.
Equivalent to the following set of actions:
- :policy-action:`s3:GetBucketLocation`
- :policy-action:`s3:GetObject`
For example, this policy specifically supports GET operations on objects at a
specific path (e.g. ``GET play/mybucket/object.file``), such as:
- :mc-cmd:`mc cp`
- :mc-cmd:`mc stat`
- :mc-cmd:`mc head`
- :mc-cmd:`mc cat`
The exclusion of listing permissions is intentional, as typical use cases
do not intend for a "read-only" role to have complete discoverability
(listing all buckets and objects) on the object storage resource.
.. userpolicy:: readwrite
Grants read and write permissions for all buckets and objects on the
MinIO server. Equivalent to :policy-action:`s3:*`.
.. userpolicy:: diagnostics
Grants permission to perform diagnostic actions on the MinIO deployment.
Specifically includes the following actions:
- :policy-action:`admin:ServerTrace`
- :policy-action:`admin:Profiling`
- :policy-action:`admin:ConsoleLog`
- :policy-action:`admin:ServerInfo`
- :policy-action:`admin:TopLocksInfo`
- :policy-action:`admin:OBDInfo`
- :policy-action:`admin:BandwidthMonitor`
- :policy-action:`admin:Prometheus`
.. userpolicy:: writeonly
Grants write-only permissions to any namespace (bucket and path to object)
the MinIO deployment. The PUT action *must* apply to a specific object
location without requiring any listing.
Equivalent to the :policy-action:`s3:PutObject` action.
Use :mc-cmd:`mc admin policy set` to associate a policy to a
user or group on a MinIO deployment.
For example, consider the following table of users. Each user is assigned
a :ref:`built-in policy <minio-policy-built-in>` or
a supported :ref:`action <minio-policy-actions>`. The table
describes a subset of operations a client could perform if authenticated
as that user:
.. list-table::
:header-rows: 1
:widths: 20 40 40
:width: 100%
* - User
- Policy
- Operations
* - ``Operations``
- | :userpolicy:`readwrite` on ``finance`` bucket
| :userpolicy:`readonly` on ``audit`` bucket
- | ``PUT`` and ``GET`` on ``finance`` bucket.
| ``PUT`` on ``audit`` bucket
* - ``Auditing``
- | :userpolicy:`readonly` on ``audit`` bucket
- ``GET`` on ``audit`` bucket
* - ``Admin``
- :policy-action:`admin:*`
- All :mc-cmd:`mc admin` commands.
Each user can access only those resources and operations which are *explicitly*
granted by the built-in role. MinIO denies access to any other resource or
action by default.
.. admonition:: ``Deny`` overrides ``Allow``
:class: note
MinIO follows the IAM policy evaluation rules where a ``Deny`` rule overrides
``Allow`` rule on the same action/resource. For example, if a user has an
explicitly assigned policy with an ``Allow`` rule for an action/resource
while one of its groups has an assigned policy with a ``Deny`` rule for that
action/resource, MinIO would apply only the ``Deny`` rule.
For more information on IAM policy evaluation logic, see the IAM
documentation on
:iam-docs:`Determining Whether a Request is Allowed or Denied Within an Account
<reference_policies_evaluation-logic.html#policy-eval-denyallow>`.
.. _minio-policy-document:
Policy Document Structure
-------------------------
MinIO policy documents use the same schema as
:aws-docs:`AWS IAM Policy <IAM/latest/UserGuide/access.html>` documents.
The following sample document provides a template for creating custom
policies for use with a MinIO deployment. For more complete documentation on IAM
policy elements, see the :aws-docs:`IAM JSON Policy Elements Reference
<IAM/latest/UserGuide/reference_policies_elements.html>`.
.. code-block:: javascript
:class: copyable
{
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Action" : [ "s3:<ActionName>", ... ],
"Resource" : "arn:aws:s3:::*",
"Condition" : { ... }
},
{
"Effect" : "Deny",
"Action" : [ "s3:<ActionName>", ... ],
"Resource" : "arn:aws:s3:::*",
"Condition" : { ... }
}
]
}
- For the ``Statement.Action`` array, specify one or more
:ref:`supported S3 API operations <minio-policy-actions>`. MinIO deployments
supports a subset of AWS S3 API operations.
- For the ``Statement.Resource`` key, you can replace the ``*`` with
the specific bucket to which the policy statement should apply.
Using ``*`` applies the statement to all resources on the MinIO deployment.
- For the ``Statement.Condition`` key, you can specify one or more
:ref:`supported Conditions <minio-policy-conditions>`. MinIO
deployments supports a subset of AWS S3 conditions.
.. _minio-policy-actions:
Supported S3 Policy Actions
---------------------------
MinIO policy documents support a subset of IAM
:iam-docs:`S3 Action keys <list_amazons3.html#amazons3-actions-as-permissions>`.
The following actions control access to common S3 operations. The remaining
subsections document actions for more advanced S3 operations:
.. policy-action:: s3:*
Selector for *all* MinIO S3 operations. Applying this action to a given
resource allows the user to perform *any* S3 operation against that
resource.
.. policy-action:: s3:CreateBucket
Controls access to the :s3-api:`CreateBucket <API_CreateBucket.html>` S3 API
operation.
.. policy-action:: s3:DeleteBucket
Controls access to the :s3-api:`DeleteBucket <API_DeleteBucket.html>` S3 API
operation.
.. policy-action:: s3:ForceDeleteBucket
Controls access to the :s3-api:`DeleteBucket <API_DeleteBucket.html>`
S3 API operation for operations with the ``x-minio-force-delete`` flag.
Required for removing non-empty buckets.
.. policy-action:: s3:GetBucketLocation
Controls access to the :s3-api:`GetBucketLocation
<API_GetBucketLocation.html>` S3 API operation.
.. policy-action:: s3:ListAllMyBuckets
Controls access to the :s3-api:`ListBuckets <API_ListBuckets.html>`
S3 API operation.
.. policy-action:: s3:DeleteObject
Controls access to the :s3-api:`DeleteObject <API_DeleteObject.html>` S3 API
operation.
.. policy-action:: s3:GetObject
Controls access to the :s3-api:`GetObject <API_GetObject.html>` S3 API
operation.
.. policy-action:: s3:ListBucket
Controls access to the :s3-api:`ListObjectsV2 <API_ListObjectsV2.html>` S3 API
operation.
.. policy-action:: s3:PutObject
Controls access to the :s3-api:`PutObject <API_PutObject.html>` S3 API
operation.
.. policy-action:: s3:PutObjectTagging
Controls access to the :s3-api:`PutObjectTagging <API_PutObjectTagging.html>`
S3 API operation.
.. policy-action:: s3:GetObjectTagging
Controls access to the :s3-api:`GetObjectTagging <API_GetObjectTagging.html>`
S3 API operation.
Bucket Configuration
~~~~~~~~~~~~~~~~~~~~
.. policy-action:: s3:GetBucketPolicy
Controls access to the :s3-api:`GetBucketPolicy <API_GetBucketPolicy.html>`
S3 API operation.
.. policy-action:: s3:PutBucketPolicy
Controls access to the :s3-api:`PutBucketPolicy <API_PutBucketPolicy.html>`
S3 API operation.
.. policy-action:: s3:DeleteBucketPolicy
Controls access to the :s3-api:`DeleteBucketPolicy
<API_DeleteBucketPolicy.html>` S3 API operation.
.. policy-action:: s3:GetBucketTagging
Controls access to the :s3-api:`GetBucketTagging <API_GetBucketTagging.html>`
S3 API operation.
.. policy-action:: s3:PutBucketTagging
Controls access to the :s3-api:`PutBucketTagging <API_PutBucketTagging.html>`
S3 API operation.
Multipart Upload
~~~~~~~~~~~~~~~~
.. policy-action:: s3:AbortMultipartUpload
Controls access to the :s3-api:`AbortMultipartUpload
<API_AbortMultipartUpload.html>` S3 API operation.
.. policy-action:: s3:ListMultipartUploadParts
Controls access to the :s3-api:`ListParts <API_ListParts.html>` S3 API
operation.
.. policy-action:: s3:ListBucketMultipartUploads
Controls access to the :s3-api:`ListMultipartUploads
<API_ListMultipartUploads.html>` S3 API operation.
Versioning and Retention
~~~~~~~~~~~~~~~~~~~~~~~~
.. policy-action:: s3:PutBucketVersioning
Controls access to the :s3-api:`PutBucketVersioning
<API_PutBucketVersioning.html>` S3 API operation.
.. policy-action:: s3:GetBucketVersioning
Controls access to the :s3-api:`GetBucketVersioning
<API_GetBucketVersioning.html>` S3 API operation.
.. policy-action:: s3:DeleteObjectVersion
Controls access to the :s3-api:`DeleteObjectVersion
<API_DeleteObjectVersion.html>` S3 API operation.
.. policy-action:: s3:DeleteObjectVersionTagging
Controls access to the :s3-api:`DeleteObjectVersionTagging
<API_DeleteObjectVersionTagging.html>` S3 API operation.
.. policy-action:: s3:GetObjectVersion
Controls access to the :s3-api:`GetObjectVersion
<API_GetObjectVersion.html>` S3 API operation.
.. policy-action:: s3:BypassGovernanceRetention
Controls access to the following S3 API operations on objects
locked under :mc-cmd:`GOVERNANCE <mc retention set MODE>`
retention mode:
- ``PutObjectRetention``
- ``PutObject``
- ``DeleteObject``
See the S3 documentation on :s3-docs:`s3:BypassGovernanceRetention
<object-lock-managing.html#object-lock-managing-bypass>` for more
information.
.. policy-action:: s3:PutObjectRetention
Controls access to the :s3-api:`PutObjectRetention
<API_PutObjectRetention.html>` S3 API operation.
Required for any ``PutObject`` operation that specifies
:ref:`retention metadata <minio-object-locking>`.
.. policy-action:: s3:GetObjectRetention
Controls access to the :s3-api:`GetObjectRetention
<API_GetObjectRetention.html>` S3 API operation.
Required for including :ref:`object locking metadata <minio-object-locking>`
as part of the response to a ``GetObject`` or ``HeadObject`` operation.
.. policy-action:: s3:GetObjectLegalHold
Controls access to the :s3-api:`GetObjectLegalHold
<API_GetObjectLegalHold.html>` S3 API operation.
Required for including :ref:`object locking metadata <minio-object-locking>`
as part of the response to a ``GetObject`` or ``HeadObject`` operation.
.. policy-action:: s3:PutObjectLegalHold
Controls access to the :s3-api:`PutObjectLegalHold
<API_PutObjectLegalHold.html>` S3 API operation.
Required for any ``PutObject`` operation that specifies
:ref:`legal hold metadata <minio-object-locking>`.
.. policy-action:: s3:GetBucketObjectLockConfiguration
Controls access to the :s3-api:`GetObjectLockConfiguration
<API_GetObjectLockConfiguration.html>` S3 API operation.
.. policy-action:: s3:PutBucketObjectLockConfiguration
Controls access to the :s3-api:`PutObjectLockConfiguration
<API_PutObjectLockConfiguration.html>` S3 API operation.
Bucket Notifications
~~~~~~~~~~~~~~~~~~~~
.. policy-action:: s3:GetBucketNotification
Controls access to the :s3-api:`GetBucketNotification
<API_GetBucketNotification.html>` S3 API operation.
.. policy-action:: s3:PutBucketNotification
Controls access to the :s3-api:`PutBucketNotification
<API_PutBucketNotification.html>` S3 API operation.
.. policy-action:: s3:ListenNotification
MinIO Extension for controlling API operations related to MinIO Bucket
Notifications.
This action is **not** intended for use with other S3-compatible services.
.. policy-action:: s3:ListenBucketNotification
MinIO Extension for controlling API operations related to MinIO Bucket
Notifications.
This action is **not** intended for use with other S3-compatible services.
Object Lifecycle Management
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. policy-action:: s3:PutLifecycleConfiguration
Controls access to the :s3-api:`PutLifecycleConfiguration
<API_PutBucketLifecycleConfiguration.html>` S3 API operation.
.. policy-action:: s3:GetLifecycleConfiguration
Controls access to the :s3-api:`GetLifecycleConfiguration
<API_GetBucketLifecycleConfiguration.html>` S3 API operation.
Object Encryption
~~~~~~~~~~~~~~~~~
.. policy-action:: s3:PutEncryptionConfiguration
Controls access to the :s3-api:`PutEncryptionConfiguration
<API_PutBucketEncryption.html>` S3 API operation.
.. policy-action:: s3:GetEncryptionConfiguration
Controls access to the :s3-api:`GetEncryptionConfiguration
<API_GetBucketEncryption.html>` S3 API operation.
Bucket Replication
~~~~~~~~~~~~~~~~~~
.. policy-action:: s3:GetReplicationConfiguration
Controls access to the :s3-api:`GetBucketReplication
<API_GetBucketReplication.html>` S3 API operation.
.. policy-action:: s3:PutReplicationConfiguration
Controls access to the :s3-api:`PutBucketReplication
<PutBucketReplication.html>` S3 API operation.
.. policy-action:: s3:ReplicateObject
MinIO Extension for controlling API operations related to
:ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`.
Required for server-side replication.
.. policy-action:: s3:ReplicateDelete
MinIO Extension for controlling API operations related to
:ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`.
Required for synchronizing delete operations as part of server-side
replication.
.. policy-action:: s3:ReplicateTags
MinIO Extension for controlling API operations related to
:ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`.
Required for server-side replication.
.. policy-action:: s3:GetObjectVersionForReplication
MinIO Extension for controlling API operations related to
:ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`.
Required for server-side replication.
.. _minio-policy-conditions:
Supported S3 Policy Condition Keys
----------------------------------
MinIO policy documents support IAM
:iam-docs:`conditional statements <reference_policies_elements_condition.html>`.
Each condition element consists of
:iam-docs:`operators <reference_policies_elements_condition_operators.html>`
and condition keys. MinIO supports a subset of IAM condition keys. For complete
information on any listed condition key, see the
:iam-docs:`IAM Condition Element Documentation
<reference_policies_elements_condition.html>`
MinIO supports the following condition keys for all supported
:ref:`actions <minio-policy-actions>`:
- ``aws:Referer``
- ``aws:SourceIp``
- ``aws:UserAgent``
- ``aws:SecureTransport``
- ``aws:CurrentTime``
- ``aws:EpochTime``
- ``aws:PrincipalType``
- ``aws:userid``
- ``aws:username``
- ``x-amz-content-sha256``
The following table lists additional supported condition keys for specific
actions:
.. list-table::
:header-rows: 1
:widths: 30 70
:width: 100%
* - Action Key
- Condition Keys
* - :policy-action:`s3:GetObject`
- | ``x-amz-server-side-encryption``
| ``x-amz-server-side-encryption-customer-algorithm``
* - :policy-action:`s3:ListBucket`
- | ``prefix``
| ``delimiter``
| ``max-keys``
* - :policy-action:`s3:PutObject`
- | ``x-amz-copy-source``
| ``x-amz-server-side-encryption``
| ``x-amz-server-side-encryption-customer-algorithm``
| ``x-amz-metadata-directive``
| ``x-amz-storage-class``
| ``object-lock-retain-until-date``
| ``object-lock-mode``
| ``object-lock-legal-hold``
* - :policy-action:`s3:PutObjectRetention`
- | ``x-amz-object-lock-remaining-retention-days``
| ``x-amz-object-lock-retain-until-date``
| ``x-amz-object-lock-mode``
* - :policy-action:`s3:PutObjectLegalHold`
- ``object-lock-legal-hold``
* - :policy-action:`s3:BypassGovernanceRetention`
- | ``object-lock-remaining-retention-days``
| ``object-lock-retain-until-date``
| ``object-lock-mode``
| ``object-lock-legal-hold``
* - :policy-action:`s3:GetObjectVersion`
- ``versionid``
* - :policy-action:`s3:DeleteObjectVersion`
- ``versionid``
.. _minio-policy-mc-admin-actions:
``mc admin`` Policy Action Keys
-------------------------------
MinIO supports the following actions for use with defining policies
for :mc-cmd:`mc admin` operations. These actions are *only* valid for
MinIO deployments and are *not* intended for use with other S3-compatible
services:
.. policy-action:: admin:*
Selector for all admin action keys.
.. policy-action:: admin:Heal
Allows heal command
.. policy-action:: admin:StorageInfo
Allows listing server info
.. policy-action:: admin:DataUsageInfo
Allows listing data usage info
.. policy-action:: admin:TopLocksInfo
Allows listing top locks
.. policy-action:: admin:Profiling
Allows profiling
.. policy-action:: admin:ServerTrace
Allows listing server trace
.. policy-action:: admin:ConsoleLog
Allows listing console logs on terminal
.. policy-action:: admin:KMSCreateKey
Allows creating a new KMS master key
.. policy-action:: admin:KMSKeyStatus
Allows getting KMS key status
.. policy-action:: admin:ServerInfo
Allows listing server info
.. policy-action:: admin:OBDInfo
Allows obtaining cluster on-board diagnostics
.. policy-action:: admin:ServerUpdate
Allows MinIO binary update
.. policy-action:: admin:ServiceRestart
Allows restart of MinIO service.
.. policy-action:: admin:ServiceStop
Allows stopping MinIO service.
.. policy-action:: admin:ConfigUpdate
Allows MinIO config management
.. policy-action:: admin:CreateUser
Allows creating MinIO user
.. policy-action:: admin:DeleteUser
Allows deleting MinIO user
.. policy-action:: admin:ListUsers
Allows list users permission
.. policy-action:: admin:EnableUser
Allows enable user permission
.. policy-action:: admin:DisableUser
Allows disable user permission
.. policy-action:: admin:GetUser
Allows GET permission on user info
.. policy-action:: admin:AddUserToGroup
Allows adding user to group permission
.. policy-action:: admin:RemoveUserFromGroup
Allows removing user to group permission
.. policy-action:: admin:GetGroup
Allows getting group info
.. policy-action:: admin:ListGroups
Allows list groups permission
.. policy-action:: admin:EnableGroup
Allows enable group permission
.. policy-action:: admin:DisableGroup
Allows disable group permission
.. policy-action:: admin:CreatePolicy
Allows create policy permission
.. policy-action:: admin:DeletePolicy
Allows delete policy permission
.. policy-action:: admin:GetPolicy
Allows get policy permission
.. policy-action:: admin:AttachUserOrGroupPolicy
Allows attaching a policy to a user/group
.. policy-action:: admin:ListUserPolicies
Allows listing user policies
.. policy-action:: admin:CreateServiceAccount
Allows creating MinIO Service Account
.. policy-action:: admin:UpdateServiceAccount
Allows updating MinIO Service Account
.. policy-action:: admin:RemoveServiceAccount
Allows deleting MinIO Service Account
.. policy-action:: admin:ListServiceAccounts
Allows listing MinIO Service Account
.. policy-action:: admin:SetBucketQuota
Allows setting bucket quota
.. policy-action:: admin:GetBucketQuota
Allows getting bucket quota
.. policy-action:: admin:SetBucketTarget
Allows setting bucket target
.. policy-action:: admin:GetBucketTarget
Allows getting bucket targets
.. policy-action:: admin:SetTier
Allows creating and modifying remote storage tiers using the
:mc-cmd:`mc admin tier` command.
.. policy-action:: admin:ListTier
Allows listing configured remote storage tiers using the
:mc-cmd:`mc admin tier` command.
.. policy-action:: admin:BandwidthMonitor
Allows retrieving metrics related to current bandwidth consumption.
.. policy-action:: admin:Prometheus
Allows access to MinIO :ref:`metrics <minio-metrics-and-alerts-endpoints>`.
Only required if MinIO requires authentication for scraping metrics.
``mc admin`` Policy Condition Keys
----------------------------------
MinIO supports the following conditions for use with defining policies for
:mc-cmd:`mc admin` :ref:`actions <minio-policy-mc-admin-actions>`.
- ``aws:Referer``
- ``aws:SourceIp``
- ``aws:UserAgent``
- ``aws:SecureTransport``
- ``aws:CurrentTime``
- ``aws:EpochTime``
For complete information on any listed condition key, see the :iam-docs:`IAM
Condition Element Documentation <reference_policies_elements_condition.html>`

View File

@@ -0,0 +1,525 @@
.. _minio-console:
=============
MinIO Console
=============
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
The MinIO Console is a rich graphical user interface that provides similar
functionality to the :mc:`mc` command line tool.
.. image:: /images/minio-console/minio-console.png
:width: 600px
:alt: MinIO Console Landing Page provides a view of Buckets on the deployment
:align: center
You can use the MinIO Console for administration tasks like Identity and
Access Management, Metrics and Log Monitoring, or Server Configuration.
The MinIO Console is embedded as part of the MinIO Server binary starting
with :minio-release:`RELEASE.2021-07-08T01-15-01Z`. You can also deploy a
standalone MinIO Console using the instructions in the
:minio-git:`github repository <console>`.
You can explore the Console using https://play.min.io:9443. Log in with
the following credentials:
- Username: ``Q3AM3UQ867SPQQA43P2F``
- Password: ``zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG``
The Play Console connects to the MinIO Play deployment at https://play.min.io.
You can also access this deployment using :mc:`mc` and using the ``play``
alias.
This page documents the high level configuration settings and features of the
MinIO Console.
Configuration
-------------
The MinIO Console inherits the majority of its configuration settings from the
MinIO Server. The following environment variables enable specific behavior in
the MinIO Console:
.. list-table::
:header-rows: 1
:widths: 30 70
:width: 100%
* - Environment Variable
- Description
* - :envvar:`MINIO_PROMETHEUS_URL`
- The URL for a Prometheus server configured to scrape metrics from the
MinIO deployment. The MinIO Console uses this server for populating the
metrics dashboard.
See :ref:`minio-metrics-collect-using-prometheus` for a tutorial on
configuring Prometheus to collect metrics from MinIO.
* - :envvar:`MINIO_SERVER_URL`
- The URL hostname the MinIO Console uses for connecting to the MinIO
Server. The hostname *must* be resolveable and reachable for the
Console to function correctly.
The MinIO Console connects to the MinIO Server using an IP
address by default. For example, when the MinIO Server starts up,
the server logs include a line
``API: https://<IP ADDRESS 1> https://<IP ADDRESS 2>``.
The MinIO Console defaults to connecting using ``<IP ADDRESS 1>``.
The MinIO Console may require setting this variable in the following
scenarios:
- The MinIO server TLS certificates do not include the local IP address
as a :rfc:`Subject Alternative Name <5280#section-4.2.1.6>` (SAN).
Specify a hostname contained in the TLS certificate to allow the MinIO
Console to validate the TLS connection.
- The MinIO server's local IP address is not reachable by the MinIO
Console. Specify a resolveable hostname for the MinIO Server.
- A load balancer or reverse proxy controls traffic to the MinIO server,
such that the MinIO Console cannot reach the server without going
through the load balancer/proxy. Specify the load balancer/proxy
URL for the MinIO server.
* - :envvar:`MINIO_BROWSER_REDIRECT_URL`
- The externally resolvable hostname for the MinIO Console used by the
configured :ref:`external identity manager
<minio-authentication-and-identity-management>` for returning the
authentication response.
This variable is typically necessary when using a reverse proxy,
load balancer, or similar system to expose the MinIO Console to the
public internet. Specify an externally reachable hostname that resolves
to the MinIO Console.
Static vs Dynamic Port Assignment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO by default selects a random port for the MinIO Console on each server
startup. Browser clients accessing the MinIO Server are automatically
redirected to the MinIO Console on its dynamically selected port.
This behavior emulates the legacy web browser behavior while reducing the
the risk of a port collision on systems which were running MinIO *before* the
embedded Console update.
You can select an explicit static port by passing the
:mc-cmd:`minio server --console-address` commandline option when starting
each MinIO Server in the deployment.
For example, the following command starts a distributed MinIO deployment using
a static port assignment of ``9001`` for the MinIO Console. This deployment
would respond to S3 API operations on the default MinIO server port ``:9000``
and browser access on the MinIO Console port ``:9001``.
.. code-block:: shell
:class: copyable
minio server https://minio-{1...4}.example.net/mnt/disk-{1...4} \
--console-address ":9001"
Deployments behind network routing components which require static ports for
routing rules may require setting a static MinIO Console port. For example,
load balancers, reverse proxies, or Kubernetes ingress may by default block
or exhibit unexpected behavior with the the dynamic redirection behavior.
.. _minio-console-admin-buckets:
Buckets
-------
.. image:: /images/minio-console/console-object-browser.png
:width: 600px
:alt: MinIO Console Object Browser
:align: center
The Console :guilabel:`Object Browser` section displays all buckets and objects to which the authenticated user has :ref:`access <minio-policy>`.
Use the :guilabel:`Search` bar to search for specific buckets or objects.
Select the row for the bucket or object to browse.
Select :guilabel:`Create Bucket` to create a new bucket on the deployment.
Each bucket has :guilabel:`Manage` and :guilabel:`Browse` buttons.
- Select :guilabel:`Manage` to open the management interface for the bucket:
The :guilabel:`Summary` view displays a summary of the bucket's configuration.
The :guilabel:`Events` view supports configuring :ref:`notification events <minio-bucket-notifications>` using a configured notification target.
The :guilabel:`Replication` view supports creating and managing :ref:`Server Side Bucket Replication Rules <minio-bucket-replication-serverside>`.
The :guilabel:`Lifecycle` view supports creating and managing :ref:`Object Lifecycle Management Rules <minio-lifecycle-management>` for the bucket.
The :guilabel:`Access Audit` view displays all :ref:`policies <minio-policy>` and :ref:`users <minio-users>` with access to that bucket.
The :guilabel:`Access Rules` view supports creating and managing anonymous bucket policies to attach to the bucket or bucket prefix.
Anonymous rules allow clients to access the bucket or prefix without explicitly authenticating with user credentials.
- Select :guilabel:`Browse` to view the contents of the bucket.
You can view and download individual objects, upload new objects, or use the :guilabel:`Rewind` function to view only those :ref:`versions <minio-bucket-versioning>` of an object which existed at the selected timestamp.
Identity
--------
The :guilabel:`Identity` section provides a management interface for :ref:`MinIO-Managed users <minio-users>`.
The section contains the following subsections.
Some subsections may not be visible if the authenticated user does not have the :ref:`required administrative permissions <minio-policy-mc-admin-actions>`.
.. tab-set::
.. tab-item:: Users
.. image:: /images/minio-console/console-users.png
:width: 600px
:alt: MinIO Console Manage Users
:align: center
The :guilabel:`Users` section displays all MinIO-managed :ref:`users <minio-users>` on the deployment.
This section is not visible for deployments using an external identity manager such as Active Directory or an OIDC-compatible provider.
- Select :guilabel:`Create User` to create a new MinIO-managed user.
You can assign :ref:`groups <minio-groups>` and :ref:`policies <minio-policy>` to the user during creation.
- Select a user's row to view details for that user.
You can view and modify the user's assigned :ref:`groups <minio-groups>` and :ref:`policies <minio-policy>`.
You can also view and manage any :ref:`Service Accounts <minio-idp-service-account>` associated to the user.
.. tab-item:: Groups
.. image:: /images/minio-console/console-groups.png
:width: 600px
:alt: MinIO Console Manage Groups
:align: center
The :guilabel:`Groups` section displays all :ref:`groups <minio-groups>` on the MinIO deployment.
This section is not visible for deployments using an external identity manager such as Active Directory or an OIDC-compatible provider.
- Select :guilabel:`Create Group` to create a new MinIO Group.
You can assign new users to the group during creation.
You can assign policies to the group after creation.
- Select the group row to open the details for that group.
You can modify the group membership from the :guilabel:`Members` view.
You can modify the group's assigned policies from the :guilabel:`Policies` view.
Changing a user's group membership modifies the policies that user inherits. See :ref:`minio-access-management` for more information.
.. tab-item:: Service Accounts
.. image:: /images/minio-console/console-service-accounts.png
:width: 600px
:alt: MinIO Console Service Accounts
:align: center
The :guilabel:`Accounts` section displays all :ref:`minio-idp-service-account` associated to the authenticated user.
Service accounts support providing applications authentication credentials which inherit permissions from the "parent" user.
For deployments using an external identity manager such as Active Directory or an OIDC-compatible provider, service accounts provide a way for users to create long-lived credentials.
- You can select the service account row to view its custom policy, if one exists.
You can create or modify the policy from this screen.
Service account policies cannot exceed the permissions granted to the parent user.
- You can create a new service account by selecting the :guilabel:`Create service account` button.
The Console auto-generates an access key and password for the account.
You can override these values as necessary.
You can set a custom policy for the service account that further restricts the permissions granted to the account.
The Console only displays the service account credentials *once*. You cannot
change or retrieve the credentials later. To rotate credentials for an
application, create a new service account and delete the old one once the
application updates to using the new credentials.
Access
------
.. image:: /images/minio-console/console-iam.png
:width: 600px
:alt: MinIO Console Manage IAM Policies
:align: center
The :guilabel:`IAM Policies` section displays all :ref:`policies <minio-policy>` on the MinIO deployment.
This tab or its contents may not be visible if the authenticated user does not have the :ref:`required administrative permissions <minio-policy-mc-admin-actions>`.
- Select :guilabel:`+ Create Policy` to create a new MinIO Policy.
- Select the policy row to manage the policy details.
The :guilabel:`Summary` view displays a summary of the policy.
The :guilabel:`Users` view displays all users assigned to the policy.
The :guilabel:`Groups` view displays all groups assigned to the policy.
The :guilabel:`Raw Policy` view displays the raw JSON policy.
Use the :guilabel:`Identity: Users` and :guilabel:`Identity: Groups` views to assign a created policy to users and groups, respectively.
Monitoring
----------
The :guilabel:`Monitoring` section provides an interface for monitoring the MinIO deployment.
The section contains the following subsections,
Some subsections may not be visible if the authenticated user does not have the :ref:`required administrative permissions <minio-policy-mc-admin-actions>`.
.. tab-set::
.. tab-item:: Metrics
.. image:: /images/minio-console/console-metrics.png
:width: 600px
:alt: MinIO Console Metrics displaying detailed data using Prometheus
:align: center
The Console :guilabel:`Dashboard` section displays metrics for the MinIO deployment.
The Console depends on a :ref:`configured Prometheus service <minio-metrics-collect-using-prometheus>` to generate the detailed metrics shown above.
The default metrics view provides a high-level overview of the deployment status, including the uptime and availability of individual servers and drives.
.. image:: /images/minio-console/console-metrics-simple.png
:width: 600px
:alt: MinIO Console Metrics displaying simplified data
:align: center
This view requires configuring a Prometheus service to scrape the deployment metrics.
See :ref:`minio-metrics-collect-using-prometheus` for complete instructions.
.. tab-item:: Logs
.. image:: /images/minio-console/console-logs.png
:width: 600px
:alt: MinIO Console Logs displaying a list of server logs
:align: center
The Console :guilabel:`Logs` section displays :ref:`server logs <minio-logging>` generated by the MinIO Deployment.
- Use the :guilabel:`Nodes` dropdown to filter logs to a subset of server nodes in the MinIO deployment.
- Use the :guilabel:`Log Types` dropdown to filter logs to a subset of log types.
- Use the :guilabel:`Filter` to apply text filters to the log results
Select the :guilabel:`Start Logs` button to begin collecting logs using the selected filters and settings.
.. tab-item:: Audit
The Audit Log section provides an interface for viewing :ref:`audit logs <minio-logging>` collected by a configured PostgreSQL service.
The Audit Logging feature is configured and enabled automatically for MinIO deployments created using the :docs-k8s:`MinIO Kubernetes Operator <>`.
.. tab-item:: Trace
.. image:: /images/minio-console/console-trace.png
:width: 600px
:alt: MinIO Console Trace
:align: center
The :guilabel:`Trace` section provides HTTP trace functionality for a bucket or buckets on the deployment.
This section provides similar functionality to :mc:`mc admin trace`.
You can modify the trace to show only specific trace calls.
The default is to show only :guilabel:`S3` related HTTP traces.
Select :guilabel:`Filters` to open additional filters to apply to trace output, such as restricting the :guilabel:`Path` on which the trace applies to a specific bucket or bucket prefix.
.. tab-item:: Watch
.. image:: /images/minio-console/console-watch.png
:width: 600px
:alt: MinIO Console Watch
:align: center
The :guilabel:`Watch` section displays S3 events as they occur on the selected bucket.
This section provides similar functionality to :mc:`mc watch`.
.. tab-item:: Drives
.. image:: /images/minio-console/console-drives.png
:width: 600px
:alt: MinIO Console Drive Health Status
:align: center
The :guilabel:`Drives` section displays the healing status for a bucket.
MinIO automatically heals objects and drives when it detects problems, such as drive-level corruption or a replacement drive.
.. important::
MinIO does not recommend performing manual healing unless explicitly directed by support.
Support
-------
The :guilabel:`Support` section provides an interface for generating health and performance reports.
You can also register your deployment with |subnet| to allow upload of health reports directly through the MinIO Console.
This section contains the following subsections.
Some subsections may not be visible if the authenticated user does not have the :ref:`required administrative permissions <minio-policy-mc-admin-actions>`.
.. tab-set::
.. tab-item:: Register
.. image:: /images/minio-console/console-register.png
:width: 600px
:alt: MinIO Console - SUBNET Registration login form
:align: center
The :guilabel:`Register` section provides a login form to connect your MinIO deployment to |subnet|.
After registration, you can upload your deployment health reports directly to SUBNET for reference by MinIO Engineering.
.. tab-item:: Health
.. image:: /images/minio-console/console-health.png
:width: 600px
:alt: MinIO Console - Health Diagnostics
:align: center
The :guilabel:`Health` section provides an interface for running a health diagnostic for the MinIO Deployment.
The resulting health report is intended for use by MinIO Engineering via |subnet| and may contain internal or private data points such as hostnames.
Exercise caution before sending a health report to a third party or posting the health report in a public forum.
.. tab-item:: Performance
.. image:: /images/minio-console/console-performance.png
:width: 600px
:alt: MinIO Console - Performance Tests
:align: center
The :guilabel:`Performance` section provides an interface for running a performance test of the deployment.
The resulting test can provide a general guideline of deployment performance under S3 ``GET`` and ``PUT`` requests.
For more complete performance testing, consider using a combination of load-testing using your staging application environments and the MinIO :minio-git:`WARP <warp>` tool.
.. tab-item:: Profile
.. image:: /images/minio-console/console-profile.png
:width: 600px
:alt: MinIO Console - Profile Tests
:align: center
The :guilabel:`Profile` section provides an interface for running system profiling of the deployment.
The results can provide insight into the MinIO server process running on a given node.
The resulting report is intended for use by MinIO Engineering via |subnet|.
Independent or third-party use of these profiles for diagnostics and remediation is done at your own risk.
.. tab-item:: Inspect
.. image:: /images/minio-console/console-inspect.png
:width: 600px
:alt: MinIO Console - Inspect an Object
:align: center
The :guilabel:`Inspect` section provides an interface for capturing the erasure-coded metadata associated to an object or objects.
MinIO Engineering may request this output as part of diagnostics in |subnet|.
The resulting object may be read using MinIO's :minio-git:`debugging tool <minio/tree/master/docs/debugging#decoding-metadata>`.
Independent or third-party use of the output for diagnostics or remediation is done at your own risk.
You can optionally encrypt the object such that it can only be read if the generated encryption key is included as part of the debugging toolchain.
License
-------
The :guilabel:`License` section displays information on the licensing status of the MinIO deployment.
For deployments not registered via |subnet|, the Console displays a table comparison of MinIO License and Support plans:
.. image:: /images/minio-console/console-license.png
:width: 600px
:alt: MinIO Console - License Plans
:align: center
MinIO is Open Source software under the :minio-git:`GNU AGPLv3 license <mc/blob/master/LICENSE>`.
Applications using MinIO should follow local laws and regulations around licensing to ensure compliance with the AGPLv3 license, which may include open sourcing the application stack.
Proprietary application stacks can register for either the SUBNET :guilabel:`Standard` or :guilabel:`Enterprise` License and Support plan to use MinIO under a commercial license.
Settings
--------
The :guilabel:`Configuration` section displays information on MinIO server configuration settings.
This section contains the following subsections.
Some subsections may not be visible if the authenticated user does not have the :ref:`required administrative permissions <minio-policy-mc-admin-actions>`.
.. tab-set::
.. tab-item:: Configuration
.. image:: /images/minio-console/console-settings-configuration.png
:width: 600px
:alt: MinIO Console Settings - Configuration View
:align: center
The :guilabel:`Configuration` subsection provides an interface for viewing and retrieving :ref:`configuration settings <minio-server-configuration-settings>` for all MinIO Servers in the deployment.
The interface functionality mimics that of using :mc-cmd:`mc admin config get` or :mc-cmd:`mc admin config set`
Some configuration settings may require restarting the MinIO deployment to apply changes.
.. tab-item:: Notifications
.. image:: /images/minio-console/console-settings-notifications.png
:width: 600px
:alt: MinIO Console Settings - Notifications View
:align: center
The :guilabel:`Notifications` subsection provides an interface for adding and managing :ref:`bucket notification targets <minio-bucket-notifications>`.
Select the :guilabel:`Add Notification Target +` button to add a new target to the deployment.
You can select an existing notification target from the list to view its details.
.. tab-item:: Tiers
.. image:: /images/minio-console/console-settings-tiers.png
:width: 600px
:alt: MinIO Console Settings - Tiering
:align: center
The :guilabel:`Tiers` subsection provides an interface for adding and managing :ref:`remote tiers <minio-lifecycle-management-tiering>` to support lifecycle management transition rules.
Select the :guilabel:`Create Tier +` button to add a new tier to the deployment.
You can select an existing tier from the list to view its details.
.. tab-item:: Site Replication
.. image:: /images/minio-console/console-settings-site-replication.png
:width: 600px
:alt: MinIO Console Settings - Site Replication
:align: center
The :guilabel:`Site Replication` subsection provides an interface for adding and managing the site replication configuration for the deployment.
Configuring site replication requires that only a single site have existing buckets or objects (if any).

View File

@@ -0,0 +1,56 @@
===================================
Monitoring Bucket and Object Events
===================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
Bucket Notifications
--------------------
MinIO bucket notifications allow administrators to send notifications to supported external services on certain object or bucket events.
MinIO supports bucket and object-level S3 events similar to the
:s3-docs:`Amazon S3 Event Notifications <NotificationHowTo.html>`.
MinIO supports publishing bucket or object events to the following supported
targets on certain supported events.
- :ref:`minio-bucket-notifications-publish-amqp`
- :ref:`minio-bucket-notifications-publish-mqtt`
- :ref:`minio-bucket-notifications-publish-nats`
- :ref:`minio-bucket-notifications-publish-nsq`
- :ref:`minio-bucket-notifications-publish-elasticsearch`
- :ref:`minio-bucket-notifications-publish-kafka`
- :ref:`minio-bucket-notifications-publish-mysql`
- :ref:`minio-bucket-notifications-publish-postgresql`
- :ref:`minio-bucket-notifications-publish-redis`
- :ref:`minio-bucket-notifications-publish-webhook`
See :ref:`minio-bucket-notifications`
for more complete documentation on MinIO Bucket Notifications.
Deployment Metrics
------------------
MinIO provides a Prometheus-compatible endpoint for supporting time-series querying of metrics.
MinIO deployments :ref:`configured to enable Prometheus scraping <minio-metrics-and-alerts-endpoints>` provide a detailed metrics view through the MinIO Console.
Server Logs
-----------
MinIO provides the following interfaces for remotely reading server logs:
- The MinIO Console provides a server log viewer at <details>
- The :mc:`mc admin console` command returns the specified server's console output.
- MinIO supports pushing server logs to an HTTP webhook for further ingestion.
See :ref:`minio-logging-publish-server-logs` for more information.
.. toctree::
:titlesonly:
:hidden:
/administration/monitoring/bucket-notifications

View File

@@ -0,0 +1,184 @@
.. _minio-bucket-notifications:
====================
Bucket Notifications
====================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
MinIO bucket notifications allow administrators to send notifications to supported external services on certain object or bucket events.
MinIO supports bucket and object-level S3 events similar to the
:s3-docs:`Amazon S3 Event Notifications <NotificationHowTo.html>`.
Supported Notification Targets
------------------------------
MinIO supports publishing event notifications to the following targets:
.. list-table::
:header-rows: 1
:widths: 30 70
:width: 100%
* - Target
- Description
* - :guilabel:`AMQP` (RabbitMQ)
- Publish notifications to an AMQP service such as
`RabbitMQ <https://www.rabbitmq.com>`__.
See :ref:`minio-bucket-notifications-publish-amqp` for a tutorial.
* - :guilabel:`MQTT`
- Publish notifications to an `MQTT <https://www.mqtt.org/>`__ service.
See :ref:`minio-bucket-notifications-publish-mqtt` for a tutorial.
* - :guilabel:`NATS`
- Publish notifications to a `NATS <https://nats.io/>`__ service.
See :ref:`minio-bucket-notifications-publish-nats` for a tutorial.
* - :guilabel:`NSQ`
- Publish notifications to a `NSQ <https://nsq.io/>`__ service.
See :ref:`minio-bucket-notifications-publish-nsq` for a tutorial
* - :guilabel:`Elasticsearch`
- Publish notifications to a `Elasticsearch <https://www.elastic.co/>`__ service.
See :ref:`minio-bucket-notifications-publish-elasticsearch` for a tutorial.
* - :guilabel:`Kafka`
- Publish notifications to a `Kafka <https://kafka.apache.org/>`__ service.
See :ref:`minio-bucket-notifications-publish-kafka` for a tutorial.
* - :guilabel:`MySQL`
- Publish notifications to a `MySQL <https://www.mysql.com/>`__ service.
See :ref:`minio-bucket-notifications-publish-mysql` for a tutorial.
* - :guilabel:`PostgreSQL`
- Publish notifications to a `PostgreSQL <https://www.postgresql.org/>`__ service.
See :ref:`minio-bucket-notifications-publish-postgresql` for a tutorial.
* - :guilabel:`Redis`
- Publish notifications to a `Redis <https://redis.io/>`__ service.
See :ref:`minio-bucket-notifications-publish-redis` for a tutorial.
* - :guilabel:`webhook`
- Publish notifications to a `Webhook
<https://en.wikipedia.org/wiki/Webhook>`__ service.
See :ref:`minio-bucket-notifications-publish-webhook` for a tutorial.
.. _minio-bucket-notifications-event-types:
Supported S3 Event Types
------------------------
MinIO bucket notifications are compatible with :s3-docs:`Amazon S3 Event Notifications <NotificationHowTo.html>`.
This section lists all supported events.
Object Events
~~~~~~~~~~~~~
MinIO supports triggering notifications on the following S3 object events:
.. data:: s3:ObjectAccessed:Get
.. data:: s3:ObjectAccessed:GetLegalHold
.. data:: s3:ObjectAccessed:GetRetention
.. data:: s3:ObjectAccessed:Head
.. data:: s3:ObjectCreated:CompleteMultipartUpload
.. data:: s3:ObjectCreated:Copy
.. data:: s3:ObjectCreated:DeleteTagging
.. data:: s3:ObjectCreated:Post
.. data:: s3:ObjectCreated:Put
.. data:: s3:ObjectCreated:PutLegalHold
.. data:: s3:ObjectCreated:PutRetention
.. data:: s3:ObjectCreated:PutTagging
.. data:: s3:ObjectRemoved:Delete
.. data:: s3:ObjectRemoved:DeleteMarkerCreated
Specify the wildcard ``*`` character to select all events related to a prefix:
.. data:: s3:ObjectAccessed:*
Selects all ``s3:ObjectAccessed``\ -prefixed events.
.. data:: s3:ObjectCreated:*
Selects all ``s3:ObjectCreated``\ -prefixed events.
.. data:: s3:ObjectRemoved:*
Selects all ``s3:ObjectRemoved``\ -prefixed events.
Replication Events
~~~~~~~~~~~~~~~~~~
MinIO supports triggering notifications on the following S3 replication events:
.. data:: s3:Replication:OperationCompletedReplication
.. data:: s3:Replication:OperationFailedReplication
.. data:: s3:Replication:OperationMissedThreshold
.. data:: s3:Replication:OperationNotTracked
.. data:: s3:Replication:OperationReplicatedAfterThreshold
Specify the wildcard ``*`` character to select all ``s3:Replication`` events:
.. data:: s3:Replication:*
ILM Transition Events
~~~~~~~~~~~~~~~~~~~~~
MinIO supports triggering notifications on the following S3 ILM transition events:
.. data:: s3:ObjectRestore:Post
.. data:: s3:ObjectRestore:Completed
.. data:: s3:ObjectTransition:Failed
.. data:: s3:ObjectTransition:Complete
Specify the wildcard ``*`` character to select all events related to a prefix:
.. data:: s3:ObjectTransition:*
Selects all ``s3:ObjectTransition``\ -prefixed events.
.. data:: s3:ObjectRestore:*
Selects all ``s3:ObjectRestore``\ -prefixed events.
Global Events
~~~~~~~~~~~~~
MinIO supports triggering notifications on the following global events.
You can only listen to these events through the :legacy:`ListenNotification <golang-client-api-reference.html#ListenNotification>` API:
.. data:: s3:BucketCreated
.. data:: s3:BucketRemoved
.. todo
.. toctree::
:titlesonly:
:hidden:
/administration/monitoring/publish-events-to-amqp
/administration/monitoring/publish-events-to-mqtt
/administration/monitoring/publish-events-to-nats
/administration/monitoring/publish-events-to-nsq
/administration/monitoring/publish-events-to-elasticsearch
/administration/monitoring/publish-events-to-kafka
/administration/monitoring/publish-events-to-mysql
/administration/monitoring/publish-events-to-postgresql
/administration/monitoring/publish-events-to-redis
/administration/monitoring/publish-events-to-webhook

View File

@@ -0,0 +1,362 @@
.. _minio-bucket-notifications-publish-amqp:
=================================
Publish Events to AMQP (RabbitMQ)
=================================
.. default-domain:: minio
.. |ARN| replace:: ``arn:minio:sqs::primary:amqp``
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a `AMQP 0-9-1 <https://www.amqp.org/>`__
service endpoint such as `RabbitMQ <https://www.rabbitmq.com>`__.
MinIO relies on the :github:`streadway/amqp` project for AMQP connectivity. The
project is primarily tested against `RabbitMQ <https://www.rabbitmq.com/>`__
deployments, though other `AMQP 0-9-1-compatible <https://www.amqp.org/>`__
services *may* also work. The procedures on this page assume a RabbitMQ
deployment using the AMQP 0-9-1 protocol as the service endpoint.
Add an AMQP Endpoint to a MinIO Deployment
------------------------------------------
The following procedure adds a new AMQP service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~~
AMQP 0-9-1 Service Endpoint
+++++++++++++++++++++++++++
MinIO relies on the :github:`streadway/amqp` project for AMQP connectivity. The
project is primarily tested against `RabbitMQ <https://www.rabbitmq.com/>`__
deployments, though other `AMQP 0-9-1-compatible <https://www.amqp.org/>`__
services *may* also work. This procedure assumes a RabbitMQ deployment
using the 0-9-1 protocol as the service endpoint.
If the AMQP service requires authentication, you *must* provide an appropriate
username and password during the configuration process to grant MinIO access
to the service.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the AMQP Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new AMQP service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tab-set::
.. tab-item:: Environment Variables
MinIO supports specifying the AMQP service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-amqp>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an AMQP service endpoint. The minimum
*required* variables are
:envvar:`MINIO_NOTIFY_AMQP_ENABLE` and :envvar:`MINIO_NOTIFY_AMQP_URL`:
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_AMQP_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_AMQP_URL_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_AMQP_EXCHANGE_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_EXCHANGE_TYPE_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_ROUTING_KEY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_MANDATORY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_DURABLE_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_NO_WAIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_INTERNAL_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_AUTO_DELETED_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_DELIVERY_MODE_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
AMQP service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new AMQP service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing AMQP service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_amqp <mc admin config get>` to
review the currently configured AMQP endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the AMQP service endpoint.
For example:
``amqp://user:password@hostname:port``
See :ref:`AMQP Service for Bucket Notifications
<minio-server-envvar-bucket-notification-amqp>` for complete documentation
on each environment variable.
.. tab-item:: Configuration Settings
MinIO supports adding or updating AMQP endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_amqp` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
AMQP service endpoint. The minimum *required* setting is
:mc-conf:`notify_amqp url <notify_amqp.url>`:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_amqp:IDENTIFIER \
url="ENDPOINT" \
exchange="<string>" \
exchange_type="<string>" \
routing_key="<string>" \
mandatory="<string>" \
durable="<string>" \
no_wait="<string>" \
internal="<string>" \
auto_deleted="<string>" \
delivery_mode="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
AMQP service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing AMQP service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_amqp <mc admin config get>` to
review the currently configured AMQP endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the AMQP service endpoint.
For example:
``amqp://user:password@hostname:port``
See :ref:`AMQP Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-amqp>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured AMQP
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:amqp
You must specify the ARN resource when configuring bucket notifications with
the associated AMQP deployment as a target.
.. include:: /includes/common-bucket-notifications.rst
:start-after: start-bucket-notification-find-arn
:end-before: end-bucket-notification-find-arn
3) Configure Bucket Notifications using the AMQP Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured AMQP service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:amqp \
--event EVENTS
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:amqp
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the AMQP service for the notification data. The action required
depends on which :mc-cmd:`events <mc event add --event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an AMQP Endpoint in a MinIO Deployment
---------------------------------------------
The following procedure updates an existing AMQP service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~~
AMQP 0-9-1 Service Endpoint
+++++++++++++++++++++++++++
MinIO relies on the :github:`streadway/amqp` project for AMQP connectivity. The
project is primarily tested against `RabbitMQ <https://www.rabbitmq.com/>`__
deployments, though other `AMQP 0-9-1-compatible <https://www.amqp.org/>`__
services *may* also work. This procedure *assumes* a RabbitMQ deployment
as the service endpoint.
If the AMQP service requires authentication, you *must* provide an appropriate
username and password during the configuration process to grant MinIO access
to the service.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured AMQP Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured AMQP service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_amqp
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_amqp:primary delivery_mode="0" exchange_type="" no_wait="off" queue_dir="" queue_limit="0" url="amqp://user:password@hostname:port" auto_deleted="off" durable="off" exchange="" internal="off" mandatory="off" routing_key=""
notify_amqp:secondary delivery_mode="0" exchange_type="" no_wait="off" queue_dir="" queue_limit="0" url="amqp://user:password@hostname:port" auto_deleted="off" durable="off" exchange="" internal="off" mandatory="off" routing_key=""
The :mc-conf:`notify_amqp` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-amqp`. The
:mc-conf:`url <notify_amqp.url>` key specifies the AMQP service endpoint
for the given `notify_amqp` key. The ``notify_amqp:<IDENTIFIER>`` suffix
describes the unique identifier for that AMQP service endpoint.
Note the identifier for the AMQP service endpoint you want to update for
the next step.
2) Update the AMQP Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the AMQP service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_amqp:<IDENTIFIER> \
url="amqp://user:password@hostname:port" \
exchange="<string>" \
exchange_type="<string>" \
routing_key="<string>" \
mandatory="<string>" \
durable="<string>" \
no_wait="<string>" \
internal="<string>" \
auto_deleted="<string>" \
delivery_mode="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_amqp url <notify_amqp.url>` configuration setting is the
*minimum* required for an AMQP service endpoint. All other configuration
settings are *optional*. See :ref:`minio-server-config-bucket-notification-amqp`
for a complete list of AMQP configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured AMQP
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:amqp
4) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
AMQP service endpoint and check the AMQP service for the notification data. The
action required depends on which :mc-cmd:`events <mc event add --event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

View File

@@ -0,0 +1,344 @@
.. _minio-bucket-notifications-publish-elasticsearch:
===============================
Publish Events to Elasticsearch
===============================
.. default-domain:: minio
.. |ARN| replace:: ``arn:minio:sqs::primary:elasticsearch``
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to an
`Elasticsearch <https://www.elastic.co/>`__ service endpoint.
MinIO relies on the :github:`olivere/elastic` v7 project for Elastic
connectivity. The ``elastic/v7`` library specifically targets Elasticsearch
v7.0 and is *not compatible with earlier Elasticsearch versions*.
Add a Elasticsearch Endpoint to a MinIO Deployment
--------------------------------------------------
The following procedure adds a new Elasticsearch service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~
Elasticsearch v7.0 and later
++++++++++++++++++++++++++++
MinIO relies on the :github:`olivere/elastic` v7 project for Elastic
connectivity. The ``elastic/v7`` library specifically targets Elasticsearch
v7.0 and is *not compatible with earlier Elasticsearch versions*.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the Elasticsearch Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new Elasticsearch service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tab-set::
.. tab-item:: Environment Variables
MinIO supports specifying the Elasticsearch service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-elasticsearch>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an Elasticsearch service endpoint. The minimum
*required* variables are:
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_ENABLE`
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_URL`
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_INDEX`
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_FORMAT`
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_ELASTICSEARCH_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_ELASTICSEARCH_URL_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_ELASTICSEARCH_INDEX_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_FORMAT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_USERNAME_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_PASSWORD_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
TARGET service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing Elasticsearch
service endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_elasticsearch <mc admin config get>`
to review the currently configured Elasticsearch endpoints on the MinIO
deployment.
- Replace ``<ENDPOINT>`` with the URL of the Elasticsearch service endpoint.
For example:
See :ref:`Elasticsearch Service for Bucket Notifications
<minio-server-envvar-bucket-notification-elasticsearch>` for complete
documentation on each environment variable.
.. tab-item:: Configuration Settings
MinIO supports adding or updating Elasticsearch endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_elasticsearch` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
Elasticsearch service endpoint. The minimum *required* settings are:
- :mc-conf:`url <notify_elasticsearch.url>`
- :mc-conf:`index <notify_elasticsearch.index>`
- :mc-conf:`format <notify_elasticsearch.format>`
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_elasticsearch:IDENTIFIER \
url="ENDPOINT" \
index="<string>" \
format="<string>" \
username="<string>" \
password="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
Elasticsearch service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing Elasticsearch service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_elasticsearch <mc admin config get>`
to review the currently configured Elasticsearch endpoints on the MinIO
deployment.
- Replace ``ENDPOINT`` with the URL of the Elasticsearch service endpoint.
For example:
``https://user:password@hostname:port``
See :ref:`Elasticsearch Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-elasticsearch>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured Elasticsearch
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:elasticsearch
You must specify the ARN resource when configuring bucket notifications with
the associated Elasticsearch deployment as a target.
.. include:: /includes/common-bucket-notifications.rst
:start-after: start-bucket-notification-find-arn
:end-before: end-bucket-notification-find-arn
3) Configure Bucket Notifications using the Elasticsearch Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured Elasticsearch service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:elasticsearch \
--event EVENTS
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:elasticsearch
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the Elasticsearch service for the notification data. The action required
depends on which :mc-cmd:`events <mc event add --event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an Elasticsearch Endpoint in a MinIO Deployment
------------------------------------------------------
The following procedure updates an existing Elasticsearch service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~~
Elasticsearch v7.0 and later
++++++++++++++++++++++++++++
MinIO relies on the :github:`olivere/elastic` v7 project for Elastic
connectivity. The ``elastic/v7`` library specifically targets Elasticsearch
v7.0 and is *not compatible with earlier Elasticsearch versions*.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured Elasticsearch Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured Elasticsearch service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_elasticsearch
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_elasticsearch:primary queue_dir="" queue_limit="0" url="https://user:password@hostname:port" format="namespace" index=""
notify_elasticsearch:secondary queue_dir="" queue_limit="0" url="https://user:password@hostname:port" format="namespace" index=""
The :mc-conf:`notify_elasticsearch` key is the top-level configuration key for
an :ref:`minio-server-config-bucket-notification-elasticsearch`. The
:mc-conf:`url <notify_elasticsearch.url>` key specifies the Elasticsearch
service endpoint for the given `notify_elasticsearch` key. The
``notify_elasticsearch:<IDENTIFIER>`` suffix describes the unique identifier for
that Elasticsearch service endpoint.
Note the identifier for the Elasticsearch service endpoint you want to update for
the next step.
2) Update the Elasticsearch Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the Elasticsearch service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_elasticsearch:<IDENTIFIER> \
url="https://user:password@hostname:port" \
index="<string>" \
format="<string>" \
username="<string>" \
password="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_elasticsearch url <notify_elasticsearch.url>` configuration
setting is the *minimum* required for an Elasticsearch service endpoint. All
other configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-elasticsearch` for a complete list
of Elasticsearch configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured
Elasticsearch target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:elasticsearch
4) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
Elasticsearch service endpoint and check the Elasticsearch service for the
notification data. The action required depends on which
:mc-cmd:`events <mc event add --event>` were specified when configuring the bucket
notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

View File

@@ -0,0 +1,356 @@
.. _minio-bucket-notifications-publish-kafka:
=======================
Publish Events to Kafka
=======================
.. default-domain:: minio
.. |ARN| replace:: ``arn:minio:sqs::primary:kafka``
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a `Kafka <https://kafka.apache.org/>`__
service endpoint.
MinIO relies on the :github:`Shopify/sarama` project for Kafka connectivity
and shares that project's Kafka support. See the
``sarama`` :github:`Compatibility and API stability
<Shopify/sarama/#compatibility-and-api-stability>` section for more details.
Add a Kafka Endpoint to a MinIO Deployment
------------------------------------------
The following procedure adds a new Kafka service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~
Kafka Minimum Versions and Supported Versions
+++++++++++++++++++++++++++++++++++++++++++++
MinIO relies on the :github:`Shopify/sarama` project for Kafka connectivity
and shares that project's Kafka support. See the
``sarama`` :github:`Compatibility and API stability
<Shopify/sarama/#compatibility-and-api-stability>` section for more details.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the Kafka Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new Kafka service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tab-set::
.. tab-item:: Environment Variables
MinIO supports specifying the Kafka service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-kafka>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring a Kafka service endpoint. The minimum
*required* variables are
:envvar:`MINIO_NOTIFY_KAFKA_ENABLE` and
:envvar:`MINIO_NOTIFY_KAFKA_BROKERS`:
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_KAFKA_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_KAFKA_BROKERS_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_KAFKA_TOPIC_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_SASL_USERNAME_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_SASL_PASSWORD_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_SASL_MECHANISM_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_TLS_CLIENT_AUTH_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_SASL_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_TLS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_TLS_SKIP_VERIFY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_CLIENT_TLS_CERT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_CLIENT_TLS_KEY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_VERSION_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
Kafka service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing Kafka service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_kafka <mc admin config get>` to
review the currently configured Kafka endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with a comma-separated list of Kafka brokers.
For example:
``"kafka1.example.com:2021,kafka2.example.com:2021"``
See :ref:`Kafka Service for Bucket Notifications
<minio-server-envvar-bucket-notification-kafka>` for complete documentation
on each environment variable.
.. tab-item:: Configuration Settings
MinIO supports adding or updating Kafka endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_kafka` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
Kafka service endpoint. The minimum *required* setting is
:mc-conf:`notify_kafka brokers <notify_kafka.brokers>`:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_kafka:IDENTIFIER \
brokers="<ENDPOINT>" \
topic="<string>" \
sasl_username="<string>" \
sasl_password="<string>" \
sasl_mechanism="<string>" \
tls_client_auth="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
client_tls_cert="<string>" \
client_tls_key="<string>" \
version="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
Kafka service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing Kafka service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_kafka <mc admin config get>` to
review the currently configured Kafka endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with a comma separated list of Kafka brokers.
For example:
``"kafka1.example.com:2021,kafka2.example.com:2021"``
See :ref:`Kafka Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-kafka>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured
Kafka target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:kafka
You must specify the ARN resource when configuring bucket notifications with
the associated Kafka deployment as a target.
.. include:: /includes/common-bucket-notifications.rst
:start-after: start-bucket-notification-find-arn
:end-before: end-bucket-notification-find-arn
3) Configure Bucket Notifications using the Kafka Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured Kafka service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:kafka \
--event EVENTS
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:kafka
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the Kafka service for the notification data. The action required
depends on which :mc-cmd:`events <mc event add --event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update a Kafka Endpoint in a MinIO Deployment
---------------------------------------------
The following procedure updates an existing Kafka service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~~
Kafka Minimum Versions and Supported Versions
+++++++++++++++++++++++++++++++++++++++++++++
MinIO relies on the :github:`Shopify/sarama` project for Kafka connectivity
and shares that project's Kafka support. See the
``sarama`` :github:`Compatibility and API stability
<Shopify/sarama/#compatibility-and-api-stability>` section for more details.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured Kafka Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured Kafka service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_kafka
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_kafka:primary tls_skip_verify="off" queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" brokers="" topic="" client_tls_cert="" client_tls_key="" version=""
notify_kafka:secondary tls_skip_verify="off" queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" brokers="" topic="" client_tls_cert="" client_tls_key="" version=""
The :mc-conf:`notify_kafka` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-kafka`. The
:mc-conf:`brokers <notify_kafka.brokers>` key specifies the Kafka service
endpoint for the given `notify_kafka` key. The ``notify_kafka:<IDENTIFIER>``
suffix describes the unique identifier for that Kafka service endpoint.
Note the identifier for the Kafka service endpoint you want to update for
the next step.
2) Update the Kafka Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the Kafka service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_kafka:<IDENTIFIER> \
brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200" \
topic="<string>" \
sasl_username="<string>" \
sasl_password="<string>" \
sasl_mechanism="<string>" \
tls_client_auth="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
client_tls_cert="<string>" \
client_tls_key="<string>" \
version="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_kafka brokers <notify_kafka.brokers>` configuration setting
is the *minimum* required for a Kafka service endpoint. All other configuration
settings are *optional*. See
:ref:`minio-server-config-bucket-notification-kafka` for a complete list of
Kafka configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured
Kafka target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:kafka
4) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
Kafka service endpoint and check the Kafka service for the notification data.
The action required depends on which :mc-cmd:`events <mc event add --event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

View File

@@ -0,0 +1,369 @@
.. _minio-bucket-notifications-publish-mqtt:
======================
Publish Events to MQTT
======================
.. default-domain:: minio
.. |ARN| replace:: ``arn:minio:sqs::primary:mqtt``
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to `MQTT <https://www.mqtt.org/>`__
server/broker endpoint.
Add an MQTT Endpoint to a MinIO Deployment
------------------------------------------
The following procedure adds a new MQTT service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~~
MQTT 3.1 or 3.1.1 Server/Broker
+++++++++++++++++++++++++++++++
This procedure assumes an existing MQTT 3.1 or 3.1.1 server/broker to which the
MinIO deployment has connectivity. See the
`mqtt.org software listing <https://mqtt.org/software/>`__ for a list of
MQTT-compatible server/brokers.
If the MQTT service requires authentication, you *must* provide an appropriate
username and password during the configuration process to grant MinIO access
to the service.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the MQTT Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new MQTT service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tab-set::
.. tab-item:: Environment Variables
MinIO supports specifying the MQTT service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-mqtt>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an MQTT service endpoint. The minimum *required*
variables are:
- :envvar:`MINIO_NOTIFY_MQTT_ENABLE`
- :envvar:`MINIO_NOTIFY_MQTT_BROKER`
- :envvar:`MINIO_NOTIFY_MQTT_TOPIC`
- :envvar:`MINIO_NOTIFY_MQTT_USERNAME` *Required if the MQTT server/broker enforces authentication/authorization*
- :envvar:`MINIO_NOTIFY_MQTT_PASSWORD` *Required if the MQTT server/broker enforces authentication/authorization*
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_MQTT_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_MQTT_BROKER_<IDENTIFIER>="ENDPOINT"
set MINIO_NOTIFY_MQTT_TOPIC_<IDENTIFIER>="TOPIC"
set MINIO_NOTIFY_MQTT_USERNAME_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_PASSWORD_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_QOS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_KEEP_ALIVE_INTERVAL_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_RECONNECT_INTERVAL_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
MQTT service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new MQTT service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing MQTT service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_mqtt <mc admin config get>` to
review the currently configured MQTT endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the MQTT service endpoint.
For example:
``tcp://hostname:port``
- Replace ``TOPIC`` with the MQTT topic to which MinIO associates
events published to the server/broker.
See :ref:`MQTT Service for Bucket Notifications
<minio-server-envvar-bucket-notification-mqtt>` for complete documentation
on each environment variable.
.. tab-item:: Configuration Settings
MinIO supports adding or updating MQTT endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_mqtt` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
MQTT service endpoint. The following configuration settings are the
*minimum* required for an MQTT server/broker endpoint:
- :mc-conf:`~notify_mqtt.broker`
- :mc-conf:`~notify_mqtt.topic`
- :mc-conf:`~notify_mqtt.username` *Required if the MQTT server/broker enforces authentication/authorization*
- :mc-conf:`~notify_mqtt.password` *Required if the MQTT server/broker enforces authentication/authorization*
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_mqtt:IDENTIFIER \
broker="ENDPOINT" \
topic="TOPIC" \
username="username" \
password="password" \
qos="<integer>" \
keep_alive_interval="60s|m|h|d"
reconnect_interval="60s|m|h|d"
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
MQTT service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing MQTT service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_mqtt <mc admin config get>` to
review the currently configured MQTT endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the MQTT service endpoint.
For example:
``tcp://hostname:port``
- Replace ``TOPIC`` with the MQTT topic to which MinIO associates
events published to the server/broker.
See :ref:`MQTT Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-mqtt>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured MQTT
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:mqtt
You must specify the ARN resource when configuring bucket notifications with
the associated MQTT deployment as a target.
.. include:: /includes/common-bucket-notifications.rst
:start-after: start-bucket-notification-find-arn
:end-before: end-bucket-notification-find-arn
1) Configure Bucket Notifications using the MQTT Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured MQTT service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:mqtt \
--event EVENTS
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:MQTT
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the MQTT service for the notification data. The action required
depends on which :mc-cmd:`events <mc event add --event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an MQTT Endpoint in a MinIO Deployment
---------------------------------------------
The following procedure updates an existing MQTT service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~~
MQTT 3.1 or 3.1.1 Server/Broker Endpoint
++++++++++++++++++++++++++++++++++++++++
This procedure assumes an existing MQTT 3.1 or 3.1.1 server/broker to which the
MinIO deployment has connectivity. See the
`mqtt.org software listing <https://mqtt.org/software/>`__ for a list of
MQTT-compatible server/brokers.
If the MQTT service requires authentication, you *must* provide an appropriate
username and password during the configuration process to grant MinIO access
to the service.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured MQTT Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured MQTT service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_mqtt
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_mqtt:primary broker="tcp://mqtt-primary.example.net:port" password="" queue_dir="" queue_limit="0" reconnect_interval="0s" keep_alive_interval="0s" qos="0" topic="" username=""
notify_mqtt:secondary broker="tcp://mqtt-primary.example.net:port" password="" queue_dir="" queue_limit="0" reconnect_interval="0s" keep_alive_interval="0s" qos="0" topic="" username=""
The :mc-conf:`notify_mqtt` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-mqtt`. The
:mc-conf:`broker <notify_mqtt.broker>` key specifies the MQTT server/broker endpoint
for the given `notify_mqtt` key. The ``notify_mqtt:<IDENTIFIER>`` suffix
describes the unique identifier for that MQTT service endpoint.
Note the identifier for the MQTT service endpoint you want to update for
the next step.
2) Update the MQTT Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the MQTT service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_mqtt:<IDENTIFIER> \
url="MQTT://user:password@hostname:port" \
exchange="<string>" \
exchange_type="<string>" \
routing_key="<string>" \
mandatory="<string>" \
durable="<string>" \
no_wait="<string>" \
internal="<string>" \
auto_deleted="<string>" \
delivery_mode="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The following configuration settings are the *minimum* required for an
MQTT server/broker endpoint:
- :mc-conf:`~notify_mqtt.broker`
- :mc-conf:`~notify_mqtt.topic`
- :mc-conf:`~notify_mqtt.username` *Required if the MQTT server/broker enforces authentication/authorization*
- :mc-conf:`~notify_mqtt.password` *Required if the MQTT server/broker enforces authentication/authorization*
All other configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-mqtt` for a complete list of MQTT
configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured MQTT
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:mqtt
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
MQTT service endpoint and check the MQTT service for the notification data. The
action required depends on which :mc-cmd:`events <mc event add --event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

View File

@@ -0,0 +1,342 @@
.. _minio-bucket-notifications-publish-mysql:
=======================
Publish Events to MySQL
=======================
.. default-domain:: minio
.. |ARN| replace:: ``arn:minio:sqs::primary:mysql``
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a My`MySQL <https://www.mysql.com/>`__
service endpoint. MinIO supports MySQL 5.7.8 and later *only*.
Add a MySQL Endpoint to a MinIO Deployment
------------------------------------------
The following procedure adds a new MySQL service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~
MySQL 5.7.8 and later
+++++++++++++++++++++
MinIO relies on features introduced with MySQL 5.7.8.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the MySQL Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new MySQL service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tab-set::
.. tab-item:: Environment Variables
MinIO supports specifying the MySQL service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-mysql>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring a MySQL service endpoint. The minimum
*required* variables are:
- :envvar:`MINIO_NOTIFY_MYSQL_ENABLE`
- :envvar:`MINIO_NOTIFY_MYSQL_DSN_STRING`
- :envvar:`MINIO_NOTIFY_MYSQL_TABLE`
- :envvar:`MINIO_NOTIFY_MYSQL_FORMAT`
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_MYSQL_DSN_STRING_<IDENTIFIER>="on"
set MINIO_NOTIFY_MYSQL_TABLE_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_MYSQL_FORMAT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MYSQL_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MYSQL_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MYSQL_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
MySQL service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing MySQL service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_mysql <mc admin config get>` to
review the currently configured MySQL endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the DSN of the MySQL service endpoint.
MinIO expects the following format:
``<user>:<password>@tcp(<host>:<port>)/<database>``
For example:
``"username:password@tcp(mysql.example.com:3306)/miniodb"``
See :ref:`MySQL Service for Bucket Notifications
<minio-server-envvar-bucket-notification-mysql>` for complete documentation
on each environment variable.
.. tab-item:: Configuration Settings
MinIO supports adding or updating MySQL endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_mysql` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
MySQL service endpoint. The minimum *required* settings are:
- :mc-conf:`notify_mysql dsn_string <notify_mysql.dsn_string>`
- :mc-conf:`notify_mysql table <notify_mysql.table>`
- :mc-conf:`notify_mysql format <notify_mysql.format>`
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_mysql:IDENTIFIER \
dsn_string="<ENDPOINT>" \
table="<string>" \
format="<string>" \
max_open_connections="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
MySQL service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing MySQL service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_mysql <mc admin config get>` to
review the currently configured MySQL endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the DSN of the MySQL service endpoint.
MinIO expects the following format:
``<user>:<password>@tcp(<host>:<port>)/<database>``
For example:
``"username:password@tcp(mysql.example.com:3306)/miniodb"``
See :ref:`MySQL Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-mysql>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured MySQL
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:mysql
You must specify the ARN resource when configuring bucket notifications with
the associated MySQL deployment as a target.
.. include:: /includes/common-bucket-notifications.rst
:start-after: start-bucket-notification-find-arn
:end-before: end-bucket-notification-find-arn
3) Configure Bucket Notifications using the MySQL Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured MySQL service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:mysql \
--event EVENTS
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:mysql
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the MySQL service for the notification data. The action required
depends on which :mc-cmd:`events <mc event add --event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update a MySQL Endpoint in a MinIO Deployment
---------------------------------------------
The following procedure updates an existing MySQL service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~~
MySQL 5.7.8 and later
+++++++++++++++++++++
MinIO relies on features introduced with MySQL 5.7.8.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured MySQL Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured MySQL service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_mysql
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_mysql:primary format="namespace" table="minio_images" dsn_string="user:pass@tcp(mysql.example.com:3306)/miniodb"
notify_mysql:secondary format="namespace" table="minio_images" dsn_string="user:pass@tcp(mysql.example.com:3306)/miniodb"
The :mc-conf:`notify_mysql` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-mysql`. The
:mc-conf:`dsn_string <notify_mysql.dsn_string>` key specifies the MySQL service
endpoint for the given `notify_mysql` key. The ``notify_mysql:<IDENTIFIER>``
suffix describes the unique identifier for that MySQL service endpoint.
Note the identifier for the MySQL service endpoint you want to update for
the next step.
2) Update the MySQL Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the MySQL service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_mysql:IDENTIFIER \
dsn_string="<ENDPOINT>" \
table="<string>" \
format="<string>" \
max_open_connections="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The following configuration settings are the *minimum required* for a MySQL
service endpoint:
- :mc-conf:`notify_mysql dsn_string <notify_mysql.dsn_string>`
- :mc-conf:`notify_mysql table <notify_mysql.table>`
- :mc-conf:`notify_mysql format <notify_mysql.format>`
All other configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-mysql` for a complete list of
MySQL configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured MySQL
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:mysql
4) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
MySQL service endpoint and check the MySQL service for the notification data. The
action required depends on which :mc-cmd:`events <mc event add --event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

View File

@@ -0,0 +1,342 @@
.. _minio-bucket-notifications-publish-nats:
======================
Publish Events to NATS
======================
.. default-domain:: minio
.. |ARN| replace:: ``arn:minio:sqs::primary:nats``
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a
`NATS <https://nats.io/>`__ service endpoint.
Add a NATS Endpoint to a MinIO Deployment
-----------------------------------------
The following procedure adds a new NATS service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the NATS Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new NATS service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tab-set::
.. tab-item:: Environment Variables
MinIO supports specifying the NATS service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-nats>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an NATS service endpoint. The minimum
*required* variables are
:envvar:`MINIO_NOTIFY_NATS_ADDRESS` and
:envvar:`MINIO_NOTIFY_NATS_SUBJECT`:
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_NATS_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_NATS_ADDRESS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_SUBJECT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_USERNAME_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_PASSWORD_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_TOKEN_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_TLS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_TLS_SKIP_VERIFY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_PING_INTERVAL_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_STREAMING_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_STREAMING_ASYNC_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_STREAMING_MAX_PUB_ACKS_IN_FLIGHT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_STREAMING_CLUSTER_ID_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_CERT_AUTHORITY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_CLIENT_CERT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_CLIENT_KEY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
NATS service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing NATS service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_nats <mc admin config get>` to
review the currently configured NATS endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the NATS service endpoint.
For example: ``htpps://nats-endpoint.example.com:4222``
See :ref:`NATS Service for Bucket Notifications
<minio-server-envvar-bucket-notification-nats>` for complete documentation
on each environment variable.
.. tab-item:: Configuration Settings
MinIO supports adding or updating NATS endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_nats` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
NATS service endpoint. The minimum *required* setting are
:mc-conf:`notify_nats address <notify_nats.address>` and
:mc-conf:`notify_nats subject <notify_nats.subject>`:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_nats:IDENTIFIER \
address="HOSTNAME" \
subject="<string>" \
username="<string>" \
password="<string>" \
token="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
ping_interval="<string>" \
streaming="<string>" \
streaming_async="<string>" \
streaming_max_pub_acks_in_flight="<string>" \
streaming_cluster_id="<string>" \
cert_authority="<string>" \
client_cert="<string>" \
client_key="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
NATS service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing NATS service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_nats <mc admin config get>` to
review the currently configured NATS endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the NATS service endpoint.
For example: ``htpps://nats-endpoint.example.com:4222``.
See :ref:`NATS Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-nats>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured NATS
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:nats
You must specify the ARN resource when configuring bucket notifications with
the associated NATS deployment as a target.
.. include:: /includes/common-bucket-notifications.rst
:start-after: start-bucket-notification-find-arn
:end-before: end-bucket-notification-find-arn
3) Configure Bucket Notifications using the NATS Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured NATS service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:nats \
--event EVENTS
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:nats
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the NATS service for the notification data. The action required
depends on which :mc-cmd:`events <mc event add --event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an NATS Endpoint in a MinIO Deployment
---------------------------------------------
The following procedure updates an existing NATS service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured NATS Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured NATS service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_nats
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_nats:primary password="yoursecret" streaming_max_pub_acks_in_flight="10" subject="" address="nats-endpoint.example.com:4222" token="" username="yourusername" ping_interval="0" queue_limit="0" tls="off" tls_skip_verify="off" streaming_async="on" queue_dir="" streaming_cluster_id="test-cluster" streaming_enable="on"
notify_nats:secondary password="yoursecret" streaming_max_pub_acks_in_flight="10" subject="" address="nats-endpoint.example.com:4222" token="" username="yourusername" ping_interval="0" queue_limit="0" tls="off" tls_skip_verify="off" streaming_async="on" queue_dir="" streaming_cluster_id="test-cluster" streaming_enable="on"
The :mc-conf:`notify_nats` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-nats`. The
:mc-conf:`address <notify_nats.address>` key specifies the NATS service endpoint
for the given ``notify_nats`` key. The ``notify_nats:<IDENTIFIER>`` suffix
describes the unique identifier for that NATS service endpoint.
Note the identifier for the NATS service endpoint you want to update for
the next step.
2) Update the NATS Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the NATS service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_nats:IDENTIFIER \
address="HOSTNAME" \
subject="<string>" \
username="<string>" \
password="<string>" \
token="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
ping_interval="<string>" \
streaming="<string>" \
streaming_async="<string>" \
streaming_max_pub_acks_in_flight="<string>" \
streaming_cluster_id="<string>" \
cert_authority="<string>" \
client_cert="<string>" \
client_key="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_nats address <notify_nats.address>` configuration setting
is the *minimum* required for an NATS service endpoint. All other configuration
settings are *optional*. See :ref:`minio-server-config-bucket-notification-nats`
for a complete list of NATS configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured NATS
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:nats
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
NATS service endpoint and check the NATS service for the notification data. The
action required depends on which :mc-cmd:`events <mc event add --event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

View File

@@ -0,0 +1,288 @@
.. _minio-bucket-notifications-publish-nsq:
=====================
Publish Events to NSQ
=====================
.. default-domain:: minio
.. |ARN| replace:: ``arn:minio:sqs::primary:nsq``
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to `NSQ <https://nsq.io/>`__
service endpoint.
Add a NSQ Endpoint to a MinIO Deployment
----------------------------------------
The following procedure adds a new NSQ service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the NSQ Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new NSQ service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tab-set::
.. tab-item:: Environment Variables
MinIO supports specifying the NSQ service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-nsq>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an NSQ service endpoint. The minimum
*required* variables are
:envvar:`MINIO_NOTIFY_NSQ_NSQD_ADDRESS` and
:envvar:`MINIO_NOTIFY_NSQ_TOPIC`:
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_NSQ_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_NSQ_NSQD_ADDRESS_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_NSQ_TOPIC_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NSQ_TLS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NSQ_TLS_SKIP_VERIFY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NSQ_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NSQ_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NSQ_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
TARGET service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing NSQ service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_nsq <mc admin config get>` to
review the currently configured NSQ endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the NSQ service endpoint.
For example, ``https://nsq-service.example.com:4150``.
See :ref:`NSQ Service for Bucket Notifications
<minio-server-envvar-bucket-notification-nsq>` for complete documentation
on each environment variable.
.. tab-item:: Configuration Settings
MinIO supports adding or updating NSQ endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_nsq` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
NSQ service endpoint. The minimum *required* setting is
:mc-conf:`notify_nsq nsqd_address <notify_nsq.nsqd_address>` and
:mc-conf:`notify_nsq topic <notify_nsq.topic>`:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_nsq:IDENTIFIER \
nsqd_address="ENDPOINT" \
topic="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
NSQ service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing NSQ service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_nsq <mc admin config get>` to
review the currently configured NSQ endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the NSQ service endpoint.
For example:
``NSQ://user:password@hostname:port``
See :ref:`NSQ Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-nsq>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured NSQ
target similar to the following:
.. code-block:: shell
SQS ARNs: |ARN|
You must specify the ARN resource when configuring bucket notifications with the associated NSQ deployment as a target.
.. include:: /includes/common-bucket-notifications.rst
:start-after: start-bucket-notification-find-arn
:end-before: end-bucket-notification-find-arn
3) Configure Bucket Notifications using the NSQ Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification event with the configured NSQ service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:nsq \
--event EVENTS
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:nsq
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and check the NSQ service for the notification data.
The action required depends on which :mc-cmd:`events <mc event add --event>` were specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the ``s3:ObjectCreated:Put`` event, you can use the :mc-cmd:`mc cp` command to create a new object in the bucket and trigger a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an NSQ Endpoint in a MinIO Deployment
--------------------------------------------
The following procedure updates an existing NSQ service endpoint for supporting :ref:`bucket notifications <minio-bucket-notifications>` in a MinIO deployment.
Prerequisites
~~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured NSQ Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently configured NSQ service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_nsq
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_nsq:primary nsqd_address="https://nsq.example.com" queue_dir="" queue_limit="0" tls="off" tls_skip_verify="off" topic=""
notify_nsq:secondary nsqd_address="https://nsq.example.com" queue_dir="" queue_limit="0" tls="off" tls_skip_verify="off" topic=""
The :mc-conf:`notify_nsq` key is the top-level configuration key for an :ref:`minio-server-config-bucket-notification-nsq`.
The :mc-conf:`nsqd_address <notify_nsq.nsqd_address>` key specifies the NSQ service endpoint for the given `notify_nsq` key.
The ``notify_nsq:<IDENTIFIER>`` suffix describes the unique identifier for that NSQ service endpoint.
Note the identifier for the NSQ service endpoint you want to update for the next step.
2) Update the NSQ Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration for the NSQ service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_nsq:<IDENTIFIER> \
nsqd_address="NSQ://user:password@hostname:port" \
topic="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_nsq nsqd_address <notify_nsq.nsqd_address>` configuration setting is the *minimum* required for an NSQ service endpoint.
All other configuration settings are *optional*.
See :ref:`minio-server-config-bucket-notification-nsq` for a complete list of NSQ configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to restart.
The :mc:`minio server` process prints a line on startup for each configured NSQ target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:NSQ
4) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated NSQ service endpoint and check the NSQ service for the notification data.
The action required depends on which :mc-cmd:`events <mc event add --event>` were specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the ``s3:ObjectCreated:Put`` event, you can use the :mc-cmd:`mc cp` command to create a new object in the bucket and trigger a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

View File

@@ -0,0 +1,346 @@
.. _minio-bucket-notifications-publish-postgresql:
============================
Publish Events to PostgreSQL
============================
.. default-domain:: minio
.. |ARN| replace:: ``arn:minio:sqs::primary:postgresql``
.. contents:: Table of Contents
:local:
:depth: 1
.. |postgresql-uri-reference| replace:: `PostgreSQL Connection String <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING>`__
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to
`PostgreSQL <https://www.postgresql.org/>`__. MinIO supports
PostgreSQL 9.5 and later *only*.
Add a PostgreSQL Endpoint to a MinIO Deployment
-----------------------------------------------
The following procedure adds a new PostgreSQL service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~
PostgreSQL 9.5 and later
++++++++++++++++++++++++
MinIO relies on features introduced with PostgreSQL 9.5.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the PostgreSQL Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new PostgreSQL service endpoint using either environment
variables *or* by setting runtime configuration settings.
.. tab-set::
.. tab-item:: Environment Variables
MinIO supports specifying the PostgreSQL service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-postgresql>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring a PostgreSQL service endpoint. The minimum
*required* variables are:
- :envvar:`MINIO_NOTIFY_POSTGRESQL_CONNECTION_STRING`
- :envvar:`MINIO_NOTIFY_POSTGRESQL_TABLE`
- :envvar:`MINIO_NOTIFY_POSTGRESQL_FORMAT`
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_CONNECTION_STRING_<IDENTIFIER>="on"
set MINIO_NOTIFY_TABLE_<IDENTIFIER>="ENDPOINT"
set MINIO_NOTIFY_FORMAT_<IDENTIFIER>=""
set MINIO_NOTIFY_MAX_OPEN_CONNECTIONS_<IDENTIFIER>=""
set MINIO_NOTIFY_QUEUE_DIR_<IDENTIFIER>=""
set MINIO_NOTIFY_QUEUE_LIMIT_<IDENTIFIER>=""
set MINIO_NOTIFY_COMMENT_<IDENTIFIER>=""
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
PostgreSQL service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing PostgreSQL service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_postgresql <mc admin config get>` to
review the currently configured PostgreSQL endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the |postgresql-uri-reference|
for PostgreSQL service endpoint. MinIO supports ``key=value`` format for
the connection string. For example:
``"host=https://postgresql.example.com port=5432 ..."``
For more complete documentation on supported PostgreSQL connection
string parameters, see |postgresql-uri-reference|.
See :ref:`PostgreSQL Service for Bucket Notifications
<minio-server-envvar-bucket-notification-postgresql>` for complete
documentation on each environment variable.
.. tab-item:: Configuration Settings
MinIO supports adding or updating PostgreSQL endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_postgresql` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
PostgreSQL service endpoint. The minimum *required* setting are:
- :mc-conf:`notify_postgresql connection_string
<notify_postgresql.connection_string>`
- :mc-conf:`notify_postgresql table <notify_postgresql.table>`
- :mc-conf:`notify_postgresql format <notify_postgresql.format>`
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_postgresql:IDENTIFIER \
connection_string="ENDPOINT" \
table="<string>" \
format="<string>" \
max_open_connections="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
PostgreSQL service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing PostgreSQL service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_postgresql <mc admin config get>` to
review the currently configured PostgreSQL endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the `PostgreSQL URI connection string
<https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING>`__
of the PostgreSQL service endpoint. MinIO supports ``key=value`` format
for the PostgreSQL connection string. For example:
``"host=https://postgresql.example.com port=5432 ..."``
For more complete documentation on supported PostgreSQL connection
string parameters, see |postgresql-uri-reference|.
See :ref:`PostgreSQL Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-postgresql>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured PostgreSQL
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:postgresql
You must specify the ARN resource when configuring bucket notifications with
the associated PostgreSQL deployment as a target.
.. include:: /includes/common-bucket-notifications.rst
:start-after: start-bucket-notification-find-arn
:end-before: end-bucket-notification-find-arn
3) Configure Bucket Notifications using the PostgreSQL Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured PostgreSQL service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:postgresql \
--event EVENTS
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the ßevent.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:postgresql
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the PostgreSQL service for the notification data. The action required
depends on which :mc-cmd:`events <mc event add --event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update a PostgreSQL Endpoint in a MinIO Deployment
---------------------------------------------------
The following procedure updates an existing PostgreSQL service endpoint for
supporting :ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~
PostgreSQL 9.5 and later
++++++++++++++++++++++++
MinIO relies on features introduced with PostgreSQL 9.5.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured PostgreSQL Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured PostgreSQL service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_postgresql
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_postgresql:primary queue_dir="" connection_string="postgresql://" queue_limit="0" table="" format="namespace"
notify_postgresql:secondary queue_dir="" connection_string="" queue_limit="0" table="" format="namespace"
The :mc-conf:`notify_postgresql` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-postgresql`. The
:mc-conf:`connection_string <notify_postgresql.connection_string>` key specifies
the PostgreSQL service endpoint for the given `notify_postgresql` key. The
``notify_postgresql:<IDENTIFIER>`` suffix describes the unique identifier for
that PostgreSQL service endpoint.
Note the identifier for the PostgreSQL service endpoint you want to update for
the next step.
2) Update the PostgreSQL Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the PostgreSQL service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_postgresql:IDENTIFIER \
connection_string="ENDPOINT" \
table="<string>" \
format="<string>" \
max_open_connections="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The following configuration settings are the *minimum* required for a
PostgreSQL service endpoint:
- :mc-conf:`notify_postgresql connection_string
<notify_postgresql.connection_string>`
- :mc-conf:`notify_postgresql table <notify_postgresql.table>`
- :mc-conf:`notify_postgresql format <notify_postgresql.format>`
All other configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-postgresql` for a complete list of
PostgreSQL configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured PostgreSQL
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:postgresql
4) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
PostgreSQL service endpoint and check the PostgreSQL service for the notification data. The
action required depends on which :mc-cmd:`events <mc event add --event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

View File

@@ -0,0 +1,315 @@
.. _minio-bucket-notifications-publish-redis:
=======================
Publish Events to Redis
=======================
.. default-domain:: minio
.. |ARN| replace:: ``arn:minio:sqs::primary:redis``
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a `Redis <https://redis.io/>`__
service endpoint.
Add a Redis Endpoint to a MinIO Deployment
-------------------------------------------
The following procedure adds a new Redis service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the Redis Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new Redis service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tab-set::
.. tab-item:: Environment Variables
MinIO supports specifying the Redis service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-redis>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an Redis service endpoint. The minimum
*required* variables are:
- :envvar:`MINIO_NOTIFY_REDIS_ENABLE`
- :envvar:`MINIO_NOTIFY_REDIS_ADDRESS`
- :envvar:`MINIO_NOTIFY_REDIS_KEY`
- :envvar:`MINIO_NOTIFY_REDIS_FORMAT`
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_REDIS_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_REDIS_ADDRESS_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_REDIS_KEY_<IDENTIFIER>="<STRING>"
set MINIO_NOTIFY_REDIS_FORMAT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_REDIS_PASSWORD_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_REDIS_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_REDIS_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_REDIS_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
TARGET service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing Redis service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_redis <mc admin config get>` to
review the currently configured Redis endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the Redis service endpoint.
For example: ``https://redis.example.com:6369``
See :ref:`Redis Service for Bucket Notifications
<minio-server-envvar-bucket-notification-redis>` for complete documentation
on each environment variable.
.. tab-item:: Configuration Settings
MinIO supports adding or updating Redis endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_redis` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
Redis service endpoint. The minimum *required* settings are:
- :mc-conf:`notify_redis address <notify_redis.address>`
- :mc-conf:`notify_redis key <notify_redis.key>`
- :mc-conf:`notify_redis format <notify_redis.format>`
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_redis:IDENTIFIER \
address="ENDPOINT" \
key="<string>" \
format="<string>" \
password="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
Redis service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing Redis service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_redis <mc admin config get>` to
review the currently configured Redis endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the Redis service endpoint.
For example: ``https://redis.example.com:6369``
See :ref:`Redis Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-redis>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured Redis
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:redis
You must specify the ARN resource when configuring bucket notifications with
the associated Redis deployment as a target.
.. include:: /includes/common-bucket-notifications.rst
:start-after: start-bucket-notification-find-arn
:end-before: end-bucket-notification-find-arn
3) Configure Bucket Notifications using the Redis Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured Redis service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:redis \
--event EVENTS
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:redis
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the Redis service for the notification data. The action required
depends on which :mc-cmd:`events <mc event add --event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an Redis Endpoint in a MinIO Deployment
----------------------------------------------
The following procedure updates an existing Redis service endpoint for
supporting :ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured Redis Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured Redis service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_redis
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_redis:primary address="https://redis.example.com:6369" format="namespace" key="minioevent" password="" queue_dir="" queue_limit="0"
notify_redis:secondary address="https://redis.example.com:6369" format="namespace" key="minioevent" password="" queue_dir="" queue_limit="0"
The :mc-conf:`notify_redis` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-redis`. The
:mc-conf:`address <notify_redis.address>` key specifies the Redis service endpoint
for the given `notify_redis` key. The ``notify_redis:<IDENTIFIER>`` suffix
describes the unique identifier for that Redis service endpoint.
Note the identifier for the Redis service endpoint you want to update for
the next step.
2) Update the Redis Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the Redis service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_redis:IDENTIFIER \
address="ENDPOINT" \
key="<string>" \
format="<string>" \
password="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_redis address <notify_redis.address>` configuration setting
is the *minimum* required for an Redis service endpoint. All other configuration
settings are *optional*. See
:ref:`minio-server-config-bucket-notification-redis` for a complete list of
Redis configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured Redis
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:redis
4) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
Redis service endpoint and check the Redis service for the notification data. The
action required depends on which :mc-cmd:`events <mc event add --event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

View File

@@ -0,0 +1,313 @@
.. _minio-bucket-notifications-publish-webhook:
=========================
Publish Events to Webhook
=========================
.. default-domain:: minio
.. |ARN| replace:: ``arn:minio:sqs::primary:webhook``
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a
`Webhook <https://en.wikipedia.org/wiki/Webhook>`__ service endpoint.
Add a Webhook Endpoint to a MinIO Deployment
--------------------------------------------
The following procedure adds a new Webhook service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the Webhook Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new Webhook service endpoint using either environment
variables *or* by setting runtime configuration settings.
.. tab-set::
.. tab-item:: Environment Variables
MinIO supports specifying the Webhook service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-webhook>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an Webhook service endpoint. The minimum
*required* variables are
:envvar:`MINIO_NOTIFY_WEBHOOK_ENABLE` and
:envvar:`MINIO_NOTIFY_WEBHOOK_ENDPOINT`:
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_WEBHOOK_ENABLE_<IDENTIFIER>_="on"
set MINIO_NOTIFY_WEBHOOK_ENDPOINT_<IDENTIFIER>_="ENDPOINT"
set MINIO_NOTIFY_WEBHOOK_AUTH_TOKEN_<IDENTIFIER>_="<string>"
set MINIO_NOTIFY_WEBHOOK_QUEUE_DIR_<IDENTIFIER>_="<string>"
set MINIO_NOTIFY_WEBHOOK_QUEUE_LIMIT_<IDENTIFIER>_="<string>"
set MINIO_NOTIFY_WEBHOOK_CLIENT_CERT_<IDENTIFIER>_="<string>"
set MINIO_NOTIFY_WEBHOOK_CLIENT_KEY_<IDENTIFIER>_="<string>"
set MINIO_NOTIFY_WEBHOOK_COMMENT_<IDENTIFIER>_="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
Webhook service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing Webhook service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_webhook <mc admin config get>` to
review the currently configured Webhook endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the Webhook service endpoint.
For example:
``https://webhook.example.com``
See :ref:`Webhook Service for Bucket Notifications
<minio-server-envvar-bucket-notification-webhook>` for complete documentation
on each environment variable.
.. tab-item:: Configuration Settings
MinIO supports adding or updating Webhook endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_webhook` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
Webhook service endpoint. The minimum *required* setting is
:mc-conf:`notify_webhook endpoint <notify_webhook.endpoint>`:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_webhook:IDENTIFIER \
endpoint="<ENDPOINT>" \
auth_token="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
client_cert="<string>" \
client_key="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
Webhook service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing Webhook service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_webhook <mc admin config get>` to
review the currently configured Webhook endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the Webhook service endpoint.
For example:
``https://webhook.example.com``
See :ref:`Webhook Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-webhook>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured
Webhook target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:webhook
You must specify the ARN resource when configuring bucket notifications with
the associated Webhook deployment as a target.
.. include:: /includes/common-bucket-notifications.rst
:start-after: start-bucket-notification-find-arn
:end-before: end-bucket-notification-find-arn
3) Configure Bucket Notifications using the Webhook Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured Webhook service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:webhook \
--event EVENTS
- Replace ``ALIAS`` with the :ref:`alias <alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:webhook
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the Webhook service for the notification data. The action required
depends on which :mc-cmd:`events <mc event add --event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an Webhook Endpoint in a MinIO Deployment
------------------------------------------------
The following procedure updates an existing Webhook service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
Prerequisites
~~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured Webhook Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured Webhook service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_webhook
Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_webhook:primary endpoint="https://webhook.example.com" auth_token="" queue_limit="0" queue_dir="" client_cert="" client_key=""
notify_webhook:secondary endpoint="https://webhook.example.com" auth_token="" queue_limit="0" queue_dir="" client_cert="" client_key=""
The :mc-conf:`notify_webhook` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-webhook`. The
:mc-conf:`endpoint <notify_webhook.endpoint>` key specifies the Webhook service
endpoint for the given `notify_webhook` key. The ``notify_webhook:<IDENTIFIER>``
suffix describes the unique identifier for that Webhook service endpoint.
Note the identifier for the Webhook service endpoint you want to update for
the next step.
2) Update the Webhook Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the Webhook service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_webhook:IDENTIFIER \
endpoint="<ENDPOINT>" \
auth_token="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
client_cert="<string>" \
client_key="<string>" \
comment="<string>"
The :mc-conf:`notify_webhook endpoint <notify_webhook.endpoint>` configuration
setting is the *minimum* required for an Webhook service endpoint. All other
configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-webhook` for a complete list of
Webhook configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured Webhook
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:webhook
4) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
Webhook service endpoint and check the Webhook service for the notification
data. The action required depends on which :mc-cmd:`events <mc event add --event>`
were specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

View File

@@ -0,0 +1,102 @@
=================
Object Management
=================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
.. _objects:
An :ref:`object <objects>` is binary data, sometimes referred to as a Binary
Large OBject (BLOB). Blobs can be images, audio files, spreadsheets, or even
binary executable code. Object Storage platforms like MinIO provide dedicated
tools and capabilities for storing, retrieving, and searching for blobs.
.. _buckets:
MinIO Object Storage uses :ref:`buckets <buckets>` to organize objects.
A bucket is similar to a folder or directory in a filesystem, where each
bucket can hold an arbitrary number of objects. MinIO buckets provide the
same functionality as AWS S3 buckets.
For example, consider an application that hosts a web blog. The application
needs to store a variety of blobs, including rich multimedia like videos and
images. The structure of objects on the MinIO server might look similar to the
following:
.. code-block:: text
/ #root
/images/
2020-01-02-MinIO-Diagram.png
2020-01-03-MinIO-Advanced-Deployment.png
MinIO-Logo.png
/videos/
2020-01-04-MinIO-Interview.mp4
/articles/
/john.doe/
2020-01-02-MinIO-Object-Storage.md
2020-01-02-MinIO-Object-Storage-comments.json
/jane.doe/
2020-01-03-MinIO-Advanced-Deployment.png
2020-01-02-MinIO-Advanced-Deployment-comments.json
2020-01-04-MinIO-Interview.md
MinIO supports multiple levels of nested directories and objects to support
even the most dynamic object storage workloads.
Object Versioning
-----------------
MinIO supports keeping multiple "versions" of an object in a single bucket.
Write operations which would normally overwrite an existing object instead result in the creation of a new versioned object.
.. image:: /images/retention/minio-versioning-multiple-versions.svg
:alt: Object with Multiple Versions
:align: center
MinIO versioning protects from unintended overwrites and deletions while providing support for "undoing" a write operation.
See :ref:`minio-bucket-versioning` for more complete documentation.
Object Retention
----------------
MinIO Object Locking ("Object Retention") enforces Write-Once Read-Many (WORM) immutability to protect :ref:`versioned objects <minio-bucket-versioning>` from deletion.
MinIO supports both :ref:`duration based object retention <minio-object-locking-retention-modes>` and :ref:`indefinite Legal Hold retention <minio-object-locking-legalhold>`.
.. image:: /images/retention/minio-object-locking.svg
:alt: 30 Day Locked Objects
:align: center
:width: 600px
See :ref:`minio-object-locking` for more complete documentation.
MinIO Object Locking provides key data retention compliance and meets SEC17a-4(f), FINRA 4511(C), and CFTC 1.31(c)-(d) requirements as per `Cohasset Associates <https://min.io/cohasset?ref=docs>`__.
Object Lifecycle Management
---------------------------
MinIO Object Lifecycle Management allows creating rules for time or date
based automatic transition or expiry of objects. For object transition,
MinIO automatically moves the object to a configured remote storage
tier. For object expiry, MinIO automatically deletes the object.
MinIO lifecycle management is built for behavior and syntax compatibility with
:s3-docs:`AWS S3 Lifecycle Management <object-lifecycle-mgmt.html>`. For
example, you can export S3 lifecycle management rules and import them into
MinIO or vice-versa. MinIO uses JSON to describe lifecycle management rules,
and conversion to or from XML may be required.
See :ref:`minio-lifecycle-management` for more complete documentation.
.. toctree::
:titlesonly:
:hidden:
/administration/object-management/object-versioning
/administration/object-management/object-retention
/administration/object-management/object-lifecycle-management

View File

@@ -0,0 +1,112 @@
.. _minio-lifecycle-management-create-expiry-rule:
===========================
Automatic Object Expiration
===========================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
Each procedure on this page creates a new object lifecycle management rule that
expires objects on a MinIO bucket. This procedure supports use cases like
removing "old" objects after a certain time period or calendar date.
.. todo: diagram
Requirements
------------
Install and Configure ``mc``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses :mc:`mc` for performing operations on the MinIO cluster.
Install :mc:`mc` on a machine with network access to both source and destination
clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
instructions on downloading and installing ``mc``.
Use the :mc:`mc alias` command to create an alias for the source MinIO cluster
and the destination S3-compatible service. Alias creation requires specifying an
access key for a user on the source and destination clusters. The specified
users must have :ref:`permissions
<minio-lifecycle-management-create-expiry-rule-permissions>` for configuring
and applying expiry operations.
.. _minio-lifecycle-management-create-expiry-rule-permissions:
Required Permissions
~~~~~~~~~~~~~~~~~~~~
MinIO requires the following permissions scoped to the bucket or buckets
for which you are creating lifecycle management rules.
- :policy-action:`s3:PutLifecycleConfiguration`
- :policy-action:`s3:GetLifecycleConfiguration`
MinIO also requires the following administrative permissions on the cluster
in which you are creating remote tiers for object transition lifecycle
management rules:
- :policy-action:`admin:SetTier`
- :policy-action:`admin:ListTier`
For example, the following policy provides permission for configuring object
transition lifecycle management rules on any bucket in the cluster:.
.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json
:language: json
:class: copyable
Expire Objects after Number of Days
-----------------------------------
Use :mc-cmd:`mc ilm add` with :mc-cmd:`~mc ilm add --expiry-days` to
expire bucket contents a number of days after object creation:
.. code-block:: shell
:class: copyable
mc ilm add ALIAS/PATH --expiry-days "DAYS"
- Replace :mc-cmd:`ALIAS <mc ilm add ALIAS>` with the
:mc:`alias <mc alias>` of the S3-compatible host.
- Replace :mc-cmd:`PATH <mc ilm add ALIAS>` with the path to the bucket on the
S3-compatible host.
- Replace :mc-cmd:`DAYS <mc ilm add --expiry-days>` with the number of days after
which to expire the object. For example, specify ``30`` to expire the
object 30 days after creation.
Expire Versioned Objects
------------------------
Use :mc-cmd:`mc ilm add` to expiring noncurrent object versions and object
delete markers:
- To expire noncurrent object versions after a specific duration in days,
include :mc-cmd:`~mc ilm add --noncurrentversion-expiration-days`.
- To expire delete markers for objects with no remaining versions,
include :mc-cmd:`~mc ilm add --expired-object-delete-marker`.
.. code-block:: shell
:class: copyable
mc ilm add ALIAS/PATH \
--noncurrentversion-expiration-days NONCURRENT_DAYS \
--expired-object-delete-marker
- Replace :mc-cmd:`ALIAS <mc ilm add ALIAS>` with the
:mc:`alias <mc alias>` of the S3-compatible host.
- Replace :mc-cmd:`PATH <mc ilm add ALIAS>` with the path to the bucket on the
S3-compatible host.
- Replace :mc-cmd:`NONCURRENT_DAYS
<mc ilm add --noncurrentversion-expiration-days>` with the number of days after
which to expire noncurrent object versions. For example, specify ``30d`` to
expire a version after it has been noncurrent for at least 30 days.

View File

@@ -0,0 +1,144 @@
.. _minio-lifecycle-management:
===========================
Object Lifecycle Management
===========================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO Object Lifecycle Management allows creating rules for time or date
based automatic transition or expiry of objects. For object transition,
MinIO automatically moves the object to a configured remote storage
tier. For object expiry, MinIO automatically deletes the object.
MinIO lifecycle management is built for behavior and syntax compatibility with
:s3-docs:`AWS S3 Lifecycle Management <object-lifecycle-mgmt.html>`. For
example, you can export S3 lifecycle management rules and import them into
MinIO or vice-versa. MinIO uses JSON to describe lifecycle management rules,
and conversion to or from XML may be required.
.. _minio-lifecycle-management-tiering:
Object Transition ("Tiering")
-----------------------------
MinIO supports creating object transition lifecycle management rules, where
MinIO can automatically move an object to a remote storage "tier". MinIO
supports any S3-compatible service as a remote tier *in addition to* the
following public cloud storage services:
- :ref:`Amazon S3 <minio-lifecycle-management-transition-to-s3>`
- :ref:`Google Cloud Storage <minio-lifecycle-management-transition-to-gcs>`
- :ref:`Microsoft Azure Blob Storage
<minio-lifecycle-management-transition-to-azure>`
MinIO object transition supports use cases like moving aged data from MinIO
clusters in private or public cloud infrastructure to low-cost private or public
cloud storage solutions. MinIO manages retrieving tiered objects on-the-fly
without any additional application-side logic.
Use the :mc-cmd:`mc admin tier` command to create a remote target for tiering
data to a supported Cloud Service Provider object storage. You can then use the
:mc-cmd:`mc ilm add --transition-days` command to transition objects to the
remote tier after a specified number of calendar days.
Exclusive Access to Remote Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-transition-bucket-access-desc
:end-before: end-transition-bucket-access-desc
Availability of Remote Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-transition-data-loss-desc
:end-before: end-transition-data-loss-desc
Versioned Buckets
~~~~~~~~~~~~~~~~~
MinIO adopts :s3-docs:`S3 behavior
<intro-lifecycle-rules.html#intro-lifecycle-rules-actions>` for transition rules
on :ref:`versioned buckets <minio-bucket-versioning>`. Specifically, MinIO by
default applies the transition operation to the *current* object version.
To transition noncurrent object versions, specify the
:mc-cmd:`~mc ilm add --noncurrentversion-transition-days` and
:mc-cmd:`~mc ilm add --noncurrentversion-transition-storage-class` options
when creating the transition rule.
.. _minio-lifecycle-management-expiration:
Object Expiration
-----------------
MinIO lifecycle management supports expiring objects on a bucket. Object
"expiration" involves performing a ``DELETE`` operation on the object. For
example, you can create a lifecycle management rule to expire any object
older than 365 days.
.. todo: Diagram of MinIO Expiration
Use :mc-cmd:`mc ilm add --expiry-days` to expire objects after a specified
number of calendar days.
For buckets with :ref:`replication <minio-bucket-replication>` configured, MinIO
does not replicate objects deleted by a lifecycle management expiration rule.
See :ref:`minio-replication-behavior-delete` for more information.
Versioned Buckets
~~~~~~~~~~~~~~~~~
MinIO adopts :s3-docs:`S3 behavior
<intro-lifecycle-rules.html#intro-lifecycle-rules-actions>` for expiration rules
on :ref:`versioned buckets <minio-bucket-versioning>`. MinIO has two
specific default behaviors for versioned buckets:
- MinIO applies the expiration option to only the *current* object version by
creating a ``DeleteMarker`` as is normal with versioned delete.
To expire noncurrent object versions, specify the
:mc-cmd:`~mc ilm add --noncurrentversion-expiration-days` option
when creating the expiration rule.
- MinIO does not expire ``DeleteMarkers`` *even if* no other versions of
that object exist.
To expire delete markers when there are no remaining versions for that
object, specify the :mc-cmd:`~mc ilm add --expired-object-delete-marker`
option when creating the expiration rule.
.. _minio-lifecycle-management-scanner:
Lifecycle Management Object Scanner
-----------------------------------
MinIO uses a built-in scanner to actively check objects against all
configured lifecycle management rules. The scanner is a low-priority process
that yields to high IO workloads to prevent performance spikes triggered
by rule timing. The scanner may therefore not detect an object as eligible
for a configured transition or expiration lifecycle rule until *after*
the lifecycle rule period has passed.
Delayed application of lifecycle management rules is typically associated to
limited node resources and cluster size. Scanner speed tends to slow as
clusters grow as more time is required to visit all buckets and objects.
This can be exacerbated if the cluster hardware is undersized for regular
workloads, as the scanner will yield to high cluster load to avoid performance
loss. Consider regularly checking cluster metrics, capacity, and resource
usage to ensure the cluster hardware is scaling alongside cluster and workload
growth.
.. toctree::
:hidden:
/administration/object-management/transition-objects-to-s3.rst
/administration/object-management/transition-objects-to-gcs.rst
/administration/object-management/transition-objects-to-azure.rst
/administration/object-management/create-lifecycle-management-expiration-rule.rst

View File

@@ -0,0 +1,433 @@
.. _minio-object-locking:
.. _minio-object-retention:
====================
MinIO Object Locking
====================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
Overview
--------
MinIO Object Locking ("Object Retention") enforces Write-Once Read-Many (WORM)
immutability to protect :ref:`versioned objects <minio-bucket-versioning>` from
deletion. MinIO supports both
:ref:`duration based object retention <minio-object-locking-retention-modes>`
and
:ref:`indefinite Legal Hold retention <minio-object-locking-legalhold>`.
MinIO Object Locking provides key data retention compliance and meets
SEC17a-4(f), FINRA 4511(C), and CFTC 1.31(c)-(d) requirements as per
`Cohasset Associates <https://min.io/cohasset?ref-docs>`__.
.. card-carousel:: 1
.. card:: Bucket Without Locking
.. image:: /images/retention/minio-versioning-delete-object.svg
:alt: Deleting an Object
:align: center
:width: 600px
MinIO versioning preserves the full history of object mutations.
However, applications can explicitly delete specific object versions.
.. card:: Bucket With Locking
.. image:: /images/retention/minio-object-locking.svg
:alt: 30 Day Locked Objects
:align: center
:width: 600px
Applying a default 30 Day WORM lock to objects in the bucket ensures
a minimum period of retention and protection for all object versions.
.. card:: Delete Operations in Locked Bucket
.. image:: /images/retention/minio-object-locking-delete.svg
:alt: Delete Operation in Locked Bucket
:align: center
:width: 600px
Delete operations follow normal behavior in
:ref:`versioned buckets <minio-bucket-versioning-delete>`, where MinIO
creates a ``DeleteMarker`` for the object. However, non-Delete Marker
versions of the object remain under the retention rules and are protected
from any specific deletion or overwrite attempts.
.. card:: Versioned Delete Operations in Locked Bucket
.. image:: /images/retention/minio-object-locking-delete-version.svg
:alt: Versioned Delete Operation in a Locked Bucket
:align: center
:width: 600px
MinIO blocks any attempt to delete a specific object version held under
WORM lock. The earliest possible time after which a client may delete
the version is when the lock expires.
MinIO object locking is
:s3-docs:`feature and API compatible with AWS S3 <object-lock.html>`.
This page summarizes Object Locking / Retention concepts as implemented by
MinIO. See the AWS S3 documentation on
:s3-docs:`How S3 Object Lock works <object-lock.html>` for additional
resources.
You can only enable object locking during bucket creation as per
:s3-docs:`S3 behavior <object-lock-overview.html#object-lock-bucket-config>`.
You cannot enable object locking on a bucket created without locking
enabled. You can then configure object retention rules at any time.
Object locking requires :ref:`versioning <minio-bucket-versioning>` and
enables the feature implicitly.
.. _minio-bucket-locking-interactions-versioning:
Interaction with Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Objects held under WORM locked are immutable until the lock expires or is
explicitly lifted. Locking is per-object version, where each version is
independently immutable.
If an application performs an unversioned delete operation on a locked object,
the operation produces a :ref:`delete marker <minio-bucket-versioning-delete>`.
Attempts to explicitly delete any WORM-locked object fail with an error.
Delete Markers are *not* eligible for protection under WORM locking.
See the S3 documentation on
:s3-docs:`Managing delete markers and object lifecycles
<object-lock-managing.html#object-lock-managing-lifecycle>` for more
information.
For example, consider the following bucket with
:ref:`minio-object-locking-governance` locking enabled by default:
.. code-block:: shell
$ mc ls --versions play/locking-guide
[DATETIME] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
[DATETIME] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
[DATETIME] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv
Attempting to perform a delete on a *specific version* of ``data.csv`` fails
due to the object locking settings:
.. code-block:: shell
$ mc rm --version-id 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 play/data.csv
Removing `play/locking-guide/data.csv` (versionId=62429eb1-9cb7-4dc5-b507-9cc23d0cc691).
mc: <ERROR> Failed to remove `play/locking-guide/data.csv`.
Object, 'data.csv (Version ID=62429eb1-9cb7-4dc5-b507-9cc23d0cc691)' is
WORM protected and cannot be overwritten
Attempting to perform an unversioned delete on ``data.csv`` succeeds and creates
a new ``DeleteMarker`` for the object:
.. code-block:: shell
$ mc rm play/locking-guide/data.csv
[DATETIME] 0B acce329f-ad32-46d9-8649-5fe8bf4ec6e0 v4 DEL data.csv
[DATETIME] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
[DATETIME] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
[DATETIME] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv
Interaction with Lifecycle Management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO :ref:`object expiration <minio-lifecycle-management-expiration>`
respects any active object lock and retention settings for objects covered by
the expiration rule.
- For expiration rules operating on only the *current* object version,
MinIO creates a Delete Marker for the locked object.
- For expiration rules operating on *non-current object versions*,
MinIO can only expire the non-current versions *after* the retention period
has passed *or* has been explicitly lifted (e.g. Legal Holds).
For example, consider the following bucket with
:ref:`minio-object-locking-governance` locking enabled by default for 45 days:
.. code-block:: shell
$ mc ls --versions play/locking-guide
[7D] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
[30D] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
[60D] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv
Creating an expiration rule for *current* objects older than 7 days results in
a Delete Marker for the object:
.. code-block:: shell
$ mc ls --versions play/locking-guide
[0D] 0B acce329f-ad32-46d9-8649-5fe8bf4ec6e0 v4 DEL data.csv
[7D] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
[30D] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
[60D] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv
However, an expiration rule for *non-current* objects older than 7 days would
only take effect *after* the configured WORM lock expires. Since the bucket
has a 45 day ``GOVERNANCE`` retention set, only the ``v1`` version of
``data.csv`` is unlocked and therefore eligible for deletion.
Tutorials
---------
Create Bucket with Object Locking Enabled
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must enable object locking during bucket creation as per S3 behavior.
You can create a bucket with object locking enabled using the MinIO Console,
the MinIO :mc:`mc` CLI, or using an S3-compatible SDK.
.. tab-set::
.. tab-item:: MinIO Console
:sync: console
Select the :guilabel:`Buckets` section of the MinIO Console to access
bucket creation and management functions. Select the bucket row from the
list of buckets. You can use the :octicon:`search` :guilabel:`Search` bar
to filter the list.
.. image:: /images/minio-console/console-bucket.png
:width: 600px
:alt: MinIO Console Bucket Management
:align: center
Click the :guilabel:`Create Bucket` button to open the bucket creation
modal. Toggle the :guilabel:`Object Locking` selector to enable object
locking on the bucket.
.. image:: /images/minio-console/console-bucket-create-bucket-with-locking.png
:width: 600px
:alt: MinIO Console Bucket Management
:align: center
.. tab-item:: MinIO CLI
:sync: cli
Use the :mc-cmd:`mc mb` command with the :mc-cmd:`~mc mb --with-lock`
option to create a bucket with object locking enabled:
.. code-block:: shell
:class: copyable
mc mb --with-lock ALIAS/BUCKET
- Replace ``ALIAS`` with the :mc:`alias <mc alias>` of a configured
MinIO deployment.
- Replace ``BUCKET`` with the
:mc:`name <mc version ALIAS>` of the bucket to create.
Configure Bucket-Default Object Retention
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure object locking rules ("object retention") using the
MinIO Console, the MinIO :mc:`mc` CLI, or using an S3-compatible SDK.
MinIO supports setting both bucket-default *and* per-object retention rules.
The following examples set bucket-default retention. For per-object retention
settings, defer to the documentation for the ``PUT`` operation used by your
preferred SDK.
.. tab-set::
.. tab-item:: MinIO Console
:sync: console
Select the :guilabel:`Buckets` section of the MinIO Console to access bucket creation and management functions. You can use the :octicon:`search` :guilabel:`Search` bar to filter the list.
.. image:: /images/minio-console/console-bucket.png
:width: 600px
:alt: MinIO Console Bucket Management
:align: center
Each bucket row has a :guilabel:`Manage` button that opens the management view for that bucket.
.. image:: /images/minio-console/console-bucket-manage.png
:width: 600px
:alt: MinIO Console Bucket Management
:align: center
From the :guilabel:`Retention` section, select :guilabel:`Enabled`.
This section is only visible for buckets created with object locking enabled.
From the :guilabel:`Set Retention Configuration` modal, set the desired bucket default retention settings.
- For :guilabel:`Retention Mode`, select either :ref:`COMPLIANCE <minio-object-locking-compliance>` or :ref:`GOVERNANCE <minio-object-locking-governance>`.
- For :guilabel:`Duration`, select the retention duration units of :guilabel:`Days` or :guilabel:`Years`.
- For :guilabel:`Retention Validity`, set the duration of time for which MinIO holds objects under the specified retention mode for the bucket.
.. tab-item:: MinIO CLI
:sync: cli
Use the :mc-cmd:`mc retention set` command with the
:mc-cmd:`--recursive <mc retention set --recursive>` and
:mc-cmd:`--default <mc retention set --default>` options to set the
default retention mode for a bucket:
.. code-block:: shell
:class: copyable
mc retention set --recursive --default MODE DURATION ALIAS/BUCKET
- Replace :mc-cmd:`MODE <mc retention set MODE>` with either either
:ref:`COMPLIANCE <minio-object-locking-compliance>` or
:ref:`GOVERNANCE <minio-object-locking-governance>`.
- Replace :mc-cmd:`DURATION <mc retention set VALIDITY>` with the
duration for which the object lock remains in effect.
- Replace :mc-cmd:`ALIAS <mc retention set ALIAS>` with the
:mc:`alias <mc alias>` of a configured MinIO deployment.
- Replace :mc-cmd:`BUCKET <mc retention set ALIAS>` with the
name of the bucket on which to set the default retention rule.
Enable Legal Hold Retention
~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can enable or disable indefinite Legal Hold retention for an object using the MinIO Console, the MinIO :mc:`mc` CLI, or using an S3-compatible SDK.
You can place a legal hold on an object already held under a :ref:`COMPLIANCE <minio-object-locking-compliance>` or :ref:`GOVERNANCE <minio-object-locking-governance>` lock.
The object remains WORM locked under the legal hold even when the retention lock expires.
You or another user with the necessary permissions must explicitly lift the legal hold to remove the WORM lock.
.. tab-set::
.. tab-item:: MinIO Console
:sync: console
Select the :guilabel:`Buckets` section of the MinIO Console to access bucket creation and management functions.
You can use the :octicon:`search` :guilabel:`Search` bar to filter the list.
.. image:: /images/minio-console/console-bucket.png
:width: 600px
:alt: MinIO Console Bucket Management
:align: center
Each bucket row has a :guilabel:`Manage` button that opens the management view for that bucket.
.. image:: /images/minio-console/console-object-browser.png
:width: 600px
:alt: MinIO Console Bucket Object Browser
:align: center
Browse to the object and select it to open the object details view.
Select the :guilabel:`Legal Hold` button to toggle the Legal Hold status of the object.
.. tab-item:: MinIO CLI
:sync: cli
Use the :guilabel:`mc legalhold set` command to toggle the legal hold status on an object.
.. code-block:: shell
:class: copyable
mc legalhold set ALIAS/PATH
- Replace :mc-cmd:`ALIAS <mc legalhold set ALIAS>` with the
:mc:`alias <mc alias>` of a configured MinIO deployment.
- Replace :mc-cmd:`PATH <mc legalhold set ALIAS>` with the
path to the object for which to enable the legal hold.
.. _minio-object-locking-retention-modes:
Object Retention Modes
----------------------
MinIO implements the following
:s3-docs:`S3 Object Locking Modes <object-lock-overview.html>`:
.. list-table::
:header-rows: 1
:widths: 40 60
:width: 100%
* - Mode
- Summary
* - :ref:`minio-object-locking-governance`
- Prevents any operation that would mutate or modify the object or its
locking settings by non-privileged users.
Users with the :policy-action:`s3:BypassGovernanceRetention` permission
on the bucket or object can modify the object or its locking settings.
MinIO lifts the lock automatically after the configured retention rule
duration has passed.
* - :ref:`minio-object-locking-compliance`
- Prevents any operation that would mutate or modify the object or its
locking settings.
No MinIO user can modify the object or its settings, including the
:ref:`MinIO root <minio-users-root>` user.
MinIO lifts the lock automatically after the configured retention rule
duration has passed.
.. _minio-object-locking-governance:
GOVERNANCE Mode
~~~~~~~~~~~~~~~
An object under ``GOVERNANCE`` lock is protected from write operations by
non-privileged users.
``GOVERNANCE`` locked objects enforce managed-immutability for locked objects,
where users with the :policy-action:`s3:BypassGovernanceRetention` action can
modify the locked object, change the retention duration, or lift the lock
entirely. Bypassing ``GOVERNANCE`` retention also requires setting the
``x-amz-bypass-governance-retention:true`` header as part of the request.
The MinIO ``GOVERNANCE`` lock is functionally identical to the
:s3-docs:`S3 GOVERNANCE mode
<object-lock-overview.html#object-lock-retention-modes>`.
.. _minio-object-locking-compliance:
COMPLIANCE Mode
~~~~~~~~~~~~~~~
An object under ``COMPLIANCE`` lock is protected from write operations by *all*
users, including the :ref:`MinIO root <minio-users-root>` user.
``COMPLIANCE`` locked objects enforce complete immutability for locked objects.
You cannot change or remove the lock before the configured retention
duration has passed.
The MinIO ``COMPLIANCE`` lock is functionally identical to the
:s3-docs:`S3 GOVERNANCE mode
<object-lock-overview.html#object-lock-retention-modes>`.
.. _minio-object-locking-legalhold:
Legal Hold
----------
An object under Legal Hold is protected from write operations by *all*
users, including the :ref:`MinIO root <minio-users-root>` user.
Legal Holds are indefinite and enforce complete immutability for locked objects.
Only privileged users with the :policy-action:`s3:PutObjectLegalHold` permission can set or lift the Legal Hold.
Legal holds are complementary to both :ref:`minio-object-locking-governance` and :ref:`minio-object-locking-compliance` retention settings.
An object held under both legal hold *and* a ``GOVERNANCE/COMPLIANCE`` retention rule remains WORM locked until the legal hold is lifted *and* the rule expires.
For ``GOVERNANCE`` locked objects, the legal hold prevents mutating the object *even if* the user has the necessary privileges to bypass retention.

View File

@@ -0,0 +1,306 @@
.. _minio-bucket-versioning:
=================
Bucket Versioning
=================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
Overview
--------
MinIO supports keeping multiple "versions" of an object in a single bucket.
Write operations which would normally overwrite an existing object instead
result in the creation of a new versioned object. MinIO versioning protects from
unintended overwrites and deletions while providing support for "undoing" a
write operation. Bucket versioning is a prerequisite for configuring
:ref:`object locking and retention rules <minio-object-locking>`.
For versioned buckets, any write operation that mutates an object results in a
new version of that object with a unique version ID. MinIO marks the "latest"
version of the object that clients retrieve by default. Clients can then
explicitly choose to list, retrieve, or remove a specific object version.
.. card-carousel:: 1
.. card:: Object with Single Version
.. image:: /images/retention/minio-versioning-single-version.svg
:alt: Object with single version
:align: center
MinIO adds a unique version ID to each object as part of write operations.
.. card:: Object with Multiple Versions
.. image:: /images/retention/minio-versioning-multiple-versions.svg
:alt: Object with Multiple Versions
:align: center
MinIO retains all versions of an object and marks the most recent
version as the "latest".
.. card:: Retrieving the Latest Object Version
.. image:: /images/retention/minio-versioning-retrieve-latest-version.svg
:alt: Object with Multiple Versions
:align: center
.. card:: Retrieving a Specific Object Version
.. image:: /images/retention/minio-versioning-retrieve-single-version.svg
:alt: Object with Multiple Versions
:align: center
:ref:`Deleting <minio-bucket-versioning-delete>` an object results in a special
``DeleteMarker`` tombstone that marks an object as deleted while retaining
all previous versions of that object.
Versioning is Per-Namespace
~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO uses the full namespace (the bucket and path to an object) for each object
as part of determining object uniqueness. For example, all of the following
namespaces are "unique" objects, where mutations of each object result in
the creation of new object versions *at that namespace*:
.. code-block:: shell
databucket/object.blob
databucket/blobs/object.blob
blobbucket/object.blob
blobbucket/blobs/object.blob
While ``object.blob`` might be the same binary across all namespaces,
MinIO only enforces versioning with a specific namespace and therefore
considers each ``object.blob`` above as distinct and unique.
Versioning and Storage Capacity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO does not perform incremental or differential-type versioning. For
mutation-heavy workloads, this may result in substantial disk usage by
older or aged object versions.
For example, consider a 1GB object containing log data. An application
appends 100MB of data to the log and uploads to MinIO. MinIO would then contain
both the 1GB and 1.1GB versions of the object. If the application repeated
this process every day for 10 days, the bucket would eventually contain more
than 14GB of data associated to a single object.
MinIO supports configuring configuring :ref:`object lifecycle management rules
<minio-lifecycle-management>` to automatically expire or transition aged
object versions and free up storage capacity. For example, you can configure
a rule to automatically expire object versions 90 days after they become
non-current (i.e. no longer the "latest" version of that object). See
:ref:`MinIO Object Expiration <minio-lifecycle-management-expiration>` for
more information.
You can alternatively perform manual removal of object versions using the
following commands:
- :mc-cmd:`mc rm --versions` - Removes all versions of an object.
- :mc-cmd:`mc rm --versions --older-than <mc rm --older-than>` -
Removes all versions of an object older than the specified calendar date.
.. _minio-bucket-versioning-id:
Version ID Generation
~~~~~~~~~~~~~~~~~~~~~
MinIO generates a unique and immutable identifier for each versioned object as
part of write operations. Each object version ID consists of a 128-bit
fixed-size :rfc:`UUIDv4 <4122#section-4.4>`. UUID generation is sufficiently
random to ensure high likelihood of uniqueness for any environment, are
computationally difficult to guess, and do not require centralized registration
process and authority to guarantee uniqueness.
.. image:: /images/retention/minio-versioning-multiple-versions.svg
:alt: Object with Multiple Versions
:width: 600px
:align: center
MinIO does not support client-managed version ID allocation. All version ID
generation is handled by the MinIO server process.
For objects created while versioning is disabled or suspended, MinIO
uses a ``null`` version ID. You can access or remove these objects by specifying
``null`` as the version ID as part of S3 operations.
.. _minio-bucket-versioning-delete:
Versioned Delete Operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Performing a ``DELETE`` operation on a versioned object creates a
0-byte ``DeleteMarker`` as the latest version of that object. Clients performing
``GET`` operations on that object do not return any results, as MinIO does not
return the ``DeleteMarker`` back as part of the response. Similarly, performing
a ``LIST`` operation by default returns only objects which are *not* a
``DeleteMarker``.
To permanently delete an object version, perform the ``DELETE`` operation and
specify the version ID of the object to delete. Versioned delete operations
are **irreversible**.
.. card-carousel:: 1
.. card:: Deleting an Object
.. image:: /images/retention/minio-versioning-delete-object.svg
:alt: Deleting an Object
:align: center
Performing a ``DELETE`` operation on a versioned object produces a
``DeleteMarker`` for that object.
.. card:: Reading a Deleted Object
.. image:: /images/retention/minio-versioning-retrieve-deleted-object.svg
:alt: Object with Multiple Versions
:align: center
Clients by default retrieve the "latest" object version. MinIO returns
a ``404``-like response if the latest version is a ``DeleteMarker``.
.. card:: Retrieve Previous Version of Deleted Object
.. image:: /images/retention/minio-versioning-retrieve-version-before-delete.svg
:alt: Retrieve Version of Deleted Object
:align: center
Clients can retrieve any previous version of the object by specifying the
version ID, even if the "Latest" version is a ``DeleteMarker``.
.. card:: Delete a Specific Object Version
.. image:: /images/retention/minio-versioning-delete-specific-version.svg
:alt: Retrieve Version of Deleted Object
:align: center
Clients can delete a specific object version by specifying the version ID
as part of the ``DELETE`` operation. Deleting a specific version is
**permanent** and does not result in the creation of a ``DeleteMarker``.
The following :mc:`mc` commands operate on ``DeleteMarkers`` or versioned
objects:
- Use :mc-cmd:`mc ls --versions` to view all versions of an object,
including delete markers.
- Use :mc-cmd:`mc cp --version-id=UUID ... <mc cp --version-id>` to
retrieve the version of the "deleted" object with matching ``UUID``.
- Use :mc-cmd:`mc rm --version-id=UUID ... <mc rm --version-id>` to delete
the version of the object with matching ``UUID``.
- Use :mc-cmd:`mc rm --versions` to delete *all* versions of an object.
Tutorials
---------
Enable Bucket Versioning
~~~~~~~~~~~~~~~~~~~~~~~~
You can enable versioning using the MinIO Console, the MinIO :mc:`mc` CLI, or
using an S3-compatible SDK. Versioning is a bucket-scoped feature. You cannot
enable versioning on only a prefix or subset of objects in a bucket.
.. tab-set::
.. tab-item:: MinIO Console
Select the :guilabel:`Buckets` section of the MinIO Console to access bucket creation and management functions. You can use the :octicon:`search` :guilabel:`Search` bar to filter the list.
.. image:: /images/minio-console/console-bucket.png
:width: 600px
:alt: MinIO Console Bucket Management
:align: center
Each bucket row has a :guilabel:`Manage` button that opens the management view for that bucket.
.. image:: /images/minio-console/console-bucket-manage.png
:width: 600px
:alt: MinIO Console Bucket Management
:align: center
Toggle the :guilabel:`Versioning` field to enable versioning on the bucket.
The MinIO Console also supports enabling versioning as part of bucket
creation. See :ref:`minio-console-admin-buckets` for more information on
bucket management using the MinIO Console.
.. tab-item:: MinIO CLI
Use the :mc-cmd:`mc version enable` command to enable versioning on an
existing bucket:
.. code-block:: shell
:class: copyable
mc version ALIAS/BUCKET
- Replace ``ALIAS`` with the :mc:`alias <mc alias>` of a configured
MinIO deployment.
- Replace ``BUCKET`` with the
:mc:`target bucket <mc version ALIAS>` on which to enable
versioning.
Objects created prior to enabling versioning have a
``null`` :ref:`version ID <minio-bucket-versioning-id>`.
Suspend Bucket Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~
You can suspend bucket versioning at any time using the MinIO Console, the
MinIO :mc:`mc` CLI, or using an S3-compatible SDK.
.. tab-set::
.. tab-item:: MinIO Console
Select the :guilabel:`Buckets` section of the MinIO Console to access bucket creation and management functions. You can use the :octicon:`search` :guilabel:`Search` bar to filter the list.
.. image:: /images/minio-console/console-bucket.png
:width: 600px
:alt: MinIO Console Bucket Management
:align: center
Each bucket row has a :guilabel:`Manage` button that opens the management view for that bucket.
.. image:: /images/minio-console/console-bucket-manage.png
:width: 600px
:alt: MinIO Console Bucket Management
:align: center
Select the :guilabel:`Versioning` field and follow the instructions to suspend versioning in the bucket.
See :ref:`minio-console-admin-buckets` for more information on bucket
management using the MinIO Console.
.. tab-item:: MinIO CLI
Use the :mc-cmd:`mc version suspend` command to enable versioning on an
existing bucket:
.. code-block:: shell
:class: copyable
mc version suspend ALIAS/BUCKET
- Replace ``ALIAS`` with the :mc:`alias <mc alias>` of a configured
MinIO deployment.
- Replace ``BUCKET`` with the
:mc:`target bucket <mc version ALIAS>` on which to disable
versioning.
Objects created while versioning is suspended are assigned a ``null`` :ref:`version ID <minio-bucket-versioning-id>`.
Any mutations to an object while versioning is suspended result in overwriting that ``null`` versioned object.
MinIO does not remove or otherwise alter existing versioned objects as part of suspending versioning.
Clients can continue interacting with any existing object versions in the bucket.

View File

@@ -0,0 +1,213 @@
.. _minio-lifecycle-management-transition-to-azure:
======================================
Transition Objects from MinIO to Azure
======================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
The procedure on this page creates a new object lifecycle management rule that
transition objects from a MinIO bucket to a remote storage tier on the
:abbr:`Azure (Microsoft Azure)` storage backend. This procedure supports use
cases like moving aged data to low-cost public cloud storage solutions after a
certain time period or calendar date.
.. todo: diagram
Requirements
------------
Install and Configure ``mc``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses :mc:`mc` for performing operations on the MinIO cluster.
Install :mc:`mc` on a machine with network access to both source and destination
clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
instructions on downloading and installing ``mc``.
Use the :mc:`mc alias` command to create an alias for the source MinIO cluster.
Alias creation requires specifying an access key for a user on the source and
destination clusters. The specified users must have :ref:`permissions
<minio-lifecycle-management-transition-to-azure-permissions>` for configuring
and applying transition operations.
.. _minio-lifecycle-management-transition-to-azure-permissions:
Required MinIO Permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO requires the following permissions scoped to the bucket or buckets
for which you are creating lifecycle management rules.
- :policy-action:`s3:PutLifecycleConfiguration`
- :policy-action:`s3:GetLifecycleConfiguration`
MinIO also requires the following administrative permissions on the cluster
in which you are creating remote tiers for object transition lifecycle
management rules:
- :policy-action:`admin:SetTier`
- :policy-action:`admin:ListTier`
For example, the following policy provides permission for configuring object
transition lifecycle management rules on any bucket in the cluster:.
.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json
:language: json
:class: copyable
.. _minio-lifecycle-management-transition-to-azure-permissions-remote:
Required Azure Permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~
Object transition lifecycle management rules require additional permissions on
the remote storage tier. Specifically, MinIO requires the
:abbr:`Azure (Microsoft Azure)` credentials provide read, write, list, and
delete permissions for the remote bucket.
Refer to the `Azure RBAC
<https://docs.microsoft.com/en-us/azure/role-based-access-control/>`__
documentation for more complete guidance on configuring the required
permissions.
Considerations
--------------
Exclusive Access to Remote Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-transition-bucket-access-desc
:end-before: end-transition-bucket-access-desc
.. important::
MinIO does *not* support changing the account name associated to an Azure
remote tier. Azure storage backends are tied to the account, such that
changing the account would change the storage backend and prevent access
to any objects transitioned to the original account/backend.
Please contact `MinIO Support <https://min.io/pricing?ref=docs>`__ if you need
situation-specific guidance around configuring Azure remote tiers.
Availability of Remote Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-transition-data-loss-desc
:end-before: end-transition-data-loss-desc
Procedure
---------
1) Configure User Accounts and Policies for Lifecycle Management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. |permissions| replace:: :ref:`permissions <minio-lifecycle-management-transition-to-azure-permissions>`
.. include:: /includes/common-minio-tiering.rst
:start-after: start-create-transition-user-desc
:end-before: end-create-transition-user-desc
2) Configure the Remote Storage Tier
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin tier add` command to add a new remote storage tier:
.. code-block:: shell
:class: copyable
mc admin tier add --azure TARGET TIER_NAME \
--endpoint https://HOSTNAME
--bucket BUCKET \
--prefix PREFIX
--account-name ACCOUNT \
--account-key KEY \
--region REGION
The example above uses the following arguments:
.. list-table::
:header-rows: 1
:widths: 30 70
:width: 100%
* - Argument
- Description
* - :mc-cmd:`TARGET <mc admin tier add TARGET>`
- The :mc:`alias <mc alias>` of the MinIO deployment on which to configure
the remote tier.
* - :mc-cmd:`TIER_NAME <mc admin tier add TIER_NAME>`
- The name to associate with the new :abbr:`Azure (Microsoft Azure)` blob
remote storage tier. Specify the name in all-caps, e.g. ``AZURE_TIER``.
This value is required in the next step.
* - :mc-cmd:`HOSTNAME <mc admin tier add --endpoint>`
- The URL endpoint for the :abbr:`Azure (Microsoft Azure)` storage
backend.
* - :mc-cmd:`BUCKET <mc admin tier add --bucket>`
- The name of the bucket on the :abbr:`Azure (Microsoft Azure)` storage
backend to which MinIO transitions objects.
* - :mc-cmd:`PREFIX <mc admin tier add --prefix>`
- The optional bucket prefix within which MinIO transitions objects.
MinIO stores all transitioned objects in the specified ``BUCKET`` under a
unique per-deployment prefix value. Omit this argument to use only that
value for isolating and organizing data within the remote storage.
MinIO recommends specifying this optional prefix for remote storage tiers
which contain other data, including transitioned objects from other MinIO
deployments. This prefix should provide a clear reference back to the
source MinIO deployment to faciliate ease of operations related to
diagnostics, maintenance, or disaster recovery.
* - :mc-cmd:`ACCOUNT <mc admin tier add --account-name>`
- The account name MinIO uses to access the bucket. The account name
*must* correspond to an :abbr:`Azure (Microsoft Azure)` user with the
required :ref:`permissions
<minio-lifecycle-management-transition-to-azure-permissions-remote>`.
You cannot change this account name after creating the tier.
* - :mc-cmd:`KEY <mc admin tier add --account-key>`
- The corresponding key for the specified ``ACCOUNT``.
* - :mc-cmd:`REGION <mc admin tier add --region>`
- The :abbr:`Azure (Microsoft Azure)` blob storage region of the specified
``BUCKET``. You can safely omit this option if the ``HOSTNAME`` includes
the region.
3) Create and Apply the Transition Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-create-transition-rule-desc
:end-before: end-create-transition-rule-desc
4) Verify the Transition Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc ilm ls` command to review the configured transition
rules:
.. code-block:: shell
:class: copyable
mc ilm ls ALIAS/PATH --transition
- Replace :mc-cmd:`ALIAS <mc ilm ls ALIAS>` with the :mc:`alias <mc alias>`
of the MinIO deployment.
- Replace :mc-cmd:`PATH <mc ilm ls ALIAS>` with the name of the bucket for
which to retrieve the configured lifecycle management rules.

View File

@@ -0,0 +1,207 @@
.. _minio-lifecycle-management-transition-to-gcs:
====================================
Transition Objects from MinIO to GCS
====================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
The procedure on this page creates a new object lifecycle management rule that
transition objects from a MinIO bucket to a remote storage tier on the Google
Cloud Storage backend. This procedure supports use cases like moving aged data
to low-cost public cloud storage solutions after a certain time period or
calendar date.
.. todo: diagram
Requirements
------------
Install and Configure ``mc``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses :mc:`mc` for performing operations on the MinIO cluster.
Install :mc:`mc` on a machine with network access to both source and destination
clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
instructions on downloading and installing ``mc``.
Use the :mc:`mc alias` command to create an alias for the source MinIO cluster.
Alias creation requires specifying an access key for a user on the source and
destination clusters. The specified users must have :ref:`permissions
<minio-lifecycle-management-transition-to-gcs-permissions>` for configuring and
applying transition operations.
.. _minio-lifecycle-management-transition-to-gcs-permissions:
Required MinIO Permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO requires the following permissions scoped to the bucket or buckets
for which you are creating lifecycle management rules.
- :policy-action:`s3:PutLifecycleConfiguration`
- :policy-action:`s3:GetLifecycleConfiguration`
MinIO also requires the following administrative permissions on the cluster
in which you are creating remote tiers for object transition lifecycle
management rules:
- :policy-action:`admin:SetTier`
- :policy-action:`admin:ListTier`
For example, the following policy provides permission for configuring object
transition lifecycle management rules on any bucket in the cluster:.
.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json
:language: json
:class: copyable
.. _minio-lifecycle-management-transition-to-gcs-permissions-remote:
Required GCS Permissions
~~~~~~~~~~~~~~~~~~~~~~~~
Object transition lifecycle management rules require additional permissions on
the remote storage tier. Specifically, MinIO requires the
:abbr:`GCS (Google Cloud Storage)` credentials provide read, write, list, and
delete permissions for the remote bucket.
Refer to the `GCS IAM permissions
<https://cloud.google.com/storage/docs/access-control/iam-permissions>`__
documentation for more complete guidance on configuring the required
permissions.
Considerations
--------------
Lifecycle Management Object Scanner
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO uses a scanner process to check objects against all configured
lifecycle management rules. Slow scanning due to high IO workloads or
limited system resources may delay application of lifecycle management
rules. See :ref:`minio-lifecycle-management-scanner` for more information.
Exclusive Access to Remote Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-transition-bucket-access-desc
:end-before: end-transition-bucket-access-desc
Availability of Remote Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-transition-data-loss-desc
:end-before: end-transition-data-loss-desc
Procedure
---------
1) Configure User Accounts and Policies for Lifecycle Management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. |permissions| replace:: :ref:`permissions <minio-lifecycle-management-transition-to-gcs-permissions>`
.. include:: /includes/common-minio-tiering.rst
:start-after: start-create-transition-user-desc
:end-before: end-create-transition-user-desc
2) Configure the Remote Storage Tier
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin tier add` command to add a new Google Cloud Storage
service as the remote storage tier:
.. code-block:: shell
:class: copyable
mc admin tier add --gcs TARGET TIER_NAME \
--endpoint https://HOSTNAME \
--bucket BUCKET \
--prefix PREFIX \
--credentials-file CREDENTIALS \
--region REGION
The example above uses the following arguments:
.. list-table::
:header-rows: 1
:widths: 30 70
:width: 100%
* - Argument
- Description
* - :mc-cmd:`TARGET <mc admin tier add TARGET>`
- The :mc:`alias <mc alias>` of the MinIO deployment on which to configure
the :abbr:`GCS (Google Cloud Storage)` remote tier.
* - :mc-cmd:`TIER_NAME <mc admin tier add TIER_NAME>`
- The name to associate with the new :abbr:`GCS (Google Cloud Storage)`
remote storage tier. Specify the name in all-caps, e.g. ``GCS_TIER``.
This value is required in the next step.
* - :mc-cmd:`HOSTNAME <mc admin tier add --endpoint>`
- The URL endpoint for the :abbr:`GCS (Google Cloud Storage)` storage
backend.
* - :mc-cmd:`BUCKET <mc admin tier add --bucket>`
- The name of the bucket on the :abbr:`GCS (Google Cloud Storage)` storage
backend to which MinIO transitions objects.
* - :mc-cmd:`PREFIX <mc admin tier add --prefix>`
- The optional bucket prefix within which MinIO transitions objects.
MinIO stores all transitioned objects in the specified ``BUCKET`` under a
unique per-deployment prefix value. Omit this argument to use only that
value for isolating and organizing data within the remote storage.
MinIO recommends specifying this optional prefix for remote storage tiers
which contain other data, including transitioned objects from other MinIO
deployments. This prefix should provide a clear reference back to the
source MinIO deployment to faciliate ease of operations related to
diagnostics, maintenance, or disaster recovery.
* - :mc-cmd:`CREDENTIALS <mc admin tier add --credentials-file>`
- The `credential file
<https://cloud.google.com/docs/authentication/getting-started>`__ for a
user on the remote GCS tier. The specified user credentials *must*
correspond to a GCS user with the required
:ref:`permissions
<minio-lifecycle-management-transition-to-gcs-permissions-remote>`.
* - :mc-cmd:`REGION <mc admin tier add --region>`
- The :abbr:`GCS (Google Cloud Storage)` region of the specified
``BUCKET``. You can safely omit this
option if the ``HOSTNAME`` includes the region.
3) Create and Apply the Transition Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-create-transition-rule-desc
:end-before: end-create-transition-rule-desc
4) Verify the Transition Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc ilm ls` command to review the configured transition
rules:
.. code-block:: shell
:class: copyable
mc ilm ls ALIAS/PATH --transition
- Replace :mc-cmd:`ALIAS <mc ilm ls ALIAS>` with the :mc:`alias <mc alias>`
of the MinIO deployment.
- Replace :mc-cmd:`PATH <mc ilm ls ALIAS>` with the name of the bucket for
which to retrieve the configured lifecycle management rules.

View File

@@ -0,0 +1,225 @@
.. _minio-lifecycle-management-transition-to-s3:
===================================
Transition Objects from MinIO to S3
===================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
The procedure on this page creates a new object lifecycle management rule that
transition objects from a MinIO bucket to a remote storage tier on the Amazon
Web Services S3 storage backend *or* an S3-compatible service. This procedure
supports use cases such as tiering objects to low-cost or archival storage after
a certain time period or calendar date.
.. todo: diagram
Requirements
------------
Install and Configure ``mc``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses :mc:`mc` for performing operations on the MinIO cluster.
Install :mc:`mc` on a machine with network access to both source and destination
clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
instructions on downloading and installing ``mc``.
Use the :mc:`mc alias` command to create an alias for the source MinIO cluster.
Alias creation requires specifying an access key for a user on the source and
destination clusters. The specified users must have :ref:`permissions
<minio-lifecycle-management-transition-to-s3-permissions>` for configuring and
applying transition operations.
.. _minio-lifecycle-management-transition-to-s3-permissions:
Required MinIO Permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO requires the following permissions scoped to the bucket or buckets
for which you are creating lifecycle management rules.
- :policy-action:`s3:PutLifecycleConfiguration`
- :policy-action:`s3:GetLifecycleConfiguration`
MinIO also requires the following administrative permissions on the cluster
in which you are creating remote tiers for object transition lifecycle
management rules:
- :policy-action:`admin:SetTier`
- :policy-action:`admin:ListTier`
For example, the following policy provides permission for configuring object
transition lifecycle management rules on any bucket in the cluster:.
.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json
:language: json
:class: copyable
.. _minio-lifecycle-management-transition-to-s3-permissions-remote:
Required S3 Permissions
~~~~~~~~~~~~~~~~~~~~~~~
Object transition lifecycle management rules require additional permissions
on the remote storage tier. Specifically, MinIO requires the remote
tier credentials provide read, write, list, and delete permissions for the
remote bucket.
For example, the following policy provides the necessary permission
for transitioning objects into and out of the remote tier:
.. literalinclude:: /extra/examples/LifecycleManagementUser.json
:language: json
:class: copyable
Modify the ``Resource`` for the bucket into which MinIO tiers objects.
Refer to the :aws-docs:`Amazon S3 Permissions
<service-authorization/latest/reference/list_amazons3.html#amazons3-actions-as-permissions>`
documentation for more complete guidance on configuring the required
permissions.
Considerations
--------------
Lifecycle Management Object Scanner
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO uses a scanner process to check objects against all configured
lifecycle management rules. Slow scanning due to high IO workloads or
limited system resources may delay application of lifecycle management
rules. See :ref:`minio-lifecycle-management-scanner` for more information.
Exclusive Access to Remote Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-transition-bucket-access-desc
:end-before: end-transition-bucket-access-desc
Availability of Remote Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-transition-data-loss-desc
:end-before: end-transition-data-loss-desc
Procedure
---------
1) Configure User Accounts and Policies for Lifecycle Management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. |permissions| replace:: :ref:`permissions <minio-lifecycle-management-transition-to-s3-permissions>`
.. include:: /includes/common-minio-tiering.rst
:start-after: start-create-transition-user-desc
:end-before: end-create-transition-user-desc
2) Configure the Remote Storage Tier
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin tier add` command to add an Amazon S3 service as the
new remote storage tier:
.. code-block:: shell
:class: copyable
mc admin tier add --s3 TARGET TIER_NAME \
--endpoint https://HOSTNAME \
--bucket BUCKET \
--prefix PREFIX
--access-key ACCESS_KEY \
--secret-key SECRET_KEY \
--region REGION \
--storage-class STORAGE_CLASS
The example above uses the following arguments:
.. list-table::
:header-rows: 1
:widths: 30 70
:width: 100%
* - Argument
- Description
* - :mc-cmd:`TARGET <mc admin tier add TARGET>`
- The :mc:`alias <mc alias>` of the MinIO deployment on which to configure
the S3 remote tier.
* - :mc-cmd:`TIER_NAME <mc admin tier add TIER_NAME>`
- The name to associate with the new S3 remote storage tier. Specify the
name in all-caps, e.g. ``S3_TIER``. This value is required in the next
step.
* - :mc-cmd:`HOSTNAME <mc admin tier add --endpoint>`
- The URL endpoint for the S3 storage backend.
* - :mc-cmd:`BUCKET <mc admin tier add --bucket>`
- The name of the bucket on the S3 storage backend to which MinIO
transitions objects.
* - :mc-cmd:`PREFIX <mc admin tier add --prefix>`
- The optional bucket prefix within which MinIO transitions objects.
MinIO stores all transitioned objects in the specified ``BUCKET`` under a
unique per-deployment prefix value. Omit this argument to use only that
value for isolating and organizing data within the remote storage.
MinIO recommends specifying this optional prefix for remote storage tiers
which contain other data, including transitioned objects from other MinIO
deployments. This prefix should provide a clear reference back to the
source MinIO deployment to faciliate ease of operations related to
diagnostics, maintenance, or disaster recovery.
* - :mc-cmd:`ACCESS_KEY <mc admin tier add --access-key>`
- The S3 access key MinIO uses to access the bucket. The
access key *must* correspond to an IAM user with the
required
:ref:`permissions
<minio-lifecycle-management-transition-to-s3-permissions-remote>`.
* - :mc-cmd:`SECRET_KEY <mc admin tier add --secret-key>`
- The corresponding secret key for the specified ``ACCESS_KEY``.
* - :mc-cmd:`REGION <mc admin tier add --region>`
- The AWS S3 region of the specified ``BUCKET``. You can safely omit this
option if the ``HOSTNAME`` includes the region.
* - :mc-cmd:`STORAGE_CLASS <mc admin tier add --storage-class>`
- The S3 storage class to which MinIO transitions objects. Specify
one of the following supported storage classes:
- ``STANDARD``
- ``REDUCED``
3) Create and Apply the Transition Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-create-transition-rule-desc
:end-before: end-create-transition-rule-desc
4) Verify the Transition Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc ilm ls` command to review the configured transition
rules:
.. code-block:: shell
:class: copyable
mc ilm ls ALIAS/PATH --transition
- Replace :mc-cmd:`ALIAS <mc ilm ls ALIAS>` with the :mc:`alias <mc alias>`
of the MinIO deployment.
- Replace :mc-cmd:`PATH <mc ilm ls ALIAS>` with the name of the bucket for
which to retrieve the configured lifecycle management rules.

View File

@@ -0,0 +1,195 @@
.. _minio-sse:
.. _minio-encryption-overview:
=================================
Server-Side Encryption of Objects
=================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
.. |EK| replace:: :abbr:`EK (External Key)`
.. |SSE| replace:: :abbr:`SSE (Server-Side Encryption)`
.. |KMS| replace:: :abbr:`KMS (Key Management System)`
MinIO Server-Side Encryption (SSE) protects objects as part of write operations,
allowing clients to take advantage of server processing power to secure objects
at the storage layer (encryption-at-rest). SSE also provides key functionality
to regulatory and compliance requirements around secure locking and erasure.
MinIO SSE uses the :minio-git:`MinIO Key Encryption Service (KES) <kes>` and an
external Key Management Service (KMS) for performing secured cryptographic
operations at scale. MinIO also supports client-managed key management, where
the application takes full responsibility for creating and managing encryption
keys for use with MinIO SSE.
MinIO SSE is feature and API compatible with
:s3-docs:`AWS Server-Side Encryption <server-side-encryption.html>` and
supports the following encryption strategies:
.. tab-set::
.. tab-item:: SSE-KMS *Recommended*
:sync: sse-kms
MinIO supports enabling automatic SSE-KMS encryption of all objects
written to a bucket using a specific External Key (EK) stored on the
external |KMS|. Clients can override the bucket-default |EK| by specifying
an explicit key as part of the write operation.
For buckets without automatic SSE-KMS encryption, clients can specify
an |EK| as part of the write operation instead.
SSE-KMS provides more granular and customizable encryption compared to
SSE-S3 and SSE-C and is recommended over the other supported encryption
methods.
For a tutorial on enabling SSE-KMS in a local (non-production) MinIO
Deployment, see :ref:`minio-encryption-sse-kms-quickstart`. For
production MinIO deployments, use one of the following guides:
- :ref:`AWS SecretsManager <minio-sse-aws>`
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
- :ref:`Azure Key Vault <minio-sse-azure>`
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
.. tab-item:: SSE-S3
:sync: sse-s3
MinIO supports enabling automatic SSE-S3 encryption of all objects
written to a bucket using an |EK| stored on the external |KMS|. MinIO
SSE-S3 supports *one* |EK| for the entire deployment.
For buckets without automatic SSE-S3 encryption, clients can request
SSE encryption as part of the write operation instead.
For a tutorial on enabling SSE-s3 in a local (non-production) MinIO
Deployment, see :ref:`minio-encryption-sse-s3-quickstart`. For
production MinIO deployments, use one of the following guides:
- :ref:`AWS SecretsManager <minio-sse-aws>`
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
- :ref:`Azure Key Vault <minio-sse-azure>`
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
.. tab-item:: SSE-C
:sync: sse-c
Clients specify an |EK| as part of the write operation for an object.
MinIO uses the specified |EK| to perform SSE-S3.
SSE-C does not support bucket-default encryption settings and requires
clients perform all key management operations.
MinIO SSE requires enabling :ref:`minio-tls`.
.. _minio-encryption-sse-secure-erasure-locking:
Secure Erasure and Locking
--------------------------
MinIO requires access to the Encryption Key (EK) *and* external Key Management
System (KMS) used as part of SSE operations to decrypt an object. You can use
this dependency to securely erase and lock objects from access by disabling
access to the EK or KMS used for encryption.
General strategies include, but are not limited to:
- Seal the |KMS| such that it cannot be accessed by MinIO server anymore. This
locks all SSE-KMS or SSE-S3 encrypted objects protected by any |EK| stored on
the KMS. The encrypted objects remain unreadable as long as the KMS remains
sealed.
- Seal/Unmount an |EK|. This locks all SSE-KMS or SSE-S3 encrypted objects
protected by that EK. The encrypted objects remain unreadable as long
as the CMK(s) remains sealed.
- Delete an |EK|. This renders all SSE-KMS or SSE-S3 encrypted objects protected
by that EK as permanently unreadable. The combination of deleting an EK and
deleting the data may fulfill regulatory requirements around secure deletion
of data.
Deleting an |EK| is typically irreversible. Exercise extreme caution
before intentionally deleting a master key.
For more information, see:
- :ref:`SSE-KMS Secure Erasure and Locking
<minio-encryption-sse-kms-erasure-locking>`
- :ref:`SSE-S3 Secure Erasure and Locking
<minio-encryption-sse-s3-erasure-locking>`
- :ref:`SSE-C Secure Erasure and Locking
<minio-encryption-sse-c-erasure-locking>`
Encryption Internals
--------------------
.. note::
The following section describes MinIO internal logic and functionality.
This information is purely educational and is not necessary for
configuring or implementing any MinIO feature.
.. _minio-encryption-sse-content-encryption:
Content Encryption
~~~~~~~~~~~~~~~~~~
The MinIO server uses an authenticated encryption scheme
(:ref:`AEAD <minio-encryption-sse-primitives>`) to en/decrypt and authenticate
the object content. The AEAD is combined with some state to build a
**Secure Channel**. A Secure Channel is a cryptographic construction that
ensures confidentiality and integrity of the processed data. In particular, the
Secure Channel splits the plaintext content into fixed size chunks and
en/decrypts each chunk separately using an unique key-nonce combination.
The following text diagram illustrates Secure Channel Construction of an
encrypted object:
The Secure Channel splits the object content into chunks of a fixed size of
``65536`` bytes. The last chunk may be smaller to avoid adding additional
overhead and is treated specially to prevent truncation attacks. The nonce
value is ``96`` bits long and generated randomly per object / multi-part part.
The Secure Channel supports plaintexts up to ``65536 * 2^32 = 256 TiB``.
For S3 multi-part operations, each object part is en/decrypted with the Secure
Channel Construction scheme shown above. For each part, MinIO generates a secret
key derived from the Object Encryption Key (OEK) and the part number using a
pseudo-random function (:ref:`PRF <minio-encryption-sse-primitives>`), such that
``key = PRF(OEK, part_id)``.
.. _minio-encryption-sse-primitives:
Cryptographic Primitives
~~~~~~~~~~~~~~~~~~~~~~~~
The MinIO server uses the following cryptographic primitive implementations:
.. list-table::
:header-rows: 1
:widths: 40 60
:width: 100%
* -
- Primitives
* - Pseudo-Random Functions (PRF)
- HMAC-SHA-256
* - :ref:`AEAD <minio-encryption-sse-content-encryption>`
- ``ChaCha20Poly-1305`` by default.
``AES-256-GCM`` for x86-64 CPUs with the AES-NI extension.
.. toctree::
:titlesonly:
:hidden:
/administration/server-side-encryption/server-side-encryption-sse-kms
/administration/server-side-encryption/server-side-encryption-sse-s3
/administration/server-side-encryption/server-side-encryption-sse-c

View File

@@ -0,0 +1,174 @@
.. _minio-encryption-sse-c:
=======================================================
Server-Side Encryption with Client-Managed Keys (SSE-C)
=======================================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
.. |EK| replace:: :abbr:`EK (External Key)`
.. |DEK| replace:: :abbr:`DEK (Data Encryption Key)`
.. |KEK| replace:: :abbr:`KEK (Key Encryption Key)`
.. |OEK| replace:: :abbr:`OEK (Object Encryption Key)`
.. |SSE| replace:: :abbr:`SSE (Server-Side Encryption)`
.. |KMS| replace:: :abbr:`KMS (Key Management Service)`
.. |KES| replace:: :abbr:`KES (Key Encryption Service)`
MinIO Server-Side Encryption (SSE) protects objects as part of write operations,
allowing clients to take advantage of server processing power to secure objects
at the storage layer (encryption-at-rest). SSE also provides key functionality
to regulatory and compliance requirements around secure locking and erasure.
The procedure on this page configures and enables Server-Side Encryption
with Client-Managed Keys (SSE-C). MinIO SSE-C supports client-driven
encryption of objects *before* writing the object to disk. Clients must
specify the correct key to decrypt objects for read operations.
MinIO SSE-C is functionally compatible with Amazon
:s3-docs:`Server-Side Encryption with Customer-Provided Keys
<ServerSideEncryptionCustomerKeys.html>`.
.. _minio-encryption-sse-c-erasure-locking:
Secure Erasure and Locking
--------------------------
SSE-C protects objects using an |EK| specified by the client as part
of the write operation. Assuming the client-side key management
supports disabling or deleting these keys:
- Disabling the |EK| temporarily locks any objects encrypted using that
|EK| by rendering them unreadable. You can later enable the |EK| to
resume normal read operations on those objects.
- Deleting the |EK| renders all objects encrypted by that |EK|
*permanently* unreadable. If the client-side KMS does not support
backups of the |EK|, this process is *irreversible*.
The scope of a single |EK| depends on the number of write operations
which specified that |EK| when requesting SSE-C encryption.
Considerations
--------------
SSE-C is Incompatible with Bucket Replication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SSE-C encrypted objects are not compatible with MinIO
:ref:`bucket replication <minio-bucket-replication>`. Use
:ref:`SSE-KMS <minio-encryption-sse-kms>` or
:ref:`SSE-S3 <minio-encryption-sse-s3>` to ensure encrypted
objects are compatible with bucket replication.
SSE-C Overrides SSE-S3 and SSE-KMS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Encrypting an object using SSE-C prevents MinIO from applying
:ref:`SSE-KMS <minio-encryption-sse-kms>` or
:ref:`SSE-S3 <minio-encryption-sse-s3>` encryption to that object.
Quickstart
----------
MinIO SSE-C requires the client to perform all key creation and storage
operations.
This procedure uses :mc:`mc` for performing operations on the source MinIO
deployment. Install :mc:`mc` on a machine with network access to the source
deployment. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
instructions on downloading and installing ``mc``.
The SSE-C key *must* be a 256-bit base64-encoded string. The client
application is responsible for generation and storage of the encryption key.
MinIO does *not* store SSE-C encryption keys and cannot decrypt SSE-C
encrypted objects without the client-managed key.
1) Generate the Encryption Key
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Generate the 256-bit base64-encoded string for use as the encryption key.
The following example generates a string that meets the encryption key
requirements. The resulting string is appropriate for non-production
environments:
.. code-block:: shell
:class: copyable
cat /dev/urandom | head -c 32 | base64 -
Defer to your organizations requirements for generating cryptographically
secure encryption keys.
Copy the encryption key for use in the next step.
2) Encrypt an Object using SSE-C
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports the following AWS S3 headers for specifying SSE-C encryption:
- ``X-Amz-Server-Side-Encryption-Customer-Algorithm`` set to ``AES256``.
- ``X-Amz-Server-Side-Encryption-Customer-Key`` set to the encryption key value.
- ``X-Amz-Server-Side-Encryption-Customer-Key-MD5`` to the 128-bit MD5 digest of
the encryption key.
The MinIO :mc:`mc` commandline tool S3-compatible SDKs include specific syntax
for setting headers. Certain :mc:`mc` commands like :mc:`mc cp` include specific
arguments for enabling SSE-S3 encryption:
.. code-block:: shell
:class: copyable
mc cp ~/data/mydata.json ALIAS/BUCKET/mydata.json \
--encrypt-key "ALIAS/BUCKET/=c2VjcmV0ZW5jcnlwdGlvbmtleWNoYW5nZW1lMTIzNAo="
- Replace :mc-cmd:`ALIAS <mc encrypt set ALIAS>` with the
:mc:`alias <mc alias>` of the MinIO deployment on which you want to write
the SSE-C encrypted object.
- Replace :mc-cmd:`BUCKET <mc encrypt set ALIAS>` with the full path to the
bucket or bucket prefix to which you want to write the SSE-C encrypted object.
3) Copy an SSE-C Encrypted Object
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports the following AWS S3 headers for copying an SSE-C encrypted
object to another S3-compatible service:
- ``X-Amz-Copy-Source-Server-Side-Encryption-Algorithm`` set to ``AES256``
- ``X-Amz-Copy-Source-Server-Side-Encryption-Key`` set to the encryption key
value. The copy operation will fail if the specified key does not match
the key used to SSE-C encrypt the object.
- ``X-Amz-Copy-Source-Server-Side-Encryption-Key-MD5`` set to the 128-bit MD5
digest of the encryption key.
The MinIO :mc:`mc` commandline tool S3-compatible SDKs include specific syntax
for setting headers. Certain :mc:`mc` commands like :mc:`mc cp` include specific
arguments for enabling SSE-S3 encryption:
.. code-block:: shell
:class: copyable
mc cp SOURCE/BUCKET/mydata.json TARGET/BUCKET/mydata.json \
--encrypt-key "SOURCE/BUCKET/=c2VjcmV0ZW5jcnlwdGlvbmtleWNoYW5nZW1lMTIzNAo=" \
--encrypt-key "TARGET/BUCKET/=c2VjcmV0ZW5jcnlwdGlvbmtleWNoYW5nZW1lMTIzNAo="
- Replace :mc-cmd:`SOURCE/BUCKET <mc encrypt set ALIAS>` with the
:mc:`alias <mc alias>` of the MinIO deployment from which you are reading the
encrypted object and the full path to the
bucket or bucket prefix from which you want to read the SSE-C encrypted
object.
- Replace :mc-cmd:`TARGET/BUCKET <mc encrypt set ALIAS>` with the
:mc:`alias <mc alias>` of the MinIO deployment from which you are writing the
encrypted object and the full path to the
bucket or bucket prefix to which you want to write the SSE-C encrypted
object.

View File

@@ -0,0 +1,323 @@
.. _minio-encryption-sse-kms:
=====================================================
Server-Side Encryption with Per-Bucket Keys (SSE-KMS)
=====================================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
.. |EK| replace:: :abbr:`EK (External Key)`
.. |DEK| replace:: :abbr:`DEK (Data Encryption Key)`
.. |KEK| replace:: :abbr:`KEK (Key Encryption Key)`
.. |OEK| replace:: :abbr:`OEK (Object Encryption Key)`
.. |SSE| replace:: :abbr:`SSE (Server-Side Encryption)`
.. |KMS| replace:: :abbr:`KMS (Key Management Service)`
.. |KES| replace:: :abbr:`KES (Key Encryption Service)`
MinIO Server-Side Encryption (SSE) protects objects as part of write operations,
allowing clients to take advantage of server processing power to secure objects
at the storage layer (encryption-at-rest). SSE also provides key functionality
to regulatory and compliance requirements around secure locking and erasure.
MinIO SSE uses the :minio-git:`MinIO Key Encryption Service (KES) <kes>` and an
external Key Management Service (KMS) for performing secured cryptographic
operations at scale. MinIO also supports client-managed key management, where
the application takes full responsibility for creating and managing encryption
keys for use with MinIO SSE.
MinIO SSE-KMS en/decrypts objects using an External Key (EK) managed by a Key
Management System (KMS). Each bucket and object can have a separate |EK|,
supporting more granular cryptographic operations in the deployment. MinIO can
only decrypt an object if it can access both the KMS *and* the |EK| used to
encrypt that object.
You can enable bucket-default SSE-KMS encryption using the
:mc-cmd:`mc encrypt set` command:
.. code-block:: shell
:class: copyable
mc encrypt set sse-kms EXTERNALKEY play/mybucket
- Replace ``EXTERNALKEY`` with the name of the |EK| to use for encrypting
objects in the bucket.
- Replace ``play/mybucket`` with the :mc-cmd:`alias <mc alias>` and bucket
on which you want to enable automatic SSE-KMS encryption.
MinIO SSE-KMS is functionally compatible with AWS S3
:s3-docs:`Server-Side Encryption with KMS keys stored in AWS
<UsingKMSEncryption.html>` while expanding support to include the
following KMS providers:
- :ref:`AWS SecretsManager <minio-sse-aws>`
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
- :ref:`Azure Key Vault <minio-sse-azure>`
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
- Thales CipherTrust (formerly Gemalto KeySecure)
.. _minio-encryption-sse-kms-quickstart:
Quickstart
----------
The following procedure uses the ``play`` MinIO |KES| sandbox for
supporting |SSE| with SSE-KMS in evaluation and early development environments.
For extended development or production environments, use one of the following
supported external Key Management Services (KMS):
- :ref:`AWS SecretsManager <minio-sse-aws>`
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
- :ref:`Azure Key Vault <minio-sse-azure>`
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
.. include:: /includes/common-minio-kes.rst
:start-after: start-kes-play-sandbox-warning
:end-before: end-kes-play-sandbox-warning
This procedure requires the following components:
- Install :mc:`mc` on a machine with network access to the source
deployment. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
instructions on downloading and installing ``mc``.
- Install :minio-git:`MinIO Key Encryption Service (KES) <kes>` on a machine
with internet access. See the ``kes``
:minio-git:`Getting Started <kes/wiki/Getting-Started>` guide for
instructions on downloading, installing, and configuring KES.
1) Create an Encryption Key for SSE-KMS Encryption
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :minio-git:`kes <kes>` commandline tool to create a new External Key
(EK) for use with SSE-KMS Encryption.
Issue the following command to retrieve the root
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>` for the KES
server:
.. code-block:: shell
:class: copyable
curl -sSL --tlsv1.2 \
-O 'https://raw.githubusercontent.com/minio/kes/master/root.key' \
-O 'https://raw.githubusercontent.com/minio/kes/master/root.cert'
Set the following environment variables in the terminal or shell:
.. code-block:: shell
:class: copyable
export KES_CLIENT_KEY=root.key
export KES_CLIENT_CERT=root.cert
.. list-table::
:stub-columns: 1
:widths: 40 60
:width: 100%
* - ``KES_CLIENT_KEY``
- The private key for an :minio-git:`identity
<kes/wiki/Configuration#policy-configuration>` on the KES server.
The identity must grant access to at minimum the ``/v1/create``,
``/v1/generate``, and ``/v1/list`` :minio-git:`API endpoints
<kes/wiki/Server-API#api-overview>`. This step uses the root
identity for the MinIO ``play`` KES sandbox, which provides access
to all operations on the KES server.
* - ``KES_CLIENT_CERT``
- The corresponding certificate for the :minio-git:`identity
<kes/wiki/Configuration#policy-configuration>` on the KES server.
This step uses the root identity for the MinIO ``play`` KES
sandbox, which provides access to all operations on the KES server.
Issue the following command to create a new |EK| through
KES.
.. code-block:: shell
:class: copyable
kes key create my-minio-sse-kms-key
This tutorial uses the example ``my-minio-sse-kms-key`` name for ease of
reference. Specify a unique key name to prevent collision
with existing keys.
2) Configure MinIO for SSE-KMS Object Encryption
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Specify the following environment variables in the shell or terminal on each
MinIO server host in the deployment:
.. code-block:: shell
:class: copyable
export MINIO_KMS_KES_ENDPOINT=https://play.min.io:7373
export MINIO_KMS_KES_KEY_FILE=root.key
export MINIO_KMS_KES_CERT_FILE=root.cert
export MINIO_KMS_KES_KEY_NAME=my-minio-sse-kms-key
.. list-table::
:stub-columns: 1
:widths: 30 80
* - :envvar:`MINIO_KMS_KES_ENDPOINT`
- The endpoint for the MinIO ``Play`` KES service.
* - :envvar:`MINIO_KMS_KES_KEY_FILE`
- The private key file corresponding to an
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>`
on the KES service. The identity must grant permission to
create, generate, and decrypt keys. Specify the same
identity key file as the ``KES_KEY_FILE`` environment variable
in the previous step.
* - :envvar:`MINIO_KMS_KES_CERT_FILE`
- The public certificate file corresponding to an
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>`
on the KES service. The identity must grant permission to
create, generate, and decrypt keys. Specify the same
identity certificate as the ``KES_CERT_FILE`` environment
variable in the previous step.
* - :envvar:`MINIO_KMS_KES_KEY_NAME`
- The name of the External Key (EK) to use for
performing SSE encryption operations. KES retrieves the |EK| from
the configured Key Management Service (KMS). Specify the name of the
key created in the previous step.
3) Restart the MinIO Deployment to Enable SSE-KMS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
4) Configure Automatic Bucket Encryption
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc encrypt set` command to enable automatic SSE-KMS protection
of all objects written to a specific bucket.
.. code-block:: shell
:class: copyable
mc encrypt set sse-kms my-minio-sse-kms-key ALIAS/BUCKET
- Replace :mc-cmd:`ALIAS <mc encrypt set ALIAS>` with the
:mc:`alias <mc alias>` of the MinIO deployment on which you enabled SSE-KMS.
- Replace :mc-cmd:`BUCKET <mc encrypt set ALIAS>` with the full path to the
bucket or bucket prefix on which you want to enable automatic SSE-KMS.
Objects written to the specified bucket are automatically encrypted using
the specified |EK|
Repeat this step for each bucket on which you want to enable automatic
SSE-KMS encryption. You can generate additional keys per bucket or bucket
prefix, such that the scope of each |EK| is limited to a subset of objects.
.. _minio-encryption-sse-kms-erasure-locking:
Secure Erasure and Locking
--------------------------
SSE-KMS protects objects using an |EK| specified either as part of the
bucket automatic encryption settings *or* as part of the write operation.
MinIO therefore *requires* access to that |EK| for decrypting that object.
- Disabling the |EK| temporarily locks objects encrypted with that |EK| by
rendering them unreadable. You can later enable the |EK| to resume
normal read operations on those objects.
- Deleting the |EK| renders all objects encrypted by that |EK|
*permanently* unreadable. If the KMS does not have or support backups
of the |EK|, this process is *irreversible*.
The scope of a single |EK| depends on:
- Which buckets specified that |EK| for automatic SSE-KMS encryption,
*and*
- Which write operations specified that |EK| when requesting SSE-KMS
encryption.
For example, consider a MinIO deployment using one |EK| per bucket.
Disabling a single |EK| renders all objects in the associated bucket
unreadable without affecting other buckets. If the deployment instead used
one |EK| for all objects and buckets, disabling that |EK| renders all
objects in the deployment unreadable.
.. _minio-encryption-sse-kms-encryption-process:
Encryption Process
------------------
.. note::
This section describes MinIO internal logic and functionality.
This information is purely educational and is not a prerequisite for
configuring or implementing any MinIO feature.
SSE-KMS uses an External Key (EK) managed by the configured Key Management
System (KMS) for performing cryptographic operations and protecting objects.
The table below describes each stage of the encryption process:
.. list-table::
:header-rows: 1
:widths: 30 70
* - Stage
- Description
* - SSE-Enabled Write Operation
- MinIO receives a write operation requesting SSE-KMS encryption.
The write operation *must* have an associated External Key (EK) to use
for encrypting the object.
- For write operations in buckets with automatic SSE-KMS enabled,
MinIO uses the bucket |EK|. If the write operation includes an
explicit EK, MinIO uses that *instead* of the bucket EK.
- For write operations in buckets *without* automatic SSE-KMS enabled,
MinIO uses the |EK| specified to the write operation.
* - Generate the Data Encryption Key (DEK)
- .. include:: /includes/common-minio-sse.rst
:start-after: start-sse-dek
:end-before: end-sse-dek
* - Generate the Key Encryption Key (KEK)
- .. include:: /includes/common-minio-sse.rst
:start-after: start-sse-kek
:end-before: end-sse-kek
* - Generate the Object Encryption Key (OEK)
- .. include:: /includes/common-minio-sse.rst
:start-after: start-sse-oek
:end-before: end-sse-oek
* - Encrypt the Object
- MinIO uses the |OEK| to encrypt the object *prior* to storing the
object to disk. MinIO then encrypts the |OEK| with the |KEK|.
MinIO stores the encrypted representation of the |OEK| and |DEK| as part
of the metadata.
For read operations, MinIO decrypts the object by retrieving the |EK| to
decrypt the |DEK|. MinIO then regenerates the |KEK|, decrypts the |OEK|, and
decrypts the object.

View File

@@ -0,0 +1,296 @@
.. _minio-encryption-sse-s3:
==================================================
Server-Side Encryption Per-Deployment Key (SSE-S3)
==================================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
.. |EK| replace:: :abbr:`EK (External Key)`
.. |DEK| replace:: :abbr:`DEK (Data Encryption Key)`
.. |KEK| replace:: :abbr:`KEK (Key Encryption Key)`
.. |OEK| replace:: :abbr:`OEK (Object Encryption Key)`
.. |SSE| replace:: :abbr:`SSE (Server-Side Encryption)`
.. |KMS| replace:: :abbr:`KMS (Key Management Service)`
.. |KES| replace:: :abbr:`KES (Key Encryption Service)`
MinIO Server-Side Encryption (SSE) protects objects as part of write operations,
allowing clients to take advantage of server processing power to secure objects
at the storage layer (encryption-at-rest). SSE also provides key functionality
to regulatory and compliance requirements around secure locking and erasure.
MinIO SSE uses the :minio-git:`MinIO Key Encryption Service (KES) <kes>` and an
external Key Management Service (KMS) for performing secured cryptographic
operations at scale. MinIO also supports client-managed key management, where
the application takes full responsibility for creating and managing encryption
keys for use with MinIO SSE.
MinIO SSE-S3 en/decrypts objects using an External Key (EK) managed by a
Key Management System (KMS). You must specify the |EK| using the
:envvar:`MINIO_KMS_KES_KEY_NAME` environment variable when starting up the
MinIO server. MinIO uses the same EK for *all* SSE-S3 cryptographic operations.
You can enable bucket-default SSE-S3 encryption using the
:mc-cmd:`mc encrypt set` command:
.. code-block:: shell
:class: copyable
mc encrypt set sse-s3 play/mybucket
- Replace ``play/mybucket`` with the :mc-cmd:`alias <mc alias>` and bucket
on which you want to enable automatic SSE-KMS encryption.
MinIO SSE-S3 is functionally compatible with AWS S3
:s3-docs:`Server-Side Encryption with Amazon S3-Managed Keys
<UsingServerSideEncryption.html>` while expanding support to include the
following KMS providers:
- :ref:`AWS SecretsManager <minio-sse-aws>`
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
- :ref:`Azure Key Vault <minio-sse-azure>`
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
- Thales CipherTrust (formerly Gemalto KeySecure)
.. _minio-encryption-sse-s3-quickstart:
Quickstart
----------
The following procedure uses the ``play`` MinIO |KES| sandbox for
supporting |SSE| with SSE-S3 in evaluation and early development environments.
For extended development or production environments, use one of the following
supported external Key Management Services (KMS):
- :ref:`AWS SecretsManager <minio-sse-aws>`
- :ref:`Google Cloud SecretManager <minio-sse-gcp>`
- :ref:`Azure Key Vault <minio-sse-azure>`
- :ref:`Hashicorp KeyVault <minio-sse-vault>`
- Thales CipherTrust (formerly Gemalto KeySecure)
.. include:: /includes/common-minio-kes.rst
:start-after: start-kes-play-sandbox-warning
:end-before: end-kes-play-sandbox-warning
This procedure requires the following components:
- Install :mc:`mc` on a machine with network access to the source
deployment. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
instructions on downloading and installing ``mc``.
- Install :minio-git:`MinIO Key Encryption Service (KES) <kes>` on a machine
with internet access. See the ``kes``
:minio-git:`Getting Started <kes/wiki/Getting-Started>` guide for
instructions on downloading, installing, and configuring KES.
1) Create an Encryption Key for SSE-S3 Encryption
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :minio-git:`kes <kes>` commandline tool to create a new External Key
(EK) for use with SSE-S3 Encryption.
Issue the following command to retrieve the root
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>` for the KES
server:
.. code-block:: shell
:class: copyable
curl -sSL --tlsv1.2 \
-O 'https://raw.githubusercontent.com/minio/kes/master/root.key' \
-O 'https://raw.githubusercontent.com/minio/kes/master/root.cert'
Set the following environment variables in the terminal or shell:
.. code-block:: shell
:class: copyable
export KES_CLIENT_KEY=root.key
export KES_CLIENT_CERT=root.cert
.. list-table::
:stub-columns: 1
:widths: 40 60
:width: 100%
* - ``KES_CLIENT_KEY``
- The private key for an :minio-git:`identity
<kes/wiki/Configuration#policy-configuration>` on the KES server.
The identity must grant access to at minimum the ``/v1/create``,
``/v1/generate``, and ``/v1/list`` :minio-git:`API endpoints
<kes/wiki/Server-API#api-overview>`. This step uses the root
identity for the MinIO ``play`` KES sandbox, which provides access
to all operations on the KES server.
* - ``KES_CLIENT_CERT``
- The corresponding certificate for the :minio-git:`identity
<kes/wiki/Configuration#policy-configuration>` on the KES server.
This step uses the root identity for the MinIO ``play`` KES
sandbox, which provides access to all operations on the KES server.
Issue the following command to create a new |EK| through
KES:
.. code-block:: shell
:class: copyable
kes key create my-minio-sse-s3-key
This tutorial uses the example ``my-minio-sse-s3-key`` name for ease of
reference. Specify a unique key name to prevent collision with existing keys.
2) Configure MinIO for SSE-S3 Object Encryption
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Specify the following environment variables in the shell or terminal on each
MinIO server host in the deployment:
.. code-block:: shell
:class: copyable
export MINIO_KMS_KES_ENDPOINT=https://play.min.io:7373
export MINIO_KMS_KES_KEY_FILE=root.key
export MINIO_KMS_KES_CERT_FILE=root.cert
export MINIO_KMS_KES_KEY_NAME=my-minio-sse-s3-key
.. list-table::
:stub-columns: 1
:widths: 30 80
* - :envvar:`MINIO_KMS_KES_ENDPOINT`
- The endpoint for the MinIO ``Play`` KES service.
* - :envvar:`MINIO_KMS_KES_KEY_FILE`
- The private key file corresponding to an
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>`
on the KES service. The identity must grant permission to
create, generate, and decrypt keys. Specify the same
identity key file as the ``KES_KEY_FILE`` environment variable
in the previous step.
* - :envvar:`MINIO_KMS_KES_CERT_FILE`
- The public certificate file corresponding to an
:minio-git:`identity <kes/wiki/Configuration#policy-configuration>`
on the KES service. The identity must grant permission to
create, generate, and decrypt keys. Specify the same
identity certificate as the ``KES_CERT_FILE`` environment
variable in the previous step.
* - :envvar:`MINIO_KMS_KES_KEY_NAME`
- The name of the External Key (EK) to use for
performing SSE encryption operations. KES retrieves the |EK| from
the configured Key Management System (KMS). Specify the name of the
key created in the previous step.
3) Restart the MinIO Deployment to Enable SSE-S3
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :ref:`alias <alias>` of the deployment to
restart.
4) Configure Automatic Bucket Encryption
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*Optional*
You can skip this step if you intend to use only client-driven SSE-S3.
Use the :mc-cmd:`mc encrypt set` command to enable automatic SSE-S3 protection
of all objects written to a specific bucket.
.. code-block:: shell
:class: copyable
mc encrypt set sse-s3 ALIAS/BUCKET
- Replace :mc-cmd:`ALIAS <mc encrypt set ALIAS>` with the
:mc:`alias <mc alias>` of the MinIO deployment on which you enabled SSE-S3.
- Replace :mc-cmd:`BUCKET <mc encrypt set ALIAS>` with the full path to the
bucket or bucket prefix on which you want to enable automatic SSE-S3.
.. _minio-encryption-sse-s3-erasure-locking:
Secure Erasure and Locking
--------------------------
SSE-S3 protects objects using an |EK| specified at server startup
using the :envvar:`MINIO_KMS_KES_KEY_NAME` environment variable. MinIO
therefore *requires* access to that |EK| for decrypting that object.
- Disabling the |EK| temporarily locks SSE-S3-encrypted objects in the
deployment by rendering them unreadable. You can later enable the |EK|
to resume normal read operations.
- Deleting the |EK| renders all SSE-S3-encrypted objects in the deployment
*permanently* unreadable. If the KMS does not have or support backups
of the |EK|, this process is *irreversible*.
The scope of the |EK| depends on:
- Which buckets specified automatic SSE-S3 encryption, *and*
- Which write operations requested SSE-S3 encryption.
.. _minio-encryption-sse-s3-encryption-process:
Encryption Process
------------------
.. note::
The following section describes MinIO internal logic and functionality.
This information is purely educational and is not necessary for
configuring or implementing any MinIO feature.
SSE-S3 uses an External Key (EK) managed by the configured Key Management
System (KMS) for performing cryptographic operations and protecting objects.
The table below describes each stage of the encryption process:
.. list-table::
:header-rows: 1
:widths: 30 70
* - Stage
- Description
* - SSE-Enabled Write Operation
- MinIO receives a write operation requesting SSE-S3 encryption.
MinIO uses the key name specified to
:envvar:`MINIO_KMS_KES_KEY_NAME` as the External Key (EK).
* - Generate the Data Encryption Key (DEK)
- .. include:: /includes/common-minio-sse.rst
:start-after: start-sse-dek
:end-before: end-sse-dek
* - Generate the Key Encryption Key (KEK)
- .. include:: /includes/common-minio-sse.rst
:start-after: start-sse-kek
:end-before: end-sse-kek
* - Generate the Object Encryption Key (OEK)
- .. include:: /includes/common-minio-sse.rst
:start-after: start-sse-oek
:end-before: end-sse-oek
* - Encrypt the Object
- MinIO uses the |OEK| to encrypt the object *prior* to storing the
object to disk. MinIO then encrypts the |OEK| with the |KEK|.
MinIO stores the encrypted representation of the |OEK| and |DEK| as part
of the metadata.