minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-07-30 07:03:26 +03:00
Code Activity

Bucket Notification Import

Browse Source 
CR fixes

Fixups
This commit is contained in:
ravindk89
2021-02-12 13:58:20 -05:00
committed by Ravind Kumar
parent 0e54583047
commit f1574f14ae
24 changed files with 8298 additions and 1562 deletions
Download Patch File Download Diff File Expand all files Collapse all files

167
source/monitoring/bucket-notifications/bucket-notifications.rst Normal file
View File

@ -0,0 +1,167 @@
.. _minio-bucket-notifications:
====================
Bucket Notifications
====================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
MinIO bucket notifications allow administrators to send notifications to
supported external services on certain object or bucket events. MinIO
supports bucket and object-level S3 events similar to the
:s3-docs:`Amazon S3 Event Notifications <NotificationHowTo.html>`.
MinIO bucket notifications are available *only* with
:mc:`minio server` deployments. MinIO :ref:`Gateway <minio-gateway>`
does *not* support bucket notifications.
Supported Notification Targets
------------------------------
MinIO supports publishing event notifications to the following targets:
.. list-table::
:header-rows: 1
:widths: 30 70
:width: 100%
* - Target
- Description
* - :guilabel:`AMQP` (RabbitMQ)
- Publish notifications to an AMQP service such as
`RabbitMQ <https://www.rabbitmq.com>`__.
See :ref:`minio-bucket-notifications-publish-amqp` for a tutorial.
* - :guilabel:`MQTT`
- Publish notifications to an `MQTT <https://www.mqtt.org/>`__
service.
See :ref:`minio-bucket-notifications-publish-mqtt` for a tutorial.
* - :guilabel:`NATS`
- Publish notifications to a `NATS <https://nats.io/>`__ service.
See :ref:`minio-bucket-notifications-publish-nats` for a tutorial.
* - :guilabel:`NSQ`
- Publish notifications to a `NSQ <https://nsq.io/>`__ service.
See :ref:`minio-bucket-notifications-publish-nsq` for a tutorial
* - :guilabel:`Elasticsearch`
- Publish notifications to a `Elasticsearch <https://www.elastic.co/>`__
service.
See :ref:`minio-bucket-notifications-publish-elasticsearch` for a
tutorial.
* - :guilabel:`Kafka`
- Publish notifications to a `Kafka <https://kafka.apache.org/>`__ service.
See :ref:`minio-bucket-notifications-publish-kafka` for a tutorial.
* - :guilabel:`MySQL`
- Publish notifications to a `MySQL <https://www.mysql.com/>`__ service.
See :ref:`minio-bucket-notifications-publish-mysql` for a tutorial.
* - :guilabel:`PostgreSQL`
- Publish notifications to a `PostgreSQL <https://www.postgresql.org/>`__
service.
See :ref:`minio-bucket-notifications-publish-postgresql` for a tutorial.
* - :guilabel:`Redis`
- Publish notifications to a `Redis <https://redis.io/>`__ service.
See :ref:`minio-bucket-notifications-publish-redis` for a tutorial.
* - :guilabel:`webhook`
- Publish notifications to a `Webhook
<https://en.wikipedia.org/wiki/Webhook>`__ service.
See :ref:`minio-bucket-notifications-publish-webhook` for a tutorial.
.. _minio-bucket-notifications-event-types:
Supported S3 Event Types
------------------------
MinIO bucket notifications are compatible with
:s3-docs:`Amazon S3 Event Notifications <NotificationHowTo.html>`. This
section lists all supported events.
Object Events
~~~~~~~~~~~~~
MinIO supports triggering notifications on the following S3 object events:
.. data:: s3:ObjectRemoved:DeleteMarkerCreated
.. data:: s3:ObjectRemoved:Delete
.. data:: s3:ObjectCreated:PutRetention
.. data:: s3:ObjectCreated:PutLegalHold
.. data:: s3:ObjectCreated:Put
.. data:: s3:ObjectCreated:Post
.. data:: s3:ObjectCreated:Copy
.. data:: s3:ObjectCreated:CompleteMultipartUpload
.. data:: s3:ObjectAccessed:Head
.. data:: s3:ObjectAccessed:GetRetention
.. data:: s3:ObjectAccessed:GetLegalHold
.. data:: s3:ObjectAccessed:Get
Replication Events
~~~~~~~~~~~~~~~~~~
MinIO supports triggering notifications on the following S3 replication
events:
.. data:: s3:Replication:OperationCompletedReplication
.. data:: s3:Replication:OperationFailedReplication
.. data:: s3:Replication:OperationMissedThreshold
.. data:: s3:Replication:OperationNotTracked
.. data:: s3:Replication:OperationReplicatedAfterThreshold
ILM Transition Events
~~~~~~~~~~~~~~~~~~~~~
MinIO supports triggering notifications on the following S3 ILM transition
events:
.. data:: s3:ObjectRestore:Post
.. data:: s3:ObjectRestore:Completed
Global Events
~~~~~~~~~~~~~
MinIO supports triggering notifications on the following global events.
You can only listen to these events through the :legacy:`ListenNotification
<golang-client-api-reference.html#ListenNotification>` API:
.. data:: s3:BucketCreated
.. data:: s3:BucketRemoved
.. todo
.. toctree::
:titlesonly:
:hidden:
/monitoring/bucket-notifications/publish-events-to-amqp
/monitoring/bucket-notifications/publish-events-to-mqtt
/monitoring/bucket-notifications/publish-events-to-nats
/monitoring/bucket-notifications/publish-events-to-nsq
/monitoring/bucket-notifications/publish-events-to-elasticsearch
/monitoring/bucket-notifications/publish-events-to-kafka
/monitoring/bucket-notifications/publish-events-to-mysql
/monitoring/bucket-notifications/publish-events-to-postgresql
/monitoring/bucket-notifications/publish-events-to-redis
/monitoring/bucket-notifications/publish-events-to-webhook

382
source/monitoring/bucket-notifications/publish-events-to-amqp.rst Normal file
View File

@ -0,0 +1,382 @@
.. _minio-bucket-notifications-publish-amqp:
=================================
Publish Events to AMQP (RabbitMQ)
=================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a `AMQP 0-9-1 <https://www.amqp.org/>`__
service endpoint such as `RabbitMQ <https://www.rabbitmq.com>`__.
MinIO relies on the :github:`streadway/amqp` project for AMQP connectivity. The
project is primarily tested against `RabbitMQ <https://www.rabbitmq.com/>`__
deployments, though other `AMQP 0-9-1-compatible <https://www.amqp.org/>`__
services *may* also work. The procedures on this page assume a RabbitMQ
deployment using the AMQP 0-9-1 protocol as the service endpoint.
Add an AMQP Endpoint to a MinIO Deployment
------------------------------------------
The following procedure adds a new AMQP service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
AMQP 0-9-1 Service Endpoint
+++++++++++++++++++++++++++
MinIO relies on the :github:`streadway/amqp` project for AMQP connectivity. The
project is primarily tested against `RabbitMQ <https://www.rabbitmq.com/>`__
deployments, though other `AMQP 0-9-1-compatible <https://www.amqp.org/>`__
services *may* also work. This procedure assumes a RabbitMQ deployment
using the 0-9-1 protocol as the service endpoint.
If the AMQP service requires authentication, you *must* provide an appropriate
username and password during the configuration process to grant MinIO access
to the service.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the AMQP Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new AMQP service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tabs::
.. tab:: Environment Variables
MinIO supports specifying the AMQP service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-amqp>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an AMQP service endpoint. The minimum
*required* variables are
:envvar:`MINIO_NOTIFY_AMQP_ENABLE` and :envvar:`MINIO_NOTIFY_AMQP_URL`:
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_AMQP_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_AMQP_URL_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_AMQP_EXCHANGE_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_EXCHANGE_TYPE_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_ROUTING_KEY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_MANDATORY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_DURABLE_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_NO_WAIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_INTERNAL_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_AUTO_DELETED_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_DELIVERY_MODE_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_AMQP_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
AMQP service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new AMQP service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing AMQP service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_amqp <mc admin config get>` to
review the currently configured AMQP endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the AMQP service endpoint.
For example:
``amqp://user:password@hostname:port``
See :ref:`AMQP Service for Bucket Notifications
<minio-server-envvar-bucket-notification-amqp>` for complete documentation
on each environment variable.
.. tab:: Configuration Settings
MinIO supports adding or updating AMQP endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_amqp` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
AMQP service endpoint. The minimum *required* setting is
:mc-conf:`notify_amqp url <notify_amqp.url>`:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_amqp:IDENTIFIER \
url="ENDPOINT" \
exchange="<string>" \
exchange_type="<string>" \
routing_key="<string>" \
mandatory="<string>" \
durable="<string>" \
no_wait="<string>" \
internal="<string>" \
auto_deleted="<string>" \
delivery_mode="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
AMQP service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing AMQP service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_amqp <mc admin config get>` to
review the currently configured AMQP endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the AMQP service endpoint.
For example:
``amqp://user:password@hostname:port``
See :ref:`AMQP Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-amqp>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured AMQP
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:amqp
You must specify the ARN resource when configuring bucket notifications with
the associated AMQP deployment as a target.
3) Configure Bucket Notifications using the AMQP Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured AMQP service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:amqp \
--event EVENTS
- Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:amqp
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the AMQP service for the notification data. The action required
depends on which :mc-cmd:`events <mc-event-add-event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an AMQP Endpoint in a MinIO Deployment
---------------------------------------------
The following procedure updates an existing AMQP service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
AMQP 0-9-1 Service Endpoint
+++++++++++++++++++++++++++
MinIO relies on the :github:`streadway/amqp` project for AMQP connectivity. The
project is primarily tested against `RabbitMQ <https://www.rabbitmq.com/>`__
deployments, though other `AMQP 0-9-1-compatible <https://www.amqp.org/>`__
services *may* also work. This procedure *assumes* a RabbitMQ deployment
as the service endpoint.
If the AMQP service requires authentication, you *must* provide an appropriate
username and password during the configuration process to grant MinIO access
to the service.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured AMQP Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured AMQP service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_amqp
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_amqp:primary delivery_mode="0" exchange_type="" no_wait="off" queue_dir="" queue_limit="0" url="amqp://user:password@hostname:port" auto_deleted="off" durable="off" exchange="" internal="off" mandatory="off" routing_key=""
notify_amqp:secondary delivery_mode="0" exchange_type="" no_wait="off" queue_dir="" queue_limit="0" url="amqp://user:password@hostname:port" auto_deleted="off" durable="off" exchange="" internal="off" mandatory="off" routing_key=""
The :mc-conf:`notify_amqp` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-amqp`. The
:mc-conf:`url <notify_amqp.url>` key specifies the AMQP service endpoint
for the given `notify_amqp` key. The ``notify_amqp:<IDENTIFIER>`` suffix
describes the unique identifier for that AMQP service endpoint.
Note the identifier for the AMQP service endpoint you want to update for
the next step.
2) Update the AMQP Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the AMQP service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_amqp:<IDENTIFIER> \
url="amqp://user:password@hostname:port" \
exchange="<string>" \
exchange_type="<string>" \
routing_key="<string>" \
mandatory="<string>" \
durable="<string>" \
no_wait="<string>" \
internal="<string>" \
auto_deleted="<string>" \
delivery_mode="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_amqp url <notify_amqp.url>` configuration setting is the
*minimum* required for an AMQP service endpoint. All other configuration
settings are *optional*. See :ref:`minio-server-config-bucket-notification-amqp`
for a complete list of AMQP configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured AMQP
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:amqp
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
AMQP service endpoint and check the AMQP service for the notification data. The
action required depends on which :mc-cmd:`events <mc-event-add-event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

363
source/monitoring/bucket-notifications/publish-events-to-elasticsearch.rst Normal file
View File

@ -0,0 +1,363 @@
.. _minio-bucket-notifications-publish-elasticsearch:
===============================
Publish Events to Elasticsearch
===============================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to an
`Elasticsearch <https://www.elastic.co/>`__ service endpoint.
MinIO relies on the :github:`olivere/elastic` v7 project for Elastic
connectivity. The ``elastic/v7`` library specifically targets Elasticsearch
v7.0 and is *not compatible with earlier Elasticsearch versions*.
Add a Elasticsearch Endpoint to a MinIO Deployment
--------------------------------------------------
The following procedure adds a new Elasticsearch service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~
Elasticsearch v7.0 and later
++++++++++++++++++++++++++++
MinIO relies on the :github:`olivere/elastic` v7 project for Elastic
connectivity. The ``elastic/v7`` library specifically targets Elasticsearch
v7.0 and is *not compatible with earlier Elasticsearch versions*.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the Elasticsearch Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new Elasticsearch service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tabs::
.. tab:: Environment Variables
MinIO supports specifying the Elasticsearch service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-elasticsearch>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an Elasticsearch service endpoint. The minimum
*required* variables are:
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_ENABLE`
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_URL`
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_INDEX`
- :envvar:`MINIO_NOTIFY_ELASTICSEARCH_FORMAT`
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_ELASTICSEARCH_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_ELASTICSEARCH_URL_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_ELASTICSEARCH_INDEX_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_FORMAT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_USERNAME_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_PASSWORD_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_ELASTICSEARCH_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
TARGET service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing Elasticsearch
service endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_elasticsearch <mc admin config get>`
to review the currently configured Elasticsearch endpoints on the MinIO
deployment.
- Replace ``<ENDPOINT>`` with the URL of the Elasticsearch service endpoint.
For example:
See :ref:`Elasticsearch Service for Bucket Notifications
<minio-server-envvar-bucket-notification-elasticsearch>` for complete
documentation on each environment variable.
.. tab:: Configuration Settings
MinIO supports adding or updating Elasticsearch endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_elasticsearch` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
Elasticsearch service endpoint. The minimum *required* settings are:
- :mc-conf:`url <notify_elasticsearch.url>`
- :mc-conf:`index <notify_elasticsearch.index>`
- :mc-conf:`format <notify_elasticsearch.format>`
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_elasticsearch:IDENTIFIER \
url="ENDPOINT" \
index="<string>" \
format="<string>" \
username="<string>" \
password="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
Elasticsearch service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing Elasticsearch service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_elasticsearch <mc admin config get>`
to review the currently configured Elasticsearch endpoints on the MinIO
deployment.
- Replace ``ENDPOINT`` with the URL of the Elasticsearch service endpoint.
For example:
``https://user:password@hostname:port``
See :ref:`Elasticsearch Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-elasticsearch>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured Elasticsearch
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:elasticsearch
You must specify the ARN resource when configuring bucket notifications with
the associated Elasticsearch deployment as a target.
3) Configure Bucket Notifications using the Elasticsearch Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured Elasticsearch service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:elasticsearch \
--event EVENTS
- Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:Elasticsearch
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the Elasticsearch service for the notification data. The action required
depends on which :mc-cmd:`events <mc-event-add-event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an Elasticsearch Endpoint in a MinIO Deployment
------------------------------------------------------
The following procedure updates an existing Elasticsearch service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
Elasticsearch v7.0 and later
++++++++++++++++++++++++++++
MinIO relies on the :github:`olivere/elastic` v7 project for Elastic
connectivity. The ``elastic/v7`` library specifically targets Elasticsearch
v7.0 and is *not compatible with earlier Elasticsearch versions*.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured Elasticsearch Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured Elasticsearch service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_elasticsearch
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_elasticsearch:primary queue_dir="" queue_limit="0" url="https://user:password@hostname:port" format="namespace" index=""
notify_elasticsearch:secondary queue_dir="" queue_limit="0" url="https://user:password@hostname:port" format="namespace" index=""
The :mc-conf:`notify_elasticsearch` key is the top-level configuration key for
an :ref:`minio-server-config-bucket-notification-elasticsearch`. The
:mc-conf:`url <notify_elasticsearch.url>` key specifies the Elasticsearch
service endpoint for the given `notify_elasticsearch` key. The
``notify_elasticsearch:<IDENTIFIER>`` suffix describes the unique identifier for
that Elasticsearch service endpoint.
Note the identifier for the Elasticsearch service endpoint you want to update for
the next step.
2) Update the Elasticsearch Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the Elasticsearch service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_elasticsearch:<IDENTIFIER> \
url="https://user:password@hostname:port" \
index="<string>" \
format="<string>" \
username="<string>" \
password="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_elasticsearch url <notify_elasticsearch.url>` configuration
setting is the *minimum* required for an Elasticsearch service endpoint. All
other configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-elasticsearch` for a complete list
of Elasticsearch configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured
Elasticsearch target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:elasticsearch
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
Elasticsearch service endpoint and check the Elasticsearch service for the
notification data. The action required depends on which
:mc-cmd:`events <mc-event-add-event>` were specified when configuring the bucket
notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

375
source/monitoring/bucket-notifications/publish-events-to-kafka.rst Normal file
View File

@ -0,0 +1,375 @@
.. _minio-bucket-notifications-publish-kafka:
=======================
Publish Events to Kafka
=======================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a `Kafka <https://kafka.apache.org/>`__
service endpoint.
MinIO relies on the :github:`Shopify/sarama` project for Kafka connectivity
and shares that project's Kafka support. See the
``sarama`` :github:`Compatibility and API stability
<Shopify/sarama/#compatibility-and-api-stability>` section for more details.
Add a Kafka Endpoint to a MinIO Deployment
------------------------------------------
The following procedure adds a new Kafka service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~
Kafka Minimum Versions and Supported Versions
+++++++++++++++++++++++++++++++++++++++++++++
MinIO relies on the :github:`Shopify/sarama` project for Kafka connectivity
and shares that project's Kafka support. See the
``sarama`` :github:`Compatibility and API stability
<Shopify/sarama/#compatibility-and-api-stability>` section for more details.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the Kafka Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new Kafka service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tabs::
.. tab:: Environment Variables
MinIO supports specifying the Kafka service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-kafka>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring a Kafka service endpoint. The minimum
*required* variables are
:envvar:`MINIO_NOTIFY_KAFKA_ENABLE` and
:envvar:`MINIO_NOTIFY_KAFKA_BROKERS`:
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_KAFKA_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_KAFKA_BROKERS_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_KAFKA_TOPIC_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_SASL_USERNAME_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_SASL_PASSWORD_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_SASL_MECHANISM_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_TLS_CLIENT_AUTH_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_SASL_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_TLS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_TLS_SKIP_VERIFY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_CLIENT_TLS_CERT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_CLIENT_TLS_KEY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_VERSION_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_KAFKA_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
Kafka service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing Kafka service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_kafka <mc admin config get>` to
review the currently configured Kafka endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with a comma-separated list of Kafka brokers.
For example:
``"kafka1.example.com:2021,kafka2.example.com:2021"``
See :ref:`Kafka Service for Bucket Notifications
<minio-server-envvar-bucket-notification-kafka>` for complete documentation
on each environment variable.
.. tab:: Configuration Settings
MinIO supports adding or updating Kafka endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_kafka` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
Kafka service endpoint. The minimum *required* setting is
:mc-conf:`notify_kafka brokers <notify_kafka.brokers>`:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_kafka:IDENTIFIER \
brokers="<ENDPOINT>" \
topic="<string>" \
sasl_username="<string>" \
sasl_password="<string>" \
sasl_mechanism="<string>" \
tls_client_auth="<string>" \
sasl="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
client_tls_cert="<string>" \
client_tls_key="<string>" \
version="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
Kafka service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing Kafka service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_kafka <mc admin config get>` to
review the currently configured Kafka endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with a comma separated list of Kafka brokers.
For example:
``"kafka1.example.com:2021,kafka2.example.com:2021"``
See :ref:`Kafka Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-kafka>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured
Kafka target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:kafka
You must specify the ARN resource when configuring bucket notifications with
the associated Kafka deployment as a target.
3) Configure Bucket Notifications using the Kafka Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured Kafka service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:kafka \
--event EVENTS
- Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:kafka
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the Kafka service for the notification data. The action required
depends on which :mc-cmd:`events <mc-event-add-event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update a Kafka Endpoint in a MinIO Deployment
---------------------------------------------
The following procedure updates an existing Kafka service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
Kafka Minimum Versions and Supported Versions
+++++++++++++++++++++++++++++++++++++++++++++
MinIO relies on the :github:`Shopify/sarama` project for Kafka connectivity
and shares that project's Kafka support. See the
``sarama`` :github:`Compatibility and API stability
<Shopify/sarama/#compatibility-and-api-stability>` section for more details.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured Kafka Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured Kafka service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_kafka
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_kafka:primary tls_skip_verify="off" queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" brokers="" topic="" client_tls_cert="" client_tls_key="" version=""
notify_kafka:secondary tls_skip_verify="off" queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" brokers="" topic="" client_tls_cert="" client_tls_key="" version=""
The :mc-conf:`notify_kafka` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-kafka`. The
:mc-conf:`brokers <notify_kafka.brokers>` key specifies the Kafka service
endpoint for the given `notify_kafka` key. The ``notify_kafka:<IDENTIFIER>``
suffix describes the unique identifier for that Kafka service endpoint.
Note the identifier for the Kafka service endpoint you want to update for
the next step.
2) Update the Kafka Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the Kafka service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_kafka:<IDENTIFIER> \
url="https://kafka1.example.net:9200, https://kafka2.example.net:9200" \
exchange="<string>" \
exchange_type="<string>" \
routing_key="<string>" \
mandatory="<string>" \
durable="<string>" \
no_wait="<string>" \
internal="<string>" \
auto_deleted="<string>" \
delivery_mode="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_kafka brokers <notify_kafka.brokers>` configuration setting
is the *minimum* required for a Kafka service endpoint. All other configuration
settings are *optional*. See
:ref:`minio-server-config-bucket-notification-kafka` for a complete list of
Kafka configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured
Kafka target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:kafka
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
Kafka service endpoint and check the Kafka service for the notification data.
The action required depends on which :mc-cmd:`events <mc-event-add-event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

389
source/monitoring/bucket-notifications/publish-events-to-mqtt.rst Normal file
View File

@ -0,0 +1,389 @@
.. _minio-bucket-notifications-publish-mqtt:
======================
Publish Events to MQTT
======================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to `MQTT <https://www.mqtt.org/>`__
server/broker endpoint.
Add an MQTT Endpoint to a MinIO Deployment
------------------------------------------
The following procedure adds a new MQTT service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
MQTT 3.1 or 3.1.1 Server/Broker
+++++++++++++++++++++++++++++++
This procedure assumes an existing MQTT 3.1 or 3.1.1 server/broker to which the
MinIO deployment has connectivity. See the
`mqtt.org software listing <https://mqtt.org/software/>`__ for a list of
MQTT-compatible server/brokers.
If the MQTT service requires authentication, you *must* provide an appropriate
username and password during the configuration process to grant MinIO access
to the service.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the MQTT Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new MQTT service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tabs::
.. tab:: Environment Variables
MinIO supports specifying the MQTT service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-mqtt>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an MQTT service endpoint. The minimum *required*
variables are:
- :envvar:`MINIO_NOTIFY_MQTT_ENABLE`
- :envvar:`MINIO_NOTIFY_MQTT_BROKER`
- :envvar:`MINIO_NOTIFY_MQTT_TOPIC`
- :envvar:`MINIO_NOTIFY_MQTT_USERNAME` *Required if the MQTT server/broker enforces authentication/authorization*
- :envvar:`MINIO_NOTIFY_MQTT_PASSWORD` *Required if the MQTT server/broker enforces authentication/authorization*
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_MQTT_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_MQTT_BROKER_<IDENTIFIER>="ENDPOINT"
set MINIO_NOTIFY_MQTT_TOPIC_<IDENTIFIER>="TOPIC"
set MINIO_NOTIFY_MQTT_USERNAME_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_PASSWORD_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_QOS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_KEEP_ALIVE_INTERVAL_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_RECONNECT_INTERVAL_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MQTT_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
MQTT service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new MQTT service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing MQTT service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_mqtt <mc admin config get>` to
review the currently configured MQTT endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the MQTT service endpoint.
For example:
``tcp://hostname:port``
- Replace ``TOPIC`` with the MQTT topic to which MinIO associates
events published to the server/broker.
See :ref:`MQTT Service for Bucket Notifications
<minio-server-envvar-bucket-notification-mqtt>` for complete documentation
on each environment variable.
.. tab:: Configuration Settings
MinIO supports adding or updating MQTT endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_mqtt` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
MQTT service endpoint. The following configuration settings are the
*minimum* required for an MQTT server/broker endpoint:
- :mc-conf:`~notify_mqtt.broker`
- :mc-conf:`~notify_mqtt.topic`
- :mc-conf:`~notify_mqtt.username` *Required if the MQTT server/broker enforces authentication/authorization*
- :mc-conf:`~notify_mqtt.password` *Required if the MQTT server/broker enforces authentication/authorization*
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_mqtt:IDENTIFIER \
broker="ENDPOINT" \
topic="TOPIC" \
username="username" \
password="password" \
qos="<integer>" \
keep_alive_interval="60s|m|h|d"
reconnect_interval="60s|m|h|d"
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
MQTT service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing MQTT service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_mqtt <mc admin config get>` to
review the currently configured MQTT endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the MQTT service endpoint.
For example:
``tcp://hostname:port``
- Replace ``TOPIC`` with the MQTT topic to which MinIO associates
events published to the server/broker.
See :ref:`MQTT Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-mqtt>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured MQTT
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:mqtt
You must specify the ARN resource when configuring bucket notifications with
the associated MQTT deployment as a target.
3) Configure Bucket Notifications using the MQTT Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured MQTT service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:mqtt \
--event EVENTS
- Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:MQTT
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the MQTT service for the notification data. The action required
depends on which :mc-cmd:`events <mc-event-add-event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an MQTT Endpoint in a MinIO Deployment
---------------------------------------------
The following procedure updates an existing MQTT service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
MQTT 3.1 or 3.1.1 Server/Broker Endpoint
++++++++++++++++++++++++++++++++++++++++
This procedure assumes an existing MQTT 3.1 or 3.1.1 server/broker to which the
MinIO deployment has connectivity. See the
`mqtt.org software listing <https://mqtt.org/software/>`__ for a list of
MQTT-compatible server/brokers.
If the MQTT service requires authentication, you *must* provide an appropriate
username and password during the configuration process to grant MinIO access
to the service.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured MQTT Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured MQTT service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_mqtt
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_mqtt:primary broker="tcp://mqtt-primary.example.net:port" password="" queue_dir="" queue_limit="0" reconnect_interval="0s" keep_alive_interval="0s" qos="0" topic="" username=""
notify_mqtt:secondary broker="tcp://mqtt-primary.example.net:port" password="" queue_dir="" queue_limit="0" reconnect_interval="0s" keep_alive_interval="0s" qos="0" topic="" username=""
The :mc-conf:`notify_mqtt` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-mqtt`. The
:mc-conf:`broker <notify_mqtt.broker>` key specifies the MQTT server/broker endpoint
for the given `notify_mqtt` key. The ``notify_mqtt:<IDENTIFIER>`` suffix
describes the unique identifier for that MQTT service endpoint.
Note the identifier for the MQTT service endpoint you want to update for
the next step.
2) Update the MQTT Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the MQTT service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_mqtt:<IDENTIFIER> \
url="MQTT://user:password@hostname:port" \
exchange="<string>" \
exchange_type="<string>" \
routing_key="<string>" \
mandatory="<string>" \
durable="<string>" \
no_wait="<string>" \
internal="<string>" \
auto_deleted="<string>" \
delivery_mode="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The following configuration settings are the *minimum* required for an
MQTT server/broker endpoint:
- :mc-conf:`~notify_mqtt.broker`
- :mc-conf:`~notify_mqtt.topic`
- :mc-conf:`~notify_mqtt.username` *Required if the MQTT server/broker enforces authentication/authorization*
- :mc-conf:`~notify_mqtt.password` *Required if the MQTT server/broker enforces authentication/authorization*
All other configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-mqtt` for a complete list of MQTT
configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured MQTT
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:mqtt
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
MQTT service endpoint and check the MQTT service for the notification data. The
action required depends on which :mc-cmd:`events <mc-event-add-event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

362
source/monitoring/bucket-notifications/publish-events-to-mysql.rst Normal file
View File

@ -0,0 +1,362 @@
.. _minio-bucket-notifications-publish-mysql:
=======================
Publish Events to MySQL
=======================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a My`MySQL <https://www.mysql.com/>`__
service endpoint. MinIO supports MySQL 5.7.8 and later *only*.
Add a MySQL Endpoint to a MinIO Deployment
------------------------------------------
The following procedure adds a new MySQL service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~
MySQL 5.7.8 and later
+++++++++++++++++++++
MinIO relies on features introduced with MySQL 5.7.8.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the MySQL Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new MySQL service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tabs::
.. tab:: Environment Variables
MinIO supports specifying the MySQL service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-mysql>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring a MySQL service endpoint. The minimum
*required* variables are:
- :envvar:`MINIO_NOTIFY_MYSQL_ENABLE`
- :envvar:`MINIO_NOTIFY_MYSQL_DSN_STRING`
- :envvar:`MINIO_NOTIFY_MYSQL_TABLE`
- :envvar:`MINIO_NOTIFY_MYSQL_FORMAT`
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_MYSQL_DSN_STRING_<IDENTIFIER>="on"
set MINIO_NOTIFY_MYSQL_TABLE_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_MYSQL_FORMAT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MYSQL_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MYSQL_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_MYSQL_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
MySQL service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing MySQL service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_mysql <mc admin config get>` to
review the currently configured MySQL endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the DSN of the MySQL service endpoint.
MinIO expects the following format:
``<user>:<password>@tcp(<host>:<port>)/<database>``
For example:
``"username:password@tcp(mysql.example.com:3306)/miniodb"``
See :ref:`MySQL Service for Bucket Notifications
<minio-server-envvar-bucket-notification-mysql>` for complete documentation
on each environment variable.
.. tab:: Configuration Settings
MinIO supports adding or updating MySQL endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_mysql` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
MySQL service endpoint. The minimum *required* settings are:
- :mc-conf:`notify_mysql dsn_string <notify_mysql.dsn_string>`
- :mc-conf:`notify_mysql table <notify_mysql.table>`
- :mc-conf:`notify_mysql format <notify_mysql.format>`
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_mysql:IDENTIFIER \
dsn_string="<ENDPOINT>" \
table="<string>" \
format="<string>" \
max_open_connections="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
MySQL service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing MySQL service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_mysql <mc admin config get>` to
review the currently configured MySQL endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the DSN of the MySQL service endpoint.
MinIO expects the following format:
``<user>:<password>@tcp(<host>:<port>)/<database>``
For example:
``"username:password@tcp(mysql.example.com:3306)/miniodb"``
See :ref:`MySQL Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-mysql>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured MySQL
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:mysql
You must specify the ARN resource when configuring bucket notifications with
the associated MySQL deployment as a target.
3) Configure Bucket Notifications using the MySQL Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured MySQL service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:mysql \
--event EVENTS
- Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:mysql
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the MySQL service for the notification data. The action required
depends on which :mc-cmd:`events <mc-event-add-event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update a MySQL Endpoint in a MinIO Deployment
---------------------------------------------
The following procedure updates an existing MySQL service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
MySQL 5.7.8 and later
+++++++++++++++++++++
MinIO relies on features introduced with MySQL 5.7.8.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured MySQL Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured MySQL service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_mysql
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_mysql:primary format="namespace" table="minio_images" dsn_string="user:pass@tcp(mysql.example.com:3306)/miniodb"
notify_mysql:secondary format="namespace" table="minio_images" dsn_string="user:pass@tcp(mysql.example.com:3306)/miniodb"
The :mc-conf:`notify_mysql` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-mysql`. The
:mc-conf:`dsn_string <notify_mysql.dsn_string>` key specifies the MySQL service
endpoint for the given `notify_mysql` key. The ``notify_mysql:<IDENTIFIER>``
suffix describes the unique identifier for that MySQL service endpoint.
Note the identifier for the MySQL service endpoint you want to update for
the next step.
2) Update the MySQL Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the MySQL service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_mysql:IDENTIFIER \
dsn_string="<ENDPOINT>" \
table="<string>" \
format="<string>" \
max_open_connections="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The following configuration settings are the *minimum required* for a MySQL
service endpoint:
- :mc-conf:`notify_mysql dsn_string <notify_mysql.dsn_string>`
- :mc-conf:`notify_mysql table <notify_mysql.table>`
- :mc-conf:`notify_mysql format <notify_mysql.format>`
All other configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-mysql` for a complete list of
MySQL configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured MySQL
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:mysql
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
MySQL service endpoint and check the MySQL service for the notification data. The
action required depends on which :mc-cmd:`events <mc-event-add-event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

362
source/monitoring/bucket-notifications/publish-events-to-nats.rst Normal file
View File

@ -0,0 +1,362 @@
.. _minio-bucket-notifications-publish-nats:
======================
Publish Events to NATS
======================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a
`NATS <https://nats.io/>`__ service endpoint.
Add a NATS Endpoint to a MinIO Deployment
-----------------------------------------
The following procedure adds a new NATS service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the NATS Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new NATS service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tabs::
.. tab:: Environment Variables
MinIO supports specifying the NATS service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-nats>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an NATS service endpoint. The minimum
*required* variables are
:envvar:`MINIO_NOTIFY_NATS_ADDRESS` and
:envvar:`MINIO_NOTIFY_NATS_SUBJECT`:
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_NATS_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_NATS_ADDRESS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_SUBJECT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_USERNAME_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_PASSWORD_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_TOKEN_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_TLS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_TLS_SKIP_VERIFY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_PING_INTERVAL_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_STREAMING_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_STREAMING_ASYNC_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_STREAMING_MAX_PUB_ACKS_IN_FLIGHT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_STREAMING_CLUSTER_ID_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_CERT_AUTHORITY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_CLIENT_CERT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_CLIENT_KEY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NATS_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
NATS service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing NATS service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_nats <mc admin config get>` to
review the currently configured NATS endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the NATS service endpoint.
For example: ``htpps://nats-endpoint.example.com:4222``
See :ref:`NATS Service for Bucket Notifications
<minio-server-envvar-bucket-notification-nats>` for complete documentation
on each environment variable.
.. tab:: Configuration Settings
MinIO supports adding or updating NATS endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_nats` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
NATS service endpoint. The minimum *required* setting are
:mc-conf:`notify_nats address <notify_nats.address>` and
:mc-conf:`notify_nats subject <notify_nats.subject>`:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_nats:IDENTIFIER \
address="HOSTNAME" \
subject="<string>" \
username="<string>" \
password="<string>" \
token="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
ping_interval="<string>" \
streaming="<string>" \
streaming_async="<string>" \
streaming_max_pub_acks_in_flight="<string>" \
streaming_cluster_id="<string>" \
cert_authority="<string>" \
client_cert="<string>" \
client_key="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
NATS service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing NATS service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_nats <mc admin config get>` to
review the currently configured NATS endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the NATS service endpoint.
For example: ``htpps://nats-endpoint.example.com:4222``.
See :ref:`NATS Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-nats>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured NATS
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:nats
You must specify the ARN resource when configuring bucket notifications with
the associated NATS deployment as a target.
3) Configure Bucket Notifications using the NATS Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured NATS service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:nats \
--event EVENTS
- Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:nats
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the NATS service for the notification data. The action required
depends on which :mc-cmd:`events <mc-event-add-event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an NATS Endpoint in a MinIO Deployment
---------------------------------------------
The following procedure updates an existing NATS service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured NATS Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured NATS service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_nats
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_nats:primary password="yoursecret" streaming_max_pub_acks_in_flight="10" subject="" address="nats-endpoint.example.com:4222" token="" username="yourusername" ping_interval="0" queue_limit="0" tls="off" tls_skip_verify="off" streaming_async="on" queue_dir="" streaming_cluster_id="test-cluster" streaming_enable="on"
notify_nats:secondary password="yoursecret" streaming_max_pub_acks_in_flight="10" subject="" address="nats-endpoint.example.com:4222" token="" username="yourusername" ping_interval="0" queue_limit="0" tls="off" tls_skip_verify="off" streaming_async="on" queue_dir="" streaming_cluster_id="test-cluster" streaming_enable="on"
The :mc-conf:`notify_nats` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-nats`. The
:mc-conf:`address <notify_nats.address>` key specifies the NATS service endpoint
for the given ``notify_nats`` key. The ``notify_nats:<IDENTIFIER>`` suffix
describes the unique identifier for that NATS service endpoint.
Note the identifier for the NATS service endpoint you want to update for
the next step.
2) Update the NATS Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the NATS service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_nats:IDENTIFIER \
address="HOSTNAME" \
subject="<string>" \
username="<string>" \
password="<string>" \
token="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
ping_interval="<string>" \
streaming="<string>" \
streaming_async="<string>" \
streaming_max_pub_acks_in_flight="<string>" \
streaming_cluster_id="<string>" \
cert_authority="<string>" \
client_cert="<string>" \
client_key="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_nats address <notify_nats.address>` configuration setting
is the *minimum* required for an NATS service endpoint. All other configuration
settings are *optional*. See :ref:`minio-server-config-bucket-notification-nats`
for a complete list of NATS configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured NATS
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:nats
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
NATS service endpoint and check the NATS service for the notification data. The
action required depends on which :mc-cmd:`events <mc-event-add-event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

333
source/monitoring/bucket-notifications/publish-events-to-nsq.rst Normal file
View File

@ -0,0 +1,333 @@
.. _minio-bucket-notifications-publish-nsq:
=====================
Publish Events to NSQ
=====================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to `NSQ <https://nsq.io/>`__
service endpoint.
Add a NSQ Endpoint to a MinIO Deployment
----------------------------------------
The following procedure adds a new NSQ service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the NSQ Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new NSQ service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tabs::
.. tab:: Environment Variables
MinIO supports specifying the NSQ service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-nsq>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an NSQ service endpoint. The minimum
*required* variables are
:envvar:`MINIO_NOTIFY_NSQ_NSQD_ADDRESS` and
:envvar:`MINIO_NOTIFY_NSQ_TOPIC`:
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_NSQ_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_NSQ_NSQD_ADDRESS_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_NSQ_TOPIC_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NSQ_TLS_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NSQ_TLS_SKIP_VERIFY_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NSQ_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NSQ_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_NSQ_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
TARGET service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing NSQ service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_nsq <mc admin config get>` to
review the currently configured NSQ endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the NSQ service endpoint.
For example, ``https://nsq-service.example.com:4150``.
See :ref:`NSQ Service for Bucket Notifications
<minio-server-envvar-bucket-notification-nsq>` for complete documentation
on each environment variable.
.. tab:: Configuration Settings
MinIO supports adding or updating NSQ endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_nsq` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
NSQ service endpoint. The minimum *required* setting is
:mc-conf:`notify_nsq nsqd_address <notify_nsq.nsqd_address>` and
:mc-conf:`notify_nsq topic <notify_nsq.topic>`:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_nsq:IDENTIFIER \
nsqd_address="ENDPOINT" \
topic="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
NSQ service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing NSQ service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_nsq <mc admin config get>` to
review the currently configured NSQ endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the NSQ service endpoint.
For example:
``NSQ://user:password@hostname:port``
See :ref:`NSQ Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-nsq>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured NSQ
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:nsq
You must specify the ARN resource when configuring bucket notifications with
the associated NSQ deployment as a target.
3) Configure Bucket Notifications using the NSQ Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured NSQ service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:nsq \
--event EVENTS
- Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:nsq
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the NSQ service for the notification data. The action required
depends on which :mc-cmd:`events <mc-event-add-event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an NSQ Endpoint in a MinIO Deployment
--------------------------------------------
The following procedure updates an existing NSQ service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured NSQ Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured NSQ service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_nsq
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_nsq:primary nsqd_address="https://nsq.example.com" queue_dir="" queue_limit="0" tls="off" tls_skip_verify="off" topic=""
notify_nsq:secondary nsqd_address="https://nsq.example.com" queue_dir="" queue_limit="0" tls="off" tls_skip_verify="off" topic=""
The :mc-conf:`notify_nsq` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-nsq`. The :mc-conf:`nsqd_address
<notify_nsq.nsqd_address>` key specifies the NSQ service endpoint for the given
`notify_nsq` key. The ``notify_nsq:<IDENTIFIER>`` suffix describes the unique
identifier for that NSQ service endpoint.
Note the identifier for the NSQ service endpoint you want to update for
the next step.
2) Update the NSQ Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the NSQ service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_nsq:<IDENTIFIER> \
nsqd_address="NSQ://user:password@hostname:port" \
topic="<string>" \
tls="<string>" \
tls_skip_verify="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_nsq nsqd_address <notify_nsq.nsqd_address>` configuration
setting is the *minimum* required for an NSQ service endpoint. All other
configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-nsq` for a complete list of NSQ
configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured NSQ
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:NSQ
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
NSQ service endpoint and check the NSQ service for the notification data. The
action required depends on which :mc-cmd:`events <mc-event-add-event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

367
source/monitoring/bucket-notifications/publish-events-to-postgresql.rst Normal file
View File

@ -0,0 +1,367 @@
.. _minio-bucket-notifications-publish-postgresql:
============================
Publish Events to PostgreSQL
============================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
.. |postgresql-uri-reference| replace:: `PostgreSQL Connection String <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING>`__
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to
`PostgreSQL <https://www.postgresql.org/>`__. MinIO supports
PostgreSQL 9.5 and later *only*.
Add a PostgreSQL Endpoint to a MinIO Deployment
-----------------------------------------------
The following procedure adds a new PostgreSQL service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~
PostgreSQL 9.5 and later
++++++++++++++++++++++++
MinIO relies on features introduced with PostgreSQL 9.5.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the PostgreSQL Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new PostgreSQL service endpoint using either environment
variables *or* by setting runtime configuration settings.
.. tabs::
.. tab:: Environment Variables
MinIO supports specifying the PostgreSQL service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-postgresql>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring a PostgreSQL service endpoint. The minimum
*required* variables are:
- :envvar:`MINIO_NOTIFY_POSTGRESQL_CONNECTION_STRING`
- :envvar:`MINIO_NOTIFY_POSTGRESQL_TABLE`
- :envvar:`MINIO_NOTIFY_POSTGRESQL_FORMAT`
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_CONNECTION_STRING_<IDENTIFIER>="on"
set MINIO_NOTIFY_TABLE_<IDENTIFIER>="ENDPOINT"
set MINIO_NOTIFY_FORMAT_<IDENTIFIER>=""
set MINIO_NOTIFY_MAX_OPEN_CONNECTIONS_<IDENTIFIER>=""
set MINIO_NOTIFY_QUEUE_DIR_<IDENTIFIER>=""
set MINIO_NOTIFY_QUEUE_LIMIT_<IDENTIFIER>=""
set MINIO_NOTIFY_COMMENT_<IDENTIFIER>=""
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
PostgreSQL service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing PostgreSQL service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_postgresql <mc admin config get>` to
review the currently configured PostgreSQL endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the |postgresql-uri-reference|
for PostgreSQL service endpoint. MinIO supports ``key=value`` format for
the connection string. For example:
``"host=https://postgresql.example.com port=5432 ..."``
For more complete documentation on supported PostgreSQL connection
string parameters, see |postgresql-uri-reference|.
See :ref:`PostgreSQL Service for Bucket Notifications
<minio-server-envvar-bucket-notification-postgresql>` for complete
documentation on each environment variable.
.. tab:: Configuration Settings
MinIO supports adding or updating PostgreSQL endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_postgresql` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
PostgreSQL service endpoint. The minimum *required* setting are:
- :mc-conf:`notify_postgresql connection_string
<notify_postgresql.connection_string>`
- :mc-conf:`notify_postgresql table <notify_postgresql.table>`
- :mc-conf:`notify_postgresql format <notify_postgresql.format>`
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_postgresql:IDENTIFIER \
connection_string="ENDPOINT" \
table="<string>" \
format="<string>" \
max_open_connections="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
PostgreSQL service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing PostgreSQL service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_postgresql <mc admin config get>` to
review the currently configured PostgreSQL endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the `PostgreSQL URI connection string
<https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING>`__
of the PostgreSQL service endpoint. MinIO supports ``key=value`` format
for the PostgreSQL connection string. For example:
``"host=https://postgresql.example.com port=5432 ..."``
For more complete documentation on supported PostgreSQL connection
string parameters, see |postgresql-uri-reference|.
See :ref:`PostgreSQL Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-postgresql>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured PostgreSQL
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:postgresql
You must specify the ARN resource when configuring bucket notifications with
the associated PostgreSQL deployment as a target.
3) Configure Bucket Notifications using the PostgreSQL Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured PostgreSQL service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:postgresql \
--event EVENTS
- Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:postgresql
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the PostgreSQL service for the notification data. The action required
depends on which :mc-cmd:`events <mc-event-add-event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update a PostgreSQL Endpoint in a MinIO Deployment
---------------------------------------------------
The following procedure updates an existing PostgreSQL service endpoint for
supporting :ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~
PostgreSQL 9.5 and later
++++++++++++++++++++++++
MinIO relies on features introduced with PostgreSQL 9.5.
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured PostgreSQL Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured PostgreSQL service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_postgresql
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_postgresql:primary queue_dir="" connection_string="postgresql://" queue_limit="0" table="" format="namespace"
notify_postgresql:secondary queue_dir="" connection_string="" queue_limit="0" table="" format="namespace"
The :mc-conf:`notify_postgresql` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-postgresql`. The
:mc-conf:`connection_string <notify_postgresql.connection_string>` key specifies
the PostgreSQL service endpoint for the given `notify_postgresql` key. The
``notify_postgresql:<IDENTIFIER>`` suffix describes the unique identifier for
that PostgreSQL service endpoint.
Note the identifier for the PostgreSQL service endpoint you want to update for
the next step.
2) Update the PostgreSQL Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the PostgreSQL service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_postgresql:IDENTIFIER \
connection_string="ENDPOINT" \
table="<string>" \
format="<string>" \
max_open_connections="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The following configuration settings are the *minimum* required for a
PostgreSQL service endpoint:
- :mc-conf:`notify_postgresql connection_string
<notify_postgresql.connection_string>`
- :mc-conf:`notify_postgresql table <notify_postgresql.table>`
- :mc-conf:`notify_postgresql format <notify_postgresql.format>`
All other configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-postgresql` for a complete list of
PostgreSQL configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured PostgreSQL
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:postgresql
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
PostgreSQL service endpoint and check the PostgreSQL service for the notification data. The
action required depends on which :mc-cmd:`events <mc-event-add-event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

335
source/monitoring/bucket-notifications/publish-events-to-redis.rst Normal file
View File

@ -0,0 +1,335 @@
.. _minio-bucket-notifications-publish-redis:
=======================
Publish Events to Redis
=======================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a `Redis <https://redis.io/>`__
service endpoint.
Add a Redis Endpoint to a MinIO Deployment
-------------------------------------------
The following procedure adds a new Redis service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the Redis Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new Redis service endpoint using either environment variables
*or* by setting runtime configuration settings.
.. tabs::
.. tab:: Environment Variables
MinIO supports specifying the Redis service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-redis>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an Redis service endpoint. The minimum
*required* variables are:
- :envvar:`MINIO_NOTIFY_REDIS_ENABLE`
- :envvar:`MINIO_NOTIFY_REDIS_ADDRESS`
- :envvar:`MINIO_NOTIFY_REDIS_KEY`
- :envvar:`MINIO_NOTIFY_REDIS_FORMAT`
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_REDIS_ENABLE_<IDENTIFIER>="on"
set MINIO_NOTIFY_REDIS_ADDRESS_<IDENTIFIER>="<ENDPOINT>"
set MINIO_NOTIFY_REDIS_KEY_<IDENTIFIER>="<STRING>"
set MINIO_NOTIFY_REDIS_FORMAT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_REDIS_PASSWORD_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_REDIS_QUEUE_DIR_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_REDIS_QUEUE_LIMIT_<IDENTIFIER>="<string>"
set MINIO_NOTIFY_REDIS_COMMENT_<IDENTIFIER>="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
TARGET service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing Redis service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_redis <mc admin config get>` to
review the currently configured Redis endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the Redis service endpoint.
For example: ``https://redis.example.com:6369``
See :ref:`Redis Service for Bucket Notifications
<minio-server-envvar-bucket-notification-redis>` for complete documentation
on each environment variable.
.. tab:: Configuration Settings
MinIO supports adding or updating Redis endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_redis` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
Redis service endpoint. The minimum *required* settings are:
- :mc-conf:`notify_redis address <notify_redis.address>`
- :mc-conf:`notify_redis key <notify_redis.key>`
- :mc-conf:`notify_redis format <notify_redis.format>`
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_redis:IDENTIFIER \
address="ENDPOINT" \
key="<string>" \
format="<string>" \
password="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
Redis service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing Redis service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_redis <mc admin config get>` to
review the currently configured Redis endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the Redis service endpoint.
For example: ``https://redis.example.com:6369``
See :ref:`Redis Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-redis>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured Redis
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:redis
You must specify the ARN resource when configuring bucket notifications with
the associated Redis deployment as a target.
3) Configure Bucket Notifications using the Redis Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured Redis service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:redis \
--event EVENTS
- Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:redis
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the Redis service for the notification data. The action required
depends on which :mc-cmd:`events <mc-event-add-event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an Redis Endpoint in a MinIO Deployment
----------------------------------------------
The following procedure updates an existing Redis service endpoint for
supporting :ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured Redis Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured Redis service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_redis
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_redis:primary address="https://redis.example.com:6369" format="namespace" key="minioevent" password="" queue_dir="" queue_limit="0"
notify_redis:secondary address="https://redis.example.com:6369" format="namespace" key="minioevent" password="" queue_dir="" queue_limit="0"
The :mc-conf:`notify_redis` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-redis`. The
:mc-conf:`address <notify_redis.address>` key specifies the Redis service endpoint
for the given `notify_redis` key. The ``notify_redis:<IDENTIFIER>`` suffix
describes the unique identifier for that Redis service endpoint.
Note the identifier for the Redis service endpoint you want to update for
the next step.
2) Update the Redis Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the Redis service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_redis:IDENTIFIER \
address="ENDPOINT" \
key="<string>" \
format="<string>" \
password="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
comment="<string>"
The :mc-conf:`notify_redis address <notify_redis.address>` configuration setting
is the *minimum* required for an Redis service endpoint. All other configuration
settings are *optional*. See
:ref:`minio-server-config-bucket-notification-redis` for a complete list of
Redis configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured Redis
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:redis
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
Redis service endpoint and check the Redis service for the notification data. The
action required depends on which :mc-cmd:`events <mc-event-add-event>` were
specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET

333
source/monitoring/bucket-notifications/publish-events-to-webhook.rst Normal file
View File

@ -0,0 +1,333 @@
.. _minio-bucket-notifications-publish-webhook:
=========================
Publish Events to Webhook
=========================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
MinIO supports publishing :ref:`bucket notification
<minio-bucket-notifications>` events to a
`Webhook <https://en.wikipedia.org/wiki/Webhook>`__ service endpoint.
Add a Webhook Endpoint to a MinIO Deployment
--------------------------------------------
The following procedure adds a new Webhook service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) Add the Webhook Endpoint to MinIO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can configure a new Webhook service endpoint using either environment
variables *or* by setting runtime configuration settings.
.. tabs::
.. tab:: Environment Variables
MinIO supports specifying the Webhook service endpoint and associated
configuration settings using
:ref:`environment variables
<minio-server-envvar-bucket-notification-webhook>`. The
:mc:`minio server` process applies the specified settings on its
next startup.
The following example code sets *all* environment variables
related to configuring an Webhook service endpoint. The minimum
*required* variables are
:envvar:`MINIO_NOTIFY_WEBHOOK_ENABLE` and
:envvar:`MINIO_NOTIFY_WEBHOOK_ENDPOINT`:
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_WEBHOOK_ENABLE_<IDENTIFIER>_="on"
set MINIO_NOTIFY_WEBHOOK_ENDPOINT_<IDENTIFIER>_="ENDPOINT"
set MINIO_NOTIFY_WEBHOOK_AUTH_TOKEN_<IDENTIFIER>_="<string>"
set MINIO_NOTIFY_WEBHOOK_QUEUE_DIR_<IDENTIFIER>_="<string>"
set MINIO_NOTIFY_WEBHOOK_QUEUE_LIMIT_<IDENTIFIER>_="<string>"
set MINIO_NOTIFY_WEBHOOK_CLIENT_CERT_<IDENTIFIER>_="<string>"
set MINIO_NOTIFY_WEBHOOK_CLIENT_KEY_<IDENTIFIER>_="<string>"
set MINIO_NOTIFY_WEBHOOK_COMMENT_<IDENTIFIER>_="<string>"
- Replace ``<IDENTIFIER>`` with a unique descriptive string for the
Webhook service endpoint. Use the same ``<IDENTIFIER>`` value for all
environment variables related to the new target service endpoint.
The following examples assume an identifier of ``PRIMARY``.
If the specified ``<IDENTIFIER>`` matches an existing Webhook service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_webhook <mc admin config get>` to
review the currently configured Webhook endpoints on the MinIO deployment.
- Replace ``<ENDPOINT>`` with the URL of the Webhook service endpoint.
For example:
``https://webhook.example.com``
See :ref:`Webhook Service for Bucket Notifications
<minio-server-envvar-bucket-notification-webhook>` for complete documentation
on each environment variable.
.. tab:: Configuration Settings
MinIO supports adding or updating Webhook endpoints on a running
:mc:`minio server` process using the :mc-cmd:`mc admin config set` command
and the :mc-conf:`notify_webhook` configuration key. You must restart the
:mc:`minio server` process to apply any new or updated configuration
settings.
The following example code sets *all* settings related to configuring an
Webhook service endpoint. The minimum *required* setting is
:mc-conf:`notify_webhook endpoint <notify_webhook.endpoint>`:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_webhook:IDENTIFIER \
endpoint="<ENDPOINT>" \
auth_token="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
client_cert="<string>" \
client_key="<string>" \
comment="<string>"
- Replace ``IDENTIFIER`` with a unique descriptive string for the
Webhook service endpoint. The following examples in this procedure
assume an identifier of ``PRIMARY``.
If the specified ``IDENTIFIER`` matches an existing Webhook service
endpoint on the MinIO deployment, the new settings *override*
any existing settings for that endpoint. Use
:mc-cmd:`mc admin config get notify_webhook <mc admin config get>` to
review the currently configured Webhook endpoints on the MinIO deployment.
- Replace ``ENDPOINT`` with the URL of the Webhook service endpoint.
For example:
``https://webhook.example.com``
See :ref:`Webhook Bucket Notification Configuration Settings
<minio-server-config-bucket-notification-webhook>` for complete
documentation on each setting.
2) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured
Webhook target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:webhook
You must specify the ARN resource when configuring bucket notifications with
the associated Webhook deployment as a target.
3) Configure Bucket Notifications using the Webhook Endpoint as a Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc event add` command to add a new bucket notification
event with the configured Webhook service as a target:
.. code-block:: shell
:class: copyable
mc event add ALIAS/BUCKET arn:minio:sqs::primary:webhook \
--event EVENTS
- Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of a MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket in which to configure the
event.
- Replace ``EVENTS`` with a comma-separated list of :ref:`events
<mc-event-supported-events>` for which MinIO triggers notifications.
Use :mc-cmd:`mc event list` to view all configured bucket events for
a given notification target:
.. code-block:: shell
:class: copyable
mc event list ALIAS/BUCKET arn:minio:sqs::primary:webhook
4) Validate the Configured Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on the bucket for which you configured the new event and
check the Webhook service for the notification data. The action required
depends on which :mc-cmd:`events <mc-event-add-event>` were specified
when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET
Update an Webhook Endpoint in a MinIO Deployment
------------------------------------------------
The following procedure updates an existing Webhook service endpoint for supporting
:ref:`bucket notifications <minio-bucket-notifications>` in a MinIO
deployment.
.. include:: /includes/common-admonitions.rst
:start-after: start-restart-downtime
:end-before: end-restart-downtime
Prerequisites
~~~~~~~~~~~~~~
MinIO ``mc`` Command Line Tool
++++++++++++++++++++++++++++++
This procedure uses the :mc:`mc` command line tool for certain actions.
See the ``mc`` :ref:`Quickstart <mc-install>` for installation instructions.
1) List Configured Webhook Endpoints In The Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config get` command to list the currently
configured Webhook service endpoints in the deployment:
.. code-block:: shell
:class: copyable
mc admin config get ALIAS/ notify_webhook
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the MinIO deployment.
The command output resembles the following:
.. code-block:: shell
notify_webhook:primary endpoint="https://webhook.example.com" auth_token="" queue_limit="0" queue_dir="" client_cert="" client_key=""
notify_webhook:secondary endpoint="https://webhook.example.com" auth_token="" queue_limit="0" queue_dir="" client_cert="" client_key=""
The :mc-conf:`notify_webhook` key is the top-level configuration key for an
:ref:`minio-server-config-bucket-notification-webhook`. The
:mc-conf:`endpoint <notify_webhook.endpoint>` key specifies the Webhook service
endpoint for the given `notify_webhook` key. The ``notify_webhook:<IDENTIFIER>``
suffix describes the unique identifier for that Webhook service endpoint.
Note the identifier for the Webhook service endpoint you want to update for
the next step.
2) Update the Webhook Endpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin config set` command to set the new configuration
for the Webhook service endpoint:
.. code-block:: shell
:class: copyable
mc admin config set ALIAS/ notify_webhook:IDENTIFIER \
endpoint="<ENDPOINT>" \
auth_token="<string>" \
queue_dir="<string>" \
queue_limit="<string>" \
client_cert="<string>" \
client_key="<string>" \
comment="<string>"
The :mc-conf:`notify_webhook endpoint <notify_webhook.endpoint>` configuration
setting is the *minimum* required for an Webhook service endpoint. All other
configuration settings are *optional*. See
:ref:`minio-server-config-bucket-notification-webhook` for a complete list of
Webhook configuration settings.
3) Restart the MinIO Deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You must restart the MinIO deployment to apply the configuration changes.
Use the :mc-cmd:`mc admin service restart` command to restart the deployment.
.. important::
MinIO restarts *all* :mc:`minio server` processes associated to the
deployment at the same time. Applications may experience a brief period of
downtime during the restart process.
Consider scheduling the restart during a maintenance period to minimize
interruption of services.
.. code-block:: shell
:class: copyable
mc admin service restart ALIAS
Replace ``ALIAS`` with the :mc:`alias <mc-alias>` of the deployment to
restart.
The :mc:`minio server` process prints a line on startup for each configured Webhook
target similar to the following:
.. code-block:: shell
SQS ARNs: arn:minio:sqs::primary:webhook
3) Validate the Changes
~~~~~~~~~~~~~~~~~~~~~~~
Perform an action on a bucket which has an event configuration using the updated
Webhook service endpoint and check the Webhook service for the notification
data. The action required depends on which :mc-cmd:`events <mc-event-add-event>`
were specified when configuring the bucket notification.
For example, if the bucket notification configuration includes the
``s3:ObjectCreated:Put`` event, you can use the
:mc-cmd:`mc cp` command to create a new object in the bucket and trigger
a notification.
.. code-block:: shell
:class: copyable
mc cp ~/data/new-object.txt ALIAS/BUCKET