minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-08-06 14:42:56 +03:00
Code Activity
Files
relax-event-restriction
docs/source/reference/minio-server/settings/notifications/kafka.rst
Ravind Kumar 4edb3f7bae Improve documentation on MINIO_DOMAIN (#1131) 
# Summary

As per a recent customer issue, our guidance on `MINIO_DOMAIN` and
path/virtual bucket lookups needs some attention.

There are two main areas to address:

1. We need to guide users to avoid namespace collision within the
`MINIO_DOMAIN` , as this causes bucket lookup issues with certain MinIO
services/features
2. We need to generally improve docs on setting `MINIO_DOMAIN` correctly

There is also a side objective from Engineering to ensure we have a
simple admonition to direct users to test wherever possible changes to
config settings before applying to production.

---------

Co-authored-by: Daryl White <53910321+djwfyi@users.noreply.github.com>
2024-02-14 17:16:58 -05:00

527 lines
12 KiB
ReStructuredText
Raw Permalink Blame History

.. _minio-server-envvar-bucket-notification-kafka:
.. _minio-server-config-bucket-notification-kafka:
===========================
Kafka Notification Settings
===========================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
This page documents settings for configuring an Kafka service as a target for :ref:`Bucket Notifications <minio-bucket-notifications>`.
See :ref:`minio-bucket-notifications-publish-kafka` for a tutorial on using these settings.
.. include:: /includes/common-mc-admin-config.rst
:start-after: start-minio-settings-defined
:end-before: end-minio-settings-defined
.. include:: /includes/common-mc-admin-config.rst
:start-after: start-minio-settings-test-before-prod
:end-before: end-minio-settings-test-before-prod
Multiple Kafka Targets
----------------------
You can specify multiple Kafka service endpoints by appending a unique identifier ``_ID`` for each set of related Kafka settings on to the top level key.
Examples
~~~~~~~~
For example, the following commands set two distinct Kafka service endpoints as ``PRIMARY`` and ``SECONDARY`` respectively:
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. code-block:: shell
:class: copyable
set MINIO_NOTIFY_KAFKA_ENABLE_PRIMARY="on"
set MINIO_NOTIFY_KAFKA_BROKERS_PRIMARY="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
set MINIO_NOTIFY_KAFKA_ENABLE_SECONDARY="on"
set MINIO_NOTIFY_KAFKA_BROKERS_SECONDARY="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
.. tab-item:: Configuration Setting
:sync: config
.. code-block:: shell
mc admin config set notify_kafka:primary \
brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
[ARGUMENT=VALUE ...]
mc admin config set notify_kafka:secondary \
brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
[ARGUMENT=VALUE ...]
Notice that for configuration settings, the unique identifier appends to ``notify_kafka`` only, not to each individual argument.
Settings
--------
Enable
~~~~~~
*Required*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_ENABLE
Specify ``on`` to enable publishing bucket notifications to a Kafka service endpoint.
Defaults to ``off``.
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka
The top-level configuration key for defining an Kafka service endpoint for use with :ref:`MinIO bucket notifications <minio-bucket-notifications>`.
Use :mc-cmd:`mc admin config set` to set or update an Kafka service endpoint.
The :mc-conf:`~notify_kafka.brokers` argument is *required* for each target.
Specify additional optional arguments as a whitespace (``" "``)-delimited list.
.. code-block:: shell
:class: copyable
mc admin config set notify_kafka \
brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
[ARGUMENT="VALUE"] ... \
Brokers
~~~~~~~
*Required*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_BROKERS
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka brokers
:delimiter: " "
Specify a comma-separated list of Kafka broker addresses.
For example:
``"kafka1.example.com:2021,kafka2.example.com:2021"``
.. include:: /includes/linux/minio-server.rst
:start-after: start-notify-target-online-desc
:end-before: end-notify-target-online-desc
Topic
~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_TOPIC
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka topic
:delimiter: " "
Specify the name of the Kafka topic to which MinIO publishes bucket events.
SASL
~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_SASL
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka sasl
:delimiter: " "
Specify ``on`` to enable SASL authentication.
SASL Username
~~~~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_SASL_USERNAME
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka sasl_username
:delimiter: " "
Specify the username for performing SASL/PLAIN or SASL/SCRAM authentication to the Kafka broker(s).
SASL Password
~~~~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_SASL_PASSWORD
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka sasl_password
:delimiter: " "
Specify the password for performing SASL/PLAIN or SASL/SCRAM authentication to the Kafka broker(s).
.. versionchanged:: RELEASE.2023-06-23T20-26-00Z
MinIO redacts this value when returned as part of :mc-cmd:`mc admin config get`.
SASL Mechanism
~~~~~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_SASL_MECHANISM
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka sasl_mechanism
:delimiter: " "
Specify the SASL mechanism to use for authenticating to the Kafka broker(s).
MinIO supports the following mechanisms:
- ``PLAIN`` (Default)
- ``SHA256``
- ``SHA512``
TLS Client Auth
~~~~~~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_TLS_CLIENT_AUTH
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka tls_client_auth
:delimiter: " "
Specify the client authentication type of the Kafka broker(s).
The following table lists the supported values and their mappings
.. list-table::
:header-rows: 1
:widths: 20 80
:width: 100%
* - Value
- Authentication Type
* - 0
- ``NoClientCert``
* - 1
- ``RequestClientCert``
* - 2
- ``RequireAnyClientCert``
* - 3
- ``VerifyClientCertIfGiven``
* - 4
- ``RequireAndVerifyClientCert``
See `ClientAuthType <https://golang.org/pkg/crypto/tls/#ClientAuthType>`__ for more information on each client auth type.
TLS
~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_TLS
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka tls
:delimiter: " "
Specify ``on`` to enable TLS connectivity to the Kafka broker(s).
TLS Skip Verify
~~~~~~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_TLS_SKIP_VERIFY
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka tls_skip_verify
:delimiter: " "
Enables or disables TLS verification of the NATS service endpoint TLS certificates.
- Specify ``on`` to disable TLS verification *(Default)*.
- Specify ``off`` to enable TLS verification.
Client TLS Cert
~~~~~~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_CLIENT_TLS_CERT
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka client_tls_cert
:delimiter: " "
Specify the path to the client certificate to use for performing mTLS authentication to the Kafka broker(s).
Client TLS Key
~~~~~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_CLIENT_TLS_KEY
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka client_tls_key
:delimiter: " "
Specify the path to the client private key to use for performing mTLS authentication to the Kafka broker(s).
Version
~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_VERSION
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka version
:delimiter: " "
Specify the version of the Kafka cluster to assume when performing operations against that cluster.
See the `sarama reference documentation <https://github.com/shopify/sarama/blob/v1.20.1/config.go#L327>`__ for more information on this field's behavior.
Batch Size
~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_BATCH_SIZE
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka batch_size
:delimiter: " "
Specify the integer value to use as the `batch size <https://kafka.apache.org/documentation/#producerconfigs_batch.size>`__ for sending records to Kafka.
.. versionchanged:: RELEASE.2023-12-02T10-51-33Z
MinIO previously limited this value to ``100``.
Queue Directory
~~~~~~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_QUEUE_DIR
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka queue_dir
:delimiter: " "
Specify the directory path to enable MinIO's persistent event store for undelivered messages, such as ``/opt/minio/events``.
MinIO stores undelivered events in the specified store while the Kafka server/broker is offline and replays the stored events when connectivity resumes.
Queue Limit
~~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_QUEUE_LIMIT
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka queue_limit
:delimiter: " "
Specify the maximum limit for undelivered messages.
Defaults to ``100000``.
Comment
~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_COMMENT
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka comment
:delimiter: " "
Specify a comment to associate with the Kafka configuration.
Compression Codec
~~~~~~~~~~~~~~~~~
.. versionadded:: MinIO Server RELEASE.2023-12-09T18-17-51Z
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_PRODUCER_COMPRESSION_CODEC
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka compression_codec
:delimiter: " "
Specify the compression codec to use when sending records to Kafka.
Supports the following values:
- ``none``
- ``snappy``
- ``gzip``
- ``lz4``
- ``zstd``
Compression Level
~~~~~~~~~~~~~~~~~
.. versionadded:: MinIO Server RELEASE.2023-12-09T18-17-51Z
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_PRODUCER_COMPRESSION_LEVEL
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka compression_level
:delimiter: " "
Controls the level of compression applied based on the configured compression codec.
Specify an integer value greater than or equal to ``0``.
The effect of the value depends on the selected codec.
View Git Blame Copy Permalink