4edb3f7bae
# Summary As per a recent customer issue, our guidance on `MINIO_DOMAIN` and path/virtual bucket lookups needs some attention. There are two main areas to address: 1. We need to guide users to avoid namespace collision within the `MINIO_DOMAIN` , as this causes bucket lookup issues with certain MinIO services/features 2. We need to generally improve docs on setting `MINIO_DOMAIN` correctly There is also a side objective from Engineering to ensure we have a simple admonition to direct users to test wherever possible changes to config settings before applying to production. --------- Co-authored-by: Daryl White <53910321+djwfyi@users.noreply.github.com>
12 KiB
Kafka Notification Settings
minio
Table of Contents
This page documents settings for configuring an Kafka service as a
target for Bucket Notifications <minio-bucket-notifications>.
See minio-bucket-notifications-publish-kafka for a
tutorial on using these settings.
Multiple Kafka Targets
You can specify multiple Kafka service endpoints by appending a
unique identifier _ID for each set of related Kafka
settings on to the top level key.
Examples
For example, the following commands set two distinct Kafka service
endpoints as PRIMARY and SECONDARY
respectively:
Environment Variable
set MINIO_NOTIFY_KAFKA_ENABLE_PRIMARY="on"
set MINIO_NOTIFY_KAFKA_BROKERS_PRIMARY="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
set MINIO_NOTIFY_KAFKA_ENABLE_SECONDARY="on"
set MINIO_NOTIFY_KAFKA_BROKERS_SECONDARY="https://kafka1.example.net:9200, https://kafka2.example.net:9200"Configuration Setting
mc admin config set notify_kafka:primary \
brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
[ARGUMENT=VALUE ...]
mc admin config set notify_kafka:secondary \
brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
[ARGUMENT=VALUE ...]Notice that for configuration settings, the unique identifier appends
to notify_kafka only, not to each individual argument.
Settings
Enable
Required
Environment Variable
MINIO_NOTIFY_KAFKA_ENABLE
Specify on to enable publishing bucket notifications to
a Kafka service endpoint.
Defaults to off.
Configuration Setting
notify_kafka
The top-level configuration key for defining an Kafka service
endpoint for use with MinIO bucket notifications <minio-bucket-notifications>.
Use mc admin config set to set or update an Kafka
service endpoint. The ~notify_kafka.brokers argument is
required for each target. Specify additional optional arguments
as a whitespace (" ")-delimited list.
mc admin config set notify_kafka \
brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
[ARGUMENT="VALUE"] ... \Brokers
Required
Environment Variable
MINIO_NOTIFY_KAFKA_BROKERS
Configuration Setting
notify_kafka brokers
Specify a comma-separated list of Kafka broker addresses. For example:
"kafka1.example.com:2021,kafka2.example.com:2021"
Topic
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_TOPIC
Configuration Setting
notify_kafka topic
Specify the name of the Kafka topic to which MinIO publishes bucket events.
SASL
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_SASL
Configuration Setting
notify_kafka sasl
Specify on to enable SASL authentication.
SASL Username
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_SASL_USERNAME
Configuration Setting
notify_kafka sasl_username
Specify the username for performing SASL/PLAIN or SASL/SCRAM authentication to the Kafka broker(s).
SASL Password
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_SASL_PASSWORD
Configuration Setting
notify_kafka sasl_password
Specify the password for performing SASL/PLAIN or SASL/SCRAM authentication to the Kafka broker(s).
RELEASE.2023-06-23T20-26-00Z
MinIO redacts this value when returned as part of mc admin config get.
SASL Mechanism
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_SASL_MECHANISM
Configuration Setting
notify_kafka sasl_mechanism
Specify the SASL mechanism to use for authenticating to the Kafka broker(s). MinIO supports the following mechanisms:
PLAIN(Default)SHA256SHA512
TLS Client Auth
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_TLS_CLIENT_AUTH
Configuration Setting
notify_kafka tls_client_auth
Specify the client authentication type of the Kafka broker(s). The following table lists the supported values and their mappings
| Value | Authentication Type |
|---|---|
| 0 | NoClientCert |
| 1 | RequestClientCert |
| 2 | RequireAnyClientCert |
| 3 | VerifyClientCertIfGiven |
| 4 | RequireAndVerifyClientCert |
See ClientAuthType for more information on each client auth type.
TLS
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_TLS
Configuration Setting
notify_kafka tls
Specify on to enable TLS connectivity to the Kafka
broker(s).
TLS Skip Verify
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_TLS_SKIP_VERIFY
Configuration Setting
notify_kafka tls_skip_verify
Enables or disables TLS verification of the NATS service endpoint TLS certificates.
- Specify
onto disable TLS verification (Default). - Specify
offto enable TLS verification.
Client TLS Cert
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_CLIENT_TLS_CERT
Configuration Setting
notify_kafka client_tls_cert
Specify the path to the client certificate to use for performing mTLS authentication to the Kafka broker(s).
Client TLS Key
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_CLIENT_TLS_KEY
Configuration Setting
notify_kafka client_tls_key
Specify the path to the client private key to use for performing mTLS authentication to the Kafka broker(s).
Version
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_VERSION
Configuration Setting
notify_kafka version
Specify the version of the Kafka cluster to assume when performing operations against that cluster. See the sarama reference documentation for more information on this field's behavior.
Batch Size
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_BATCH_SIZE
Configuration Setting
notify_kafka batch_size
Specify the integer value to use as the batch size for sending records to Kafka.
RELEASE.2023-12-02T10-51-33Z
MinIO previously limited this value to 100.
Queue Directory
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_QUEUE_DIR
Configuration Setting
notify_kafka queue_dir
Specify the directory path to enable MinIO's persistent event store
for undelivered messages, such as /opt/minio/events.
MinIO stores undelivered events in the specified store while the Kafka server/broker is offline and replays the stored events when connectivity resumes.
Queue Limit
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_QUEUE_LIMIT
Configuration Setting
notify_kafka queue_limit
Specify the maximum limit for undelivered messages. Defaults to
100000.
Comment
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_COMMENT
Configuration Setting
notify_kafka comment
Specify a comment to associate with the Kafka configuration.
Compression Codec
MinIO Server RELEASE.2023-12-09T18-17-51Z
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_PRODUCER_COMPRESSION_CODEC
Configuration Setting
notify_kafka compression_codec
Specify the compression codec to use when sending records to Kafka.
Supports the following values:
nonesnappygziplz4zstd
Compression Level
MinIO Server RELEASE.2023-12-09T18-17-51Z
Optional
Environment Variable
MINIO_NOTIFY_KAFKA_PRODUCER_COMPRESSION_LEVEL
Configuration Setting
notify_kafka compression_level
Controls the level of compression applied based on the configured compression codec.
Specify an integer value greater than or equal to 0. The
effect of the value depends on the selected codec.