docs/source/kubernetes/minio-operator-reference.rst

MinIO Kubernetes Operator

minio

Table of Contents

Overview

The MinIO Kubernetes Operator ("MinIO Operator") brings native support for deploying and managing MinIO deployments ("MinIO Tenant") on a Kubernetes cluster.

The MinIO Operator requires familiarity with interacting with a Kubernetes cluster, including but not limited to using the kubectl command line tool and interacting with Kubernetes YAML objects. Users who would prefer a more simplified experience should use the minio-kubernetes for deploying and managing MinIO Tenants.

Deploying the MinIO Operator

The following operations deploy the MinIO operator using kustomize templates. Users who would prefer a more simplified deployment experience that does not require familiarity with kustomize should use the minio-kubernetes for deploying and managing MinIO Tenants.

kubectl

Use the following command to deploy the MinIO Operator using kubectl and kustomize templates:

kubectl apply -k github.com/minio/operator/\?ref\=|minio-operator-latest-version|

kustomize

Use kustomize <kubernetes-sigs/kustomize> to deploy the MinIO Operator using kustomize templates:

kustomize build github.com/minio/operator/\?ref\=|minio-operator-latest-version| \
   > minio-operator-|minio-operator-latest-version|.yaml

MinIO Tenant Object

The following example Kubernetes object describes a MinIO Tenant with the following resources:

• 4 minio server processes.
• 4 Volumes per server.
• 2 MinIO Console Service (MCS) processes.

apiVersion: minio.min.io/v1
kind: Tenant
metadata:
  creationTimestamp: null
  name: minio-tenant-1
  namespace: minio-tenant-1
scheduler:
  name: ""
spec:
  certConfig: {}
  console:
    consoleSecret:
      name: minio-tenant-1-console-secret
    image: minio/console:v0.3.14
    metadata:
      creationTimestamp: null
      name: minio-tenant-1
    replicas: 2
    resources: {}
  credsSecret:
    name: minio-tenant-1-creds-secret
  image: minio/minio:RELEASE.2020-09-26T03-44-56Z
  imagePullSecret: {}
  liveness:
    initialDelaySeconds: 10
    periodSeconds: 1
    timeoutSeconds: 1
  mountPath: /export
  requestAutoCert: true
  serviceName: minio-tenant-1-internal-service
  zones:
  - resources: {}
    servers: 4
    volumeClaimTemplate:
      apiVersion: v1
      kind: persistentvolumeclaims
      metadata:
        creationTimestamp: null
      spec:
        accessModes:
        - ReadWriteOnce
        storageClassName: local-storage
        resources:
          requests:
            storage: 10Gi
      status: {}
    volumesPerServer: 4

MinIO Operator YAML Reference

The MinIO Operator adds a CustomResourceDefinition <#customresourcedefinition-v1-apiextensions-k8s-io> that extends the Kubernetes Object API to support creating MinIO Tenant objects.

All Top-Level Fields

The following YAML block describes a MinIO Tenant object and its top-level fields.

apiVersion: minio.min.io/v1 kind: Tenant metadata: ~metadata.name: minio ~metadata.namespace: <string> ~metadata.labels: app: minio ~metadata.annotations: prometheus.io/path: <string> prometheus.io/port: "<string>" prometheus.io/scrape: "<bool>" spec: ~spec.certConfig: <object> ~spec.console: <object> ~spec.credsSecret: <object> ~spec.env: <object> ~spec.externalCertSecret: <array> ~spec.externalClientCertSecret: <object> ~spec.image: minio/minio:latest ~spec.imagePullPolicy: IfNotPresent ~spec.kes: <object> ~spec.mountPath: <string> ~spec.podManagementPolicy: <string> ~spec.priorityClassName: <string> ~spec.requestAutoCert: <boolean> ~spec.s3: <object> ~spec.securityContext: <object> ~spec.serverSet: <array> ~spec.serviceAccountName: <string> ~spec.subPath: <string> ~spec.serverSet: <array>

Minimum Required Fields

Minimum Required Fields

apiVersion: minio.min.io/v1 kind: Tenant metadata: ~metadata.name: minio ~metadata.labels: app: minio spec: ~spec.serverSet : - ~spec.serverSet.servers : <int> ~spec.serverSet.volumeClaimTemplate: ~spec.serverSet.volumeClaimTemplate.spec: ~spec.serverSet.volumeClaimTemplate.spec.accessModes: <string> ~spec.serverSet.volumeClaimTemplate.spec.resources: requests: storage: <string> ~spec.serverSet.volumesPerServer: <int>

Core Fields

The following fields describe the core settings used to deploy a MinIO Tenant.

apiVersion: minio.min.io/v1 kind: Tenant metadata: ~metadata.name: <string> ~metadata.namespace: <string> ~metadata.labels: app: minio ~metadata.annotations: - prometheus.io/path: <string> - prometheus.io/port: <string> - prometheus.io/scrape: <string> spec: ~spec.credsSecret: <object> ~spec.env: <object>

~spec.serverSet:

• ~spec.serverSet.affinity: <object> ~spec.serverSet.name: <string> ~spec.serverSet.nodeSelector: <object> ~spec.serverSet.resources: <object> ~spec.serverSet.servers: <int> ~spec.serverSet.tolerations: <array> ~spec.serverSet.volumeClaimTemplate: <object> ~spec.serverSet.volumesPerServer: <integer>

apiVersion

Required

The API Version of the MinIO Tenant Object.

Specify minio.min.io/v1.

kind

Required

The REST resource the object represents. Specify Tenant.

metadata

The root field for describing metadata related to the Tenant object.

metadata.name

Required

The name of the Tenant resource. The name must be unique within the target namespace.

metadata.namespace

Required

The namespace in which Kubernetes deploys the Tenant resource. Omit to use the "Default" namespace. MinIO recommends creating a namespace for each MinIO Tenant deployed in the Kubernetes cluster.

metadata.labels

The Kubernetes labels <concepts/overview/working-with-objects/labels> to apply to the MinIO Tenant Object.

Specify at minimum the following key-value pair:

metadata:
   labels:
      app: minio

metadata.annotations

One or more Kubernetes annotations <user-guide/annotations> to associate with the MinIO Tenant Object.

MinIO Tenants support the following annotations:

• prometheus.io/path: <string>
• prometheus.io/port: <string>
• prometheus.io/scrape: <bool>

spec

The root field for the MinIO Tenant Specification.

spec.credsSecret

The Kubernetes secret containing values to use for setting the MinIO access key (MINIO_ACCESS_KEY) and secret key (MINIO_SECRET_KEY). The MinIO Operator automatically generates the secret along with values for the access and secret key if this field is omitted.

Specify an object where the name field contains the name of the Kubernetes secret to use:

spec:
   credsSecret:
      name: minio-secret

The Kubernetes secret should contain the following values:

• data.accesskey - the Access Key for each minio server in the Tenant.
• data.secretkey - the Secret Key for each minio server in the Tenant.

spec.env

The environment variables available for use by the MinIO Tenant.

spec.mountPath

Optional

The mount path for Persistent Volumes bound to minio pods in the MinIO Tenant.

Defaults to /export.

spec.s3

Optional

The S3-related features enabled on the MinIO Tenant.

Specify any of the following supported features as part of the ~spec.s3 object:

• bucketDNS: <boolean> - specify true to enable DNS lookup of buckets on the MinIO Tenant.

spec.subPath

Optional

The sub path appended to the spec.mountPath. The resulting full path is the directory in which MinIO stores data.

For example, given a ~spec.mountPath of export and a ~spec.subPath of minio, the full mount path is export/minio.

Defaults to empty ("").

spec.serverSet

Required

The configuration for each MinIO Server Set deployed in the MinIO Tenant. A Server Set consists of one or more minio servers.

Each element in the ~spec.serverSet array is an object that must contain the following fields:

• ~spec.serverSet.servers
• ~spec.serverSet.volumeClaimTemplate
• ~spec.serverSet.volumesPerServer

~spec.serverSet must have at least one element in the array.

spec.serverSet.affinity

Optional

The configuration for node affinity, pod affinity, and pod anti-affinity applied to each pod in the Server Set.

spec.serverSet.name

Optional

The name of the MinIO Server Set object.

The MinIO Operator automatically generates the Server Set name if this field is omitted.

spec.serverSet.nodeSelector

Optional

The filter to apply when selecting which node or nodes on which to deploy each pod in the Server Set. See the Kubernetes documentation on Assigning Pods to Nodes <concepts/scheduling-eviction/assign-pod-node> for more information.

spec.serverSet.resources

Optional

The resources <concepts/configuration/manage-resources-containers/> each pod in the Server Set requests.

spec.serverSet.servers

Required

The number of minio pods to deploy in the Zone.

The minimum number of servers is 2. MinIO recommends a minimum of 4 servers for optimal availability and distribution of data in the Server Set.

spec.serverSet.tolerations

Optional

The Tolerations <concepts/scheduling-eviction/taint-and-toleration/> applied to pods deployed in the Server Set.

spec.serverSet.volumeClaimTemplate

Required

The configuration template to apply to each Persistent Volume Claim (PVC) created as part of the Server Set.

See spec.serverSet.volumeClaimTemplate for more complete documentation on the full specification of the volumeClaimTemplate object.

The MinIO Operator calculates the number of PVC to generate by multiplying spec.serverSet.volumesPerServer by spec.serverSet.servers.

spec.serverSet.volumesPerServer

Required

The number of Persistent Volume Claims (PVC) to create for each server <spec.serverSet.servers> in the Server Set.

The total number of volumes in the Server Set must be greater than 4. Specifically:

~spec.serverSet.servers X ~spec.serverSet.volumesPerServer > 4

The MinIO Operator calculates the number of PVC to generate by multiplying spec.serverSet.volumesPerServer by spec.serverSet.servers.

Volume Claim Template

The following fields describe the template used to generate Persistent Volume Claims (PVC) for use in the MinIO Tenant.

spec:

serverSet: - ~spec.serverSet.volumeClaimTemplate ~spec.serverSet.volumeClaimTemplate.apiVersion: <string> ~spec.serverSet.volumeClaimTemplate.kind: <string> ~spec.serverSet.volumeClaimTemplate.metadata: <object> ~spec.serverSet.volumeClaimTemplate.spec: ~spec.serverSet.volumeClaimTemplate.spec.accessModes: <array> ~spec.serverSet.volumeClaimTemplate.spec.dataSource: <object> ~spec.serverSet.volumeClaimTemplate.spec.resources: <object> ~spec.serverSet.volumeClaimTemplate.spec.selector: <object> ~spec.serverSet.volumeClaimTemplate.spec.storageClassName: <string> ~spec.serverSet.volumeClaimTemplate.spec.volumeMode: <string> ~spec.serverSet.volumeClaimTemplate.spec.volumeName: <string> status: <object>

spec.serverSet.volumeClaimTemplate

Required

The configuration template to apply to each Persistent Volume Claim (PVC) created as part of a Server Set <spec.serverSet>. The ~spec.serverSet.volumeClaimTemplate dictates which Persistent Volumes (PV) the generated PVC can bind to.

The ~spec.serverSet.volumeClaimTemplate requires at minimum the following fields:

• ~spec.serverSet.volumeClaimTemplate.spec.resources
• ~spec.serverSet.volumeClaimTemplate.spec.accessModes

The MinIO Operator calculates the number of PVC to generate by multiplying spec.serverSet.volumesPerServer by spec.serverSet.servers.

spec.serverSet.volumeClaimTemplate.apiVersion

Optional

The API Version of the ~spec.serverSet.volumeClaimTemplate.

Specify minio.min.io/v1.

spec.serverSet.volumeClaimTemplate.kind

Optional

The REST resource the object represents.

spec.serverSet.volumeClaimTemplate.metadata

Optional

The metadata for the ~spec.serverSet.volumeClaimTemplate.

spec.serverSet.volumeClaimTemplate.spec

The specification applied to each Persistent Volume Claim (PVC) created using the ~spec.serverSet.volumeClaimTemplate.

spec.serverSet.volumeClaimTemplate.spec.accessModes

Required

The desired access mode <concepts/storage/persistent-volumes#access-modes-1> for each Persistent Volume Claim (PVC) created using the ~spec.serverSet.volumeClaimTemplate.

spec.serverSet.volumeClaimTemplate.spec.dataSource

Optional

The data source to use for each Persistent Volume Claim (PVC) created using the ~spec.serverSet.volumeClaimTemplate.

spec.serverSet.volumeClaimTemplate.spec.resources

Required

The resources requested by each Persistent Volume Claim (PVC) created using the ~spec.serverSet.volumeClaimTemplate.

The ~spec.serverSet.volumeClaimTemplate.spec.resources object must include a requests.storage object:

spec:
   serverSet:
      - name: minio-server-set-1
        volumeClaimTemplate:
           spec: 
              resources:
                 requests:
                    storage: <string>

The following table lists the supported units for the storage capacity.

Suffix	Unit Size
k	KB (Kilobyte, 1000 Bytes)
m	MB (Megabyte, 1000 Kilobytes)
g	GB (Gigabyte, 1000 Megabytes)
t	TB (Terrabyte, 1000 Gigabytes)
ki	KiB (Kibibyte, 1024 Bites)
mi	MiB (Mebibyte, 1024 Kibibytes)
gi	GiB (Gibibyte, 1024 Mebibytes)
ti	TiB (Tebibyte, 1024 Gibibytes)

spec.serverSet.volumeClaimTemplate.spec.selector

Optional

The selector logic to apply when querying available Persistent Volumes (PV) for binding to the Persistent Volume Claim (PVC).

spec.serverSet.volumeClaimTemplate.spec.storageClassName

Optional

The storage class to apply to each Persistent Volume Claim (PVC) created using the ~spec.serverSet.volumeClaimTemplate.

spec.serverSet.volumeClaimTemplate.spec.volumeMode

Optional

The type of Persistent Volume (PV) required by the claim. Defaults to Filesystem if omitted.

spec.serverSet.volumeClaimTemplate.spec.volumeName

Optional

The name to apply to each Persistent Volume Claim (PVC) created using the ~spec.serverSet.volumeClaimTemplate.

MinIO Docker Image

The following fields describe the Docker settings used by the MinIO Tenant.

spec:

~spec.image: <string> ~spec.imagePullPolicy: <string> ~spec.imagePullSecret: <string>

spec.image

The Docker image to use for the minio server process.

Defaults to the latest stable release of minio:minio if omitted.

spec.imagePullPolicy

The Docker pull policy to use for the specified spec.image.

Specify one of the following values:

• Always - Always pull the image.
• Never - Never pull the image.
• IfNotPresent - Pull the image if not already present.

Defaults to IfNotPresent if omitted.

spec.imagePullSecret

The secret to use for pulling images from private Docker repositories.

Transport Layer Encryption (TLS)

The following fields describe the Transport Layer Encryption (TLS) settings of a MinIO Tenant, including automatic TLS certificate generation.

spec:

~spec.requestAutoCert: <boolean> ~spec.certConfig: ~spec.certConfig.commonName: <string> ~spec.certConfig.dnsNames: <string> ~spec.certConfig.organizationName: <string> ~spec.externalCertSecret: - name: <string> type: kubernetes.io/tls ~spec.externalClientCertSecret: name: <string> type: kubernetes.io/tls

spec.requestAutoCert

Optional

Specify true to enable automatic TLS certificate generation and signing using the Kubernetes certificates.k8s.io API. The MinIO Operator generates self-signed x.509 certificates.

See the Kubernetes documentation on Manage TLS Certificates in a Cluster <tasks/tls/managing-tls-in-a-cluster/> for more information.

This field is mutually exclusive with spec.externalCertSecret.

spec.certConfig

Optional

The configuration settings to use when auto-generating x.509 certificates for TLS encryption.

Omit to allow the MinIO Operator to generate required fields in each auto-generate x.509 certificates.

If spec.requestAutoCert is false or omitted, this field has no effect.

spec.certConfig.commonName

Optional

The x.509 Common Name to use when generating x.509 certificates for TLS encryption. Use wildcard patterns when constructing the commonName to ensure the generated certificates match the Kubernetes-generated DNS names of Tenant resources. See the Kubernetes documentation on DNS for Services and Pods <concepts/services-networking/dns-pod-service/> for more information on Kubernetes DNS.

If spec.requestAutoCert is false or omitted, this field has no effect.

spec.certConfig.dnsNames

Optional

The DNS names to use when generating x.509 certificates for TLS encryption.

If spec.requestAutoCert is false or omitted, this field has no effect.

spec.certConfig.organizationName

Optional

The x.509 Organization Name to use when generating x.509 certificates for TLS encryption.

If spec.requestAutoCert is false or omitted, this field has no effect.

spec.externalCertSecret

Optional

One or more Kubernetes secrets that contain custom TLS certificate and private key pairs. Use this field for specifying certificates signed by a Certificate Authority (CA) of your choice.

Each item in the array contains an object where:

• names specifies the name of the Kubernetes secret, and
• types specifies kubernetes.io/tls

Use wildcard patterns when constructing the DNS-related fields to ensure the generated certificates match the Kubernetes-generated DNS names of Tenant resources. See the Kubernetes documentation on DNS for Services and Pods <concepts/services-networking/dns-pod-service/> for more information on Kubernetes DNS.

spec:
   externalCertSecret:
      - name: tenant-external-cert-secret-name
        type: kubernetes.io/tls

This field is mutually exclusive with spec.requestAutoCert.

spec.externalClientCertSecret

Optional

The Kubernetes secret that contains the custom Certificate Authority certificate and private key used to sign x.509 certificates used by clients connecting to the MinIO Tenant.

Specify an object where:

• names specifies the name of the Kubernetes secret, and
• types specifies kubernetes.io/tls

spec:
   externalClientCertSecret:
      name: tenant-external-client-cert-secret-name
      type: kubernetes.io/tls

MinIO Console Service

The following fields describe the settings for deploying the MinIO Console in the MinIO Tenant.

spec:

~spec.console:

~spec.console.annotations: <object> ~spec.console.consoleSecret: name: <string> ~spec.console.env: <array> ~spec.console.externalCertSecret: name: <string> type: kubernetes.io/tls ~spec.console.image: <string> ~spec.console.imagePullPolicy: <string> ~spec.console.labels: <object> ~spec.console.nodeSelector: <object> ~spec.console.replicas: <int> ~spec.console.resources: <object> ~spec.console.serviceAccountName: <string>

spec.console

Optional

The root field for describing MinIO Console-related configuration information.

Omit to deploy the MinIO Tenant without an attached Console service.

spec.console.consoleSecret

Required if specifying spec.console.

The Kubernetes Secret object that contains all environment variables required by the MinIO Console. Specify the name of the secret as a subfield:

spec:
   console:
      consoleSecret:
         name: console-secret-name

spec.console.annotations

Optional

One or more Kubernetes annotations <user-guide/annotations> to associate with the MinIO Console object.

spec.console.env

Optional

The environment variables available for use by the MinIO Console.

spec.console.externalCertSecret

Optional

The name of the Kubernetes secret containing the custom Certificate Authority certificate and private key to use for configuring TLS on the Console object. Specify an object where names specifies the name of the secret and types specifies kubernetes.io/tls:

spec:
   console:
      externalCertSecret:
         name: console-external-secret-cert-name
         type: kubernetes.io/tls

spec.console.image

Optional

The name of the Docker image to use for deploying the MinIO Console.

Defaults to the latest release of MinIO Console.

spec.console.imagePullPolicy

Optional

The pull policy for the Docker image. Defaults to IfNotPresent.

spec.console.labels

Optional

The Kubernetes labels <concepts/overview/working-with-objects/labels> to apply to the MinIO Console object.

spec.console.nodeSelector

Optional

The filter to apply when selecting which node or nodes on which to deploy the MinIO Console. See the Kubernetes documentation on Assigning Pods to Nodes <concepts/scheduling-eviction/assign-pod-node> for more information.

spec.console.replicas

Optional

The number of MinIO Console pods to create in the cluster.

spec.console.resources

Optional

The resources <concepts/configuration/manage-resources-containers/> each MinIO Console object requests.

spec.console.serviceAccountName

Optional

The name of the Service Account <reference/access-authn-authz/service-accounts-admin/> used to run all MinIO Console pods created as part of the Tenant.

MinIO Key Encryption Service

The following fields describe the settings for deploying the MinIO Key Encryption Service (KES) in the MinIO Tenant.

spec:

kes:

annotations: <object> labels: <object> clientCertSecret: <object> name: <string> type: kubernetes.io/tls externalCertSecret: <object> name: <string> type: kubernetes.io/tls image: <string> imagePullPolicy: <string> kesSecret: <string> nodeSelector: <object> replicas: <integer> serviceAccountName: <string>

spec.kes

Optional

The root field for describing MinIO Key Encryption Service-related configuration information.

Omit to deploy the MinIO Tenant without an attached KES service.

spec.kes.kesSecret

Required if specifying spec.kes.

The Kubernetes Secret object that contains all environment variables required by the MinIO KES. Specify the name of the secret as a subfield:

spec:
   kes:
      kesSecret:
         name: kes-secret-name

spec.kes.annotations

Optional

One or more Kubernetes annotations <user-guide/annotations> to associate with the MinIO KES object.

spec.kes.env

Optional

The environment variables available for use by the MinIO KES.

spec.kes.externalCertSecret

Optional

The name of the Kubernetes secret containing the custom Certificate Authority certificate and private key to use for configuring TLS on the KES object. Specify an object where names specifies the name of the secret and types specifies kubernetes.io/tls:

spec:
   kes:
      externalCertSecret:
         name: kes-external-secret-cert-name
         type: kubernetes.io/tls

spec.kes.image

Optional

The name of the Docker image to use for deploying MinIO KES.

Defaults to the latest release of MinIO KES.

spec.kes.imagePullPolicy

Optional

The pull policy for the Docker image. Defaults to IfNotPresent.

spec.kes.labels

Optional

The Kubernetes labels <concepts/overview/working-with-objects/labels> to apply to the MinIO KES object.

spec.kes.nodeSelector

Optional

The filter to apply when selecting which node or nodes on which to deploy MinIO KES. See the Kubernetes documentation on Assigning Pods to Nodes <concepts/scheduling-eviction/assign-pod-node> for more information.

spec.kes.replicas

Optional

The number of MinIO Console pods to create in the cluster.

spec.kes.serviceAccountName

Optional

The name of the Service Account <reference/access-authn-authz/service-accounts-admin/> used to run all MinIO KES pods created as part of the Tenant.

Pod Security, Scheduling, and Management

The following fields describe the settings for Pod Security, Pod Scheduling, and Pod Management in the MinIO Tenant.

spec:

~spec.securityContext: <object> ~spec.serviceAccountName: <string> ~spec.podManagementPolicy: <object> ~spec.priorityClassName: <string>

spec.securityContext

Optional

Root field for configuring the Security Context <tasks/configure-pod-container/security-context> of pods created as part of the MinIO Tenant.

The MinIO Operator supports the following PodSecurityContext <#podsecuritycontext-v1-core> fields:

• fsGroup
• fsGroupChangePolicy
• runAsGroup
• runAsNonRoot
• runAsUser
• seLinuxOptions

spec.serviceAccountName

Optional

The name of the Service Account <reference/access-authn-authz/service-accounts-admin/> used to run all MinIO server minio pods created as part of the Tenant.

spec.podManagementPolicy

Optional

The Pod Management Policy <concepts/workloads/controllers/statefulset/#pod-management-policies> used for pods created as part of the MinIO Tenant.

spec.priorityClassName

Optional

The Pod Priority Class <concepts/configuration/pod-priority-preemption/#priorityclass> to apply to pods created as part of the MinIO Tenant.