mirror of
https://github.com/minio/docs.git
synced 2025-07-30 07:03:26 +03:00
Operator sidecar (#838)
This adds information about Operator component changes in v5.0.x. Closes #791 Staged: http://192.241.195.202:9000/staging/operator-sidecar/operations/installation.html
This commit is contained in:
Daryl White
GitHub
BIN
source/images/k8s/OperatorsComponent-Diagram.png
Normal file
BIN
source/images/k8s/OperatorsComponent-Diagram.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 82 KiB |
@ -14,29 +14,48 @@ Deploy the MinIO Operator
|
||||
Overview
|
||||
--------
|
||||
|
||||
MinIO is a Kubernetes-native high performance object store with an S3-compatible
|
||||
API. The MinIO Kubernetes Operator supports deploying MinIO Tenants onto private
|
||||
and public cloud infrastructures ("Hybrid" Cloud).
|
||||
MinIO is a Kubernetes-native high performance object store with an S3-compatible API.
|
||||
The MinIO Kubernetes Operator supports deploying MinIO Tenants onto private and public cloud infrastructures ("Hybrid" Cloud).
|
||||
|
||||
The following procedure installs the latest stable version
|
||||
(|operator-version-stable|) of the MinIO Operator and MinIO Plugin on Kubernetes
|
||||
infrastructure:
|
||||
The following procedure installs the latest stable version (|operator-version-stable|) of the MinIO Operator and MinIO Plugin on Kubernetes infrastructure:
|
||||
|
||||
- The MinIO Operator installs a :kube-docs:`Custom Resource Document (CRD)
|
||||
<concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions>`
|
||||
to support describing MinIO tenants as a Kubernetes :kube-docs:`object
|
||||
<concepts/overview/working-with-objects/kubernetes-objects/>`. See the MinIO
|
||||
Operator :minio-git:`CRD Reference <operator/blob/master/docs/tenant_crd.adoc>` for
|
||||
complete documentation on the MinIO CRD.
|
||||
- The MinIO Operator installs a :kube-docs:`Custom Resource Definition (CRD) <concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions>` to support describing MinIO tenants as a Kubernetes :kube-docs:`object <concepts/overview/working-with-objects/kubernetes-objects/>`.
|
||||
See the MinIO Operator :minio-git:`CRD Reference <operator/blob/master/docs/tenant_crd.adoc>` for complete documentation on the MinIO CRD.
|
||||
|
||||
- The MinIO Kubernetes Plugin brings native support for deploying and managing
|
||||
MinIO tenants on a Kubernetes cluster using the :mc:`kubectl minio` command.
|
||||
- The MinIO Kubernetes Plugin brings native support for deploying and managing MinIO tenants on a Kubernetes cluster using the :mc:`kubectl minio` command.
|
||||
|
||||
This documentation assumes familiarity with all referenced Kubernetes
|
||||
concepts, utilities, and procedures. While this documentation *may*
|
||||
provide guidance for configuring or deploying Kubernetes-related resources
|
||||
on a best-effort basis, it is not a replacement for the official
|
||||
:kube-docs:`Kubernetes Documentation <>`.
|
||||
This documentation assumes familiarity with referenced Kubernetes concepts, utilities, and procedures.
|
||||
While this documentation *may* provide guidance for configuring or deploying Kubernetes-related resources on a best-effort basis, it is not a replacement for the official :kube-docs:`Kubernetes Documentation <>`.
|
||||
|
||||
MinIO Operator Components
|
||||
-------------------------
|
||||
|
||||
The MinIO Operator exists in its own namespace.
|
||||
|
||||
Within the Operator's namespace, the MinIO Operator utilizes two pods:
|
||||
- The Operator pod for the base Operator functions to deploy, manage, modify, and maintain tenants.
|
||||
- Console pod for the Operator's Graphical User Interface, the Operator Console.
|
||||
|
||||
When you use the Operator to create a tenant, the tenant *must* have its own namespace.
|
||||
Within that namespace, the Operator generates the pods required by the tenant configuration.
|
||||
|
||||
Each pod runs three containers:
|
||||
|
||||
- MinIO Container that runs all of the standard MinIO functions, equivalent to basic MinIO installation on baremetal.
|
||||
This container stores and retrieves objects in the provided mount points (persistent volumes).
|
||||
|
||||
- InitContainer that only exists during the launch of the pod to manage configuration secrets during startup.
|
||||
Once startup completes, this container terminates.
|
||||
|
||||
- SideCar container that monitors configuration secrets for the tenant and updates them as they change.
|
||||
This container also monitors for root credentials and creates an error if it does not find root credentials.
|
||||
|
||||
The tenant utilizes Persistent Volume Claims to talk to the Persistent Volumes that store the objects.
|
||||
|
||||
.. image:: /images/k8s/OperatorsComponent-Diagram.png
|
||||
:width: 600px
|
||||
:alt: A diagram of the namespaces and pods used by or maintained by the MinIO Operator.
|
||||
:align: center
|
||||
|
||||
Prerequisites
|
||||
-------------
|
||||
@ -44,13 +63,13 @@ Prerequisites
|
||||
Kubernetes Version 1.19.0
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Starting with v4.0.0, the MinIO Operator and MinIO Kubernetes Plugin require
|
||||
Kubernetes 1.19.0 and later. The Kubernetes infrastructure *and* the
|
||||
``kubectl`` CLI tool must have the same version of 1.19.0+.
|
||||
Starting with v4.0.0, the MinIO Operator and MinIO Kubernetes Plugin **require** Kubernetes 1.19.0 and later.
|
||||
The Kubernetes infrastructure *and* the ``kubectl`` CLI tool must have the same version of 1.19.0+.
|
||||
|
||||
Prior to v4.0.0, the MinIO Operator and Plugin required Kubernetes 1.17.0. You
|
||||
*must* upgrade your Kubernetes infrastructure to 1.19.0 or later to use
|
||||
the MinIO Operator or Plugin v4.0.0 or later.
|
||||
Prior to v4.0.0, the MinIO Operator and Plugin required Kubernetes 1.17.0.
|
||||
You *must* upgrade your Kubernetes infrastructure to 1.19.0 or later to use the MinIO Operator or Plugin v4.0.0 or later.
|
||||
|
||||
Starting with v5.0.0, MinIO *recommends* Kubernetes 1.21.0 or later for both the infrastructure and the ``kubectl`` CLI tool.
|
||||
|
||||
``kubectl`` Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
@ -62,9 +81,18 @@ This procedure assumes that your local host machine has both the correct version
|
||||
Kubernetes TLS Certificate API
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The MinIO Operator automatically generates TLS Certificate Signing Requests (CSR) and uses the Kubernetes ``certificates.k8s.io`` :kube-docs:`TLS certificate management API <tasks/tls/managing-tls-in-a-cluster/>` to create signed TLS certificates.
|
||||
.. versionchanged:: Operator v.5.0.0
|
||||
|
||||
The MinIO Operator therefore *requires* that the Kubernetes ``kube-controller-manager`` configuration include the following :kube-docs:`configuration settings <reference/command-line-tools-reference/kube-controller-manager/#options>`:
|
||||
The MinIO Operator manages TLS Certificate Signing Requests (CSR) using the Kubernetes ``certificates.k8s.io`` :kube-docs:`TLS certificate management API <tasks/tls/managing-tls-in-a-cluster/>` to create signed TLS certificates in the following circumstances:
|
||||
|
||||
- When ``autoCert`` is enabled.
|
||||
- For the MinIO Console when the :envvar:`OPERATOR_CONSOLE_TLS_ENABLE` environment variable is set to ``on``.
|
||||
- For :ref:`STS service <minio-security-token-service>` when :envvar:`OPERATOR_STS_ENABLED` environment variable is set to ``on``.
|
||||
- For retrieving the health of the cluster.
|
||||
|
||||
The MinIO Operator reads certificates inside the ``operator-ca-tls`` secret and syncs this secret within the tenant namespace to trust private certificate authorities, such as when using cert-manager.
|
||||
|
||||
For any of these circumstances, the MinIO Operator *requires* that the Kubernetes ``kube-controller-manager`` configuration include the following :kube-docs:`configuration settings <reference/command-line-tools-reference/kube-controller-manager/#options>`:
|
||||
|
||||
- ``--cluster-signing-key-file`` - Specify the PEM-encoded RSA or ECDSA private key used to sign cluster-scoped certificates.
|
||||
|
||||
@ -74,13 +102,10 @@ The Kubernetes TLS API uses the CA signature algorithm for generating new TLS ce
|
||||
MinIO recommends ECDSA (e.g. `NIST P-256 curve <https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.186-4.pdf>`__) or EdDSA (e.g. :rfc:`Curve25519 <7748>`) TLS private keys/certificates due to their lower computation requirements compared to RSA.
|
||||
See :ref:`minio-TLS-supported-cipher-suites` for a complete list of supported TLS Cipher Suites.
|
||||
|
||||
The Operator cannot complete initialization if the Kubernetes cluster is
|
||||
not configured to respond to a generated CSR. Certain Kubernetes
|
||||
providers do not specify these configuration values by default.
|
||||
If the Kubernetes cluster is not configured to respond to a generated :abbr:`CSR (Certificate Signing Request)`, the Operator cannot complete initialization.
|
||||
Some Kubernetes providers do not specify these configuration values by default.
|
||||
|
||||
To verify whether the ``kube-controller-manager`` has the required
|
||||
settings, use the following command. Replace ``$CLUSTER-NAME`` with the name
|
||||
of the Kubernetes cluster:
|
||||
To check whether the ``kube-controller-manager`` specifies the cluster signing key and certificate files, use the following command:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
@ -88,8 +113,10 @@ of the Kubernetes cluster:
|
||||
kubectl get pod kube-controller-manager-$CLUSTERNAME-control-plane \
|
||||
-n kube-system -o yaml
|
||||
|
||||
Confirm that the output contains the highlighted lines. The output of
|
||||
the example command above may differ from the output in your terminal:
|
||||
- Replace ``$CLUSTERNAME`` with the name of the Kubernetes cluster.
|
||||
|
||||
Confirm that the output contains the highlighted lines.
|
||||
The output of the example command above may differ from the output in your terminal:
|
||||
|
||||
.. code-block:: shell
|
||||
:emphasize-lines: 12,13
|
||||
@ -111,18 +138,60 @@ the example command above may differ from the output in your terminal:
|
||||
|
||||
.. important::
|
||||
|
||||
The MinIO Operator automatically generates TLS certificates for all
|
||||
MinIO Tenant pods using the specified Certificate Authority (CA).
|
||||
Clients external to the Kubernetes cluster must trust the
|
||||
Kubernetes cluster CA to connect to the MinIO Operator or MinIO Tenants.
|
||||
The MinIO Operator automatically generates TLS certificates for all MinIO Tenant pods using the specified Certificate Authority (CA).
|
||||
Clients external to the Kubernetes cluster must trust the Kubernetes cluster CA to connect to the MinIO Operator or MinIO Tenants.
|
||||
|
||||
Clients which cannot trust the Kubernetes cluster CA can try disabling TLS
|
||||
validation for connections to the MinIO Operator or a MinIO Tenant.
|
||||
Clients which cannot trust the Kubernetes cluster CA can disable TLS validation for connections to the MinIO Operator or a MinIO Tenant.
|
||||
|
||||
Alternatively, you can generate x.509 TLS certificates signed by a known
|
||||
and trusted CA and pass those certificates to MinIO Tenants.
|
||||
Alternatively, you can generate x.509 TLS certificates signed by a known and trusted CA and pass those certificates to MinIO Tenants.
|
||||
See :ref:`minio-tls` for more complete documentation.
|
||||
|
||||
Configure MinIO Operator to Trust Custom Certificates
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If you use custom certificates for your deployment, add the certificate so that MinIO Operator trusts it.
|
||||
|
||||
This procedure assumes you have an existing custom certificate.
|
||||
|
||||
1. Use the following command to generate a secret from the certificate:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
kubectl create secret generic MY-CUSTOM-TLS -n MY-CLUSTER-NAMESPACE --from-file=<path/to/public.crt>
|
||||
|
||||
Replace the following placeholders in the above command:
|
||||
- ``MY-CUSTOM-TLS`` with the name of your secrets file
|
||||
- ``MY-CLUSTER-NAMESPACE`` with your cluster's namespace
|
||||
- ``<path/to/public.crt>`` with the relative path to the public certificate to use to create the secret
|
||||
|
||||
2. Add a volume to the yaml for your cluster under ``.spec.template.spec``
|
||||
|
||||
.. code-block:: yaml
|
||||
:class: copyable
|
||||
|
||||
volumes:
|
||||
- name: tls-certificates
|
||||
projected:
|
||||
defaultMode: 420
|
||||
sources:
|
||||
- secret:
|
||||
items:
|
||||
- key: public.crt
|
||||
path: CAs/custom-public.crt
|
||||
name: MY-CUSTOM-TLS
|
||||
|
||||
- replace ``MY-CUSTOM-TLS`` with the name of your secrets file.
|
||||
|
||||
3. Add a ``volumeMount`` to the yaml for your cluster under ``.spec.template.spec.container[0]``
|
||||
|
||||
.. code-block:: yaml
|
||||
:class: copyable
|
||||
|
||||
volumeMounts:
|
||||
- mountPath: /tmp/certs
|
||||
name: tls-certificates
|
||||
|
||||
Procedure
|
||||
---------
|
||||
|
||||
|
||||
@ -176,7 +176,6 @@ The Operator Console still requires using the generated JWT token for access, wh
|
||||
After deploying the MinIO Operator, you can create a new MinIO Tenant.
|
||||
To deploy a MinIO Tenant using OpenShift, see :ref:`deploy-minio-tenant-redhat-openshift`.
|
||||
|
||||
.. uncomment this block when upgrade operator procedure is complete
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
|
||||
@ -43,7 +43,12 @@ The MinIO Operator Console supports deploying, managing, and monitoring MinIO Te
|
||||
|
||||
You can :ref:`deploy a MinIO Tenant <minio-k8s-deploy-minio-tenant>` through the Operator Console.
|
||||
|
||||
The Operator Console automatically detects MinIO Tenants deployed on the cluster when provisioned through the Operator Console, the :ref:`MinIO Kubernetes Plugin <minio-k8s-deploy-minio-tenant-commandline>`, Helm, or Kustomize.
|
||||
The Operator Console automatically detects MinIO Tenants deployed on the cluster when provisioned through:
|
||||
|
||||
- Operator Console
|
||||
- :ref:`MinIO Kubernetes Plugin <minio-k8s-deploy-minio-tenant-commandline>`
|
||||
- Helm
|
||||
- Kustomize
|
||||
|
||||
Select a listed tenant to open an in-browser view of that tenant's MinIO Console.
|
||||
You can use this view to directly manage, modify, expand, upgrade, and delete the tenant through the Operator UI.
|
||||
@ -56,12 +61,12 @@ You can use this view to directly manage, modify, expand, upgrade, and delete th
|
||||
The ZIP archive contains status, events, and log information for each pool on the deployment.
|
||||
The archive also includes a summary yaml file describing the deployment.
|
||||
|
||||
|subnet| users can upload this file for analysis by MinIO Engineers.
|
||||
|subnet| users relying on the commercial license should register the MinIO tenants to their SUBNET account, which can be done through the Operator Console.
|
||||
|
||||
Tenant Registration
|
||||
-------------------
|
||||
|
||||
|subnet| users relying on the commercial license should register your MinIO tenants to your SUBNET account, which can be done through the Operator Console.
|
||||
|subnet| users relying on the commercial license should register the MinIO tenants to their SUBNET account, which can be done through the Operator Console.
|
||||
|
||||
.. image:: /images/k8s/operator-console-register.png
|
||||
:align: center
|
||||
|
||||
@ -65,7 +65,7 @@ The following changes apply for Operator v5.0.0 or later:
|
||||
This change impacts the Tenant :abbr:`CRD <CustomResourceDefinition>` and only impacts users editing a tenant YAML directly, such as through Helm or Kustomize.
|
||||
- Both the **Log Search API** (``.spec.log``) and **Prometheus** (``.spec.prometheus``) deployments have been removed.
|
||||
However, existing deployments are left running as standalone deployments / statefulsets with no connection to the Tenant CR.
|
||||
If the Tenant CRD is deleted, this does not cascade to the log or Prometheus deployments.
|
||||
Deleting the Tenant :abbr:`CRD (Custom Resource Definition)` does **not** cascade to the log or Prometheus deployments.
|
||||
|
||||
.. important::
|
||||
|
||||
@ -170,7 +170,7 @@ Upgrade Operator to |operator-version-stable|
|
||||
"name": "minio-operator"
|
||||
}
|
||||
|
||||
3. Download the Latest Stable Version of the MinIO Kubernetes Plugin
|
||||
3. Download the latest stable version of the MinIO Kubernetes Plugin
|
||||
|
||||
.. include:: /includes/k8s/install-minio-kubectl-plugin.rst
|
||||
|
||||
|
||||
@ -93,7 +93,30 @@ Environment Variables
|
||||
---------------------
|
||||
|
||||
The :ref:`MinIO Operator <minio-operator-installation>` uses the following environment variables during startup to set configuration settings.
|
||||
Configure these variables on the machine running the kubectl plugin.
|
||||
Configure these variables in the ``minio-operator`` container.
|
||||
|
||||
Setting Environment Variables in Kubernetes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To set these environment variables, modify the operator container's yaml at ``.spec.env`` or use the following ``kubectl`` command syntax:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
kubectl set env -n minio-operator deployment/minio-operator <ENV_VARIABLE>=<value> ... <ENV_VARIABLE2>=<value2>
|
||||
|
||||
Replace:
|
||||
|
||||
- ``minio-operator`` with the namespace for your Operator, if not using the default value.
|
||||
- ``deployment/minio-operator`` with the deployment for your Operator, if not the default value.
|
||||
(This is not common.)
|
||||
- ``<ENV_VARIABLE>`` with the environment variable to set or modify.
|
||||
- ``<value>`` with the value to use for the environment variable.
|
||||
|
||||
You can set or modify multiple environment variables by separating each ``VARIABLE=value`` pair with a space.
|
||||
|
||||
Available MinIO Operator Environment Variables
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. envvar:: MINIO_OPERATOR_CERTIFICATES_VERSION
|
||||
|
||||
@ -126,7 +149,7 @@ Configure these variables on the machine running the kubectl plugin.
|
||||
|
||||
.. envvar:: MINIO_OPERATOR_DEPLOYMENT_NAME
|
||||
|
||||
Specifies the namespace to create and use for Operator
|
||||
Specifies the namespace to create and use for Operator.
|
||||
|
||||
When not specified, the default value is ``minio-operator``.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user