DOCS-610: Update KES procedures for KES 0.21.0, minor cleanups for Vault (#612)

Closes #610 

Did a QA pass after reports of some issues with the vault setup.

Identified an issue where my local testing was done with 0.20.0,
resulting in some errors on 0.21.0 due to dropping --mlock.

Also did a few other general tune ups as I ran end-to-ends again using
Vault + Systemd instead of vault in dev mode.

Staging views in #minio-docs channel

source/extra/kes.service

@@ -14,7 +14,7 @@ User=kes
 Group=kes
 ProtectProc=invisible
 
-ExecStart=/usr/local/bin/kes server --config=/opt/kes/config.yaml --mlock
+ExecStart=/usr/local/bin/kes server --config=/opt/kes/config.yaml
 
 # Let systemd restart this service always
 Restart=always

source/includes/common/common-minio-kes-hashicorp.rst

@@ -1,7 +1,8 @@
 .. start-kes-configuration-hashicorp-vault-desc
 
 |KES| uses a YAML-formatted configuration file. 
-The following YAML provides the minimum required fields for using Hashicorp Vault as the root |KMS| and is intended for use in this tutorial.
+The following YAML provides the minimum required fields for using Hashicorp Vault as the root |KMS|. 
+You must modify this YAML to reflect your deployment environment.
 
 .. code-block:: shell
    :class: copyable
@@ -30,7 +31,7 @@ The following YAML provides the minimum required fields for using Hashicorp Vaul
        - /v1/key/generate/* # e.g. '/minio-'
        - /v1/key/decrypt/*
        identities:
-       - ${MINIO_IDENTITY_HASH} # Replace with the output of 'kes identity of minio-kes.cert'
+       - MINIO_IDENTITY_HASH # Replace with the output of 'kes identity of minio-kes.cert'
                                 # In production environments, each client connecting to KES must
                                 # Have their TLS hash listed under at least one `policy`.
 
@@ -64,5 +65,24 @@ This procedure assumes an existing `Hashicorp Vault <https://www.vaultproject.io
 The Vault `Quick Start <https://learn.hashicorp.com/tutorials/vault/getting-started-install>`__ provides a sufficient foundation for the purposes of this procedure.
 Defer to the `Vault Documentation <https://learn.hashicorp.com/vault>`__ for guidance on deployment and configuration.
 
+.. admonition:: KMS Key Creation Requires Unsealed Vault
+   :class: important
+
+   KES requires unsealing the Vault instance for creating new External Keys (EK) for supporting SSE operations.
+   A sealed Vault blocks key creation operations and results in errors on both KES and the KES client (MinIO).
+
+   You can safely seal the Vault after KES completes any key creation operations.
+   See the Vault documentation on `Seal/Unseal <https://www.vaultproject.io/docs/concepts/seal>`__ for more information.
+
 .. end-kes-prereq-hashicorp-vault-desc
 
+.. start-kes-vault-seal-unseal-desc
+
+.. admonition:: KMS Key Creation Requires Unsealed Vault
+   :class: important
+
+   You must unseal the backing Vault instance to allow KES to create new External Keys (EK) for supporting SSE operations.
+   This step requires an unsealed Vault to complete successfully.
+   See the Vault documentation on `Seal/Unseal <https://www.vaultproject.io/docs/concepts/seal>`__ for more information.
+
+.. end-kes-vault-seal-unseal-desc

source/includes/common/common-minio-kes.rst

@@ -48,13 +48,13 @@ The following commands create two TLS certificates that expire within 30 days of
    # These commands output keys to |kescertpath|
    # and |miniocertpath| respectively
 
-   kes identity new  \
+   kes identity new kes_server \
      --key  |kescertpath|/kes-server.key  \
      --cert |kescertpath|/kes-server.cert  \
      --ip   "127.0.0.1"  \
      --dns  localhost
 
-   kes identity new  \
+   kes identity new minio_server \
      --key  |miniocertpath|/minio-kes.key  \
      --cert |miniocertpath|/minio-kes.cert  \
      --ip   "127.0.0.1"  \
@@ -89,12 +89,13 @@ Run the following commands in a terminal or shell to start the KES server as a f
 
    sudo setcap cap_ipc_lock=+ep $(readlink -f $(which kes))
 
-   kes server --mlock  \
+   kes server --auth=off --config=|kesconfigpath|/kes-config.yaml
-               --config=|kesconfigpath|kes-config.yaml  \
+               
-               --auth=off
 
 The first command allows |KES| to use the `mlock <http://man7.org/linux/man-pages/man2/mlock.2.html>`__ system call without running as root. 
 ``mlock`` ensures the OS does not write in-memory data to disk (swap memory) and mitigates the risk of cryptographic operations being written to unsecured disk at any time.
+KES 0.21.0 and later automatically detect and enable ``mlock`` if supported by the host OS. 
+Versions 0.20.0 and earlier required specifying the ``--mlock`` argument to KES.
 
 The second command starts the KES server in the foreground using the configuration file created in the last step. 
 The ``--auth=off`` disables strict validation of client TLS certificates.
@@ -131,8 +132,6 @@ The following command uses the ``kes key create`` command to add a new External
 Add the following lines to the MinIO Environment file on each MinIO host.
 See the tutorials for :ref:`minio-snsd`, :ref:`minio-snmd`, or :ref:`minio-mnmd` for more detailed descriptions of a base MinIO environment file.
 
-This command assumes the ``minio-kes.cert``, ``minio-kes.key``, and ``kes-server.cert`` certificates are accessible at the specified location:
-
 .. code-block:: shell
    :class: copyable
    :substitutions:
@@ -142,16 +141,19 @@ This command assumes the ``minio-kes.cert``, ``minio-kes.key``, and ``kes-server
    MINIO_KMS_KES_ENDPOINT=https://HOSTNAME:7373
    MINIO_KMS_KES_CERT_FILE=|miniocertpath|/minio-kes.cert
    MINIO_KMS_KES_KEY_FILE=|miniocertpath|/minio-kes.key
-   MINIO_KMS_KES_CAPATH=|miniocertpath|/kes-server.cert
-   MINIO_KMS_KES_KEY_NAME=minio-backend-default-key
 
-   minio server [ARGUMENTS]
+   # Allows validation of the KES Server Certificate (Self-Signed or Third-Party CA)
+   # Change this path to the location of the KES CA Path
+   MINIO_KMS_KES_CAPATH=|kescertpath|/kes-server.cert
+
+   # Sets the default KMS key for the backend and SSE-KMS/SSE-S3 Operations)
+   MINIO_KMS_KES_KEY_NAME=minio-backend-default-key
 
 Replace ``HOSTNAME`` with the IP address or hostname of the KES server.
 If the MinIO server host machines cannot resolve or reach the specified ``HOSTNAME``, the deployment may return errors or fail to start.
 
 - If using a single KES server host, specify the IP or hostname of that host
-- If using multiple KES server hosts, specify the load balancer or reverse proxy managing connections to those hosts.
+- If using multiple KES server hosts, specify a comma-separated list of IPs or hostnames of each host
 
 MinIO uses the :envvar:`MINIO_KMS_KES_KEY_NAME` key for the following cryptographic operations:
 

source/includes/container/common-minio-kes.rst

@@ -25,7 +25,7 @@ The commands in this section create the following resources:
      -e KES_CLIENT_KEY=/certs/kes-server.key  \
      -e KES_CLIENT_CERT=/certs/kes-server.cert  \
      quay.io/minio/kes:|kes-stable| server  \
-       --mlock --auth  \
+       --auth  \
        --config=/etc/default/kes-config.yaml  \
 
    sudo podman run -dt  \
@@ -73,14 +73,14 @@ The following commands create two TLS certificates that expire within 30 days of
 
     podman run --rm  \
       -v |kescertpath|:/certs  \
-      quay.io/minio/kes:|kes-stable| identity new  \
+      quay.io/minio/kes:|kes-stable| identity new  kes_server \
         --key  /certs/kes-server.key  \
         --cert /certs/kes-server.cert  \
         kes-server
 
     podman run --rm  \
       -v |miniocertpath|:/certs  \
-      quay.io/minio/kes:|kes-stable| identity new  \
+      quay.io/minio/kes:|kes-stable| identity new minio_server \
         --key  /certs/minio-kes.key  \
         --cert /certs/minio-kes.cert  \
         minio-server
@@ -123,6 +123,12 @@ KES automatically creates this key if it does not already exist on the root KMS.
 
 .. start-kes-generate-key-desc
 
+.. admonition:: Unseal Vault Before Creating Key
+   :class: important
+
+   You must unseal the backing Vault instance before creating new encryption keys.
+   See the Vault documentation on `Seal/Unseal <https://www.vaultproject.io/docs/concepts/seal>`__ for more information.
+
 MinIO requires that the |EK| exist on the root KMS *before* performing
 |SSE| operations using that key. Use ``kes key create`` *or*
 :mc-cmd:`mc admin kms key create` to create a new |EK| for use with |SSE|.

source/includes/k8s/common-minio-kes.rst

@@ -80,6 +80,12 @@ You can use either the MinIO Tenant Console or the MinIO :mc:`mc` CLI to enable
 
 .. start-kes-generate-key-desc
 
+.. admonition:: Unseal Vault Before Creating Key
+   :class: important
+
+   You must unseal the backing Vault instance before creating new encryption keys.
+   See the Vault documentation on `Seal/Unseal <https://www.vaultproject.io/docs/concepts/seal>`__ for more information.
+
 MinIO requires that the |EK| for a given bucket or object exist on the root KMS *before* performing |SSE| operations using that key.
 You can use the :mc-cmd:`mc admin kms key create` command against the MinIO Tenant.
 

source/includes/linux/common-minio-kes.rst

@@ -11,14 +11,15 @@ The following example code downloads the latest Linux AMD64-compatible binary an
    :class: copyable
    :substitutions:
 
-   wget https://github.com/minio/kes/releases/download/|kes-stable|/kes-linux-amd64 -O /tmp/kes
+   curl --retry 10 https://github.com/minio/kes/releases/download/|kes-stable|/kes-linux-amd64 -o /tmp/kes
    chmod +x /tmp/kes
    sudo mv /tmp/kes /usr/local/bin
 
    kes --version
 
 For distributed KES topologies, repeat this step and all following KES-specific instructions for each host on which you want to deploy KES.
-MinIO strongly recommends configuring a load balancer with a "Least Connections" configuration to manage connections to distributed KES hosts.
+MinIO uses a round-robin approach by default for routing connections to multiple configured KES servers.
+For more granular controls, deploy a dedicated load balancer to manage connections to distributed KES hosts.
 
 .. end-kes-download-desc
 
@@ -92,66 +93,39 @@ Depending on your Vault configuration, you may also need to create a dedicated s
 Defer to your organizations best practices around generating production-ready TLS certificates.
 
 Place the certificates and corresponding private keys an appropriate directory such that the MinIO and KES service users can access and read their contents.
-This procedure assumes a structure similar to the following:
+The following example structure uses the folder hierarchy suggested in the beginning of this procedure:
 
-  .. code-block:: shell
+.. tab-set::
-     :substitutions:
 
-     # For the MinIO Hosts
+   .. tab-item:: KES Hosts
-     -rw-r--r-- 1 minio-user:minio-user |miniocertpath|/minio-kes.cert
-     -rw-r--r-- 1 minio-user:minio-user |miniocertpath|/minio-kes.key
 
-     # If KES certs are self-signed or use a non-global CA
+      .. code-block:: shell
-     # Include the CA certs as well
+         :substitutions:
-     -rw-r--r-- 1 minio-user:minio-user |miniocertpath|/kes-server.cert
 
-     # For the KES Hosts
+         -rw-r--r-- 1 kes:kes |kescertpath|/kes-server.cert
-     -rw-r--r-- 1 kes:kes |kescertpath|/kes-server.cert
+         -rw-r--r-- 1 kes:kes |kescertpath|/kes-server.key
-     -rw-r--r-- 1 kes:kes |kescertpath|/kes-server.key
 
-If the KES certificates are self-signed *or* signed by Certificate Authority (CA) that is *not* globally trusted, you **must** add the CA certificate to the |miniocertpath| directory such that each MinIO server can properly validate the KES certificates.
+         # If the Vault certs are self-signed or use a non-global CA
+         # Include those CA certs as well
+
+         -rw-r--r-- 1 kes:kes |kescertpath|/vault-CA.cert
+
+   .. tab-item:: MinIO Hosts
+
+      .. code-block:: shell
+         :substitutions:
+
+         -rw-r--r-- 1 minio-user:minio-user |miniocertpath|/minio-kes.cert
+         -rw-r--r-- 1 minio-user:minio-user |miniocertpath|/minio-kes.key
+
+         # If KES certs are self-signed or use a non-global CA
+         # Include the CA certs as well
+         -rw-r--r-- 1 minio-user:minio-user |miniocertpath|/kes-server.cert
+
+The general strategy for cert management is to ensure that each process (MinIO, KES, and Vault) have their own mTLS certificates *and* the Certificate Authority (CA) used to sign each client certificate.
 
 .. end-kes-generate-kes-certs-prod-desc
 
-.. start-kes-configuration-minio-desc
-
-Add the following lines to the MinIO Environment file on each MinIO host.
-See the tutorials for :ref:`minio-snsd`, :ref:`minio-snmd`, or :ref:`minio-mnmd` for more detailed descriptions of a base MinIO environment file.
-
-This command assumes the ``minio-kes.cert``, ``minio-kes.key``, and ``kes-server.cert`` certificates are accessible at the specified location:
-
-.. code-block:: shell
-   :class: copyable
-   :substitutions:
-
-   # Add these environment variables to the existing environment file
-
-   MINIO_KMS_KES_ENDPOINT=https://HOSTNAME:7373
-   MINIO_KMS_KES_CERT_FILE=|miniocertpath|/minio-kes.cert
-   MINIO_KMS_KES_KEY_FILE=|miniocertpath|/minio-kes.key
-   MINIO_KMS_KES_CAPATH=|kescertpath|/kes-server.cert
-   MINIO_KMS_KES_KEY_NAME=minio-backend-default-key
-
-   minio server [ARGUMENTS]
-
-Replace ``HOSTNAME`` with the IP address or hostname of the KES server.
-If the MinIO server host machines cannot resolve or reach the specified ``HOSTNAME``, the deployment may return errors or fail to start.
-
-- If using a single KES server host, specify the IP or hostname of that host
-- If using multiple KES server hosts, specify the load balancer or reverse proxy managing connections to those hosts.
-
-MinIO uses the :envvar:`MINIO_KMS_KES_KEY_NAME` key for the following cryptographic operations:
-
-- Encrypting the MinIO backend (IAM, configuration, etc.)
-- Encrypting objects using :ref:`SSE-KMS <minio-encryption-sse-kms>` if the request does not 
-  include a specific |EK|.
-- Encrypting objects using :ref:`SSE-S3 <minio-encryption-sse-s3>`.
-
-The ``minio-kes`` certificates enable mTLS between the MinIO deployment and the KES server *only*.
-They do not otherwise enable TLS for other client connections to MinIO.
-
-.. end-kes-configuration-minio-desc
-
 .. start-kes-generate-key-desc
 
 MinIO requires that the |EK| exist on the root KMS *before* performing |SSE| operations using that key. 

source/includes/linux/steps-configure-minio-kes-aws.rst

@@ -72,7 +72,7 @@ b. Configure the MinIO Environment File
    MinIO defaults to expecting this file at ``/etc/default/minio``.
    If you modified your deployment to use a different location for the environment file, modify the file at that location.
 
-   .. include:: /includes/linux/common-minio-kes.rst
+   .. include:: /includes/common/common-minio-kes.rst
       :start-after: start-kes-configuration-minio-desc
       :end-before: end-kes-configuration-minio-desc
 
@@ -99,7 +99,7 @@ b. Start the MinIO Server
 5) Generate a New Encryption Key
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. include:: /includes/linux/common-minio-kes.rst
+.. include:: /includes/common/common-minio-kes.rst
    :start-after: start-kes-generate-key-desc
    :end-before: end-kes-generate-key-desc
 

source/includes/linux/steps-configure-minio-kes-azure.rst

@@ -71,7 +71,7 @@ b. Configure the MinIO Environment File
    MinIO defaults to expecting this file at ``/etc/default/minio``.
    If you modified your deployment to use a different location for the environment file, modify the file at that location.
 
-   .. include:: /includes/linux/common-minio-kes.rst
+   .. include:: /includes/common/common-minio-kes.rst
       :start-after: start-kes-configuration-minio-desc
       :end-before: end-kes-configuration-minio-desc
 
@@ -98,7 +98,7 @@ b. Start the MinIO Server
 5) Generate a New Encryption Key
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. include:: /includes/linux/common-minio-kes.rst
+.. include:: /includes/common/common-minio-kes.rst
    :start-after: start-kes-generate-key-desc
    :end-before: end-kes-generate-key-desc
 

source/includes/linux/steps-configure-minio-kes-gcp.rst

@@ -71,7 +71,7 @@ b. Configure the MinIO Environment File
    MinIO defaults to expecting this file at ``/etc/default/minio``.
    If you modified your deployment to use a different location for the environment file, modify the file at that location.
 
-   .. include:: /includes/linux/common-minio-kes.rst
+   .. include:: /includes/common/common-minio-kes.rst
       :start-after: start-kes-configuration-minio-desc
       :end-before: end-kes-configuration-minio-desc
 
@@ -98,7 +98,7 @@ b. Start the MinIO Server
 5) Generate a New Encryption Key
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. include:: /includes/linux/common-minio-kes.rst
+.. include:: /includes/common/common-minio-kes.rst
    :start-after: start-kes-generate-key-desc
    :end-before: end-kes-generate-key-desc
 

source/includes/linux/steps-configure-minio-kes-hashicorp-quick.rst

@@ -1,17 +1,31 @@
-Deploy MinIO and KES with Server-Side Encryption using Hashicorp Vault for Local Development
+Enable Server-Side Encryption using Hashicorp Vault for Local Development
---------------------------------------------------------------------------------------------
+-------------------------------------------------------------------------
 
-Prior to starting these steps, create the following folders:
+This procedure assumes deploying MinIO and KES onto the same host.
+You can use and modify the information below to better suit your local development environment.
 
-.. code-block:: shell
+This procedure assumes the following:
-   :class: copyable
-   :substitutions:
 
-   mkdir -P |kescertpath|
+- An existing Vault deployment
-   mkdir -P |kesconfigpath|
+- A single host machine for deploying KES and MinIO
-   mkdir -P |miniocertpath|
+
-   mkdir -P |minioconfigpath|
+.. admonition:: Set Up Folder Hierarchy
-   mkdir -P |miniodatapath|
+   :class: note
+
+   Create the following folders on the host machine if they do not already exist.
+
+   .. code-block:: shell
+      :class: copyable
+      :substitutions:
+
+      mkdir -P |kescertpath|
+      mkdir -P |kesconfigpath|
+      mkdir -P |miniocertpath|
+      mkdir -P |minioconfigpath|
+      mkdir -P |miniodatapath|
+
+   This procedure uses these paths in the following steps. 
+   If you use different paths, make the necessary modifications for each step to reflect those changes
 
 1) Download the KES Server Binary
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -34,65 +48,73 @@ Defer to the client documentation for instructions on trusting a third-party CA.
 3) Create the KES and MinIO Configurations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-a. Create the KES Configuration File
+.. container:: procedure
 
-   Create the configuration file using your preferred text editor.
+   a. Create the KES Configuration File
-   The following example uses ``nano``:
 
-   .. code-block:: shell
+      Create the configuration file using your preferred text editor.
-      :substitutions:
+      The following example uses ``nano``:
-
-      nano |kesconfigpath|/kes-config.yaml
-
-   .. include:: /includes/common/common-minio-kes-hashicorp.rst
-      :start-after: start-kes-configuration-hashicorp-vault-desc
-      :end-before: end-kes-configuration-hashicorp-vault-desc
-
-   - Set ``MINIO_IDENTITY_HASH`` to the identity hash of the MinIO mTLS certificate.
-
-      The following command computes the necessary hash:
 
       .. code-block:: shell
-         :class: copyable
          :substitutions:
 
-         kes identity of |miniocertpath|/minio-kes.cert
+         nano |kesconfigpath|/kes-config.yaml
 
-   - Replace the ``vault.endpoint`` with the hostname of the Vault server(s).
+      .. include:: /includes/common/common-minio-kes-hashicorp.rst
-   - Set the ``vault.engine`` and ``vault.version`` to the appropriate values for the Vault K/V Engine configuration
+         :start-after: start-kes-configuration-hashicorp-vault-desc
-   - Replace the ``VAULTAPPID`` and ``VAULTAPPSECRET`` with the appropriate :ref:`Vault AppRole credentials <minio-sse-vault-prereq-vault>`.
+         :end-before: end-kes-configuration-hashicorp-vault-desc
 
-b. Create the MinIO Environment File
+      - Set ``MINIO_IDENTITY_HASH`` to the identity hash of the MinIO mTLS certificate.
 
-   Create or modify the environment file for the MinIO deployment using your preferred text editor.
+        The following command computes the necessary hash:
-   The following example uses ``nano``:
 
-   .. code-block:: shell
+        .. code-block:: shell
-      :substitutions:
+           :class: copyable
+           :substitutions:
 
-      nano |minioconfigpath|/minio
+           kes identity of |miniocertpath|/minio-kes.cert
 
-   .. include:: /includes/common/common-minio-kes.rst
+      - Replace the ``vault.endpoint`` with the hostname of the Vault server(s).
-      :start-after: start-kes-configuration-minio-desc
+      - Set the ``vault.engine`` and ``vault.version`` to the appropriate values for the Vault K/V Engine configuration
-      :end-before: end-kes-configuration-minio-desc
+      - Replace the ``VAULTAPPID`` and ``VAULTAPPSECRET`` with the appropriate :ref:`Vault AppRole credentials <minio-sse-vault-prereq-vault>`.
+
+   b. Create the MinIO Environment File
+
+      Create or modify the environment file for the MinIO deployment using your preferred text editor.
+      The following example uses ``nano``:
+
+      .. code-block:: shell
+         :substitutions:
+
+         nano |minioconfigpath|/minio
+
+      .. include:: /includes/common/common-minio-kes.rst
+         :start-after: start-kes-configuration-minio-desc
+         :end-before: end-kes-configuration-minio-desc
 
 4) Start KES and MinIO
 ~~~~~~~~~~~~~~~~~~~~~~
 
+.. include:: /includes/common/common-minio-kes-hashicorp.rst
+   :start-after: start-kes-vault-seal-unseal-desc
+   :end-before: end-kes-vault-seal-unseal-desc
+
 You must start KES *before* starting MinIO. 
 The MinIO deployment requires access to KES as part of its startup.
 
-a. Start the KES Server
+.. container:: procedure
 
-   .. include:: /includes/common/common-minio-kes.rst
+   a. Start the KES Server
-      :start-after: start-kes-start-server-desc
-      :end-before: end-kes-start-server-desc
 
-b. Start the MinIO Server
+      .. include:: /includes/common/common-minio-kes.rst
+         :start-after: start-kes-start-server-desc
+         :end-before: end-kes-start-server-desc
 
-   .. include:: /includes/common/common-minio-kes.rst
+   b. Start the MinIO Server
-      :start-after: start-kes-minio-start-server-desc
+
-      :end-before: end-kes-minio-start-server-desc
+      .. include:: /includes/common/common-minio-kes.rst
+         :start-after: start-kes-minio-start-server-desc
+         :end-before: end-kes-minio-start-server-desc
 
 Foreground processes depend on the shell or terminal in which they run.
 Exiting or terminating the shell/terminal instance also kills the attached process.
@@ -101,6 +123,10 @@ Defer to your operating system best practices for running processes in the backg
 5) Generate a New Encryption Key
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+.. include:: /includes/common/common-minio-kes-hashicorp.rst
+   :start-after: start-kes-vault-seal-unseal-desc
+   :end-before: end-kes-vault-seal-unseal-desc
+
 .. include:: /includes/common/common-minio-kes.rst
    :start-after: start-kes-generate-key-desc
    :end-before: end-kes-generate-key-desc

source/includes/linux/steps-configure-minio-kes-hashicorp.rst

@@ -1,30 +1,56 @@
-Deploy MinIO and KES with Server-Side Encryption using Hashicorp Vault for Production
+Enable Server-Side Encryption using Hashicorp Vault in Production Environments
--------------------------------------------------------------------------------------
+------------------------------------------------------------------------------
 
-Prior to starting these steps, create the following folders if they do not already exist:
+This procedure provides instructions for configuring and enabling Server-Side Encryption using Hashicorp Vault in production environments. 
+Specifically, this procedure assumes the following:
 
-.. code-block:: shell
+- An existing production-grade Vault deployment
-   :class: copyable
+- One or more hosts for deploying KES
-   :substitutions:
+- One or more hosts for a new or existing MinIO deployment
 
-   mkdir -P |kescertpath|
+.. admonition:: Set Up Folder Hierarchy
-   mkdir -P |kesconfigpath|
+   :class: note
-   mkdir -P |miniocertpath|
+
+   Create the following folders on the KES and MinIO host machines if they do not already exist:
+
+   .. tab-set::
+
+      .. tab-item:: KES Hosts
+
+         .. code-block:: shell
+            :class: copyable
+            :substitutions:
+
+            mkdir -P |kescertpath|
+            mkdir -P |kesconfigpath|
+
+      .. tab-item:: MinIO Hosts
+
+         .. code-block:: shell
+            :class: copyable
+            :substitutions:
+
+            mkdir -P |miniocertpath|
+
+   This procedure uses these paths in the following steps. 
+   If you use different paths, make the necessary modifications for each step to reflect those changes
 
 1) Download KES and Create the Service File
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-a. Download KES
+.. container:: procedure
 
-   .. include:: /includes/linux/common-minio-kes.rst
+   a. Download KES
-      :start-after: start-kes-download-desc
-      :end-before: end-kes-download-desc
 
-b. Create the Service File
+      .. include:: /includes/linux/common-minio-kes.rst
+         :start-after: start-kes-download-desc
+         :end-before: end-kes-download-desc
 
-   .. include:: /includes/linux/common-minio-kes.rst
+   b. Create the Service File
-      :start-after: start-kes-service-file-desc
+
-      :end-before: end-kes-service-file-desc
+      .. include:: /includes/linux/common-minio-kes.rst
+         :start-after: start-kes-service-file-desc
+         :end-before: end-kes-service-file-desc
 
 2) Generate TLS Certificates for KES and MinIO
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -40,48 +66,58 @@ Defer to the client documentation for instructions on trusting a third-party CA.
 3) Create the KES and MinIO Configurations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-a. Create the KES Configuration File
+.. container:: procedure
 
-   Create the configuration file using your preferred text editor.
+   a. Create the KES Configuration File
-   The following example uses ``nano``:
 
-   .. code-block:: shell
+      Create the configuration file using your preferred text editor.
-      :substitutions:
+      The following example uses ``nano``:
-
-      nano /opt/kes/config.yaml
-
-   .. include:: /includes/common/common-minio-kes-hashicorp.rst
-      :start-after: start-kes-configuration-hashicorp-vault-desc
-      :end-before: end-kes-configuration-hashicorp-vault-desc
-
-   - Set ``MINIO_IDENTITY_HASH`` to the identity hash of the MinIO mTLS certificate.
-
-      The following command computes the necessary hash:
 
       .. code-block:: shell
-         :class: copyable
          :substitutions:
 
-         kes identity of |miniocertpath|/minio-kes.cert
+         nano /opt/kes/config.yaml
 
-   - Replace the ``vault.endpoint`` with the hostname of the Vault server(s).
+      .. include:: /includes/common/common-minio-kes-hashicorp.rst
+         :start-after: start-kes-configuration-hashicorp-vault-desc
+         :end-before: end-kes-configuration-hashicorp-vault-desc
 
-   - Replace the ``VAULTAPPID`` and ``VAULTAPPSECRET`` with the appropriate :ref:`Vault AppRole credentials <minio-sse-vault-prereq-vault>`.
+      - Set ``MINIO_IDENTITY_HASH`` to the identity hash of the MinIO mTLS certificate.
 
-b. Configure the MinIO Environment File
+        The following command computes the necessary hash:
 
-   Modify the MinIO Server environment file for all hosts in the target deployment to include the following environment variables.
+        .. code-block:: shell
+           :class: copyable
+           :substitutions:
 
-   MinIO defaults to expecting this file at ``/etc/default/minio``.
+           kes identity of |miniocertpath|/minio-kes.cert
-   If you modified your deployment to use a different location for the environment file, modify the file at that location.
 
-   .. include:: /includes/linux/common-minio-kes.rst
+      - Replace the ``keystore.vault.endpoint`` with the hostname of the Vault server(s).
-      :start-after: start-kes-configuration-minio-desc
+
-      :end-before: end-kes-configuration-minio-desc
+      - Replace ``keystore.vault.engine`` and ``keystore.vault.version`` with the path and version of the KV engine used for storing secrets.
+
+      - Replace the ``VAULTAPPID`` and ``VAULTAPPSECRET`` with the appropriate :ref:`Vault AppRole credentials <minio-sse-vault-prereq-vault>`.
+
+      - Modify the ``keystore.vault.tls.ca`` value to correspond to the path to the Vault :abbr:`CA (Certificate Authority)` certificate used to sign the Vault TLS keys.
+
+   b. Configure the MinIO Environment File
+
+      Modify the MinIO Server environment file for all hosts in the target deployment to include the following environment variables.
+
+      MinIO defaults to expecting this file at ``/etc/default/minio``.
+      If you modified your deployment to use a different location for the environment file, modify the file at that location.
+
+      .. include:: /includes/common/common-minio-kes.rst
+         :start-after: start-kes-configuration-minio-desc
+         :end-before: end-kes-configuration-minio-desc
 
 4) Start KES and MinIO
 ~~~~~~~~~~~~~~~~~~~~~~
 
+.. include:: /includes/common/common-minio-kes-hashicorp.rst
+   :start-after: start-kes-vault-seal-unseal-desc
+   :end-before: end-kes-vault-seal-unseal-desc
+
 You must start KES *before* starting MinIO. 
 The MinIO deployment requires access to KES as part of its startup.
 
@@ -102,7 +138,11 @@ b. Start the MinIO Server
 5) Generate a New Encryption Key
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. include:: /includes/linux/common-minio-kes.rst
+.. include:: /includes/common/common-minio-kes-hashicorp.rst
+   :start-after: start-kes-vault-seal-unseal-desc
+   :end-before: end-kes-vault-seal-unseal-desc
+
+.. include:: /includes/common/common-minio-kes.rst
    :start-after: start-kes-generate-key-desc
    :end-before: end-kes-generate-key-desc
 

source/includes/windows/common-minio-kes.rst

@@ -134,8 +134,6 @@ This command assumes the ``minio-kes.cert``, ``minio-kes.key``, and ``kes-server
    MINIO_KMS_KES_CAPATH=|miniocertpath|\kes-server.cert
    MINIO_KMS_KES_KEY_NAME=minio-backend-default-key
 
-   minio.exe server [ARGUMENTS]
-
 MinIO uses the :envvar:`MINIO_KMS_KES_KEY_NAME` key for the following cryptographic operations:
 
 - Encrypting the MinIO backend (IAM, configuration, etc.)

source/operations/server-side-encryption/configure-minio-kes-hashicorp.rst

@@ -128,59 +128,73 @@ Deploy or Ensure Access to a Hashicorp Vault Service
       :start-after: start-kes-prereq-hashicorp-vault-desc
       :end-before: end-kes-prereq-hashicorp-vault-desc
 
-MinIO |KES| supports both the V1 and V2 Vault engines.
+MinIO |KES| supports either the V1 or V2 Vault `K/V engines <https://www.vaultproject.io/docs/secrets/kv>`__.
-Select the corresponding tab to the engine used by your Vault deployment for instructions on configuring the necessary permissions:
 
-.. tab-set::
+MinIO KES requires using AppRole authentication to the Vault server. 
+You must create an AppRole, assign it a policy that the necessary permissions, and retrieve the AppRole ID and Secret for use in configuring KES.
 
-   .. tab-item:: Vault Engine V1
+You can use the following steps to enable AppRole authentication and create the necessary policies to support core KES functionality against Vault:
 
-      Create an access policy ``kes-policy.hcl`` with a configuration similar to the following:
+1. Enable AppRole Authentication
 
-      .. code-block:: shell
+   .. code-block:: shell
-         :class: copyable
+      :class: copyable
 
-         path "kv/*" {
+      vault auth enable approle
-               capabilities = [ "create", "read", "delete" ]
-         }
 
-      Write the policy to Vault using ``vault policy write kes-policy kes-policy.hcl``.
+#. Create a Policy for KES
 
-   .. tab-item:: Vault Engine V2
+   Create a `policy with necessary capabilities <https://www.vaultproject.io/docs/concepts/policies#capabilities>`__ for KES to use when accessing Vault.
+   Select the tab corresponding to the KV engine used for storing KES secrets:
  
-      Create an access policy ``kes-policy.hcl`` with a configuration similar to the following:
+   .. tab-set::
 
-      .. code-block:: shell
+      .. tab-item:: Vault Engine V1
-         :class: copyable
 
-         path "kv/data/*" {
+         Create an access policy ``kes-policy.hcl`` with a configuration similar to the following:
-               capabilities = [ "create", "read"]
-         }
             
-         path "kv/metadata/*" {
+         .. code-block:: shell
-               capabilities = [ "list", "delete"]
+            :class: copyable
-         }
 
-      Write the policy to Vault using ``vault policy write kes-policy kes-policy.hcl``
+            path "kv/*" {
+                  capabilities = [ "create", "read", "delete" ]
+            }
 
-MinIO requires using AppRole authentication for secure communication with the Vault server.
+         Write the policy to Vault using ``vault policy write kes-policy kes-policy.hcl``.
-The following commands:
 
-- Enable AppRole Authentication
+      .. tab-item:: Vault Engine V2
-- Create an App Role ID for |KES|
-- Binds that role to the created KES policy
-- Requests a RoleID and SecretID
 
-  .. code-block:: shell
+         Create an access policy ``kes-policy.hcl`` with a configuration similar to the following:
-     :class: copyable
 
-     vault auth enable approle
+         .. code-block:: shell
-     vault write    auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m
+            :class: copyable
-     vault write    auth/approle/role/kes-role policies=kes-policy
+
-     vault read     auth/approle/role/kes-role/role-id
+            path "kv/data/*" {
-     vault write -f auth/approle/role/kes-role/secret-id
+                  capabilities = [ "create", "read"]
+            }
+
+            path "kv/metadata/*" {
+                  capabilities = [ "list", "delete"]
+            }
+            
+         Write the policy to Vault using ``vault policy write kes-policy kes-policy.hcl``
+
+#. Create an AppRole for KES and assign it the created policy
+
+   .. code-block:: shell
+      :class: copyable
+
+      vault write    auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m
+      vault write    auth/approle/role/kes-role policies=kes-policy
+
+#. Retrieve the AppRole ID and Secret
+
+   .. code-block:: shell
+      :class: copyable
+
+      vault read     auth/approle/role/kes-role/role-id
+      vault write -f auth/approle/role/kes-role/secret-id
 
-You must specify both RoleID and SecretID as part of this procedure.
 
 .. cond:: linux or macos or windows
 
@@ -232,7 +246,7 @@ You must specify both RoleID and SecretID as part of this procedure.
 .. cond:: macos
 
    .. |kescertpath|        replace:: ~/minio-kes-vault/certs
-   .. |kesconfigpath|      replace:: ~/minio-kes-vault/config/
+   .. |kesconfigpath|      replace:: ~/minio-kes-vault/config
    .. |kesconfigcertpath|  replace:: ~/minio-kes-vault/certs
    .. |miniocertpath|      replace:: ~/minio-kes-vault/certs
    .. |minioconfigpath|    replace:: ~/minio-kes-vault/config
@@ -300,6 +314,10 @@ using Hashicorp Vault as the root Key Management Service (KMS) for |SSE|:
          keystore:
            vault:
              endpoint: https://vault.example.net:8200
+             engine: "kv"
+             version: "v1"
+             namespace: "minio"
+             prefix: "keys"
              approle:
                id: ${KES_APPROLE_ID}
                secret: ${KES_APPROLE_SECRET}
@@ -307,6 +325,8 @@ using Hashicorp Vault as the root Key Management Service (KMS) for |SSE|:
              status:
                ping: 10s
              tls:
+               key: "kes-mtls.key"
+               cert: "kes-mtls.cert"
                ca: vault-tls.cert
 
    .. tab-item:: Reference
@@ -348,15 +368,24 @@ using Hashicorp Vault as the root Key Management Service (KMS) for |SSE|:
            - The configuration for the Hashicorp Vault keystore. The following
              fields are *required*:
 
-             - ``endpoint`` - The hostname for the vault server(s). The 
+             - ``endpoint`` - The hostname for the vault server(s). 
-               hostname *must* be resolvable by the KES server host.
+               The hostname *must* be resolvable by the KES server host.
 
-             - ``approle`` - The `AppRole 
+             - ``engine`` - The path to the K/V engine to use.
-               <https://www.vaultproject.io/docs/auth/approle>`__ used by 
+               Defaults to ``kv``
-               KES for performing authenticated operations against Vault.
 
-               The specified AppRole must have the
+             - ``version`` - The version of the K/V engine to use.
-               appropriate :ref:`permissions <minio-sse-vault-prereq-vault>`
+               
+               Specify either ``v1`` or ``v2``. 
+               Defaults to ``v1``.
+
+             - ``namespace`` - The Vault namespace to use for secret storage.
+
+             - ``prefix`` - The prefix to use for secret storage.
+
+             - ``approle`` - The `AppRole <https://www.vaultproject.io/docs/auth/approle>`__ used by KES for performing authenticated operations against Vault.
+
+               The specified AppRole must have the appropriate :ref:`permissions <minio-sse-vault-prereq-vault>`
 
              - ``tls.ca`` - The Certificate Authority used to sign the 
                Vault TLS certificates. Typically required if the Vault