DOCS-912: Part 1: Cleaning up Vault (#949)

Staged:
http://192.241.195.202:9000/staging/DOCS-912/linux/operations/server-side-encryption/configure-minio-kes-hashicorp.html

---------

Co-authored-by: Andrea Longo <feorlen@users.noreply.github.com>
Co-authored-by: Daryl White <53910321+djwfyi@users.noreply.github.com>

source/default-conf.py

@@ -75,7 +75,7 @@ extlinks = {
     'gcs-docs'        : ('https://cloud.google.com/storage/docs/%s', None),
     'aks-docs'        : ('https://learn.microsoft.com/en-us/azure/aks/%s', None),
     'azure-docs'      : ('https://learn.microsoft.com/en-us/azure/%s', None),
-
+    'kes-docs'        : ('https://min.io/docs/kes/%s', None),
 }
 
 suppress_warnings = [

source/includes/common/common-minio-kes-aws.rst

@@ -34,7 +34,7 @@ Manager:
        - /v1/key/generate/* # e.g. '/minio-'
        - /v1/key/decrypt/*
        - /v1/key/bulk/decrypt
-       - /v1/key/list
+       - /v1/key/list/*
        - /v1/status
        - /v1/metrics
        - /v1/log/audit

source/includes/common/common-minio-kes-azure.rst

@@ -35,7 +35,7 @@ Manager:
        - /v1/key/generate/* # e.g. '/minio-'
        - /v1/key/decrypt/*
        - /v1/key/bulk/decrypt
-       - /v1/key/list
+       - /v1/key/list/*
        - /v1/status
        - /v1/metrics
        - /v1/log/audit

source/includes/common/common-minio-kes-gcp.rst

@@ -34,7 +34,7 @@ Manager:
        - /v1/key/generate/* # e.g. '/minio-'
        - /v1/key/decrypt/*
        - /v1/key/bulk/decrypt
-       - /v1/key/list
+       - /v1/key/list/*
        - /v1/status
        - /v1/metrics
        - /v1/log/audit

source/includes/common/common-minio-kes-hashicorp.rst

@@ -16,12 +16,17 @@ You must modify this YAML to reflect your deployment environment.
      identity: disabled
 
    # Specify the TLS keys generated in the previous step here
-   # For production environments, use keys signed by a known and trusted
+   # For production environments, use keys signed by a known and trusted Certificate Authority (CA).
-   # Certificate Authority (CA).
    tls:
      key:  |kesconfigcertpath|kes-server.key
      cert: |kesconfigcertpath|kes-server.cert
 
+     # Specify the path to CAs used by KES for validating client certificates
+     # This can alternatively be a single CA
+     # KES uses these CAs in addition to the system trust store for validating client certificates.
+
+     ca: |kesconfigcertpath|CAs/
+
    # Sets access policies for KES
    # The `minio` policy grants access to the listed APIs.
    policy:
@@ -31,15 +36,13 @@ You must modify this YAML to reflect your deployment environment.
        - /v1/key/generate/* # e.g. '/minio-'
        - /v1/key/decrypt/*
        - /v1/key/bulk/decrypt
-       - /v1/key/list
+       - /v1/key/list/*
        - /v1/status
        - /v1/metrics
        - /v1/log/audit
        - /v1/log/error
        identities:
-       - MINIO_IDENTITY_HASH # Replace with the output of 'kes identity of minio-kes.cert'
+       - MINIO_API_KEY_HASH # Replace with the hash output returned from kes identity new 
-                                # In production environments, each client connecting to KES must
-                                # Have their TLS hash listed under at least one `policy`.
 
    # Specify the connection information for the Vault server.
    # The endpoint should be resolvable from the host.

source/includes/common/common-minio-kes.rst

@@ -113,17 +113,12 @@ If you run |KES| without tying it to the current shell session (e.g. with ``nohu
 MinIO requires that the |EK| exist on the root KMS *before* performing |SSE| operations using that key. 
 Use ``kes key create`` *or* :mc-cmd:`mc admin kms key create` to add a new |EK| for use with |SSE|.
 
-The following command uses the ``kes key create`` command to add a new External Key (EK) stored on the root KMS server for use with encrypting the MinIO backend.
+The following command uses the :mc-cmd:`mc admin kms key create` command to add a new External Key (EK) stored on the root KMS server for use with encrypting the MinIO backend.
 
 .. code-block:: shell
    :class: copyable
-   :substitutions:
 
-   export KES_SERVER=https://127.0.0.1:7373
+   mc admin kms key create ALIAS KEYNAME
-   export KES_CLIENT_KEY=|miniocertpath|/minio-kes.key
-   export KES_CLIENT_CERT=|miniocertpath|/minio-kes.cert
-
-   kes key create -k encrypted-bucket-key
 
 .. end-kes-generate-key-desc
 
@@ -139,8 +134,7 @@ See the tutorials for :ref:`minio-snsd`, :ref:`minio-snmd`, or :ref:`minio-mnmd`
    # Add these environment variables to the existing environment file
 
    MINIO_KMS_KES_ENDPOINT=https://HOSTNAME:7373
-   MINIO_KMS_KES_CERT_FILE=|miniocertpath|/minio-kes.cert
+   MINIO_KMS_KES_API_KEY="kes:v1:ACTpAsNoaGf2Ow9o5gU8OmcaG6Af/VcZ1Mt7ysuKoBjv"
-   MINIO_KMS_KES_KEY_FILE=|miniocertpath|/minio-kes.key
 
    # Allows validation of the KES Server Certificate (Self-Signed or Third-Party CA)
    # Change this path to the location of the KES CA Path
@@ -150,7 +144,7 @@ See the tutorials for :ref:`minio-snsd`, :ref:`minio-snmd`, or :ref:`minio-mnmd`
    MINIO_KMS_KES_KEY_NAME=minio-backend-default-key
 
    # Optional, defines the name for the KES server enclave to use.
-   MINIO_KMS_KES_ENCLAVE=<name>
+   # MINIO_KMS_KES_ENCLAVE=<name>
 
 Replace ``HOSTNAME`` with the IP address or hostname of the KES server.
 If the MinIO server host machines cannot resolve or reach the specified ``HOSTNAME``, the deployment may return errors or fail to start.
@@ -168,14 +162,11 @@ MinIO uses the :envvar:`MINIO_KMS_KES_KEY_NAME` key for the following cryptograp
 MinIO uses the :envvar:`MINIO_KMS_KES_ENCLAVE` key to define the name of the KES enclave to use.
 
 - Replace ``<name>`` with the name of the :term:`enclave` to use.
-- If not defined, MinIO does not send any enclave information.
+  If not defined, MinIO does not send any enclave information.
   This may result in using the default enclave for stateful KES servers.
 
   A KES :term:`enclave` provides an isolated space for its associated keys separate from other enclaves on a stateful KES server.
 
-The ``minio-kes`` certificates enable mTLS between the MinIO deployment and the KES server *only*.
-They do not otherwise enable TLS for other client connections to MinIO.
-
 .. end-kes-configuration-minio-desc
 
 .. start-kes-enable-sse-kms-desc

source/includes/linux/common-minio-kes.rst

@@ -25,7 +25,7 @@ For more granular controls, deploy a dedicated load balancer to manage connectio
 
 .. start-kes-service-file-desc
 
-Create the ``/etc/systemd/system/kes.service`` file on all KES hosts:
+Create the ``/lib/systemd/system/kes.service`` file on all KES hosts:
 
 .. literalinclude:: /extra/kes.service
    :language: shell
@@ -88,16 +88,15 @@ For existing MinIO deployments, run the following command on each MinIO host to
 
 .. start-kes-generate-kes-certs-prod-desc
 
-Enabling connectivity between MinIO and KES requires at minimum one TLS certificate for performing mutual TLS (mTLS) authentication.
+KES requires TLS connectivity for all client connections, including those originating from MinIO.
+See :ref:`minio-tls` for more information on enabling TLS for the MinIO deployment.
+
 Depending on your Vault configuration, you may also need to create a dedicated set of TLS certificates for KES to connect and authenticate to Vault.
-Defer to your organizations best practices around generating production-ready TLS certificates.
 
-Place the certificates and corresponding private keys an appropriate directory such that the MinIO and KES service users can access and read their contents.
+Defer to your organization's best practices around generating production-ready TLS certificates.
-The following example structure uses the folder hierarchy suggested in the beginning of this procedure:
 
-.. tab-set::
+Place the certificates and corresponding private keys in a directory that the KES service user has permissions to access and read the directory's contents.
-
+For example:
-   .. tab-item:: KES Hosts
 
 .. code-block:: shell
    :substitutions:
@@ -110,20 +109,6 @@ The following example structure uses the folder hierarchy suggested in the begin
 
    -rw-r--r-- 1 kes:kes |kescertpath|/vault-CA.cert
 
-   .. tab-item:: MinIO Hosts
-
-      .. code-block:: shell
-         :substitutions:
-
-         -rw-r--r-- 1 minio-user:minio-user |miniocertpath|/minio-kes.cert
-         -rw-r--r-- 1 minio-user:minio-user |miniocertpath|/minio-kes.key
-
-         # If KES certs are self-signed or use a non-global CA
-         # Include the CA certs as well
-         -rw-r--r-- 1 minio-user:minio-user |miniocertpath|/kes-server.cert
-
-The general strategy for cert management is to ensure that each process (MinIO, KES, and Vault) have their own mTLS certificates *and* the Certificate Authority (CA) used to sign each client certificate.
-
 .. end-kes-generate-kes-certs-prod-desc
 
 .. start-kes-generate-key-desc
@@ -135,12 +120,7 @@ The following command uses the ``kes key create`` command to add a new External
 
 .. code-block:: shell
    :class: copyable
-   :substitutions:
 
-   export KES_SERVER=https://127.0.0.1:7373
+   mc admin kms key create ALIAS KEYNAME
-   export KES_CLIENT_KEY=|miniocertpath|/minio-kes.key
-   export KES_CLIENT_CERT=|miniocertpath|/minio-kes.cert
-
-   kes key create -k encrypted-bucket-key
 
 .. end-kes-generate-key-desc

source/includes/linux/steps-configure-minio-kes-hashicorp-quick.rst

@@ -1,139 +0,0 @@
-Enable Server-Side Encryption using Hashicorp Vault for Local Development
--------------------------------------------------------------------------
-
-This procedure assumes deploying MinIO and KES onto the same host.
-You can use and modify the information below to better suit your local development environment.
-
-This procedure assumes the following:
-
-- An existing Vault deployment
-- A single host machine for deploying KES and MinIO
-
-.. admonition:: Set Up Folder Hierarchy
-   :class: note
-
-   Create the following folders on the host machine if they do not already exist.
-
-   .. code-block:: shell
-      :class: copyable
-      :substitutions:
-
-      mkdir -P |kescertpath|
-      mkdir -P |kesconfigpath|
-      mkdir -P |miniocertpath|
-      mkdir -P |minioconfigpath|
-      mkdir -P |miniodatapath|
-
-   This procedure uses these paths in the following steps. 
-   If you use different paths, make the necessary modifications for each step to reflect those changes
-
-1) Download the KES Server Binary
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. include:: /includes/linux/common-minio-kes.rst
-   :start-after: start-kes-download-desc
-   :end-before: end-kes-download-desc
-
-2) Generate TLS Certificates for KES and MinIO
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. include:: /includes/common/common-minio-kes.rst
-   :start-after: start-kes-generate-kes-certs-desc
-   :end-before: end-kes-generate-kes-certs-desc
-
-Depending on your Vault configuration, you may need to pass the ``kes-server.cert`` certificate as a trusted Certificate Authority. 
-See the `Hashicorp Server Configuration Documentation <https://www.vaultproject.io/docs/configuration/listener/tcp#tls_client_ca_file>`__ for more information.
-Defer to the client documentation for instructions on trusting a third-party CA.
-
-3) Create the KES and MinIO Configurations
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. container:: procedure
-
-   a. Create the KES Configuration File
-
-      Create the configuration file using your preferred text editor.
-      The following example uses ``nano``:
-
-      .. code-block:: shell
-         :substitutions:
-
-         nano |kesconfigpath|/kes-config.yaml
-
-      .. include:: /includes/common/common-minio-kes-hashicorp.rst
-         :start-after: start-kes-configuration-hashicorp-vault-desc
-         :end-before: end-kes-configuration-hashicorp-vault-desc
-
-      - Set ``MINIO_IDENTITY_HASH`` to the identity hash of the MinIO mTLS certificate.
-
-        The following command computes the necessary hash:
-
-        .. code-block:: shell
-           :class: copyable
-           :substitutions:
-
-           kes identity of |miniocertpath|/minio-kes.cert
-
-      - Replace the ``vault.endpoint`` with the hostname of the Vault server(s).
-      - Set the ``vault.engine`` and ``vault.version`` to the appropriate values for the Vault K/V Engine configuration
-      - Replace the ``VAULTAPPID`` and ``VAULTAPPSECRET`` with the appropriate :ref:`Vault AppRole credentials <minio-sse-vault-prereq-vault>`.
-
-   b. Create the MinIO Environment File
-
-      Create or modify the environment file for the MinIO deployment using your preferred text editor.
-      The following example uses ``nano``:
-
-      .. code-block:: shell
-         :substitutions:
-
-         nano |minioconfigpath|/minio
-
-      .. include:: /includes/common/common-minio-kes.rst
-         :start-after: start-kes-configuration-minio-desc
-         :end-before: end-kes-configuration-minio-desc
-
-4) Start KES and MinIO
-~~~~~~~~~~~~~~~~~~~~~~
-
-.. include:: /includes/common/common-minio-kes-hashicorp.rst
-   :start-after: start-kes-vault-seal-unseal-desc
-   :end-before: end-kes-vault-seal-unseal-desc
-
-You must start KES *before* starting MinIO. 
-The MinIO deployment requires access to KES as part of its startup.
-
-.. container:: procedure
-
-   a. Start the KES Server
-
-      .. include:: /includes/common/common-minio-kes.rst
-         :start-after: start-kes-start-server-desc
-         :end-before: end-kes-start-server-desc
-
-   b. Start the MinIO Server
-
-      .. include:: /includes/common/common-minio-kes.rst
-         :start-after: start-kes-minio-start-server-desc
-         :end-before: end-kes-minio-start-server-desc
-
-Foreground processes depend on the shell or terminal in which they run.
-Exiting or terminating the shell/terminal instance also kills the attached process.
-Defer to your operating system best practices for running processes in the background.
-
-5) Generate a New Encryption Key
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. include:: /includes/common/common-minio-kes-hashicorp.rst
-   :start-after: start-kes-vault-seal-unseal-desc
-   :end-before: end-kes-vault-seal-unseal-desc
-
-.. include:: /includes/common/common-minio-kes.rst
-   :start-after: start-kes-generate-key-desc
-   :end-before: end-kes-generate-key-desc
-
-6) Enable SSE-KMS for a Bucket
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. include:: /includes/common/common-minio-kes.rst
-   :start-after: start-kes-enable-sse-kms-desc
-   :end-before: end-kes-enable-sse-kms-desc

source/includes/linux/steps-configure-minio-kes-hashicorp.rst

@@ -1,5 +1,5 @@
-Enable Server-Side Encryption using Hashicorp Vault in Production Environments
+Procedure
-------------------------------------------------------------------------------
+---------
 
 This procedure provides instructions for configuring and enabling Server-Side Encryption using Hashicorp Vault in production environments. 
 Specifically, this procedure assumes the following:
@@ -8,33 +8,6 @@ Specifically, this procedure assumes the following:
 - One or more hosts for deploying KES
 - One or more hosts for a new or existing MinIO deployment
 
-.. admonition:: Set Up Folder Hierarchy
-   :class: note
-
-   Create the following folders on the KES and MinIO host machines if they do not already exist:
-
-   .. tab-set::
-
-      .. tab-item:: KES Hosts
-
-         .. code-block:: shell
-            :class: copyable
-            :substitutions:
-
-            mkdir -P |kescertpath|
-            mkdir -P |kesconfigpath|
-
-      .. tab-item:: MinIO Hosts
-
-         .. code-block:: shell
-            :class: copyable
-            :substitutions:
-
-            mkdir -P |miniocertpath|
-
-   This procedure uses these paths in the following steps. 
-   If you use different paths, make the necessary modifications for each step to reflect those changes
-
 1) Download KES and Create the Service File
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -52,8 +25,8 @@ Specifically, this procedure assumes the following:
          :start-after: start-kes-service-file-desc
          :end-before: end-kes-service-file-desc
 
-2) Generate TLS Certificates for KES and MinIO
+2) Generate TLS Certificates for KES
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. include:: /includes/linux/common-minio-kes.rst
    :start-after: start-kes-generate-kes-certs-prod-desc
@@ -63,7 +36,21 @@ Depending on your Vault configuration, you may also need to specify the CA used
 See the `Hashicorp Vault Configuration Docs <https://www.vaultproject.io/docs/configuration/listener/tcp#tls_client_ca_file>`__ for more information.
 Defer to the client documentation for instructions on trusting a third-party CA.
 
-3) Create the KES and MinIO Configurations
+3) Generate a KES API Key for use by MinIO
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Starting with KES version :minio-git:`2023-02-15T14-54-37Z <kes/releases/tag/2023-02-15T14-54-37Z>`, you can generate an API key to use for authenticating to the KES server.
+
+Use the :kes-docs:`kes identity new <cli/kes-identity/new>` command to generate a new API key for use by the MinIO Server:
+
+.. code-block:: shell
+   :class: copyable
+
+   kes identity new
+
+The output includes both the API Key for use with MinIO and the Identity hash for use with the :kes-docs:`KES Policy configuration <tutorials/configuration/#policy-configuration>`.
+
+4) Create the KES and MinIO Configurations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. important::
@@ -87,15 +74,14 @@ Defer to the client documentation for instructions on trusting a third-party CA.
          :start-after: start-kes-configuration-hashicorp-vault-desc
          :end-before: end-kes-configuration-hashicorp-vault-desc
 
-      - Set ``MINIO_IDENTITY_HASH`` to the identity hash of the MinIO mTLS certificate.
+      - Set ``MINIO_IDENTITY_HASH`` to the identity hash of the API Key generated in the previous step.
 
-        The following command computes the necessary hash:
+        The following command recomputes the necessary hash from the API key:
 
         .. code-block:: shell
            :class: copyable
-           :substitutions:
 
-           kes identity of |miniocertpath|/minio-kes.cert
+           kes identity of kes:v1:KEY/KEY
 
       - Replace the ``keystore.vault.endpoint`` with the hostname of the Vault server(s).
 
@@ -107,16 +93,16 @@ Defer to the client documentation for instructions on trusting a third-party CA.
 
    b. Configure the MinIO Environment File
 
-      Modify the MinIO Server environment file for all hosts in the target deployment to include the following environment variables.
+      Create or modify the MinIO Server environment file for all hosts in the target deployment to include the following environment variables:
-
-      MinIO defaults to expecting this file at ``/etc/default/minio``.
-      If you modified your deployment to use a different location for the environment file, modify the file at that location.
 
       .. include:: /includes/common/common-minio-kes.rst
          :start-after: start-kes-configuration-minio-desc
          :end-before: end-kes-configuration-minio-desc
 
-4) Start KES and MinIO
+      MinIO defaults to expecting this file at ``/etc/default/minio``.
+      If you modified your deployment to use a different location for the environment file, modify the file at that location.
+
+5) Start KES and MinIO
 ~~~~~~~~~~~~~~~~~~~~~~
 
 .. include:: /includes/common/common-minio-kes-hashicorp.rst

source/operations/network-encryption.rst

@@ -283,6 +283,8 @@ MinIO supports the following TLS 1.2 and 1.3 cipher suites as supported by `Go <
       - ``TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256``
       - ``TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384``      
 
+.. _minio-TLS-third-party-ca:
+
 Third-Party Certificate Authorities
 -----------------------------------
 

source/operations/server-side-encryption/configure-minio-kes-aws.rst

@@ -275,7 +275,7 @@ The following section describes each of the |KES-git| configuration settings for
              - /v1/key/generate/*
              - /v1/key/decrypt/*
              - /v1/key/bulk/decrypt
-             - /v1/key/list
+             - /v1/key/list/*
              - /v1/status
              - /v1/metrics
              - /v1/log/audit

source/operations/server-side-encryption/configure-minio-kes-azure.rst

@@ -259,7 +259,7 @@ using Azure Key Vault as the root Key Management Service
              - /v1/key/generate/*
              - /v1/key/decrypt/*
              - /v1/key/bulk/decrypt
-             - /v1/key/list
+             - /v1/key/list/*
              - /v1/status
              - /v1/metrics
              - /v1/log/audit

source/operations/server-side-encryption/configure-minio-kes-gcp.rst

@@ -265,7 +265,7 @@ The following section describes each of the |KES-git| configuration settings for
              - /v1/key/generate/*
              - /v1/key/decrypt/*
              - /v1/key/bulk/decrypt
-             - /v1/key/list
+             - /v1/key/list/*
              - /v1/status
              - /v1/metrics
              - /v1/log/audit

source/operations/server-side-encryption/configure-minio-kes-hashicorp.rst

@@ -18,18 +18,11 @@ Server-Side Object Encryption with Hashicorp Vault Root KMS
 .. |rootkms|       replace:: `Hashicorp Vault <https://vaultproject.io/>`__
 .. |rootkms-short| replace:: Vault
 
-MinIO Server-Side Encryption (SSE) protects objects as part of write operations, allowing clients to take advantage of server processing power to secure objects at the storage layer (encryption-at-rest). 
-SSE also provides key functionality to regulatory and compliance requirements around secure locking and erasure.
-
-MinIO SSE uses |KES-git| and an external root Key Management Service (KMS) for performing secured cryptographic operations at scale. 
-The root KMS provides stateful and secured storage of External Keys (EK) while |KES| is stateless and derives additional cryptographic keys from the root-managed |EK|. 
-
 .. Conditionals to handle the slight divergences in procedures between platforms.
 
 .. cond:: linux
 
-   This procedure provides guidance for deploying and configuring KES at scale for a supporting |SSE| on a production MinIO deployment.
+   This procedure provides guidance for deploying MinIO configured to use KES and enable :ref:`Server Side Encryption <minio-sse-data-encryption>`.
-   You can also use this procedure for deploying to local environments for testing and evaluation.
 
    As part of this procedure, you will:
 
@@ -43,8 +36,6 @@ The root KMS provides stateful and secured storage of External Keys (EK) while |
 
    #. Configure automatic bucket-default :ref:`SSE-KMS <minio-encryption-sse-kms>`
 
-   For production orchestrated environments, use the MinIO Kubernetes Operator to deploy a tenant with |SSE| enabled and configured for use with Hashicorp Vault.
-
 .. cond:: macos or windows
 
    This procedure assumes a single local host machine running the MinIO and KES processes.
@@ -239,8 +230,6 @@ You can use the following steps to enable AppRole authentication and create the
    .. |minioconfigpath|    replace:: /opt/minio/config
    .. |miniodatapath|      replace:: ~/minio
 
-   .. include:: /includes/linux/steps-configure-minio-kes-hashicorp-quick.rst
-
    .. include:: /includes/linux/steps-configure-minio-kes-hashicorp.rst
 
 .. cond:: macos
@@ -296,7 +285,8 @@ The following section describes each of the |KES-git| configuration settings for
       .. code-block:: yaml
 
          address: 0.0.0.0:7373
-         root: ${ROOT_IDENTITY}
+         admin: 
+           identity: ${ROOT_IDENTITY}
 
          tls:
            key: kes-server.key