Updates for MinIO RELEASE.2023-04-07T05-28-58Z (#823)

- Adds new environment variable for ILM expiration workers
- Adds new metrics for locks
- Adds keyrotate batch job type
- Adds info about batch replicate from remote to local
    
Closes #806

Staged:
http://192.241.195.202:9000/staging/minio-2023-04-07/administration/batch-framework.html

source/administration/batch-framework.rst

@@ -40,6 +40,9 @@ The MinIO Batch Framework supports the following job types:
    * - ``replicate``
      - Perform a one-time replication procedure from one MinIO location to another MinIO location.
 
+   * - ``keyrotate``
+     - Perform a one-time process to cycle the :ref:`sse-s3 or sse-kms <minio-sse-data-encryption>` cryptographic keys on objects.
+
 MinIO Batch CLI
 ---------------
 
@@ -84,10 +87,16 @@ Job Types
 Replicate
 ~~~~~~~~~
 
-Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location.
+Use the ``replicate`` job type to create a batch job that replicates objects from one MinIO deployment to another MinIO location.
+At least one of the deployment locations, either the source or the target, must be ``local``.
 The definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects.
 
-For example, you can use a batch job to perform a one-time replication sync of objects from ``minio-alpha/invoices/`` to ``minio-baker/invoices``.
+.. versionchanged:: MinIO RELEASE.2023-04-07T05-28-58Z
+
+   You can replicate from a remote MinIO deployment to the local deployment that runs the batch job.
+
+For example, you can use a batch job to perform a one-time replication sync to push objects from a bucket on a local deployment at ``minio-local/invoices/`` to a bucket on a remote deployment at ``minio-remote/invoices``.
+You can also pull objects from the remote deployment at ``minio-remote/invoices`` to the local deployment at ``minio-local/invoices``.
 
 The advantages of Batch Replication over :mc:`mc mirror` include:
 
@@ -106,4 +115,22 @@ Sample YAML Description File for a ``replicate`` Job Type
 Create a basic ``replicate`` job definition file you can edit with :mc:`mc batch generate`.
 
 .. literalinclude:: /includes/code/replicate.yaml
-   :language: yaml
+   :language: yaml
+
+Key Rotate
+~~~~~~~~~~
+
+.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z
+
+Use the ``keyrotate`` job type to create a batch job that cycles the :ref:`sse-s3 or sse-kms keys <minio-sse-data-encryption>` for encrypted objects.
+
+The YAML configuration supports filters to restrict key rotation to  a specific set of objects by creation date, tags, metadata, or kms key.
+You can also define retry attempts or set a notification endpoint and token.
+
+Sample YAML Description File for a ``keyrotate`` Job Type
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+Create a basic ``keyrotate`` job definition file you can edit with :mc:`mc batch generate`.
+
+.. literalinclude:: /includes/code/replicate.yaml
+   :language: yaml

source/includes/code/keyrotate.yaml

@@ -0,0 +1,37 @@
+keyrotate:
+  apiVersion: v1
+  bucket: bucket
+  prefix: 
+  encryption:
+    type: sse-kms # valid values are sse-s3 and sse-kms
+    
+    # The following encryption values only apply for sse-kms type.
+    # For sse-s3 key types, MinIO uses the key provided by the MINIO_KMS_KES_KEY_FILE environment variable.
+    # The following two values are ignored if type is set to sse-s3.
+    key: my-new-keys2 # valid only for sse-kms
+    context: <new-kms-key-context> # valid only for sse-kms
+
+  # optional flags based filtering criteria
+  flags:
+    filter:
+      newerThan: "84h" # match objects newer than this value (e.g. 7d10h31s)
+      olderThan: "80h" # match objects older than this value (e.g. 7d10h31s)
+      createdAfter: "2023-03-02T15:04:05Z07:00" # match objects created after "date"
+      createdBefore: "2023-03-02T15:04:05Z07:00" # match objects created before "date"
+      tags:
+        - key: "name"
+          value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
+      metadata:
+        - key: "content-type"
+          value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
+      kmskey: "key-id" # match objects with KMS key-id (applicable only for sse-kms)
+  
+  # optional entries to add notifications for the job
+  notify:
+    endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
+    token: "Bearer xxxxx" # optional authentication token for the notification endpoint
+  
+  # optional entries to add retry attempts if the job is interrupted
+  retry:
+    attempts: 10 # number of retries for the job before giving up
+    delay: "500ms" # least amount of delay between each retry

source/operations/monitoring/metrics-and-alerts.rst

@@ -577,6 +577,21 @@ Software and Process Metrics
    
    Resident memory size in bytes.
 
+Lock Metrics
+~~~~~~~~~~~~
+
+.. metric:: minio_locks_total
+
+   Total number of current locks on the peer.
+
+.. metric:: minio_locks_write_total
+
+   Number of current WRITE locks on the peer.
+
+.. metric:: minio_locks_read_total
+
+   Number of current READ locks on the peer.
+
 .. toctree::
    :titlesonly:
    :hidden:

source/operations/server-side-encryption.rst

@@ -1,3 +1,5 @@
+.. _minio-sse-data-encryption:
+
 =====================
 Data Encryption (SSE)
 =====================

source/reference/minio-mc-admin/mc-admin-rebalance.rst

@@ -28,7 +28,7 @@ Rebalancing redistributes objects across all pools in the deployment.
 .. end-mc-admin-rebalance-desc
 
 MinIO does not automatically rebalance objects when adding a new server pool.
-Instead, MinIO ref:`writes new objects <minio-writing-files>` to the pool with relatively more free space compared to the other available pools on the deployment.
+Instead, MinIO :ref:`writes new objects <minio-writing-files>` to the pool with relatively more free space compared to the other available pools on the deployment.
 Triggering a manual rebalancing procedure prompts MinIO to scan the entire deployment and move objects as necessary to achieve a similar available free space across all pools.
 
 This is an expensive and time consuming operation.

source/reference/minio-mc/mc-batch-generate.rst

@@ -74,7 +74,7 @@ Parameters
    
    The type of job to generate a YAML document for.
    
-   Currently, :mc:`mc batch` only supports the ``replicate`` job type.
+   Currently, :mc:`mc batch` supports the ``replicate`` and ``keyrotate`` job types.
 
 
 Global Flags
@@ -101,7 +101,7 @@ The following command generates a YAML blueprint for a replicate type batch job
 
 - Replace ``replicate`` with the type of job to generate a yaml file for.
  
-  At the time of release, :mc:``mc batch`` only supports the ``replicate`` job type. 
+  :mc:``mc batch`` supports the ``replicate`` and ``keyrotate`` job types. 
 
 
 S3 Compatibility
@@ -123,7 +123,11 @@ Job Types
   Replicate objects between two MinIO deployments.
   Provides similar functionality to :ref:`bucket replication <minio-bucket-replication>` as a batch job rather than continual scanning function.
 
-MinIO may add more job types in the future.
+- ``keyrotate``
+
+  .. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z 
+  
+  Rotate the sse-s3 or sse-kms keys for objects at rest on a MinIO deployment.
 
 ``replicate``
 ~~~~~~~~~~~~~
@@ -131,8 +135,13 @@ MinIO may add more job types in the future.
 Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location.
 
 The YAML **must** define the source and target deployments.
+If the _source_ deployment is remote, then the _target_ deployment **must** be ``local``.
 Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.
 
+.. versionchanged:: MinIO RELEASE.2023-04-07T05-28-58Z
+
+   You can replicate from a remote MinIO deployment to the local deployment that runs the batch job.
+
 For the **source deployment**
 
 - Required information
@@ -156,7 +165,8 @@ For the **source deployment**
        - The prefix on the object(s) that should replicate.
 
      * - ``endpoint:`` 
-       - | Location of the source deployment, must be ``local``.
+       - | Location of the source deployment.
+         | If the location is not remote, use ``local``.
 
      * - ``credentials:`` 
        - The ``accesskey:`` and ``secretKey:`` or the ``sessionToken:`` that grants access to the object(s).
@@ -186,6 +196,7 @@ For the **target deployment**
      * - ``endpoint:`` 
        - | The location of the source deployment.
          | If the location is not remote, use ``local``.
+         | If the location of the source is remote, the source for target **must** be ``local``. 
 
      * - ``credentials:`` 
        - The ``accesskey`` and ``secretKey`` or the ``sessionToken`` that grants access to the object(s).
@@ -245,3 +256,93 @@ Sample YAML
 
 .. literalinclude:: /includes/code/replicate.yaml
    :language: yaml
+
+``keyrotate``
+~~~~~~~~~~~~~
+
+.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z 
+
+Use the ``keyrotate`` job type to create a batch job that cycles the :ref:`sse-s3 or sse-kms keys <minio-sse-data-encryption>` for encrypted objects.
+
+Required information
+++++++++++++++++++++
+
+  .. list-table::
+     :widths: 25 75
+     :width: 100%
+
+     * - ``type:`` 
+       - Either ``sse-s3`` or ``sse-kms``.
+     * - ``key:`` 
+       - Only for use with the ``sse-kms`` type. 
+         The key to use to unseal the key vault.
+     * - ``context:``
+       - Only for use with the ``sse-kms`` type.
+         The context within which to perform actions. 
+
+   
+Optional information
+++++++++++++++++++++
+
+For **flag based filters**
+
+.. list-table::
+   :widths: 25 75
+   :width: 100%
+
+   * - ``newerThan:`` 
+     - A string representing a length of time in ``#d#h#s`` format.
+       
+       Keys rotate only for objects newer than the specified length of time.
+       For example, ``7d``, ``24h``, ``5d12h30s`` are valid strings.
+   * - ``olderThan:`` 
+     - A string representing a length of time in ``#d#h#s`` format.
+       
+       Keys rotate only for objects older than the specified length of time.
+   * - ``createdAfter:`` 
+     - A date in ``YYYY-MM-DD`` format.
+  
+       Keys rotate only for objects created after the date.
+   * - ``createdBefore:`` 
+     - A date in ``YYYY-MM-DD`` format.
+       
+       Keys rotate only for objects created prior to the date.
+   * - ``tags:``
+     - Rotate keys only for objects with tags that match the specified ``key:`` and ``value:``.  
+   * - ``metadtaa:``
+     - Rotate keys only for objects with metadata that match the specified ``key:`` and ``value:``.  
+   * - ``kmskey:``
+     - Rotate keys only for objects with a KMS key-id that match the specified value.  
+       This is only applicable for the ``sse-kms`` type. 
+
+For **notifications**
+
+.. list-table::
+   :widths: 25 75
+   :width: 100%
+
+   * - ``endpoint:`` 
+     - The predefined endpoint to send events for notifications.
+   * - ``token:`` 
+     - An optional :abbr:`JWT <JSON Web Token>` to access the ``endpoint``.
+
+For **retry attempts**
+
+If something interrupts the job, you can define a maximum number of retry attempts.
+For each retry, you can also define how long to wait between attempts.
+
+.. list-table::
+   :widths: 25 75
+   :width: 100%
+
+   * - ``attempts:`` 
+     - Number of tries to complete the batch job before giving up.
+   * - ``delay:`` 
+     - The amount of time to wait between each attempt.
+
+
+Sample YAML
++++++++++++
+
+.. literalinclude:: /includes/code/keyrotate.yaml
+   :language: yaml

source/reference/minio-mc/mc-mv.rst

@@ -222,7 +222,7 @@ Parameters
    Recursively move the contents of each bucket or directory
    :mc-cmd:`~mc mv SOURCE` to the :mc-cmd:`~mc mv TARGET` bucket.
 
-.. mc-cmd:: --storage-class, sc
+.. mc-cmd:: --storage-class
    :optional:
 
    Set the storage class for the new object(s) on the 
@@ -313,7 +313,7 @@ Move Bucket Between S3-Compatible Services
 Move File to S3-Compatible Host with Specific Storage Class
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Use :mc:`mc mv` with the :mc-cmd:`~mc mv storage-class` option to set
+Use :mc:`mc mv` with the :mc-cmd:`~mc mv --storage-class` option to set
 the storage class on the destination S3-compatible host.
 
 .. code-block:: shell
@@ -321,7 +321,7 @@ the storage class on the destination S3-compatible host.
 
    mc mv --storage-class CLASS FILEPATH ALIAS/PATH
 
-- Replace :mc-cmd:`CLASS <mc mv storage-class>` with the storage class to 
+- Replace :mc-cmd:`CLASS <mc mv --storage-class>` with the storage class to 
   associate to the files.
 
 - Replace :mc-cmd:`FILEPATH <mc mv SOURCE>` with the full file path to the

source/reference/minio-server/minio-server.rst

@@ -222,6 +222,11 @@ Core Configuration
 
    Specifies the full path to the file the MinIO server process uses for loading environment variables.
 
+.. envvar:: MINIO_ILM_EXPIRY_WORKERS
+
+   Specifies the number of workers to make available to expire objects configured with ILM rules for expiration.
+   When not set, MinIO defaults to using up to half of the available processing cores available.
+
 Root Credentials
 ~~~~~~~~~~~~~~~~