diff --git a/source/administration/batch-framework.rst b/source/administration/batch-framework.rst index d2e0d30a..74b1d85b 100644 --- a/source/administration/batch-framework.rst +++ b/source/administration/batch-framework.rst @@ -40,6 +40,9 @@ The MinIO Batch Framework supports the following job types: * - ``replicate`` - Perform a one-time replication procedure from one MinIO location to another MinIO location. + * - ``keyrotate`` + - Perform a one-time process to cycle the :ref:`sse-s3 or sse-kms ` cryptographic keys on objects. + MinIO Batch CLI --------------- @@ -84,10 +87,16 @@ Job Types Replicate ~~~~~~~~~ -Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location. +Use the ``replicate`` job type to create a batch job that replicates objects from one MinIO deployment to another MinIO location. +At least one of the deployment locations, either the source or the target, must be ``local``. The definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects. -For example, you can use a batch job to perform a one-time replication sync of objects from ``minio-alpha/invoices/`` to ``minio-baker/invoices``. +.. versionchanged:: MinIO RELEASE.2023-04-07T05-28-58Z + + You can replicate from a remote MinIO deployment to the local deployment that runs the batch job. + +For example, you can use a batch job to perform a one-time replication sync to push objects from a bucket on a local deployment at ``minio-local/invoices/`` to a bucket on a remote deployment at ``minio-remote/invoices``. +You can also pull objects from the remote deployment at ``minio-remote/invoices`` to the local deployment at ``minio-local/invoices``. The advantages of Batch Replication over :mc:`mc mirror` include: @@ -106,4 +115,22 @@ Sample YAML Description File for a ``replicate`` Job Type Create a basic ``replicate`` job definition file you can edit with :mc:`mc batch generate`. .. literalinclude:: /includes/code/replicate.yaml - :language: yaml \ No newline at end of file + :language: yaml + +Key Rotate +~~~~~~~~~~ + +.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z + +Use the ``keyrotate`` job type to create a batch job that cycles the :ref:`sse-s3 or sse-kms keys ` for encrypted objects. + +The YAML configuration supports filters to restrict key rotation to a specific set of objects by creation date, tags, metadata, or kms key. +You can also define retry attempts or set a notification endpoint and token. + +Sample YAML Description File for a ``keyrotate`` Job Type ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Create a basic ``keyrotate`` job definition file you can edit with :mc:`mc batch generate`. + +.. literalinclude:: /includes/code/replicate.yaml + :language: yaml diff --git a/source/includes/code/keyrotate.yaml b/source/includes/code/keyrotate.yaml new file mode 100644 index 00000000..2b4ae76e --- /dev/null +++ b/source/includes/code/keyrotate.yaml @@ -0,0 +1,37 @@ +keyrotate: + apiVersion: v1 + bucket: bucket + prefix: + encryption: + type: sse-kms # valid values are sse-s3 and sse-kms + + # The following encryption values only apply for sse-kms type. + # For sse-s3 key types, MinIO uses the key provided by the MINIO_KMS_KES_KEY_FILE environment variable. + # The following two values are ignored if type is set to sse-s3. + key: my-new-keys2 # valid only for sse-kms + context: # valid only for sse-kms + + # optional flags based filtering criteria + flags: + filter: + newerThan: "84h" # match objects newer than this value (e.g. 7d10h31s) + olderThan: "80h" # match objects older than this value (e.g. 7d10h31s) + createdAfter: "2023-03-02T15:04:05Z07:00" # match objects created after "date" + createdBefore: "2023-03-02T15:04:05Z07:00" # match objects created before "date" + tags: + - key: "name" + value: "pick*" # match objects with tag 'name', with all values starting with 'pick' + metadata: + - key: "content-type" + value: "image/*" # match objects with 'content-type', with all values starting with 'image/' + kmskey: "key-id" # match objects with KMS key-id (applicable only for sse-kms) + + # optional entries to add notifications for the job + notify: + endpoint: "https://notify.endpoint" # notification endpoint to receive job status events + token: "Bearer xxxxx" # optional authentication token for the notification endpoint + + # optional entries to add retry attempts if the job is interrupted + retry: + attempts: 10 # number of retries for the job before giving up + delay: "500ms" # least amount of delay between each retry \ No newline at end of file diff --git a/source/operations/monitoring/metrics-and-alerts.rst b/source/operations/monitoring/metrics-and-alerts.rst index 2ef7aaeb..0b153198 100644 --- a/source/operations/monitoring/metrics-and-alerts.rst +++ b/source/operations/monitoring/metrics-and-alerts.rst @@ -577,6 +577,21 @@ Software and Process Metrics Resident memory size in bytes. +Lock Metrics +~~~~~~~~~~~~ + +.. metric:: minio_locks_total + + Total number of current locks on the peer. + +.. metric:: minio_locks_write_total + + Number of current WRITE locks on the peer. + +.. metric:: minio_locks_read_total + + Number of current READ locks on the peer. + .. toctree:: :titlesonly: :hidden: diff --git a/source/operations/server-side-encryption.rst b/source/operations/server-side-encryption.rst index 729975ce..f50757b9 100644 --- a/source/operations/server-side-encryption.rst +++ b/source/operations/server-side-encryption.rst @@ -1,3 +1,5 @@ +.. _minio-sse-data-encryption: + ===================== Data Encryption (SSE) ===================== diff --git a/source/reference/minio-mc-admin/mc-admin-rebalance.rst b/source/reference/minio-mc-admin/mc-admin-rebalance.rst index 47183aa0..94ec823e 100644 --- a/source/reference/minio-mc-admin/mc-admin-rebalance.rst +++ b/source/reference/minio-mc-admin/mc-admin-rebalance.rst @@ -28,7 +28,7 @@ Rebalancing redistributes objects across all pools in the deployment. .. end-mc-admin-rebalance-desc MinIO does not automatically rebalance objects when adding a new server pool. -Instead, MinIO ref:`writes new objects ` to the pool with relatively more free space compared to the other available pools on the deployment. +Instead, MinIO :ref:`writes new objects ` to the pool with relatively more free space compared to the other available pools on the deployment. Triggering a manual rebalancing procedure prompts MinIO to scan the entire deployment and move objects as necessary to achieve a similar available free space across all pools. This is an expensive and time consuming operation. diff --git a/source/reference/minio-mc/mc-batch-generate.rst b/source/reference/minio-mc/mc-batch-generate.rst index e66bb63c..5840d7f4 100644 --- a/source/reference/minio-mc/mc-batch-generate.rst +++ b/source/reference/minio-mc/mc-batch-generate.rst @@ -74,7 +74,7 @@ Parameters The type of job to generate a YAML document for. - Currently, :mc:`mc batch` only supports the ``replicate`` job type. + Currently, :mc:`mc batch` supports the ``replicate`` and ``keyrotate`` job types. Global Flags @@ -101,7 +101,7 @@ The following command generates a YAML blueprint for a replicate type batch job - Replace ``replicate`` with the type of job to generate a yaml file for. - At the time of release, :mc:``mc batch`` only supports the ``replicate`` job type. + :mc:``mc batch`` supports the ``replicate`` and ``keyrotate`` job types. S3 Compatibility @@ -123,7 +123,11 @@ Job Types Replicate objects between two MinIO deployments. Provides similar functionality to :ref:`bucket replication ` as a batch job rather than continual scanning function. -MinIO may add more job types in the future. +- ``keyrotate`` + + .. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z + + Rotate the sse-s3 or sse-kms keys for objects at rest on a MinIO deployment. ``replicate`` ~~~~~~~~~~~~~ @@ -131,8 +135,13 @@ MinIO may add more job types in the future. Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location. The YAML **must** define the source and target deployments. +If the _source_ deployment is remote, then the _target_ deployment **must** be ``local``. Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job. +.. versionchanged:: MinIO RELEASE.2023-04-07T05-28-58Z + + You can replicate from a remote MinIO deployment to the local deployment that runs the batch job. + For the **source deployment** - Required information @@ -156,7 +165,8 @@ For the **source deployment** - The prefix on the object(s) that should replicate. * - ``endpoint:`` - - | Location of the source deployment, must be ``local``. + - | Location of the source deployment. + | If the location is not remote, use ``local``. * - ``credentials:`` - The ``accesskey:`` and ``secretKey:`` or the ``sessionToken:`` that grants access to the object(s). @@ -186,6 +196,7 @@ For the **target deployment** * - ``endpoint:`` - | The location of the source deployment. | If the location is not remote, use ``local``. + | If the location of the source is remote, the source for target **must** be ``local``. * - ``credentials:`` - The ``accesskey`` and ``secretKey`` or the ``sessionToken`` that grants access to the object(s). @@ -245,3 +256,93 @@ Sample YAML .. literalinclude:: /includes/code/replicate.yaml :language: yaml + +``keyrotate`` +~~~~~~~~~~~~~ + +.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z + +Use the ``keyrotate`` job type to create a batch job that cycles the :ref:`sse-s3 or sse-kms keys ` for encrypted objects. + +Required information +++++++++++++++++++++ + + .. list-table:: + :widths: 25 75 + :width: 100% + + * - ``type:`` + - Either ``sse-s3`` or ``sse-kms``. + * - ``key:`` + - Only for use with the ``sse-kms`` type. + The key to use to unseal the key vault. + * - ``context:`` + - Only for use with the ``sse-kms`` type. + The context within which to perform actions. + + +Optional information +++++++++++++++++++++ + +For **flag based filters** + +.. list-table:: + :widths: 25 75 + :width: 100% + + * - ``newerThan:`` + - A string representing a length of time in ``#d#h#s`` format. + + Keys rotate only for objects newer than the specified length of time. + For example, ``7d``, ``24h``, ``5d12h30s`` are valid strings. + * - ``olderThan:`` + - A string representing a length of time in ``#d#h#s`` format. + + Keys rotate only for objects older than the specified length of time. + * - ``createdAfter:`` + - A date in ``YYYY-MM-DD`` format. + + Keys rotate only for objects created after the date. + * - ``createdBefore:`` + - A date in ``YYYY-MM-DD`` format. + + Keys rotate only for objects created prior to the date. + * - ``tags:`` + - Rotate keys only for objects with tags that match the specified ``key:`` and ``value:``. + * - ``metadtaa:`` + - Rotate keys only for objects with metadata that match the specified ``key:`` and ``value:``. + * - ``kmskey:`` + - Rotate keys only for objects with a KMS key-id that match the specified value. + This is only applicable for the ``sse-kms`` type. + +For **notifications** + +.. list-table:: + :widths: 25 75 + :width: 100% + + * - ``endpoint:`` + - The predefined endpoint to send events for notifications. + * - ``token:`` + - An optional :abbr:`JWT ` to access the ``endpoint``. + +For **retry attempts** + +If something interrupts the job, you can define a maximum number of retry attempts. +For each retry, you can also define how long to wait between attempts. + +.. list-table:: + :widths: 25 75 + :width: 100% + + * - ``attempts:`` + - Number of tries to complete the batch job before giving up. + * - ``delay:`` + - The amount of time to wait between each attempt. + + +Sample YAML ++++++++++++ + +.. literalinclude:: /includes/code/keyrotate.yaml + :language: yaml \ No newline at end of file diff --git a/source/reference/minio-mc/mc-mv.rst b/source/reference/minio-mc/mc-mv.rst index de7c512b..43f5bf29 100644 --- a/source/reference/minio-mc/mc-mv.rst +++ b/source/reference/minio-mc/mc-mv.rst @@ -222,7 +222,7 @@ Parameters Recursively move the contents of each bucket or directory :mc-cmd:`~mc mv SOURCE` to the :mc-cmd:`~mc mv TARGET` bucket. -.. mc-cmd:: --storage-class, sc +.. mc-cmd:: --storage-class :optional: Set the storage class for the new object(s) on the @@ -313,7 +313,7 @@ Move Bucket Between S3-Compatible Services Move File to S3-Compatible Host with Specific Storage Class ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Use :mc:`mc mv` with the :mc-cmd:`~mc mv storage-class` option to set +Use :mc:`mc mv` with the :mc-cmd:`~mc mv --storage-class` option to set the storage class on the destination S3-compatible host. .. code-block:: shell @@ -321,7 +321,7 @@ the storage class on the destination S3-compatible host. mc mv --storage-class CLASS FILEPATH ALIAS/PATH -- Replace :mc-cmd:`CLASS ` with the storage class to +- Replace :mc-cmd:`CLASS ` with the storage class to associate to the files. - Replace :mc-cmd:`FILEPATH ` with the full file path to the diff --git a/source/reference/minio-server/minio-server.rst b/source/reference/minio-server/minio-server.rst index d9f0374f..4b8c5184 100644 --- a/source/reference/minio-server/minio-server.rst +++ b/source/reference/minio-server/minio-server.rst @@ -222,6 +222,11 @@ Core Configuration Specifies the full path to the file the MinIO server process uses for loading environment variables. +.. envvar:: MINIO_ILM_EXPIRY_WORKERS + + Specifies the number of workers to make available to expire objects configured with ILM rules for expiration. + When not set, MinIO defaults to using up to half of the available processing cores available. + Root Credentials ~~~~~~~~~~~~~~~~