minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-07-30 07:03:26 +03:00
Code Activity

DOCS-1083: MinIO Server Release RELEASE.2023-12-02T10-51-33Z (#1097)

Browse Source 
# Summary

Closes #1083 

Also reorganizes the batch framework pages. Could use additional
refinement, but considering that out of scope for now.

---------

Co-authored-by: Daryl White <53910321+djwfyi@users.noreply.github.com>
This commit is contained in:
Ravind Kumar
2024-01-02 14:25:04 -05:00
committed by GitHub
parent 69315f3810
commit a065b7a29f
9 changed files with 743 additions and 304 deletions
Download Patch File Download Diff File Expand all files Collapse all files

186
source/administration/batch-framework-job-expire.rst Normal file
View File

@ -0,0 +1,186 @@
.. _minio-batch-framework-expire-job:
=================
Batch Expiration
=================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
.. versionadded:: MinIO RELEASE.2023-12-02T10-51-33Z
The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a "batch file").
The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the :ref:`MinIO Client <minio-client>`.
The ``expire`` batch job applies :ref:`minio-lifecycle-management-create-expiry-rule` behavior to a single bucket.
The job determines expiration eligibility based on the provided configuration, independent of any configured expiration rules.
Behavior
--------
Immediate Expiration of Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Batch expiration occurs immediately as part of the batch job, as compared to the :ref:`passive scanner-based application of expiration rules <minio-lifecycle-management-scanner>`.
Specifically, batch expiration does not yield to application I/O and may impact performance of regular read/write operations on the deployment.
Expiration Eligibility Determined at Batch-Run
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The batch expiration works per-bucket and runs once to completion.
The job determines expiration eligibility at the time the job runs, and does *not* rescan or recheck for new objects periodically.
To capture any new objects eligible for expiration, re-run the batch job.
Expiry Rules Check Latest Object Only
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The batch expiration job only checks the latest or "current" version of each object against each batch expiration rule.
.. _minio-batch-framework-expire-job-ref:
Expire Batch Job Reference
--------------------------
.. list-table::
:widths: 25 75
:width: 100%
* - Field
- Description
* - ``expire``
- *Required*
Top-level field for the expiration job type.
* - ``apiVersion``
- *Required*
Set to ``v1``.
* - ``bucket``
- *Required*
Specify the name of the bucket in which the job runs.
* - ``prefix``
- *Optional*
Specify the bucket prefix in which the job runs.
* - ``rules``
- *Required*
An array of one or more expiration rules to apply to objects in the specified ``bucket`` and ``prefix`` (if any).
* - ``rules.[n].type``
- *Required*
Supports one of the following two values:
- ``object`` - Applies only to objects which do **not** have a ``DeleteMarker`` as the current version.
- ``deleted`` - Applies only to objects which **do** Have a ``DeleteMarker`` as the current version.
See :ref:`minio-bucket-versioning-delete` for more complete documentation on ``DeleteMarker`` or delete operations in versioned buckets.
* - ``rules.[n].name``
- *Optional*
Specify a match string to use for filtering objects.
Supports glob-style wildcards (``*``, ``?``).
* - ``rules.[n].olderThan``
- *Optional*
Specify the age of objects for filtering objects.
The rule applies to only those objects older than the specified unit of time.
For example, ``72h`` or ``3d`` selects objects older than three days.
* - ``rules.[n].createdBefore``
- *Optional*
Specify an ISO-8601 timestamp for filtering objects.
The rule applies to only those objects created *before* the specified timestamp.
* - ``rules.[n].tags``
- *Optional*
Specify an array of key-value pairs describing object tags to use for filtering objects.
The ``value`` entry supports glob-style wildcards (``*``, ``?``).
For example, the following filters the rule to only objects with matching tags:
.. code-block:: yaml
tags:
- key: archive
value: True
This key is incompatible with ``rules.[n].type: deleted``.
* - ``rules.[n].metadata``
- *Optional*
Specify an array of key-value pairs describing object metadata to use for filtering objects.
The ``value`` key supports glob-style wildcards (``*``, ``?``).
For example, the following filters the rule to only objects with matching metadata:
.. code-block:: yaml
metadata:
- key: content-type
value: image/*
This key is incompatible with ``rules.[n].type: deleted``.
* - ``rules.[n].size``
- *Optional*
Specify the range of object sizes for filtering objects.
- ``lessThan`` - matches objects with size less than the specified amount (e.g. ``MiB``, ``GiB``).
- ``greaterThan`` - matches objects with size greater than the specified amount (e.g. ``MiB``, ``GiB``).
* - ``rules.[n].purge.retainVersions``
- *Optional*
Specify the number of object versions to retain when applying expiration.
Defaults to ``0`` for deleting all object versions (fastest).
* - ``notify.endpoint``
- *Optional*
The predefined endpoint to send events for notifications.
* - ``notify.token``
- *Optional*
An optional JSON Web Token (JWT) to access the ``notify.endpoint``.
* - ``retry.attempts``
- *Optional*
The number of tries to complete the batch job before giving up.
* - ``retry.delay``
- *Optional*
The amount of time to wait between each attempt (``ms``).
Sample YAML Description for an ``expire`` Job Type
--------------------------------------------------
Use :mc:`mc batch generate` to create a basic ``expire`` batch job for further customization.
.. literalinclude:: /includes/code/expire.yaml
:language: yaml

115
source/administration/batch-framework-job-keyrotate.rst Normal file
View File

@ -0,0 +1,115 @@
.. _minio-batch-framework-keyrotate-job:
==================
Batch Key Rotation
==================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z
The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a "batch file").
The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the :ref:`MinIO Client <minio-client>`.
The ``keyrotate`` batch job type cycles the :ref:`sse-s3 or sse-kms keys <minio-sse-data-encryption>` for encrypted objects on a MinIO deployment.
The YAML configuration supports filters to restrict key rotation to a specific set of objects by creation date, tags, metadata, or kms key.
You can also define retry attempts or set a notification endpoint and token.
.. _minio-batch-framework-keyrotate-job-ref:
Key Rotate Batch Job Reference
------------------------------
.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z
Use the ``keyrotate`` job type to create a batch job that cycles the :ref:`sse-s3 or sse-kms keys <minio-sse-data-encryption>` for encrypted objects.
Required Fields
~~~~~~~~~~~~~~~
.. list-table::
:widths: 25 75
:width: 100%
* - ``type:``
- Either ``sse-s3`` or ``sse-kms``.
* - ``key:``
- Only for use with the ``sse-kms`` type.
The key to use to unseal the key vault.
* - ``context:``
- Only for use with the ``sse-kms`` type.
The context within which to perform actions.
Optional Fields
~~~~~~~~~~~~~~~
For **flag based filters**
.. list-table::
:widths: 25 75
:width: 100%
* - ``newerThan:``
- A string representing a length of time in ``#d#h#s`` format.
Keys rotate only for objects newer than the specified length of time.
For example, ``7d``, ``24h``, ``5d12h30s`` are valid strings.
* - ``olderThan:``
- A string representing a length of time in ``#d#h#s`` format.
Keys rotate only for objects older than the specified length of time.
* - ``createdAfter:``
- A date in ``YYYY-MM-DD`` format.
Keys rotate only for objects created after the date.
* - ``createdBefore:``
- A date in ``YYYY-MM-DD`` format.
Keys rotate only for objects created prior to the date.
* - ``tags:``
- Rotate keys only for objects with tags that match the specified ``key:`` and ``value:``.
* - ``metadata:``
- Rotate keys only for objects with metadata that match the specified ``key:`` and ``value:``.
* - ``kmskey:``
- Rotate keys only for objects with a KMS key-id that match the specified value.
This is only applicable for the ``sse-kms`` type.
For **notifications**
.. list-table::
:widths: 25 75
:width: 100%
* - ``endpoint:``
- The predefined endpoint to send events for notifications.
* - ``token:``
- An optional JSON Web Token (JWT) to access the ``endpoint``.
For **retry attempts**
If something interrupts the job, you can define a maximum number of retry attempts.
For each retry, you can also define how long to wait between attempts.
.. list-table::
:widths: 25 75
:width: 100%
* - ``attempts:``
- Number of tries to complete the batch job before giving up.
* - ``delay:``
- The amount of time to wait between each attempt.
Sample YAML Description File for a ``keyrotate`` Job Type
---------------------------------------------------------
Use :mc:`mc batch generate` to create a basic ``keyrotate`` batch job for further customization:
.. literalinclude:: /includes/code/keyrotate.yaml
:language: yaml

262
source/administration/batch-framework-job-replicate.rst Normal file
View File

@ -0,0 +1,262 @@
.. _minio-batch-framework-replicate-job:
=================
Batch Replication
=================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
.. versionadded:: MinIO RELEASE.2022-10-08T20-11-00Z
The Batch Framework was introduced with the ``replicate`` job type in the :mc:`mc` :mc-release:`RELEASE.2022-10-08T20-11-00Z`.
The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a "batch file").
The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the :ref:`MinIO Client <minio-client>`.
The ``replicate`` batch job replicates objects from one MinIO deployment (the ``source`` deployment) to another MinIO deployment (the ``target`` deployment).
Either the ``source`` or the ``target`` **must** be the :ref:`local <minio-batch-local>` deployment.
Batch Replication between MinIO deployments have the following advantages over using :mc:`mc mirror`:
- Removes the client to cluster network as a potential bottleneck
- A user only needs access to starting a batch job with no other permissions, as the job runs entirely server side on the cluster
- The job provides for retry attempts in event that objects do not replicate
- Batch jobs are one-time, curated processes allowing for fine control replication
- (MinIO to MinIO only) The replication process copies object versions from source to target
.. versionchanged:: MinIO Server RELEASE.2023-02-17T17-52-43Z
Run batch replication with multiple workers in parallel by specifying the :envvar:`MINIO_BATCH_REPLICATION_WORKERS` environment variable.
Starting with the MinIO Server ``RELEASE.2023-05-04T21-44-30Z``, the other deployment can be either another MinIO deployment or any S3-compatible location using a realtime storage class.
Use filtering options in the replication ``YAML`` file to exclude objects stored in locations that require rehydration or other restoration methods before serving the requested object.
Batch replication to these types of remotes uses ``mc mirror`` behavior.
Behavior
--------
Access Control and Requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Batch replication shares similar access and permission requirements as :ref:`bucket replication <minio-bucket-replication-requirements>`.
The credentials for the "source" deployment must have a policy similar to the following:
.. literalinclude:: /extra/examples/ReplicationAdminPolicy.json
:class: copyable
:language: json
The credentials for the "remote" deployment must have a policy similar to the following:
.. literalinclude:: /extra/examples/ReplicationRemoteUserPolicy.json
:class: copyable
:language: json
See :mc:`mc admin user`, :mc:`mc admin user svcacct`, and :mc:`mc admin policy` for more complete documentation on adding users, access keys, and policies to a MinIO deployment.
MinIO deployments configured for :ref:`Active Directory/LDAP <minio-external-identity-management-ad-ldap>` or :ref:`OpenID Connect <minio-external-identity-management-openid>` user management can instead create dedicated :ref:`access keys <minio-idp-service-account>` for supporting batch replication.
Filter Replication Targets
~~~~~~~~~~~~~~~~~~~~~~~~~~
The batch job definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects.
The access to objects and buckets for the replication process may be restricted by the credentials you provide in the YAML for either the source or target destinations.
.. versionchanged:: MinIO Server RELEASE.2023-04-07T05-28-58Z
You can replicate from a remote MinIO deployment to the local deployment that runs the batch job.
For example, you can use a batch job to perform a one-time replication sync to push objects from a bucket on a local deployment at ``minio-local/invoices/`` to a bucket on a remote deployment at ``minio-remote/invoices``.
You can also pull objects from the remote deployment at ``minio-remote/invoices`` to the local deployment at ``minio-local/invoices``.
Small File Optimization
~~~~~~~~~~~~~~~~~~~~~~~
Starting with :minio-release:`RELEASE.2023-12-09T18-17-51Z`, batch replication by default automatically batches and compresses objects smaller than 5MiB to efficiently transfer data between the source and remote.
The remote MinIO deployment can check and immediately apply lifecycle management tiering rules to batched objects.
The functionality resembles that offered by S3 Snowball Edge small file batching.
You can modify the compression settings in the :ref:`replicate <minio-batch-job-types>` job configuration.
.. _minio-batch-framework-replicate-job-ref:
Replicate Batch Job Reference
-----------------------------
The YAML **must** define the source and target deployments.
If the *source* deployment is remote, then the *target* deployment **must** be ``local``.
Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.
.. versionchanged:: MinIO RELEASE.2023-04-07T05-28-58Z
You can replicate from a remote MinIO deployment to the local deployment that runs the batch job.
For the **source deployment**
- Required information
.. list-table::
:widths: 25 75
:width: 100%
* - ``type:``
- Must be ``minio``.
* - ``bucket:``
- The bucket on the deployment.
- Optional information
.. list-table::
:widths: 25 75
:width: 100%
* - ``prefix:``
- The prefix on the object(s) that should replicate.
* - ``endpoint:``
- | Location of the deployment to use for either the source or the target of a replication batch job.
| For example, ``https://minio.example.net``.
|
| If the deployment is the :ref:`alias` specified to the command, omit this field to direct MinIO to use that alias for the endpoint and credentials values.
| Either the source deployment *or* the remote deployment *must* be the :ref:`"local" <minio-batch-local>` alias.
| The non-"local" deployment must specify the ``endpoint`` and ``credentials``.
* - ``path:``
- | Directs MinIO to use Path or Virtual Style (DNS) lookup of the bucket.
|
| - Specify ``on`` for Path style
| - Specify ``off`` for Virtual style
| - Specify ``auto`` to let MinIO determine the correct lookup style.
|
| Defaults to ``auto``.
* - ``credentials:``
- | The ``accesskey:`` and ``secretKey:`` or the ``sessionToken:`` that grants access to the object(s).
| Only specify for the deployment that is not the :ref:`local <minio-batch-local>` deployment.
* - ``snowball``
- | *version added*: RELEASE.2023-12-09T18-17-51Z
|
| Configuration options for controlling the batch-and-compress functionality.
* - ``snowball.disable``
- | Specify ``true`` to disable the batch-and-compress functionality during replication.
| Defaults to ``false``.
* - ``snowball.batch``
- | Specify the maximum integer number of objects to batch for compression.
| Defaults to ``100``.
* - ``snowball.inmemory``
- | Specify ``false`` to stage archives using local storage or ``true`` to stage to memory (RAM).
| Defaults to ``true``.
* - ``snowball.compress``
- | Specify ``true`` to generate compress batched objects over the wire using the `S2/Snappy compression algorithm <https://en.wikipedia.org/wiki/Snappy_(compression)>`__.
| Defaults to ``false`` or no compression.
* - ``snowball.smallerThan``
- | Specify the size of object in Megabits (MiB) under which MinIO should batch objects.
| Defaults to ``5MiB``.
* - ``snowball.skipErrs``
- | Specify ``false`` to direct MinIO to halt on any object which produces errors on read.
| Defaults to ``true``.
For the **target deployment**
- Required information
.. list-table::
:widths: 25 75
:width: 100%
* - ``type:``
- Must be ``minio``.
* - ``bucket:``
- The bucket on the deployment.
- Optional information
.. list-table::
:widths: 25 75
:width: 100%
* - ``prefix:``
- The prefix on the object(s) to replicate.
* - ``endpoint:``
- | The location of the target deployment.
|
| If the target is the :ref:`alias <alias>` specified to the command, you can omit this and the ``credentials`` fields.
| If the target is "local", the source *must* specify the remote deployment with ``endpoint`` and ``credentials``.
* - ``credentials:``
- The ``accesskey`` and ``secretKey`` or the ``sessionToken`` that grants access to the object(s).
For **filters**
.. list-table::
:widths: 25 75
:width: 100%
* - ``newerThan:``
- A string representing a length of time in ``#d#h#s`` format.
Only objects newer than the specified length of time replicate.
For example, ``7d``, ``24h``, ``5d12h30s`` are valid strings.
* - ``olderThan:``
- A string representing a length of time in ``#d#h#s`` format.
Only objects older than the specified length of time replicate.
* - ``createdAfter:``
- A date in ``YYYY-MM-DD`` format.
Only objects created after the date replicate.
* - ``createdBefore:``
- A date in ``YYYY-MM-DD`` format.
Only objects created prior to the date replicate.
For **notifications**
.. list-table::
:widths: 25 75
:width: 100%
* - ``endpoint:``
- The predefined endpoint to send events for notifications.
* - ``token:``
- An optional :abbr:`JWT <JSON Web Token>` to access the ``endpoint``.
For **retry attempts**
If something interrupts the job, you can define how many attempts to retry the job batch.
For each retry, you can also define how long to wait between attempts.
.. list-table::
:widths: 25 75
:width: 100%
* - ``attempts:``
- Number of tries to complete the batch job before giving up.
* - ``delay:``
- The least amount of time to wait between each attempt.
Sample YAML Description File for a ``replicate`` Job Type
---------------------------------------------------------
Use :mc:`mc batch generate` to create a basic ``replicate`` batch job for further customization.
For the :ref:`local <minio-batch-local>` deployment, do not specify the endpoint or credentials.
Either delete or comment out those lines for the source or the target section, depending on which is the ``local``.
.. literalinclude:: /includes/code/replicate.yaml
:language: yaml

96
source/administration/batch-framework.rst
View File

@ -10,9 +10,7 @@ Batch Framework
:local:
:depth: 2
.. versionadded:: MinIO RELEASE.2022-10-08T20-11-00Z
The Batch Framework was introduced with the ``replicate`` job type in the :mc:`mc` RELEASES.2022-10-08T20-11-00Z.
Overview
--------
@ -37,12 +35,15 @@ The MinIO Batch Framework supports the following job types:
* - Job Type
- Description
* - ``replicate``
* - :ref:`replicate <minio-batch-framework-replicate-job>`
- Perform a one-time replication procedure from one MinIO location to another MinIO location.
* - ``keyrotate``
* - :ref:`keyrotate <minio-batch-framework-keyrotate-job>`
- Perform a one-time process to cycle the :ref:`sse-s3 or sse-kms <minio-sse-data-encryption>` cryptographic keys on objects.
* - :ref:`expire <minio-batch-framework-expire-job>`
- Perform a one-time immediate expiration of objects in a bucket.
MinIO Batch CLI
---------------
@ -76,16 +77,16 @@ The :mc:`mc batch` commands include
:start-after: start-mc-batch-describe-desc
:end-before: end-mc-batch-describe-desc
.. _minio-batch-framework-access:
Access to ``mc batch``
----------------------
A user's access keys and policies do not restrict the the buckets, prefixes, or objects the batch function can access or the types of actions the process can perform on any objects.
Each batch job executes using the credentials specified in the batch definition.
The success of a given batch job depends on those credentials having the appropriate :ref:`permissions <minio-policy>` to perform all requested actions.
For some job types, the credentials passed to the batch job through the YAML file do restrict the objects that the job can access.
However, any restrictions to the job are from the credentials in the YAML, not policies attached to the user who starts the job.
Use MinIO's :ref:`Policy Based Access Control <minio-policy>` and the :ref:`administrative policy actions <minio-policy-mc-admin-actions>` to restrict who can perform various batch job functions.
MinIO provides the following admin policy actions for Batch Jobs:
The user executing the batch job must have the following permissions.
You can alternatively restrict users from accessing these functions by blocking or limiting access to these actions:
``admin:ListBatchJobs``
Grants the user the ability to see batch jobs currently in process.
@ -104,77 +105,18 @@ You can assign any of these actions to users independently or in any combination
The built-in ``ConsoleAdmin`` policy includes sufficient access to perform all of these types of batch job actions.
Job Types
---------
.. note::
Depending on the job type, the success or failure of any batch job may be impacted by the credentials given in the batch job's YAML for the source or target deployments.
.. _minio-batch-local:
``Local`` Deployment
~~~~~~~~~~~~~~~~~~~~
--------------------
You run a batch job against a particular deployment by passing an ``alias`` to the :mc:`mc batch` command.
The deployment you specify in the command becomes the ``local`` deployment within the context of that batch job.
Replicate
~~~~~~~~~
Use the ``replicate`` job type to create a batch job that replicates objects from one MinIO deployment (the ``source`` deployment) to another MinIO deployment (the ``target`` deployment).
Either the ``source`` or the ``target`` **must** be the :ref:`local <minio-batch-local>` deployment.
Starting with the MinIO Server ``RELEASE.2023-05-04T21-44-30Z``, the other deployment can be either another MinIO deployment or any S3-compatible location using a realtime storage class.
Use filtering options in the replication ``YAML`` file to exclude objects stored in locations that require rehydration or other restoration methods before serving the requested object.
The batch job definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects.
The access to objects and buckets for the replication process may be restricted by the credentials you provide in the YAML for either the source or target destinations.
.. versionchanged:: MinIO Server RELEASE.2023-04-07T05-28-58Z
You can replicate from a remote MinIO deployment to the local deployment that runs the batch job.
For example, you can use a batch job to perform a one-time replication sync to push objects from a bucket on a local deployment at ``minio-local/invoices/`` to a bucket on a remote deployment at ``minio-remote/invoices``.
You can also pull objects from the remote deployment at ``minio-remote/invoices`` to the local deployment at ``minio-local/invoices``.
The advantages of Batch Replication over :mc:`mc mirror` include:
- Removes the client to cluster network as a potential bottleneck
- A user only needs access to starting a batch job with no other permissions, as the job runs entirely server side on the cluster
- The job provides for retry attempts in event that objects do not replicate
- Batch jobs are one-time, curated processes allowing for fine control replication
- (MinIO to MinIO only) The replication process copies object versions from source to target
.. versionchanged:: MinIO Server RELEASE.2023-02-17T17-52-43Z
Run batch replication with multiple workers in parallel by specifying the :envvar:`MINIO_BATCH_REPLICATION_WORKERS` environment variable.
Sample YAML Description File for a ``replicate`` Job Type
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Create a basic ``replicate`` job definition file you can edit with :mc:`mc batch generate`.
For the :ref:`local <minio-batch-local>` deployment, do not specify the endpoint or credentials.
Either delete or comment out those lines for the source or the target section, depending on which is the ``local``.
.. literalinclude:: /includes/code/replicate.yaml
:language: yaml
Key Rotate
~~~~~~~~~~
.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z
Use the ``keyrotate`` job type to create a batch job that cycles the :ref:`sse-s3 or sse-kms keys <minio-sse-data-encryption>` for encrypted objects.
The YAML configuration supports filters to restrict key rotation to a specific set of objects by creation date, tags, metadata, or kms key.
You can also define retry attempts or set a notification endpoint and token.
Sample YAML Description File for a ``keyrotate`` Job Type
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Create a basic ``keyrotate`` job definition file you can edit with :mc:`mc batch generate`.
.. literalinclude:: /includes/code/keyrotate.yaml
:language: yaml
.. toctree::
:titlesonly:
:hidden:
/administration/batch-framework-job-replicate
/administration/batch-framework-job-keyrotate
/administration/batch-framework-job-expire

1
source/administration/identity-access-management/policy-based-access-control.rst
View File

@ -664,6 +664,7 @@ MinIO supports the following condition keys for all supported
- ``aws:userid``
- ``aws:username``
- ``x-amz-content-sha256``
- ``s3:signatureAge``
.. warning::

37
source/includes/code/expire.yaml Normal file
View File

@ -0,0 +1,37 @@
expire:
apiVersion: v1
bucket: mybucket # Bucket where this job will expire matching objects from
prefix: myprefix # (Optional) Prefix under which this job will expire objects matching the rules below.
rules:
- type: object # objects with zero ore more older versions
name: NAME # match object names that satisfy the wildcard expression.
olderThan: 70h # match objects older than this value
createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
tags:
- key: name
value: pick* # match objects with tag 'name', all values starting with 'pick'
metadata:
- key: content-type
value: image/* # match objects with 'content-type', all values starting with 'image/'
size:
lessThan: 10MiB # match objects with size less than this value (e.g. 10MiB)
greaterThan: 1MiB # match objects with size greater than this value (e.g. 1MiB)
purge:
# retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
# retainVersions: 5 # keep the latest 5 versions of the object.
- type: deleted # objects with delete marker as their latest version
name: NAME # match object names that satisfy the wildcard expression.
olderThan: 10h # match objects older than this value (e.g. 7d10h31s)
createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
purge:
# retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
# retainVersions: 5 # keep the latest 5 versions of the object including delete markers.
notify:
endpoint: https://notify.endpoint # notification endpoint to receive job completion status
token: Bearer xxxxx # optional authentication token for the notification endpoint
retry:
attempts: 10 # number of retries for the job before giving up
delay: 500ms # least amount of delay between each retry

8
source/includes/code/replicate.yaml
View File

@ -12,7 +12,13 @@ replicate:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# snowball:
# disable: true|false
# batch: 100
# inmemory: true|false
# compress: true|false
# smallerThan: 5MiB
# skipErrs: true|false
# target where the objects must be replicated
# if target is not the local deployment for the command, provide the endpoint and credentials
target:

261
source/reference/minio-mc/mc-batch-generate.rst
View File

@ -74,8 +74,11 @@ Parameters
The type of job to generate a YAML document for.
Currently, :mc:`mc batch` supports the ``replicate`` and ``keyrotate`` job types.
Supports the following values:
- :ref:`minio-mc-batch-generate-replicate-job`
- :ref:`minio-mc-batch-generate-keyrotate-job`
- :ref:`minio-mc-batch-generate-expire-job` (Added ``mc.RELEASE.2023-12-02T11-24-10Z``)
Global Flags
~~~~~~~~~~~~
@ -118,247 +121,55 @@ Job Types
:mc:`mc batch` currently supports the following job task types:
- ``replicate``
- :ref:`minio-mc-batch-generate-replicate-job`
Replicate objects between two MinIO deployments.
Provides similar functionality to :ref:`bucket replication <minio-bucket-replication>` as a batch job rather than continual scanning function.
- ``keyrotate``
- :ref:`minio-mc-batch-generate-keyrotate-job`
.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z
Rotate the sse-s3 or sse-kms keys for objects at rest on a MinIO deployment.
- :ref:`minio-mc-batch-generate-expire-job`
.. versionadded:: MinIO RELEASE.2023-12-02T10-51-33Z
Expire objects based using similar semantics as :ref:`minio-lifecycle-management-create-expiry-rule`.
.. _minio-mc-batch-generate-replicate-job:
``replicate``
~~~~~~~~~~~~~
Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location.
The YAML **must** define the source and target deployments.
If the _source_ deployment is remote, then the _target_ deployment **must** be ``local``.
Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.
.. versionchanged:: MinIO RELEASE.2023-04-07T05-28-58Z
You can replicate from a remote MinIO deployment to the local deployment that runs the batch job.
For the **source deployment**
- Required information
.. list-table::
:widths: 25 75
:width: 100%
* - ``type:``
- Must be ``minio``.
* - ``bucket:``
- The bucket on the deployment.
- Optional information
.. list-table::
:widths: 25 75
:width: 100%
* - ``prefix:``
- The prefix on the object(s) that should replicate.
* - ``endpoint:``
- | Location of the deployment to use for either the source or the target of a replication batch job.
| For example, ``https://minio.example.net``.
|
| If the deployment is the :ref:`alias` specified to the command, omit this field to direct MinIO to use that alias for the endpoint and credentials values.
| Either the source deployment *or* the remote deployment *must* be the :ref:`"local" <minio-batch-local>` alias.
| The non-"local" deployment must specify the ``endpoint`` and ``credentials``.
* - ``path:``
- | Directs MinIO to use Path or Virtual Style (DNS) lookup of the bucket.
|
| - Specify ``on`` for Path style
| - Specify ``off`` for Virtual style
| - Specify ``auto`` to let MinIO determine the correct lookup style.
|
| Defaults to ``auto``.
* - ``credentials:``
- | The ``accesskey:`` and ``secretKey:`` or the ``sessionToken:`` that grants access to the object(s).
| Only specify for the deployment that is not the :ref:`local <minio-batch-local>` deployment.
For the **target deployment**
- Required information
.. list-table::
:widths: 25 75
:width: 100%
* - ``type:``
- Must be ``minio``.
* - ``bucket:``
- The bucket on the deployment.
- Optional information
.. list-table::
:widths: 25 75
:width: 100%
* - ``prefix:``
- The prefix on the object(s) to replicate.
* - ``endpoint:``
- | The location of the target deployment.
|
| If the target is the :ref:`alias <alias>` specified to the command, you can omit this and the ``credentials`` fields.
| If the target is "local", the source *must* specify the remote deployment with ``endpoint`` and ``credentials``.
* - ``credentials:``
- The ``accesskey`` and ``secretKey`` or the ``sessionToken`` that grants access to the object(s).
For **filters**
.. list-table::
:widths: 25 75
:width: 100%
* - ``newerThan:``
- A string representing a length of time in ``#d#h#s`` format.
Only objects newer than the specified length of time replicate.
For example, ``7d``, ``24h``, ``5d12h30s`` are valid strings.
* - ``olderThan:``
- A string representing a length of time in ``#d#h#s`` format.
Only objects older than the specified length of time replicate.
* - ``createdAfter:``
- A date in ``YYYY-MM-DD`` format.
Only objects created after the date replicate.
* - ``createdBefore:``
- A date in ``YYYY-MM-DD`` format.
Only objects created prior to the date replicate.
For **notifications**
.. list-table::
:widths: 25 75
:width: 100%
* - ``endpoint:``
- The predefined endpoint to send events for notifications.
* - ``token:``
- An optional :abbr:`JWT <JSON Web Token>` to access the ``endpoint``.
For **retry attempts**
If something interrupts the job, you can define how many attempts to retry the job batch.
For each retry, you can also define how long to wait between attempts.
.. list-table::
:widths: 25 75
:width: 100%
* - ``attempts:``
- Number of tries to complete the batch job before giving up.
* - ``delay:``
- The least amount of time to wait between each attempt.
Sample YAML
+++++++++++
You can use the following example configuration as the starting point for building your own custom replication batch job:
.. literalinclude:: /includes/code/replicate.yaml
:language: yaml
See :ref:`minio-batch-framework-replicate-job-ref` for more complete documentation on each key.
.. _minio-mc-batch-generate-keyrotate-job:
``keyrotate``
~~~~~~~~~~~~~
.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z
Use the ``keyrotate`` job type to create a batch job that cycles the :ref:`sse-s3 or sse-kms keys <minio-sse-data-encryption>` for encrypted objects.
Required information
++++++++++++++++++++
.. list-table::
:widths: 25 75
:width: 100%
* - ``type:``
- Either ``sse-s3`` or ``sse-kms``.
* - ``key:``
- Only for use with the ``sse-kms`` type.
The key to use to unseal the key vault.
* - ``context:``
- Only for use with the ``sse-kms`` type.
The context within which to perform actions.
Optional information
++++++++++++++++++++
For **flag based filters**
.. list-table::
:widths: 25 75
:width: 100%
* - ``newerThan:``
- A string representing a length of time in ``#d#h#s`` format.
Keys rotate only for objects newer than the specified length of time.
For example, ``7d``, ``24h``, ``5d12h30s`` are valid strings.
* - ``olderThan:``
- A string representing a length of time in ``#d#h#s`` format.
Keys rotate only for objects older than the specified length of time.
* - ``createdAfter:``
- A date in ``YYYY-MM-DD`` format.
Keys rotate only for objects created after the date.
* - ``createdBefore:``
- A date in ``YYYY-MM-DD`` format.
Keys rotate only for objects created prior to the date.
* - ``tags:``
- Rotate keys only for objects with tags that match the specified ``key:`` and ``value:``.
* - ``metadata:``
- Rotate keys only for objects with metadata that match the specified ``key:`` and ``value:``.
* - ``kmskey:``
- Rotate keys only for objects with a KMS key-id that match the specified value.
This is only applicable for the ``sse-kms`` type.
For **notifications**
.. list-table::
:widths: 25 75
:width: 100%
* - ``endpoint:``
- The predefined endpoint to send events for notifications.
* - ``token:``
- An optional :abbr:`JWT <JSON Web Token>` to access the ``endpoint``.
For **retry attempts**
If something interrupts the job, you can define a maximum number of retry attempts.
For each retry, you can also define how long to wait between attempts.
.. list-table::
:widths: 25 75
:width: 100%
* - ``attempts:``
- Number of tries to complete the batch job before giving up.
* - ``delay:``
- The amount of time to wait between each attempt.
Sample YAML
+++++++++++
You can use the following example configuration as the starting point for building your own custom key rotation batch job:
.. literalinclude:: /includes/code/keyrotate.yaml
:language: yaml
:language: yaml
See :ref:`minio-batch-framework-keyrotate-job-ref` for more complete documentation on each key.
.. _minio-mc-batch-generate-expire-job:
``expire``
~~~~~~~~~~
You can use the following example configuration as a starting point for building your own custom expiration batch job:
.. literalinclude:: /includes/code/keyrotate.yaml
:language: yaml
See :ref:`minio-batch-framework-expire-job-ref` for more complete documentation on each key.

81
source/reference/minio-server/settings/notifications/kafka.rst
View File

@ -380,6 +380,30 @@ Version
Specify the version of the Kafka cluster to assume when performing operations against that cluster.
See the `sarama reference documentation <https://github.com/shopify/sarama/blob/v1.20.1/config.go#L327>`__ for more information on this field's behavior.
Batch Size
~~~~~~~~~~
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_BATCH_SIZE
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka batch_size
:delimiter: " "
Specify the integer value to use as the `batch size <https://kafka.apache.org/documentation/#producerconfigs_batch.size>`__ for sending records to Kafka.
.. versionchanged:: RELEASE.2023-12-02T10-51-33Z
MinIO previously limited this value to ``100``.
Queue Directory
~~~~~~~~~~~~~~~
@ -441,4 +465,59 @@ Comment
.. mc-conf:: notify_kafka comment
:delimiter: " "
Specify a comment to associate with the Kafka configuration.
Specify a comment to associate with the Kafka configuration.
Compression Codec
~~~~~~~~~~~~~~~~~
.. versionadded:: MinIO Server RELEASE.2023-12-09T18-17-51Z
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_PRODUCER_COMPRESSION_CODEC
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka compression_codec
:delimiter: " "
Specify the compression codec to use when sending records to Kafka.
Supports the following values:
- ``none``
- ``snappy``
- ``gzip``
- ``lz4``
- ``zstd``
Compression Level
~~~~~~~~~~~~~~~~~
.. versionadded:: MinIO Server RELEASE.2023-12-09T18-17-51Z
*Optional*
.. tab-set::
.. tab-item:: Environment Variable
:sync: envvar
.. envvar:: MINIO_NOTIFY_KAFKA_PRODUCER_COMPRESSION_LEVEL
.. tab-item:: Configuration Setting
:sync: config
.. mc-conf:: notify_kafka compression_level
:delimiter: " "
Controls the level of compression applied based on the configured compression codec.
Specify an integer value greater than or equal to ``0``.
The effect of the value depends on the selected codec.