minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-07-30 07:03:26 +03:00
Code Activity

Clean up of batch framework and mc batch docs (#749)

Browse Source 
- Removes bad copy paste info about ilm from batch command docs.
- Clarifies that batch files are one task only.
- Fixes source type from `s3` to `minio`.
- Cleans up indentation errors in replicate yaml example.
- Replicate task is local to remote only.

Closes #748
This commit is contained in:
Daryl White
2023-03-06 16:12:44 -06:00
committed by GitHub
parent 732b6c3de7
commit f37aee8416
7 changed files with 88 additions and 138 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
source/administration/batch-framework.rst
View File

@ -20,8 +20,7 @@ Overview
The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a "batch file").
The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the :ref:`MinIO Client <minio-client>`.
A batch file can define any number of job tasks.
MinIO does not limit on the number of job tasks that you can define in a batch file.
A batch file defines one job task.
Once started, MinIO starts processing the job.
Time to completion depends on the resources available to the deployment.
@ -82,10 +81,10 @@ You can use MinIO's :ref:`Policy Based Access Control <minio-policy>` and the :r
Job Types
---------
``replicate``
~~~~~~~~~~~~~
Replicate
~~~~~~~~~
The ``replicate`` job type performs a single-run replication of objects from one MinIO deployment to another.
Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location.
The definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects.
For example, you can use a batch job to perform a one-time replication sync of objects from ``minio-alpha/invoices/`` to ``minio-baker/invoices``.
@ -106,57 +105,5 @@ Sample YAML Description File for a ``replicate`` Job Type
Create a basic ``replicate`` job definition file you can edit with :mc:`mc batch generate`.
.. code-block:: yaml
replicate:
apiVersion: v1
# source of the objects to be replicated
source:
type: TYPE # valid values are "s3"
bucket: BUCKET
prefix: PREFIX
# NOTE: if source is remote then target must be "local"
# endpoint: ENDPOINT
# credentials:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# target where the objects must be replicated
target:
type: TYPE # valid values are "s3"
bucket: BUCKET
prefix: PREFIX
# NOTE: if target is remote then source must be "local"
# endpoint: ENDPOINT
# credentials:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# optional flags based filtering criteria
# for all source objects
flags:
filter:
newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
createdAfter: "date" # match objects created after "date"
createdBefore: "date" # match objects created before "date"
## NOTE: tags are not supported when "source" is remote.
# tags:
# - key: "name"
# value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
## NOTE: metadata filter not supported when "source" is non MinIO.
# metadata:
# - key: "content-type"
# value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
notify:
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
retry:
attempts: 10 # number of retries for the job before giving up
delay: "500ms" # least amount of delay between each retry
.. literalinclude:: /includes/code/replicate.yaml
:language: yaml

49
source/includes/code/replicate.yaml Normal file
View File

@ -0,0 +1,49 @@
replicate:
apiVersion: v1
# source of the objects to be replicated
source:
type: TYPE # valid values are "s3"
bucket: BUCKET
prefix: PREFIX
# endpoint: ENDPOINT
# credentials:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# target where the objects must be replicated
target:
type: TYPE # valid values are "s3"
bucket: BUCKET
prefix: PREFIX
# endpoint: ENDPOINT
# credentials:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# optional flags based filtering criteria
# for all source objects
flags:
filter:
newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
createdAfter: "date" # match objects created after "date"
createdBefore: "date" # match objects created before "date"
# tags:
# - key: "name"
# value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
## NOTE: metadata filter not supported when "source" is non MinIO.
# metadata:
# - key: "content-type"
# value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
notify:
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
retry:
attempts: 10 # number of retries for the job before giving up
delay: "500ms" # least amount of delay between each retry

5
source/reference/minio-mc/mc-batch-describe.rst
View File

@ -106,3 +106,8 @@ S3 Compatibility
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-s3-compatibility
:end-before: end-minio-mc-s3-compatibility
Permissions
-----------
You must have the :policy-action:`admin:DescribeBatchJobs` permission to describe jobs on the deployment.

84
source/reference/minio-mc/mc-batch-generate.rst
View File

@ -24,9 +24,7 @@ The :mc:`mc batch generate` command creates a basic YAML-formatted template file
.. end-mc-batch-generate-desc
After MinIO creates the file, open it in your preferred text editor tool to further customize.
- You can add multiple job task definitions to a batch file.
- MinIO imposes no limit to the number of job task definitions per batch file.
You can define one job task definition per batch file.
See :ref:`job types <minio-batch-job-types>` for the supported jobs you can generate.
@ -62,15 +60,14 @@ Parameters
.. mc-cmd:: TARGET
:required:
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
The :ref:`alias <alias>` used to generate the YAML template file.
The specified ``alias`` does not restrict the deployment(s) where you can use the generated file.
For example:
.. code-block:: none
mc batch generate myminio/mydata replicate
You can use just an alias without specifying a bucket to perform the job at the deployment's root.
mc batch generate myminio replicate
.. mc-cmd:: JOBTYPE
:required:
@ -98,12 +95,9 @@ The following command generates a YAML blueprint for a replicate type batch job
.. code-block:: shell
:class: copyable
mc batch generate alias/<bucket> replicate
mc batch generate alias replicate > replicate.yaml
- Replace ``alias`` with the :mc:`alias <mc alias>` and (optionally) bucket of the MinIO deployment that is the source of the replication job.
- Replace ``<bucket>`` with the bucket on the alias where the batch should run.
The bucket can be left blank to perform the action on the alias deployment's root location.
- Replace ``alias`` with the :mc:`alias <mc alias>` to use to generate the yaml file.
- Replace ``replicate`` with the type of job to generate a yaml file for.
@ -134,7 +128,7 @@ MinIO may add more job types in the future.
``replicate``
~~~~~~~~~~~~~
Use the ``replicate`` job type to create a batch job that performs replicates objects between two MinIO deployments.
Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location.
The YAML **must** define the source and target deployments.
Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.
@ -148,7 +142,7 @@ For the **source deployment**
:width: 100%
* - ``type:``
- Must be ``s3``.
- Must be ``minio``.
* - ``bucket:``
- The bucket on the deployment.
@ -162,8 +156,7 @@ For the **source deployment**
- The prefix on the object(s) that should replicate.
* - ``endpoint:``
- | Location of the source deployment.
| If the location is not remote, use ``local``.
- | Location of the source deployment, must be ``local``.
* - ``credentials:``
- The ``accesskey:`` and ``secretKey:`` or the ``sessionToken:`` that grants access to the object(s).
@ -177,7 +170,7 @@ For the **target deployment**
:width: 100%
* - ``type:``
- Must be ``s3``.
- Must be ``minio``.
* - ``bucket:``
- The bucket on the deployment.
@ -250,58 +243,5 @@ For each retry, you can also define how long to wait between attempts.
Sample YAML
+++++++++++
.. code-block:: yaml
replicate:
apiVersion: v1
# source of the objects to be replicated
source:
type: TYPE # valid values are "s3"
bucket: BUCKET
prefix: PREFIX
# NOTE: if source is remote then target must be "local"
# endpoint: ENDPOINT
# credentials:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# target where the objects must be replicated
target:
type: TYPE # valid values are "s3"
bucket: BUCKET
prefix: PREFIX
# NOTE: if target is remote then source must be "local"
# endpoint: ENDPOINT
# credentials:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# optional flags based filtering criteria
# for all source objects
flags:
filter:
newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
createdAfter: "date" # match objects created after "date"
createdBefore: "date" # match objects created before "date"
## NOTE: tags are not supported when "source" is remote.
# tags:
# - key: "name"
# value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
## NOTE: metadata filter not supported when "source" is non MinIO.
# metadata:
# - key: "content-type"
# value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
notify:
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
retry:
attempts: 10 # number of retries for the job before giving up
delay: "500ms" # least amount of delay between each retry
.. literalinclude:: /includes/code/replicate.yaml
:language: yaml

9
source/reference/minio-mc/mc-batch-list.rst
View File

@ -55,7 +55,7 @@ Parameters
.. mc-cmd:: TARGET
:required:
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
The :ref:`alias <alias>` of the deployment for which you want to list jobs in progress.
.. mc-cmd:: --type
:optional:
@ -75,7 +75,7 @@ Example
List all ``replicate`` type batch jobs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following command starts the batch of job(s) defined in the file ``replication.yaml`` on the deployment at :mc:`alias <mc alias>` ``myminio``:
The following command lists the ``replicate``` type job(s) on the deployment at :mc:`alias <mc alias>` ``myminio``:
.. code-block:: shell
:class: copyable
@ -101,3 +101,8 @@ S3 Compatibility
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-s3-compatibility
:end-before: end-minio-mc-s3-compatibility
Permissions
-----------
You must have the :policy-action:`admin:ListBatchJobs` permission to list jobs on the deployment.

12
source/reference/minio-mc/mc-batch-start.rst
View File

@ -64,15 +64,13 @@ Parameters
.. mc-cmd:: TARGET
:required:
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
The :ref:`alias <alias>` on which to start the batch job.
For example:
.. code-block:: none
mc batch start myminio/mydata replicate.yaml
You can use just an alias without specifying a bucket to perform the job at the deployment's root.
mc batch start myminio replicate.yaml
.. mc-cmd:: JOBFILE
:required:
@ -117,3 +115,9 @@ S3 Compatibility
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-s3-compatibility
:end-before: end-minio-mc-s3-compatibility
Permissions
-----------
You must have the :policy-action:`admin:StartBatchJob` permission on the deployment to start jobs.

2
source/reference/minio-mc/mc-batch-status.rst
View File

@ -55,7 +55,7 @@ Parameters
.. mc-cmd:: TARGET
:required:
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
The :ref:`alias <alias>` for which to display batch job statuses.
.. mc-cmd:: JOBID
:optional: