diff --git a/source/administration/batch-framework.rst b/source/administration/batch-framework.rst index baddb1a1..d2e0d30a 100644 --- a/source/administration/batch-framework.rst +++ b/source/administration/batch-framework.rst @@ -20,8 +20,7 @@ Overview The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a "batch file"). The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the :ref:`MinIO Client `. -A batch file can define any number of job tasks. -MinIO does not limit on the number of job tasks that you can define in a batch file. +A batch file defines one job task. Once started, MinIO starts processing the job. Time to completion depends on the resources available to the deployment. @@ -82,10 +81,10 @@ You can use MinIO's :ref:`Policy Based Access Control ` and the :r Job Types --------- -``replicate`` -~~~~~~~~~~~~~ +Replicate +~~~~~~~~~ -The ``replicate`` job type performs a single-run replication of objects from one MinIO deployment to another. +Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location. The definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects. For example, you can use a batch job to perform a one-time replication sync of objects from ``minio-alpha/invoices/`` to ``minio-baker/invoices``. @@ -106,57 +105,5 @@ Sample YAML Description File for a ``replicate`` Job Type Create a basic ``replicate`` job definition file you can edit with :mc:`mc batch generate`. -.. code-block:: yaml - - replicate: - apiVersion: v1 - # source of the objects to be replicated - source: - type: TYPE # valid values are "s3" - bucket: BUCKET - prefix: PREFIX - # NOTE: if source is remote then target must be "local" - # endpoint: ENDPOINT - # credentials: - # accessKey: ACCESS-KEY - # secretKey: SECRET-KEY - # sessionToken: SESSION-TOKEN # Available when rotating credentials are used - - # target where the objects must be replicated - target: - type: TYPE # valid values are "s3" - bucket: BUCKET - prefix: PREFIX - # NOTE: if target is remote then source must be "local" - # endpoint: ENDPOINT - # credentials: - # accessKey: ACCESS-KEY - # secretKey: SECRET-KEY - # sessionToken: SESSION-TOKEN # Available when rotating credentials are used - - # optional flags based filtering criteria - # for all source objects - flags: - filter: - newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s) - olderThan: "7d" # match objects older than this value (e.g. 7d10h31s) - createdAfter: "date" # match objects created after "date" - createdBefore: "date" # match objects created before "date" - - ## NOTE: tags are not supported when "source" is remote. - # tags: - # - key: "name" - # value: "pick*" # match objects with tag 'name', with all values starting with 'pick' - - ## NOTE: metadata filter not supported when "source" is non MinIO. - # metadata: - # - key: "content-type" - # value: "image/*" # match objects with 'content-type', with all values starting with 'image/' - - notify: - endpoint: "https://notify.endpoint" # notification endpoint to receive job status events - token: "Bearer xxxxx" # optional authentication token for the notification endpoint - - retry: - attempts: 10 # number of retries for the job before giving up - delay: "500ms" # least amount of delay between each retry \ No newline at end of file +.. literalinclude:: /includes/code/replicate.yaml + :language: yaml \ No newline at end of file diff --git a/source/includes/code/replicate.yaml b/source/includes/code/replicate.yaml new file mode 100644 index 00000000..4d80b9d2 --- /dev/null +++ b/source/includes/code/replicate.yaml @@ -0,0 +1,49 @@ +replicate: + apiVersion: v1 + # source of the objects to be replicated + source: + type: TYPE # valid values are "s3" + bucket: BUCKET + prefix: PREFIX + # endpoint: ENDPOINT + # credentials: + # accessKey: ACCESS-KEY + # secretKey: SECRET-KEY + # sessionToken: SESSION-TOKEN # Available when rotating credentials are used + + # target where the objects must be replicated + target: + type: TYPE # valid values are "s3" + bucket: BUCKET + prefix: PREFIX + # endpoint: ENDPOINT + # credentials: + # accessKey: ACCESS-KEY + # secretKey: SECRET-KEY + # sessionToken: SESSION-TOKEN # Available when rotating credentials are used + + # optional flags based filtering criteria + # for all source objects + flags: + filter: + newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s) + olderThan: "7d" # match objects older than this value (e.g. 7d10h31s) + createdAfter: "date" # match objects created after "date" + createdBefore: "date" # match objects created before "date" + + # tags: + # - key: "name" + # value: "pick*" # match objects with tag 'name', with all values starting with 'pick' + + ## NOTE: metadata filter not supported when "source" is non MinIO. + # metadata: + # - key: "content-type" + # value: "image/*" # match objects with 'content-type', with all values starting with 'image/' + + notify: + endpoint: "https://notify.endpoint" # notification endpoint to receive job status events + token: "Bearer xxxxx" # optional authentication token for the notification endpoint + + retry: + attempts: 10 # number of retries for the job before giving up + delay: "500ms" # least amount of delay between each retry diff --git a/source/reference/minio-mc/mc-batch-describe.rst b/source/reference/minio-mc/mc-batch-describe.rst index 6e2130c3..99b35011 100644 --- a/source/reference/minio-mc/mc-batch-describe.rst +++ b/source/reference/minio-mc/mc-batch-describe.rst @@ -106,3 +106,8 @@ S3 Compatibility .. include:: /includes/common-minio-mc.rst :start-after: start-minio-mc-s3-compatibility :end-before: end-minio-mc-s3-compatibility + +Permissions +----------- + +You must have the :policy-action:`admin:DescribeBatchJobs` permission to describe jobs on the deployment. diff --git a/source/reference/minio-mc/mc-batch-generate.rst b/source/reference/minio-mc/mc-batch-generate.rst index dfd1bfee..e66bb63c 100644 --- a/source/reference/minio-mc/mc-batch-generate.rst +++ b/source/reference/minio-mc/mc-batch-generate.rst @@ -24,9 +24,7 @@ The :mc:`mc batch generate` command creates a basic YAML-formatted template file .. end-mc-batch-generate-desc After MinIO creates the file, open it in your preferred text editor tool to further customize. - -- You can add multiple job task definitions to a batch file. -- MinIO imposes no limit to the number of job task definitions per batch file. +You can define one job task definition per batch file. See :ref:`job types ` for the supported jobs you can generate. @@ -62,15 +60,14 @@ Parameters .. mc-cmd:: TARGET :required: - The :ref:`alias ` and bucket on the MinIO deployment to which to add the object lifecycle management rule. + The :ref:`alias ` used to generate the YAML template file. + The specified ``alias`` does not restrict the deployment(s) where you can use the generated file. For example: .. code-block:: none - mc batch generate myminio/mydata replicate - - You can use just an alias without specifying a bucket to perform the job at the deployment's root. + mc batch generate myminio replicate .. mc-cmd:: JOBTYPE :required: @@ -98,12 +95,9 @@ The following command generates a YAML blueprint for a replicate type batch job .. code-block:: shell :class: copyable - mc batch generate alias/ replicate + mc batch generate alias replicate > replicate.yaml -- Replace ``alias`` with the :mc:`alias ` and (optionally) bucket of the MinIO deployment that is the source of the replication job. - -- Replace ```` with the bucket on the alias where the batch should run. - The bucket can be left blank to perform the action on the alias deployment's root location. +- Replace ``alias`` with the :mc:`alias ` to use to generate the yaml file. - Replace ``replicate`` with the type of job to generate a yaml file for. @@ -134,7 +128,7 @@ MinIO may add more job types in the future. ``replicate`` ~~~~~~~~~~~~~ -Use the ``replicate`` job type to create a batch job that performs replicates objects between two MinIO deployments. +Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location. The YAML **must** define the source and target deployments. Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job. @@ -148,7 +142,7 @@ For the **source deployment** :width: 100% * - ``type:`` - - Must be ``s3``. + - Must be ``minio``. * - ``bucket:`` - The bucket on the deployment. @@ -162,8 +156,7 @@ For the **source deployment** - The prefix on the object(s) that should replicate. * - ``endpoint:`` - - | Location of the source deployment. - | If the location is not remote, use ``local``. + - | Location of the source deployment, must be ``local``. * - ``credentials:`` - The ``accesskey:`` and ``secretKey:`` or the ``sessionToken:`` that grants access to the object(s). @@ -177,7 +170,7 @@ For the **target deployment** :width: 100% * - ``type:`` - - Must be ``s3``. + - Must be ``minio``. * - ``bucket:`` - The bucket on the deployment. @@ -250,58 +243,5 @@ For each retry, you can also define how long to wait between attempts. Sample YAML +++++++++++ -.. code-block:: yaml - - replicate: - apiVersion: v1 - # source of the objects to be replicated - source: - type: TYPE # valid values are "s3" - bucket: BUCKET - prefix: PREFIX - # NOTE: if source is remote then target must be "local" - # endpoint: ENDPOINT - # credentials: - # accessKey: ACCESS-KEY - # secretKey: SECRET-KEY - # sessionToken: SESSION-TOKEN # Available when rotating credentials are used - - # target where the objects must be replicated - target: - type: TYPE # valid values are "s3" - bucket: BUCKET - prefix: PREFIX - # NOTE: if target is remote then source must be "local" - # endpoint: ENDPOINT - # credentials: - # accessKey: ACCESS-KEY - # secretKey: SECRET-KEY - # sessionToken: SESSION-TOKEN # Available when rotating credentials are used - - # optional flags based filtering criteria - # for all source objects - flags: - filter: - newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s) - olderThan: "7d" # match objects older than this value (e.g. 7d10h31s) - createdAfter: "date" # match objects created after "date" - createdBefore: "date" # match objects created before "date" - - ## NOTE: tags are not supported when "source" is remote. - # tags: - # - key: "name" - # value: "pick*" # match objects with tag 'name', with all values starting with 'pick' - - ## NOTE: metadata filter not supported when "source" is non MinIO. - # metadata: - # - key: "content-type" - # value: "image/*" # match objects with 'content-type', with all values starting with 'image/' - - notify: - endpoint: "https://notify.endpoint" # notification endpoint to receive job status events - token: "Bearer xxxxx" # optional authentication token for the notification endpoint - - retry: - attempts: 10 # number of retries for the job before giving up - delay: "500ms" # least amount of delay between each retry - \ No newline at end of file +.. literalinclude:: /includes/code/replicate.yaml + :language: yaml diff --git a/source/reference/minio-mc/mc-batch-list.rst b/source/reference/minio-mc/mc-batch-list.rst index 4360428d..1137e3da 100644 --- a/source/reference/minio-mc/mc-batch-list.rst +++ b/source/reference/minio-mc/mc-batch-list.rst @@ -55,7 +55,7 @@ Parameters .. mc-cmd:: TARGET :required: - The :ref:`alias ` and bucket on the MinIO deployment to which to add the object lifecycle management rule. + The :ref:`alias ` of the deployment for which you want to list jobs in progress. .. mc-cmd:: --type :optional: @@ -75,7 +75,7 @@ Example List all ``replicate`` type batch jobs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The following command starts the batch of job(s) defined in the file ``replication.yaml`` on the deployment at :mc:`alias ` ``myminio``: +The following command lists the ``replicate``` type job(s) on the deployment at :mc:`alias ` ``myminio``: .. code-block:: shell :class: copyable @@ -101,3 +101,8 @@ S3 Compatibility .. include:: /includes/common-minio-mc.rst :start-after: start-minio-mc-s3-compatibility :end-before: end-minio-mc-s3-compatibility + +Permissions +----------- + +You must have the :policy-action:`admin:ListBatchJobs` permission to list jobs on the deployment. diff --git a/source/reference/minio-mc/mc-batch-start.rst b/source/reference/minio-mc/mc-batch-start.rst index 3640fadb..de87854b 100644 --- a/source/reference/minio-mc/mc-batch-start.rst +++ b/source/reference/minio-mc/mc-batch-start.rst @@ -64,15 +64,13 @@ Parameters .. mc-cmd:: TARGET :required: - The :ref:`alias ` and bucket on the MinIO deployment to which to add the object lifecycle management rule. + The :ref:`alias ` on which to start the batch job. For example: .. code-block:: none - mc batch start myminio/mydata replicate.yaml - - You can use just an alias without specifying a bucket to perform the job at the deployment's root. + mc batch start myminio replicate.yaml .. mc-cmd:: JOBFILE :required: @@ -117,3 +115,9 @@ S3 Compatibility .. include:: /includes/common-minio-mc.rst :start-after: start-minio-mc-s3-compatibility :end-before: end-minio-mc-s3-compatibility + + +Permissions +----------- + +You must have the :policy-action:`admin:StartBatchJob` permission on the deployment to start jobs. diff --git a/source/reference/minio-mc/mc-batch-status.rst b/source/reference/minio-mc/mc-batch-status.rst index 701259ee..8d92c982 100644 --- a/source/reference/minio-mc/mc-batch-status.rst +++ b/source/reference/minio-mc/mc-batch-status.rst @@ -55,7 +55,7 @@ Parameters .. mc-cmd:: TARGET :required: - The :ref:`alias ` and bucket on the MinIO deployment to which to add the object lifecycle management rule. + The :ref:`alias ` for which to display batch job statuses. .. mc-cmd:: JOBID :optional: