mirror of
https://github.com/minio/docs.git
synced 2025-07-28 19:42:10 +03:00
Adds mc batch commands to the docs (#620)
Closes #605 - Adds the five `mc batch` subcommands - Adds a batch framework page to the Administration section of docs
This commit is contained in:
Daryl White
GitHub
158
source/administration/batch-framework.rst
Normal file
158
source/administration/batch-framework.rst
Normal file
@ -0,0 +1,158 @@
|
||||
.. _minio-batch-framework:
|
||||
|
||||
===============
|
||||
Batch Framework
|
||||
===============
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
.. versionadded:: MinIO RELEASE.2022-10-08T20-11-00Z
|
||||
|
||||
The Batch Framework was introduced with the ``replicate`` job type in the :mc:`mc` RELEASES.2022-10-08T20-11-00Z.
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a "batch file").
|
||||
The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the :ref:`MinIO Client <minio-client>`.
|
||||
|
||||
A batch file can define any number of job tasks.
|
||||
MinIO does not limit on the number of job tasks that you can define in a batch file.
|
||||
|
||||
Once started, MinIO starts processing the job.
|
||||
Time to completion depends on the resources available to the deployment.
|
||||
|
||||
If any portion of the job fails, MinIO retries the job up to the number of times defined in the job definition.
|
||||
|
||||
The MinIO Batch Framework supports the following job types:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
:width: 100%
|
||||
|
||||
* - Job Type
|
||||
- Description
|
||||
|
||||
* - ``replicate``
|
||||
- Perform a one-time replication procedure from one MinIO location to another MinIO location.
|
||||
|
||||
MinIO Batch CLI
|
||||
---------------
|
||||
|
||||
- Install the :ref:`MinIO Client <minio-client>`
|
||||
- Define an :mc-cmd:`alias <mc alias set>` for the MinIO deployment
|
||||
|
||||
The :mc:`mc batch` commands include
|
||||
|
||||
.. list-table::
|
||||
:widths: 30 70
|
||||
:width: 90%
|
||||
|
||||
* - :mc:`mc batch generate`
|
||||
- .. include:: /reference/minio-mc/mc-batch-generate.rst
|
||||
:start-after: start-mc-batch-generate-desc
|
||||
:end-before: end-mc-batch-generate-desc
|
||||
* - :mc:`mc batch start`
|
||||
- .. include:: /reference/minio-mc/mc-batch-start.rst
|
||||
:start-after: start-mc-batch-start-desc
|
||||
:end-before: end-mc-batch-start-desc
|
||||
* - :mc:`mc batch list`
|
||||
- .. include:: /reference/minio-mc/mc-batch-list.rst
|
||||
:start-after: start-mc-batch-list-desc
|
||||
:end-before: end-mc-batch-list-desc
|
||||
* - :mc:`mc batch status`
|
||||
- .. include:: /reference/minio-mc/mc-batch-status.rst
|
||||
:start-after: start-mc-batch-status-desc
|
||||
:end-before: end-mc-batch-status-desc
|
||||
* - :mc:`mc batch describe`
|
||||
- .. include:: /reference/minio-mc/mc-batch-describe.rst
|
||||
:start-after: start-mc-batch-describe-desc
|
||||
:end-before: end-mc-batch-describe-desc
|
||||
|
||||
Access to ``mc batch``
|
||||
----------------------
|
||||
|
||||
You can use MinIO's :ref:`Policy Based Access Control <minio-policy>` and the :ref:`administrative policy actions <minio-policy-mc-admin-actions>` to restrict who can start a batch job, retrieve a list of running jobs, or describe a running job.
|
||||
|
||||
Job Types
|
||||
---------
|
||||
|
||||
``replicate``
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The ``replicate`` job type performs a single-run replication of objects from one MinIO deployment to another.
|
||||
The definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects.
|
||||
|
||||
For example, you can use a batch job to perform a one-time replication sync of objects from ``minio-alpha/invoices/`` to ``minio-baker/invoices``.
|
||||
|
||||
The advantages of Batch Replication over :mc:`mc mirror` include:
|
||||
|
||||
- Removes the client to cluster network as a potential bottleneck
|
||||
- A user only needs access to starting a batch job with no other permissions, as the job runs entirely server side on the cluster
|
||||
- The job provides for retry attempts in event that objects do not replicate
|
||||
- Batch jobs are one-time, curated processes allowing for fine control replication
|
||||
|
||||
Sample YAML Description File for a ``replicate`` Job Type
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
Create a basic ``replicate`` job definition file you can edit with :mc-cmd:`mc batch generate`.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
replicate:
|
||||
apiVersion: v1
|
||||
# source of the objects to be replicated
|
||||
source:
|
||||
type: TYPE # valid values are "s3"
|
||||
bucket: BUCKET
|
||||
prefix: PREFIX
|
||||
# NOTE: if source is remote then target must be "local"
|
||||
# endpoint: ENDPOINT
|
||||
# credentials:
|
||||
# accessKey: ACCESS-KEY
|
||||
# secretKey: SECRET-KEY
|
||||
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
|
||||
|
||||
# target where the objects must be replicated
|
||||
target:
|
||||
type: TYPE # valid values are "s3"
|
||||
bucket: BUCKET
|
||||
prefix: PREFIX
|
||||
# NOTE: if target is remote then source must be "local"
|
||||
# endpoint: ENDPOINT
|
||||
# credentials:
|
||||
# accessKey: ACCESS-KEY
|
||||
# secretKey: SECRET-KEY
|
||||
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
|
||||
|
||||
# optional flags based filtering criteria
|
||||
# for all source objects
|
||||
flags:
|
||||
filter:
|
||||
newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
|
||||
olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
|
||||
createdAfter: "date" # match objects created after "date"
|
||||
createdBefore: "date" # match objects created before "date"
|
||||
|
||||
## NOTE: tags are not supported when "source" is remote.
|
||||
# tags:
|
||||
# - key: "name"
|
||||
# value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
|
||||
|
||||
## NOTE: metadata filter not supported when "source" is non MinIO.
|
||||
# metadata:
|
||||
# - key: "content-type"
|
||||
# value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
|
||||
|
||||
notify:
|
||||
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
|
||||
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
|
||||
|
||||
retry:
|
||||
attempts: 10 # number of retries for the job before giving up
|
||||
delay: "500ms" # least amount of delay between each retry
|
||||
@ -761,6 +761,18 @@ services:
|
||||
Allows access to MinIO :ref:`metrics <minio-metrics-and-alerts-endpoints>`.
|
||||
Only required if MinIO requires authentication for scraping metrics.
|
||||
|
||||
.. policy-action:: admin:ListBatchJobs
|
||||
|
||||
Allows access to list the active batch jobs.
|
||||
|
||||
.. policy-action:: admin:DescribeBatchJobs
|
||||
|
||||
Allows access to the see the definition details of a running batch job.
|
||||
|
||||
.. policy-action:: admin:StartBatchJob
|
||||
|
||||
Allows user to begin a batch job run.
|
||||
|
||||
``mc admin`` Policy Condition Keys
|
||||
----------------------------------
|
||||
|
||||
|
||||
@ -94,6 +94,7 @@ and non-protected.
|
||||
/administration/identity-access-management
|
||||
/administration/server-side-encryption
|
||||
/administration/bucket-replication
|
||||
/administration/batch-framework
|
||||
/administration/concepts
|
||||
|
||||
|
||||
@ -119,6 +120,7 @@ and non-protected.
|
||||
/administration/identity-access-management
|
||||
/administration/server-side-encryption
|
||||
/administration/bucket-replication
|
||||
/administration/batch-framework
|
||||
/administration/concepts
|
||||
|
||||
.. cond:: k8s or openshift or container or macos or windows
|
||||
|
||||
@ -174,6 +174,16 @@ The following table lists :mc-cmd:`mc` commands:
|
||||
action granted by the policy without :ref:`authentication
|
||||
<minio-authentication-and-identity-management>`.
|
||||
|
||||
* - | :mc:`mc batch generate`
|
||||
| :mc:`mc batch start`
|
||||
| :mc:`mc batch list`
|
||||
| :mc:`mc batch status`
|
||||
| :mc:`mc batch describe`
|
||||
- The ``mc batch`` commands allow you to run one or more job tasks on a MinIO deployment.
|
||||
Specify the jobs to run in a YAML-formatted file you can start generating with :mc:`mc batch generate`.
|
||||
Use :mc:`mc batch start` to when you are ready to begin the job(s) defined in the file.
|
||||
Display current jobs with :mc:`mc batch list`, see current status with :mc:`mc batch status`, and output the full definition for a current job with :mc:`mc batch describe`.
|
||||
|
||||
* - :mc:`mc cat`
|
||||
- .. include:: /reference/minio-mc/mc-cat.rst
|
||||
:start-after: start-mc-cat-desc
|
||||
@ -471,6 +481,11 @@ All :ref:`commands <minio-mc-commands>` support the following global options:
|
||||
/reference/minio-mc/mc-anonymous-links
|
||||
/reference/minio-mc/mc-anonymous-get-json
|
||||
/reference/minio-mc/mc-anonymous-set-json
|
||||
/reference/minio-mc/mc-batch-describe
|
||||
/reference/minio-mc/mc-batch-generate
|
||||
/reference/minio-mc/mc-batch-list
|
||||
/reference/minio-mc/mc-batch-start
|
||||
/reference/minio-mc/mc-batch-status
|
||||
/reference/minio-mc/mc-cat
|
||||
/reference/minio-mc/mc-cp
|
||||
/reference/minio-mc/mc-diff
|
||||
|
||||
108
source/reference/minio-mc/mc-batch-describe.rst
Normal file
108
source/reference/minio-mc/mc-batch-describe.rst
Normal file
@ -0,0 +1,108 @@
|
||||
.. _minio-mc-batch-describe:
|
||||
|
||||
=====================
|
||||
``mc batch describe``
|
||||
=====================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
.. mc:: mc batch describe
|
||||
|
||||
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
|
||||
|
||||
Syntax
|
||||
------
|
||||
|
||||
.. start-mc-batch-describe-desc
|
||||
|
||||
The :mc:`mc batch describe` command outputs the job definition for a specified job ID.
|
||||
|
||||
.. end-mc-batch-describe-desc
|
||||
|
||||
You must specify the job ID.
|
||||
To find the job ID, use :mc-cmd:`mc batch list`.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: EXAMPLE
|
||||
|
||||
The following command outputs the job definition for the job identified as ``KwSysDpxcBU9FNhGkn2dCf``.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc batch describe myminio KwSysDpxcBU9FNhGkn2dCf
|
||||
|
||||
.. tab-item:: SYNTAX
|
||||
|
||||
The command has the following syntax:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc [GLOBALFLAGS] batch describe TARGET \
|
||||
JOBID
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-syntax
|
||||
:end-before: end-minio-syntax
|
||||
|
||||
Parameters
|
||||
~~~~~~~~~~
|
||||
|
||||
.. mc-cmd:: TARGET
|
||||
:required:
|
||||
|
||||
The :ref:`alias <alias>` for the MinIO deployment to look for the Job ID.
|
||||
|
||||
.. mc-cmd:: JOBID
|
||||
:required:
|
||||
|
||||
The unique identifier of a job to describe.
|
||||
To find the ID of a job, use :mc:`mc batch list`.
|
||||
|
||||
Global Flags
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-globals
|
||||
:end-before: end-minio-mc-globals
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
Show the Definition of an In Progress Batch Job
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following command provides the full job definition of a specific job at :mc:`alias <mc alias>` ``myminio``:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc batch describe myminio KwSysDpxcBU9FNhGkn2dCf
|
||||
|
||||
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
|
||||
- Replace ``KwSysDpxcBU9FNhGkn2dCf`` with the ID of the job to define.
|
||||
|
||||
The output of the above command is similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mc batch describe myminio KwSysDpxcBU9FNhGkn2dCf
|
||||
replicate:
|
||||
apiVersion: v1
|
||||
...
|
||||
|
||||
Note, this example is truncated.
|
||||
The output is the full job definition for the specified job.
|
||||
|
||||
S3 Compatibility
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-s3-compatibility
|
||||
:end-before: end-minio-mc-s3-compatibility
|
||||
309
source/reference/minio-mc/mc-batch-generate.rst
Normal file
309
source/reference/minio-mc/mc-batch-generate.rst
Normal file
@ -0,0 +1,309 @@
|
||||
.. _minio-mc-batch-generate:
|
||||
|
||||
=====================
|
||||
``mc batch generate``
|
||||
=====================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
.. mc:: mc batch
|
||||
|
||||
.. mc:: mc batch generate
|
||||
|
||||
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
|
||||
|
||||
Syntax
|
||||
------
|
||||
|
||||
.. start-mc-batch-generate-desc
|
||||
|
||||
The :mc:`mc batch generate` command creates a basic YAML-formatted template file for the specified job type.
|
||||
|
||||
.. end-mc-batch-generate-desc
|
||||
|
||||
After MinIO creates the file, open it in your preferred text editor tool to further customize.
|
||||
|
||||
- You can add multiple job task definitions to a batch file.
|
||||
- MinIO imposes no limit to the number of job task definitions per batch file.
|
||||
|
||||
See :ref:`job types <minio-batch-job-types>` for the supported jobs you can generate.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: EXAMPLE
|
||||
|
||||
The following command creates a basic YAML file for a replicate job on the ``mybucket`` bucket of the ``myminio`` alias.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc batch generate myminio/mybucket replicate
|
||||
|
||||
.. tab-item:: SYNTAX
|
||||
|
||||
The command has the following syntax:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc [GLOBALFLAGS] batch generate \
|
||||
TARGET \
|
||||
JOBTYPE
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-syntax
|
||||
:end-before: end-minio-syntax
|
||||
|
||||
Parameters
|
||||
~~~~~~~~~~
|
||||
|
||||
.. mc-cmd:: TARGET
|
||||
:required:
|
||||
|
||||
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
mc batch generate myminio/mydata replicate
|
||||
|
||||
You can use just an alias without specifying a bucket to perform the job at the deployment's root.
|
||||
|
||||
.. mc-cmd:: JOBTYPE
|
||||
:required:
|
||||
|
||||
The type of job to generate a YAML document for.
|
||||
|
||||
Currently, :mc:`mc batch` only supports the ``replicate`` job type.
|
||||
|
||||
|
||||
Global Flags
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-globals
|
||||
:end-before: end-minio-mc-globals
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Generate a ``yaml`` File for a Replicate Job Type
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following command generates a YAML blueprint for a replicate type batch job and names the file ``replicate`` with the ``.yaml`` extension:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc batch generate alias/<bucket> replicate
|
||||
|
||||
- Replace ``alias`` with the :mc:`alias <mc alias>` and (optionally) bucket of the MinIO deployment that is the source of the replication job.
|
||||
|
||||
- Replace ``<bucket>`` with the bucket on the alias where the batch should run.
|
||||
The bucket can be left blank to perform the action on the alias deployment's root location.
|
||||
|
||||
- Replace ``replicate`` with the type of job to generate a yaml file for.
|
||||
|
||||
At the time of release, :mc:``mc batch`` only supports the ``replicate`` job type.
|
||||
|
||||
|
||||
S3 Compatibility
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-s3-compatibility
|
||||
:end-before: end-minio-mc-s3-compatibility
|
||||
|
||||
.. _minio-batch-job-types:
|
||||
|
||||
Job Types
|
||||
---------
|
||||
|
||||
:mc:`mc batch` currently supports the following job task types:
|
||||
|
||||
- ``replicate``
|
||||
|
||||
Replicate objects between two MinIO deployments.
|
||||
Provides similar functionality to :ref:`bucket replication <minio-bucket-replication>` as a batch job rather than continual scanning function.
|
||||
|
||||
MinIO may add more job types in the future.
|
||||
|
||||
``replicate``
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Use the ``replicate`` job type to create a batch job that performs replicates objects between two MinIO deployments.
|
||||
|
||||
The YAML **must** define the source and target deployments.
|
||||
Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.
|
||||
|
||||
For the **source deployment**
|
||||
|
||||
- Required information
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:width: 100%
|
||||
|
||||
* - ``type:``
|
||||
- Must be ``s3``.
|
||||
* - ``bucket:``
|
||||
- The bucket on the deployment.
|
||||
|
||||
- Optional information
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:width: 100%
|
||||
|
||||
* - ``prefix:``
|
||||
- The prefix on the object(s) that should replicate.
|
||||
|
||||
* - ``endpoint:``
|
||||
- | Location of the source deployment.
|
||||
| If the location is not remote, use ``local``.
|
||||
|
||||
* - ``credentials:``
|
||||
- The ``accesskey:`` and ``secretKey:`` or the ``sessionToken:`` that grants access to the object(s).
|
||||
|
||||
For the **target deployment**
|
||||
|
||||
- Required information
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:width: 100%
|
||||
|
||||
* - ``type:``
|
||||
- Must be ``s3``.
|
||||
* - ``bucket:``
|
||||
- The bucket on the deployment.
|
||||
|
||||
- Optional information
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:width: 100%
|
||||
|
||||
* - ``prefix:``
|
||||
- The prefix on the object(s) to replicate.
|
||||
|
||||
* - ``endpoint:``
|
||||
- | The location of the source deployment.
|
||||
| If the location is not remote, use ``local``.
|
||||
|
||||
* - ``credentials:``
|
||||
- The ``accesskey`` and ``secretKey`` or the ``sessionToken`` that grants access to the object(s).
|
||||
|
||||
For **filters**
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:width: 100%
|
||||
|
||||
* - ``newerThan:``
|
||||
- A string representing a length of time in ``#d#h#s`` format.
|
||||
|
||||
Only objects newer than the specified length of time replicate.
|
||||
For example, ``7d``, ``24h``, ``5d12h30s`` are valid strings.
|
||||
* - ``olderThan:``
|
||||
- A string representing a length of time in ``#d#h#s`` format.
|
||||
|
||||
Only objects older than the specified length of time replicate.
|
||||
* - ``createdAfter:``
|
||||
- A date in ``YYYY-MM-DD`` format.
|
||||
|
||||
Only objects created after the date replicate.
|
||||
* - ``createdBefore:``
|
||||
- A date in ``YYYY-MM-DD`` format.
|
||||
|
||||
Only objects created prior to the date replicate.
|
||||
|
||||
For **notifications**
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:width: 100%
|
||||
|
||||
* - ``endpoint:``
|
||||
- The predefined endpoint to send events for notifications.
|
||||
* - ``token:``
|
||||
- An optional :abbr:`JWT <JSON Web Token>` to access the ``endpoint``.
|
||||
|
||||
For **retry attempts**
|
||||
|
||||
If something interrupts the job, you can define how many attempts to retry the job batch.
|
||||
For each retry, you can also define how long to wait between attempts.
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:width: 100%
|
||||
|
||||
* - ``attempts:``
|
||||
- Number of tries to complete the batch job before giving up.
|
||||
* - ``delay:``
|
||||
- The least amount of time to wait between each attempt.
|
||||
|
||||
|
||||
Sample YAML
|
||||
+++++++++++
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
replicate:
|
||||
apiVersion: v1
|
||||
# source of the objects to be replicated
|
||||
source:
|
||||
type: TYPE # valid values are "s3"
|
||||
bucket: BUCKET
|
||||
prefix: PREFIX
|
||||
# NOTE: if source is remote then target must be "local"
|
||||
# endpoint: ENDPOINT
|
||||
# credentials:
|
||||
# accessKey: ACCESS-KEY
|
||||
# secretKey: SECRET-KEY
|
||||
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
|
||||
|
||||
# target where the objects must be replicated
|
||||
target:
|
||||
type: TYPE # valid values are "s3"
|
||||
bucket: BUCKET
|
||||
prefix: PREFIX
|
||||
# NOTE: if target is remote then source must be "local"
|
||||
# endpoint: ENDPOINT
|
||||
# credentials:
|
||||
# accessKey: ACCESS-KEY
|
||||
# secretKey: SECRET-KEY
|
||||
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
|
||||
|
||||
# optional flags based filtering criteria
|
||||
# for all source objects
|
||||
flags:
|
||||
filter:
|
||||
newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
|
||||
olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
|
||||
createdAfter: "date" # match objects created after "date"
|
||||
createdBefore: "date" # match objects created before "date"
|
||||
|
||||
## NOTE: tags are not supported when "source" is remote.
|
||||
# tags:
|
||||
# - key: "name"
|
||||
# value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
|
||||
|
||||
## NOTE: metadata filter not supported when "source" is non MinIO.
|
||||
# metadata:
|
||||
# - key: "content-type"
|
||||
# value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
|
||||
|
||||
notify:
|
||||
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
|
||||
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
|
||||
|
||||
retry:
|
||||
attempts: 10 # number of retries for the job before giving up
|
||||
delay: "500ms" # least amount of delay between each retry
|
||||
|
||||
103
source/reference/minio-mc/mc-batch-list.rst
Normal file
103
source/reference/minio-mc/mc-batch-list.rst
Normal file
@ -0,0 +1,103 @@
|
||||
.. _minio-mc-batch-list:
|
||||
|
||||
=================
|
||||
``mc batch list``
|
||||
=================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
.. mc:: mc batch list
|
||||
|
||||
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
|
||||
|
||||
Syntax
|
||||
------
|
||||
|
||||
.. start-mc-batch-list-desc
|
||||
|
||||
The :mc:`mc batch list` command outputs a list of the batch jobs currently in progress on a deployment.
|
||||
|
||||
.. end-mc-batch-list-desc
|
||||
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: EXAMPLE
|
||||
|
||||
The following command outputs a list of all jobs currently in progress on the ``myminio`` alias.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc batch list myminio
|
||||
|
||||
.. tab-item:: SYNTAX
|
||||
|
||||
The command has the following syntax:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc [GLOBALFLAGS] batch list TARGET \
|
||||
--type "string"
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-syntax
|
||||
:end-before: end-minio-syntax
|
||||
|
||||
Parameters
|
||||
~~~~~~~~~~
|
||||
|
||||
.. mc-cmd:: TARGET
|
||||
:required:
|
||||
|
||||
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
|
||||
|
||||
.. mc-cmd:: --type
|
||||
:optional:
|
||||
|
||||
List batch jobs only of a certain type.
|
||||
|
||||
Global Flags
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-globals
|
||||
:end-before: end-minio-mc-globals
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
List all ``replicate`` type batch jobs
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following command starts the batch of job(s) defined in the file ``replication.yaml`` on the deployment at :mc:`alias <mc alias>` ``myminio``:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc batch list myminio --type "replicate"
|
||||
|
||||
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
|
||||
|
||||
- Replace ``replicate`` with the job type to output.
|
||||
|
||||
Currently, :mc:`mc batch` only supports the ``replicate`` job type.
|
||||
|
||||
The output of the above command is similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
ID TYPE USER STARTED
|
||||
E24HH4nNMcgY5taynaPfxu replicate minioadmin 1 minute ago
|
||||
|
||||
S3 Compatibility
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-s3-compatibility
|
||||
:end-before: end-minio-mc-s3-compatibility
|
||||
119
source/reference/minio-mc/mc-batch-start.rst
Normal file
119
source/reference/minio-mc/mc-batch-start.rst
Normal file
@ -0,0 +1,119 @@
|
||||
.. _minio-mc-batch-start:
|
||||
|
||||
==================
|
||||
``mc batch start``
|
||||
==================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
.. mc:: mc batch start
|
||||
|
||||
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
|
||||
|
||||
Syntax
|
||||
------
|
||||
|
||||
.. start-mc-batch-start-desc
|
||||
|
||||
The :mc:`mc batch start` command launches a batch job from a job batch YAML file.
|
||||
|
||||
.. end-mc-batch-start-desc
|
||||
|
||||
The batch job runs to completion (or up to the number of retries specified in the file) one time.
|
||||
To run the batch job again after completion, you must start it again.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: EXAMPLE
|
||||
|
||||
The following command creates a basic YAML file for a replicate job on the ``mybucket`` bucket of the ``myminio`` alias.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc batch start myminio/mybucket jobfile.yaml
|
||||
|
||||
The output of the above command is something similar to:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
Successfully start 'replicate' job `B34HHqnNMcg1taynaPfxu` on '2022-10-24 17:19:06.296974771 -0700 PDT'
|
||||
|
||||
.. tab-item:: SYNTAX
|
||||
|
||||
The command has the following syntax:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc [GLOBALFLAGS] batch start \
|
||||
TARGET \
|
||||
JOBFILE
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-syntax
|
||||
:end-before: end-minio-syntax
|
||||
|
||||
Parameters
|
||||
~~~~~~~~~~
|
||||
|
||||
.. mc-cmd:: TARGET
|
||||
:required:
|
||||
|
||||
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
mc batch start myminio/mydata replicate.yaml
|
||||
|
||||
You can use just an alias without specifying a bucket to perform the job at the deployment's root.
|
||||
|
||||
.. mc-cmd:: JOBFILE
|
||||
:required:
|
||||
|
||||
A YAML-defined batch job.
|
||||
The job may have as many tasks as desired; there is no predefined limit.
|
||||
|
||||
Global Flags
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-globals
|
||||
:end-before: end-minio-mc-globals
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
Start a Batch Job
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following command starts the batch of job(s) defined in the file ``replication.yaml`` on the deployment at :mc:`alias <mc alias>` ``myminio``:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc batch start myminio ./replication.yaml
|
||||
|
||||
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
|
||||
|
||||
- Replace ``./replication.yaml`` with the yaml-formatted file that describes the batch job.
|
||||
Use the file path relative to your current location.
|
||||
|
||||
The output of the above command is similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
Successfully start 'replicate' job `E24HH4nNMcgY5taynaPfxu` on '2022-09-26 17:19:06.296974771 -0700 PDT'
|
||||
|
||||
S3 Compatibility
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-s3-compatibility
|
||||
:end-before: end-minio-mc-s3-compatibility
|
||||
105
source/reference/minio-mc/mc-batch-status.rst
Normal file
105
source/reference/minio-mc/mc-batch-status.rst
Normal file
@ -0,0 +1,105 @@
|
||||
.. _minio-mc-batch-status:
|
||||
|
||||
===================
|
||||
``mc batch status``
|
||||
===================
|
||||
|
||||
.. default-domain:: minio
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
.. mc:: mc batch status
|
||||
|
||||
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
|
||||
|
||||
Syntax
|
||||
------
|
||||
|
||||
.. start-mc-batch-status-desc
|
||||
|
||||
The :mc:`mc batch status` command outputs real-time summaries of job events on a MinIO server.
|
||||
|
||||
.. end-mc-batch-status-desc
|
||||
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: EXAMPLE
|
||||
|
||||
The following command outputs a list of all jobs currently in progress on the ``myminio`` alias.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc batch status myminio KwSysDpxcBU9FNhGkn2dCf
|
||||
|
||||
.. tab-item:: SYNTAX
|
||||
|
||||
The command has the following syntax:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc [GLOBALFLAGS] batch list TARGET \
|
||||
JOBID
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-syntax
|
||||
:end-before: end-minio-syntax
|
||||
|
||||
Parameters
|
||||
~~~~~~~~~~
|
||||
|
||||
.. mc-cmd:: TARGET
|
||||
:required:
|
||||
|
||||
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
|
||||
|
||||
.. mc-cmd:: JOBID
|
||||
:optional:
|
||||
|
||||
The unique identifier of a job to summarize.
|
||||
To find the ID of a job, use :mc:`mc batch list`.
|
||||
|
||||
Global Flags
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-globals
|
||||
:end-before: end-minio-mc-globals
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
Summary the Events of a Replicate Job
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following command provides the real-time summary of an active job on the deployment at :mc:`alias <mc alias>` ``myminio``:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc batch status myminio KwSysDpxcBU9FNhGkn2dCf
|
||||
|
||||
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
|
||||
|
||||
The output of the above command is similar to the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
●∙∙
|
||||
Objects: 28766
|
||||
Versions: 28766
|
||||
Throughput: 3.0 MiB/s
|
||||
Transferred: 406 MiB
|
||||
Elapsed: 2m14.227222868s
|
||||
CurrObjName: share/doc/xml-core/examples/foo.xmlcatalogs
|
||||
|
||||
S3 Compatibility
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-s3-compatibility
|
||||
:end-before: end-minio-mc-s3-compatibility
|
||||
Reference in New Issue
Block a user