mirror of
https://github.com/minio/docs.git
synced 2025-07-28 19:42:10 +03:00
Adds mc batch commands to the docs (#620)
Closes #605 - Adds the five `mc batch` subcommands - Adds a batch framework page to the Administration section of docs
This commit is contained in:
158
source/administration/batch-framework.rst
Normal file
158
source/administration/batch-framework.rst
Normal file
@ -0,0 +1,158 @@
|
|||||||
|
.. _minio-batch-framework:
|
||||||
|
|
||||||
|
===============
|
||||||
|
Batch Framework
|
||||||
|
===============
|
||||||
|
|
||||||
|
.. default-domain:: minio
|
||||||
|
|
||||||
|
.. contents:: Table of Contents
|
||||||
|
:local:
|
||||||
|
:depth: 2
|
||||||
|
|
||||||
|
.. versionadded:: MinIO RELEASE.2022-10-08T20-11-00Z
|
||||||
|
|
||||||
|
The Batch Framework was introduced with the ``replicate`` job type in the :mc:`mc` RELEASES.2022-10-08T20-11-00Z.
|
||||||
|
|
||||||
|
Overview
|
||||||
|
--------
|
||||||
|
|
||||||
|
The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a "batch file").
|
||||||
|
The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the :ref:`MinIO Client <minio-client>`.
|
||||||
|
|
||||||
|
A batch file can define any number of job tasks.
|
||||||
|
MinIO does not limit on the number of job tasks that you can define in a batch file.
|
||||||
|
|
||||||
|
Once started, MinIO starts processing the job.
|
||||||
|
Time to completion depends on the resources available to the deployment.
|
||||||
|
|
||||||
|
If any portion of the job fails, MinIO retries the job up to the number of times defined in the job definition.
|
||||||
|
|
||||||
|
The MinIO Batch Framework supports the following job types:
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:header-rows: 1
|
||||||
|
:widths: 30 70
|
||||||
|
:width: 100%
|
||||||
|
|
||||||
|
* - Job Type
|
||||||
|
- Description
|
||||||
|
|
||||||
|
* - ``replicate``
|
||||||
|
- Perform a one-time replication procedure from one MinIO location to another MinIO location.
|
||||||
|
|
||||||
|
MinIO Batch CLI
|
||||||
|
---------------
|
||||||
|
|
||||||
|
- Install the :ref:`MinIO Client <minio-client>`
|
||||||
|
- Define an :mc-cmd:`alias <mc alias set>` for the MinIO deployment
|
||||||
|
|
||||||
|
The :mc:`mc batch` commands include
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:widths: 30 70
|
||||||
|
:width: 90%
|
||||||
|
|
||||||
|
* - :mc:`mc batch generate`
|
||||||
|
- .. include:: /reference/minio-mc/mc-batch-generate.rst
|
||||||
|
:start-after: start-mc-batch-generate-desc
|
||||||
|
:end-before: end-mc-batch-generate-desc
|
||||||
|
* - :mc:`mc batch start`
|
||||||
|
- .. include:: /reference/minio-mc/mc-batch-start.rst
|
||||||
|
:start-after: start-mc-batch-start-desc
|
||||||
|
:end-before: end-mc-batch-start-desc
|
||||||
|
* - :mc:`mc batch list`
|
||||||
|
- .. include:: /reference/minio-mc/mc-batch-list.rst
|
||||||
|
:start-after: start-mc-batch-list-desc
|
||||||
|
:end-before: end-mc-batch-list-desc
|
||||||
|
* - :mc:`mc batch status`
|
||||||
|
- .. include:: /reference/minio-mc/mc-batch-status.rst
|
||||||
|
:start-after: start-mc-batch-status-desc
|
||||||
|
:end-before: end-mc-batch-status-desc
|
||||||
|
* - :mc:`mc batch describe`
|
||||||
|
- .. include:: /reference/minio-mc/mc-batch-describe.rst
|
||||||
|
:start-after: start-mc-batch-describe-desc
|
||||||
|
:end-before: end-mc-batch-describe-desc
|
||||||
|
|
||||||
|
Access to ``mc batch``
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
You can use MinIO's :ref:`Policy Based Access Control <minio-policy>` and the :ref:`administrative policy actions <minio-policy-mc-admin-actions>` to restrict who can start a batch job, retrieve a list of running jobs, or describe a running job.
|
||||||
|
|
||||||
|
Job Types
|
||||||
|
---------
|
||||||
|
|
||||||
|
``replicate``
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The ``replicate`` job type performs a single-run replication of objects from one MinIO deployment to another.
|
||||||
|
The definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects.
|
||||||
|
|
||||||
|
For example, you can use a batch job to perform a one-time replication sync of objects from ``minio-alpha/invoices/`` to ``minio-baker/invoices``.
|
||||||
|
|
||||||
|
The advantages of Batch Replication over :mc:`mc mirror` include:
|
||||||
|
|
||||||
|
- Removes the client to cluster network as a potential bottleneck
|
||||||
|
- A user only needs access to starting a batch job with no other permissions, as the job runs entirely server side on the cluster
|
||||||
|
- The job provides for retry attempts in event that objects do not replicate
|
||||||
|
- Batch jobs are one-time, curated processes allowing for fine control replication
|
||||||
|
|
||||||
|
Sample YAML Description File for a ``replicate`` Job Type
|
||||||
|
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
Create a basic ``replicate`` job definition file you can edit with :mc-cmd:`mc batch generate`.
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
replicate:
|
||||||
|
apiVersion: v1
|
||||||
|
# source of the objects to be replicated
|
||||||
|
source:
|
||||||
|
type: TYPE # valid values are "s3"
|
||||||
|
bucket: BUCKET
|
||||||
|
prefix: PREFIX
|
||||||
|
# NOTE: if source is remote then target must be "local"
|
||||||
|
# endpoint: ENDPOINT
|
||||||
|
# credentials:
|
||||||
|
# accessKey: ACCESS-KEY
|
||||||
|
# secretKey: SECRET-KEY
|
||||||
|
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
|
||||||
|
|
||||||
|
# target where the objects must be replicated
|
||||||
|
target:
|
||||||
|
type: TYPE # valid values are "s3"
|
||||||
|
bucket: BUCKET
|
||||||
|
prefix: PREFIX
|
||||||
|
# NOTE: if target is remote then source must be "local"
|
||||||
|
# endpoint: ENDPOINT
|
||||||
|
# credentials:
|
||||||
|
# accessKey: ACCESS-KEY
|
||||||
|
# secretKey: SECRET-KEY
|
||||||
|
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
|
||||||
|
|
||||||
|
# optional flags based filtering criteria
|
||||||
|
# for all source objects
|
||||||
|
flags:
|
||||||
|
filter:
|
||||||
|
newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
|
||||||
|
olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
|
||||||
|
createdAfter: "date" # match objects created after "date"
|
||||||
|
createdBefore: "date" # match objects created before "date"
|
||||||
|
|
||||||
|
## NOTE: tags are not supported when "source" is remote.
|
||||||
|
# tags:
|
||||||
|
# - key: "name"
|
||||||
|
# value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
|
||||||
|
|
||||||
|
## NOTE: metadata filter not supported when "source" is non MinIO.
|
||||||
|
# metadata:
|
||||||
|
# - key: "content-type"
|
||||||
|
# value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
|
||||||
|
|
||||||
|
notify:
|
||||||
|
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
|
||||||
|
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
|
||||||
|
|
||||||
|
retry:
|
||||||
|
attempts: 10 # number of retries for the job before giving up
|
||||||
|
delay: "500ms" # least amount of delay between each retry
|
@ -761,6 +761,18 @@ services:
|
|||||||
Allows access to MinIO :ref:`metrics <minio-metrics-and-alerts-endpoints>`.
|
Allows access to MinIO :ref:`metrics <minio-metrics-and-alerts-endpoints>`.
|
||||||
Only required if MinIO requires authentication for scraping metrics.
|
Only required if MinIO requires authentication for scraping metrics.
|
||||||
|
|
||||||
|
.. policy-action:: admin:ListBatchJobs
|
||||||
|
|
||||||
|
Allows access to list the active batch jobs.
|
||||||
|
|
||||||
|
.. policy-action:: admin:DescribeBatchJobs
|
||||||
|
|
||||||
|
Allows access to the see the definition details of a running batch job.
|
||||||
|
|
||||||
|
.. policy-action:: admin:StartBatchJob
|
||||||
|
|
||||||
|
Allows user to begin a batch job run.
|
||||||
|
|
||||||
``mc admin`` Policy Condition Keys
|
``mc admin`` Policy Condition Keys
|
||||||
----------------------------------
|
----------------------------------
|
||||||
|
|
||||||
|
@ -94,6 +94,7 @@ and non-protected.
|
|||||||
/administration/identity-access-management
|
/administration/identity-access-management
|
||||||
/administration/server-side-encryption
|
/administration/server-side-encryption
|
||||||
/administration/bucket-replication
|
/administration/bucket-replication
|
||||||
|
/administration/batch-framework
|
||||||
/administration/concepts
|
/administration/concepts
|
||||||
|
|
||||||
|
|
||||||
@ -119,6 +120,7 @@ and non-protected.
|
|||||||
/administration/identity-access-management
|
/administration/identity-access-management
|
||||||
/administration/server-side-encryption
|
/administration/server-side-encryption
|
||||||
/administration/bucket-replication
|
/administration/bucket-replication
|
||||||
|
/administration/batch-framework
|
||||||
/administration/concepts
|
/administration/concepts
|
||||||
|
|
||||||
.. cond:: k8s or openshift or container or macos or windows
|
.. cond:: k8s or openshift or container or macos or windows
|
||||||
|
@ -174,6 +174,16 @@ The following table lists :mc-cmd:`mc` commands:
|
|||||||
action granted by the policy without :ref:`authentication
|
action granted by the policy without :ref:`authentication
|
||||||
<minio-authentication-and-identity-management>`.
|
<minio-authentication-and-identity-management>`.
|
||||||
|
|
||||||
|
* - | :mc:`mc batch generate`
|
||||||
|
| :mc:`mc batch start`
|
||||||
|
| :mc:`mc batch list`
|
||||||
|
| :mc:`mc batch status`
|
||||||
|
| :mc:`mc batch describe`
|
||||||
|
- The ``mc batch`` commands allow you to run one or more job tasks on a MinIO deployment.
|
||||||
|
Specify the jobs to run in a YAML-formatted file you can start generating with :mc:`mc batch generate`.
|
||||||
|
Use :mc:`mc batch start` to when you are ready to begin the job(s) defined in the file.
|
||||||
|
Display current jobs with :mc:`mc batch list`, see current status with :mc:`mc batch status`, and output the full definition for a current job with :mc:`mc batch describe`.
|
||||||
|
|
||||||
* - :mc:`mc cat`
|
* - :mc:`mc cat`
|
||||||
- .. include:: /reference/minio-mc/mc-cat.rst
|
- .. include:: /reference/minio-mc/mc-cat.rst
|
||||||
:start-after: start-mc-cat-desc
|
:start-after: start-mc-cat-desc
|
||||||
@ -471,6 +481,11 @@ All :ref:`commands <minio-mc-commands>` support the following global options:
|
|||||||
/reference/minio-mc/mc-anonymous-links
|
/reference/minio-mc/mc-anonymous-links
|
||||||
/reference/minio-mc/mc-anonymous-get-json
|
/reference/minio-mc/mc-anonymous-get-json
|
||||||
/reference/minio-mc/mc-anonymous-set-json
|
/reference/minio-mc/mc-anonymous-set-json
|
||||||
|
/reference/minio-mc/mc-batch-describe
|
||||||
|
/reference/minio-mc/mc-batch-generate
|
||||||
|
/reference/minio-mc/mc-batch-list
|
||||||
|
/reference/minio-mc/mc-batch-start
|
||||||
|
/reference/minio-mc/mc-batch-status
|
||||||
/reference/minio-mc/mc-cat
|
/reference/minio-mc/mc-cat
|
||||||
/reference/minio-mc/mc-cp
|
/reference/minio-mc/mc-cp
|
||||||
/reference/minio-mc/mc-diff
|
/reference/minio-mc/mc-diff
|
||||||
|
108
source/reference/minio-mc/mc-batch-describe.rst
Normal file
108
source/reference/minio-mc/mc-batch-describe.rst
Normal file
@ -0,0 +1,108 @@
|
|||||||
|
.. _minio-mc-batch-describe:
|
||||||
|
|
||||||
|
=====================
|
||||||
|
``mc batch describe``
|
||||||
|
=====================
|
||||||
|
|
||||||
|
.. default-domain:: minio
|
||||||
|
|
||||||
|
.. contents:: Table of Contents
|
||||||
|
:local:
|
||||||
|
:depth: 2
|
||||||
|
|
||||||
|
.. mc:: mc batch describe
|
||||||
|
|
||||||
|
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
|
||||||
|
|
||||||
|
Syntax
|
||||||
|
------
|
||||||
|
|
||||||
|
.. start-mc-batch-describe-desc
|
||||||
|
|
||||||
|
The :mc:`mc batch describe` command outputs the job definition for a specified job ID.
|
||||||
|
|
||||||
|
.. end-mc-batch-describe-desc
|
||||||
|
|
||||||
|
You must specify the job ID.
|
||||||
|
To find the job ID, use :mc-cmd:`mc batch list`.
|
||||||
|
|
||||||
|
.. tab-set::
|
||||||
|
|
||||||
|
.. tab-item:: EXAMPLE
|
||||||
|
|
||||||
|
The following command outputs the job definition for the job identified as ``KwSysDpxcBU9FNhGkn2dCf``.
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc batch describe myminio KwSysDpxcBU9FNhGkn2dCf
|
||||||
|
|
||||||
|
.. tab-item:: SYNTAX
|
||||||
|
|
||||||
|
The command has the following syntax:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc [GLOBALFLAGS] batch describe TARGET \
|
||||||
|
JOBID
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-syntax
|
||||||
|
:end-before: end-minio-syntax
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
.. mc-cmd:: TARGET
|
||||||
|
:required:
|
||||||
|
|
||||||
|
The :ref:`alias <alias>` for the MinIO deployment to look for the Job ID.
|
||||||
|
|
||||||
|
.. mc-cmd:: JOBID
|
||||||
|
:required:
|
||||||
|
|
||||||
|
The unique identifier of a job to describe.
|
||||||
|
To find the ID of a job, use :mc:`mc batch list`.
|
||||||
|
|
||||||
|
Global Flags
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-mc-globals
|
||||||
|
:end-before: end-minio-mc-globals
|
||||||
|
|
||||||
|
Example
|
||||||
|
-------
|
||||||
|
|
||||||
|
Show the Definition of an In Progress Batch Job
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The following command provides the full job definition of a specific job at :mc:`alias <mc alias>` ``myminio``:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc batch describe myminio KwSysDpxcBU9FNhGkn2dCf
|
||||||
|
|
||||||
|
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
|
||||||
|
- Replace ``KwSysDpxcBU9FNhGkn2dCf`` with the ID of the job to define.
|
||||||
|
|
||||||
|
The output of the above command is similar to the following:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
|
||||||
|
mc batch describe myminio KwSysDpxcBU9FNhGkn2dCf
|
||||||
|
replicate:
|
||||||
|
apiVersion: v1
|
||||||
|
...
|
||||||
|
|
||||||
|
Note, this example is truncated.
|
||||||
|
The output is the full job definition for the specified job.
|
||||||
|
|
||||||
|
S3 Compatibility
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-mc-s3-compatibility
|
||||||
|
:end-before: end-minio-mc-s3-compatibility
|
309
source/reference/minio-mc/mc-batch-generate.rst
Normal file
309
source/reference/minio-mc/mc-batch-generate.rst
Normal file
@ -0,0 +1,309 @@
|
|||||||
|
.. _minio-mc-batch-generate:
|
||||||
|
|
||||||
|
=====================
|
||||||
|
``mc batch generate``
|
||||||
|
=====================
|
||||||
|
|
||||||
|
.. default-domain:: minio
|
||||||
|
|
||||||
|
.. contents:: Table of Contents
|
||||||
|
:local:
|
||||||
|
:depth: 2
|
||||||
|
|
||||||
|
.. mc:: mc batch
|
||||||
|
|
||||||
|
.. mc:: mc batch generate
|
||||||
|
|
||||||
|
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
|
||||||
|
|
||||||
|
Syntax
|
||||||
|
------
|
||||||
|
|
||||||
|
.. start-mc-batch-generate-desc
|
||||||
|
|
||||||
|
The :mc:`mc batch generate` command creates a basic YAML-formatted template file for the specified job type.
|
||||||
|
|
||||||
|
.. end-mc-batch-generate-desc
|
||||||
|
|
||||||
|
After MinIO creates the file, open it in your preferred text editor tool to further customize.
|
||||||
|
|
||||||
|
- You can add multiple job task definitions to a batch file.
|
||||||
|
- MinIO imposes no limit to the number of job task definitions per batch file.
|
||||||
|
|
||||||
|
See :ref:`job types <minio-batch-job-types>` for the supported jobs you can generate.
|
||||||
|
|
||||||
|
.. tab-set::
|
||||||
|
|
||||||
|
.. tab-item:: EXAMPLE
|
||||||
|
|
||||||
|
The following command creates a basic YAML file for a replicate job on the ``mybucket`` bucket of the ``myminio`` alias.
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc batch generate myminio/mybucket replicate
|
||||||
|
|
||||||
|
.. tab-item:: SYNTAX
|
||||||
|
|
||||||
|
The command has the following syntax:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc [GLOBALFLAGS] batch generate \
|
||||||
|
TARGET \
|
||||||
|
JOBTYPE
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-syntax
|
||||||
|
:end-before: end-minio-syntax
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
.. mc-cmd:: TARGET
|
||||||
|
:required:
|
||||||
|
|
||||||
|
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
.. code-block:: none
|
||||||
|
|
||||||
|
mc batch generate myminio/mydata replicate
|
||||||
|
|
||||||
|
You can use just an alias without specifying a bucket to perform the job at the deployment's root.
|
||||||
|
|
||||||
|
.. mc-cmd:: JOBTYPE
|
||||||
|
:required:
|
||||||
|
|
||||||
|
The type of job to generate a YAML document for.
|
||||||
|
|
||||||
|
Currently, :mc:`mc batch` only supports the ``replicate`` job type.
|
||||||
|
|
||||||
|
|
||||||
|
Global Flags
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-mc-globals
|
||||||
|
:end-before: end-minio-mc-globals
|
||||||
|
|
||||||
|
Examples
|
||||||
|
--------
|
||||||
|
|
||||||
|
Generate a ``yaml`` File for a Replicate Job Type
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The following command generates a YAML blueprint for a replicate type batch job and names the file ``replicate`` with the ``.yaml`` extension:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc batch generate alias/<bucket> replicate
|
||||||
|
|
||||||
|
- Replace ``alias`` with the :mc:`alias <mc alias>` and (optionally) bucket of the MinIO deployment that is the source of the replication job.
|
||||||
|
|
||||||
|
- Replace ``<bucket>`` with the bucket on the alias where the batch should run.
|
||||||
|
The bucket can be left blank to perform the action on the alias deployment's root location.
|
||||||
|
|
||||||
|
- Replace ``replicate`` with the type of job to generate a yaml file for.
|
||||||
|
|
||||||
|
At the time of release, :mc:``mc batch`` only supports the ``replicate`` job type.
|
||||||
|
|
||||||
|
|
||||||
|
S3 Compatibility
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-mc-s3-compatibility
|
||||||
|
:end-before: end-minio-mc-s3-compatibility
|
||||||
|
|
||||||
|
.. _minio-batch-job-types:
|
||||||
|
|
||||||
|
Job Types
|
||||||
|
---------
|
||||||
|
|
||||||
|
:mc:`mc batch` currently supports the following job task types:
|
||||||
|
|
||||||
|
- ``replicate``
|
||||||
|
|
||||||
|
Replicate objects between two MinIO deployments.
|
||||||
|
Provides similar functionality to :ref:`bucket replication <minio-bucket-replication>` as a batch job rather than continual scanning function.
|
||||||
|
|
||||||
|
MinIO may add more job types in the future.
|
||||||
|
|
||||||
|
``replicate``
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Use the ``replicate`` job type to create a batch job that performs replicates objects between two MinIO deployments.
|
||||||
|
|
||||||
|
The YAML **must** define the source and target deployments.
|
||||||
|
Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.
|
||||||
|
|
||||||
|
For the **source deployment**
|
||||||
|
|
||||||
|
- Required information
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:widths: 25 75
|
||||||
|
:width: 100%
|
||||||
|
|
||||||
|
* - ``type:``
|
||||||
|
- Must be ``s3``.
|
||||||
|
* - ``bucket:``
|
||||||
|
- The bucket on the deployment.
|
||||||
|
|
||||||
|
- Optional information
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:widths: 25 75
|
||||||
|
:width: 100%
|
||||||
|
|
||||||
|
* - ``prefix:``
|
||||||
|
- The prefix on the object(s) that should replicate.
|
||||||
|
|
||||||
|
* - ``endpoint:``
|
||||||
|
- | Location of the source deployment.
|
||||||
|
| If the location is not remote, use ``local``.
|
||||||
|
|
||||||
|
* - ``credentials:``
|
||||||
|
- The ``accesskey:`` and ``secretKey:`` or the ``sessionToken:`` that grants access to the object(s).
|
||||||
|
|
||||||
|
For the **target deployment**
|
||||||
|
|
||||||
|
- Required information
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:widths: 25 75
|
||||||
|
:width: 100%
|
||||||
|
|
||||||
|
* - ``type:``
|
||||||
|
- Must be ``s3``.
|
||||||
|
* - ``bucket:``
|
||||||
|
- The bucket on the deployment.
|
||||||
|
|
||||||
|
- Optional information
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:widths: 25 75
|
||||||
|
:width: 100%
|
||||||
|
|
||||||
|
* - ``prefix:``
|
||||||
|
- The prefix on the object(s) to replicate.
|
||||||
|
|
||||||
|
* - ``endpoint:``
|
||||||
|
- | The location of the source deployment.
|
||||||
|
| If the location is not remote, use ``local``.
|
||||||
|
|
||||||
|
* - ``credentials:``
|
||||||
|
- The ``accesskey`` and ``secretKey`` or the ``sessionToken`` that grants access to the object(s).
|
||||||
|
|
||||||
|
For **filters**
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:widths: 25 75
|
||||||
|
:width: 100%
|
||||||
|
|
||||||
|
* - ``newerThan:``
|
||||||
|
- A string representing a length of time in ``#d#h#s`` format.
|
||||||
|
|
||||||
|
Only objects newer than the specified length of time replicate.
|
||||||
|
For example, ``7d``, ``24h``, ``5d12h30s`` are valid strings.
|
||||||
|
* - ``olderThan:``
|
||||||
|
- A string representing a length of time in ``#d#h#s`` format.
|
||||||
|
|
||||||
|
Only objects older than the specified length of time replicate.
|
||||||
|
* - ``createdAfter:``
|
||||||
|
- A date in ``YYYY-MM-DD`` format.
|
||||||
|
|
||||||
|
Only objects created after the date replicate.
|
||||||
|
* - ``createdBefore:``
|
||||||
|
- A date in ``YYYY-MM-DD`` format.
|
||||||
|
|
||||||
|
Only objects created prior to the date replicate.
|
||||||
|
|
||||||
|
For **notifications**
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:widths: 25 75
|
||||||
|
:width: 100%
|
||||||
|
|
||||||
|
* - ``endpoint:``
|
||||||
|
- The predefined endpoint to send events for notifications.
|
||||||
|
* - ``token:``
|
||||||
|
- An optional :abbr:`JWT <JSON Web Token>` to access the ``endpoint``.
|
||||||
|
|
||||||
|
For **retry attempts**
|
||||||
|
|
||||||
|
If something interrupts the job, you can define how many attempts to retry the job batch.
|
||||||
|
For each retry, you can also define how long to wait between attempts.
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:widths: 25 75
|
||||||
|
:width: 100%
|
||||||
|
|
||||||
|
* - ``attempts:``
|
||||||
|
- Number of tries to complete the batch job before giving up.
|
||||||
|
* - ``delay:``
|
||||||
|
- The least amount of time to wait between each attempt.
|
||||||
|
|
||||||
|
|
||||||
|
Sample YAML
|
||||||
|
+++++++++++
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
replicate:
|
||||||
|
apiVersion: v1
|
||||||
|
# source of the objects to be replicated
|
||||||
|
source:
|
||||||
|
type: TYPE # valid values are "s3"
|
||||||
|
bucket: BUCKET
|
||||||
|
prefix: PREFIX
|
||||||
|
# NOTE: if source is remote then target must be "local"
|
||||||
|
# endpoint: ENDPOINT
|
||||||
|
# credentials:
|
||||||
|
# accessKey: ACCESS-KEY
|
||||||
|
# secretKey: SECRET-KEY
|
||||||
|
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
|
||||||
|
|
||||||
|
# target where the objects must be replicated
|
||||||
|
target:
|
||||||
|
type: TYPE # valid values are "s3"
|
||||||
|
bucket: BUCKET
|
||||||
|
prefix: PREFIX
|
||||||
|
# NOTE: if target is remote then source must be "local"
|
||||||
|
# endpoint: ENDPOINT
|
||||||
|
# credentials:
|
||||||
|
# accessKey: ACCESS-KEY
|
||||||
|
# secretKey: SECRET-KEY
|
||||||
|
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
|
||||||
|
|
||||||
|
# optional flags based filtering criteria
|
||||||
|
# for all source objects
|
||||||
|
flags:
|
||||||
|
filter:
|
||||||
|
newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
|
||||||
|
olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
|
||||||
|
createdAfter: "date" # match objects created after "date"
|
||||||
|
createdBefore: "date" # match objects created before "date"
|
||||||
|
|
||||||
|
## NOTE: tags are not supported when "source" is remote.
|
||||||
|
# tags:
|
||||||
|
# - key: "name"
|
||||||
|
# value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
|
||||||
|
|
||||||
|
## NOTE: metadata filter not supported when "source" is non MinIO.
|
||||||
|
# metadata:
|
||||||
|
# - key: "content-type"
|
||||||
|
# value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
|
||||||
|
|
||||||
|
notify:
|
||||||
|
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
|
||||||
|
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
|
||||||
|
|
||||||
|
retry:
|
||||||
|
attempts: 10 # number of retries for the job before giving up
|
||||||
|
delay: "500ms" # least amount of delay between each retry
|
||||||
|
|
103
source/reference/minio-mc/mc-batch-list.rst
Normal file
103
source/reference/minio-mc/mc-batch-list.rst
Normal file
@ -0,0 +1,103 @@
|
|||||||
|
.. _minio-mc-batch-list:
|
||||||
|
|
||||||
|
=================
|
||||||
|
``mc batch list``
|
||||||
|
=================
|
||||||
|
|
||||||
|
.. default-domain:: minio
|
||||||
|
|
||||||
|
.. contents:: Table of Contents
|
||||||
|
:local:
|
||||||
|
:depth: 2
|
||||||
|
|
||||||
|
.. mc:: mc batch list
|
||||||
|
|
||||||
|
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
|
||||||
|
|
||||||
|
Syntax
|
||||||
|
------
|
||||||
|
|
||||||
|
.. start-mc-batch-list-desc
|
||||||
|
|
||||||
|
The :mc:`mc batch list` command outputs a list of the batch jobs currently in progress on a deployment.
|
||||||
|
|
||||||
|
.. end-mc-batch-list-desc
|
||||||
|
|
||||||
|
|
||||||
|
.. tab-set::
|
||||||
|
|
||||||
|
.. tab-item:: EXAMPLE
|
||||||
|
|
||||||
|
The following command outputs a list of all jobs currently in progress on the ``myminio`` alias.
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc batch list myminio
|
||||||
|
|
||||||
|
.. tab-item:: SYNTAX
|
||||||
|
|
||||||
|
The command has the following syntax:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc [GLOBALFLAGS] batch list TARGET \
|
||||||
|
--type "string"
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-syntax
|
||||||
|
:end-before: end-minio-syntax
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
.. mc-cmd:: TARGET
|
||||||
|
:required:
|
||||||
|
|
||||||
|
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
|
||||||
|
|
||||||
|
.. mc-cmd:: --type
|
||||||
|
:optional:
|
||||||
|
|
||||||
|
List batch jobs only of a certain type.
|
||||||
|
|
||||||
|
Global Flags
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-mc-globals
|
||||||
|
:end-before: end-minio-mc-globals
|
||||||
|
|
||||||
|
Example
|
||||||
|
-------
|
||||||
|
|
||||||
|
List all ``replicate`` type batch jobs
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The following command starts the batch of job(s) defined in the file ``replication.yaml`` on the deployment at :mc:`alias <mc alias>` ``myminio``:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc batch list myminio --type "replicate"
|
||||||
|
|
||||||
|
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
|
||||||
|
|
||||||
|
- Replace ``replicate`` with the job type to output.
|
||||||
|
|
||||||
|
Currently, :mc:`mc batch` only supports the ``replicate`` job type.
|
||||||
|
|
||||||
|
The output of the above command is similar to the following:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
|
||||||
|
ID TYPE USER STARTED
|
||||||
|
E24HH4nNMcgY5taynaPfxu replicate minioadmin 1 minute ago
|
||||||
|
|
||||||
|
S3 Compatibility
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-mc-s3-compatibility
|
||||||
|
:end-before: end-minio-mc-s3-compatibility
|
119
source/reference/minio-mc/mc-batch-start.rst
Normal file
119
source/reference/minio-mc/mc-batch-start.rst
Normal file
@ -0,0 +1,119 @@
|
|||||||
|
.. _minio-mc-batch-start:
|
||||||
|
|
||||||
|
==================
|
||||||
|
``mc batch start``
|
||||||
|
==================
|
||||||
|
|
||||||
|
.. default-domain:: minio
|
||||||
|
|
||||||
|
.. contents:: Table of Contents
|
||||||
|
:local:
|
||||||
|
:depth: 2
|
||||||
|
|
||||||
|
.. mc:: mc batch start
|
||||||
|
|
||||||
|
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
|
||||||
|
|
||||||
|
Syntax
|
||||||
|
------
|
||||||
|
|
||||||
|
.. start-mc-batch-start-desc
|
||||||
|
|
||||||
|
The :mc:`mc batch start` command launches a batch job from a job batch YAML file.
|
||||||
|
|
||||||
|
.. end-mc-batch-start-desc
|
||||||
|
|
||||||
|
The batch job runs to completion (or up to the number of retries specified in the file) one time.
|
||||||
|
To run the batch job again after completion, you must start it again.
|
||||||
|
|
||||||
|
.. tab-set::
|
||||||
|
|
||||||
|
.. tab-item:: EXAMPLE
|
||||||
|
|
||||||
|
The following command creates a basic YAML file for a replicate job on the ``mybucket`` bucket of the ``myminio`` alias.
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc batch start myminio/mybucket jobfile.yaml
|
||||||
|
|
||||||
|
The output of the above command is something similar to:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
|
||||||
|
Successfully start 'replicate' job `B34HHqnNMcg1taynaPfxu` on '2022-10-24 17:19:06.296974771 -0700 PDT'
|
||||||
|
|
||||||
|
.. tab-item:: SYNTAX
|
||||||
|
|
||||||
|
The command has the following syntax:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc [GLOBALFLAGS] batch start \
|
||||||
|
TARGET \
|
||||||
|
JOBFILE
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-syntax
|
||||||
|
:end-before: end-minio-syntax
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
.. mc-cmd:: TARGET
|
||||||
|
:required:
|
||||||
|
|
||||||
|
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
.. code-block:: none
|
||||||
|
|
||||||
|
mc batch start myminio/mydata replicate.yaml
|
||||||
|
|
||||||
|
You can use just an alias without specifying a bucket to perform the job at the deployment's root.
|
||||||
|
|
||||||
|
.. mc-cmd:: JOBFILE
|
||||||
|
:required:
|
||||||
|
|
||||||
|
A YAML-defined batch job.
|
||||||
|
The job may have as many tasks as desired; there is no predefined limit.
|
||||||
|
|
||||||
|
Global Flags
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-mc-globals
|
||||||
|
:end-before: end-minio-mc-globals
|
||||||
|
|
||||||
|
Example
|
||||||
|
-------
|
||||||
|
|
||||||
|
Start a Batch Job
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The following command starts the batch of job(s) defined in the file ``replication.yaml`` on the deployment at :mc:`alias <mc alias>` ``myminio``:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc batch start myminio ./replication.yaml
|
||||||
|
|
||||||
|
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
|
||||||
|
|
||||||
|
- Replace ``./replication.yaml`` with the yaml-formatted file that describes the batch job.
|
||||||
|
Use the file path relative to your current location.
|
||||||
|
|
||||||
|
The output of the above command is similar to the following:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
|
||||||
|
Successfully start 'replicate' job `E24HH4nNMcgY5taynaPfxu` on '2022-09-26 17:19:06.296974771 -0700 PDT'
|
||||||
|
|
||||||
|
S3 Compatibility
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-mc-s3-compatibility
|
||||||
|
:end-before: end-minio-mc-s3-compatibility
|
105
source/reference/minio-mc/mc-batch-status.rst
Normal file
105
source/reference/minio-mc/mc-batch-status.rst
Normal file
@ -0,0 +1,105 @@
|
|||||||
|
.. _minio-mc-batch-status:
|
||||||
|
|
||||||
|
===================
|
||||||
|
``mc batch status``
|
||||||
|
===================
|
||||||
|
|
||||||
|
.. default-domain:: minio
|
||||||
|
|
||||||
|
.. contents:: Table of Contents
|
||||||
|
:local:
|
||||||
|
:depth: 2
|
||||||
|
|
||||||
|
.. mc:: mc batch status
|
||||||
|
|
||||||
|
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
|
||||||
|
|
||||||
|
Syntax
|
||||||
|
------
|
||||||
|
|
||||||
|
.. start-mc-batch-status-desc
|
||||||
|
|
||||||
|
The :mc:`mc batch status` command outputs real-time summaries of job events on a MinIO server.
|
||||||
|
|
||||||
|
.. end-mc-batch-status-desc
|
||||||
|
|
||||||
|
|
||||||
|
.. tab-set::
|
||||||
|
|
||||||
|
.. tab-item:: EXAMPLE
|
||||||
|
|
||||||
|
The following command outputs a list of all jobs currently in progress on the ``myminio`` alias.
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc batch status myminio KwSysDpxcBU9FNhGkn2dCf
|
||||||
|
|
||||||
|
.. tab-item:: SYNTAX
|
||||||
|
|
||||||
|
The command has the following syntax:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc [GLOBALFLAGS] batch list TARGET \
|
||||||
|
JOBID
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-syntax
|
||||||
|
:end-before: end-minio-syntax
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
.. mc-cmd:: TARGET
|
||||||
|
:required:
|
||||||
|
|
||||||
|
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
|
||||||
|
|
||||||
|
.. mc-cmd:: JOBID
|
||||||
|
:optional:
|
||||||
|
|
||||||
|
The unique identifier of a job to summarize.
|
||||||
|
To find the ID of a job, use :mc:`mc batch list`.
|
||||||
|
|
||||||
|
Global Flags
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-mc-globals
|
||||||
|
:end-before: end-minio-mc-globals
|
||||||
|
|
||||||
|
Example
|
||||||
|
-------
|
||||||
|
|
||||||
|
Summary the Events of a Replicate Job
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The following command provides the real-time summary of an active job on the deployment at :mc:`alias <mc alias>` ``myminio``:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc batch status myminio KwSysDpxcBU9FNhGkn2dCf
|
||||||
|
|
||||||
|
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
|
||||||
|
|
||||||
|
The output of the above command is similar to the following:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
|
||||||
|
●∙∙
|
||||||
|
Objects: 28766
|
||||||
|
Versions: 28766
|
||||||
|
Throughput: 3.0 MiB/s
|
||||||
|
Transferred: 406 MiB
|
||||||
|
Elapsed: 2m14.227222868s
|
||||||
|
CurrObjName: share/doc/xml-core/examples/foo.xmlcatalogs
|
||||||
|
|
||||||
|
S3 Compatibility
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. include:: /includes/common-minio-mc.rst
|
||||||
|
:start-after: start-minio-mc-s3-compatibility
|
||||||
|
:end-before: end-minio-mc-s3-compatibility
|
Reference in New Issue
Block a user