minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-07-30 07:03:26 +03:00
Code Activity

Adds mc batch commands to the docs (#620)

Browse Source 
Closes #605

- Adds the five `mc batch` subcommands
- Adds a batch framework page to the Administration section of docs
This commit is contained in:
Daryl White
2022-11-14 17:40:21 -06:00
committed by GitHub
parent 33d5c70922
commit e2f30fb5fd
9 changed files with 931 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
source/reference/minio-mc.rst
View File

@ -174,6 +174,16 @@ The following table lists :mc-cmd:`mc` commands:
action granted by the policy without :ref:`authentication
<minio-authentication-and-identity-management>`.
* - | :mc:`mc batch generate`
| :mc:`mc batch start`
| :mc:`mc batch list`
| :mc:`mc batch status`
| :mc:`mc batch describe`
- The ``mc batch`` commands allow you to run one or more job tasks on a MinIO deployment.
Specify the jobs to run in a YAML-formatted file you can start generating with :mc:`mc batch generate`.
Use :mc:`mc batch start` to when you are ready to begin the job(s) defined in the file.
Display current jobs with :mc:`mc batch list`, see current status with :mc:`mc batch status`, and output the full definition for a current job with :mc:`mc batch describe`.
* - :mc:`mc cat`
- .. include:: /reference/minio-mc/mc-cat.rst
:start-after: start-mc-cat-desc
@ -471,6 +481,11 @@ All :ref:`commands <minio-mc-commands>` support the following global options:
/reference/minio-mc/mc-anonymous-links
/reference/minio-mc/mc-anonymous-get-json
/reference/minio-mc/mc-anonymous-set-json
/reference/minio-mc/mc-batch-describe
/reference/minio-mc/mc-batch-generate
/reference/minio-mc/mc-batch-list
/reference/minio-mc/mc-batch-start
/reference/minio-mc/mc-batch-status
/reference/minio-mc/mc-cat
/reference/minio-mc/mc-cp
/reference/minio-mc/mc-diff

108
source/reference/minio-mc/mc-batch-describe.rst Normal file
View File

@ -0,0 +1,108 @@
.. _minio-mc-batch-describe:
=====================
``mc batch describe``
=====================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
.. mc:: mc batch describe
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
Syntax
------
.. start-mc-batch-describe-desc
The :mc:`mc batch describe` command outputs the job definition for a specified job ID.
.. end-mc-batch-describe-desc
You must specify the job ID.
To find the job ID, use :mc-cmd:`mc batch list`.
.. tab-set::
.. tab-item:: EXAMPLE
The following command outputs the job definition for the job identified as ``KwSysDpxcBU9FNhGkn2dCf``.
.. code-block:: shell
:class: copyable
mc batch describe myminio KwSysDpxcBU9FNhGkn2dCf
.. tab-item:: SYNTAX
The command has the following syntax:
.. code-block:: shell
:class: copyable
mc [GLOBALFLAGS] batch describe TARGET \
JOBID
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-syntax
:end-before: end-minio-syntax
Parameters
~~~~~~~~~~
.. mc-cmd:: TARGET
:required:
The :ref:`alias <alias>` for the MinIO deployment to look for the Job ID.
.. mc-cmd:: JOBID
:required:
The unique identifier of a job to describe.
To find the ID of a job, use :mc:`mc batch list`.
Global Flags
~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-globals
:end-before: end-minio-mc-globals
Example
-------
Show the Definition of an In Progress Batch Job
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following command provides the full job definition of a specific job at :mc:`alias <mc alias>` ``myminio``:
.. code-block:: shell
:class: copyable
mc batch describe myminio KwSysDpxcBU9FNhGkn2dCf
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
- Replace ``KwSysDpxcBU9FNhGkn2dCf`` with the ID of the job to define.
The output of the above command is similar to the following:
.. code-block:: shell
mc batch describe myminio KwSysDpxcBU9FNhGkn2dCf
replicate:
apiVersion: v1
...
Note, this example is truncated.
The output is the full job definition for the specified job.
S3 Compatibility
~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-s3-compatibility
:end-before: end-minio-mc-s3-compatibility

309
source/reference/minio-mc/mc-batch-generate.rst Normal file
View File

@ -0,0 +1,309 @@
.. _minio-mc-batch-generate:
=====================
``mc batch generate``
=====================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
.. mc:: mc batch
.. mc:: mc batch generate
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
Syntax
------
.. start-mc-batch-generate-desc
The :mc:`mc batch generate` command creates a basic YAML-formatted template file for the specified job type.
.. end-mc-batch-generate-desc
After MinIO creates the file, open it in your preferred text editor tool to further customize.
- You can add multiple job task definitions to a batch file.
- MinIO imposes no limit to the number of job task definitions per batch file.
See :ref:`job types <minio-batch-job-types>` for the supported jobs you can generate.
.. tab-set::
.. tab-item:: EXAMPLE
The following command creates a basic YAML file for a replicate job on the ``mybucket`` bucket of the ``myminio`` alias.
.. code-block:: shell
:class: copyable
mc batch generate myminio/mybucket replicate
.. tab-item:: SYNTAX
The command has the following syntax:
.. code-block:: shell
:class: copyable
mc [GLOBALFLAGS] batch generate \
TARGET \
JOBTYPE
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-syntax
:end-before: end-minio-syntax
Parameters
~~~~~~~~~~
.. mc-cmd:: TARGET
:required:
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
For example:
.. code-block:: none
mc batch generate myminio/mydata replicate
You can use just an alias without specifying a bucket to perform the job at the deployment's root.
.. mc-cmd:: JOBTYPE
:required:
The type of job to generate a YAML document for.
Currently, :mc:`mc batch` only supports the ``replicate`` job type.
Global Flags
~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-globals
:end-before: end-minio-mc-globals
Examples
--------
Generate a ``yaml`` File for a Replicate Job Type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following command generates a YAML blueprint for a replicate type batch job and names the file ``replicate`` with the ``.yaml`` extension:
.. code-block:: shell
:class: copyable
mc batch generate alias/<bucket> replicate
- Replace ``alias`` with the :mc:`alias <mc alias>` and (optionally) bucket of the MinIO deployment that is the source of the replication job.
- Replace ``<bucket>`` with the bucket on the alias where the batch should run.
The bucket can be left blank to perform the action on the alias deployment's root location.
- Replace ``replicate`` with the type of job to generate a yaml file for.
At the time of release, :mc:``mc batch`` only supports the ``replicate`` job type.
S3 Compatibility
~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-s3-compatibility
:end-before: end-minio-mc-s3-compatibility
.. _minio-batch-job-types:
Job Types
---------
:mc:`mc batch` currently supports the following job task types:
- ``replicate``
Replicate objects between two MinIO deployments.
Provides similar functionality to :ref:`bucket replication <minio-bucket-replication>` as a batch job rather than continual scanning function.
MinIO may add more job types in the future.
``replicate``
~~~~~~~~~~~~~
Use the ``replicate`` job type to create a batch job that performs replicates objects between two MinIO deployments.
The YAML **must** define the source and target deployments.
Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.
For the **source deployment**
- Required information
.. list-table::
:widths: 25 75
:width: 100%
* - ``type:``
- Must be ``s3``.
* - ``bucket:``
- The bucket on the deployment.
- Optional information
.. list-table::
:widths: 25 75
:width: 100%
* - ``prefix:``
- The prefix on the object(s) that should replicate.
* - ``endpoint:``
- | Location of the source deployment.
| If the location is not remote, use ``local``.
* - ``credentials:``
- The ``accesskey:`` and ``secretKey:`` or the ``sessionToken:`` that grants access to the object(s).
For the **target deployment**
- Required information
.. list-table::
:widths: 25 75
:width: 100%
* - ``type:``
- Must be ``s3``.
* - ``bucket:``
- The bucket on the deployment.
- Optional information
.. list-table::
:widths: 25 75
:width: 100%
* - ``prefix:``
- The prefix on the object(s) to replicate.
* - ``endpoint:``
- | The location of the source deployment.
| If the location is not remote, use ``local``.
* - ``credentials:``
- The ``accesskey`` and ``secretKey`` or the ``sessionToken`` that grants access to the object(s).
For **filters**
.. list-table::
:widths: 25 75
:width: 100%
* - ``newerThan:``
- A string representing a length of time in ``#d#h#s`` format.
Only objects newer than the specified length of time replicate.
For example, ``7d``, ``24h``, ``5d12h30s`` are valid strings.
* - ``olderThan:``
- A string representing a length of time in ``#d#h#s`` format.
Only objects older than the specified length of time replicate.
* - ``createdAfter:``
- A date in ``YYYY-MM-DD`` format.
Only objects created after the date replicate.
* - ``createdBefore:``
- A date in ``YYYY-MM-DD`` format.
Only objects created prior to the date replicate.
For **notifications**
.. list-table::
:widths: 25 75
:width: 100%
* - ``endpoint:``
- The predefined endpoint to send events for notifications.
* - ``token:``
- An optional :abbr:`JWT <JSON Web Token>` to access the ``endpoint``.
For **retry attempts**
If something interrupts the job, you can define how many attempts to retry the job batch.
For each retry, you can also define how long to wait between attempts.
.. list-table::
:widths: 25 75
:width: 100%
* - ``attempts:``
- Number of tries to complete the batch job before giving up.
* - ``delay:``
- The least amount of time to wait between each attempt.
Sample YAML
+++++++++++
.. code-block:: yaml
replicate:
apiVersion: v1
# source of the objects to be replicated
source:
type: TYPE # valid values are "s3"
bucket: BUCKET
prefix: PREFIX
# NOTE: if source is remote then target must be "local"
# endpoint: ENDPOINT
# credentials:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# target where the objects must be replicated
target:
type: TYPE # valid values are "s3"
bucket: BUCKET
prefix: PREFIX
# NOTE: if target is remote then source must be "local"
# endpoint: ENDPOINT
# credentials:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# optional flags based filtering criteria
# for all source objects
flags:
filter:
newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
createdAfter: "date" # match objects created after "date"
createdBefore: "date" # match objects created before "date"
## NOTE: tags are not supported when "source" is remote.
# tags:
# - key: "name"
# value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
## NOTE: metadata filter not supported when "source" is non MinIO.
# metadata:
# - key: "content-type"
# value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
notify:
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
retry:
attempts: 10 # number of retries for the job before giving up
delay: "500ms" # least amount of delay between each retry

103
source/reference/minio-mc/mc-batch-list.rst Normal file
View File

@ -0,0 +1,103 @@
.. _minio-mc-batch-list:
=================
``mc batch list``
=================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
.. mc:: mc batch list
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
Syntax
------
.. start-mc-batch-list-desc
The :mc:`mc batch list` command outputs a list of the batch jobs currently in progress on a deployment.
.. end-mc-batch-list-desc
.. tab-set::
.. tab-item:: EXAMPLE
The following command outputs a list of all jobs currently in progress on the ``myminio`` alias.
.. code-block:: shell
:class: copyable
mc batch list myminio
.. tab-item:: SYNTAX
The command has the following syntax:
.. code-block:: shell
:class: copyable
mc [GLOBALFLAGS] batch list TARGET \
--type "string"
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-syntax
:end-before: end-minio-syntax
Parameters
~~~~~~~~~~
.. mc-cmd:: TARGET
:required:
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
.. mc-cmd:: --type
:optional:
List batch jobs only of a certain type.
Global Flags
~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-globals
:end-before: end-minio-mc-globals
Example
-------
List all ``replicate`` type batch jobs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following command starts the batch of job(s) defined in the file ``replication.yaml`` on the deployment at :mc:`alias <mc alias>` ``myminio``:
.. code-block:: shell
:class: copyable
mc batch list myminio --type "replicate"
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
- Replace ``replicate`` with the job type to output.
Currently, :mc:`mc batch` only supports the ``replicate`` job type.
The output of the above command is similar to the following:
.. code-block:: shell
ID TYPE USER STARTED
E24HH4nNMcgY5taynaPfxu replicate minioadmin 1 minute ago
S3 Compatibility
~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-s3-compatibility
:end-before: end-minio-mc-s3-compatibility

119
source/reference/minio-mc/mc-batch-start.rst Normal file
View File

@ -0,0 +1,119 @@
.. _minio-mc-batch-start:
==================
``mc batch start``
==================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
.. mc:: mc batch start
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
Syntax
------
.. start-mc-batch-start-desc
The :mc:`mc batch start` command launches a batch job from a job batch YAML file.
.. end-mc-batch-start-desc
The batch job runs to completion (or up to the number of retries specified in the file) one time.
To run the batch job again after completion, you must start it again.
.. tab-set::
.. tab-item:: EXAMPLE
The following command creates a basic YAML file for a replicate job on the ``mybucket`` bucket of the ``myminio`` alias.
.. code-block:: shell
:class: copyable
mc batch start myminio/mybucket jobfile.yaml
The output of the above command is something similar to:
.. code-block:: shell
Successfully start 'replicate' job `B34HHqnNMcg1taynaPfxu` on '2022-10-24 17:19:06.296974771 -0700 PDT'
.. tab-item:: SYNTAX
The command has the following syntax:
.. code-block:: shell
:class: copyable
mc [GLOBALFLAGS] batch start \
TARGET \
JOBFILE
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-syntax
:end-before: end-minio-syntax
Parameters
~~~~~~~~~~
.. mc-cmd:: TARGET
:required:
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
For example:
.. code-block:: none
mc batch start myminio/mydata replicate.yaml
You can use just an alias without specifying a bucket to perform the job at the deployment's root.
.. mc-cmd:: JOBFILE
:required:
A YAML-defined batch job.
The job may have as many tasks as desired; there is no predefined limit.
Global Flags
~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-globals
:end-before: end-minio-mc-globals
Example
-------
Start a Batch Job
~~~~~~~~~~~~~~~~~
The following command starts the batch of job(s) defined in the file ``replication.yaml`` on the deployment at :mc:`alias <mc alias>` ``myminio``:
.. code-block:: shell
:class: copyable
mc batch start myminio ./replication.yaml
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
- Replace ``./replication.yaml`` with the yaml-formatted file that describes the batch job.
Use the file path relative to your current location.
The output of the above command is similar to the following:
.. code-block:: shell
Successfully start 'replicate' job `E24HH4nNMcgY5taynaPfxu` on '2022-09-26 17:19:06.296974771 -0700 PDT'
S3 Compatibility
~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-s3-compatibility
:end-before: end-minio-mc-s3-compatibility

105
source/reference/minio-mc/mc-batch-status.rst Normal file
View File

@ -0,0 +1,105 @@
.. _minio-mc-batch-status:
===================
``mc batch status``
===================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
.. mc:: mc batch status
.. versionchanged:: MinIO RELEASE.2022-10-08T20-11-00Z or later
Syntax
------
.. start-mc-batch-status-desc
The :mc:`mc batch status` command outputs real-time summaries of job events on a MinIO server.
.. end-mc-batch-status-desc
.. tab-set::
.. tab-item:: EXAMPLE
The following command outputs a list of all jobs currently in progress on the ``myminio`` alias.
.. code-block:: shell
:class: copyable
mc batch status myminio KwSysDpxcBU9FNhGkn2dCf
.. tab-item:: SYNTAX
The command has the following syntax:
.. code-block:: shell
:class: copyable
mc [GLOBALFLAGS] batch list TARGET \
JOBID
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-syntax
:end-before: end-minio-syntax
Parameters
~~~~~~~~~~
.. mc-cmd:: TARGET
:required:
The :ref:`alias <alias>` and bucket on the MinIO deployment to which to add the object lifecycle management rule.
.. mc-cmd:: JOBID
:optional:
The unique identifier of a job to summarize.
To find the ID of a job, use :mc:`mc batch list`.
Global Flags
~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-globals
:end-before: end-minio-mc-globals
Example
-------
Summary the Events of a Replicate Job
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following command provides the real-time summary of an active job on the deployment at :mc:`alias <mc alias>` ``myminio``:
.. code-block:: shell
:class: copyable
mc batch status myminio KwSysDpxcBU9FNhGkn2dCf
- Replace ``myminio`` with the :mc:`alias <mc alias>` of the MinIO deployment that should run the job.
The output of the above command is similar to the following:
.. code-block:: shell
●∙∙
Objects: 28766
Versions: 28766
Throughput: 3.0 MiB/s
Transferred: 406 MiB
Elapsed: 2m14.227222868s
CurrObjName: share/doc/xml-core/examples/foo.xmlcatalogs
S3 Compatibility
~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-s3-compatibility
:end-before: end-minio-mc-s3-compatibility