minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-05-21 11:53:41 +03:00
Code
docs/source/administration/batch-framework.rst
Daryl White f37aee8416
Clean up of batch framework and mc batch docs (#749) 
- Removes bad copy paste info about ilm from batch command docs.
- Clarifies that batch files are one task only.
- Fixes source type from `s3` to `minio`.
- Cleans up indentation errors in replicate yaml example.
- Replicate task is local to remote only.

Closes #748
2023-03-06 16:12:44 -06:00

4.0 KiB
Raw Blame History

Batch Framework

minio

Table of Contents

MinIO RELEASE.2022-10-08T20-11-00Z

The Batch Framework was introduced with the replicate job type in the mc RELEASES.2022-10-08T20-11-00Z.

Overview

The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a "batch file"). The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the MinIO Client <minio-client>.

A batch file defines one job task.

Once started, MinIO starts processing the job. Time to completion depends on the resources available to the deployment.

If any portion of the job fails, MinIO retries the job up to the number of times defined in the job definition.

The MinIO Batch Framework supports the following job types:

Job Type Description
replicate Perform a one-time replication procedure from one MinIO location to another MinIO location.

MinIO Batch CLI

  • Install the MinIO Client <minio-client>
  • Define an alias <mc alias set> for the MinIO deployment

The mc batch commands include

mc batch generate
mc batch start
mc batch list
mc batch status
mc batch describe

Access to mc batch

You can use MinIO's Policy Based Access Control <minio-policy> and the administrative policy actions <minio-policy-mc-admin-actions> to restrict who can start a batch job, retrieve a list of running jobs, or describe a running job.

Job Types

Replicate

Use the replicate job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location. The definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects.

For example, you can use a batch job to perform a one-time replication sync of objects from minio-alpha/invoices/ to minio-baker/invoices.

The advantages of Batch Replication over mc mirror include:

  • Removes the client to cluster network as a potential bottleneck
  • A user only needs access to starting a batch job with no other permissions, as the job runs entirely server side on the cluster
  • The job provides for retry attempts in event that objects do not replicate
  • Batch jobs are one-time, curated processes allowing for fine control replication

RELEASE.2023-02-17T17-52-43Z

Run batch replication with multiple workers in parallel by specifying the MINIO_BATCH_REPLICATION_WORKERS environment variable.

Sample YAML Description File for a replicate Job Type

Create a basic replicate job definition file you can edit with mc batch generate.

/includes/code/replicate.yaml