minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-07-30 07:03:26 +03:00
Code Activity

Updates for MinIO RELEASE.2023-04-07T05-28-58Z (#823)

Browse Source 
- Adds new environment variable for ILM expiration workers
- Adds new metrics for locks
- Adds keyrotate batch job type
- Adds info about batch replicate from remote to local
    
Closes #806

Staged:
http://192.241.195.202:9000/staging/minio-2023-04-07/administration/batch-framework.html
This commit is contained in:
Daryl White
2023-05-01 16:42:13 -05:00
committed by GitHub
parent 101ad648dd
commit 363e688617
8 changed files with 198 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
source/administration/batch-framework.rst
View File

@ -40,6 +40,9 @@ The MinIO Batch Framework supports the following job types:
* - ``replicate``
- Perform a one-time replication procedure from one MinIO location to another MinIO location.
* - ``keyrotate``
- Perform a one-time process to cycle the :ref:`sse-s3 or sse-kms <minio-sse-data-encryption>` cryptographic keys on objects.
MinIO Batch CLI
---------------
@ -84,10 +87,16 @@ Job Types
Replicate
~~~~~~~~~
Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location.
Use the ``replicate`` job type to create a batch job that replicates objects from one MinIO deployment to another MinIO location.
At least one of the deployment locations, either the source or the target, must be ``local``.
The definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects.
For example, you can use a batch job to perform a one-time replication sync of objects from ``minio-alpha/invoices/`` to ``minio-baker/invoices``.
.. versionchanged:: MinIO RELEASE.2023-04-07T05-28-58Z
You can replicate from a remote MinIO deployment to the local deployment that runs the batch job.
For example, you can use a batch job to perform a one-time replication sync to push objects from a bucket on a local deployment at ``minio-local/invoices/`` to a bucket on a remote deployment at ``minio-remote/invoices``.
You can also pull objects from the remote deployment at ``minio-remote/invoices`` to the local deployment at ``minio-local/invoices``.
The advantages of Batch Replication over :mc:`mc mirror` include:
@ -106,4 +115,22 @@ Sample YAML Description File for a ``replicate`` Job Type
Create a basic ``replicate`` job definition file you can edit with :mc:`mc batch generate`.
.. literalinclude:: /includes/code/replicate.yaml
:language: yaml
:language: yaml
Key Rotate
~~~~~~~~~~
.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z
Use the ``keyrotate`` job type to create a batch job that cycles the :ref:`sse-s3 or sse-kms keys <minio-sse-data-encryption>` for encrypted objects.
The YAML configuration supports filters to restrict key rotation to a specific set of objects by creation date, tags, metadata, or kms key.
You can also define retry attempts or set a notification endpoint and token.
Sample YAML Description File for a ``keyrotate`` Job Type
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Create a basic ``keyrotate`` job definition file you can edit with :mc:`mc batch generate`.
.. literalinclude:: /includes/code/replicate.yaml
:language: yaml

37
source/includes/code/keyrotate.yaml Normal file
View File

@ -0,0 +1,37 @@
keyrotate:
apiVersion: v1
bucket: bucket
prefix:
encryption:
type: sse-kms # valid values are sse-s3 and sse-kms
# The following encryption values only apply for sse-kms type.
# For sse-s3 key types, MinIO uses the key provided by the MINIO_KMS_KES_KEY_FILE environment variable.
# The following two values are ignored if type is set to sse-s3.
key: my-new-keys2 # valid only for sse-kms
context: <new-kms-key-context> # valid only for sse-kms
# optional flags based filtering criteria
flags:
filter:
newerThan: "84h" # match objects newer than this value (e.g. 7d10h31s)
olderThan: "80h" # match objects older than this value (e.g. 7d10h31s)
createdAfter: "2023-03-02T15:04:05Z07:00" # match objects created after "date"
createdBefore: "2023-03-02T15:04:05Z07:00" # match objects created before "date"
tags:
- key: "name"
value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
metadata:
- key: "content-type"
value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
kmskey: "key-id" # match objects with KMS key-id (applicable only for sse-kms)
# optional entries to add notifications for the job
notify:
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
# optional entries to add retry attempts if the job is interrupted
retry:
attempts: 10 # number of retries for the job before giving up
delay: "500ms" # least amount of delay between each retry

15
source/operations/monitoring/metrics-and-alerts.rst
View File

@ -577,6 +577,21 @@ Software and Process Metrics
Resident memory size in bytes.
Lock Metrics
~~~~~~~~~~~~
.. metric:: minio_locks_total
Total number of current locks on the peer.
.. metric:: minio_locks_write_total
Number of current WRITE locks on the peer.
.. metric:: minio_locks_read_total
Number of current READ locks on the peer.
.. toctree::
:titlesonly:
:hidden:

2
source/operations/server-side-encryption.rst
View File

@ -1,3 +1,5 @@
.. _minio-sse-data-encryption:
=====================
Data Encryption (SSE)
=====================

2
source/reference/minio-mc-admin/mc-admin-rebalance.rst
View File

@ -28,7 +28,7 @@ Rebalancing redistributes objects across all pools in the deployment.
.. end-mc-admin-rebalance-desc
MinIO does not automatically rebalance objects when adding a new server pool.
Instead, MinIO ref:`writes new objects <minio-writing-files>` to the pool with relatively more free space compared to the other available pools on the deployment.
Instead, MinIO :ref:`writes new objects <minio-writing-files>` to the pool with relatively more free space compared to the other available pools on the deployment.
Triggering a manual rebalancing procedure prompts MinIO to scan the entire deployment and move objects as necessary to achieve a similar available free space across all pools.
This is an expensive and time consuming operation.

109
source/reference/minio-mc/mc-batch-generate.rst
View File

@ -74,7 +74,7 @@ Parameters
The type of job to generate a YAML document for.
Currently, :mc:`mc batch` only supports the ``replicate`` job type.
Currently, :mc:`mc batch` supports the ``replicate`` and ``keyrotate`` job types.
Global Flags
@ -101,7 +101,7 @@ The following command generates a YAML blueprint for a replicate type batch job
- Replace ``replicate`` with the type of job to generate a yaml file for.
At the time of release, :mc:``mc batch`` only supports the ``replicate`` job type.
:mc:``mc batch`` supports the ``replicate`` and ``keyrotate`` job types.
S3 Compatibility
@ -123,7 +123,11 @@ Job Types
Replicate objects between two MinIO deployments.
Provides similar functionality to :ref:`bucket replication <minio-bucket-replication>` as a batch job rather than continual scanning function.
MinIO may add more job types in the future.
- ``keyrotate``
.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z
Rotate the sse-s3 or sse-kms keys for objects at rest on a MinIO deployment.
``replicate``
~~~~~~~~~~~~~
@ -131,8 +135,13 @@ MinIO may add more job types in the future.
Use the ``replicate`` job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location.
The YAML **must** define the source and target deployments.
If the _source_ deployment is remote, then the _target_ deployment **must** be ``local``.
Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.
.. versionchanged:: MinIO RELEASE.2023-04-07T05-28-58Z
You can replicate from a remote MinIO deployment to the local deployment that runs the batch job.
For the **source deployment**
- Required information
@ -156,7 +165,8 @@ For the **source deployment**
- The prefix on the object(s) that should replicate.
* - ``endpoint:``
- | Location of the source deployment, must be ``local``.
- | Location of the source deployment.
| If the location is not remote, use ``local``.
* - ``credentials:``
- The ``accesskey:`` and ``secretKey:`` or the ``sessionToken:`` that grants access to the object(s).
@ -186,6 +196,7 @@ For the **target deployment**
* - ``endpoint:``
- | The location of the source deployment.
| If the location is not remote, use ``local``.
| If the location of the source is remote, the source for target **must** be ``local``.
* - ``credentials:``
- The ``accesskey`` and ``secretKey`` or the ``sessionToken`` that grants access to the object(s).
@ -245,3 +256,93 @@ Sample YAML
.. literalinclude:: /includes/code/replicate.yaml
:language: yaml
``keyrotate``
~~~~~~~~~~~~~
.. versionadded:: MinIO RELEASE.2023-04-07T05-28-58Z
Use the ``keyrotate`` job type to create a batch job that cycles the :ref:`sse-s3 or sse-kms keys <minio-sse-data-encryption>` for encrypted objects.
Required information
++++++++++++++++++++
.. list-table::
:widths: 25 75
:width: 100%
* - ``type:``
- Either ``sse-s3`` or ``sse-kms``.
* - ``key:``
- Only for use with the ``sse-kms`` type.
The key to use to unseal the key vault.
* - ``context:``
- Only for use with the ``sse-kms`` type.
The context within which to perform actions.
Optional information
++++++++++++++++++++
For **flag based filters**
.. list-table::
:widths: 25 75
:width: 100%
* - ``newerThan:``
- A string representing a length of time in ``#d#h#s`` format.
Keys rotate only for objects newer than the specified length of time.
For example, ``7d``, ``24h``, ``5d12h30s`` are valid strings.
* - ``olderThan:``
- A string representing a length of time in ``#d#h#s`` format.
Keys rotate only for objects older than the specified length of time.
* - ``createdAfter:``
- A date in ``YYYY-MM-DD`` format.
Keys rotate only for objects created after the date.
* - ``createdBefore:``
- A date in ``YYYY-MM-DD`` format.
Keys rotate only for objects created prior to the date.
* - ``tags:``
- Rotate keys only for objects with tags that match the specified ``key:`` and ``value:``.
* - ``metadtaa:``
- Rotate keys only for objects with metadata that match the specified ``key:`` and ``value:``.
* - ``kmskey:``
- Rotate keys only for objects with a KMS key-id that match the specified value.
This is only applicable for the ``sse-kms`` type.
For **notifications**
.. list-table::
:widths: 25 75
:width: 100%
* - ``endpoint:``
- The predefined endpoint to send events for notifications.
* - ``token:``
- An optional :abbr:`JWT <JSON Web Token>` to access the ``endpoint``.
For **retry attempts**
If something interrupts the job, you can define a maximum number of retry attempts.
For each retry, you can also define how long to wait between attempts.
.. list-table::
:widths: 25 75
:width: 100%
* - ``attempts:``
- Number of tries to complete the batch job before giving up.
* - ``delay:``
- The amount of time to wait between each attempt.
Sample YAML
+++++++++++
.. literalinclude:: /includes/code/keyrotate.yaml
:language: yaml

6
source/reference/minio-mc/mc-mv.rst
View File

@ -222,7 +222,7 @@ Parameters
Recursively move the contents of each bucket or directory
:mc-cmd:`~mc mv SOURCE` to the :mc-cmd:`~mc mv TARGET` bucket.
.. mc-cmd:: --storage-class, sc
.. mc-cmd:: --storage-class
:optional:
Set the storage class for the new object(s) on the
@ -313,7 +313,7 @@ Move Bucket Between S3-Compatible Services
Move File to S3-Compatible Host with Specific Storage Class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use :mc:`mc mv` with the :mc-cmd:`~mc mv storage-class` option to set
Use :mc:`mc mv` with the :mc-cmd:`~mc mv --storage-class` option to set
the storage class on the destination S3-compatible host.
.. code-block:: shell
@ -321,7 +321,7 @@ the storage class on the destination S3-compatible host.
mc mv --storage-class CLASS FILEPATH ALIAS/PATH
- Replace :mc-cmd:`CLASS <mc mv storage-class>` with the storage class to
- Replace :mc-cmd:`CLASS <mc mv --storage-class>` with the storage class to
associate to the files.
- Replace :mc-cmd:`FILEPATH <mc mv SOURCE>` with the full file path to the

5
source/reference/minio-server/minio-server.rst
View File

@ -222,6 +222,11 @@ Core Configuration
Specifies the full path to the file the MinIO server process uses for loading environment variables.
.. envvar:: MINIO_ILM_EXPIRY_WORKERS
Specifies the number of workers to make available to expire objects configured with ILM rules for expiration.
When not set, MinIO defaults to using up to half of the available processing cores available.
Root Credentials
~~~~~~~~~~~~~~~~