minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-07-30 07:03:26 +03:00
Code Activity

Doc updates for mc RELEASE.2024-01-05T05-04-32Z (#1112)

Browse Source 
Doc updates for these `mc` changes:

- New option for `mc mirror`: `--summary`
- Clarify `mc sql --json-input` also works with `.ndjson`

NDJSON clarified with a discussion in `mc sql` and an example of setting
`content-type` in `mc cp`.

Staged:

http://192.241.195.202:9000/staging/DOCS-1104/linux/reference/minio-mc/mc-mirror.html#mc.mirror.-summary


http://192.241.195.202:9000/staging/DOCS-1104/linux/reference/minio-mc/mc-sql.html#mc.sql.-json-input

http://192.241.195.202:9000/staging/DOCS-1104/linux/reference/minio-mc/mc-sql.html#input-formats

http://192.241.195.202:9000/staging/DOCS-1104/linux/reference/minio-mc/mc-cp.html#add-a-content-type-value

Fixes https://github.com/minio/docs/issues/1104
This commit is contained in:
Andrea Longo
2024-01-23 08:58:14 -07:00
committed by GitHub
parent 6f00040411
commit 2d00cf06e5
3 changed files with 181 additions and 161 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
source/reference/minio-mc/mc-cp.rst
View File

@ -163,7 +163,7 @@ Parameters
Enclose the entire list of key-value pairs passed to :mc-cmd:`~mc cp --encrypt` in double-quotes ``"``. Enclose the entire list of key-value pairs passed to :mc-cmd:`~mc cp --encrypt` in double-quotes ``"``.
:mc-cmd:`~mc cp --encrypt` can use the ``MC_ENCRYPT`` environment variable for retrieving a list of encryption key-value pairs as an alternative to specifying them on the command line. :mc-cmd:`~mc cp --encrypt` can use the :envvar:`MC_ENCRYPT` environment variable for retrieving a list of encryption key-value pairs as an alternative to specifying them on the command line.
.. mc-cmd:: --encrypt-key .. mc-cmd:: --encrypt-key
:optional: :optional:
@ -178,7 +178,7 @@ Parameters
Enclose the entire list of key-value pairs passed to Enclose the entire list of key-value pairs passed to
:mc-cmd:`~mc cp --encrypt-key` in double quotes ``"``. :mc-cmd:`~mc cp --encrypt-key` in double quotes ``"``.
:mc-cmd:`~mc cp --encrypt-key` can use the ``MC_ENCRYPT_KEY`` :mc-cmd:`~mc cp --encrypt-key` can use the :envvar:`MC_ENCRYPT_KEY`
environment variable for retrieving a list of encryption key-value pairs environment variable for retrieving a list of encryption key-value pairs
as an alternative to specifying them on the command line. as an alternative to specifying them on the command line.
@ -449,6 +449,35 @@ command only applies to S3-to-S3 copy.
:start-after: start-versioning-admonition :start-after: start-versioning-admonition
:end-before: end-versioning-admonition :end-before: end-versioning-admonition
Add a ``content-type`` Value
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use :mc-cmd:`mc cp --attr` to add a ``content-type`` value.
This command only applies to S3-to-S3 copy.
.. code-block:: shell
:class: copyable
mc cp --attr="content-type=CONTENT-TYPE" SRCALIAS/SRCPATH TGTALIAS/TGTPATH
- Replace ``CONTENT-TYPE`` with the desired content type (also called a `media type <https://www.iana.org/assignments/media-types/media-types.xhtml>`__).
- Replace :mc-cmd:`SRCALIAS <mc cp SOURCE>` with the :mc:`alias <mc alias>` of a source S3-compatible host.
- Replace :mc-cmd:`SRCPATH <mc cp SOURCE>` with the path to the object on the source S3-compatible host.
- Replace :mc-cmd:`TGTALIAS <mc cp TARGET>` with the :mc:`alias <mc alias>` of a target S3-compatible host.
- Replace :mc-cmd:`TGTPATH <mc cp TARGET>` with the path to the object on the target S3-compatible host.
Omit the object name to use the ``SRCPATH`` object name.
The following example sets a ``content-type`` of ``application/json``:
.. code-block::
:class: copyable
mc cp data.ndjson --attr="content-type=application/json" myminio/mybucket
Behavior Behavior
-------- --------

94
source/reference/minio-mc/mc-mirror.rst
View File

@ -21,7 +21,7 @@ The :mc:`mc mirror` command synchronizes content to MinIO deployment, similar to
.. end-mc-mirror-desc .. end-mc-mirror-desc
.. note:: .. note::
:mc:`mc mirror` only synchronizes the current object without any version information or metadata. :mc:`mc mirror` only synchronizes the current object without any version information or metadata.
To synchronize an object's version history and metadata, consider using :mc:`mc replicate` for :ref:`bucket replication <minio-bucket-replication-serverside>` or :mc:`mc admin replicate` for :ref:`site replication <minio-site-replication-overview>`. To synchronize an object's version history and metadata, consider using :mc:`mc replicate` for :ref:`bucket replication <minio-bucket-replication-serverside>` or :mc:`mc admin replicate` for :ref:`site replication <minio-site-replication-overview>`.
@ -38,7 +38,7 @@ The :mc:`mc mirror` command synchronizes content to MinIO deployment, similar to
mc mirror --watch ~/mydata myminio/mydata mc mirror --watch ~/mydata myminio/mydata
The command "watches" for files added or removed on the local filesystem and synchronizes those operations to MinIO until explicitly terminated. The command "watches" for files added or removed on the local filesystem and synchronizes those operations to MinIO until explicitly terminated.
:mc-cmd:`mc mirror --watch` updates files changed on the local filesystem to MinIO (see :mc-cmd:`~mc mirror --overwrite`). :mc-cmd:`mc mirror --watch` updates files changed on the local filesystem to MinIO (see :mc-cmd:`~mc mirror --overwrite`).
``--watch`` does not remove other files from MinIO not present on the local filesystem (see :mc-cmd:`~mc mirror --remove`). ``--watch`` does not remove other files from MinIO not present on the local filesystem (see :mc-cmd:`~mc mirror --remove`).
@ -67,7 +67,7 @@ The :mc:`mc mirror` command synchronizes content to MinIO deployment, similar to
[--remove] \ [--remove] \
[--storage-class "string"] \ [--storage-class "string"] \
[--watch] \ [--watch] \
SOURCE \ SOURCE \
TARGET TARGET
.. include:: /includes/common-minio-mc.rst .. include:: /includes/common-minio-mc.rst
@ -78,8 +78,9 @@ Parameters
~~~~~~~~~~ ~~~~~~~~~~
.. mc-cmd:: SOURCE .. mc-cmd:: SOURCE
:required:
*REQUIRED* The file(s) or object(s) to synchronize to the :mc-cmd:`~mc mirror TARGET` S3 host. The file(s) or object(s) to synchronize to the :mc-cmd:`~mc mirror TARGET` S3 host.
For objects on S3-compatible hosts, specify the path to the object as ``ALIAS/PATH``, where: For objects on S3-compatible hosts, specify the path to the object as ``ALIAS/PATH``, where:
@ -100,8 +101,9 @@ Parameters
If specifying a directory, :mc:`mc mirror` synchronizes all files in the directory. If specifying a directory, :mc:`mc mirror` synchronizes all files in the directory.
.. mc-cmd:: TARGET .. mc-cmd:: TARGET
:required:
*REQUIRED* The full path to bucket to which :mc:`mc mirror` synchronizes SOURCE objects. Specify the ``TARGET`` as ``ALIAS/PATH``, where: The full path to bucket to which :mc:`mc mirror` synchronizes SOURCE objects. Specify the ``TARGET`` as ``ALIAS/PATH``, where:
- ``ALIAS`` is the :mc:`alias <mc alias>` of a configured S3-compatible host, *and* - ``ALIAS`` is the :mc:`alias <mc alias>` of a configured S3-compatible host, *and*
@ -114,23 +116,29 @@ Parameters
:mc:`mc mirror` uses the object or file names from the :mc-cmd:`~mc mirror SOURCE` when synchronizing to the ``TARGET`` bucket. :mc:`mc mirror` uses the object or file names from the :mc-cmd:`~mc mirror SOURCE` when synchronizing to the ``TARGET`` bucket.
.. mc-cmd:: --attr .. mc-cmd:: --attr
:optional:
Add custom metadata for mirrored objects. Specify key-value pairs as ``KEY=VALUE\;``. Add custom metadata for mirrored objects. Specify key-value pairs as ``KEY=VALUE\;``.
For example, ``--attr key1=value1\;key2=value2\;key3=value3``. For example, ``--attr key1=value1\;key2=value2\;key3=value3``.
.. mc-cmd:: --disable-multipart .. mc-cmd:: --disable-multipart
:optional:
Disables multipart upload for the synchronization session. Disables multipart upload for the synchronization session.
.. mc-cmd:: --dry-run
:optional:
Perform a mock mirror operation.
Use this operation to test that the :mc:`mc mirror` operation will only mirror the desired objects or buckets.
.. mc-cmd:: --encrypt-key .. mc-cmd:: --encrypt-key
:optional:
Encrypt or decrypt objects using server-side encryption with client-specified keys. Encrypt or decrypt objects using server-side encryption with client-specified keys.
Specify key-value pairs as ``KEY=VALUE``. Specify key-value pairs as ``KEY=VALUE``.
- Each ``KEY`` represents a bucket or object. - Each ``KEY`` represents a bucket or object.
- Each ``VALUE`` represents the data key to use for encrypting object(s). - Each ``VALUE`` represents the data key to use for encrypting object(s).
Enclose the entire list of key-value pairs passed to :mc-cmd:``~mc mirror --encrypt-key`` in double quotes ``"``. Enclose the entire list of key-value pairs passed to :mc-cmd:``~mc mirror --encrypt-key`` in double quotes ``"``.
@ -140,24 +148,18 @@ Parameters
You can only delete encrypted objects if you specify the correct :mc-cmd:`~mc mirror --encrypt-key` secret key. You can only delete encrypted objects if you specify the correct :mc-cmd:`~mc mirror --encrypt-key` secret key.
.. mc-cmd:: --exclude .. mc-cmd:: --exclude
:optional:
Exclude object(s) in the :mc-cmd:`~mc mirror SOURCE` path that match the specified object name pattern. Exclude object(s) in the :mc-cmd:`~mc mirror SOURCE` path that match the specified object name pattern.
.. mc-cmd:: --exclude-storageclass .. mc-cmd:: --exclude-storageclass
:optional:
Exclude object(s) on the :mc-cmd:`~mc mirror SOURCE` that have the specified storage class. Exclude object(s) on the :mc-cmd:`~mc mirror SOURCE` that have the specified storage class.
You can use this flag multiple times in a command to exclude objects from more than one storage class. You can use this flag multiple times in a command to exclude objects from more than one storage class.
Use this to exclude objects with storage classes that require rehydration or restoration of objects, such as migrating from an AWS S3 bucket where some objects have the ``GLACIER`` or ``DEEP_ARCHIVE`` storage classes. Use this to exclude objects with storage classes that require rehydration or restoration of objects, such as migrating from an AWS S3 bucket where some objects have the ``GLACIER`` or ``DEEP_ARCHIVE`` storage classes.
.. mc-cmd:: --dry-run
Perform a mock mirror operation.
Use this operation to test that the :mc:`mc mirror` operation will only mirror the desired objects or buckets.
.. --limit-download and --limit-upload included here .. --limit-download and --limit-upload included here
.. include:: /includes/linux/minio-client.rst .. include:: /includes/linux/minio-client.rst
@ -165,36 +167,36 @@ Parameters
:end-before: end-mc-limit-flags-desc :end-before: end-mc-limit-flags-desc
.. mc-cmd:: --md5 .. mc-cmd:: --md5
:optional:
Forces all uploads to calculate MD5 checksums. Forces all uploads to calculate MD5 checksums.
.. mc-cmd:: --monitoring-address .. mc-cmd:: --monitoring-address
:optional:
Creates a `Prometheus <https://prometheus.io/>`__ endpoint for monitoring mirroring activity. Creates a `Prometheus <https://prometheus.io/>`__ endpoint for monitoring mirroring activity.
Specify the local network adapter and port address on which to create the scraping endpoint. Specify the local network adapter and port address on which to create the scraping endpoint.
Defaults to ``localhost:8081``). Defaults to ``localhost:8081``).
.. mc-cmd:: --newer-than .. mc-cmd:: --newer-than
:optional:
Mirror object(s) newer than the specified number of days. Mirror object(s) newer than the specified number of days.
Specify a string in ``#d#hh#mm#ss`` format. Specify a string in ``#d#hh#mm#ss`` format
For example: ``--newer-than 1d2hh3mm4ss``. For example: ``--newer-than 1d2hh3mm4ss``.
.. mc-cmd:: --older-than .. mc-cmd:: --older-than
:optional:
Mirror object(s) older than the specified time limit. Mirror object(s) older than the specified time limit.
Specify a string in ``#d#hh#mm#ss`` format. Specify a string in ``#d#hh#mm#ss`` format.
For example: ``--older-than 1d2hh3mm4ss``. For example: ``--older-than 1d2hh3mm4ss``.
Defaults to ``0`` (all objects). Defaults to ``0`` (all objects).
.. mc-cmd:: --overwrite .. mc-cmd:: --overwrite
:optional:
Overwrites object(s) on the :mc-cmd:`~mc mirror TARGET`. Overwrites object(s) on the :mc-cmd:`~mc mirror TARGET`.
For example, consider an active ``mc mirror --overwrite`` synchronizing content from Source to Destination. For example, consider an active ``mc mirror --overwrite`` synchronizing content from Source to Destination.
@ -205,21 +207,21 @@ Parameters
``mc mirror`` logs an error and continues to synchronize other objects. ``mc mirror`` logs an error and continues to synchronize other objects.
.. mc-cmd:: --preserve, a .. mc-cmd:: --preserve, a
:optional:
Preserve file system attributes and bucket policy rules of the :mc-cmd:`~mc mirror SOURCE` on the :mc-cmd:`~mc mirror TARGET`. Preserve file system attributes and bucket policy rules of the :mc-cmd:`~mc mirror SOURCE` on the :mc-cmd:`~mc mirror TARGET`.
.. mc-cmd:: --region .. mc-cmd:: --region
:optional:
Specify the ``string`` region when creating new bucket(s) on the target. Specify the ``string`` region when creating new bucket(s) on the target.
Defaults to ``"us-east-1"``. Defaults to ``"us-east-1"``.
.. mc-cmd:: --remove .. mc-cmd:: --remove
:optional:
Removes object(s) on the Target that do not exist on the Source. Removes object(s) on the Target that do not exist on the Source.
Use the ``--remove`` flag to have the same list of objects on both Source and Target. Use the ``--remove`` flag to have the same list of objects on both Source and Target.
@ -241,18 +243,24 @@ Parameters
In prior versions, specifying ``/path/to/directory`` would result in the removal of the ``/path/to`` folder if ``directory`` did not exist. In prior versions, specifying ``/path/to/directory`` would result in the removal of the ``/path/to`` folder if ``directory`` did not exist.
.. mc-cmd:: --retry .. mc-cmd:: --retry
:optional:
In case of errors during mirror process, retry on each errored object. In case of errors during mirror process, retry on each errored object.
.. mc-cmd:: storage-class, sc .. mc-cmd:: --storage-class, sc
:optional:
Set the storage class for the new object(s) on the :mc-cmd:`~mc mirror TARGET`. Set the storage class for the new object(s) on the :mc-cmd:`~mc mirror TARGET`.
See the Amazon documentation on :aws-docs:`Storage Classes <AmazonS3/latest/dev/storage-class-intro.html>` for more information on S3 storage classses. See the Amazon documentation on :aws-docs:`Storage Classes <AmazonS3/latest/dev/storage-class-intro.html>` for more information on S3 storage classses.
.. mc-cmd:: --summary
:optional:
On completion, output a summary of the data that was synchronized.
.. mc-cmd:: --watch, w .. mc-cmd:: --watch, w
:optional:
Use ``--watch`` flag to mirror objects from Source to Target, where the Target may also have additional objects not present on the Source. Use ``--watch`` flag to mirror objects from Source to Target, where the Target may also have additional objects not present on the Source.
@ -264,13 +272,13 @@ Parameters
For example, object A and B exist on the watched Source. For example, object A and B exist on the watched Source.
Objects A, B, and C exist on the watched Target. Objects A, B, and C exist on the watched Target.
A client writes object D to Source and removes object B. A client writes object D to Source and removes object B.
After the operation, objects A and D exist on the Source. After the operation, objects A and D exist on the Source.
Objects A, C, and D exist on the Target. Objects A, C, and D exist on the Target.
Global Flags Global Flags
~~~~~~~~~~~~ ~~~~~~~~~~~~
@ -338,7 +346,7 @@ Use :mc:`mc mirror` with :mc-cmd:`~mc mirror --exclude-storageclass` to mirror o
.. code-block:: .. code-block::
:class: copyable :class: copyable
mc mirror --exclude-storageclass GLACIER \ mc mirror --exclude-storageclass GLACIER \
--exclude-storageclass DEEP_ARCHIVE SRCALIAS/SRCPATH TGALIAS/TGPATH --exclude-storageclass DEEP_ARCHIVE SRCALIAS/SRCPATH TGALIAS/TGPATH

215
source/reference/minio-mc/mc-sql.rst
View File

@ -15,21 +15,17 @@ Syntax
.. start-mc-sql-desc .. start-mc-sql-desc
The :mc:`mc sql` command provides an S3 Select interface for performing sql The :mc:`mc sql` command provides an S3 Select interface for performing sql queries on objects in the specified MinIO deployment.
queries on objects in the specified MinIO deployment.
.. end-mc-sql-desc .. end-mc-sql-desc
See :s3-docs:`Selecting content from objects See :s3-docs:`Selecting content from objects <selecting-content-from-objects>` for more information on S3 Select behavior and limitations.
<selecting-content-from-objects>` for more information on S3 Select behavior
and limitations.
.. tab-set:: .. tab-set::
.. tab-item:: EXAMPLE .. tab-item:: EXAMPLE
The following command queries all objects in the The following command queries all objects in the ``mydata`` bucket on the ``myminio`` MinIO deployment:
``mydata`` bucket on the ``myminio`` MinIO deployment:
.. code-block:: shell .. code-block:: shell
:class: copyable :class: copyable
@ -63,36 +59,36 @@ Parameters
~~~~~~~~~~ ~~~~~~~~~~
.. mc-cmd:: ALIAS .. mc-cmd:: ALIAS
:required:
*Required* The full path to the bucket or object to run the SQL query The full path to the bucket or object to run the SQL query against.
against. Specify the :ref:`alias <alias>` of a configured Specify the :ref:`alias <alias>` of a configured S3 service as the prefix to the ``ALIAS`` path.
S3 service as the prefix to the ``ALIAS`` path. For example: For example:
.. code-block:: shell .. code-block:: shell
mc sql [FLAGS] play/mybucket mc sql [FLAGS] play/mybucket
.. mc-cmd:: --query, e .. mc-cmd:: --query, e
:required:
*Required* The SQL statement to execute on the specified The SQL statement to execute on the specified :mc-cmd:`~mc sql ALIAS` directory or object.
:mc-cmd:`~mc sql ALIAS` directory or object. Wrap the entire SQL query in Wrap the entire SQL query in double quotes ``"``.
double quotes ``"``.
Defaults to ``"select * from S3Object"``. Defaults to ``"select * from S3Object"``.
.. mc-cmd:: --csv-input .. mc-cmd:: --csv-input
:optional:
*Optional* The data format for ``.csv`` input objects. Specify a string of The data format for ``.csv`` input objects.
comma-seperated ``key=value,...`` pairs. See :ref:`mc-sql-csv-format` for Specify a string of comma-seperated ``key=value,...`` pairs.
more information on valid keys. See :ref:`mc-sql-csv-format` for more information on valid keys.
.. mc-cmd:: --compression .. mc-cmd:: --compression
:optional:
*Optional* The compression type of the input object. Specify one of the The compression type of the input object.
following supported values: Specify one of the following supported values:
- ``GZIP`` - ``GZIP``
- ``BZIP2`` - ``BZIP2``
@ -106,71 +102,62 @@ Parameters
- ``SNAPPY`` `Snappy <http://google.github.io/snappy/>`__ framed stream - ``SNAPPY`` `Snappy <http://google.github.io/snappy/>`__ framed stream
.. mc-cmd:: --csv-output .. mc-cmd:: --csv-output
:optional:
*Optional* The data format for ``.csv`` output. Specify a string of The data format for ``.csv`` output.
comma-seperated ``key=value,...`` pairs. See :ref:`mc-sql-csv-format` for Specify a string of comma-seperated ``key=value,...`` pairs.
more information on valid keys. See :ref:`mc-sql-csv-format` for more information on valid keys.
See the S3 API :s3-api:`CSVOutput <API_CSVOutput.html>` for more See the S3 API :s3-api:`CSVOutput <API_CSVOutput.html>` for more information.
information.
.. mc-cmd:: --csv-output-header .. mc-cmd:: --csv-output-header
:optional:
*Optional* The header row of the ``.csv`` output file. Specify a string of The header row of the ``.csv`` output file.
comma-separated fields as ``field1,field2,...``. Specify a string of comma-separated fields as ``field1,field2,...``.
Omit to output a ``.csv`` with no header row. Omit to output a ``.csv`` with no header row.
.. mc-cmd:: --encrypt-key .. mc-cmd:: --encrypt-key
:optional:
*Optional* The encryption key to use for performing Server-Side Encryption The encryption key to use for performing Server-Side Encryption with Client Keys (SSE-C).
with Client Keys (SSE-C). Specify comma seperated key-value pairs as Specify comma seperated key-value pairs as ``KEY=VALUE,...``.
``KEY=VALUE,...``.
- For ``KEY``, specify the MinIO deployment
:mc-cmd:`alias <mc alias>` and full path to the bucket, including any
bucket prefixes. Separate the alias and bucket path with a forward slash
``\``. For example, ``play/mybucket``
- For ``VALUE``, specify the data key to use for encryption object(s) in - For ``KEY``, specify the MinIO deployment :mc-cmd:`alias <mc alias>` and full path to the bucket, including any bucket prefixes.
the bucket or bucket prefix specified to ``KEY``. Separate the alias and bucket path with a forward slash ``\``.
For example, ``play/mybucket``.
:mc-cmd:`~mc sql --encrypt-key` can use the ``MC_ENCRYPT_KEY`` - For ``VALUE``, specify the data key to use for encryption object(s) in the bucket or bucket prefix specified to ``KEY``.
environment variable for populating the list of encryption key-value
pairs as an alternative to specifying them on the command line. :mc-cmd:`~mc sql --encrypt-key` can use the :envvar:`MC_ENCRYPT_KEY` environment variable for populating the list of encryption key-value pairs as an alternative to specifying them on the command line.
.. mc-cmd:: --json-input .. mc-cmd:: --json-input
:optional:
*Optional* The data format for ``.json`` input objects. Specify the type of The data format for ``.json`` or ``.ndjson`` input objects.
the JSON contents as ``type=<VALUE>``. The value can be either: Specify the type of the JSON contents as ``type=<VALUE>``.
The value can be either:
- ``DOCUMENT`` - JSON `document <https://www.json.org/json-en.html>`__. - ``DOCUMENT`` - JSON `document <https://www.json.org/json-en.html>`__.
- ``LINES`` - JSON `lines <http://jsonlines.org/>`__. - ``LINES`` - JSON `lines <http://jsonlines.org/>`__.
See the S3 API :s3-api:`JSONInput <API_JSONInput.html>` for more See the S3 API :s3-api:`JSONInput <API_JSONInput.html>` for more information.
information.
.. mc-cmd:: --json-output .. mc-cmd:: --json-output
:optional:
*Optional* The data format for the ``.json`` output. Supports the The data format for the ``.json`` output.
``rd=value`` key, where ``rd`` is the ``RecordDelimiter`` for the JSON Supports the ``rd=value`` key, where ``rd`` is the ``RecordDelimiter`` for the JSON document.
document.
Omit to use the default newline character ``\n``. Omit to use the default newline character ``\n``.
See the S3 API :s3-api:`JSONOutput <API_JSONOutput.html>` for more See the S3 API :s3-api:`JSONOutput <API_JSONOutput.html>` for more information.
information.
.. mc-cmd:: --recursive, r .. mc-cmd:: --recursive, r
:optional:
*Optional* Recursively searches the specified :mc-cmd:`~mc sql ALIAS` Recursively searches the specified :mc-cmd:`~mc sql ALIAS` directory using the :mc-cmd:`~mc sql --query` SQL statement.
directory using the :mc-cmd:`~mc sql --query` SQL statement.
Global Flags Global Flags
~~~~~~~~~~~~ ~~~~~~~~~~~~
@ -185,36 +172,29 @@ Examples
Select all Columns in all Objects in a Bucket Select all Columns in all Objects in a Bucket
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use :mc:`mc sql` with the :mc-cmd:`~mc sql --recursive` and Use :mc:`mc sql` with the :mc-cmd:`~mc sql --recursive` and :mc-cmd:`~mc sql --query` options to apply the query to all objects in a bucket:
:mc-cmd:`~mc sql --query` options to apply the query to all objects
in a bucket:
.. code-block:: shell .. code-block:: shell
:class: copyable :class: copyable
mc sql --recursive --query "select * from S3Object" ALIAS/PATH mc sql --recursive --query "select * from S3Object" ALIAS/PATH
- Replace :mc-cmd:`ALIAS <mc sql ALIAS>` with the - Replace :mc-cmd:`ALIAS <mc sql ALIAS>` with the :ref:`alias <alias>` of the MinIO deployment.
:ref:`alias <alias>` of the MinIO deployment.
- Replace :mc-cmd:`PATH <mc sql ALIAS>` with the path to the bucket - Replace :mc-cmd:`PATH <mc sql ALIAS>` with the path to the bucket on the MinIO deployment.
on the MinIO deployment.
Run an Aggregation Query on an Object Run an Aggregation Query on an Object
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use :mc:`mc sql` with the :mc-cmd:`~mc sql --query` option to query Use :mc:`mc sql` with the :mc-cmd:`~mc sql --query` option to query an object on an MinIO deployment:
an object on an MinIO deployment:
.. code-block:: shell .. code-block:: shell
mc sql --query "select count(s.power) from S3Object" ALIAS/PATH mc sql --query "select count(s.power) from S3Object" ALIAS/PATH
- Replace :mc-cmd:`ALIAS <mc sql ALIAS>` with the - Replace :mc-cmd:`ALIAS <mc sql ALIAS>` with the :ref:`alias <alias>` of the MinIO deployment.
:ref:`alias <alias>` of the MinIO deployment.
- Replace :mc-cmd:`PATH <mc sql ALIAS>` with the path to the object - Replace :mc-cmd:`PATH <mc sql ALIAS>` with the path to the object on the MinIO deployment.
on the MinIO deployment.
Behavior Behavior
-------- --------
@ -224,32 +204,44 @@ Input Formats
:mc:`mc sql` supports the following input formats: :mc:`mc sql` supports the following input formats:
- ``.csv`` .. list-table:: Input Format Types
- ``.json`` :header-rows: 1
- ``.parquet``
* - Type
- ``content-type`` Value
For ``.csv`` file types, use :mc-cmd:`mc sql --csv-input` to * - ``.csv``
specify the CSV data format. See :ref:`mc-sql-csv-format` for more - ``text/csv``
information on CSV formatting fields.
For ``.json`` file types, use :mc-cmd:`mc sql --json-input` to specify * - ``.json``
the JSON data format. - ``application/json``
For ``.parquet`` file types, :mc:`mc sql` automatically interprets the * - ``.parquet``
data format. - none
For ``.csv`` file types, use :mc-cmd:`mc sql --csv-input` to specify the CSV data format.
See :ref:`mc-sql-csv-format` for more information on CSV formatting fields.
For ``.json`` file types, use :mc-cmd:`mc sql --json-input` to specify the JSON data format.
For ``.parquet`` file types, :mc:`mc sql` automatically interprets the data format.
:mc:`mc sql` determines the type by the file extension of the target object.
For example, an object named ``data.json`` is interpreted as a JSON file.
You can query data of a supported type but a different extension if the object has the appropriate ``content-type``.
For more information, see :mc-cmd:`mc cp --attr`.
.. _mc-sql-csv-format: .. _mc-sql-csv-format:
CSV Formatting Fields CSV Formatting Fields
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
The following table lists valid key-value pairs for use with The following table lists valid key-value pairs for use with :mc-cmd:`mc sql --csv-input` and :mc-cmd:`mc sql --csv-output`.
:mc-cmd:`mc sql --csv-input` and :mc-cmd:`mc sql --csv-output`. Certain key pairs are only valid for :mc-cmd:`~mc sql --csv-input`.
Certain key pairs are only valid for :mc-cmd:`~mc sql --csv-input` See the documentation for S3 API :s3-api:`CSVInput <API_CSVInput.html>` for more information on S3 CSV formatting.
See the documentation for S3 API :s3-api:`CSVInput <API_CSVInput.html>` for more
information on S3 CSV formatting.
.. list-table:: .. list-table::
:header-rows: 1 :header-rows: 1
:widths: 20 20 60 :widths: 20 20 60
:width: 100% :width: 100%
@ -260,71 +252,62 @@ information on S3 CSV formatting.
* - ``rd`` * - ``rd``
- -
- The character that seperates each record (row) in the input ``.csv`` - The character that seperates each record (row) in the input ``.csv`` file.
file.
Corresponds to ``RecordDelimiter`` in the S3 API ``CSVInput``. Corresponds to ``RecordDelimiter`` in the S3 API ``CSVInput``.
* - ``fd`` * - ``fd``
- -
- The character that seperates each field in a record. Defaults to - The character that seperates each field in a record. Defaults to ``,``.
``,``.
Corresponds to ``FieldDelimeter`` in the S3 API ``CSVInput``. Corresponds to ``FieldDelimeter`` in the S3 API ``CSVInput``.
* - ``qc`` * - ``qc``
- -
- The character used for escaping when the ``fd`` character is part of - The character used for escaping when the ``fd`` character is part of a value. Defaults to ``"``.
a value. Defaults to ``"``.
Corresponds to ``QuoteCharacter`` in the S3 API ``CSVInput``. Corresponds to ``QuoteCharacter`` in the S3 API ``CSVInput``.
* - ``qec`` * - ``qec``
- -
- The character used for escaping a quotation mark ``"`` character - The character used for escaping a quotation mark ``"`` character inside an already escaped value.
inside an already escaped value.
Corresponds to ``QuoteEscapeCharacter`` in the S3 API ``CSVInput``. Corresponds to ``QuoteEscapeCharacter`` in the S3 API ``CSVInput``.
* - ``fh`` * - ``fh``
- Yes - Yes
- The content of the first line in the ``.csv`` file. - The content of the first line in the ``.csv`` file.
Specify one of the following supported values: Specify one of the following supported values:
- ``NONE`` - The first line is not a header. - ``NONE`` - The first line is not a header.
- ``IGNORE`` - Ignore the first line. - ``IGNORE`` - Ignore the first line.
- ``USE`` - The first line is a header. - ``USE`` - The first line is a header.
For ``NONE`` or ``IGNORE``, you must specify column positions For ``NONE`` or ``IGNORE``, you must specify column positions ``_#`` to identify a column in the :mc-cmd:`~mc sql --query` statement.
``_#`` to identify a column in the :mc-cmd:`~mc sql --query`
statement.
For ``USE``, you can specify header values to identify a column in For ``USE``, you can specify header values to identify a column in the :mc-cmd:`~mc sql --query` statement.
the :mc-cmd:`~mc sql --query` statement.
Corresponds to ``FieldHeaderInfo`` in the S3 API ``CSVInput``. Corresponds to ``FieldHeaderInfo`` in the S3 API ``CSVInput``.
* - ``cc`` * - ``cc``
- Yes - Yes
- The character used to indicate a record should be ignored. The - The character used to indicate a record should be ignored.
character *must* appear at the beginning of the record. The character *must* appear at the beginning of the record.
Corresponds to ``Comment`` in the S3 API ``CSVInput``. Corresponds to ``Comment`` in the S3 API ``CSVInput``.
* - ``qrd`` * - ``qrd``
- Yes - Yes
- Specify ``TRUE`` to indicate that fields may contain record delimiter - Specify ``TRUE`` to indicate that fields may contain record delimiter values (``rd``).
values (``rd``).
Defaults to ``FALSE``. Defaults to ``FALSE``.
Corresponds to ``AllowQuotedRecordDelimiter`` in the S3 API Corresponds to ``AllowQuotedRecordDelimiter`` in the S3 API ``CSVInput``.
``CSVInput``.
S3 Compatibility S3 Compatibility
~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-mc.rst .. include:: /includes/common-minio-mc.rst
:start-after: start-minio-mc-s3-compatibility :start-after: start-minio-mc-s3-compatibility
:end-before: end-minio-mc-s3-compatibility :end-before: end-minio-mc-s3-compatibility