mirror of
https://github.com/minio/docs.git
synced 2025-07-28 19:42:10 +03:00
Doc updates for mc RELEASE.2024-01-05T05-04-32Z (#1112)
Doc updates for these `mc` changes: - New option for `mc mirror`: `--summary` - Clarify `mc sql --json-input` also works with `.ndjson` NDJSON clarified with a discussion in `mc sql` and an example of setting `content-type` in `mc cp`. Staged: http://192.241.195.202:9000/staging/DOCS-1104/linux/reference/minio-mc/mc-mirror.html#mc.mirror.-summary http://192.241.195.202:9000/staging/DOCS-1104/linux/reference/minio-mc/mc-sql.html#mc.sql.-json-input http://192.241.195.202:9000/staging/DOCS-1104/linux/reference/minio-mc/mc-sql.html#input-formats http://192.241.195.202:9000/staging/DOCS-1104/linux/reference/minio-mc/mc-cp.html#add-a-content-type-value Fixes https://github.com/minio/docs/issues/1104
This commit is contained in:
@ -163,7 +163,7 @@ Parameters
|
||||
|
||||
Enclose the entire list of key-value pairs passed to :mc-cmd:`~mc cp --encrypt` in double-quotes ``"``.
|
||||
|
||||
:mc-cmd:`~mc cp --encrypt` can use the ``MC_ENCRYPT`` environment variable for retrieving a list of encryption key-value pairs as an alternative to specifying them on the command line.
|
||||
:mc-cmd:`~mc cp --encrypt` can use the :envvar:`MC_ENCRYPT` environment variable for retrieving a list of encryption key-value pairs as an alternative to specifying them on the command line.
|
||||
|
||||
.. mc-cmd:: --encrypt-key
|
||||
:optional:
|
||||
@ -178,7 +178,7 @@ Parameters
|
||||
Enclose the entire list of key-value pairs passed to
|
||||
:mc-cmd:`~mc cp --encrypt-key` in double quotes ``"``.
|
||||
|
||||
:mc-cmd:`~mc cp --encrypt-key` can use the ``MC_ENCRYPT_KEY``
|
||||
:mc-cmd:`~mc cp --encrypt-key` can use the :envvar:`MC_ENCRYPT_KEY`
|
||||
environment variable for retrieving a list of encryption key-value pairs
|
||||
as an alternative to specifying them on the command line.
|
||||
|
||||
@ -449,6 +449,35 @@ command only applies to S3-to-S3 copy.
|
||||
:start-after: start-versioning-admonition
|
||||
:end-before: end-versioning-admonition
|
||||
|
||||
Add a ``content-type`` Value
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use :mc-cmd:`mc cp --attr` to add a ``content-type`` value.
|
||||
This command only applies to S3-to-S3 copy.
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc cp --attr="content-type=CONTENT-TYPE" SRCALIAS/SRCPATH TGTALIAS/TGTPATH
|
||||
|
||||
- Replace ``CONTENT-TYPE`` with the desired content type (also called a `media type <https://www.iana.org/assignments/media-types/media-types.xhtml>`__).
|
||||
|
||||
- Replace :mc-cmd:`SRCALIAS <mc cp SOURCE>` with the :mc:`alias <mc alias>` of a source S3-compatible host.
|
||||
|
||||
- Replace :mc-cmd:`SRCPATH <mc cp SOURCE>` with the path to the object on the source S3-compatible host.
|
||||
|
||||
- Replace :mc-cmd:`TGTALIAS <mc cp TARGET>` with the :mc:`alias <mc alias>` of a target S3-compatible host.
|
||||
|
||||
- Replace :mc-cmd:`TGTPATH <mc cp TARGET>` with the path to the object on the target S3-compatible host.
|
||||
Omit the object name to use the ``SRCPATH`` object name.
|
||||
|
||||
The following example sets a ``content-type`` of ``application/json``:
|
||||
|
||||
.. code-block::
|
||||
:class: copyable
|
||||
|
||||
mc cp data.ndjson --attr="content-type=application/json" myminio/mybucket
|
||||
|
||||
|
||||
Behavior
|
||||
--------
|
||||
|
@ -21,7 +21,7 @@ The :mc:`mc mirror` command synchronizes content to MinIO deployment, similar to
|
||||
.. end-mc-mirror-desc
|
||||
|
||||
.. note::
|
||||
|
||||
|
||||
:mc:`mc mirror` only synchronizes the current object without any version information or metadata.
|
||||
To synchronize an object's version history and metadata, consider using :mc:`mc replicate` for :ref:`bucket replication <minio-bucket-replication-serverside>` or :mc:`mc admin replicate` for :ref:`site replication <minio-site-replication-overview>`.
|
||||
|
||||
@ -38,7 +38,7 @@ The :mc:`mc mirror` command synchronizes content to MinIO deployment, similar to
|
||||
mc mirror --watch ~/mydata myminio/mydata
|
||||
|
||||
The command "watches" for files added or removed on the local filesystem and synchronizes those operations to MinIO until explicitly terminated.
|
||||
|
||||
|
||||
:mc-cmd:`mc mirror --watch` updates files changed on the local filesystem to MinIO (see :mc-cmd:`~mc mirror --overwrite`).
|
||||
``--watch`` does not remove other files from MinIO not present on the local filesystem (see :mc-cmd:`~mc mirror --remove`).
|
||||
|
||||
@ -67,7 +67,7 @@ The :mc:`mc mirror` command synchronizes content to MinIO deployment, similar to
|
||||
[--remove] \
|
||||
[--storage-class "string"] \
|
||||
[--watch] \
|
||||
SOURCE \
|
||||
SOURCE \
|
||||
TARGET
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
@ -78,8 +78,9 @@ Parameters
|
||||
~~~~~~~~~~
|
||||
|
||||
.. mc-cmd:: SOURCE
|
||||
:required:
|
||||
|
||||
*REQUIRED* The file(s) or object(s) to synchronize to the :mc-cmd:`~mc mirror TARGET` S3 host.
|
||||
The file(s) or object(s) to synchronize to the :mc-cmd:`~mc mirror TARGET` S3 host.
|
||||
|
||||
For objects on S3-compatible hosts, specify the path to the object as ``ALIAS/PATH``, where:
|
||||
|
||||
@ -100,8 +101,9 @@ Parameters
|
||||
If specifying a directory, :mc:`mc mirror` synchronizes all files in the directory.
|
||||
|
||||
.. mc-cmd:: TARGET
|
||||
:required:
|
||||
|
||||
*REQUIRED* The full path to bucket to which :mc:`mc mirror` synchronizes SOURCE objects. Specify the ``TARGET`` as ``ALIAS/PATH``, where:
|
||||
The full path to bucket to which :mc:`mc mirror` synchronizes SOURCE objects. Specify the ``TARGET`` as ``ALIAS/PATH``, where:
|
||||
|
||||
- ``ALIAS`` is the :mc:`alias <mc alias>` of a configured S3-compatible host, *and*
|
||||
|
||||
@ -114,23 +116,29 @@ Parameters
|
||||
:mc:`mc mirror` uses the object or file names from the :mc-cmd:`~mc mirror SOURCE` when synchronizing to the ``TARGET`` bucket.
|
||||
|
||||
.. mc-cmd:: --attr
|
||||
|
||||
:optional:
|
||||
|
||||
Add custom metadata for mirrored objects. Specify key-value pairs as ``KEY=VALUE\;``.
|
||||
For example, ``--attr key1=value1\;key2=value2\;key3=value3``.
|
||||
|
||||
.. mc-cmd:: --disable-multipart
|
||||
|
||||
:optional:
|
||||
|
||||
Disables multipart upload for the synchronization session.
|
||||
|
||||
.. mc-cmd:: --dry-run
|
||||
:optional:
|
||||
|
||||
Perform a mock mirror operation.
|
||||
Use this operation to test that the :mc:`mc mirror` operation will only mirror the desired objects or buckets.
|
||||
|
||||
.. mc-cmd:: --encrypt-key
|
||||
|
||||
:optional:
|
||||
|
||||
Encrypt or decrypt objects using server-side encryption with client-specified keys.
|
||||
Specify key-value pairs as ``KEY=VALUE``.
|
||||
|
||||
- Each ``KEY`` represents a bucket or object.
|
||||
|
||||
- Each ``KEY`` represents a bucket or object.
|
||||
- Each ``VALUE`` represents the data key to use for encrypting object(s).
|
||||
|
||||
Enclose the entire list of key-value pairs passed to :mc-cmd:``~mc mirror --encrypt-key`` in double quotes ``"``.
|
||||
@ -140,24 +148,18 @@ Parameters
|
||||
You can only delete encrypted objects if you specify the correct :mc-cmd:`~mc mirror --encrypt-key` secret key.
|
||||
|
||||
.. mc-cmd:: --exclude
|
||||
|
||||
:optional:
|
||||
|
||||
Exclude object(s) in the :mc-cmd:`~mc mirror SOURCE` path that match the specified object name pattern.
|
||||
|
||||
.. mc-cmd:: --exclude-storageclass
|
||||
|
||||
:optional:
|
||||
|
||||
Exclude object(s) on the :mc-cmd:`~mc mirror SOURCE` that have the specified storage class.
|
||||
You can use this flag multiple times in a command to exclude objects from more than one storage class.
|
||||
|
||||
Use this to exclude objects with storage classes that require rehydration or restoration of objects, such as migrating from an AWS S3 bucket where some objects have the ``GLACIER`` or ``DEEP_ARCHIVE`` storage classes.
|
||||
|
||||
.. mc-cmd:: --dry-run
|
||||
|
||||
|
||||
Perform a mock mirror operation.
|
||||
Use this operation to test that the :mc:`mc mirror` operation will only mirror the desired objects or buckets.
|
||||
|
||||
.. --limit-download and --limit-upload included here
|
||||
|
||||
.. include:: /includes/linux/minio-client.rst
|
||||
@ -165,36 +167,36 @@ Parameters
|
||||
:end-before: end-mc-limit-flags-desc
|
||||
|
||||
.. mc-cmd:: --md5
|
||||
|
||||
:optional:
|
||||
|
||||
Forces all uploads to calculate MD5 checksums.
|
||||
Forces all uploads to calculate MD5 checksums.
|
||||
|
||||
.. mc-cmd:: --monitoring-address
|
||||
|
||||
:optional:
|
||||
|
||||
Creates a `Prometheus <https://prometheus.io/>`__ endpoint for monitoring mirroring activity.
|
||||
Specify the local network adapter and port address on which to create the scraping endpoint.
|
||||
Defaults to ``localhost:8081``).
|
||||
|
||||
.. mc-cmd:: --newer-than
|
||||
|
||||
:optional:
|
||||
|
||||
Mirror object(s) newer than the specified number of days.
|
||||
Specify a string in ``#d#hh#mm#ss`` format.
|
||||
Mirror object(s) newer than the specified number of days.
|
||||
Specify a string in ``#d#hh#mm#ss`` format
|
||||
For example: ``--newer-than 1d2hh3mm4ss``.
|
||||
|
||||
.. mc-cmd:: --older-than
|
||||
|
||||
:optional:
|
||||
|
||||
Mirror object(s) older than the specified time limit.
|
||||
Specify a string in ``#d#hh#mm#ss`` format.
|
||||
Mirror object(s) older than the specified time limit.
|
||||
Specify a string in ``#d#hh#mm#ss`` format.
|
||||
For example: ``--older-than 1d2hh3mm4ss``.
|
||||
|
||||
|
||||
Defaults to ``0`` (all objects).
|
||||
|
||||
.. mc-cmd:: --overwrite
|
||||
|
||||
|
||||
:optional:
|
||||
|
||||
Overwrites object(s) on the :mc-cmd:`~mc mirror TARGET`.
|
||||
|
||||
For example, consider an active ``mc mirror --overwrite`` synchronizing content from Source to Destination.
|
||||
@ -205,21 +207,21 @@ Parameters
|
||||
``mc mirror`` logs an error and continues to synchronize other objects.
|
||||
|
||||
.. mc-cmd:: --preserve, a
|
||||
|
||||
:optional:
|
||||
|
||||
Preserve file system attributes and bucket policy rules of the :mc-cmd:`~mc mirror SOURCE` on the :mc-cmd:`~mc mirror TARGET`.
|
||||
|
||||
.. mc-cmd:: --region
|
||||
|
||||
:optional:
|
||||
|
||||
Specify the ``string`` region when creating new bucket(s) on the target.
|
||||
Specify the ``string`` region when creating new bucket(s) on the target.
|
||||
|
||||
Defaults to ``"us-east-1"``.
|
||||
|
||||
.. mc-cmd:: --remove
|
||||
|
||||
:optional:
|
||||
|
||||
Removes object(s) on the Target that do not exist on the Source.
|
||||
Removes object(s) on the Target that do not exist on the Source.
|
||||
|
||||
Use the ``--remove`` flag to have the same list of objects on both Source and Target.
|
||||
|
||||
@ -241,18 +243,24 @@ Parameters
|
||||
In prior versions, specifying ``/path/to/directory`` would result in the removal of the ``/path/to`` folder if ``directory`` did not exist.
|
||||
|
||||
.. mc-cmd:: --retry
|
||||
:optional:
|
||||
|
||||
In case of errors during mirror process, retry on each errored object.
|
||||
|
||||
.. mc-cmd:: storage-class, sc
|
||||
|
||||
.. mc-cmd:: --storage-class, sc
|
||||
:optional:
|
||||
|
||||
Set the storage class for the new object(s) on the :mc-cmd:`~mc mirror TARGET`.
|
||||
|
||||
|
||||
See the Amazon documentation on :aws-docs:`Storage Classes <AmazonS3/latest/dev/storage-class-intro.html>` for more information on S3 storage classses.
|
||||
|
||||
.. mc-cmd:: --summary
|
||||
:optional:
|
||||
|
||||
On completion, output a summary of the data that was synchronized.
|
||||
|
||||
.. mc-cmd:: --watch, w
|
||||
|
||||
:optional:
|
||||
|
||||
Use ``--watch`` flag to mirror objects from Source to Target, where the Target may also have additional objects not present on the Source.
|
||||
|
||||
@ -264,13 +272,13 @@ Parameters
|
||||
|
||||
For example, object A and B exist on the watched Source.
|
||||
Objects A, B, and C exist on the watched Target.
|
||||
|
||||
|
||||
A client writes object D to Source and removes object B.
|
||||
|
||||
After the operation, objects A and D exist on the Source.
|
||||
Objects A, C, and D exist on the Target.
|
||||
|
||||
|
||||
|
||||
|
||||
Global Flags
|
||||
~~~~~~~~~~~~
|
||||
|
||||
@ -338,7 +346,7 @@ Use :mc:`mc mirror` with :mc-cmd:`~mc mirror --exclude-storageclass` to mirror o
|
||||
|
||||
.. code-block::
|
||||
:class: copyable
|
||||
|
||||
|
||||
mc mirror --exclude-storageclass GLACIER \
|
||||
--exclude-storageclass DEEP_ARCHIVE SRCALIAS/SRCPATH TGALIAS/TGPATH
|
||||
|
||||
|
@ -15,21 +15,17 @@ Syntax
|
||||
|
||||
.. start-mc-sql-desc
|
||||
|
||||
The :mc:`mc sql` command provides an S3 Select interface for performing sql
|
||||
queries on objects in the specified MinIO deployment.
|
||||
The :mc:`mc sql` command provides an S3 Select interface for performing sql queries on objects in the specified MinIO deployment.
|
||||
|
||||
.. end-mc-sql-desc
|
||||
|
||||
See :s3-docs:`Selecting content from objects
|
||||
<selecting-content-from-objects>` for more information on S3 Select behavior
|
||||
and limitations.
|
||||
See :s3-docs:`Selecting content from objects <selecting-content-from-objects>` for more information on S3 Select behavior and limitations.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: EXAMPLE
|
||||
|
||||
The following command queries all objects in the
|
||||
``mydata`` bucket on the ``myminio`` MinIO deployment:
|
||||
The following command queries all objects in the ``mydata`` bucket on the ``myminio`` MinIO deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
@ -63,36 +59,36 @@ Parameters
|
||||
~~~~~~~~~~
|
||||
|
||||
.. mc-cmd:: ALIAS
|
||||
:required:
|
||||
|
||||
*Required* The full path to the bucket or object to run the SQL query
|
||||
against. Specify the :ref:`alias <alias>` of a configured
|
||||
S3 service as the prefix to the ``ALIAS`` path. For example:
|
||||
The full path to the bucket or object to run the SQL query against.
|
||||
Specify the :ref:`alias <alias>` of a configured S3 service as the prefix to the ``ALIAS`` path.
|
||||
For example:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mc sql [FLAGS] play/mybucket
|
||||
|
||||
.. mc-cmd:: --query, e
|
||||
|
||||
:required:
|
||||
|
||||
*Required* The SQL statement to execute on the specified
|
||||
:mc-cmd:`~mc sql ALIAS` directory or object. Wrap the entire SQL query in
|
||||
double quotes ``"``.
|
||||
The SQL statement to execute on the specified :mc-cmd:`~mc sql ALIAS` directory or object.
|
||||
Wrap the entire SQL query in double quotes ``"``.
|
||||
|
||||
Defaults to ``"select * from S3Object"``.
|
||||
|
||||
.. mc-cmd:: --csv-input
|
||||
|
||||
:optional:
|
||||
|
||||
*Optional* The data format for ``.csv`` input objects. Specify a string of
|
||||
comma-seperated ``key=value,...`` pairs. See :ref:`mc-sql-csv-format` for
|
||||
more information on valid keys.
|
||||
The data format for ``.csv`` input objects.
|
||||
Specify a string of comma-seperated ``key=value,...`` pairs.
|
||||
See :ref:`mc-sql-csv-format` for more information on valid keys.
|
||||
|
||||
.. mc-cmd:: --compression
|
||||
|
||||
:optional:
|
||||
|
||||
*Optional* The compression type of the input object. Specify one of the
|
||||
following supported values:
|
||||
The compression type of the input object.
|
||||
Specify one of the following supported values:
|
||||
|
||||
- ``GZIP``
|
||||
- ``BZIP2``
|
||||
@ -106,71 +102,62 @@ Parameters
|
||||
- ``SNAPPY`` `Snappy <http://google.github.io/snappy/>`__ framed stream
|
||||
|
||||
.. mc-cmd:: --csv-output
|
||||
|
||||
:optional:
|
||||
|
||||
*Optional* The data format for ``.csv`` output. Specify a string of
|
||||
comma-seperated ``key=value,...`` pairs. See :ref:`mc-sql-csv-format` for
|
||||
more information on valid keys.
|
||||
The data format for ``.csv`` output.
|
||||
Specify a string of comma-seperated ``key=value,...`` pairs.
|
||||
See :ref:`mc-sql-csv-format` for more information on valid keys.
|
||||
|
||||
See the S3 API :s3-api:`CSVOutput <API_CSVOutput.html>` for more
|
||||
information.
|
||||
See the S3 API :s3-api:`CSVOutput <API_CSVOutput.html>` for more information.
|
||||
|
||||
.. mc-cmd:: --csv-output-header
|
||||
|
||||
:optional:
|
||||
|
||||
*Optional* The header row of the ``.csv`` output file. Specify a string of
|
||||
comma-separated fields as ``field1,field2,...``.
|
||||
The header row of the ``.csv`` output file.
|
||||
Specify a string of comma-separated fields as ``field1,field2,...``.
|
||||
|
||||
Omit to output a ``.csv`` with no header row.
|
||||
|
||||
.. mc-cmd:: --encrypt-key
|
||||
|
||||
:optional:
|
||||
|
||||
*Optional* The encryption key to use for performing Server-Side Encryption
|
||||
with Client Keys (SSE-C). Specify comma seperated key-value pairs as
|
||||
``KEY=VALUE,...``.
|
||||
|
||||
- For ``KEY``, specify the MinIO deployment
|
||||
:mc-cmd:`alias <mc alias>` and full path to the bucket, including any
|
||||
bucket prefixes. Separate the alias and bucket path with a forward slash
|
||||
``\``. For example, ``play/mybucket``
|
||||
The encryption key to use for performing Server-Side Encryption with Client Keys (SSE-C).
|
||||
Specify comma seperated key-value pairs as ``KEY=VALUE,...``.
|
||||
|
||||
- For ``VALUE``, specify the data key to use for encryption object(s) in
|
||||
the bucket or bucket prefix specified to ``KEY``.
|
||||
- For ``KEY``, specify the MinIO deployment :mc-cmd:`alias <mc alias>` and full path to the bucket, including any bucket prefixes.
|
||||
Separate the alias and bucket path with a forward slash ``\``.
|
||||
For example, ``play/mybucket``.
|
||||
|
||||
:mc-cmd:`~mc sql --encrypt-key` can use the ``MC_ENCRYPT_KEY``
|
||||
environment variable for populating the list of encryption key-value
|
||||
pairs as an alternative to specifying them on the command line.
|
||||
- For ``VALUE``, specify the data key to use for encryption object(s) in the bucket or bucket prefix specified to ``KEY``.
|
||||
|
||||
:mc-cmd:`~mc sql --encrypt-key` can use the :envvar:`MC_ENCRYPT_KEY` environment variable for populating the list of encryption key-value pairs as an alternative to specifying them on the command line.
|
||||
|
||||
.. mc-cmd:: --json-input
|
||||
|
||||
:optional:
|
||||
|
||||
*Optional* The data format for ``.json`` input objects. Specify the type of
|
||||
the JSON contents as ``type=<VALUE>``. The value can be either:
|
||||
The data format for ``.json`` or ``.ndjson`` input objects.
|
||||
Specify the type of the JSON contents as ``type=<VALUE>``.
|
||||
The value can be either:
|
||||
|
||||
- ``DOCUMENT`` - JSON `document <https://www.json.org/json-en.html>`__.
|
||||
- ``LINES`` - JSON `lines <http://jsonlines.org/>`__.
|
||||
|
||||
See the S3 API :s3-api:`JSONInput <API_JSONInput.html>` for more
|
||||
information.
|
||||
See the S3 API :s3-api:`JSONInput <API_JSONInput.html>` for more information.
|
||||
|
||||
.. mc-cmd:: --json-output
|
||||
|
||||
:optional:
|
||||
|
||||
*Optional* The data format for the ``.json`` output. Supports the
|
||||
``rd=value`` key, where ``rd`` is the ``RecordDelimiter`` for the JSON
|
||||
document.
|
||||
The data format for the ``.json`` output.
|
||||
Supports the ``rd=value`` key, where ``rd`` is the ``RecordDelimiter`` for the JSON document.
|
||||
|
||||
Omit to use the default newline character ``\n``.
|
||||
|
||||
See the S3 API :s3-api:`JSONOutput <API_JSONOutput.html>` for more
|
||||
information.
|
||||
See the S3 API :s3-api:`JSONOutput <API_JSONOutput.html>` for more information.
|
||||
|
||||
.. mc-cmd:: --recursive, r
|
||||
|
||||
:optional:
|
||||
|
||||
*Optional* Recursively searches the specified :mc-cmd:`~mc sql ALIAS`
|
||||
directory using the :mc-cmd:`~mc sql --query` SQL statement.
|
||||
Recursively searches the specified :mc-cmd:`~mc sql ALIAS` directory using the :mc-cmd:`~mc sql --query` SQL statement.
|
||||
|
||||
Global Flags
|
||||
~~~~~~~~~~~~
|
||||
@ -185,36 +172,29 @@ Examples
|
||||
Select all Columns in all Objects in a Bucket
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use :mc:`mc sql` with the :mc-cmd:`~mc sql --recursive` and
|
||||
:mc-cmd:`~mc sql --query` options to apply the query to all objects
|
||||
in a bucket:
|
||||
Use :mc:`mc sql` with the :mc-cmd:`~mc sql --recursive` and :mc-cmd:`~mc sql --query` options to apply the query to all objects in a bucket:
|
||||
|
||||
.. code-block:: shell
|
||||
:class: copyable
|
||||
|
||||
mc sql --recursive --query "select * from S3Object" ALIAS/PATH
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc sql ALIAS>` with the
|
||||
:ref:`alias <alias>` of the MinIO deployment.
|
||||
- Replace :mc-cmd:`ALIAS <mc sql ALIAS>` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
- Replace :mc-cmd:`PATH <mc sql ALIAS>` with the path to the bucket
|
||||
on the MinIO deployment.
|
||||
- Replace :mc-cmd:`PATH <mc sql ALIAS>` with the path to the bucket on the MinIO deployment.
|
||||
|
||||
Run an Aggregation Query on an Object
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use :mc:`mc sql` with the :mc-cmd:`~mc sql --query` option to query
|
||||
an object on an MinIO deployment:
|
||||
Use :mc:`mc sql` with the :mc-cmd:`~mc sql --query` option to query an object on an MinIO deployment:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mc sql --query "select count(s.power) from S3Object" ALIAS/PATH
|
||||
|
||||
- Replace :mc-cmd:`ALIAS <mc sql ALIAS>` with the
|
||||
:ref:`alias <alias>` of the MinIO deployment.
|
||||
- Replace :mc-cmd:`ALIAS <mc sql ALIAS>` with the :ref:`alias <alias>` of the MinIO deployment.
|
||||
|
||||
- Replace :mc-cmd:`PATH <mc sql ALIAS>` with the path to the object
|
||||
on the MinIO deployment.
|
||||
- Replace :mc-cmd:`PATH <mc sql ALIAS>` with the path to the object on the MinIO deployment.
|
||||
|
||||
Behavior
|
||||
--------
|
||||
@ -224,32 +204,44 @@ Input Formats
|
||||
|
||||
:mc:`mc sql` supports the following input formats:
|
||||
|
||||
- ``.csv``
|
||||
- ``.json``
|
||||
- ``.parquet``
|
||||
.. list-table:: Input Format Types
|
||||
:header-rows: 1
|
||||
|
||||
* - Type
|
||||
- ``content-type`` Value
|
||||
|
||||
For ``.csv`` file types, use :mc-cmd:`mc sql --csv-input` to
|
||||
specify the CSV data format. See :ref:`mc-sql-csv-format` for more
|
||||
information on CSV formatting fields.
|
||||
* - ``.csv``
|
||||
- ``text/csv``
|
||||
|
||||
For ``.json`` file types, use :mc-cmd:`mc sql --json-input` to specify
|
||||
the JSON data format.
|
||||
* - ``.json``
|
||||
- ``application/json``
|
||||
|
||||
For ``.parquet`` file types, :mc:`mc sql` automatically interprets the
|
||||
data format.
|
||||
* - ``.parquet``
|
||||
- none
|
||||
|
||||
For ``.csv`` file types, use :mc-cmd:`mc sql --csv-input` to specify the CSV data format.
|
||||
See :ref:`mc-sql-csv-format` for more information on CSV formatting fields.
|
||||
|
||||
For ``.json`` file types, use :mc-cmd:`mc sql --json-input` to specify the JSON data format.
|
||||
|
||||
For ``.parquet`` file types, :mc:`mc sql` automatically interprets the data format.
|
||||
|
||||
:mc:`mc sql` determines the type by the file extension of the target object.
|
||||
For example, an object named ``data.json`` is interpreted as a JSON file.
|
||||
|
||||
You can query data of a supported type but a different extension if the object has the appropriate ``content-type``.
|
||||
For more information, see :mc-cmd:`mc cp --attr`.
|
||||
|
||||
.. _mc-sql-csv-format:
|
||||
|
||||
CSV Formatting Fields
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following table lists valid key-value pairs for use with
|
||||
:mc-cmd:`mc sql --csv-input` and :mc-cmd:`mc sql --csv-output`.
|
||||
Certain key pairs are only valid for :mc-cmd:`~mc sql --csv-input`
|
||||
See the documentation for S3 API :s3-api:`CSVInput <API_CSVInput.html>` for more
|
||||
information on S3 CSV formatting.
|
||||
The following table lists valid key-value pairs for use with :mc-cmd:`mc sql --csv-input` and :mc-cmd:`mc sql --csv-output`.
|
||||
Certain key pairs are only valid for :mc-cmd:`~mc sql --csv-input`.
|
||||
See the documentation for S3 API :s3-api:`CSVInput <API_CSVInput.html>` for more information on S3 CSV formatting.
|
||||
|
||||
.. list-table::
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 20 60
|
||||
:width: 100%
|
||||
@ -260,71 +252,62 @@ information on S3 CSV formatting.
|
||||
|
||||
* - ``rd``
|
||||
-
|
||||
- The character that seperates each record (row) in the input ``.csv``
|
||||
file.
|
||||
|
||||
- The character that seperates each record (row) in the input ``.csv`` file.
|
||||
|
||||
Corresponds to ``RecordDelimiter`` in the S3 API ``CSVInput``.
|
||||
|
||||
* - ``fd``
|
||||
-
|
||||
- The character that seperates each field in a record. Defaults to
|
||||
``,``.
|
||||
|
||||
- The character that seperates each field in a record. Defaults to ``,``.
|
||||
|
||||
Corresponds to ``FieldDelimeter`` in the S3 API ``CSVInput``.
|
||||
|
||||
|
||||
* - ``qc``
|
||||
-
|
||||
- The character used for escaping when the ``fd`` character is part of
|
||||
a value. Defaults to ``"``.
|
||||
- The character used for escaping when the ``fd`` character is part of a value. Defaults to ``"``.
|
||||
|
||||
Corresponds to ``QuoteCharacter`` in the S3 API ``CSVInput``.
|
||||
|
||||
|
||||
* - ``qec``
|
||||
-
|
||||
- The character used for escaping a quotation mark ``"`` character
|
||||
inside an already escaped value.
|
||||
- The character used for escaping a quotation mark ``"`` character inside an already escaped value.
|
||||
|
||||
Corresponds to ``QuoteEscapeCharacter`` in the S3 API ``CSVInput``.
|
||||
|
||||
|
||||
* - ``fh``
|
||||
- Yes
|
||||
- The content of the first line in the ``.csv`` file.
|
||||
|
||||
- The content of the first line in the ``.csv`` file.
|
||||
|
||||
Specify one of the following supported values:
|
||||
|
||||
- ``NONE`` - The first line is not a header.
|
||||
- ``IGNORE`` - Ignore the first line.
|
||||
- ``USE`` - The first line is a header.
|
||||
|
||||
For ``NONE`` or ``IGNORE``, you must specify column positions
|
||||
``_#`` to identify a column in the :mc-cmd:`~mc sql --query`
|
||||
statement.
|
||||
For ``NONE`` or ``IGNORE``, you must specify column positions ``_#`` to identify a column in the :mc-cmd:`~mc sql --query` statement.
|
||||
|
||||
For ``USE``, you can specify header values to identify a column in
|
||||
the :mc-cmd:`~mc sql --query` statement.
|
||||
For ``USE``, you can specify header values to identify a column in the :mc-cmd:`~mc sql --query` statement.
|
||||
|
||||
Corresponds to ``FieldHeaderInfo`` in the S3 API ``CSVInput``.
|
||||
|
||||
|
||||
* - ``cc``
|
||||
- Yes
|
||||
- The character used to indicate a record should be ignored. The
|
||||
character *must* appear at the beginning of the record.
|
||||
- The character used to indicate a record should be ignored.
|
||||
The character *must* appear at the beginning of the record.
|
||||
|
||||
Corresponds to ``Comment`` in the S3 API ``CSVInput``.
|
||||
|
||||
|
||||
* - ``qrd``
|
||||
- Yes
|
||||
- Specify ``TRUE`` to indicate that fields may contain record delimiter
|
||||
values (``rd``).
|
||||
- Specify ``TRUE`` to indicate that fields may contain record delimiter values (``rd``).
|
||||
|
||||
Defaults to ``FALSE``.
|
||||
|
||||
Corresponds to ``AllowQuotedRecordDelimiter`` in the S3 API
|
||||
``CSVInput``.
|
||||
Corresponds to ``AllowQuotedRecordDelimiter`` in the S3 API ``CSVInput``.
|
||||
|
||||
S3 Compatibility
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: /includes/common-minio-mc.rst
|
||||
:start-after: start-minio-mc-s3-compatibility
|
||||
:end-before: end-minio-mc-s3-compatibility
|
||||
:end-before: end-minio-mc-s3-compatibility
|
||||
|
Reference in New Issue
Block a user