From a5bfd27b805bbe0305c1d350f099143dba5f81c3 Mon Sep 17 00:00:00 2001 From: Andrea Longo Date: Fri, 27 Sep 2024 14:50:06 -0600 Subject: [PATCH] partial edits --- ...collect-minio-metrics-using-prometheus.rst | 2 +- .../monitoring/metrics-and-alerts.rst | 170 +-------------- ...trics-v2-deprecated.rst => metrics-v2.rst} | 37 ++-- source/operations/monitoring/metrics-v3.rst | 197 ++++++++++++++++++ 4 files changed, 228 insertions(+), 178 deletions(-) rename source/operations/monitoring/{metrics-v2-deprecated.rst => metrics-v2.rst} (92%) create mode 100644 source/operations/monitoring/metrics-v3.rst diff --git a/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst b/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst index 58011ed2..14941855 100644 --- a/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst +++ b/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst @@ -102,7 +102,7 @@ Common changes include: Some deployments require a longer scrape interval due to the number of metrics being scraped. To reduce the load on your MinIO and Prometheus servers, choose the longest interval that meets your monitoring requirements. - You can specify a ``scrape_interval`` for each job in its ``job_name`` section, or all jobs in a separate ``global`` section. + You can specify a global ``scrape_interval`` for all jobs or override a global value by setting a ``scrape_interval`` in an individual ``job_name`` section. - Set the ``job_name`` to a value associated to the MinIO deployment. diff --git a/source/operations/monitoring/metrics-and-alerts.rst b/source/operations/monitoring/metrics-and-alerts.rst index eaae840a..8bddf6d6 100644 --- a/source/operations/monitoring/metrics-and-alerts.rst +++ b/source/operations/monitoring/metrics-and-alerts.rst @@ -12,105 +12,13 @@ Metrics and Alerts :local: :depth: 2 -.. admonition:: Metrics Version 2 Deprecated - :class: note - - Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 (v3) replaces the deprecated :ref:`metrics version 2 (v2) `. +.. versionadded:: RELEASE.2024-07-15T19-02-30Z + Metrics version 3 (v3) provides additional metrics and metric types for more targeted scraping. + MinIO publishes metrics using the :prometheus-docs:`Prometheus Data Model `. You can use any scraping tool to pull metrics data from MinIO for further analysis and alerting. -For metrics version 3, all metrics are available under the base ``/minio/metrics/v3`` endpoint, optionally appending an additional path for each category. - -For example, the following endpoint returns audit metrics: - -.. code-block:: shell - :class: copyable - - http://HOSTNAME:PORT/minio/metrics/v3/audit - -Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment. -For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer. - -By default, MinIO requires authentication to scrape the metrics endpoints. -To generate the needed bearer tokens, use :mc:`mc admin prometheus generate`. -You can also disable metrics endpoint authentication by setting :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` to ``public``. - -MinIO provides the following types and scraping endpoints, relative to the base URL: - -.. list-table:: - :header-rows: 1 - :widths: 25 25 50 - :width: 100% - - * - Category - - Metric type - - Path - - * - API - - ``api`` - - ``/api/requests`` - - ``/bucket/api`` - - * - Audit - - ``audit`` - - ``/audit`` - - * - Cluster - - ``cluster`` - - ``/cluster/config`` - - ``/cluster/erasure-set`` - - ``/cluster/health`` - - ``/cluster/iam`` - - ``/cluster/usage/buckets`` - - ``/cluster/usage/objects`` - - * - Debug - - ``debug`` - - ``/debug/go`` - - * - ILM - - ``ilm`` - - ``/ilm`` - - * - Logger webhook - - ``logger`` - - ``/logger/webhook`` - - * - Notification - - ``notification`` - - ``/notification`` - - * - Replication - - ``replication`` - - ``/replication`` - - ``/bucket/replication`` - - * - Scanner - - ``scanner`` - - ``/scanner`` - - * - System - - ``system`` - - ``/system/drive`` - - ``/system/memory`` - - ``/system/cpu`` - - ``/system/network/internode`` - - ``/system/process`` - -For a complete list of metrics for each endpoint, see :ref:`Available Metrics `. - You can also access metrics using :mc-cmd:`mc admin prometheus metrics` and the metric type for the desired category. For more information, see the :mc-cmd:`MinIO Admin Client reference `. @@ -128,80 +36,18 @@ For more information, see the :mc-cmd:`MinIO Admin Client reference ` -- :ref:`Audit Metrics ` -- :ref:`Cluster Metrics ` -- :ref:`Debug Metrics ` -- :ref:`ILM Metrics ` -- :ref:`Logger webhook Metrics ` -- :ref:`Notification Metrics ` -- :ref:`Replication Metrics ` -- :ref:`Scanner Metrics ` -- :ref:`System Metrics ` - - -.. _minio-available-v3-api-metrics: - -.. include:: /includes/common-metrics-v3-api.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-audit-metrics: - -.. include:: /includes/common-metrics-v3-audit.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-cluster-metrics: - -.. include:: /includes/common-metrics-v3-cluster.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-debug-metrics: - -.. include:: /includes/common-metrics-v3-debug.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-ilm-metrics: - -.. include:: /includes/common-metrics-v3-ilm.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-logger-webhook-metrics: - -.. include:: /includes/common-metrics-v3-logger-webhook.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-notification-metrics: - -.. include:: /includes/common-metrics-v3-notification.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-replication-metrics: - -.. include:: /includes/common-metrics-v3-replication.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-scanner-metrics: - -.. include:: /includes/common-metrics-v3-scanner.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-system-metrics: - -.. include:: /includes/common-metrics-v3-system.md - :parser: myst_parser.sphinx_ +v3 metrics +v2 metrics .. toctree:: :titlesonly: :hidden: + /operations/monitoring/metrics-v3 + /operations/monitoring/metrics-v2 /operations/monitoring/collect-minio-metrics-using-prometheus /operations/monitoring/monitor-and-alert-using-influxdb diff --git a/source/operations/monitoring/metrics-v2-deprecated.rst b/source/operations/monitoring/metrics-v2.rst similarity index 92% rename from source/operations/monitoring/metrics-v2-deprecated.rst rename to source/operations/monitoring/metrics-v2.rst index 3d02b941..6501d25c 100644 --- a/source/operations/monitoring/metrics-v2-deprecated.rst +++ b/source/operations/monitoring/metrics-v2.rst @@ -1,4 +1,3 @@ -:orphan: .. _minio-metrics-v2: ================= @@ -11,15 +10,22 @@ Metrics Version 2 :local: :depth: 1 -.. admonition:: Metrics Version 2 Deprecated - :class: note - Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics :ref:`version 3 (v3) ` replaces the deprecated metrics version 2 (v2). +Metrics version 2 +----------------- -The following sections describe the deprecated version 2 endpoints and metrics. +For metrics version 2, all metrics are available under the base ``/minio/v2/metrics`` endpoint, optionally appending an additional path for each category. + +For example, the following endpoint returns bucket metrics: + +.. code-block:: shell + :class: copyable + + http://HOSTNAME:PORT/minio/v2/metrics/bucket + +Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment. +For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer. -Metrics version 2 endpoints ---------------------------- .. tab-set:: @@ -70,11 +76,11 @@ Metrics version 2 endpoints For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer. -Configure Prometheus to Collect and Alert using MinIO Metrics +Configure Prometheus to collect and alert using MinIO Metrics ------------------------------------------------------------- -1) Generate the Scrape Configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +1) Generate a v2 scrape configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Use the :mc:`mc admin prometheus generate` command to generate the scrape configuration for use by Prometheus in making scraping requests: @@ -208,7 +214,7 @@ Use the :mc:`mc admin prometheus generate` command to generate the scrape config For Prometheus deployments external to the cluster, you must specify an ingress or load balancer endpoint configured to route connections to and from the MinIO Tenant. -2) Restart Prometheus with the Updated Configuration +2) Restart Prometheus with the updated configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Append the desired ``scrape_configs`` job generated in the previous step to the configuration file: @@ -292,7 +298,7 @@ Start the Prometheus cluster using the configuration file: prometheus --config.file=prometheus.yaml -3) Analyze Collected Metrics +3) Analyze collected metrics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Prometheus includes an :prometheus-docs:`expression browser `. @@ -323,7 +329,7 @@ The following query examples return metrics collected by Prometheus every five m minio_node_drive_io_waiting{job-"minio-job"}[5m] -4) Configure an Alert Rule using MinIO Metrics +4) Configure an alert rule using MinIO Metrics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You must configure :prometheus-docs:`Alert Rules ` on the Prometheus deployment to trigger alerts based on collected MinIO metrics. @@ -370,9 +376,10 @@ Dashboards For v2 metrics, MinIO provides Grafana Dashboards to display the metrics collected by Prometheus. For more information, see :ref:`minio-grafana` +.. _minio-metrics-and-alerts-available-v2-metrics: -Available metrics ------------------ +Available v2 metrics +-------------------- - :ref:`Cluster Metrics ` - :ref:`Bucket Metrics ` diff --git a/source/operations/monitoring/metrics-v3.rst b/source/operations/monitoring/metrics-v3.rst new file mode 100644 index 00000000..2b006af0 --- /dev/null +++ b/source/operations/monitoring/metrics-v3.rst @@ -0,0 +1,197 @@ +.. _minio-metrics-v3: + +================= +Metrics version 3 +================= + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 2 + +For metrics version 3, all metrics are available under the base ``/minio/metrics/v3`` endpoint, optionally appending an additional path for each category. + +For example, the following endpoint returns audit metrics: + +.. code-block:: shell + :class: copyable + + http://HOSTNAME:PORT/minio/metrics/v3/audit + +Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment. +For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer. + +By default, MinIO requires authentication to scrape the metrics endpoints. +To generate the needed bearer tokens, use :mc:`mc admin prometheus generate`. +You can also disable metrics endpoint authentication by setting :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` to ``public``. + +MinIO provides the following v3 types and scraping endpoints, relative to the base URL: + +.. list-table:: + :header-rows: 1 + :widths: 25 25 50 + :width: 100% + + * - Category + - Metric type + - Path + + * - API + - ``api`` + - ``/api/requests`` + + ``/bucket/api`` + + * - Audit + - ``audit`` + - ``/audit`` + + * - Cluster + - ``cluster`` + - ``/cluster/config`` + + ``/cluster/erasure-set`` + + ``/cluster/health`` + + ``/cluster/iam`` + + ``/cluster/usage/buckets`` + + ``/cluster/usage/objects`` + + * - Debug + - ``debug`` + - ``/debug/go`` + + * - ILM + - ``ilm`` + - ``/ilm`` + + * - Logger webhook + - ``logger`` + - ``/logger/webhook`` + + * - Notification + - ``notification`` + - ``/notification`` + + * - Replication + - ``replication`` + - ``/replication`` + + ``/bucket/replication`` + + * - Scanner + - ``scanner`` + - ``/scanner`` + + * - System + - ``system`` + - ``/system/drive`` + + ``/system/memory`` + + ``/system/cpu`` + + ``/system/network/internode`` + + ``/system/process`` + +For a complete list of metrics for each endpoint, see :ref:`Available Metrics `. + +You can also access metrics using :mc-cmd:`mc admin prometheus metrics` and the metric type for the desired category. +For more information, see the :mc-cmd:`MinIO Admin Client reference `. + +.. cond:: k8s + + The MinIO Operator supports deploying a per-tenant Prometheus instance configured to support metrics and visualization. + + If you deploy the Tenant with this feature disabled *but* still want the historical metric views, you can instead configure an external Prometheus service to scrape the Tenant metrics. + Once configured, you can update the Tenant to query that Prometheus service to retrieve metric data: + +.. cond:: linux or container or macos or windows + + To enable historical data visualization in MinIO Console, set the following environment variables on each node in the MinIO deployment: + +- Set :envvar:`MINIO_PROMETHEUS_URL` to the URL of the Prometheus service +- Set :envvar:`MINIO_PROMETHEUS_JOB_ID` to the unique job ID assigned to the collected metrics + +.. _minio-metrics-and-alerts-available-v3-metrics: + +Available v3 metrics +-------------------- + +MinIO publishes a number of metrics at the cluster, node, or bucket levels. +Each metric includes a label for the MinIO server which generated that metric. + +- :ref:`API Metrics ` +- :ref:`Audit Metrics ` +- :ref:`Cluster Metrics ` +- :ref:`Debug Metrics ` +- :ref:`ILM Metrics ` +- :ref:`Logger webhook Metrics ` +- :ref:`Notification Metrics ` +- :ref:`Replication Metrics ` +- :ref:`Scanner Metrics ` +- :ref:`System Metrics ` + + +.. _minio-available-v3-api-metrics: + +.. include:: /includes/common-metrics-v3-api.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-audit-metrics: + +.. include:: /includes/common-metrics-v3-audit.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-cluster-metrics: + +.. include:: /includes/common-metrics-v3-cluster.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-debug-metrics: + +.. include:: /includes/common-metrics-v3-debug.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-ilm-metrics: + +.. include:: /includes/common-metrics-v3-ilm.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-logger-webhook-metrics: + +.. include:: /includes/common-metrics-v3-logger-webhook.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-notification-metrics: + +.. include:: /includes/common-metrics-v3-notification.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-replication-metrics: + +.. include:: /includes/common-metrics-v3-replication.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-scanner-metrics: + +.. include:: /includes/common-metrics-v3-scanner.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-system-metrics: + +.. include:: /includes/common-metrics-v3-system.md + :parser: myst_parser.sphinx_ + + +.. toctree:: + :titlesonly: + :hidden: + + /operations/monitoring/collect-minio-metrics-using-prometheus + /operations/monitoring/monitor-and-alert-using-influxdb