Metrics v3 (#1289)

Add metrics v3 to the Metrics and Alerts page and other relevant
locations, deprecate metrics v2.

- Updated the metrics extract script to parse the [list of v3 metrics
from
GitHub](https://github.com/minio/minio/blob/master/docs/metrics/v3.md).
- Remove references to MinIO Console metrics.
- Make `:mc-release:` show friendly release name text like
`:minio-release:` does.

Staged:

Linux

http://192.241.195.202:9000/staging/DOCS-1266/linux/operations/monitoring/metrics-and-alerts.html

k8s

http://192.241.195.202:9000/staging/DOCS-1266/k8s/operations/monitoring/metrics-and-alerts.html

fixes https://github.com/minio/docs/issues/1266

source/default-conf.py

@@ -62,7 +62,7 @@ extlinks = {
     's3-api'          : ('https://docs.aws.amazon.com/AmazonS3/latest/API/%s', None),
     'iam-docs'        : ('https://docs.aws.amazon.com/IAM/latest/UserGuide/%s', None),
     'minio-release'   : ('https://github.com/minio/minio/releases/tag/%s', '%s'),
-    'mc-release'      : ('https://github.com/minio/mc/releases/tag/%s', None),
+    'mc-release'      : ('https://github.com/minio/mc/releases/tag/%s', '%s'),
     'prometheus-docs' : ('https://prometheus.io/docs/%s', None),
     'podman-docs'     : ('https://docs.podman.io/en/latest/%s', None),
     'podman-git'      : ('https://github.com/containers/podman/%s', None),

source/operations/monitoring/collect-minio-metrics-using-prometheus.rst

@@ -375,23 +375,6 @@ In the Prometheus configuration, specify the path to the alert file in the ``rul
 
 Once triggered, Prometheus sends the alert to the configured AlertManager service.
 
-5) (Optional) Configure MinIO Console to Query Prometheus
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The Console also supports displaying time-series and historical data by querying a :prometheus-docs:`Prometheus <prometheus/latest/getting_started/>` service configured to scrape data from the MinIO deployment. 
-
-.. image:: /images/minio-console/console-metrics.png
-   :width: 600px
-   :alt: MinIO Console displaying Prometheus-backed Monitoring Data
-   :align: center
-
-To enable historical data visualization in MinIO Console, set the following environment variables on each node in the MinIO deployment:
-
-- Set :envvar:`MINIO_PROMETHEUS_URL` to the URL of the Prometheus service
-- Set :envvar:`MINIO_PROMETHEUS_JOB_ID` to the unique job ID assigned to the collected metrics
-
-Restart the MinIO deployment and visit the :ref:`Monitoring <minio-console-monitoring>` pane to see the historical data views.
-
 Dashboards
 ----------
 

source/operations/monitoring/metrics-and-alerts.rst

@@ -10,85 +10,95 @@ Metrics and Alerts
 
 .. contents:: Table of Contents
    :local:
-   :depth: 3
+   :depth: 2
+
+.. admonition:: Metrics Version 2 Deprecated
+   :class: note
+
+   Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 replaces the deprecated :ref:`metrics version 2 <minio-metrics-v2>`.
 
 MinIO publishes cluster and node metrics using the :prometheus-docs:`Prometheus Data Model <concepts/data_model/>`.
 You can use any scraping tool to pull metrics data from MinIO for further analysis and alerting.
 
-MinIO provides scraping endpoints for the following metric groups:
+For metrics version 3, all metrics are available under the base ``/minio/metrics/v3`` endpoint by appending an additional path for each category.
 
-.. tab-set::
+For example, the following endpoint returns audit metrics:
 
-   .. tab-item:: Cluster Metrics
+.. code-block:: shell
-
-      You can scrape :ref:`cluster-level metrics <minio-available-cluster-metrics>` using the following URL endpoint:
-
-      .. code-block:: shell
    :class: copyable
 
-         http://HOSTNAME:PORT/minio/v2/metrics/cluster
+   http://HOSTNAME:PORT/minio/metrics/v3/audit
 
-      Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment.
+Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment.
-      For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.
+For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.
 
+By default, MinIO requires authentication to scrape the metrics endpoints.
+To generate the needed bearer tokens, use :mc:`mc admin prometheus generate`.
+You can also disable metrics endpoint authentication by setting :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` to ``public``.
 
-   .. tab-item:: Bucket Metrics
+MinIO provides the following scraping endpoints, relative to the base URL:
 
-      .. versionchanged:: RELEASE.2023-08-31T15-31-16Z
+.. list-table::
+   :header-rows: 1
+   :widths: 30 70
+   :width: 100%
 
-      You can scrape :ref:`bucket-level metrics <minio-available-bucket-metrics>` using the following URL endpoint:
+   * - Category
+     - Path
 
-      .. code-block:: shell
+   * - API
-         :class: copyable
+     - ``/api/requests``
        
-         http://HOSTNAME:PORT/minio/v2/metrics/bucket
+       ``/bucket/api``
 
-      Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment.
+   * - Audit
-      For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.
+     - ``/audit``
 
-   .. tab-item:: Resource Metrics
+   * - Cluster
+     - ``/cluster/config``
        
-      .. versionadded:: RELEASE.2023-10-07T15-07-38Z 
+       ``/cluster/erasure-set``
        
-      You can scrape :ref:`resource metrics <minio-available-resource-metrics>` using the following URL endpoint:
+       ``/cluster/health``
        
-      .. code-block:: shell
+       ``/cluster/iam``
-         :class: copyable
        
-         http://HOSTNAME:PORT/minio/v2/metrics/resource
+       ``/cluster/usage/buckets``
        
-      Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment.
+       ``/cluster/usage/objects``
-      For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.
 
+   * - Debug
+     - ``/debug/go``
 
-MinIO by default requires authentication for scraping the metrics endpoints.
+   * - ILM
-Use the :mc:`mc admin prometheus generate` command to generate the necessary bearer tokens. 
+     - ``/ilm``
-You can alternatively disable metrics endpoint authentication by setting :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` to ``public``.
 
-.. _minio-console-metrics:
+   * - Logger webhook
+     - ``/logger/webhook``
 
-MinIO Console Metrics Dashboard
+   * - Notification
--------------------------------
+     - ``/notification``
 
-The :ref:`MinIO Console <minio-console-monitoring>` provides a point-in-time metrics dashboard by default:
+   * - Replication
+     - ``/replication``
        
-.. image:: /images/minio-console/console-metrics-simple.png
+       ``/bucket/replication``
-   :width: 600px
-   :alt: MinIO Console with Point-In-Time Metrics
-   :align: center
 
-The Console also supports displaying time-series and historical data by querying a :prometheus-docs:`Prometheus <prometheus/latest/getting_started/>` service configured to scrape data from the MinIO deployment. 
+   * - Scanner
-Specifically, the MinIO Console uses :prometheus-docs:`Prometheus query API <prometheus/latest/querying/api/>` to retrieve stored metrics data and display the following visualizations:
+     - ``/scanner``
 
-- :guilabel:`Usage` - provides historical and on-demand visualization of overall usage and status
+   * - System
-- :guilabel:`Traffic` - provides historical and on-demand visualization of network traffic
+     - ``/system/drive``
-- :guilabel:`Resources` - provides historical and on-demand visualization of  resources (compute and storage)
-- :guilabel:`Info` - provides point-in-time status of the deployment
        
-.. image:: /images/minio-console/console-metrics.png
+       ``/system/memory``
-   :width: 600px
+       
-   :alt: MinIO Console displaying Prometheus-backed Monitoring Data
+       ``/system/cpu``
-   :align: center
+       
+       ``/system/network/internode``
+       
+       ``/system/process``
+
+For a complete list of metrics for each endpoint, see :ref:`Available Metrics <minio-metrics-and-alerts-available-metrics>`.
 
 .. cond:: k8s
 
@@ -118,29 +128,69 @@ Available Metrics
 MinIO publishes a number of metrics at the cluster, node, or bucket levels.
 Each metric includes a label for the MinIO server which generated that metric.
 
-.. versionchanged:: MinIO RELEASE.2023-07-21T21-12-44Z
+- :ref:`API Metrics <minio-available-v3-api-metrics>`
+- :ref:`Audit Metrics <minio-available-v3-audit-metrics>`
+- :ref:`Cluster Metrics <minio-available-v3-cluster-metrics>`
+- :ref:`Debug Metrics <minio-available-v3-debug-metrics>`
+- :ref:`ILM Metrics <minio-available-v3-ilm-metrics>`
+- :ref:`Logger webhook Metrics <minio-available-v3-logger-webhook-metrics>`
+- :ref:`Notification Metrics <minio-available-v3-notification-metrics>`
+- :ref:`Replication Metrics <minio-available-v3-replication-metrics>`
+- :ref:`Scanner Metrics <minio-available-v3-scanner-metrics>`
+- :ref:`System Metrics <minio-available-v3-system-metrics>`
 
-   Bucket metrics have moved to use their own, separate endpoint.
 
-- :ref:`Cluster Metrics <minio-available-cluster-metrics>`
+.. _minio-available-v3-api-metrics:
-- :ref:`Bucket Metrics <minio-available-bucket-metrics>`
-- :ref:`Resource Metrics <minio-available-resource-metrics>`
 
-.. _minio-available-cluster-metrics:
+.. include:: /includes/common-metrics-v3-api.md
-
-.. include:: /includes/common-metrics-cluster.md
    :parser: myst_parser.sphinx_
 
-.. _minio-available-bucket-metrics:
+.. _minio-available-v3-audit-metrics:
 
-.. include:: /includes/common-metrics-bucket.md
+.. include:: /includes/common-metrics-v3-audit.md
    :parser: myst_parser.sphinx_
 
-.. _minio-available-resource-metrics:
+.. _minio-available-v3-cluster-metrics:
 
-.. include:: /includes/common-metrics-resource.md
+.. include:: /includes/common-metrics-v3-cluster.md
    :parser: myst_parser.sphinx_
 
+.. _minio-available-v3-debug-metrics:
+
+.. include:: /includes/common-metrics-v3-debug.md
+   :parser: myst_parser.sphinx_
+
+.. _minio-available-v3-ilm-metrics:
+
+.. include:: /includes/common-metrics-v3-ilm.md
+   :parser: myst_parser.sphinx_
+
+.. _minio-available-v3-logger-webhook-metrics:
+
+.. include:: /includes/common-metrics-v3-logger-webhook.md
+   :parser: myst_parser.sphinx_
+
+.. _minio-available-v3-notification-metrics:
+
+.. include:: /includes/common-metrics-v3-notification.md
+   :parser: myst_parser.sphinx_
+
+.. _minio-available-v3-replication-metrics:
+
+.. include:: /includes/common-metrics-v3-replication.md
+   :parser: myst_parser.sphinx_
+
+.. _minio-available-v3-scanner-metrics:
+
+.. include:: /includes/common-metrics-v3-scanner.md
+   :parser: myst_parser.sphinx_
+
+.. _minio-available-v3-system-metrics:
+
+.. include:: /includes/common-metrics-v3-system.md
+   :parser: myst_parser.sphinx_
+
+
 .. toctree::
    :titlesonly:
    :hidden:

source/operations/monitoring/metrics-v2-deprecated.rst

@@ -0,0 +1,88 @@
+:orphan:
+.. _minio-metrics-v2:
+
+=================
+Metrics Version 2
+=================
+
+.. default-domain:: minio
+
+.. contents:: Table of Contents
+   :local:
+   :depth: 3
+
+.. admonition:: Metrics Version 2 Deprecated
+   :class: note
+
+   Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, :ref:`metrics version 3 <minio-metrics-and-alerts>` replaces the deprecated metrics version 2.
+
+The following sections describe the deprecated endpoints and metrics.
+
+.. tab-set::
+
+   .. tab-item:: Cluster Metrics
+
+      You can scrape :ref:`cluster-level metrics <minio-available-cluster-metrics>` using the following URL endpoint:
+
+      .. code-block:: shell
+         :class: copyable
+
+         http://HOSTNAME:PORT/minio/v2/metrics/cluster
+
+      Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment.
+      For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.
+
+
+   .. tab-item:: Bucket Metrics
+
+      .. versionchanged:: MinIO RELEASE.2023-07-21T21-12-44Z
+
+         Bucket metrics have moved to use their own, separate endpoint.
+
+      .. versionchanged:: RELEASE.2023-08-31T15-31-16Z
+
+      You can scrape :ref:`bucket-level metrics <minio-available-bucket-metrics>` using the following URL endpoint:
+
+      .. code-block:: shell
+         :class: copyable
+
+         http://HOSTNAME:PORT/minio/v2/metrics/bucket
+
+      Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment.
+      For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.
+
+
+   .. tab-item:: Resource Metrics
+
+      .. versionadded:: RELEASE.2023-10-07T15-07-38Z 
+
+      You can scrape :ref:`resource metrics <minio-available-resource-metrics>` using the following URL endpoint:
+
+      .. code-block:: shell
+         :class: copyable
+
+         http://HOSTNAME:PORT/minio/v2/metrics/resource
+
+      Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment.
+      For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.
+
+
+- :ref:`Cluster Metrics <minio-available-cluster-metrics>`
+- :ref:`Bucket Metrics <minio-available-bucket-metrics>`
+- :ref:`Resource Metrics <minio-available-resource-metrics>`
+
+.. _minio-available-cluster-metrics:
+
+.. include:: /includes/common-metrics-cluster.md
+   :parser: myst_parser.sphinx_
+
+.. _minio-available-bucket-metrics:
+
+.. include:: /includes/common-metrics-bucket.md
+   :parser: myst_parser.sphinx_
+
+.. _minio-available-resource-metrics:
+
+.. include:: /includes/common-metrics-resource.md
+   :parser: myst_parser.sphinx_
+

sync-minio-server-docs.sh

@@ -4,6 +4,9 @@ set -e
 set -x
 
 function main() {
+
+   # Extract metrics v2 list
+
    curl --retry 10 -Ls https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/list.md | csplit - /"# Bucket Metrics"/
    mv xx00 source/includes/common-metrics-cluster.md
 
@@ -14,6 +17,58 @@ function main() {
 
    mv xx00 source/includes/common-metrics-bucket.md
    mv xx01 source/includes/common-metrics-resource.md
+
+   # Extract metrics v3 list
+
+   # Get the full list
+
+   curl --retry 10 -Ls https://raw.githubusercontent.com/minio/minio/master/docs/metrics/v3.md | csplit - /"## Metric Categories"/
+
+   # Ignore xx00, contains intro text
+   # Overwritten in second csplit anyway
+
+   # Split remainder of file into categories by H3
+   # Number of times to match must be exact
+   cat xx01 | csplit - /"^### "/ '{9}'
+
+   # Copy each category into an include file
+   # Order below must match section order in source file
+
+   # Ignore xx00 again, more intro text
+   rm xx00
+
+   # API (Renamed from Request)
+   # Hopefully ex works the same on both Mac and Linux, unlike sed
+   ex -sc '%s/### Request metrics/### API metrics/' -c 'x' xx01
+   mv xx01 source/includes/common-metrics-v3-api.md
+
+   # Audit
+   mv xx02 source/includes/common-metrics-v3-audit.md
+
+   # Cluster
+   mv xx03 source/includes/common-metrics-v3-cluster.md
+
+   # Debug
+   mv xx04 source/includes/common-metrics-v3-debug.md
+
+   # ILM
+   mv xx05 source/includes/common-metrics-v3-ilm.md
+
+   # Logger Webhook
+   mv xx06 source/includes/common-metrics-v3-logger-webhook.md
+
+   # Notification
+   mv xx07 source/includes/common-metrics-v3-notification.md
+
+   # Replication
+   mv xx08 source/includes/common-metrics-v3-replication.md
+
+   # Scanner
+   mv xx09 source/includes/common-metrics-v3-scanner.md
+
+   # System
+   mv xx10 source/includes/common-metrics-v3-system.md
+
 }
 
 main "$@"