From ab25dd2059d11a292185f012204da8f507d60e8c Mon Sep 17 00:00:00 2001 From: ravindk89 Date: Wed, 28 Jul 2021 12:43:41 -0400 Subject: [PATCH] Clean up prometheus metric collection tutorial and references --- ...collect-minio-metrics-using-prometheus.rst | 57 ++++------ .../minio-metrics-and-alerts.rst | 102 +++++++----------- .../reference/minio-server/minio-server.rst | 15 +++ 3 files changed, 70 insertions(+), 104 deletions(-) diff --git a/source/monitoring/metrics-alerts/collect-minio-metrics-using-prometheus.rst b/source/monitoring/metrics-alerts/collect-minio-metrics-using-prometheus.rst index 784e462a..93192b0d 100644 --- a/source/monitoring/metrics-alerts/collect-minio-metrics-using-prometheus.rst +++ b/source/monitoring/metrics-alerts/collect-minio-metrics-using-prometheus.rst @@ -49,8 +49,9 @@ Procedure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ MinIO by default requires authentication for requests made to the metrics -endpoints. This step is not required for MinIO deployments started with -:envvar:`MINIO_PROMETHEUS_AUTH_TYPE` set to ``"public"``. +endpoints. While step is not required for MinIO deployments started with +:envvar:`MINIO_PROMETHEUS_AUTH_TYPE` set to ``"public"``, you can still use the +command output for retrieving a Prometheus ``scrape_configs`` entry. Use the :mc-cmd:`mc admin prometheus generate` command to generate a JWT bearer token for use by Prometheus in making authenticated scraping @@ -77,30 +78,14 @@ The command returns output similar to the following: static_configs: - targets: [minio.example.net] -You can specify the output block to the +The ``targets`` array can contain the hostname for any node in the deployment. +For clusters with a load balancer managing connections between MinIO nodes, +specify the address of the load balancer. + +Specify the output block to the :prometheus-docs:`scrape_config ` section of -the Prometheus configuration. Change the ``job_name`` to match a label or -identifier for the MinIO cluster being scraped. - -The :mc-cmd:`mc admin prometheus generate` command generates a scrape config -for only the ``/minio/v2/metrics/cluster`` endpoint. You can use the generated -config to create a node-level metric scraping job: - -.. code-block:: yaml - :class: copyable - - scrape_configs: - - job_name: minio-job - bearer_token: TOKEN - metrics_path: /minio/v2/metrics/node - scheme: https - static_configs: - - targets: [minio-01.example.net] - -Change the ``job_name`` to match a label or identifier for the MinIO node -being scraped. Change the ``static_configs.targets`` to specify the -hostname for the node being scraped. +the Prometheus configuration. 2) Configure and Run Prometheus ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -119,18 +104,12 @@ configuration file: scrape_interval: 15s scrape_configs: - - job_name: minio-cluster-metrics + - job_name: minio-job bearer_token: TOKEN metrics_path: /minio/v2/metrics/cluster scheme: https static_configs: - targets: [minio.example.net] - - job_name: minio-node-01-metrics - bearer_token: TOKEN - metrics_path: /minio/v2/metrics/node - scheme: https - static_configs: - - targets: [minio-01.example.net] Start the Prometheus cluster using the configuration file: @@ -165,11 +144,6 @@ list of published metrics. 4) Visualize Collected Metrics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Prometheus includes a -:prometheus-docs:`graphing interface -` for -visualizing collected metrics. - The :minio-git:`MinIO Console ` supports visualizing collected metrics from Prometheus. Specify the URL of the Prometheus service to the :envvar:`MINIO_PROMETHEUS_URL` environment variable to each MinIO server @@ -178,7 +152,10 @@ in the deployment: .. code-block:: shell :class: copyable - set MINIO_PROMETHEUS_URL="https://prometheus.example.net" + export MINIO_PROMETHEUS_URL="https://prometheus.example.net" + +If you set a custom ``job_name`` for the Prometheus scraping job, you must also +set :envvar:`MINIO_PROMETHEUS_JOB_ID` to match that job name. Restart the deployment using :mc-cmd:`mc admin service restart` to apply the changes. @@ -195,4 +172,8 @@ MinIO also publishes a `Grafana Dashboard `_ for visualizing collected metrics. For more complete documentation on configuring a Prometheus data source for Grafana, see :prometheus-docs:`Grafana Support for Prometheus -`. \ No newline at end of file +`. + +Prometheus includes a :prometheus-docs:`graphing interface +` for +visualizing collected metrics. \ No newline at end of file diff --git a/source/monitoring/metrics-alerts/minio-metrics-and-alerts.rst b/source/monitoring/metrics-alerts/minio-metrics-and-alerts.rst index 92994961..c75d84a0 100644 --- a/source/monitoring/metrics-alerts/minio-metrics-and-alerts.rst +++ b/source/monitoring/metrics-alerts/minio-metrics-and-alerts.rst @@ -8,7 +8,7 @@ Metrics and Alerts .. contents:: Table of Contents :local: - :depth: 1 + :depth: 2 MinIO leverages `Prometheus `__ for metrics and alerts. Prometheus is an Open-Source systems and service monitoring system which @@ -36,39 +36,24 @@ process MinIO metrics for analysis, visualization, and alerting. Metrics ------- -MinIO provides a scraping endpoint for both cluster and node-level metrics. -Create a new :prometheus-docs:`scraping configuration -` for either or -both endpoints to begin collecting metrics from the MinIO deployment. - -MinIO also publishes a `Grafana Dashboard -`_ for visualizing collected -metrics. For more complete documentation on configuring a Prometheus data source -for Grafana, see :prometheus-docs:`Grafana Support for Prometheus -`. - -MinIO by default requires authentication for scraping the metrics endpoints. -Use the :mc-cmd:`mc admin prometheus generate` command to generate the -necessary bearer tokens for use with configuring the -``scrape_configs.bearer_token`` field. You can alternatively disable -metrics endpoint authentication by setting -:envvar:`MINIO_PROMETHEUS_AUTH_TYPE` to ``public``. - -Cluster Metrics -~~~~~~~~~~~~~~~ - -MinIO publishes cluster-level metrics to the following endpoint: +MinIO provides a scraping endpoint for cluster-level metrics: .. code-block:: shell :class: copyable http://minio.example.net:9000/minio/v2/metrics/cluster -Replace ``http://minio.example.net`` with the hostname for the MinIO -node. +Replace ``http://minio.example.net`` with the hostname of any node in the MinIO +deployment. For deployments with a load balancer managing connections between +MinIO nodes, specify the address of the load balancer. + +Create a new :prometheus-docs:`scraping configuration +` to begin +collecting metrics from the MinIO deployment. See +:ref:`minio-metrics-collect-using-prometheus` for a complete tutorial. The following example describes a ``scrape_configs`` entry for collecting -cluster metrics: +cluster metrics. .. code-block:: yaml :class: copyable @@ -83,7 +68,7 @@ cluster metrics: .. list-table:: :stub-columns: 1 - :widths: 30 70 + :widths: 20 80 :width: 100% * - ``job_name`` @@ -95,55 +80,40 @@ cluster metrics: Omit this field if the MinIO deployment was started with :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` set to ``public``. - * - ``static_configs[n].targets`` + * - ``targets`` - The endpoint for the MinIO deployment. You can specify any node in the deployment for collecting cluster metrics. For clusters with a load balancer managing connections between MinIO nodes, specify the address of the load balancer. -Node Metrics -~~~~~~~~~~~~ +MinIO by default requires authentication for scraping the metrics endpoints. +Use the :mc-cmd:`mc admin prometheus generate` command to generate the +necessary bearer tokens for use with configuring the +``scrape_configs.bearer_token`` field. You can alternatively disable +metrics endpoint authentication by setting +:envvar:`MINIO_PROMETHEUS_AUTH_TYPE` to ``public``. -MinIO publishes node-level metrics to the following endpoint: +Visualizing Metrics +~~~~~~~~~~~~~~~~~~~ -.. code-block:: yaml - :class: copyable +The MinIO Console uses the metrics collected by Prometheus to populate the +Dashboard metrics: - http://minio.example.net:9000/minio/v2/metrics/node +.. image:: /images/minio-console-dashboard.png + :width: 600px + :alt: MinIO Console Dashboard displaying Monitoring Data + :align: center -Replace ``http://minio.example.net`` with the hostname for the MinIO -node. +Set the :envvar:`MINIO_PROMETHEUS_URL` environment variable to the URL of the +Prometheus service to allow the Console to retrieve and display collected +metrics. See :ref:`minio-metrics-collect-using-prometheus` for a complete +example. -The following example describes a ``scrape_configs`` entry for collecting -node metrics. You must specify one job for each node in the MinIO deployment: - -.. code-block:: shell - :class: copyable - - scrape_configs: - - job_name: minio-job - bearer_token: - metrics_path: /minio/v2/metrics/node - scheme: https - static_configs: - - targets: ['minio.example.net:9000'] - -.. list-table:: - :stub-columns: 1 - :widths: 30 70 - :width: 100% - - * - ``job_name`` - - The name of the scraping job. - - * - ``bearer_token`` - - The JWT token generated by :mc-cmd:`mc admin prometheus generate`. - - Omit this field if the MinIO node was started with - :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` set to ``public``. - - * - ``static_configs[n].targets`` - - The endpoint for the MinIO node. +MinIO also publishes a `Grafana Dashboard +`_ for visualizing collected +metrics. For more complete documentation on configuring a Prometheus data source +for Grafana, see :prometheus-docs:`Grafana Support for Prometheus +`. .. _minio-metrics-and-alerts-available-metrics: diff --git a/source/reference/minio-server/minio-server.rst b/source/reference/minio-server/minio-server.rst index e52ecf9e..c53239a8 100644 --- a/source/reference/minio-server/minio-server.rst +++ b/source/reference/minio-server/minio-server.rst @@ -279,6 +279,21 @@ MinIO Console: The MinIO Console populates the :guilabel:`Dashboard` with cluster metrics using the ``minio-job`` Prometheus scraping job. + If you are using a standalone MinIO Console process, this variable + corresponds to ``CONSOLE_PROMETHEUS_URL``. + +.. envvar:: MINIO_PROMETHEUS_JOB_ID + + *Optional* + + Specify the custom Prometheus job ID used for + :ref:`scraping MinIO metrics `. + + MinIO defaults to ``minio-job``. + + If you are using a standalone MinIO Console process, this variable + corresponds to ``CONSOLE_PROMETHEUS_JOB_ID``. + .. envvar:: MINIO_LOG_QUERY_URL *Optional*