docs/source/operations/monitoring/metrics-v2.rst

Metrics Version 2

minio

Table of Contents

Metrics version 2

For metrics version 2, all metrics are available under the base /minio/v2/metrics endpoint, optionally appending an additional path for each category.

For example, the following endpoint returns bucket metrics:

http://HOSTNAME:PORT/minio/v2/metrics/bucket

Replace HOSTNAME:PORT with the FQDN (Fully Qualified Domain Name) and port of the MinIO deployment. For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.

Cluster Metrics

You can scrape cluster-level metrics <minio-available-cluster-metrics> using the following URL endpoint:

http://HOSTNAME:PORT/minio/v2/metrics/cluster

Replace HOSTNAME:PORT with the FQDN (Fully Qualified Domain Name) and port of the MinIO deployment. For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.

Bucket Metrics

MinIO RELEASE.2023-07-21T21-12-44Z

Bucket metrics have moved to use their own, separate endpoint.

RELEASE.2023-08-31T15-31-16Z

You can scrape bucket-level metrics <minio-available-bucket-metrics> using the following URL endpoint:

http://HOSTNAME:PORT/minio/v2/metrics/bucket

Replace HOSTNAME:PORT with the FQDN (Fully Qualified Domain Name) and port of the MinIO deployment. For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.

Resource Metrics

RELEASE.2023-10-07T15-07-38Z

You can scrape resource metrics <minio-available-resource-metrics> using the following URL endpoint:

http://HOSTNAME:PORT/minio/v2/metrics/resource

Replace HOSTNAME:PORT with the FQDN (Fully Qualified Domain Name) and port of the MinIO deployment. For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.

Configure Prometheus to collect and alert using MinIO Metrics

1) Generate a v2 scrape configuration

Use the mc admin prometheus generate command to generate the scrape configuration for use by Prometheus in making scraping requests:

MinIO Server

The following command scrapes metrics for the MinIO cluster.

mc admin prometheus generate ALIAS

Replace ALIAS <mc admin prometheus generate ALIAS> with the alias <mc alias> of the MinIO deployment.

The command returns output similar to the following:

global:
   scrape_interval: 60s

scrape_configs:
   - job_name: minio-job
     bearer_token: TOKEN
     metrics_path: /minio/v2/metrics/cluster
     scheme: https
     static_configs:
     - targets: [minio.example.net]

Nodes

The following command scrapes metrics for a node on the MinIO Server.

mc admin prometheus generate ALIAS node

Replace ALIAS <mc admin prometheus generate ALIAS> with the alias <mc alias> of the MinIO deployment.

global:
   scrape_interval: 60s

scrape_configs:
   - job_name: minio-job-node
     bearer_token: TOKEN
     metrics_path: /minio/v2/metrics/node
     scheme: https
     static_configs:
     - targets: [minio-1.example.net, minio-2.example.net, minio-N.example.net]

Buckets

The following command scrapes metrics for buckets on the MinIO Server.

mc admin prometheus generate ALIAS bucket

Replace ALIAS <mc admin prometheus generate ALIAS> with the alias <mc alias> of the MinIO deployment.

global:
   scrape_interval: 60s

scrape_configs:
   - job_name: minio-job-bucket
     bearer_token: TOKEN
     metrics_path: /minio/v2/metrics/bucket
     scheme: https
     static_configs:
     - targets: [minio.example.net]

Resources

RELEASE.2023-10-07T15-07-38Z

The following command scrapes metrics for resources on the MinIO Server.

mc admin prometheus generate ALIAS resource

Replace ALIAS <mc admin prometheus generate ALIAS> with the alias <mc alias> of the MinIO deployment.

global:
   scrape_interval: 60s

scrape_configs:
   - job_name: minio-job-resource
     bearer_token: TOKEN
     metrics_path: /minio/v2/metrics/resource
     scheme: https
     static_configs:
     - targets: [minio.example.net]

• Set an appropriate scrape_interval value to ensure each scraping operation completes before the next one begins. The recommended value is 60 seconds.

  Some deployments require a longer scrape interval due to the number of metrics being scraped. To reduce the load on your MinIO and Prometheus servers, choose the longest interval that meets your monitoring requirements.

• Set the job_name to a value associated to the MinIO deployment.

  Use a unique value to ensure isolation of the deployment metrics from any others collected by that Prometheus service.

• MinIO deployments started with MINIO_PROMETHEUS_AUTH_TYPE set to "public" can omit the bearer_token field.

• Set the scheme to http for MinIO deployments not using TLS.

• Set the targets array with a hostname that resolves to the MinIO deployment.

  This can be any single node, or a load balancer/proxy which handles connections to the MinIO nodes.

  k8s

  For Prometheus deployments in the same cluster as the MinIO Tenant, you can specify the service DNS name for the minio service.

  For Prometheus deployments external to the cluster, you must specify an ingress or load balancer endpoint configured to route connections to and from the MinIO Tenant.

2) Restart Prometheus with the updated configuration

Append the desired scrape_configs job generated in the previous step to the configuration file:

Cluster

Cluster metrics aggregate node-level metrics and, where appropriate, attach labels to metrics for the originating node.

global:
   scrape_interval: 60s

scrape_configs:
   - job_name: minio-job
     bearer_token: TOKEN
     metrics_path: /minio/v2/metrics/cluster
     scheme: https
     static_configs:
     - targets: [minio.example.net]

Nodes

Node metrics are specific for node-level monitoring. You need to list all MinIO nodes for this configuration.

global:
   scrape_interval: 60s

scrape_configs:
   - job_name: minio-job-node
     bearer_token: TOKEN
     metrics_path: /minio/v2/metrics/node
     scheme: https
     static_configs:
     - targets: [minio-1.example.net, minio-2.example.net, minio-N.example.net]

Bucket

global:
   scrape_interval: 60s

scrape_configs:
   - job_name: minio-job-bucket
     bearer_token: TOKEN
     metrics_path: /minio/v2/metrics/bucket
     scheme: https
     static_configs:
     - targets: [minio.example.net]

Resource

global:
   scrape_interval: 60s

scrape_configs:
   - job_name: minio-job-resource
     bearer_token: TOKEN
     metrics_path: /minio/v2/metrics/resource
     scheme: https
     static_configs:
     - targets: [minio.example.net]

Start the Prometheus cluster using the configuration file:

prometheus --config.file=prometheus.yaml

3) Analyze collected metrics

Prometheus includes an expression browser <prometheus/latest/getting_started/#using-the-expression-browser>. You can execute queries here to analyze the collected metrics.

The following query examples return metrics collected by Prometheus every five minutes for a scrape job named minio-job:

minio_node_drive_free_bytes{job-"minio-job"}[5m]
minio_node_drive_free_inodes{job-"minio-job"}[5m]

minio_node_drive_latency_us{job-"minio-job"}[5m]

minio_node_drive_offline_total{job-"minio-job"}[5m]
minio_node_drive_online_total{job-"minio-job"}[5m]

minio_node_drive_total{job-"minio-job"}[5m]

minio_node_drive_total_bytes{job-"minio-job"}[5m]
minio_node_drive_used_bytes{job-"minio-job"}[5m]

minio_node_drive_errors_timeout{job-"minio-job"}[5m]
minio_node_drive_errors_availability{job-"minio-job"}[5m]

minio_node_drive_io_waiting{job-"minio-job"}[5m]

4) Configure an alert rule using MinIO Metrics

You must configure Alert Rules <prometheus/latest/configuration/alerting_rules/> on the Prometheus deployment to trigger alerts based on collected MinIO metrics.

The following example alert rule files provide a baseline of alerts for a MinIO deployment. You can modify or otherwise use these examples as guidance in building your own alerts.

groups:
- name: minio-alerts
  rules:
  - alert: NodesOffline
    expr: avg_over_time(minio_cluster_nodes_offline_total{job="minio-job"}[5m]) > 0
    for: 10m
    labels:
      severity: warn
    annotations:
      summary: "Node down in MinIO deployment"
      description: "Node(s) in cluster {{ $labels.instance }} offline for more than 5 minutes"

  - alert: DisksOffline
    expr: avg_over_time(minio_cluster_drive_offline_total{job="minio-job"}[5m]) > 0
    for: 10m
    labels:
      severity: warn
    annotations:
      summary: "Disks down in MinIO deployment"
      description: "Disks(s) in cluster {{ $labels.instance }} offline for more than 5 minutes"

In the Prometheus configuration, specify the path to the alert file in the rule_files key:

rule_files:
- minio-alerting.yml

Once triggered, Prometheus sends the alert to the configured AlertManager service.

Dashboards

For v2 metrics, MinIO provides Grafana Dashboards to display the metrics collected by Prometheus. For more information, see minio-grafana

Available v2 metrics

• Cluster Metrics <minio-available-cluster-metrics>
• Bucket Metrics <minio-available-bucket-metrics>
• Resource Metrics <minio-available-resource-metrics>