wal: handle Aurora's pg_ls_waldir() unsupported error gracefully (#1312)

* wal: skip the WAL metrics where Aurora doesn't support pg_ls_waldir()

Some Amazon Aurora PostgreSQL versions reject pg_ls_waldir() with
feature_not_supported (SQLSTATE 0A000), so the wal collector, which is
enabled by default, failed on every scrape there. On Aurora, detected
with instance.isAurora from #1311, treat that error as no WAL metrics;
Aurora versions where pg_ls_waldir() works keep reporting them, and any
other error is still returned.

Fixes #1309.

Signed-off-by: Pavel K <megativ3@gmail.com>

* wal: return ErrNoData where Aurora doesn't support pg_ls_waldir()

Other collectors return ErrNoData when there is no data for a known
reason, e.g. Aurora collectors on instances that aren't Aurora, so the
collector is reported as not successful with a debug log instead of an
error log.

Signed-off-by: Pavel K <megativ3@gmail.com>

---------

Signed-off-by: Pavel K <megativ3@gmail.com>

CHANGELOG.md

@@ -13,6 +13,7 @@ enabling wrapping later will still create this one-time reset boundary.
 * [CHANGE] Wrap non-negative 64-bit counters at `2^53` to preserve single-unit precision by @gnanirahulnutakki in https://github.com/prometheus-community/postgres_exporter/pull/1351
 * [CHANGE] stat_replication: add `pid` label to disambiguate replication connections that otherwise share identical labels by @sysadmind in https://github.com/prometheus-community/postgres_exporter/pull/1353
 * [BUGFIX] Fix `long_running_transactions` to count only transactions older than a configurable threshold, by @ArthurSens in https://github.com/prometheus-community/postgres_exporter/pull/1379, based on the contribution by @moreinhardt in https://github.com/prometheus-community/postgres_exporter/pull/1210
+* [BUGFIX] Fix the `wal` collector failing on every scrape on Amazon Aurora PostgreSQL versions that don't support `pg_ls_waldir()` by @megative in https://github.com/prometheus-community/postgres_exporter/pull/1312
 * [FEATURE] Export `pg_settings.pending_restart` as `pg_settings_pending_restart{name}` from the `settings` collector by @MarcWort in https://github.com/prometheus-community/postgres_exporter/pull/1396
 * [FEATURE] Add `aurora_replica_status` collector for Amazon Aurora PostgreSQL by @megative in https://github.com/prometheus-community/postgres_exporter/pull/1311
 

collector/pg_wal.go

@@ -16,7 +16,9 @@ package collector
 import (
 	"context"
 	"database/sql"
+	"errors"
 
+	"github.com/lib/pq"
 	"github.com/prometheus/client_golang/prometheus"
 )
 
@@ -69,6 +71,12 @@ func (c PGWALCollector) Update(ctx context.Context, instance *instance, ch chan<
 	var size sql.NullInt64
 	err := row.Scan(&segments, &size)
 	if err != nil {
+		// Some Amazon Aurora PostgreSQL versions reject pg_ls_waldir() as
+		// feature_not_supported. Report no data there instead of failing
+		// every scrape; other Aurora versions run it as usual.
+		if instance.isAurora && isFeatureNotSupported(err) {
+			return ErrNoData
+		}
 		return err
 	}
 	ch <- prometheus.MustNewConstMetric(
@@ -83,3 +91,10 @@ func (c PGWALCollector) Update(ctx context.Context, instance *instance, ch chan<
 	}
 	return nil
 }
+
+// isFeatureNotSupported reports whether err is a PostgreSQL
+// feature_not_supported (0A000) error.
+func isFeatureNotSupported(err error) bool {
+	var pqErr *pq.Error
+	return errors.As(err, &pqErr) && pqErr.Code == "0A000"
+}

collector/pg_wal_test.go

@@ -14,9 +14,11 @@ package collector
 
 import (
 	"context"
+	"errors"
 	"testing"
 
 	"github.com/DATA-DOG/go-sqlmock"
+	"github.com/lib/pq"
 	"github.com/prometheus/client_golang/prometheus"
 	dto "github.com/prometheus/client_model/go"
 	"github.com/smartystreets/goconvey/convey"
@@ -100,3 +102,42 @@ func TestPgWALCollectorZeroSegments(t *testing.T) {
 		t.Errorf("there were unfulfilled exceptions: %s", err)
 	}
 }
+
+func TestPgWALCollectorErrors(t *testing.T) {
+	notSupported := &pq.Error{Code: "0A000", Message: "pg_ls_waldir() is currently not supported for Aurora"}
+	permissionDenied := &pq.Error{Code: "42501", Message: "permission denied for function pg_ls_waldir"}
+
+	for _, tc := range []struct {
+		name     string
+		isAurora bool
+		err      error
+		wantErr  error
+	}{
+		{name: "Aurora rejects pg_ls_waldir", isAurora: true, err: notSupported, wantErr: ErrNoData},
+		{name: "not supported outside Aurora", isAurora: false, err: notSupported, wantErr: notSupported},
+		{name: "other errors on Aurora", isAurora: true, err: permissionDenied, wantErr: permissionDenied},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			db, mock, err := sqlmock.New()
+			if err != nil {
+				t.Fatalf("Error opening a stub db connection: %s", err)
+			}
+			defer db.Close()
+
+			inst := &instance{db: db, isAurora: tc.isAurora}
+			mock.ExpectQuery(sanitizeQuery(pgWALQuery)).WillReturnError(tc.err)
+
+			ch := make(chan prometheus.Metric, 2)
+			err = PGWALCollector{}.Update(context.Background(), inst, ch)
+			if !errors.Is(err, tc.wantErr) {
+				t.Errorf("PGWALCollector.Update returned %v, want %v", err, tc.wantErr)
+			}
+			if len(ch) != 0 {
+				t.Errorf("Expected no metrics, got %d", len(ch))
+			}
+			if err := mock.ExpectationsWereMet(); err != nil {
+				t.Errorf("there were unfulfilled exceptions: %s", err)
+			}
+		})
+	}
+}

docs/configuration.md

@@ -53,7 +53,7 @@ The per-table and per-index collectors scale with your schema — one set of ser
 | `stat_replication` | enabled | Seen from the primary: for each connected standby, how many WAL bytes it is behind on sent/written/flushed/replayed, plus the primary's current WAL position. From `pg_stat_replication`. |
 | `replication_slots` | enabled | Per slot: whether it's active, and how much WAL the slot is holding back. An inactive slot quietly retaining WAL is a classic way to fill a disk. From `pg_replication_slots`. |
 | `stat_archiver` | enabled | WAL archiving: count of archived and failed segments, and seconds since the last successful archive. Catches a broken `archive_command` before it fills `pg_wal`. From `pg_stat_archiver`. |
-| `wal` | enabled | Number of WAL segment files and their total size on disk, via `pg_ls_waldir()`. |
+| `wal` | enabled | Number of WAL segment files and their total size on disk, via `pg_ls_waldir()`. On Amazon Aurora PostgreSQL versions that don't support `pg_ls_waldir()`, it reports nothing. |
 | `stat_wal_receiver` | disabled | Seen from a standby: the WAL receiver's LSNs, timeline, upstream node, and last message send/receipt times. Enable this on replicas for a view of the receiving side. From `pg_stat_wal_receiver`. |
 | `xlog_location` | disabled | Current WAL write position (or replay position on a replica) as a byte offset. Superseded by `stat_replication`; retained for old setups and pre-10 servers. |
 | `aurora_replica_status` | disabled | **Amazon Aurora PostgreSQL only.** For each instance in the cluster: replica lag behind the writer, replay latency, and pending page reads. On non-Aurora servers it reports nothing. From `aurora_replica_status()`. |