Expose how far each instance, including the cross-Region replicas of an
Aurora global database, lags behind the writer, from
aurora_global_db_instance_status(), converted to seconds. The writer
reports NULL lag, so no lag is exported for it. aws_region is a label
because the same server_id can appear in several Regions.
The collector is disabled by default and returns no data on non-Aurora
servers.
Aurora collector names get their own block in config.go, so gofmt
does not realign the main list when longer names are added.
Signed-off-by: Pavel K <megativ3@gmail.com>
* Redact DSN secrets from logged errors
Errors from the URL parser and the pq driver quote the connection string
they were given, password included, and were logged verbatim alongside
the already-redacted dsn field. Add config.RedactError to strip the
password (including encoded forms and fragments of malformed key=value
values) from such errors, and apply it at every log site that handles a
DSN error in both the exporter and collector paths. Stop dsnFromURL and
parseFingerprint from echoing their input.
Fixes#1408
Signed-off-by: Joe Adams <github@joeadams.io>
* Remove copyright dates
Signed-off-by: Joe Adams <github@joeadams.io>
* Update redact for newline support
Signed-off-by: Joe Adams <github@joeadams.io>
---------
Signed-off-by: Joe Adams <github@joeadams.io>
Expose where each database's block reads come from (Aurora storage,
the Optimized Reads cache or local storage) and the time spent on each,
from aurora_stat_database(), with times converted to seconds.
aurora_stat_database() also returns every pg_stat_database column; only
Aurora's own columns are exported, the rest is already covered by the
stat_database collector.
The function only exists in Aurora PostgreSQL 14.9+ and 15.4+, so the
collector checks for it first and returns no data on older versions.
The collector is disabled by default and returns no data on non-Aurora
servers.
Signed-off-by: Pavel K <megativ3@gmail.com>
* stat_activity_autovacuum: add datname and schemaname labels
relname is only the part of the autovacuum query after the first dot, so
workers on same-named tables in different schemas or databases emitted
identical series and failed the scrape as duplicates.
Fixes#953
Signed-off-by: spicysharkco <321699413+spicysharkco@users.noreply.github.com>
* stat_activity_autovacuum: keep spaces in schema names
The schemaname label was the last space-separated word before the first
dot, so schemas such as "A schema" and "B schema" both became "schema",
and two workers on same-named tables in them gave duplicate series again.
Strip the command prefix that autovac_report_activity and
autovac_report_workitem write (VACUUM, VACUUM ANALYZE, ANALYZE or BRIN
summarize) with an anchored REGEXP_REPLACE and take everything before
the first dot. relname is unchanged and the query stays valid on
PostgreSQL 13 and later. A dot inside a schema name still ends the
schema early.
The integration test gets two workers in schemas "A schema" and
"B schema".
Signed-off-by: spicysharkco <321699413+spicysharkco@users.noreply.github.com>
* stat_activity_autovacuum: test the ANALYZE and BRIN summarize prefixes
The integration test only had VACUUM and VACUUM ANALYZE rows, so it
still passed with ANALYZE or BRIN summarize dropped from the prefix
pattern. Add one row for each.
Signed-off-by: spicysharkco <321699413+spicysharkco@users.noreply.github.com>
---------
Signed-off-by: spicysharkco <321699413+spicysharkco@users.noreply.github.com>
Aurora PostgreSQL rejects pg_last_xact_replay_timestamp() and, on
readers, pg_last_wal_receive_lsn() as not supported (0A000), so the
replication collector failed on every scrape there, on the writer and
on readers alike.
On Aurora the collector now queries only pg_is_in_recovery() and emits
pg_replication_is_replica. The lag metrics aren't emitted there: the
replica lag on Aurora comes from the aurora_replica_status collector.
Errors of that query are returned as usual.
Aurora is checked up front rather than after the error: these functions
aren't supported on Aurora, so a fallback would only add a failed
query, and an error in the Aurora log, to every scrape.
Based on the contribution by @dannotripp in #1274.
Signed-off-by: Pavel K <megativ3@gmail.com>
* Add aurora_stat_logical_wal_cache collector for Amazon Aurora PostgreSQL
Expose WAL cache hits, misses and read requests of each logical
replication slot from aurora_stat_logical_wal_cache().
The function only exists in Aurora PostgreSQL 11.17+, 12.12+, 13.8+,
14.7+ and 15.2+. Checking for Aurora alone would make the collector fail
on every scrape on older versions, so it first checks that the function
exists with to_regproc, the same way Aurora itself is detected, and
returns no data when it doesn't. The check is a small helper that other
version-dependent Aurora collectors can reuse.
The collector is disabled by default and returns no data on non-Aurora
servers.
Aurora collector names get their own block in config.go, so gofmt
does not realign the main list when longer names are added.
Signed-off-by: Pavel K <megativ3@gmail.com>
* collector: rename auroraFunctionExists to functionExists
The helper isn't Aurora specific, so it could be used by collectors
for other types of instances as well.
Signed-off-by: Pavel K <megativ3@gmail.com>
---------
Signed-off-by: Pavel K <megativ3@gmail.com>
Closes#1372. Part of #1371.
Adds CLI flags for the parts of the Postgres connection config that
aren't secrets: --datasource.uri (replaces DATA_SOURCE_URI), plus
--datasource.uri-file, --datasource.user-file and --datasource.pass-file
(replacing the *_FILE env vars) for values sourced from mounted files.
The user and password values themselves stay env-var only
(DATA_SOURCE_USER / DATA_SOURCE_PASS), since CLI flags are readable via
/proc and the Kubernetes API and would leak the secret itself. The
*-file flags carry only a path, the same pattern Prometheus uses for
password_file, so no secret ever reaches the command line.
Flag and env var reading now lives in cmd/postgres_exporter rather than
the exporter package, which we're trying to shrink. exporter.GetDataSources
stays for embedders, marked deprecated with a migration note, and will be
removed in v0.22.0.
The four env vars that gained a flag are deprecated: the exporter warns
at startup when one is set, the CHANGELOG maps each to its replacement,
and the docs now configure the exporter by flag.
Each flag falls back to its env var when unset, so behavior is unchanged
when only env vars are set. Precedence is unchanged: DATA_SOURCE_NAME
wins over everything, *_FILE wins over the plain value.
Signed-off-by: Nicolas Takashi <nicolas.takashi@dash0.com>
Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
Expose the storage (durability) lag and RPO lag of each secondary
cluster in an Aurora global database from aurora_global_db_status(),
converted to seconds. The primary cluster reports -1, so no lag is
exported for it.
The collector is disabled by default and returns no data on non-Aurora
servers.
Aurora collector names get their own block in config.go, so gofmt
does not realign the main list when longer names are added.
Signed-off-by: Pavel K <megativ3@gmail.com>
* wal: skip the WAL metrics where Aurora doesn't support pg_ls_waldir()
Some Amazon Aurora PostgreSQL versions reject pg_ls_waldir() with
feature_not_supported (SQLSTATE 0A000), so the wal collector, which is
enabled by default, failed on every scrape there. On Aurora, detected
with instance.isAurora from #1311, treat that error as no WAL metrics;
Aurora versions where pg_ls_waldir() works keep reporting them, and any
other error is still returned.
Fixes#1309.
Signed-off-by: Pavel K <megativ3@gmail.com>
* wal: return ErrNoData where Aurora doesn't support pg_ls_waldir()
Other collectors return ErrNoData when there is no data for a known
reason, e.g. Aurora collectors on instances that aren't Aurora, so the
collector is reported as not successful with a debug log instead of an
error log.
Signed-off-by: Pavel K <megativ3@gmail.com>
---------
Signed-off-by: Pavel K <megativ3@gmail.com>
Expose replica lag, replay latency and pending read IOs from Aurora's
aurora_replica_status() function. Metrics are converted to base units
(seconds).
Aurora is detected in instance.setup() alongside current_database(),
using to_regproc('aurora_version') so that regular PostgreSQL servers
don't log an error on every scrape. The collector is disabled by
default and returns no data on non-Aurora servers.
Signed-off-by: Pavel K <megativ3@gmail.com>
Removed the newInstance function in collector/instance.go.
The internal sql.Open call with a hardcoded "postgres" driver
did not perform any DSN verification, as true connection and
DSN validation requires db.Ping().
This aligns with the previously expressed maintainers intent
to defer database connectivity checks to a later stage.
See https://github.com/prometheus-community/postgres_exporter/pull/882
Signed-off-by: chief librarian <323671055+chieflibrarian@users.noreply.github.com>
* Run database-scoped collectors against every discovered database
collector.NewRuntime only ever built one PostgresCollector, bound to
DataSourceNames[0]. This meant per-database collectors like
stat_user_tables, statio_user_tables and statio_user_indexes silently
never ran for any database beyond the first configured DSN, whether
that list came from a static, comma-separated DATA_SOURCE_NAME or
from --auto-discover-databases.
Naively running every collector against every discovered database
isn't safe either: most collectors (bgwriter, wal, stat_activity,
stat_statements, locks, ...) read catalog views or functions that
already report on the whole instance from any single connection, so
running them again from a second database would re-report the same
rows and the registry would panic on the duplicate metric (as already
happened once, for stat_activity, in #533).
This adds an explicit scope to each registered collector (clusterScope
vs databaseScope) and a multiInstanceCollector that fans the
database-scoped ones out across every DSN exporter.Exporter currently
targets (static list or, with autodiscovery enabled, freshly
enumerated databases), while cluster-scoped collectors still only ever
run once, against the originally configured DSN. The target list is
re-evaluated on every scrape, so databases created or dropped after
startup are picked up without restarting the exporter, matching the
autodiscovery behavior already used by the legacy exporter.Exporter
metrics.
Also gives statio_user_indexes a datname label, matching its sibling
per-database collectors: without it, index metrics from two databases
with a same-named index would collide once run concurrently.
Signed-off-by: Matthew Wimpelberg <120263653+mwimpelberg28@users.noreply.github.com>
* Address review: rename cluster scope to server, drop multiInstanceCollector
sysadmind flagged two issues in review: PostgreSQL doesn't have a "cluster",
so collectorScope should say "server" instead; and the multiInstanceCollector
built a full server+database collector set for every explicitly configured
DSN, which panics the registry if two of those DSNs turn out to be the same
physical server (e.g. two databases on one server, each listed separately in
DATA_SOURCE_NAME).
Follows the plan from #1326 instead: exactly one primary connection
(DataSourceNames[0]) ever runs server-scoped collectors, so they can never be
duplicated no matter how many databases are discovered. PostgresCollector
gains a WithDatabaseDiscovery option that, when auto-discovery is enabled,
queries pg_database on that same primary connection and runs only the
database-scoped collectors against every other database found, over its own
connection, reconciled on every scrape.
This drops collector's dependency on exporter.Target/TargetDSNs, since
discovery is now entirely self-contained within the collector package.
Signed-off-by: Matthew Wimpelberg <120263653+mwimpelberg28@users.noreply.github.com>
* Address review: fold database discovery into PostgresCollector, fix datname collisions
sysadmind flagged that database_discovery.go duplicated DSN-rewriting logic
that config.DSN already owns, and that the whole databaseDiscovery type
(its own collect loop, cache of *PostgresCollector, mutex) duplicated what
PostgresCollector already knows how to do: distinguish server- from
database-scoped collectors and run them.
config.DSN gains WithDatabase, built on its existing parser, and instance
gains withDatabase to use it, so no connection-string parsing is duplicated.
databaseDiscovery, buildDatabaseCollector, and onlyScope/scopeFilter are all
gone; collectFromConnection takes an optional scope filter so the single
PostgresCollector can run every collector against its primary connection and
just the database-scoped ones against each discovered database. No state
needs to persist across scrapes: the primary connection already opens and
closes a fresh sql.DB every scrape, so discovered-database connections do
the same instead of being cached.
Also fixes the brittle "unknown" datname fallback in pg_statio_user_indexes:
datname comes from current_database(), which Postgres guarantees is never
NULL, so it's scanned as a plain string instead of falling back to a shared
sentinel that could collide across databases.
Testing locally (as sysadmind asked) surfaced a related bug the redesign
didn't originally cover: pg_scrape_collector_duration_seconds and
pg_scrape_collector_success carried no datname label, so running the same
collector against multiple databases produced the exact duplicate-metric
registry error from sysadmind's repro. Fixed by adding a datname label to
both, sourced from a new instance.database field populated via
current_database() during setup().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Matthew Wimpelberg <120263653+mwimpelberg28@users.noreply.github.com>
* Address review: bound discovery concurrency, fix socket DSNs, gate statio_user_indexes datname
ArthurSens and Copilot's latest review pass flagged four issues in the
database-discovery path added by this PR:
- collectDiscoveredDatabases launched one goroutine per discovered
database with no bound, so a server with many databases could open more
connections than it can spare in a single scrape. It's now bounded by a
semaphore sized by a new AutoDiscoverDatabasesMaxConcurrency config field
(--auto-discover-databases.max-concurrency, default 10).
- The per-database goroutine called instance.setup() with no context at
all, so a stalled connect/auth could block wg.Wait() well past
CollectionTimeout. setup() (and queryVersion) now take a context, and
each discovered database gets one deadline covering both connecting and
scraping.
- config.DSN.WithDatabase always serialized through GetConnectionString's
URL form, but pq's own URL parser rejects a percent-encoded "/" in the
host — so any discovered-database connection built from a Unix-socket
primary DSN (the documented `host=/var/run/postgresql/` form) produced a
DSN string sql.Open couldn't even parse. GetConnectionString now detects
a socket-shaped host and falls back to key=value serialization, the only
form pq accepts for it.
- pg_statio_user_indexes gained an unconditional datname label last round
to fix a duplicate-metric collision, but that collision can only happen
once database discovery runs the collector concurrently against more
than one database. Installs that never enable
--auto-discover-databases got a new label on an existing metric for no
reason. The label (on this collector and on the
pg_scrape_collector_duration_seconds/success meta-metrics added for the
same reason) is now only added when WithDatabaseDiscovery is active.
Separately, ArthurSens and Copilot also asked why NewRuntime still only
passes DataSourceNames[0] into the native PostgresCollector, so a
comma-separated multi-server DATA_SOURCE_NAME still only gets
database-scoped collectors on the first server. That line is unchanged
from master and predates this PR — it's a pre-existing limitation of the
native collector framework (this PR only fans database-scoped collectors
out across databases discovered on a single server), not something to fix
here. Replying on the review thread to flag it as a good follow-up instead.
Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: Matthew Wimpelberg <120263653+mwimpelberg28@users.noreply.github.com>
* Address review: enforce CollectionTimeout across discovery batches
collectDiscoveredDatabases gave each batch its own fresh
context.WithTimeout(context.Background(), ...), so a scrape could run well
past CollectionTimeout once enough batches of discovered databases had run.
Thread a single ctx through the whole call via a new forEachDatabase
worker-pool helper so the deadline actually bounds the total time.
Signed-off-by: Matthew Wimpelberg <120263653+mwimpelberg28@users.noreply.github.com>
Assisted-by: Claude <noreply@anthropic.com>
* Address review: replace DataSourceNames []string with a single DataSourceName
DataSourceNames[0] has been hardcoded wherever the registry-based collectors
(and /probe) pick a primary connection, ever since database auto-discovery
was added; the array never actually enabled discovery across multiple
servers there. Meanwhile the legacy Exporter genuinely iterated the whole
slice to scrape multiple independent servers from one process via a
comma-separated DATA_SOURCE_NAME.
Collapse this into a single DataSourceName string everywhere, removing the
ambiguity for good. This is a breaking change: DATA_SOURCE_NAME containing a
comma is now a startup error. Multi-server scraping from a single exporter
is the job of the multi-target /probe endpoint, not this field.
Signed-off-by: Matthew Wimpelberg <120263653+mwimpelberg28@users.noreply.github.com>
Assisted-by: Claude <noreply@anthropic.com>
* Address review: drop env var for auto-discover-databases.max-concurrency
Per review feedback, new flags shouldn't gain env var bindings going
forward since they aren't required for security reasons. Keep the flag
CLI-only and drop the README mention of PG_EXPORTER_AUTO_DISCOVER_DATABASES_MAX_CONCURRENCY.
Signed-off-by: Matthew Wimpelberg <120263653+mwimpelberg28@users.noreply.github.com>
Assisted-by: Claude <noreply@anthropic.com>
* Revert "Address review: replace DataSourceNames []string with a single DataSourceName"
This reverts commit b19a55eea484bd7bf9565dd9e7f2e5bbb18c7643.
Per @ArthurSens on #1378: the legacy exporter package genuinely depends on
DATA_SOURCE_NAME accepting a comma-separated list to scrape multiple
independent servers from a single process. Collapsing it to a single DSN
would silently break that path for existing users. Go back to accepting
the array here; the actual deprecation (if any) should be its own,
intentional follow-up PR.
Signed-off-by: Matthew Wimpelberg <120263653+mwimpelberg28@users.noreply.github.com>
Assisted-by: Claude <noreply@anthropic.com>
* Address review: extract collect closure, skip redundant version query
Pull the closure passed to forEachDatabase out into a named
collectDatabase variable for readability, and have withDatabase carry
over the primary's already-known version so setup() skips re-querying
it for each discovered database on the same server.
Signed-off-by: Matthew Wimpelberg <120263653+mwimpelberg28@users.noreply.github.com>
Assisted-by: Claude <noreply@anthropic.com>
---------
Signed-off-by: Matthew Wimpelberg <120263653+mwimpelberg28@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Report settings that have been changed but only take effect after a
server restart as pg_settings_pending_restart{name} from the existing
settings collector. The flag is selected in the settings query itself
and carried in the pgSetting struct, so it is emitted from the same
loop that reports the value metrics, without an extra query.
The vartype filter moves from the SQL WHERE clause into
pgSetting.metric(): rows of every vartype are scanned, so pending
restarts are reported for settings beyond the bool/integer/real ones
exported as value metrics (e.g. wal_level is an enum); rows without a
numeric vartype are skipped silently, and the existing warning is kept
for broken values.
Signed-off-by: MarcWort <113890636+MarcWort@users.noreply.github.com>
Ensure any row preparation errors are validated by returning rows.Err()
after the rows.Next() loop finishes in collector/pg_stat_database.
Added a unit test and refactored the way functions return rows.Err()
within the codebase for consistency.
Signed-off-by: chief librarian <323671055+chieflibrarian@users.noreply.github.com>
* Create user facing documentation
Create a docs/ directory with user facing documentation. This covers getting started, running with docker, specifying secrets, configuration, and database permissions. This is all focused on the perspective of a user or someone deploying the exporter.
Signed-off-by: Joe Adams <github@joeadams.io>
* Update getting started to show multiple options
Clean up duplication on docker commands. Make the docs README more useful as a standalone page.
Signed-off-by: Joe Adams <github@joeadams.io>
* Update configuration docs
Add more details on each flag and collector to help the user understand the purpose.
Signed-off-by: Joe Adams <github@joeadams.io>
* Add frontmatter
Signed-off-by: Joe Adams <github@joeadams.io>
* Review feedback
Remove deprecated option doc and /-/reload endpoint (unimplemented).
Signed-off-by: Joe Adams <github@joeadams.io>
* Add 2 new flags in user docs configuration
Signed-off-by: Joe Adams <github@joeadams.io>
* Fix docs link
Signed-off-by: Joe Adams <github@joeadams.io>
* Fix correctness in docs
Signed-off-by: Joe Adams <github@joeadams.io>
---------
Signed-off-by: Joe Adams <github@joeadams.io>
Fix loggableDSN redaction of sensitive values when passed by query string parameters. The config.DSN was already safe for this purpose, so instead of duplicating logic, this moves loggableDSN to use that type instead.
fixes#643
Signed-off-by: Joe Adams <github@joeadams.io>
The stat_activity_autovacuum collector scanned xact_start into a plain
float64, so a NULL value (possible for autovacuum workers) failed the
whole collector with "converting NULL to float64 is unsupported". Scan
into sql.NullFloat64 and skip the row when it is NULL, matching the
pg_stat_walreceiver / pg_stat_database collectors.
Fixes#952
Signed-off-by: Dean Chen <862469039@qq.com>
Copy LICENSE and NOTICE into the Docker image, matching what the
release tarballs already ship and the prometheus/prometheus Dockerfile
convention.
Fixes#1206
Signed-off-by: Dean Chen <862469039@qq.com>
* Refactor Postgres exporter to make it re-usable as a Go library
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Re add Go and Process collectors
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Move Runtime to existing collector package
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Remove Config pointer in NewRuntime
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Remove ProbeCollector dead code
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Turn usage of unvalidated config into a compilation error
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Fail early on misconfigured timeout duration
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Consolidate collector default configuration in config package
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Stop using global for Auth config handler
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Address comments
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Add missing test
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
---------
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* stat_replication: add pid label to disambiguate replication connections
application_name, client_addr, state, and slot_name are not a unique key
for pg_stat_replication rows. Multiple standbys with the same application_name,
client_addr, state, and slot_name (empty) were causing duplicate metrics which
would get rejected by the registry.
This regressed visibly in 0.20.1: 0.20.0 had the stat_replication query
erroring on every scrape ("column slot_name does not exist", #1310), so
the collector never actually emitted rows. #1313 fixed that query error,
which let real data flow again and exposed this pre-existing label
collision for the first time.
One downside is that the additional label could increase series churn because
every time the walsender reconnects, the pid will change and therefore create
a new series.
Fixes#1352
Signed-off-by: Joe Adams <github@joeadams.io>
* Add PR# in CHANGELOG
Signed-off-by: Joe Adams <github@joeadams.io>
---------
Signed-off-by: Joe Adams <github@joeadams.io>
