* fix(collector): keep collecting temperatures without CPU power status
Apple Silicon does not implement IOPMCopyCPUPowerStatus, so
fetchCPUPowerStatus returns kIOReturnNotFound there. Update returned that
error straight away, which aborted the collector before updateTemperatures
ran, so no temperature metrics were collected at all. The error was also
not ErrNoData, so it was logged at error level on every scrape.
On an M5 Pro, Update emitted 0 metrics and failed, while updateTemperatures
on its own returned 52 temperature metrics.
Treat kIOReturnNotFound as a system that does not report CPU power status:
skip the three CPU power metrics, log at debug level and carry on to the
temperature sensors. Any other non-success return code is still returned as
an error. Systems that do report CPU power status are unaffected.
This does not make the CPU power metrics available on Apple Silicon, since
the underlying API provides no data. It only stops their absence from
suppressing the temperature metrics.
Adds a regression test covering the case.
Signed-off-by: SaiPisey2 <piseysai0202@gmail.com>
* fix(collector): disambiguate colliding thermal sensor names
Collecting temperatures on Apple Silicon surfaced a second problem that
was previously unreachable, because Update returned before
updateTemperatures ever ran.
A sensor is labelled with its IOHID "Product" property, which is not
unique. On an M5 Pro, 52 temperature-reporting services carry only 17
distinct product names, so the registry rejects the colliding samples and
logs an error on every scrape.
The collisions are of two kinds. Six "gas gauge battery" services are
distinct sensors that merely share a name; they have distinct
LocationIDs. The "PMU tdie*" services come in threes that share both the
name and the LocationID, and report readings that differ by a few tenths
of a degree, so they are separate sensing elements rather than the same
one listed repeatedly. Dropping either kind loses readings.
Following the approach taken for hwmon in #3646, read all sensors first,
then qualify a name shared by several services with its location, falling
back to the service registry ID when the location does not separate them
either. Registry IDs are unique per service, which the OS guarantees and
which held for every service observed here. Names that do not collide are
left untouched.
The endpoint now reports all 52 readings with unique label sets and no
gather errors, where before the change 35 of them were rejected.
Also update collector/fixtures/e2e-output-darwin.txt for
node_scrape_collector_success{collector="thermal"}, which is now 1
because the collector no longer fails.
resolveSensorNames is kept free of cgo so that the labelling rules are
covered by table-driven tests, which matters because the CI runner
reports no thermal sensors at all.
Signed-off-by: SaiPisey2 <piseysai0202@gmail.com>
* test(collector): satisfy the stricter linters in the thermal test
Drop a no-op drain of an already closed channel and build the metric key
with a strings.Builder.
Signed-off-by: SaiPisey2 <piseysai0202@gmail.com>
---------
Signed-off-by: SaiPisey2 <piseysai0202@gmail.com>
* Add resctrl collector
Expose the Linux resctrl monitoring counters (Intel RDT, AMD PQoS,
ARM MPAM): per L3 domain memory bandwidth (mbm_total_bytes,
mbm_local_bytes) and last level cache occupancy (llc_occupancy).
Parsing is done by the procfs resctrlfs package (procfs v0.22.0).
The collector is disabled by default because the resctrl filesystem
must be mounted first (mount -t resctrl resctrl /sys/fs/resctrl). The
mountpoint is configurable via --collector.resctrl.path.
Counters a CPU does not support (for example mbm_local_bytes on Arm
MPAM) or that read "Unavailable" are skipped per sample. The hardware
counters wrap; Prometheus treats the wrap as a counter reset, so no
correction is applied.
Signed-off-by: René Treffer <treffer@measite.de>
* Describe steps needed to use the resctrl exporter
Signed-off-by: René Treffer <treffer@measite.de>
* Shorten resctrl documentation
Signed-off-by: René Treffer <treffer@measite.de>
---------
Signed-off-by: René Treffer <treffer@measite.de>
* collector: add tainted collector for /proc/sys/kernel/tainted
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
* go.mod: bump procfs to pick up KernelTainted() (#844)
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
* go.mod: bump procfs to pick up linux build tag fix (#847)
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
* collector/tainted: use year-less copyright header
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
* collector/tainted: address review feedback and document collector
Store procfs.FS on the collector instead of opening it each scrape.
Add matching !notainted build tag on the implementation file.
Document the tainted collector and node_kernel_tainted metrics in README.
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
* README: drop tainted collector section, keep table entry only
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
* go.mod: align with master after merge (procfs v0.22.0, toolkit v0.19.0)
Fixes build and BSD e2e failures caused by stale dependency versions
left over from a bad merge.
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
Signed-off-by: Arun Srinivasan <arunpiter@gmail.com>
Co-authored-by: Arun S <arun.srinivasan@flipkart.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Bump minimum Go version
Update to Go 1.26 and fix linting issues.
Signed-off-by: Ben Kochie <superq@gmail.com>
* Bump promtool version.
Signed-off-by: Ben Kochie <superq@gmail.com>
---------
Signed-off-by: Ben Kochie <superq@gmail.com>
* Update FreeBSD release
Use lastest major version for FreeBSD.
Signed-off-by: Ben Kochie <superq@gmail.com>
* Fix FreeBSD end-to-end test
Add `node_zfs_arcstats_c_min_bytes` to non-deterministic list.
Signed-off-by: Ben Kochie <superq@gmail.com>
---------
Signed-off-by: Ben Kochie <superq@gmail.com>
The kernel reports btree_read_average_duration_us in microseconds, but the
value was scaled as nanoseconds, making the metric 1000x too small.
Signed-off-by: Finomosec <1665799+Finomosec@users.noreply.github.com>
Co-authored-by: Ben Kochie <superq@gmail.com>
The Darwin end-to-end test scrapes the real macos-latest runner and diffs
the result against collector/fixtures/e2e-output-darwin.txt. We no longer
see disk4 on the nodes and that breaks the e2e tests.
This PR removes the usage of disk4 and the related fixtures:
-node_disk_read_errors_total{device="disk4"} 0
-node_disk_read_retries_total{device="disk4"} 0
-node_disk_write_errors_total{device="disk4"} 0
-node_disk_write_retries_total{device="disk4"} 0
This PR also adds a dump of the attached interfaces and disks
so we can more easily discover issues in the future.
Signed-off-by: René Treffer <treffer@measite.de>
* fix(e2e): resolve the Darwin fixture path on bash 3.2
The per-OS fixture path was built with "${fixture_metrics::-4}". A
negative substring length needs bash 4.2, and macOS still ships bash 3.2,
where the expansion fails, fixture_metrics keeps pointing at the Linux
fixture, and ./end-to-end-test.sh -u run on a Mac overwrites
collector/fixtures/e2e-output.txt with Darwin output.
"${fixture_metrics%.txt}" is equivalent and works on both.
Also strip node_thermal_temperature_celsius along with the other
non-deterministic metrics. Its sensor names and values are per-machine,
so a developer running the suite on real hardware sees the whole sensor
list as a diff. The CI runner reports no thermal sensors, so this makes
no difference there.
Signed-off-by: SaiPisey2 <piseysai0202@gmail.com>
* Apply suggestion from @SuperQ
Signed-off-by: Ben Kochie <superq@gmail.com>
---------
Signed-off-by: SaiPisey2 <piseysai0202@gmail.com>
Signed-off-by: Ben Kochie <superq@gmail.com>
Co-authored-by: Ben Kochie <superq@gmail.com>
* netstat: Use procfs parsers for netstat, snmp and snmp6
Replace the hand-written parsers for /proc/net/netstat, /proc/net/snmp
and /proc/net/snmp6 with the parsers from prometheus/procfs
(Proc.Netstat, Proc.Snmp and Proc.Snmp6). The statistics are read via
/proc/self/net, which is equivalent to /proc/net (both are
network-namespace local views of the current process), so the collected
data is unchanged.
Metrics are exposed by iterating the procfs statistics structs, the
same approach the NFS collector uses: fields are *float64 and only the
ones present on the system are exported, matching the previous
behavior. The --collector.netstat.fields flag and the exported metric
names are unchanged; the e2e golden output is identical.
Signed-off-by: neoLsH <43921685+neoLsH@users.noreply.github.com>
* netstat: use explicit package-level metric descriptors
Replace the per-scrape prometheus.NewDesc calls with a pre-built map of
explicit descriptors (netstat_descs_linux.go, generated from the procfs
statistics structs), as requested in review. Collection still reflects over
the procfs structs but looks up the descriptor by "<protocol>_<field>", so
descriptors are allocated once and are source-visible literals for the
AST-based doc generation. Metric names, help strings and types are unchanged.
Signed-off-by: 霏承 <huangleshu.hls@alibaba-inc.com>
* Bump procfs to v0.22.0 to populate legacy TcpExt descriptors
procfs v0.22.0 includes prometheus/procfs#863, which models the 10 legacy
TcpExt counters. With this bump, the explicit descriptors already present in
netstat_descs_linux.go start populating for users whose field regex selects
them, restoring parity with older kernels.
Part of #3796.
Signed-off-by: 霏承 <huangleshu.hls@alibaba-inc.com>
* Add generator and drift test for the netstat descriptor table
netstat_descs_linux.go was previously committed without its generator. Add an
in-repo generator (netstat_descs_gen.go, a build-ignored program run via
GOOS=linux go generate ./collector) that regenerates the explicit descriptor
table from the procfs structs, mirroring the go:generate pattern used elsewhere
in the Prometheus ecosystem.
Add TestNetStatDescsInSync, a drift test that fails if the committed table
diverges from the procfs structs (e.g. after a procfs upgrade) until the table
is regenerated.
Part of #3796.
Signed-off-by: 霏承 <huangleshu.hls@alibaba-inc.com>
---------
Signed-off-by: neoLsH <43921685+neoLsH@users.noreply.github.com>
Signed-off-by: 霏承 <huangleshu.hls@alibaba-inc.com>
Both cpu_linux and cpufreq export cpu frequencies under the same name.
This makes the metric mutually exclusive.
A previous PR made it possible to enable both which can cause flapping
help texts.
This PR re-introduces mutual-exclusive behavior for the frequencies.
A new flag `collector.cpufreq.enable-cpufreq-prefix` can be used to
migrate all cpufreq metrics to node_cpufreq to avoid the collision.
Signed-off-by: René Treffer <treffer@measite.de>
Refactor node_exporter's entrypoint to build its configuration and
run via the exporter-toolkit bootstrap runner, as groundwork for
making node_exporter easier to embed outside the standalone binary.
Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>
The Debug call added in #3734 passes printf verbs and positional arguments to
slog, which does not format them. The format string is logged verbatim and the
arguments are consumed as key/value pairs, so controller and channel numbers
become attribute keys: a duplicate key when the two are equal, and otherwise a
key whose name changes on every iteration. Under --log.format=json that is an
object with a repeated key, and parsers silently drop one of the values. The
csrow number and the error, the two things worth reading, are the only parts
that survive, and they appear as values of meaningless keys.
%w compounds it: that is an fmt.Errorf directive and not a valid Printf verb,
left behind when the error return this replaced was converted to a log line.
Pass the values as named attributes instead. Metric output is unchanged.
Signed-off-by: Adam Butler <github@at-blacknight.com>
Update the ksmd collector to use pre-defined metrics descriptions
instead of auto-generated. This allows for improved help text
taken from the kernel documentation. This also avoids memory allocations
each time `NewKsmdCollector()` is called.
Signed-off-by: Ben Kochie <superq@gmail.com>
Expose /sys/kernel/mm/ksm/general_profit as node_ksmd_general_profit_bytes.
This file (available since Linux 6.4, EL9) reports the net memory saved by KSM
after subtracting the overhead of rmap_item tracking structures.
Also gracefully skip any missing sysfs files instead of failing the entire
collector, since general_profit and potentially other files may not exist
on older kernels.
Assisted-by: claude-4.6-opus
Signed-off-by: Michal Skrivanek <michal.skrivanek@redhat.com>
