Reliable Insights

Sometimes you need a machine metric that's not exported yet by the node exporter. The textfile collector can be used to quickly get such metrics graphed in Prometheus.

 

One of my computers was randomly getting turned off. Looking through the logs wasn't fruitful, and as it has always run a bit hot I suspected thermal issues. The node exporter doesn't yet export metrics related to temperature, fans and voltage so that's not something Prometheus can provide out of the box. In addition every motherboard tends to have a different layout, but in this case I only care about one machine so I can do a hack for just this one machine. Using the sensors command I can get:

atk0110-acpi-0
Adapter: ACPI interface
Vcore Voltage: +1.12 V (min = +0.85 V, max = +1.60 V)
 +3.3 Voltage: +3.29 V (min = +2.97 V, max = +3.63 V)
 +5 Voltage: +4.92 V (min = +4.50 V, max = +5.50 V)
 +12 Voltage: +12.05 V (min = +10.20 V, max = +13.80 V)
CPU FAN Speed: 0 RPM (min = 600 RPM)
CHASSIS FAN Speed: 0 RPM (min = 600 RPM)
CPU Temperature: +94.0°C (high = +60.0°C, crit = +95.0°C)
GPU Temperature: +89.0°C (high = +60.0°C, crit = +95.0°C)

A little bit of sed produces output that's compatible with the Prometheus text output format:

$ sensors | sed -ne 's/\..*//; s/ /_/; s/: *+/ /; /Temp/p'
CPU_Temperature 94
GPU_Temperature 89

Finally this can be put in a crontab to run every minute, with a little code to ensure atomicity:

* * * * * sensors | sed -ne 's/\..*//; s/ /_/; s/: *+/ /; /Temp/p' > /var/lib/node_exporter/textfile_collector/lmsensors.prom.$$ && mv /var/lib/node_exporter/textfile_collector/lmsensors.prom.$$ /var/lib/node_exporter/textfile_collector/lmsensors.prom

/var/lib/node_exporter/textfile_collector/ is the default location for the textfile collector to look at for node exporter Debian packages from deb.robustperception.io.

The next time it turns off, I'll be able to check this graph to see if there was a temperature spike beforehand.