Reliable Insights

Prometheus offers integrations with systems like PagerDuty, Email and Hipchat for alert notifications - but what if you want do something that's not supported out of the box? The Alertmanager's generic web hook has got you covered.

I'd like to be told when my internet connections go down, rather than finding out the hard way. To do this I setup alerts for connection failure, and had my media centre speak the text of alert notifications.

For the sake of brevity, I'm going to assume you already know how to download and compile Prometheus binaries. First off, I run the blackbox_exporter as root:

sudo ./blackbox_exporter

This will take in HTTP requests from Prometheus, do an ICMP ping and return the result.

Next I create a Prometheus configuration, and run it:

cat <<'EOF' > prometheus.yml
global:
  scrape_interval: 1s
  evaluation_interval: 1s

rule_files:
  - internet.rules
scrape_configs:
  - job_name: 'blackbox_magnet'
    metrics_path: /probe
    params:
      module: [icmp]
      target: [8.8.4.4]
    static_configs:
      - targets:
        - 127.0.0.1:9115
    relabel_configs:
      - source_labels: []
        regex: (.*)
        target_label: instance
        replacement: Magnet
  - job_name: 'blackbox_upc'
    metrics_path: /probe
    params:
      module: [icmp]
      target: [8.8.8.8]
    static_configs:
      - targets:
        - 127.0.0.1:9115
    relabel_configs:
      - source_labels: []
        regex: (.*)
        target_label: instance
        replacement: UPC
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - "127.0.0.1:9093"
    
EOF
cat <<'EOF' > internet.rules
groups:
- name: internet.rules
  rules:
  - alert: InternetDown
    expr: probe_success == 0
    annotations:
      DESCRIPTION: '{{$labels.instance}} internet connection down'
      SUMMARY: '{{$labels.instance}} down'
EOF
./prometheus

Once per second Prometheus will poll the blackbox_exporter to see if the connections are working, and if not will fire alerts to the alertmanager. I've configured my firewall so that 8.8.4.4 and 8.8.8.8 go out via the Magnet and UPC internet connections respectively.

I need an alertmanager to take in alerts and send alert notifications:

cat <<'EOF' > alertmanager.conf
notification_config {
  name: "audio"
  webhook_config {
    url: "http://192.168.5.42:1234/"
    send_resolved: true
  }
}
aggregation_rule {
  repeat_rate_seconds: 3600
  notification_config_name: "audio"
}
EOF
./alertmanager

And finally on my media box, a small python script to receive and process the notifications:

cat <<'EOF' > alarm.py
#!/usr/bin/python

import cgi
import json
import pipes
import subprocess
from BaseHTTPServer import BaseHTTPRequestHandler
from BaseHTTPServer import HTTPServer

class AlarmHandler(BaseHTTPRequestHandler):
    def do_POST(self):
        self.send_response(200)
        self.end_headers()
        data = json.loads(self.rfile.read(int(self.headers['Content-Length'])))
        text = "%s %s" % (data["alert"][0]["summary"], data["status"])
        print text
        subprocess.call(
            "espeak %s -p 10 --stdout | aplay -Dhdmi_complete -" % pipes.quote(text), shell=True)

if __name__ == '__main__':
   httpd = HTTPServer(('', 1234), AlarmHandler)
   httpd.serve_forever()

EOF
python alarm.py

The script uses eSpeak to convert the alert summary to audio. There's no limit to what you can do with alert notifications using the generic web hook.

Note: The alert rule and old alertmanager configuration formats have changed since this was first posted, so this will no longer work out of the box with the latest binary versions.