Reliable Insights

In the previous post we looked at testing rules. You can also test alerts.

Let's take a simple example of an alert with some templating:

wget https://github.com/prometheus/prometheus/releases/download/v2.5.0/prometheus-2.5.0.linux-amd64.tar.gz
tar -xzf prometheus-*.tar.gz
cd prometheus-*
cat >rules.yml <<'EOF'
groups:
 - name: example
   rules:
    - alert: MyAlert
      expr: avg without(instance)(up) < 0.75
      for: 2m
      labels:
        severity: page
      annotations:
        description: 'Only {{$value}} of {{$labels.job}} job is up'
EOF

To test this as before we create a test.yml, with the following content:

rule_files:
  - rules.yml
evaluation_interval: 1m
tests:
 - interval: 1m
   input_series:
    - series: 'up{job="node",instance="foo"}'
      values: '1+0x10'
    - series: 'up{job="node",instance="bar"}'
      values: '1+0x5 0+0x5'
    - series: 'up{job="prometheus",instance="foo"}'
      values: '1+0x10'
   alert_rule_test:
    - alertname: MyAlert
      eval_time: 7m
    - alertname: MyAlert
      eval_time: 8m
      exp_alerts:
       - exp_labels:
           severity: page
           job: node
         exp_annotations:
           description: 'Only 0.5 of node job is up'

Once again you can then run these tests with:

./promtool test rules test.yml

which should return:

Unit Testing: test.yml
  SUCCESS

This is the same input data as in the previous post, but here we're testing alerts instead. The first tests that no alerts are firing for MyAlert at 7 minutes in:

   alert_rule_test:
    - alertname: MyAlert
      eval_time: 7m

The second tests that a single alert is firing a 8 minutes in, and that it has all the right labels and annotations:

    - alertname: MyAlert
      eval_time: 8m
      exp_alerts:
       - exp_labels:
           severity: page
           job: node
         exp_annotations:
           description: 'Only 0.5 of node job is up'

If you expected more alerts to be firing with the same alertname, you would list all of them under exp_alerts.

 

And that's all there is to it. You can test rules and alerts together, and there are more advanced features like specifying what order rule groups should be evaluated in to help simulate rules across federation hierarchies.

Wondering what are good alerts to have? Contact us.