Prometheus 0.16.0 has been released with a whopping 312 commits and 89 changes by 18 contributors since 0.15.1. That's a lot to swallow, so let's take a look at main changes and improvements.
Service Discovery
First the features I'm most excited about: Amazon EC2, Marathon and Kubernetes have been added as options for service discovery. While all three are initially marked experimental, they offer the ability to automatically have Prometheus detect new targets if you're running on one of these systems. For example this allows Prometheus to monitor EC2 instances managed by Auto Scaling.
Additional metadata has been added to other service discovery methods to give more options for relabelling. Consul service discovery no longer requires you to specify the services to return, if you pass it an empty list of services it'll return them all.
Speaking of relabelling there's a new relabel action called labelmap. This can be used to copy labels from EC2 tags or Kubernetes labels without having to know the names of the labels in advance. It's now also possible to relabel query parameters, which is used by the blackbox_exporter and snmp_exporter. Finally the __tmp prefix for labels has been reserved for the user for use as temporary labels.
In short it's now even easier to integrate Prometheus into your existing systems and automatically scrape new targets!
Scraping
There's several new options for authenticating to targets: HTTP Basic Auth, bearer tokens and SSL client certs. In addition SSL server certificate verification can be disabled with insecure_skip_verify, and a proxy server specified with proxy_url. These offers several options for those who wish to scrape over an untrusted network.
PromQL
There's one change to the core language, the bool modifier was added as an option for comparison operators. Rather than filtering it returns 0 or 1 depending on whether the comparison failed or succeeded. For example previously to count how many values of x for each job were over 5 you'd have to do:
sum by (job)(((x > 5) * 0 + 1) or (x * 0))
which can now be replaced by:
sum by (job)(x > bool 5)
There are three new functions. predict_linear() is the most generally applicable, it allows you to predict what the value of a gauge will be in the future. Check out my previous post on reducing noise from disk space alerts for how to use it.
label_replace() allows labels to be extracted from other labels with regular expressions, most famously used in Conway's Life. Outside of esoteric programming, it's useful for when target metrics have mismatched label schemes.
vector() is the inverse of scalar(), converting a scalar into a no-label vector. This was non-trivial previously.
Things to be aware of
There are eleven breaking changes in the 0.16.0 release. The ones you're most likely to run into:
- Regexes in relabelling and selectors are now anchored (consoles template regexes remain unanchored)
- Strings now support escape sequences, and backticks allow for multi-line strings
- Queries no longer interpolate values, rather it uses the next oldest value
- Global labels are now called external labels, and are applied at alert/federation/remote storage time rather than scrape time
Other changes
There's a large number of bugfixes, tweaks and improvements. Rather than listing them all, you can read the full release notes.
No comments.