Having to maintain dashboards for every Prometheus server you have would be a bit annoying. Thankfully Grafana has a feature for this.
Variables in Grafana (previously known as templates) allow parameterisation of a dashboard via a drop-down menu. Often this is used to switch between machines or services, so that you can have per-machine dashboards without needing to create a dashboard every time a new machine appears. They're also stored in URL parameters, so could be linked from alert notifications or wiki pages.
One type of variable is the data source variable, which as the name indicates lets data sources be parameterised. I have setup some Prometheus data sources for demonstration in a Grafana 6.2.5:
Then I create a new dashboard, and go to its settings:
I Add variable, with the name Source, the type of the variable is Datasource and the type of data source is Prometheus:
All the Prometheus data sources are listed, plus the "default" data source. Here I only want the prom- Prometheus servers, so I can put in an Instance name filter regular expression of prom-.*:
By having a standard naming scheme for data sources, it's possible to select just the ones I want. For example I might use the pattern prom-ENV-SERVICE-LOCATION, likely following the information contained in external_labels.
I Add the variable, go back to the dashboard, and add a simple query of up to the initial panel, and set the Query to be against $Source:
If I now go back to the dashboard the Source is there as a dropdown, and changing it will switch which data source is queried:
There's also &var-Source=prom-dev-ie in my browser's URL, which I could also change if I wanted to link to the dashboard with a specific data source.
By using this approach it's easy to have a central Grafana that pulls data from all your individual Prometheus servers, with dashboards that can look at whichever is the required Prometheus with a click. This is a simple but powerful setup.
Wondering how to best organise your dashboards? Contact us.
No comments.