Graphs from Prometheus use the query_range endpoint, and there's a non-trivial amount of confusion that it's more magic than it actually is.
Reliable Insights
A blog on monitoring, scale and operational Sanity
For online serving systems it's fairly well known that you should look for request rate, errors and duration. What about offline processing pipelines though?
Having to maintain dashboards for every Prometheus server you have would be a bit annoying. Thankfully Grafana has a feature for this.
The machine knows its own name, couldn't Prometheus use it?
Having to reconstruct how far a failed cron job had gotten and what exact parameters it was run with can be error prone and time consuming. There is a better way.
Prometheus 2.10 has a new metric to make finding churn easier.
