Scraping targets across datacenters will make things better, right?
Reliable Insights
A blog on monitoring, scale and operational Sanity
How can you view older data, while keeping your monitoring reliable?
On a regular basis a potential Prometheus user says they need a different architecture to make things reliable or scalable. Let's look at that.
Having to reconstruct how far a failed cron job had gotten and what exact parameters it was run with can be error prone and time consuming. There is a better way.
Prometheus has gained a number of features to limit the impact of expensive PromQL queries.
While not a problem specific to Prometheus, being affected by the open files ulimit is something you're likely to run into at some point.
Worried that your application metrics might suddenly explode in cardinality? sample_limit can save you.
Prometheus is architected for reliability of alerting, how do you set it up?
When designing a monitoring system and the datastore that goes with it, it can be tempting to go straight for a clustered highly consistent approach. But is that the best approach?
