It’s been a great day at DevOps Days (#devopsdays) in Austin so far. The afternoon was devoted to the Open Space concept, and that was pretty cool. You go up on stage, propose a topic for discussion, pick a time slot, and people decide whether it’s interesting for them or not. I proposed a session on “Beyond the pretty charts – Analytics for the rest of us” and a lot of people seemed interested so I couldn’t back out of moderating it. We had such a great discussion that I wanted to capture some of the points/conclusions here. If you attended the session and want to add some points or comment, please let me know! Since I’m not one for great lyrical writing, I’ll just list these in bullet points:
We’ve moved beyond thresholds. While thresholds are important for cataclysmic events, (database is down!) they’re not so useful for a lot of other things. They make all kinds of unwarranted assumptions about the underlying systems they are monitoring. So what if my disk space is at 91% on a large capacity drive, and has been stable at 91% for a while? I’d much rather know if the disk usage is rapidly increasing. Machine Learning/Analytics is the next wave.
Context is important. Do I really want to be alerted when I know someone is performing maintenance or backups? There was argument on both sides of this one, but the consensus is no. The trick of course is how do you tie your monitoring and alerting system to other events that are occurring (such as maintenance)?
Know your data. This is something I’ve already written about, but it bears repeating. You need to understand the statistical properties of your data in order to determine what kind of analytics to use. For example, it’s important to know if your data is normally distributed. If so, you can leverage a large number of powerful tools and techniques that are geared to this kind of data.
Don’t just look at timeline charts. We’ve fallen into the trap of looking at all the pretty charts as time series charts. When we do that, we end up missing some important characteristics. For example, a simple histogram of the data, instead of just a time chart, can tell you a lot about anomalies and distribution. Using different kinds of visualization is crucial to giving us a different aspect on our data.
Is all data important? Here we got into a great debate on whether you can just look at all the data, or be picky about which feeds are important and which aren’t. This reminded me of the great debate between Noam Chomsky and Peter Norvig of Google. On the one side, some people are saying data is data, let’s analyze everything and figure out the trends. On the other, some people are saying well not all data is important, so let’s figure out what’s important first and understand the underlying model so we don’t waste resources on the rest. Unresolved as far as I know 
We all want to automate. Having humans in the way of detecting and solving DevOps issues doesn’t scale. At some point, we need systems that can detect anomalies before problems become critical, and take appropriate action. We all want to automate. The quote of the day, however, goes to Geoff George who, when asked why doesn’t he automate to the next level, replied: “Because I don’t have the budget”! How true.
I hope that covers the major points. I’m sure I missed a whole bunch of stuff, so let me know. Next post is back to analytics, I promise!
The post Beyond The Pretty Charts – A Report From #devopsdays in Austin appeared first on Metafor Software.
