Having recently perused a few trade shows and viewed lots of industry resources, there’s a lot of discussion around analytics. Looking at some IT Operations system demos, there are quite a few slick interfaces with some eye-popping reports. Really, it would seem from viewing these that an IT Ops professional could produce advanced reports for just about anything they’d need. Yet many of these systems store all of this data, and do all of the transformations and queries, without a separate data warehouse? So, why would anyone require a data warehouse, for IT Operations?
IT Operations has grown to a level of maturity, that analytics are becoming the norm. But I won’t go onto detail on that, as I’ve recently written an analytics blog post that covers it. What I do want to explore is why one would want to do those analytics on a data warehouse vs. the transactional system. For the record, Zenoss Service Dynamics does incorporate an IT Operations Data Warehouse. With that in mind, Data Warehouses make sense, for the following reasons:
- Scalability – Offload server & I/O bound tasks associated with querying and reporting on servers & I/O not used by the transaction processing systems. In other words, let your IT Ops monitoring system do its job, without getting bogged down by resource-intensive analytics stuff.
- Performance - Data models that are ideal for querying and reporting are not appropriate for transaction processing. By splitting this function, you get faster individual analytics & transaction systems, vs. a slow generic system that covers both functions.
- More Data – To store data from a longer span of time than can efficiently be held in a transaction processing system. How long should IT Ops transactional systems store their data, especially if they’re doing real or near real time polling? The reality is, without a data warehouse, most IT Ops monitoring systems must purge this data regularly, for performance reasons.
- Standardization – To provide a repository of normalized transaction data that can be reported against, without modifying the transaction processing systems
There are certainly other primers, but these may be the ones that are most significant for IT Operations. Numbers 1-3 are significant for any IT Ops system that attempts to perform analytics in their transactional systems. By adding the analytics overhead to their transactional systems, they’ll adversely impact performance and their ability to scale. And, too be effective, they’ll really have to limit the data stored. So, what’s the point? If we look at the industry now, we have a couple approaches to dealing with lots of data:
- No Data Warehouse – These vendors crunch their snazzy reports and analytics, on the transactional systems. This impacts scale and performance heavily, and will lead to organizations having to choose between effective monitoring, and good analytics. Really, it’s a dead-end game. As analytics requirements grow (eg. capacity planning, tracking SLAs, etc.), they will eventually need a warehouse. But organizations will be so invested in customized reports & such for these systems that the transition will be extremely painful.
- 3rd Party IT Ops Data Warehouse – Whether by buying one off the shelf, or by building their own, organizations can bolt a 3rd party warehouse on their IT Ops system. The standardization & normalization of data (eg. ETL) for these implementations is usually painful, and they’re hard to maintain. Though some “frameworks” promote these integrated capabilities, they are really 3rd party solutions that have been bolted on, through acquisition.
Some still see an IT Ops data warehouse as overkill, but I personally see it as overdue. It frees up the transactional side of the platform to scale to just about any infrastructure, while allowing the analytics side to do what it does best. As organizations stretch their transactional systems to deliver advanced reporting and even rudimentary analytics, they’ll eventually run into walls requiring them to bolt on a warehouse, anyhow.
Like it or not, analytics is coming to IT Ops, and it will inevitably drag its warehouse along for the ride.
