Back when I was in college, our football team was regularly featured in the Bottom 10. There wasn’t much we could do against the powerhouse teams, so our cheers changed from “Fumble Turkeys Fumble” to a more plaintive “Turkeys Turkeys Turkeys.” This edition of “I Hate Monitoring” is similar. Our friends at a media company were getting flattened by two of the Big 4, only to turn around and find out that a Tiny 1 was crushing them on the other side.
“All we wanted to do was talk sports,” said Squanto Massasoit. “But those monitoring turkeys kept us running around like our heads were cut off. Somebody thought these were the best, but my team was getting the stuffing kicked out of us trying to keep up.”
Massasoit’s team is responsible for keeping one the web presence for one of the United States largest media companies up and running. Their monitoring tools combined event management from one Big 4 vendor, response time monitoring from another, and an administrator's toolkit for availability tracking.
“Making that big turkey of an event console work was no fun. First of all, it seemed like it was frozen solid, and nobody really knew how to handle it safely. For any good to come of it we had to get deep inside and I don’t know about you but there’s nothing I like less than sticking my hands deep into stinking turkey guts. Well, change management meetings, they’re worse.”
Since the event console wasn’t easy to change, the monitoring team was left dealing with huge numbers of single events instead of nicely coordinated service issue indicators.
“You know how you get really sleepy after you’ve had a big turkey dinner?” asks Massasoit. “It’s the tryptophan. Well, when we tried to scope out our web site performance it seemed like that system got all tuckered out. It never did finish, and I never felt like our eyes were really open.”
After more than a year of trying, the team realized that the performance monitoring system simply didn’t work at the scale of their web site.
“I’ve heard that turkeys are so stupid they’ll drown in a rainstorm just looking up at the clouds. We thought that a popular administrator's toolkit had enough legs to track whether everything was working. But the more we looked into how much work it was to prepare that turkey the more we realized that that bird just couldn’t fly. Not for us.”
Static configuration files were completely inappropriate for the dynamic media company web infrastructure. In order to accommodate huge surges in visitors for holidays, special sporting events, and the like, the team would dramatically reconfigure the infrastructure. It was challenging enough to redefine server roles without also having to change all the monitoring function, too.
Lacking confidence in the existing tools and facing an extremely important large event, the media company installed Zenoss. Zenoss was instantly successful, successfullly tracking performance in the large web application environment and alerting the staff to critical events missed by prior systems. The team has implemented per-event custom Zenoss-powered dashboards for business management, tightening the relationship between the IT web operations team and senior executive staff.
“We went cold turkey on the whole lot.” Said Massasoit. “Ben Franklin thought the national bird should be a turkey, but the country went with the eagle. For us, Zenoss is an eagle, not a turkey. Pass the pumpkin pie!”
