The Digital War Room Battleground – Costs, People and Metrics
Last week the Enterprise Management Associates’ (EMA) Dennis Drogseth conducted comprehensive global research into the state of today’s IT War Room practices and presented the results in a Webinar on “Unifying IT for Digital War Room Performance”. The insights from the EMA research spanned all aspects of IT, process, organization and technology related.
We were struck by several key “battlegrounds” into which the research seemed to align:
- Cost of War Rooms
- People and Metrics involved in War Rooms
- Key gaps preventing shorter, fewer and less costly War Rooms
We decided to take a closer look at the key digital war room battlegrounds in a two-part blog series to dig a little deeper and shed some additional insight on this topic. For more details on the research itself, please visit EMA.
War Room Costs
EMA research showed the following costs associated with today’s IT War Rooms:
• “The average War Room comprises 15 people and takes 5-6 hours to respond and resolve the incident that cause it to convene. 29% of War Rooms take more than 11 hours to respond and resolve.”
With today’s fully-burdened cost of seasoned IT professionals approaching $75 per hour; this means the average War Room costs at least $7000 and in many cases is well into five figures. More costly still, of course, is the cost in terms of lost revenue and reputation associated with the service disruptions that typically initiated the War Room conclave in the first place.
• “70% of War Rooms are responding to incidents whose underlying technical causes were not diagnosed and identified before those services were impacted”.
These War Rooms were totally reactive with respect to service outages. Despite the use of multiple monitoring technologies across the stack from applications to infrastructure, service impacts were not prevented. There was no proactive or preventative management present (or, indeed possible).
People and Metrics
• “24% of the time the CIO is involved, Application Development is increasingly becoming more involved. IT Service Management typically owns and manages the War Room, but all organizations are represented in the heat of battle because of the complexity of IT environments.”
• “Issue domains ranged from Network Performance (45% of the time), Compute or Storage Performance (37%), IoT (33%), In-house Applications (29%) to Public Cloud (28%).” (Please note these do not add to 100% because it is normal that War Room issues span multiple aspects of technology under-pinning application and service outages and slow downs.)
With many personnel from differing teams and with technical expertise spanning so many domains, the very act of convening a War Room (whether Physical or Virtual) itself takes between 1-2 hours just to get started! And, at least a quarter of the time, the CIO is directly involved. This is not the recipe for an optimally performing IT organization, whether in the eyes of the CIO, nor in the eyes of the Business (or customers) directly affected. Instead, it is the recipe for high stress and burn out.
• “Performance and Latency of services is the most important technical metric”
While seemingly obvious, it bears constant repeating. If applications and services aren’t getting the work done that needs accomplishing, in the appropriate amount of time, IT isn’t delivering on its fundamental reason to exist. As such this metric must be the context by which all management must be based – across the stack and across the organization.
• “IT Team disruption is the most important Business metric to the CIO.”
Why? CIO’s are responsible for delivering on service commitments made to the business – hence the technical metric above. But, they must not only measure and manage the delivery of that service, they must also manage their organizational cost to do so. Their staff is key to their success, and not only is the disruption to their staff caused by reactive War Rooms costly in pure staff hours, it is also costly in terms of staff moral and retention. The best way to reduce IT Team Disruption in today’s organizations is to become proactive and prevent the disruptions in the first place. Ironically, the very opportunity costs of the staff disruptions prevent putting in place measures to become more proactive.
Key Battleground Take Aways
IT organizations are currently trapped into a vicious cycle. Because they are not sufficiently proactive at preventing application service disruption and degradations, their staff is constantly involved in a costly and time-consuming War Room process. The continuous reactivity prevents appropriate use of IT resources to implement automation and analytics by which they can achieve higher levels of proactivity in order to “break the cycle.”
Success, to CIO’s, is simple to articulate and measure: the continuous delivery of cost-effective IT resource needed to support appropriate application service levels. To succeed, the gaps to success in automation and analysis need to be identified and closed. Stay tuned for part two, where we will discuss this and layout a strategy for success.
A Solution for AIOps and Digital War Room Success
OpsDataStore offers a solution with an AIOps platform that delivers the whole picture, so you can to optimize performance, increase business agility, measure operational risk and drive ROI for every technology investment.
OpsDataStore uniquely collects the metrics across the application, compute, network and storage stacks and automatically and continuously creates the relationships between them. It continuously analyzes all the related data to calculate baselines of normal and abnormal behavior for every metric, in the context of all other related metrics, and all mapped to the business metrics that matter: application, transaction and service performance and response times.
Learn more about how OpsDataStore enables real-time AIOps to quickly win Digital War Room battles.
Stay tuned for part two: A Digital War Room Strategy – Closing the Gaps to War Room Efficiency…