Data Driven IT Operations

Engage in a conversation with any IT executive on the topic of big data and you will likely hear how the IT team has helped their company put in place a “Data Lake” that houses large amounts of business data and that is the basis for data driven business decisions. We know that data driven decisions underlie how airlines price seats, how hotels price rooms, and what merchandise online retailers offer each visitor based upon the available information about each visitor.

But what you will not hear is any stories about how IT Operations has embraced big data to run IT. Somehow the people responsible for building all of these big data environments for the business have neglected to put one in place for themselves.

The Need for Data Driven IT Operations

Why does IT need to embrace big data to improve its own operations. For the following reasons:

  1. The business is demanding that ever more functionality be implemented in software and put online and evolved ever more frequently. This means constant changes to software in production.
  2. The software platform is now much more diverse. It is not just a Java world anymore. In many cases, JVM’s are being replaced with PaaS frameworks like Pivotal Cloud Foundry.
  3. The infrastructure is now virtualized, automated and itself highly dynamic. Compute has been virtualized, networking and storage are being virtualized with products like VMware NSX and VMware vSAN.

In summary the modern IT stack looks a lot like the diagram below.


The stack above could not be more different than what enterprises ran as recently as 5 years ago. This new stack demands that IT Operations run differently – in fact it demands that IT Operations embrace Data Driven IT Operations.

What is Data Driven IT Operations?

Just as with business data where different sources of data are combined in order to allow for previously unattainable insights, with Data Driven IT Operations, all of the metrics that measure how IT is operating need to be brought together into a common real-time and related data store. An overview of what this looks like is below.


The Instrumentation Architecture for Data Driven IT Operations

Your modern IT environment is now changing so quickly at all layers of the IT stack that you need to rethink how you collect data from that entire stack. 20 years ago, IBM, BMC, HP and CA promised their customers that their products and their products alone could monitor and manage the entire stack. This was called Business Service Management and it was a miserable failure – for the simple reason that no single vendor could then nor can now keep up with the pace of innovation.

Enterprises reacted to the failure of BSM by buying best of breed point tools from the many great new vendors who jumped in to address new technologies and meet new customer needs. This lead to the Franken-Monitor shown below.


The first step towards implementing Data Driven IT Operations then needs to be to make sure that you have the right streams of data. You need an instrumentation architecture wherein you think through exactly what combinations of commodity and open source metrics you should combine with valuable metrics from various vendors. An instrumentation architecture is show below.


Metric Quality for Data Driven IT Operations

The modern IT stack is so dynamic from the top of the stack to the bottom that the old ways of collecting data and in some cases the old metrics themselves are either insufficient or completely inadequate. A modern metric collection approach should focus upon the following:

  • Performance should be defined as response time (latency) and throughput, not resource utilization
  • The entire stack should be instrumented for response time (latency), throughput, errors and contention
  • Metrics should be collected continuously – at least once every minute and sometimes every couple of seconds
  • Metric collection should be consistent – gaps in metric data are no longer acceptable
  •  The metrics themselves should be deterministic – not rolled up averages of other metrics
  • The metrics must be accurate – time-shifting due to virtualization should be avoided – objective “outside-in” approaches should be used wherever possible

Data Driven IT Operations Requires Related Objects and Metrics

Many enterprises have implemented big data log management solutions. What they have found is that since the log management solutions only tag logs by time stamp at ingest time, they have to know what they are looking for before the log store becomes useful. Which means that the event has to have already occurred before the log store is of any use.

The goal of Data Driven IT Operations is to play offense and detect trends and patterns that precede actual problems. In order for this to be possible the relationships between the items (the objects) in the IT stack need to be captures automatically and continuously at ingest time. This is done by the OpsDataStore Dynamic Object Model which automatically establishes the relationship between the items in the streams of metrics as the data is ingested. In the diagram below, OpsDataStore continuously relates the transactions in your environment monitored by AppDynamics (see the video) or Dynatrace (see the video) to the infrastructure upon which they run.


OpsDataStore – The Platform for Data Driven IT Operations

If all of this sounds really hard to do, it is not. OpsDataStore does the heavy lifting for you (see the 2 minute overview video). OpsDataStore has connectors to popular infrastructure management solutions like VMware vSphere, and to popular monitoring tools like AppDynamics, Dynatrace, and ExtraHop.


The Benefits of Data Driven IT Operations

Data Driven IT Operations offers revolutionary benefits to IT Operations and to the constituents of IT in the enterprise. These benefits include:

  • Putting more business functionality online (digitization) means that it is imperative that these online systems work correctly all of the time.
  • Data Driven IT Operations allows enterprises to combine metrics from best of breed tool vendors in to a related whole to ensure end-to-end service quality
  • IT organizations are under constant pressures to become more efficient and cut costs. Yet most data centers run at less than 30% utilization – and everyone is afraid to drive up the utilization out of fear of causing a performance problems. Data Driven IT Operations allows enterprise to use fine grained performance metrics to tune capacity, driven up utilization, and reap significant hard dollar savings.
  • IT organizations waste a lot of time manually extracting data and responding to requests for IT data from various constituents. Data Driven IT Operations turns this into a live self-service exercise which lets users choose their BI or query tool (see the video on how easy it is to build a custom dashboard in Tableau).
  • Finally the time and people resources that are wasted in IT in the existing processes prevents IT from being as agile as it needs to be, and prevents IT from being as effective a partner to the business as IT wants to be. Data Driven IT Operations empowers IT to make a real difference to the business.

More Information

For more information please click on one of the links below: