Unified Observability View with No Data Duplication

Share on facebook
Share on linkedin
Share on twitter
Share on email

Many organizations are striving to get to a “single pane of glass view” in order to get visibility into the health status of their business applications and infrastructure.  In reality, these organizations have multiple monitoring tools where some teams prefer one tool over the other. For example, the application team might need an application performance monitoring tool that provides very deep visibility into the application code beyond simple metrics that many generic monitoring tools provide like URL or system monitoring. 

The Chaos in using multiple monitoring tools

The major challenge with having multiple monitoring tools is that it creates some chaos within silos where every team will have visibility only to their particular component.

However, for the Application Support and SRE teams it is a strong requirement for visibility into the full ecosystem of the application. Historically people who work in IT Operations can relate to how troubleshooting works during an incident. It’s very common to have a war room with more than 20 people where each person might be looking at a separate tool or tools to troubleshoot an event. This results in: Unnecessary time taken to resolve events and incidents, which can leave you with frustrated customers, perhaps loss of revenue, and possible damage to your brand. 

Organizations are starting projects which will integrate these different toolsets to work together, but most of the time these projects are failing due to multiple reasons that can be attributed to planning, technical capabilities or costs associated with data duplication.

Answering the Why ?

When we work with our clients, we start by identifying the WHY, then assessing their maturity level to meet that WHY with a co-developed HOW and defined steps to meeting their goals. What we generally find is that the goal is simply to have a unified dashboard that the operations group and the management teams can view to get a visibility into the health status of the environment.

In this BLOG we are not talking about sophisticated use cases to apply machine learning or building a data warehouse against the collected data. We are simply expressing the ease of how you can achieve your goal and get far better visibility on the performance of your application and infrastructure by combining all of the data you are currently collecting.

Grafana Unified Dashboard

One of the dashboarding tools that we use is Grafana that was built on the concept of visualizing  the data from your data sources without storing the data.  Data can be your enterprise monitoring tool or any other data source you can connect to. It’s a free open source tool that is also available as an enterprise edition to support access to premium plugins and security features. 

This is an example of a dashboard we built that illustrates how you can visualize data from multiple sources with no data storage. As a matter of fact you can install Grafana locally on your machine and build any dashboard you want as long as you have access to the data sources. In this dashboard, we visualized data from these data sources : 

  • ServiceNow : Incident and Change records
  • Business Database: Number of placed orders from an internal database that stores customer purchase orders 
  • Moogsoft: Noise reduction for the number of received events
  • AppDynamics: Application performance and user experience metrics
  • AWS Cloud Watch: ELB Metrics
  • Elasticsearch: Logs 
Observability Grafana Dashboard

Grafana also supports a drill down if you need to investigate further in the tool where the data came from. For example, if you visualize the number of incidents from ServiceNow and you need to drill down to ServiceNow console to look at more details. 

Mixing multiple data sources in the same graph

One of the other great features in Grafana is that you can mix data from multiple data sources in the same chart. For example here, we are bringing purchase orders from an internal database used by the business to track customer orders. We are also adding performance metrics on the same chart that tracks the user experience from AppDynamics.

Multiple Data sources
Multiple Data Sources

Other Visualizations in Grafana

In addition to the common charts, Grafana has other visualization plugins like the world map that allows you to visualize and drill down for Geolocation data and impact.

Geo Map Visualization

In this article, we used Grafana as an example of an open source product that you can deliver a lot by using it, but if you have your own preferred data visualization tool you can still utilize it to achieve faster time to value and similar outcomes.