Achieve Real-Time Data Analysis with a Virtual Data Warehouse

Chetan DhapudkarJanuary 19, 2016

In today’s digital environment, the success of big data initiatives hinge not on simply collecting large volumes of data, but on a business’ ability to analyze the data and act on its insights. Companies with near real-time analytic capabilities can decipher information faster, find actionable results, and increase their overall competiveness.

 

Yet, despite the critical influence real-time data analysis can have on a company’s competitive advantage, many businesses still struggle to increase the velocity of their data. According to a PwC survey highlighted in CIO.com, 62 percent of respondents surveyed believe big data can give them a competitive advantage; however 58 percent agreed that moving from data to insight is a major challenge.

 

The Challenge of Real-Time Analytics

Why is it so hard for businesses to gain near real-time analysis? In the traditional database model, data is stored and indexed and then subsequently processed by queries, or batch processes. These queries are typically processed on a schedule—one to two times a day. Even when done frequently, there is significant lag time between the start and completion of the process.

For most companies, the only way to get near real-time data is to develop a custom solution. However, given the expense involved in maintaining custom code over time as well as limitations on functionality of the solution, most companies will find this approach not viable in the long-term.

Still, sticking with the status quo and doing nothing is not a good alternative—companies that do so risk losing their competitive edge.

 

Near Real-Time Analytics Achievable with a Virtual Database

To achieve near real-time analysis of data without the need for expensive custom coding, organizations should consider developing a virtual data warehouse using a tool like Informatica. This solution is compatible with any database or BI reporting tool such as MicroStrategy and is simple for anyone with database or ETL experience to use.

Here’s a look at how it works:

  • A virtual data warehouse allows the real-time processing of queries to distributed data sources.
  • Data is then made available in the form of virtual tables to facilitate real-time reporting.
  • Data checks are temporary and done on the fly.

Because this solution doesn’t require replication of data to a physical database, the processing time that is required in traditional batch processing or ETL is eliminated and businesses can achieve near real-time data and analysis. Additionally, a virtual data warehouse can work even in complicated data environments. In one use case, a virtual data warehouse is providing near real-time analysis with data in a multi-tenant environment involving 140 different time zones.

 

Benefits Beyond Real-Time Analysis

As already touched on, near real-time analysis has strong business benefits such as speeding decision-making and improving competitiveness. And for those reasons alone, a strong business case can be made that it’s worth investing in a solution that can provide near real-time analysis of data as needed.

 

But, there are other benefits that add additional value. These include:

  • Eliminates batch processing time – Because all real-time data is made available via data services, there is no additional time spent in ETL or batch processing execution, which typically has a long lag time and requires maintenance and follow-up.
  • Reusability – The virtual tables designed in data services are used to load data for historical reports, so there is no need to build additional ETL functionality for historical data.
  • Ease of coding – With Informatica all the data can be easily read, transformed, and made available for reports without the need to write complex map-reduce codes. The Developer tool also has an easy to use user interface (UI).
  • Maintenance and monitoring – Using a web-based Admin console, monitoring and maintenance is simple.
  • Scalability – Using an SOA-based architecture in Informatica, it’s easy to scale the infrastructure by adding nodes to the existing domain when needed.

From better understanding the customer experience to detecting fraud or security threats to tapping into the IoT to gain insights into how your organization is performing, near real-time analytics is applicable to a host of use cases that improve competiveness, operational efficiency, and to drive new revenue streams.

If your organization doesn’t have near real-time analytics capability, and are wondering how it might apply to your business, let’s schedule time to talk.