Of the five companies I interviewed at the recent NoSQL Now 2012 conference in San Jose, two were database companies, two were analytics companies, and one was a technology company in the business of accelerating the speed of others’ NoSQL databases. One of the analytics companies was Metric Insights, which is based in San Francisco. In the enterprise and elsewhere, we receive a vast amount of data in many intervals ranging from real time to hourly or even longer periods, with complexity of kind and format—or no format at all. The idea of Big Data is to derive useful information out of unmanageable data and use it for action to improve business. Business intelligence (BI) is similar to Big Data analytics but usually deals with data of more manageable volume, velocity, and complexity. This is changing as SNS and mobile computing also enter the enterprise market. Metric Insights says we need Big Data BI.
I sat down with Metric Insights’ Marius Moscovici, CEO, and Steve Mock, COO, to find out what they were up to.
From left: Marius Moscovici, CEO, and Steve Mock, COO
When I interview companies at conferences, I usually want to know what they do in relation to the conference theme, and what their differentiation, future directions, and competition are. What they do seems to be fairly easy to understand, but differentiation and competition may not always be that easy to figure out. The NoSQL-related domain is still being defined as it moves in many directions at Big Data creation speed. Certainly, we need solid database technologies, including databases themselves and any utilities that enhance them, analytics engines, and good visualization tools.
For further expansion of this market, it is vital to get buy-in from the enterprise. William McKnight in his keynote speech advocated for putting NoSQL in the enterprise market and emphasized that only then would the NoSQL market become legitimate. Riding on what the enterprise already embraces would lower entry barriers for NoSQL-related technologies and services.
Business intelligence has been a big push in the enterprise market. Even before the age of Big Data, in a typical enterprise domain there was a large set of data not shared among different individuals and departments such as call centers, engineering, marketing, and HR. If a marketing campaign reflected call-center customer feedback, a company might be able to sell more of their products and services. Metric Insights’ goal is to enhance BI capability by increasing each datum’s value by associating it with more meaningful information. Because BI is already accepted in the enterprise segment, their goal is reasonable. They want to expand transitional BI into Big Data BI by sourcing data from Big Data as well as from traditional sources.
As shown below, Metric Insights collects data from multiple heterogeneous sources and adds context (relevant attributes, as shown in the figure) to make that data more effective and valuable. Metric Insights says this creates useful insights. The collected and context-enhanced data are stored in intermediate form (JSON) in a database. (By the way, some say that JSON will push XML out completely, and others say, not so fast, because the world is not built by Web alone. But that is not the focus of this blog.) When data have more attributes or context, you can provide more effective analytics because you have more relevant information on each datum.
Metric Insights’ system consists of data collection, augmenting data with other attributes (context), analysis, and visualization. By the way, I mentioned to those gentlemen that I was more interested in how the backend works than in their user interface (UI). Yes, the backend is important, but the front-end, the UI, is crucial in the BI segment. So gentlemen, I take back what I said. My comment came from my techie point of view. When you use a BI system, the first thing people pay attention to is the UI. Because not all BI users are data scientists, BI specialists, techies, geeks, or interested in how it works, its use should be easy and intuitive without lengthy training. When you present your BI tool, if it does not communicate its ease of use and simplicity, no one will pay any attention to it.
Metric Insights prepares typical dashboards for ease of use for a given application. The example below is for a sales database. Sales reps just select a pane to get to what they want instead of creating complex queries (like “what I would like to do”) to obtain the result they are looking for.
A typical screenshot of the UI for the sales database is given in the following. This example shows product releases and the number of daily sales demos made. When a new release is given, it is likely to have an increased number of daily demo requests. But if there is any sudden increase or decrease, you can take a look at that particular point and drill down because more relevant context is available and can be added as an annotation.
Architecturally, it uses a persistent cache to accommodate real and semi-real time data speed and store data in a local Mysql database as well as a document-based store (JSON format). Since it is document based, it is easy to add more information for each datum. Their system works with some well-known Big Data storage/databases and technologies, such as Cloudera, MongoDB, and Google BigQuery. Additionally, a secondary memory-based caching layer is used to optimize end-user access speed of analytics.
Their application areas include sales, production, inventory, and finance, and they are expanding their scope to include recruiting talent. This is an interesting area. It used to be difficult to gain information on each individual because publicly available personal information was very limited. A résumé is written to cast the best light on the job applicant, and references usually provide only positive comments. Now in the era of SNS, we can gather a vast amount of information on individuals when they are at their ease and off their guard, so to speak.
In a way, Metric Insights and Fluid Operations provide a similar product. They both collect data from multiple sources, convert them to a standard form with additional information, apply analytics, and visualize results. On the surface they are similar, but their focus and implementation differ significantly. Metric Insights uses context to enhance each datum and obtain insight, then stores it in JSON-based storage, which is more common for NoSQL players and more relaxed (and easier to manage) than the semantic model Virtual Operations uses.
I think both approaches are valid, and each has its good application areas. The market is still evolving and is big enough for both of them. I asked Metric Insights if they have considered the power industry as an application area. They have not considered it yet, but I think their product can be used for that. The power industry will face multiple Big Data problems, as they will have more real-time monitor data, such as meter-read, equipment status, data feeds from other systems like weather, static information like assets and service logs, and SNS. A utilities backoffice is filled with disjointed applications without much data sharing, which can be improved very much by something like this technology. I do not know how, but that is up to folks like Metric Insights and Fluid Operations.