Data Warehouse 2.0: In Search Of Meaning

Gartner issued a press release yesterday stating their belief that data warehousing is on the verge of major changes beginning in 2011. Two key changes highlighted by the release are increased demand for new types of information formats and applications, and increased demand for real-time data. Data consumers in the most forward-thinking companies are straining to go beyond mere reporting and analysis of business numbers to get at the context or meaning of business activity.

New formats and applications

In order to find this meaning, it will be necessary to go beyond traditional number-oriented data warehouses and incorporate text and media to a much greater extent than ever before. Companies that had toyed with text and media as data sources will now embrace them as integral to their data warehouses. This in turn will challenge business intelligence architects to ditch traditional ideas of reporting and analysis in order to present the new data in a way that helps the decision maker grasp the meaning he or she seeks. One thing I can easily envision is the increased use of tag cloud interfaces similar to what you now see in many blogs.

More data, faster

The other item, which we've discussed elsewhere on this site, is real-time data and business intelligence. This is what is driving the rapidly increasing interest in columnar databases, such as the one that powers Vertica Systems, whose acquisition by HP was announced and discussed here last week.

Behind the search for meaning

Larger companies will be investing heavily in the new technologies in an attempt to match the nimbleness and responsiveness of smaller businesses without adding large numbers of skilled analysts, who will be too expensive or not available. Growing businesses will need to be aware of the need to transition into these kinds of capabilities as they scale up through time and to plan their investments in new technology and data accordingly.

Are You In The Market For An Appliance?

There's a lot of buzz in the business intelligence world these days about appliances. At first blush the term sounds confusing. It evokes images of something you would find in your kitchen rather than in the IT server room. In fact, appliances for business intelligence applications have been around in some form for decades.

So what's an appliance?

In the 1980's, the advent of personal computers and networking made it possible for small businesses to enter the computer age in great numbers for the first time. Unfortunately, most small business owners lacked either the time or skill to put together an integrated business solution. And if the business owners could find people with the time and necessary skill, they couldn't afford them. This created opportunities for systems integrators who stepped in with bundled hardware and software solutions, tailored for specific business applications. Such solutions were called "turnkey systems," because it was already installed and configured and all you needed to do was "turn the key" and drive the system.

Fast forward a few years and vendors responding to the need for "turnkey" solutions for data warehousing began building and configuring "data warehouse appliances" along very similar principles. The appliance is a bundle of hardware and software that has been pre-configured and optimized for a specific application, such as data warehousing. (Need background on data warehousing, schemas, etc.? Review Parts One and Two on processing data in the BI data life-cycle and the Data Warehouse Imperative.)

Why would I want an appliance?

You can make a very strong business case for buying an appliance rather than trying to build your business intelligence or data warehouse solution from scratch. Some of the key advantages are listed below:

The hardware and software are pre-selected and installed for you.

The system is configured and sold based on the amount of data you're pulling out of your source system, removing much of the guesswork involved in tweaking the system for performance.

Many appliances come with extras like administration consoles, to make system management easier. This can be a major consideration for small or medium sized businesses.

The appliance vendor becomes the single point of contact for all support. This can be huge. Just ask your IT person how many times he or she has stressed over troubleshooting system issues when the hardware vendor is blaming the software vendor and vice versa. And how much money has that, in turn, cost you?

So what's the catch?

You would think that for providing all this wonderful service, the integrator who puts this all together would charge a stiff premium over the cost of the hardware and software components bought separately. Actually, Gartner Research has found that this is not the case. But all is not paradise. You still have to design and maintain the database side of the data warehouse, or schema. Many vendors have pre-built schemas available, but these are only starting points for you to customize for your business.

Before I bought a data warehouse appliance (or similar - Microsoft has, for example, just released a "decision support" appliance bundle) I would ask some pointed questions of my prospective vendor partner:

Can I talk to other customers that you've worked with before to gain the benefit of their experience with you?

If I'm a growing business, how much flexibility do I have to expand as my needs change? How much is that going to cost me?

Are you (vendor) willing to commit to a long-term business relationship? And how will that work from your end?

And, of course, many others that are typical for any vendor/customer relationship, like service levels, do we get any cool logo schwag, etc.

Who are the major players?

Beside the aforementioned Microsoft, most of the big BI vendors participate in this market, either directly (Teradata) or through proxies (for example Netezza, a former independent now part of IBM).

About The Data Warehouse Concept

In recent articles we've been talking about data governance and data integration and why these are so important to your business. One of the key best practices for successful governance and integration is to keep business intelligence data separate from transaction data (such as the data in your accounting or point of sale systems).

Once you grow your business to the point where you need the concepts discussed here, you’ll probably want a technical guru to head up the implementation. Remember as we go that my idea is not to make you that guru, but to give you enough information to be an intelligent consumer of the products and services that make a sustainable BI program possible.

One very important principle behind that sustainable BI program is that we want a separate place for the data we’re going to consume, away from the sources from which we captured the data in the first place. Different people may give different names to this separate place. For our purposes right now let’s use the term “data warehouse.” This isn’t completely accurate, because the term “data warehouse” has a very specific meaning for BI professionals. So we’ll come back to this later.

But for now, as a simplification, we’ll say that we take data from where it was captured, do some processing with it, and load it into the data warehouse. There are good reasons for this. First, the systems designed to capture the data are not typically designed to get data out as easily as it gets in. Second, as we’ve hinted at previously there may be two or more sources of data that need to be combined in a way that is meaningful for your business.