Gartner issued a press release yesterday stating their belief that data warehousing is on the verge of major changes beginning in 2011. Two key changes highlighted by the release are increased demand for new types of information formats and applications, and increased demand for real-time data. Data consumers in the most forward-thinking companies are straining to go beyond mere reporting and analysis of business numbers to get at the context or meaning of business activity.
New formats and applications
In order to find this meaning, it will be necessary to go beyond traditional number-oriented data warehouses and incorporate text and media to a much greater extent than ever before. Companies that had toyed with text and media as data sources will now embrace them as integral to their data warehouses. This in turn will challenge business intelligence architects to ditch traditional ideas of reporting and analysis in order to present the new data in a way that helps the decision maker grasp the meaning he or she seeks. One thing I can easily envision is the increased use of tag cloud interfaces similar to what you now see in many blogs.
More data, faster
The other item, which we've discussed elsewhere on this site, is real-time data and business intelligence. This is what is driving the rapidly increasing interest in columnar databases, such as the one that powers Vertica Systems, whose acquisition by HP was announced and discussed here last week.
Behind the search for meaning
Larger companies will be investing heavily in the new technologies in an attempt to match the nimbleness and responsiveness of smaller businesses without adding large numbers of skilled analysts, who will be too expensive or not available. Growing businesses will need to be aware of the need to transition into these kinds of capabilities as they scale up through time and to plan their investments in new technology and data accordingly.
Showing posts with label real-time. Show all posts
Showing posts with label real-time. Show all posts
Thursday, February 24, 2011
Tuesday, February 15, 2011
HP Acquires Vertica
HP announced yesterday that it is acquiring Vertica Systems, a company I had recently highlighted as a highly successful real-time BI and analytics vendor operating "under the radar" while racking up an impressive client list. I see two takeaways from this:
1. Obviously Vertica is "under the radar" no longer, but more importantly:
2. This represents a big pivot for HP.
HP had been widely rumored to be exiting the BI/analytics market. Their NeoView product had failed to gain traction, and they recently announced an alliance with Microsoft to market a business intelligence appliance running Microsoft BI and SQL Server on HP servers. My guess is this means HP is making a heavy bet on real-time BI and analytics as the next growth business in this segment and are buying in through the Vertica acquisition.
It should be interesting to find out whether this ultimately signals HP's intent to focus on real-time business intelligence and analytics as a niche or if there are more acquisitions in the offing as HP seeks to build a new BI portfolio.
1. Obviously Vertica is "under the radar" no longer, but more importantly:
2. This represents a big pivot for HP.
HP had been widely rumored to be exiting the BI/analytics market. Their NeoView product had failed to gain traction, and they recently announced an alliance with Microsoft to market a business intelligence appliance running Microsoft BI and SQL Server on HP servers. My guess is this means HP is making a heavy bet on real-time BI and analytics as the next growth business in this segment and are buying in through the Vertica acquisition.
It should be interesting to find out whether this ultimately signals HP's intent to focus on real-time business intelligence and analytics as a niche or if there are more acquisitions in the offing as HP seeks to build a new BI portfolio.
Thursday, January 27, 2011
Real-Time BI and Analytics: Flying Under the Radar
I don't normally devote a lot of space to a single company, but while researching to write about real-time business intelligence and analytics I ran across one I had never heard of before.
Vertica Systems specializes in databases that are built specifically for real-time and near real-time business intelligence and analytics. They claim database performance that is 50-1000 times faster than traditional data warehouses. How does Vertica accomplish this? In the most non-technical terms I can muster:
1. They organize the data differently than a traditional database, using a method called "columnar orientation." This makes it easier to get at just the data that are needed, faster than traditional queries.
2. They compress the data very aggressively. This allows them to, for example, store multiple copies of the same data sorted in different ways.
3. They store data for reading by queries differently than data to be written to the database, again to optimize the speed of data reads for queries.
4. They can take advantage of multiple processors on the database server.
5. Vertica wasn't designed to run on a specific kind of server hardware, but can run on anything that runs the Linux operating system. This should help hold down hardware costs.
Even though I hadn't heard of them, Vertica must be doing something right. They've quietly racked up an impressive client list that spans multiple industries. Clients include Twitter, Verizon, Bank of America, Comcast, Blue Cross Blue Shield, Sunoco, and most recently Groupon.
Vertica Systems specializes in databases that are built specifically for real-time and near real-time business intelligence and analytics. They claim database performance that is 50-1000 times faster than traditional data warehouses. How does Vertica accomplish this? In the most non-technical terms I can muster:
1. They organize the data differently than a traditional database, using a method called "columnar orientation." This makes it easier to get at just the data that are needed, faster than traditional queries.
2. They compress the data very aggressively. This allows them to, for example, store multiple copies of the same data sorted in different ways.
3. They store data for reading by queries differently than data to be written to the database, again to optimize the speed of data reads for queries.
4. They can take advantage of multiple processors on the database server.
5. Vertica wasn't designed to run on a specific kind of server hardware, but can run on anything that runs the Linux operating system. This should help hold down hardware costs.
Even though I hadn't heard of them, Vertica must be doing something right. They've quietly racked up an impressive client list that spans multiple industries. Clients include Twitter, Verizon, Bank of America, Comcast, Blue Cross Blue Shield, Sunoco, and most recently Groupon.
Real-Time Business Intelligence: Technology and Strategy Implications
In an earlier article we discussed how increasing competition has led to the compression of the business intelligence life-cycle from a monthly to an almost continuous frequency in some cases. This challenges technology vendors and business strategists alike to respond to the new reality.
Technology Implications
When businesses updated their BI data monthly, processing of the data could be conveniently scheduled outside business hours. A processing window would be established (say, from 6 PM to 8 AM local time) for the data architects to extract, transform, and load (ETL) data from the transaction processing system such as sales orders or general ledger to the BI system. As demands for amounts and detail of BI data increased, it became more difficult to keep the processing within the allotted time window. But as long as this meant management occasionally had to wait until 10 AM or noon local time for monthly updates it wasn't a major problem. The flaw in this strategy was exposed by demands for more frequent updates.
Clearly what is needed from a technology standpoint is a more targeted and incremental approach to processing data. For example, a Microsoft SQL Server Analysis Services database (cube) can be programmed to monitor for changes in its source transaction system. After a change is detected, the cube can "listen" for a pre-determined amount of time to make sure there are no more changes to the source data, and then issue a request for only the new data. Since this represents a very small amount of data from the source system, and the portion of the cube that needs to be reprocessed is also quite small, the BI data update takes seconds instead of hours as in the old processing strategy. Other vendors are building similar capabilities. Another technique involves the use of database caching to constantly process changes to data in the background while continuing to present the latest available view of the data to the consumer.
Strategy Implications
Elsewhere on this site, I have advocated for a BI strategy that is in harmony with the way people interact with data. I introduced I concept I call the PRIME model, with five activities: Producing, Reviewing, Investigating, Monitoring, and Extrapolating. For more specifics, please feel free to click here. Based on the PRIME model, I can foresee three significant impacts on how businesses work with data as opposed to the current approach.
First, real-time BI and analytics will shift the focus of data consumption for knowledge workers, analysts and managers alike, from reviewing to monitoring. With a greatly reduced time window for the consumption of data before fresh data arrive, there simply won't be time to review and digest static reports. This will lead to a greater emphasis on business performance management. The traditional report will increasingly be boiled down to the presentation of a small number of key performance indicators (KPIs), along with strategic drivers of those KPIs.
Second, because data updates will occur too rapidly for a human analyst to investigate causes or extrapolate trends using traditional methods, there will be an increasing dependence on automated technologies for data mining and predictive analytics. The investigation and extrapolation performed through such automated technologies will feed into the monitoring I described in the last paragraph. These automated technologies will not supplant but will augment the knowledge worker, whose focus will shift toward longer-term strategic analyses.
Finally, the "24-hour BI cycle," as I described it in my previous article will drive an increasing reliance on mobile business intelligence, as managers and analysts require access to data wherever they are when business conditions change. IT will need the ability to push needed data to these knowledge workers, whether the client machine is a traditional PC, laptop, netbook, smartphone or tablet. It's not difficult to imagine that, over time the latter two devices will play a greater role as their capabilities and the wireless network infrastructure continue to improve.
Technology Implications
When businesses updated their BI data monthly, processing of the data could be conveniently scheduled outside business hours. A processing window would be established (say, from 6 PM to 8 AM local time) for the data architects to extract, transform, and load (ETL) data from the transaction processing system such as sales orders or general ledger to the BI system. As demands for amounts and detail of BI data increased, it became more difficult to keep the processing within the allotted time window. But as long as this meant management occasionally had to wait until 10 AM or noon local time for monthly updates it wasn't a major problem. The flaw in this strategy was exposed by demands for more frequent updates.
Clearly what is needed from a technology standpoint is a more targeted and incremental approach to processing data. For example, a Microsoft SQL Server Analysis Services database (cube) can be programmed to monitor for changes in its source transaction system. After a change is detected, the cube can "listen" for a pre-determined amount of time to make sure there are no more changes to the source data, and then issue a request for only the new data. Since this represents a very small amount of data from the source system, and the portion of the cube that needs to be reprocessed is also quite small, the BI data update takes seconds instead of hours as in the old processing strategy. Other vendors are building similar capabilities. Another technique involves the use of database caching to constantly process changes to data in the background while continuing to present the latest available view of the data to the consumer.
Strategy Implications
Elsewhere on this site, I have advocated for a BI strategy that is in harmony with the way people interact with data. I introduced I concept I call the PRIME model, with five activities: Producing, Reviewing, Investigating, Monitoring, and Extrapolating. For more specifics, please feel free to click here. Based on the PRIME model, I can foresee three significant impacts on how businesses work with data as opposed to the current approach.
First, real-time BI and analytics will shift the focus of data consumption for knowledge workers, analysts and managers alike, from reviewing to monitoring. With a greatly reduced time window for the consumption of data before fresh data arrive, there simply won't be time to review and digest static reports. This will lead to a greater emphasis on business performance management. The traditional report will increasingly be boiled down to the presentation of a small number of key performance indicators (KPIs), along with strategic drivers of those KPIs.
Second, because data updates will occur too rapidly for a human analyst to investigate causes or extrapolate trends using traditional methods, there will be an increasing dependence on automated technologies for data mining and predictive analytics. The investigation and extrapolation performed through such automated technologies will feed into the monitoring I described in the last paragraph. These automated technologies will not supplant but will augment the knowledge worker, whose focus will shift toward longer-term strategic analyses.
Finally, the "24-hour BI cycle," as I described it in my previous article will drive an increasing reliance on mobile business intelligence, as managers and analysts require access to data wherever they are when business conditions change. IT will need the ability to push needed data to these knowledge workers, whether the client machine is a traditional PC, laptop, netbook, smartphone or tablet. It's not difficult to imagine that, over time the latter two devices will play a greater role as their capabilities and the wireless network infrastructure continue to improve.
Tuesday, January 25, 2011
Real-Time Business Intelligence and Analytics: An Introduction
Previously we introduced the concept of a life-cycle for business intelligence data. Data are captured into transaction processing systems like the general ledger accounting system. At some interval these data are moved into a BI database and processed to make them suitable for consumption. At some point older data are archived or even removed when they are no longer useful.
At first the BI or decision support life-cycle was typically built around financial reporting, and thus depended on the quarterly or monthly accounting close. Once the books were closed for the accounting period, data were pulled from the accounting system, loaded into the BI or decision support database, reports were run, and data analyzed.
As BI moved beyond the financial realm, managers in business areas like Sales and Operations found that they could achieve competitive advantage by getting (and acting upon) more frequent updates of data as opposed to organizations that remained tied to the monthly accounting cycle. This spurred the increase in the frequency of data updates to weekly, daily, and so on. The result is an evolution toward a "24-hour BI cycle" that parallels the evolution toward a 24-hour news cycle in the news media. In some applications BI data are now updated almost continuously. Rather than being the exception, this high-frequency updating of data will increasingly become the rule.
The impetus toward real-time business intelligence and analytics has implications for both technology and business. Vendors are challenged to devise new solutions to the problems introduced by requirements for more frequent updates. With the increase in the amount and frequency of data updates, businesses need to rethink how they consume data. We'll address these issues in more detail in the next article.
At first the BI or decision support life-cycle was typically built around financial reporting, and thus depended on the quarterly or monthly accounting close. Once the books were closed for the accounting period, data were pulled from the accounting system, loaded into the BI or decision support database, reports were run, and data analyzed.
As BI moved beyond the financial realm, managers in business areas like Sales and Operations found that they could achieve competitive advantage by getting (and acting upon) more frequent updates of data as opposed to organizations that remained tied to the monthly accounting cycle. This spurred the increase in the frequency of data updates to weekly, daily, and so on. The result is an evolution toward a "24-hour BI cycle" that parallels the evolution toward a 24-hour news cycle in the news media. In some applications BI data are now updated almost continuously. Rather than being the exception, this high-frequency updating of data will increasingly become the rule.
The impetus toward real-time business intelligence and analytics has implications for both technology and business. Vendors are challenged to devise new solutions to the problems introduced by requirements for more frequent updates. With the increase in the amount and frequency of data updates, businesses need to rethink how they consume data. We'll address these issues in more detail in the next article.
Subscribe to:
Posts (Atom)