Attack of the Rampaging Data

By Bill Roberts

[Chart]: Data Growth
[Chart]: Advanced Data Access Architecture
[Sidebar]: Hard Facts About Data Growth

Enterprise MIS must handle ever-increasing amounts of data. Here are some strategies and case studies for dealing with this issue.

To remain competitive, any business must understand customers. Once you know the detailed buying patterns and demographics of customers, you can launch sales and marketing efforts into narrowly defined markets. Today those markets may be so narrow that one large bank refers to them as market segments of one.

To segment customers into slices that small, companies need data--ever larger amounts of data. In fact, corporations are acquiring data so fast that one expert estimates that mission-critical data has doubled every year for the past five years. The desires to know customers intimately and to target markets accurately are good reasons why enterprise data is overrunning the corporate landscape. Additional causes may be technological, such as distributed systems and bigger applications, or social, such as industry deregulation or litigation. Whatever the reason, it seems that executives and managers can't get enough data and tools to analyze it.

There's good news, though. Storage has never been cheaper, strategies for managing data are reasonably well tested and the tools for storing, maintaining and delivering data to end users in useful ways are proliferating. Most encouraging are reports from the front. As high as some of the hurdles are, IS professionals who accept the facts of life are able to provide business colleagues with information that contributes to revenue-generating efforts.

It's Everywhere

How fast is enterprise data growing? Hard figures aren't available, but anecdotal evidence is compelling. A manager of decision support systems (DSS) at a $1 billion laboratory instruments maker says an executive information system (EIS) that he built a mere four years ago at 10MB has grown above 25GB, and it's just one of four EISs the company has.

Richard Finkelstein, president of Performance Computing, a client/server and database consulting firm in Chicago, has been building databases for 20 years. "My last big system on a mainframe 10 years ago was one gigabyte and considered huge," he says. "Now you'd say a terabyte is large. That's a thousand-fold increase."

George Ferguson, a product manager at Hewlett-Packard in Palo Alto, CA, began working in HP's data warehousing business four years ago, when most customers were asking for 10GB systems. "Today that's 100 to 150GB at least, with about 25 percent of new data warehouse business requesting more than 500GB," he says.

Not everyone requires an industrial-strength database of those dimensions. "You'll hear people talking about terabytes," says Randy Betancourt, program manager for data warehousing at SAS Institute in Cary, NC, "but, with the exception of the Fortune 1000, the typical size of a data warehouse is about 50GB."

Even the Fortune 1000 includes modest efforts. Bernard Seban, database manager for computer systems vendor Data General Corp. in Westboro, MA, has only 4GB in a data warehouse dedicated to online analytical processing (OLAP). However, this is not a comprehensive repository. "We have only four years of history in there," he says.

Getting to Know You

As noted, the desire to know customers is what's driving data growth. Today, point-of-sale retail systems collect every piece of accessible data about a customer. Computers store it. Marketing and sales staffs demand access to it.

The goal is micro-marketing: getting the right promotions to the right people at the right time. "If you're spending $50 million on a direct-mail campaign and you can identify the 10 regions that will give you the biggest payback, you can target it that way," says Michael Guidry, director of corporate applications development for Management Science Associates, a market research and applied technology firm in Pittsburgh that works with various vertical markets, including consumer packaged goods companies. In that niche, Guidry says, data is flowing into the enterprise from points of sale; store-level marketing studies that rival the U.S. Census in detail and breadth; syndicated information; and other sources.

It has gotten to the point that, in retailing, the sale itself may no longer be the most important thing. "Sometimes the most valuable commodity that comes out of the exchange is the demographics you capture from it," says Betancourt of SAS. "There's a keen desire to capture as much information as possible about that transaction so you can go back and sell to [that customer again]."

The demand for data seems endless. "You no longer just need to know your customers, but the marketplace and the competitors. And the competitors come from different places," says Terril Retter, a senior analyst at consultant Price Waterhouse in Menlo Park, CA. "The kinds of questions CEOs ask are expanding dramatically." This demand has trickled down, too. Vice presidents, department managers and many others want access to the corporate data stores.

Despite all this activity, even more data is out there. Jim Johnson, chairman of the Standish Group International, a research firm in Dennis, MA, believes business has barely made a dent in collecting data about the customer. He recalls a client, a Las Vegas casino, which knew practically everything about its "high rollers" but almost nothing about the next tier of customers. Businesses can no longer afford this sort of ignorance.

The Big Picture

A host of other factors also contribute to data growth. There's the technology itself. Relational databases have made information more useful; graphical user interfaces have made it easier to use; and today's end-user tools make analysis efficient and thorough. As the technology gets cheaper and easier to use, even more data is collected and analyzed. Existing applications in some cases are doubling in size, and new apps are needed quickly.

Consider the Federal Express customer service application. It used to be you called up and got one of 50 customer service reps who would report the progress of your package. Then the company extended the application to thousands of drivers with hand-held devices. Next it went to top customers, and there were 50,000 users. Now it's on the Internet, accessible by millions of customers. Each iteration requires more data to make the application work.

What we might call meta-factors also cause data to proliferate. David Flaxman, a partner in the Radnor, PA, office of consultant KPMG Peat Marwick, believes that most industries run in five-year cycles. When the industry moves into a new cycle, it ushers in a whole new set of data requirements. He points to watershed changes in financial services (Wall Street in the 1980s), retailing (thanks to WalMart), telecommunications (the breakup of the Bell system) and commercial banking (in the present day) as examples that demanded new and better data throughout the industry.

Johnson of the Standish Group believes deregulation in certain industries is a key factor. "The more things get deregulated, the more information government requires about the entity. As soon as the airline industry was deregulated, the government wanted more information on things like maintenance and on-time records." A twin cause is litigation, he notes; more information is being archived for much longer periods on a CYA basis.

Not least of all, corporate reengineering or downsizing, and the distributed systems they call into play, are culprits. A downsized company needs to empower the few to do the work once handled by many. IT and better and more data are necessary to do that. Data General's Seban says a driving force in his company's data warehouse strategy was the downsizing of the workforce from more than 18,000 five years ago to 5,000 today. And distributed systems themselves result in small individual or workgroup databases that eventually must be replicated to a wider audience, producing duplicate databases in various locations on the corporate network.

Controlling the Uncontrollable

IS professionals must bear the brunt of managing this data rampage: storing data, moving it into relational databases or warehouses where it can be accessed by end users, cleaning and maintaining data, acquiring or building applications for the end users, and training support staff and end users. "How do you control something that is out of control?" asks Katherine Hammer, president and CEO of Evolutionary Technologies in Austin, TX, which develops software tools for data migration and maintenance. "The IS pro needs to accept that he's running a distributed, heterogeneous space and that he's responsible for it." For example, IS has to determine what is the database of record for each different application and control what information goes where and to whom.

This, of course, is easier said than done. The headaches are enormous. Sterling Makishima, HP's data warehouse manager for global customer support, says his biggest problems are maintaining data integrity, assuring the timeliness of the data being demanded by the end user and supporting a worldwide data warehouse infrastructure.

IS people are increasingly turning to some version of the data warehouse to manage this flow. "To improve the look at traditional data, you put it in a data warehouse," says Price Waterhouse's Retter. "You pull it from OLTP, rationalize it and put it on an OLAP for EIS or DDS." This also is not simple to do. "Most IT pros are struggling to integrate data across stovepipe legacy apps," he says.

Whatever the problems for IS, management looks for the advantages. "Data warehouses have a good payoff based on productivity," says Paul Cubbage, a senior analyst at Dataquest, a research firm in San Jose, CA. "You can get metrics that show how much money you're going to save by going with the data warehouse concept."

In a growing number of cases, the business unit--not IS--is proposing and paying for data solutions such as warehouses. In Retter's view, reengineering changed the perspective from using technology for business functions (vertical) to using it across business processes (horizontal). But vertical data structures don't match horizontal data needs. "For 30 years the heads of data processing departments made technology sound mystical," he says. "Now that technology is more important to the health of the enterprise, business has to pay attention."

This restructuring makes it imperative that IS pros and business users talk to each other. "Business units are driving the data warehouse phenomenon," says Betancourt of SAS. "IT has to spend a lot of energy transferring the data logic. To do that properly, they have to go out to the business users and ask, 'How do you want it transformed? What are the business rules?'"

To provide insight into how leaders in technology implementation are solving their data growth and access problems, we present the following somewhat related, yet uniquely defined case studies.

HP Worldwide Customer Service Organization

Asking the Unaskable

Ferguson, the product manager at HP, points out that decision support systems have been around for years, but answering previously unasked questions about customers was a painful process. Because it would take IS two to three months to do a new report, "very few questions were asked," he says.

Today, thanks to several data warehouses and tools designed for end users, Ferguson and others can get detailed answers quickly. "We can look at our customers more broadly and see the business we're doing with them," he says. "We ask a lot more questions."

That's what Makishima, the data warehouse manager, intended when his team began building the Worldwide Customer Service Organization warehouse two years ago. Customer support was once merely reactive, responding when the customer called. "Now when the customer's contract expires, we can go out and inform them of new programs and see if they are interested," Makishima says. "It gives us a competitive edge."

Customer service used to be handled by a mainframe at corporate headquarters. With users demanding more detailed information, the system took too long to load data and too long to respond to queries, and it cost too much to support. For those reasons and obvious technological preferences, HP decided to migrate off mainframes.

After investigating several products, Makishima's team chose software from Red Brick Systems of Los Gatos, CA, to run on HP 9000 servers. Makishima says IS wanted to get out of the ad hoc reporting role and provide data so users could select what they wanted. Red Brick's software loads millions of rows of data speedily and supports new data access tools.

There was strong management sponsorship. Marketing and finance lent people full-time to the project. "We had a good idea how users would use it, but when we modeled the Red Brick system we were able to check with them," Makishima explains. When the team began to build the warehouse, the design called for 20GB. It's now at 100GB with room to grow to 200GB. "Our users have realized the importance of historical data," he says.

The biggest challenges appear to be assuring the integrity and timeliness of data, and providing support to users worldwide. "At first we had a handful of users. In a little over a year we're at more than 400 and growing at 18 end users per month," Makishima says. "Originally the end users were just in customer support. When the word got out we had all this consolidated information, other people started using it." The demand is more than Makishima's staff of six can support. HP is adding individual business information managers around the globe, under the budgets of the workgroups they support.

Another problem was that management wanted the job done in three months. "We got it up in about four or five. That was pretty good, because warehouses were taking more than a year," Makishima recalls. "That's one reason we don't always use HP products. We don't have time to reinvent the wheel, so we leverage partnerships with other companies."

Perkin-Elmer Corp.

Out with the Greenbars

In the beginning there were greenbars: those mainframe-generated reports printed on reams of green-and-white-lined paper. The greenbars begat a DOS-based EIS. EIS begat a Unix-based relational database, which begat four regional data warehouses. Now Perkin-Elmer Corp. is looking at a World Wide Web solution to give one-fourth of its 5,500 employees access to rich data stores.

In 1990, an executive with technical savvy was put in charge of the product group that accounted for half of the $1 billion revenues at the Wilton, CT-based laboratory instruments manufacturer. He wanted to know what products were ordered where. But all data resided on a mainframe; greenbars came out monthly. Daily greenbars could be cranked out only painfully.

John Stoveken, a consultant, led a team that took a couple of years' worth of data off the mainframe and built a DOS-based EIS. "There were probably 10,000 rows of information," recalls Stoveken, who today is the company's DSS manager. "It was world order information, by country and by product. It started off under 10MB. It was accessed by a small group and grew to 100 users."

The users demanded more. The system grew to 800 product groups. Next, users asked for information on 15,000 to 20,000 active parts. When they asked for the entire parts list of 130,000, IS saw that it had reached the limits of DOS. About three years ago, a decision was made to move to a data warehouse. A first step was selecting the Oracle RDBMS on a Sun Sparc 1000. "We started to grow the data systems and to look for tools to manage the data we were bringing in," Stoveken says.

During the pilot test, Stoveken discovered CrossTarget, a client/server multidimensional analysis and reporting system from Dimensional Insight of Burlington, MA. The CrossTarget engine uses array indexing to transform data from spreadsheets, SQL databases and legacy systems into multidimensional models. The client, called Diver, lets users view data as they wish, perform analysis and "drill down" into subsets. The models, of 250MB or less, are built from data stored on Oracle and go out over the LAN to be used by anyone who has the client software. This way, Stoveken's 10-person staff keeps control of the data in the warehouses.

U.S. users were happy. "Then people around the world started asking for more detail," Stoveken says. "We said it's time to do data warehousing on a global basis." They chose four HP 9000s running Oracle at manufacturing sites in California, Connecticut, Germany and the United Kingdom. "When you have 40 different order-entry systems around the world, there are coding inconsistencies," Stoveken explains. "So we use the data warehouse as a central point where we can clean data, take out the coding problems and make sure it's a well-defined set."

What's next? With a goal of giving as many as 1,500 employees access, Stoveken is looking at CrossTarget's new Hypertext Markup Language (HTML) version and considering using it over the company Web. "The company runs a WAN that stretches across the U.S. to Germany to Japan. Through our intranet, we can open up some of these larger models to the global users."

CoreStates Financial Corp.

Getting Intimate with Data

If a customer doesn't answer the letter that offered a special interest rate on a new credit card, CoreStates Financial Corp. wants to know why. That level of data analysis requires a staff that is more intimate with the data than are typical IS staffers.

At CoreStates, a Philadelphia-based bank holding company with $30 billion in assets, the retail credit risk technology unit not only analyzes the data, it manages it, develops its own applications and is building a data warehouse. "Our group is comprised of analytical people, used to dealing with large quantities of data and turning it into competitive advantage," says Jeffrey Oulton, vice president of retail credit risk technology. "They're not reading batch runs and turning them into spreadsheets. They're merging and analyzing data for decision-makers."

Within the 20-person team are people dedicated to building a data warehouse. "They like data and like to code, but they understand business," says Oulton. "They're our liaison to the technology side of the house; their mission is data acquisition and making it available to others."

At the heart of credit risk analysis is scoring. Within the group, one team works on the front end of the business--determining whether to extend credit. Another concentrates on the back end, traditionally known as collections but increasingly a source of new revenue, including offering bigger lines of credit. These two groups were separate and worked off different sets of data. As a result of a reengineering effort last year, they combined. "The piece that is completely new is the data warehousing," says Oulton.

They're migrating data off a mainframe with the DB2 RDBMS to an IBM RS/6000 server running Oracle. The LAN is Novell NetWare. "The entire corporation is also building its own warehouse. There will be links between them," says Oulton. "We're ahead of the rest of the corporation. The corporate IT guys are not as comfortable with the Unix environment."

Oulton praises the IS side of the house. "They understand that some end users understand the data better than they do. IT is very supportive of the project. We play on their expertise. Data modeling, loading in the DB2 environment, the decision whether to move to Oracle and how--that's their call."

Using tools and consultants from SAS Institute, Oulton's team has built application suites for three types of users. The EIS is a highly summarized GUI with some drill-down capabilities for about 20 executives. The power-user version satisfies the people in his unit. And a middle-of-the-road suite serves about 200 others who need access to data. "We've been able with one tool to address three levels of user," he says. "That's important from cost justification and support standpoints."

Bill Roberts is a free-lance writer who covers business, technology and management issues. He can be reached at