Intelligent Storage:The Big Picture

As traditional storage strategies become increasingly inadequate, new alternatives are rising to the challenge

By Arun Taneja

Executive Summary

Arun Taneja Most enterprises are doubling the amount of data they must manage every year. Unfortunately, the existing storage infrastructures are ill-equipped to deal with this explosion of data. New technologies, including NAS, SAN, Fibre Channel, InfiniBand, iSCSI, and virtualization can help you better manage your data and prepare for a future escalation in your storage needs.

Once upon a time, storage was boring; it was simply a disk drive storing your data. Application servers were the center of the universe: they required strategic planning. After all, once you decided on the server vendor, you just bought the storage that came with the server. Times have changed. Today, storage rules.

Now, your first decision is which storage vendor you should use, then the application vendor. Finally, you are ready to choose the servers. Visualize these elements with storage in the center, applications in the next ring, and servers in the outermost ring. Numerous reasons for these shifts exist, but primarily it comes down to the following factors:

Storage costs as a percentage of the total IT budget have steadily climbed and now make up its largest piece. In the early 1990s the split between server and storage spending for enterprise class servers was 70/30; now the ratio is reversed. Also, the significance of data has evolved from being an important part of an organization to being the lifeblood of a company today. The sheer competitiveness of your company depends upon how much relevant customer and market data you collect and, equally importantly, how you manipulate it to extract information for decision-making. Clearly, the type and quality of storage infrastructure is more important than ever before.

The Internet has been single-handedly responsible for the explosion of data. Enterprises adding clicks to bricks and those based purely upon clicks are adding huge amounts of data - with no end in sight. Most organizations are doubling the amount of data they have to manage every year. To get a sense of this incredible growth, consider the fact that in 2001 alone, enterprises will add 400 to 500PB (1 petabyte equals 1,000TB) of new data, which is only slightly less than all the historical data that enterprises have ever created. Also, companies have a greater need to maintain more data online for business or legal reasons.

The type of data is also changing dramatically; it is no longer predominately textual and numeric. The user now expects to see graphics, audio, and video file attachments. A simple audio file can easily take up 1MB: video files are even hungrier for storage. A two-minute video file can require 10MB of storage.

One hundred percent uptime is the only acceptable status for your Web site in the global economy. Your Web site must always be open for business. And just think, only a few years ago, vendors talked grandly about a 99 percent uptime. In the past, vendors and the IT community would distinguish between unplanned vs. planned downtime. Somehow planned downtime was acceptable, or at least condoned by the end users. Today, the end user could care less about the distinction and has zero tolerance for any downtime whatsoever. Consider online buyers: If they go to a Web site and the site is unavailable, for whatever reasons, they move on and the sales go to the next vendor.

Time to market means life or death to an enterprise in today's Internet economy. If your company can introduce and ship a product a quarter earlier, it could potentially mean millions of dollars in terms of incremental revenue and other benefits associated with a first-mover advantage.

All of these factors combine to make effective gathering, moving, and storing of data the most important aspect of an enterprise. That means as an IT executive, if you don't make the right strategic decision regarding storage, you are potentially jeopardizing your company's entire IT infrastructure. The day of storage is finally here.

Unfortunately, existing storage architectures are practically useless for managing vast amounts of data - further complicating the situation. So what is an IT executive to do? Before I provide you with some guidelines for success, let's look at the current state of storage infrastructures and then examine what's new today and what technology is almost here.

Infrastructure Now

The dominant current storage infrastructure is called direct attached storage (DAS). Basically, this infrastructure means that application servers, database servers, and file servers are on a network, typically a TCP/IP-based Ethernet, interconnected with routers or switches, with each server - usually using a SCSI bus - with its own storage attached directly to it. The client computers are mostly PCs, Macs, or Unix workstations, also on these same networks, and with varying amounts of local disk storage. In this environment, because each server has a maximum capacity for storage, once you reach that limit, you have to add another server, regardless of the remaining processing power of the CPU (see Figure 1). Similarly, if the server runs out of processing power, you have to add another one, even if the existing server has plenty of storage capacity left. This one-to-one relationship between server and storage is a fundamental issue for scalability and manageability of storage required for today's economy.

Because data on a storage unit is "owned" by the server to which that the unit is attached, accessing information simultaneously from multiple storage units involves all associated servers - a highly inefficient process for your databases and many e-commerce applications. Also, consider the impact when one of these servers is down: Data associated with that server is unavailable. This server-storage association also requires that you manage each storage unit separately, making overall storage management in a large enterprise with hundreds or thousands of servers an absolute nightmare. Small wonder that the IT community, already reeling from budget crunches and lack of storage-centric personnel resources, is fighting a losing battle and crying for help.