Enterprise storage is currently in the midst of a great transformation the likes of which has not been seen in well over two decades. There’s been an unprecedented amount of activity in the storage space this year, with a host of promising new companies launching, and major storage vendors beginning to drastically retool their architectures to accommodate the modern IT realities of virtualization and cloud computing.
With that in mind, here’s a look at the innovations and trends that are driving this phenomenon and reshaping the storage landscape.
Moving the Bottleneck
Revolutions in IT are often linked to advances in underlying component technologies. Some technologies, such as CPUs, improve predictably every year like clockwork. Other technologies, such as networking, improve in larger steps once every decade or so, and still others, such as disk access times, improve very slowly, and have not changed dramatically in the past 30 years. Although large changes are more likely to cause disruption than small changes, this is not always the case. Whether a change, large or small, will be disruptive is less dependent on its size and more on whether it moves the bottleneck.
A bottleneck is the component of a system that most limits the usability of the system, whether it is measured in terms of capacity, performance or manageability. It is usually the most expensive component of the system and the one that requires the most time and resources to maintain. Because of its importance, entire industries are created to address the bottleneck.
Now, imagine what happens if a new technology comes along and suddenly moves the bottleneck to an entirely different component in the system. Users would experience an immediate improvement in the usability of their system, but much of the resources invested in existing products, which targeted the old bottleneck, would become far less useful. And because such legacy products have become specialized and more complex over time to better address the old bottleneck, they would be difficult to modify to address the new bottleneck.
The Bottleneck Is the Storage
In today’s data centers, the bottleneck is the storage. The storage system is the most expensive component to purchase, the most costly to maintain, and the source of the most difficult management and performance problems. Storage today is also under rapid change due to the advent of multi-core CPUs, 10 Gb Ethernet and flash, which are rapidly replacing hard disks in performance-intensive applications.
After spending decades designing and optimizing storage systems to work around scarce CPU cycles, low-bandwidth networks and slow mechanical hard disks, these problems have now disappeared along with many of the advantages existing vendors had created with decades of investment in existing storage technologies.
The Rise of Flash
Flash has emerged as one of the most disruptive technologies in IT in recent years. It has significantly increased performance capabilities for enterprise storage for a number of reasons, including:
- More attractive economics: According to IDC, flash storage prices have dropped from about US$21/GB five years ago to less than $2/GB for MLC flash today. This is still far more expensive than disk, but flash provides 1,000 times more performance in a single drive.
- Lack of compatibility between flash and legacy storage: Mainstream enterprise storage systems are struggling to effectively incorporate flash into their architectures. Their storage systems were designed in the 1980s and 1990s, well before flash became relevant. These storage systems were architected to handle arrays of hundreds or even thousands of hard disks that behave very differently than flash and often hinder the efficient use of flash.
- New flash-specific requirements: Efficient use of flash requires the application of new technologies, such as inline de-duplication and compression, wear-leveling, garbage collection, and managing write amplification and latency spikes. These new technologies are difficult to retrofit into existing storage architectures that have already been made complex by decades of optimizations aimed at improving the performance of mechanical disk drives. Imagine attempting to use a storage system built for tape drives to manage hard disks. This is equivalent to traditional disk systems attempting to adapt to flash with “bolt-on” techniques — which are extremely expensive, to boot.
- Modern solutions are designed for flash: The recent wave of storage startups aren’t burdened by legacy file systems and a large installed pool of disk-based arrays. They can design their systems to make much more efficient use of flash and fully leverage its potential.
- Hybrid storage systems: Although flash has many advantages compared with hard disks, flash capacity is still far more expensive than hard disk. Therefore, most data centers will use a combination of flash and hard disk, creating additional work in managing the use of these storage resources. Hybrid storage systems which automatically place the data can greatly reduce these overheads.
Other Key Innovations for Enterprise Storage
In addition to flash, there have been a number of other notable IT innovations that have enabled this new wave of flash-based storage. Multi-core processors, when combined with a modern file system design, enable previously unworkable technologies like inline deduplication and compression. This has a transformative effect on flash cost-effectiveness and system cost/performance.
Another recent innovation that has aided the evolution of storage is 10 Gb Ethernet. While it was first introduced in 2002, 10 GbE has only recently become more cost-effective and thus easier to integrate into storage offerings. Inexpensive 10GbE now offers higher throughput than Fibre Channel with far less cost and complexity. Cost per port continues to drop quickly, and the advent of excellent non-blocking 10GbE switches clearly positions Ethernet as the dominant networking topology.
New Fronts in the IT Landscape
Individually, each of these technology shifts is important, but the combination represents a revolution in the IT landscape.
Will the establishment quickly retrofit their already complex products with these new technologies and create innovative solutions to repel the invaders, or will they be overrun by them?
Loading ...
