RAID++: Lessons from the 1990's market collapse for Large Expensive Disk Drives

Following a previous post, this is an examination of the causes of the collapse in IBM's Mainframe DASD (Direct Access Storage Device) business, from an 80% share of an $11.B market in 1990, to ~25% a decade later. (Source PDF). Notably, IBM's DASD business wasn't the only line to suddenly collapse: the Mainframe business as a whole collapsed, creating a loss across the whole business, the first since 1914.

Mainframe market share 1991 – 1998: % total Terabytes shipped

	1991	1992	1993	1994	1995	1996	1997	1998	Avg. ∆ / year
EMC	2	5	17	32	40	42	48	51	+59%
IBM	66	61	48	36	36	29	30	27	-12%
Hitachi	22	23	25	19	11	18	18	18	-3%
StorTek	2	1	0	5	10	9	0	0	N/A
Amdahl	8	10	10	8	3	2	4	4	-9%
Total TB	-	-	703	906	1,581	2,546	3,367	5,019	+48%
Total $MM	-	-	4,467	3,937	4,176	3,684	3,121	2,964	-8%
Avg $k/GB	-	-	6.35	4.34	2.64	1.45	0.93	0.59	-38%

EMC and Storage Technology (StorTek or STK), took different paths to releasing Mainframe RAID products while NCR focussed on Unix/Mini computer products with  a SCSI device priced $20,000-$100,00.

EMC and StorTek both already built Mainframe 'Channel' attached devices (e.g. ESCON) and a sales presence at customer site. Stortek's 9200 Iceberg may have been released in 1994 (or 1991 above), while EMC's 4200, a much smaller and simpler model, was for sale in late 1990. In 1996, Stortek and IBM signed an agreement to resell the Iceberg exclusively as IBM's RAMAC.

EMC first model wasn't a high-reliability RAID, but was small, fast, more expensive per GB than IBM's 3390 and targeted at a very specific niche of users. It released new models ever year, increasing features and capacity. It leveraged its first mover advantage to good effect.

The Mainframe market was demanding with Performance and Reliability/Availability were prime metrics, with per-unit capacity and Price per GB were less important.

RAID only became possible in the Mainframe market because of three developments elsewhere:

• The 1987/8 Berkeley paper on RAID providing, without patent licensing restrictions, the theoretical base for RAID-5 and 6, allowing arbitrarily large capacity, reliability or performance (random IO/sec) devices to be built, albeit at a premium. Operationally, these were compelling reasons.

• The rise of micro- and mini-computer (or low- and mid-range) markets, with a strong market of independent Storage vendors charging normal commercial margins of 20%-30%, not the "premium-priced" Mainframe products with 50%-70% gross margins. The competition between vendors grew both the size of the market, kept margins down and increased the rate drive capacity increased, as per-unit input costs were relatively static, pushing vendors to seek higher storage densities.

• The creation of Industry Standards for disk sizes & connectors and disk controller interfaces:
• SCSI series for mini-computers. Initially 5Mhz and 8-bits wide for 40Mbps transfers using standard connectors, and
• IDE (now called ATA) as the common disk interface.
• Specialist vendors, such as Adaptec, built disk controllers for many types of computers and different drive interfaces, IDE and SCSI. 
• The two drives mentioned in the Berkeley 1987/8 paper, the larger 600MB Fujitsu 'Super Eagle' were 5.25", and smaller PC drives, such as the 100MB Conner Peripherals were  3.5".
• As the PC market expanded, demand for 3.5" drives, by units and total capacity, rose dramatically faster than all other segments.
• Increased volumes and intense competition drove the PC-drive vendors to improve products, yield, capacity, performance and prices through automation and improved production facilities.
• Around 1995, 3.5" drives supplanted 5.25" drives in RAID arrays.

Mainframe vendors, specifically EMC and Stortek, saw the potential for the potential for substituting low-cost, low-performance and lower-reliability mass-market drives for the expensive 'DASD' drives sold by IBM.

We know from NCR and HP's efforts in the low and mid-range markets that those clients also valued new features that RAID could bring then, even though it was more expensive per GB because low- and mid-range computers were already directly using the 3.5" and 5.25" assembled in RAID arrays.

Data General's Clarion line of RAID devices for mid-range (Unix) systems was acquired by EMC within the decade, allowing them to access that market segment as well.

There wasn't a widespread, strong demand in any market segment for all the benefits provided by RAID. However, in almost all market segments, there was one or two compelling features of RAID that compensated for the additional cost and complexity of RAID arrays, allowing the large vendors to continue with 50+% gross margins for some time.

The Mainframe segment is not mostly price-determined. While not insensitive to price, they are sophisticated consumers with deep pockets, taking time to assess vendor offerings against each other on multiple criteria and able to gain deep discounts on published prices. While we can see the aggregated sales figures for EMC et al, we don't know the discounts they offered to major clients to win sales.

In a "premium-priced" market, the idea of deep discounting seems odd, but is a sales tactic used by many vendors.

Summary:

• The replacement of SLED's by RAID in the Mainframe market was fast, and for IBM bloody: an archetypal Christensen "Disruptive Innovation".
• This sudden change was primarily due to commercial, not technological factors.
• EMC conclusively demonstrated that RAID's competitive advantage wasn't just price, either per-unit or per GB.
• The technological conditions creating the base for RAID took nearly 2 decades to build. They started with the competition between mini-computer vendors (e.g. DEC) of the early 1970's forcing innovation across all their product lines.
• Two necessary critical conditions were:
• The license-free Berkeley RAID design of 1987/8, and
• the ramp-up in IBM pricing on all its Mainframe products during the 1980's prior to the 1991 collapse in profits,
• The large and vibrant market in low- and mid-range computers, pushing innovation and preventing excessive margins.
• The existence of multiple "plug-compatible" vendors in the Mainframe market, with established sales channels and good reputations, and that could quickly develop new products.
• The development of high-end RAID products helped create a RAID market for mid-range computers.

The IBM management team that created this perfect storm during the 1980's cannot be criticised: they followed the MBA "milk the cash-cow" strategy perfectly.

Only with hindsight can anyone criticise their actions and decisions: from the data available to them at the time, there was nothing to indicate they should be acting differently. Clayton Christensen and others have written extensively on this effect.