The NBN and defending against Cyber warfare attacks.

CYBER-WARFARE and the Australian NBN.

We know from "Stuxnet" that Nation States are actively building and deploying Cyber warfare tools, applying them to National Security concerns and running them as Military, not technical, operations. This includes accurate reconnaissance and network topology and vulnerability mapping. The worst case is that attackers will gain access to the network control tools and infrastructure.

Recent coverage suggests that Obama denied a US Military request to launch a cyber attack on Syria's infrastructure during the recent 'troubles'.

From the "slammer" worm, we know that any Cyber warfare attack will be fully developed within 3 minutes, and any attack will be launched at the worst possible time for defenders, possibly accompanied by physical distractions.

Recovery from "munitions grade" worm/malware compromise will be long and expensive. Experience is that malware infections is as damaging to businesses as a fire: Within 12 months of a fire, 80-90% of small businesses fail.

The NBN as an Essential Strategic Defence for Cyber-warfare.

Whilst reading this piece today I 'had a thought'.

"Militarisation of cyberspace: how the global power struggle moved

online",  Nick Hopkins, guardian.co.uk, Monday 16 April 2012 15.00 BST

One argument in support of the NBN I've not heard is about Security, but not the "how to keep your bank account and credit card safe" kind - the usual direct theft or Identity Fraud talked about at Cyber-Security conferences.

The National Security kind that interest the Intelligence agencies and Military, a.k.a. "Cyber-warfare".

This is as far removed from normal Cyber-security as guarding bank vaults is from fighting a war. Attack, and hence Defence, is taken to a whole new level: because the resources employed and what is at stake is taken to a whole new level.

The NBN we had to have, and the one we'll likely get.

Yesterday I read Robert Gottliebsen's, "Who wants a poor man's NBN?" and thought, "Am I the only person who remembers why we had to have the NBN?"

We got the NBN for a number of reasons, few of them necessary or good, in my opinion:

Smart-Grids and Carbon Trading: Enough for an economic Negawatt scheme?

Can the Smart-Grid and Carbon Trading create a new, economically viable marketplace in saving power?

In 1997, Amory B. Lovins co-authored a book with a radical new idea to address the Energy Crisis and our Environmental problems (now it might be "Climate Change"):

the negawatt negative watts of power consumed by creating lower demand through more efficient power use.

It is still much cheaper to "save a watt" than for a Power Generator to "build a watt", and the marginal cost of production and distribution is zero to the Generator. The consumer still has to maintain and replace their infrastructure investment.

So why hasn't the negawatt market happened? What's different now that it could work?

CyberWars, Governments and Internet Security

There's an 800-lb Gorilla in Internet Security that nobody discusses or acknowledges:

If Governments decide to apply their Technical and Military Intelligence skills to the Internet, not only won't we know, we won't be able to do anything about it.

Talking to a friend recently, off the top of my head I outlined 4 levels of Internet attackers/exploits (highest level/most competent at the top):

• [4] National Military and Commercial Intelligence: surveillance, espionage, counter-espionage, targeted cyber-attack.
• [3] Commercial Espionage and "Exploit as a business": Exploits and SPAM as a Service, botnets, Credit Card and Identity trading.
• [2] small-scale, "hobbyist" and semi-professional technical creators. Some sales to level [3].
• [1] script-kiddies, Internet "graffiti"/vanity attackers, customers of level [3].

These levels may or may not be "official" and may not be complete. But they are roughly right.

Security threats "in the network": detection and countering

A new Internet Security report for July-Dec 2011 from M86 is out: "New M86 Security Labs Report Reveals Spread of Malware Growing via Social Media, Targeted Attacks and Exploit Kits" [PDF]

It triggered a thought that first occurred to me during the "No Internet Censorship" campaign:

The perfect place for those wanting to hide illegal activities is "within the network", to work as Admins for Internet Providers. They can monitor, avoid and intercept Law Enforcement etc. requests and respond in many subtle ways.

This thought arose after two rather disconcerting incidents for me:

• A TV documentary on Internet Porn mentioned the officers have to view these images and that it can lead to desensitisation over time, and
• an unprovoked personal attack within an Admin's forum by a "security professional" upon an individual. Sexually explicit language was used and that the language went unremarked by the entire forum was gob-smacking for me.

 We also have the phenomena of the group "Anonymous" acting as Internet Vigilantes. Again, an Admin (with unlimited access rights) within a major Internet Provider is the perfect place for such groups and their activities.

Microsoft Troubles: XV. My prediction not fulfilled

In 2006 I looked at the Microsoft Annual Report and thought "I've seen this before, their market is being taken by newer 'substitutes'. They'll hit a 'financial pothole' within 5 years, give or take."

Well, that hasn't happened, so my prediction was wrong.

Microsoft Troubles XIV: iPhone sales bigger than total MSFT Revenues

Another interesting piece from Business Insider: "Apple's iPhone Business Alone Is Now Bigger Than All Of Microsoft".

Steve Ballmer can't be a happy man... His competitor that almost died, but they helped save, looks like it's "done an end-run" around their business and is trouncing him convincingly.

Why Fibre Channel SAN's will be dead in 5 years.

I won't be buying shares in any Fibre Channel-based tech stocks as I think the technology will be dead within 5 years for two reasons:

• Enterprise Storage Arrays will stop being used for high-intensity random I/O, instead being used for "seek and stream", and
• PCI Flash or SCM Storage will become the low-latency "Tier 0" Storage of choice because of speed (latency), cost and simplicity.

Update 1-Feb-2012:
Another article by Fusion-io, Getting the most out of flash storage  provides extra links:

• Flash storage moves closer to CPUs: STEC is moving into PCI Flash cards.
• EMC: Flash could spell doom for Fibre Channel but don't talk about PCI Flash challenges.
• Flash storage in post-PC devices advances, general background on Flash memory.

---

Elsewhere I've written Jim Gray's observation, "Disk is the new Tape" should be "Disk is the new CD".
That is, Enterprise Storage is best suited for "Seek and Stream", not random I/O.
Enterprise Storage Arrays will need to provide in this future:

• reliable persistent storage and archives,
• high capacity and best $ per MB, and
• high bandwidth streaming IO.
• and if you were being honest, vendor-neutral management and protocols, in-place upgrades, any-time snapshots/backups and flexible zero-downtime expansion and reconfiguration.
  The current need for "fork-lift upgrades" and vendor incompatibility are disadvantageous to customers.

How to create a Storage Network that needs to be fast, cheap, simple, robust/reliable, secure and scalable when "super low-latency, zero jitter and non-blocking IO" is taken out of the mix?

Ethernet and nothing else.

Whether Layer 2 protocols, like Coraid's ATA-over-Ethernet, or Layer 3 protocols, e.g. the slower, higher overhead but routable iSCSI, dominate is still an open question.
Both have strengths and weaknesses and can be used together very effectively without conflicts to maximise ROI's and minimise Enterprise Storage costs, both CapEx and OpEx.

Ethernet is around 10-100 times cheaper than Fibre Channel to install and configure and requires only a fraction of the Administration and support, because Enterprises already have well resourced and competent Networking teams. Network Engineers are in much better supply than SAN specialists, so wages are more reasonable and availability much, much higher.
Their competency/capability is also much better able to be assessed by technical managers when both hiring and firing.

As well, Ethernet has a current growth path of 40Gbps and 100Gbps with 10Gbps widely available now for servers.
Fibre Channel may improve sometime in the future to 12Gbps, but that's an uncertain roadmap and with a global market in 10,000's vs millions for ethernet, the cost differential will only grow.

Fibre Channel has become a very poor choice when bandwidth is the primary "figure of merit".

In access latency, PCI-based Flash Memory, such as Fusion-io's, will always beat SAN-based Storage Arrays by a rather large margin.

It's there in the physics and unbeatable...

All the interfaces, line delays, buffering and switching - out and back on a SAN - means if even the Storage Array SSD's had zero latency, it would be many-times slower.

This Q and A with Matt Young of Fusion-io on "Making Flash Fast",  says it well:

Q: How does your ioMemory technology differ from Solid State Disks?  And how does it compare performance wise?

A: Solid State Disks or SSDs are used to store data with the intention of constant use – similar to that of a hard drive. These SSDs generally use disk-based protocols that introduce unnecessary latency into the system. Fusion’s ioMemory technology differs in that it doesn’t act as a hard drive. It performs as an extension of the memory hierarchy for servers. This means that they provide a tighter integration with host systems and applications, helping you to work more productively.

Fusion-io products offer the industry’s lowest latencies, which maximise performance and scalability, while delivering enterprise reliability.

and

Q: Can you provide some typical I/O performance figures for ioMemory compared to DRAM, and solid state disk?

A: With some generalisation, the order of memory, fastest first is as follows,

• DRAM with 100-300 nanosecond access,
• ioMemory with 15 microsecond access,
• NAND appliances with around 500 microsecond access and
• then SSD’s with around 1ms [1000  microsecond] access. [66 times slower...]

There are of course a number of factors that need to be considered in these times such as payload size, load, etc, however, in simple terms with all things equal and a well-designed product, latency is ultimately affected by the distance data must travel to get to where it is useful.
So, the closer your technology resides in relation to the CPU the better the response time.

That’s why that even though two products may use the same NAND chips and be connected on the PCI Express bus, you see markedly different latency characteristics.

finally:

Q: As well as I/O performance, what other attributes of ioMemory are finding favour among customers?

A: One of the things that our customers tell us provides a major benefit in addition to performance is the reliability of ioMemory and the cost savings generated from implementing Fusion-io solutions. Fusion-io products are uniquely reliable enough to be offered by all major OEM manufacturers, including Dell, HP and IBM.
[snip]

Finally, many customers tell us that they save a lot of money on CapEx and OpEx, since ioMemory takes so much less power, cooling and real estate than traditional, scaled-out storage infrastructures.

---

Declaration of Interest:

I have no shares or other financial interest in Fusion-io or any companies or their competitors mentioned in this piece.
I am not employed now, nor have ever been, by Fusion-io or any of its related/associated entities.
I receive no remuneration for writing these opinions/analyses.

(signed) Steve Jenkin, 31-Jan-2012.

How they got to 1 Billion IO per second.

Is Fusion-io's demonstration of "1 Billion IO operations per second" the same sort of game-changer that the 1987/8 RAID paper by Patterson, Katz and Gibson was?

Within 5 years all "Single Large Expensive Disks" (SLED's) were out of production, will we see Flash disks in Storage Arrays and low-latency SAN's out-of-production by 2017?

A more interesting "real world" demo by Fusion-io in early 2012 was loading MS-SQL in 16 virtual machines running Windows 2008 under VMware. They achieved a 2.8-fold improvement in throughput with a possible (unstated) 5-10 fold access-time improvement.

Updated 16:00, 31-Jan-2012. A little more interpretation of the demo descriptions and detailed PCI bus analysis.

Fusion used a total of 64 cards in 8 servers running against a "custom load generator", or 16 million IO/sec per card.
There are two immediate problems:

• How did they get the IO results off the servers? Presumably ethernet and TCP IP. [No, internal load generator, no external I/O.]
• The spces on those cards (2.4TB ioDrive2 Duo) only rate them for 0.93M or 0.86 sequential IO/sec (write, read) with 512 byte, a 16-fold shortfall.

The IO's used in the Demo were small, 64 byte, plus they avoided the Linux block driver sub-system, using their Direct Memory Mapping scheme, "Auto Commit Memory" (ACM) versus their Read-only Cache, "ioTurbine".
The card spec sheet also quotes throughput of 3GB/sec (24Gbps) for large read and 2.5GB/sec (20Gbps) for large writes (1MB I/O).

There was no information on the read/write ratio of the load, nor if they were random or sequential.
From a piece on Read Caching also speeding up writes by 3-10 fold, Fusion show they are savvy systems designers, they use a 70:30 (R:W) workload as representative of real-world loads.

That nothing was said about the workload suggests it may have pure read or pure write - whichever was faster with ACM under Linux. If the cards' ACM performance tracks the quoted specs (via a block mode interface), this would be pure sequential write.

The workload must have been 50:50 to allow full utilisation of the single shared PCI bus on each system, otherwise the bus would've saturated.

Also, as this is as much a demonstration of ACM, integrated with the Virtual Memory system to cause "page faults",  the transfers to/from the Fusion cards were probably in whole VM pages. The VM page size isn't stated.
In Linux pages default to 4KB or 8KB, but are configurable to at least 256MB. Again, these are savvy techs with highly competent kernel developers involved, so an optimum page file for the Fusion card architecture, potentially 1-4MB, was chosen for the demo. [Later, 64KB is used for PCI bus calculations.]

The Fusion write does not say how they checked the IO's were correctly written. With 153.6TB total storage and 64GB/sec in the test work load, the tests could've run 2,500 seconds (40min) before filling the cards. Perhaps they read-back the contents and compared that to the generated input, though nothing is said. In the best of all worlds, there would've been a real-time read-back check, i.e. a 50:50 R:W workload.

The 64GB/sec total I/O throughput gives 8GB/sec, or 64Gbps, per server.
The HP ProLiant DL370 servers used in the tests, according to the detailed specs, only support 9 PCI-e v 2 cards, most slots are 4 lane ('x4'), at 4Gbps per lane, bi-directionally.  With 8 slots taken by the Fusion-io cards, only one (x16?) slot was available for the network card needed to supplement the 4 1Gbps on-board ethernet ports.
80-100Gbps ethernet capacity would normally be needed to support 64Gbps of IP traffic.

Reading the "datasheet" carefully, including inferring from the diagram which has no external connections between systems and no external load-gen host, an internal load-generator was used, one per host. There may have been some co-ordination between hosts of the load generators, such as partitioning work units. From the datasheet commentary:

Custom load generator that exercises memory-mapped I/O at a rate of approximately 125 million operations per second on each server. Each operation is a 64-byte packet.

A little more information is available in a blog entry.

PCI Express v2 is ~4Gbps per lane, or 64Gbps for x16, each direction. Potentially just enough if an on-board ToE was used either with a 8-way 10Gbps card, a dual-port 40Gbps card or a single-port 100Gbps card.

From this, we're can't be sure if the load generator was internal to the Linux servers or external. 

Even though not used, HP support dual 10Gbps ethernet cards, PCI-e v2 x8, in the DL370 G6, but with a maximum of two per system. This suggests a normal operational limit of the PCI backplane of 20-40Gbps per direction. The aggregate 64Gbps is achievable if split into 32Gbps in each direction.

The Fusion cards "half-length" and are x8 (so will work with x1, x2, and x4 slots as well).
From the DL370 specs, the system has 8 available full-length slots:

• 2 * x16,
• 1 * x8 and
• 5 * x4.

The other half-length slot is probably x8.

The per-server load is 64Gbps, spread amongst 8 cards, or 8Gbps/card, which is only 2 lanes (x2).
The per-card bandwidth would be possible if the single shared PCI bus wasn't saturated.
Per direction, the x4 PCI-e lanes would only support 16Gbps and the x8 32Gbps.
The simple average (5 * 16 + 3 * 32)/8, or 22Gbps, is insufficient for the load.

The two x16 slots and x8 slot could support the maximum transfer rate of the Fusion cards, 32Gbps per direction, or an aggregate of 64Gbps. The cards' spec sheet allows 20-24Gbps large (1MB) transfers per card, which with some load-generator tuning, could've resulted in 60Gbps aggregate from just 3 cards.

If the I/O total load, 64Gbps, is split evenly between cards, each card must process an aggregate 8Gbps, with equal read/write loads, or 4Gbps per direction.

If 64KB pages are read/written, then each card will need to process 64K (65,536) pages per second per direction.

The x4 slots, with 16Gbps available in each direction (aggregate of 32Gbps), will transfer 64KB in 15.28 usec.
The x8 slots will transfer a 64KB half that time, 7.63 usec.

The average 64KB transfer time for the mix of cards (5 * x4, 3* x8) in the system is:

( (5 * 15.28) + (3 * 7.63)) / 8, or 12.40 usec,

or 80,659 64KB pages per second per direction, leaving 25% headroom for other traffic and bus controller overheads.

The required 32Gbps per direction seems feasible.

This either says the DL370 has multiple PCI-e buses, not mentioned in the spec sheet, or something else happened.

Smart Internet+Smart-Grid: making money and reducing carbon footprint

Two technology/commercial trends are coming our way in Australia:

• the Internet everywhere (3G wireless or NBN), means "smart controllers" will be cheap, simple and everywhere. They will be able to trivially hook into 3-7 day local forecasts, especially useful for air-conditioning units.
• "Smart Power" metering will start to charge power at different prices during the day, rather than the disconnected traditional pricing of "a single price whenever you use it".
  You can see real-time wholesale electricity prices on-line and a Pretty graph in 30-min periods.
  Yesterday (12/12/11), the 30-min price varied from $52/MWhr @ 4PM to $16/MWhr @ 2AM. In 5-min periods, the price ranged from $95 to $16.

There's a bit of background you may or may not know about Power Generators: they over-build capacity to meet any and all demands placed on them. There aren't just no incentives for Power Generators or their customers to reduce either aggregate or instantaneous demand, but the reverse: significant economic disincentives to reducing demand, and hence to lowered income and profit. This is a perverse economic outcome costing us a lot of money and burning carbon unnecessarily.

From "An EnergySmart Plan. Positioning Queensland for a Diversified Energy Future 2010 - 2050" [original dead link] (Nov 2010 report for Queensland Government):

Ergon and ENERGEX will each spend $6 billion (that is $12B combined) in capital expenditure over the next five years to cope with extraordinary consumption during a fraction of the year, rather than the average consumption over the course of the year.

To put this into perspective, ENERGEX has over $900M in assets that are only used for approximately 3.5 days per year. (Mark Paterson, ENERGEX, The SPRA Standard).

That's not good business, but how can a solution be converted into useful products that make a profit?

Microsoft Troubles XIII: "Business Insider" articles

A couple of articles discussing Microsoft's future, directly and indirectly.
Others are starting to forecast the potential for a collapse and "Microsoft 2012 == IBM 1990".

Steve Jobs Was Right: Google IS Turning Into Microsoft
(reminding us that Microsoft has tried to get into TV, cable, music and a bunch of other things)

(Larry Pages asked Steve Jobs for advice.)
Jobs told him to focus on fewer things and do them really well.

Apple needs to invent 'The Brick': screenless high-performance CPU, Graphics and Storage

Two new things appeared in my world recently:

• A friend showed me Google 'Jump Desktop', connecting his iPad to his Desktop. [Apple have their own 'Remote Desktop' product which I haven't researched. Not sure if it works over the wider Internet as well.]
• Joe Wilcox's piece (Jan-2011) on Apple dumping desktops sometime (soon) as they become less relevant.

Wilcox wonders what will happen to "Power Users" if Apple move on Desktops, as they've moved on from the rack-mount X-serve line. Customers needing servers now have the choice of a Mac Mini Server or a "Mac Pro" (tower).

Can Apple afford to cut-adrift and ignore the needs of good folk who rely on their product for their business/livelihood, some of whom may have used Macs for 20+ years?? Would seem a Bad Idea to alienate such core and influential users.

Clearly Apple look to the future, and like the floppy drive they expunged long ago in favour of Optical drives (now also obsolete), Desktops as we know them are disappearing from mainstream appeal and usefulness.

I think there are two markets that Apple needs to consider:

• One they haven't won yet: Corporate Desktops, and
• One that's been part of their core business for decades: High-end Graphics/Media

Thunderbolt on laptops means big, even dual, monitors are simple for Corporate Desktops, addressing a large part of the demand/needs. While Apple retain the Mac Mini line, they have a viable PC Desktop replacement for those organisations that like the "modular PC" model, especially those that don't want laptops walking out the door.

The simplicity and elegance of Just One Plug of the iMac makes it unbeatable in certain niche applications, such as public use PC's in Libraries or battery workstations in call centres.

Can Apple produce a "power" laptop with the processing, graphics and storage size/performance that meets the needs of High-end Media folk?
A: No, never. Because the fastest, most-powerful CPU's, GPU's, most RAM and largest/fastest storage only ever come with high-power and big footprint: you need a big box with a big power supply: The definition of a Desktop or Workstation.

One solution would be to licence OS/X to "tier 1" PC vendors like Dell or HP  for use on certified systems. But that's not going to happen, Apple is a hardware/manufacturing company - they will never go there.

Hence "the Brick" that is mainly accessed via "Remote Desktop".
My suggestions are a modular design, not dissimilar to the NGEN's expandable 'slices':

• CPU's and RAM in a housing with capacity to gang together for scale-up.
• GPU's in a PCI-slot chassis, with Thunderbolt available for physical displays.
• Local storage via e-SATA, SAS or Thunderbolt.
• remote bulk storage over the network
• External power-supply, or part of a base-unit (CPU, RAM, PCI-slot, network, Thunderbolt).

The point of "the Brick" is ComputePower-on-Demand and Universal-Workspace-View, not unlike SUN's 1993 "Starfire video" prototype.
It can live in a (locked) cupboard, or many can be hosted on a server cluster as one of many Virtual Machines. For even a modest operation, high-power servers running VMware makes operational and economic sense. VM's mean another licensing deal. Perhaps VMware, part of EMC, might have the clout to do a deal like this with Apple. Or not.

Jim Gray authored a paper in 2004, "TerraServer Bricks" as an alternative architecture. The concept is not new/original and more than the usual low-power appliances.

---

An aside on "Jump Desktop", it uses well established (and secure) remote desktop protocols (RDP, VNC). But for Unix/Linux users interested in security and control, this is important:

Jump also supports SSH tunneling for RDP and VNC connections which also adds a layer of encryption but this must be configured manually.

Surprises reading up on RAID and Disk Storage

Researching the history of Disk Storage and RAID since Patterson et al's 1988 paper has given me some surprises. Strictly personal viewpoint, YMMV.

1. Aerodynamic drag of (disk) platters is ∝ ω³ r⁵  (RPM^3 * radius^5)
   • If you double the RPM of a drive, spindle drive power consumption is cubed. All that power is put into moving the air, which in a closed system, heats it.
     Ergo, 15K drives run hot!
   • If you halve the size of a platter, spindle drive power consumption is reduced by the fifth-power. This is why 2½ inch drives use under 5W (and can be powered by USB bus).
   There's a 2003 paper by Gurumurthi on using this effect to dynamically vary drive RPM and save power. Same author in 2008 suggests disks benefit from 2 or 4 sets of heads/actuators. Either to increase streaming rate or seek time, or reduce RPM and maintain seek times.
   
   The Dynamic RPM paper to be the genesis of the current lines of "Green" drives. Western Digital quote RPM as "intellidrive", but class these as 7,200RPM drives. Access time just got harder to predict.
   
   This is also the reason that 2.5" and 3.5" 15K drives use the same size platters.
   
   
2. In Patterson's 1988 RAID paper. They compare 3 different drives and invented the term "SLED" - Single Large Expensive Disk to describe the IBM mainframe drives of the time.

   Name	Capacity	Drive Size	Mb per Rack Unit	Platter Size	Power	Specific Power
   IBM 3380	7.5Gb	whole cabinet	180Mb/RU	14"	6.6kW	0.9 W/Mb
   Fujitsu Super Eagle	600Mb	6RU, 610mm deep	60Mb/RU	10.5"	600W	1.0 W/Mb
   Conner CP-3100	100Mb	4in x 1.63in, 150-250mm deep	350Mb/RU	3.5"	6-10W	0.1 W/Mb

   And two smaller surprises, all these drives had 30-50,000 MTBF and the two non-SLED drives were both SCSI, capable of 7 devices per bus.
   8 or 9 3.5" drives could be fitted vertically in 3RU, or horizontally, 4 per RU.
   Because of the SCSI bus 7-device limit, and the need for 'check disks' in RAID, a natural organisation would be 7-active+1-spare in 2RU.
3. 2½ inch drives aren't all the same thickness! Standard is ~ 70mmx100mm x 7-15mm
   • 9.5mm thick drives are currently 'standard' for laptops (2 platters)
   • 7mm drives, single platter, are used by many netbooks.
   • 7mm can also be form-factor of SSD's
   • "Enterprise" drives can be 12.5mm (½ inch) or 15mm (more common)
   • Upshot is, drives ain't drives. You probably can't put a high-spec Enterprise drive into your laptop.
4. IBM invented the disk drive (RAMAC) in 1956. 50 platters of 100KB (= 5Mb). Platters loaded singly to read/write station.
   IBM introduced it's last SLED line, the 3390, in 1989. The last version, "Model 9" 34Gb, was introduced in 1993. Last production date not listed by IBM.
   IBM introduced the 9341/9345 Disk Array, a 3390 "compatible", in 1991.
   When Chen, Patterson et al published their follow-up RAID paper in 1994, they'd already spawned a whole industry and caused the demise of the SLED.
   IBM sold its Disk Storage division to Hitachi in 2003 after creating the field and leading it for 4 decades.
5. RAID-6 was initially named "RAID P+Q" in the 1994 Chen, Patterson et al paper.
   The two parity blocks must be calculated differently to support any two drive failures, they aren't simply two copies of "XOR".
   Coming up with alternate parity schemes, the 'Q', is tricky - they can be computationally intensive.
   Meaning RAID-6 is not only the slowest type of RAID because of extra disk accesses (notionally, 3 physical writes per logical-block update), but it also consumes the most CPU resource.
6. IBM didn't invent the Compact-Flash format "microdrive", but did lead its development and adoption. The most curious use was the 4Gb microdrive in the Apple iPod mini.
   In 2000, the largest Compact Flash was the 1Gb microdrive.
   By 2006, Hitachi, after acquiring from IBM in 2003, had increased the capacity to 8Gb, its last evolution.
   According to wikipedia, by 2009, development of 1.3", 1" and 0.85" drives was abandoned by all manufacturers.
7. Leventhal in 2009 pointed out if capacities kept doubling every 2 years, then by 2020 (5 doublings or *32), then RAID would need to adopt triple-parity (and suggests "RAID-7").
   What I found disturbing is that the 1993 RAID-5 and 2009 RAID-6 calculations for the probability of a successful RAID rebuild after a single drive failure is 99.2%.
   
   I find an almost 1% chance of a RAID rebuild failing rather disturbing.
   No wonder Google invented it's own way of providing Data Protection!
8. The UER (Unrecoverable Error Reading) quoted for SSD's is "1 sector in 10^15".
   We know that flash memory is organised as blocks, typically 64kB, so how can they only lose a single sector? Or do they really mean "lose 128 sectors every 10^17 reads"?
9. Disk specs now have"load/unload cycles" quoted (60-600,000).
   Disk platters these days have a plastic unload ramp at the edge of the disk, and the drive will retract the heads there after a period of inactivity.
   Linux servers with domestic SATA drives apparently have a reputation for exceeding load/unload cycles. Cycles are reported by S.M.A.R.T., if you're concerned.
10. Rebuild times of current RAID sets are 5hours to over 24 hours.
    Part of this is due to the large size of "groups", ~50. In 1988, Patterson et al expected 10-20 drives per group.
    As well, the time-to-scan a single drive has risen from ~100 seconds to ~6,000 seconds.
11. One of the related problems with disks is archiving data. Drives have a 3-5 year service life.
    A vendor claims to have a writeable DVD-variant with a "1,000 year life".
    They use a carbon-layer (also called "synthetic stone") instead of a dye layer.
    There is also speculation that flash-memory used as 'write-once' might be a good archival medium. Keep those flash drives!

Update 11-Nov-2011:

Something new I learnt last night:
 The 1.8" disk format is very much alive and well.
 they're used in mobile appliances.
 I wonder if we'll see them "move up" into laptops, desktops or servers?
 Already I've seen a 2.5" SSD which is a 1.8" module in a carrier...

Another factoid:
 For the last 2 years, HP has only shipped servers with 2.5" internal
drives.

Apple lead the desktop world twice in this fashion:
  Mac's skipped 5.25" floppies, only ever 3.5".
  Mac removed floppy drives well before PC's.

Does the 'Air' w/o optical drive count too?
The Mac Classic used SCSI devices, which seemed like a very good idea at the time. But not great for consumer-level devices and they've gone to SATA now.
Apple did invent Firewire (IEEE 1394 a.k.a. "iLink"), which took off in the video market, and I believe still support it on most devices. 

---

Articles

"Triple-Parity RAID and Beyond", ACM Queue
 Adam Leventhal (SUN), December 17, 2009

"Calculating Mean Time To Data Loss (and probability of silent data corruption)"
Jeff Whitehead, Zetta, June 10, 2009

"A Better RAID Strategy for High Capacity Drives in Mainframe Storage"  [PDF],
ORACLE Corporation, Sept 2010.

"Comparison Test: Storage Vendor Drive Rebuild Times and Application Performance Implications"
Dennis Martin,  Feb 18, 2009

"Considerations for RAID-6 Availability and Format/Rebuild Performance on the DS5000" [PDF]
IBM, March 2010.

"Your Useable Capacity May Vary ..."
Chuck Hollis, EMC Corp, August 28, 2008.


"Five ways to control RAID rebuild times" [requires login. Only intro read]
George Crump. July, 2011 ???

 In a recent test we conducted, a RAID 5 array with five 500 GB SATA drives took approximately 24 hours to rebuild. 

 With nine 500 GB drives and almost the exact same data set, it took fewer than eight hours.

"DRPM: Dynamic Speed Control for Power Management in Server Class Disks",  Gurumurthi, Sivasubramaniam,  Kandemir, Franke, 2003, International Symposium on Computer Architecture (ISCA).

"Intra-Disk Parallelism: An Idea Whose Time Has Come", Sankar, Gurumurthi, Mircea R. Stan, ISCA, 2008.

The importance of Design Rules

This started with an aside in "Crypto", Stephen Levy (2000), about Rivest's first attempt at creating an RSA Crypto chip failing because whilst the design worked perfectly on the simulator, it didn't work when fabricated.

[p134] Alderman blames the failure on their overreliance on Carver Mead's publications...

Carver Mead and Lynn Conway at CalTech revolutionised VLSI design and production around 1980, publishing "Introduction to VLSI System Design" and providing access to fabrication lines for students and academics. This has been widely written about:
e.g. in "The Power of Modularity", a short piece on the birth of the microchip from Longview Institute, and a 2007 Computerworld piece on the importance of Mead and Conway's work.

David A. Patterson wrote of a further, related, effect in Scientific American, September 1995, p63, "Microprocessors in 2020"

Every 18 months microprocessors double in speed. Within 25 years, one computer will be as powerful as all those in Silicon Valley today

Most recently, microprocessors have become more powerful, thanks to a change in the design approach.
Following the lead of researchers at universities and laboratories across the U.S., commercial chip designers now take a quantitative approach to computer architecture.
Careful experiments precede hardware development, and engineers use sensible metrics to judge their success.
Computer companies acted in concert to adopt this design strategy during the 1980s, and as a result, the rate of improvement in microprocessor technology has risen from 35 percent a year only a decade ago to its current high of approximately 55 percent a year, or almost 4 percent each month.
Processors are now three times faster than had been predicted in the early 1980s;
it is as if our wish was granted, and we now have machines from the year 2000.

Copyright 1995 Scientific American, Inc.

The important points are:

• These acts, capturing expert knowledge in formal Design Rules, were intentional and deliberate.
• These rules weren't an arbitrary collection just thrown together, they were a three-part approach, 1) the dimensionless scalable design rules, 2) the partitioning of tasks and 3) system integration and testing activities.
• The impact, through a compounding rate effect, has been immense e.g. through Moore's Law doubling time, bringing CPU improvements forward 20 years.
• The Design Rules have become embedded in software design and simulation tools, allowing new silicon devices to be designed much faster, with more complexity and with orders fewer errors and faults.
• It's a very successful model that's been replicated in other areas of I.T.

So I'm wondering why vendors don't push this model in other areas?
Does it not work, not scale or is not considered 'useful' or 'necessary'?

There are some tools that contain embedded expert knowledge, e.g. for server storage configuration. But they are tightly tied to particular vendors and product families.

Update 13-Nov-2011: What makes/defines a Design Rule (DR)?

Design Rules fall in the middle ground between  "Rules-of-Thumb" used in Art/Craft of Practice and  the authoritative, abstract models/equations of Science.

They define the middle ground  of Engineering:
 more formal than R-o-T's but more general and directly applicable than the theories models and equations of pure Science, suitable for creating and costing Engineering designs.

This "The Design Rule for I.T./Computing" approach is modelled after the VLSI technique used for many decades, but is not a slavish derivation of it.

Every well understood field of Engineering has one definitive/authoritative "XXX Engineering Handbook" publication that covers all the sub-fields/specialities, recites all the formal Knowledge, Equations, Models, Relationships and Techniques, provides Case Studies, Tutorials, necessary Tables/Charts and worked examples. Plus basic material of ancillary, related or supporting fields.

The object of these "Engineering Handbooks" is that any capable, competent, certified Engineer in a field can rely on its material to solve problems, projects or designs that come their way. They have a reference they can rely upon for their field.

Quantifying specific costs and materials/constraints comes from vendor/product specifications and contracts or price lists. These numbers are used for the detailed calculations and pricing using the techniques/models/equations given in The Engineering Handbook.

A collection  of "Design Rules for I.T. and Computing" may serve the same need.

What are the requirements of a DR?:

• Explicitly list aspects covered and not covered by the DR:
   eg. Persistent Data Storage vs Permanent Archival Storage
• Constraints and Limits of the DR:
  What's the largest, smallest or complex system applicable.
• Complete: all Engineering factors named and quantified.
• Inputs and Outputs: Power, Heat, Air/Water, ...
• Scalable: How to scale the DR up and down.
• Accounting costs: Whole of Life, CapEx and Opex models.
• Environmental Requirements: 
• Availability and Serviceability:
• Contamination/Pollution: Production, Supply and Operation.
• Waste generation and disposal.
• Consumables, Maintenance, Operation and Administration
• Training, Staffing, User education.
• Deployment, Installation/Cutover, Removal/Replacement.
• Compatibility with systems, components and people.
• Optimisable in multiple dimensions.  Covers all the aspects traded off in Engineering decisions:
  • Cost: per unit, 'specific metric' (eg $$/Gb),
  • Speed/Performance:  how it's defined, measured, reported and compared.
  • 'Space' (Speed and 'Space' in the sense of Algorithmn trade-off)
  • Size, Weight, and other Physical characteristics
  • 'Quality' (of design and execution, not the simplistic "fault/error rate")
  • Product compliance to specification, repeatability of 'performance'. (manufacturing defects, variance, problems, ...)
  • Usability
  • Safety/Security
  • Reliability/Recovery
• other factors will be needed to achieve a model/rule that is:
   {Correct, Consistent, Complete, Canonical (ie min size)}

QPRSUCOST: Software Design has more dimensions than 'Functionality'

Summary: There are multiple Essential Dimensions of Software Design besides "Functionality".

There are three Essential External Dimensions, {Function, Time, Money} and multiple Internal Dimensions.
I'm ot sure where/how "Real-Time" is covered, it isn't just "Performance": the necessary concurrency (not just "parallelism") and asynchronous events/processing require 10-100 times the cognitive capacity to deal with, and problems scale-up extraordinarily (faster than exponential) due to this added complexity. This is why Operating Systems and embedded critical systems (health/medicine, aerospace control, nuclear, Telecomms, Routers/Switches, Storage Devices, ...) are so difficult and expensive.

Not understanding and enumerating these multiple Dimensions whilst seemingly teaching Functionality only is perhaps currently the single biggest single failure of the discipline of Software Engineering.

The Necessary or Essential Dimensions of the further Software phases of Software Construction, Software Deployment and Software Maintenance besides the meta-processes of Software Project Management and I.T. Operations are beyond the scope of this piece.

This non-exhaustive taxonomy implies that there are additional Essential Dimensions, such as Maintainability and Manageability, elsewhere in the Computing/I.T. milieu.

My apologies in advance that this piece is in itself a first pass and not yet definitive.


+++ Need to deal with "Documentation" vs "Literate Programming" vs "Slices & tools"
+++ Dev - Ops. Infrastructure is part of the deliverable. Scripts on PRD/DEV/TST must be same. Software Config Mgt and Migration/Fail-back/Fail-over are different and essential/necessary

---

Software Design:
I'm using Software Design in an unconventional sense:
  everything that precedes and defines Coding and Construction.

While noting that Software Design and Construction are closely intertwined and inter-dependent and that all Software Projects are iterative, especially after notional Deployment and during Software Maintenance.

The acts of coding and testing uncover/reveal failings, errors, assumptions, blind-spots and omissions in the Design and its underlying models and concepts.

Where do the various Testing activities belong?
Wherever your Process or Project Methodology define them to be.
Many Software Design problems are revealed when first attempting to construct tests and later in performing them. Thus creating feedback, corrections and additional requirements/constraints.


What's an "Essential Dimension"?
In Formal Logic and Maths, there's the notion of "necessary and sufficient conditions" for a relationship or dependency to hold.
It is in this sense that I'm defining an "Essential Dimension" of elements or phases in the Software  process, that they individually be Necessary and together be Sufficient for a complete solution/result.

A Dimension is Essential if it's removal, omission or non-performance results in Defective, Incomplete, Ineffective, Non-Performing or Non-Compliant Software and Systems.
Or more positively, a Dimension is Essential if it must be performed to achieve the desired/specified process and product outputs and outcomes.

A marker of an Essential is

Defective, or colloquially "Buggy", Software, has many aspects, not just "Erroneous, Invalid or Inconsistent Results".

The term is meant to be parsed against each of the Essential Design Dimensions for specific meanings, such as "Hacked or Compromised" (Security), "Failure to Proceed or Complete" (i.e. crash or infinite loop: Quality), "Too Slow" (Performance),  "Corrupt or Lose Data" (Quality), "Unmaintainable" (Quality) and "Maxed out" (Scalability).


Initial candidate Essential Dimensions.
From my experience and observations of the full Software cycle and I.T. Operations, a first cut, not in order of importance:

• F - Functionality
  
• Q - Quality
• P - Performance
• R - Reliability/Recovery
• S - Security/Safety
• U - Usability 
• C - Concurrency/Synchronousness
• O - Operability/Manageability 
• S - Scalability
• T - Testability

Relative Importance of the Design Dimensions
Which Dimension is most important?
All and None: it depends on the specific project or task and its goals, constraints and requirements.

An essential outcome of the Specification phase of Software Design is to precisely define:

• The criteria for each  Essential Design Dimensions for the Product, Project, all Tasks and every Component.
• The relative importance of the Dimensions.
• How to assess final compliance to these criteria in both Business and Technical realms.

The one universally applicable Design Dimension is Quality.

Which of its many aspects are critical for any project, sub-system, task, phase or component, and how they will be monitored, controlled and confirmed, must be defined by your meta-processes or derived through the execution of your Methodology.

Minimally, any Professionally produced Software component or product must be shown to conform both to the Zeroth Law requirements (keep running, terminate, Do no Damage  and produce results) and its written Functional Requirements/Specifications.


Quality


Zeroth Law requirements (keep running, terminate, Do no Damage and produce results)

From "The quality of software", Hoare, Software-Practice and Experience Vol 2, 1972 p103-5 

Hoare's Software Quality Criteria:

(1) Clear definition of purpose

(2) Simplicity of use
(3) Ruggedness
(4) Early availability
(5) Reliability

(6) Extensibility and improvability in light of experience

(7) Adaptability and easy extension to different configurations

(8) Suitability to each individual configuration of the range
(9) Brevity
(10) Efficiency (speed)

(11) Operating ease

(12) Adaptability to wide range of applications
(13) Coherence and consistency with other programs
(14) Minimum cost to develop
(15) Conformity to national and international standards

(16) Early and valid sales documentation

(17) Clear accurate and precise user’s documents

Security/Safety

Performance

Usability

Reliability/Recovery

Scalability

Testability

• Functional Testing or Specification Compliance Testing?
• Load Testing
• Regression Testing, post-Release esp.
• Acceptance Testing. Commercial Compliance?
• Others?

Concurrency/Asynchronousity

Operability/Manageability

A new inflection point? Definitive Commodity Server Organisation/Design Rules

Summary:

For the delivery of general purpose and wide-scale Compute/Internet Services there now seems to be a definitive hardware organisation for servers, typified by the E-bay "pod" contract.

For decades there have been well documented "Design Rules" for producing Silicon devices using specific technologies/fabrication techniques. This is an attempt to capture some rules for current server farms. [Update 06-Nov-11: "Design Rules" are important: Patterson in a Sept. 1995 Scientific American article notes that the adoption of a quantitative design approach in the 1980's led to an improvement in microprocessor speedup from 35%pa to 55%pa. After a decade, processors were 3 times faster than forecast.]

Commodity Servers have exactly three possible CPU configurations, based on "scale-up" factors:

• single CPU, with no coupling/coherency between App instances. e.g. pure static web-server.
• dual CPU, with moderate coupling/coherency. e.g. web-servers with dynamic content from local databases. [LAMP-style].
• multi-CPU, with high coupling/coherency. e.g. "Enterprise" databases with complex queries.

If you're not running your Applications and Databases in Virtual Machines, why not?
[Update 06-Nov-11: Because Oracle insists some feature sets must run on raw hardware. Sometimes vendors won't support your (preferred) VM solution.]

VM products are close to free and offer incontestable Admin and Management advantages, like 'teleportation' or live-migration of running instances and local storage.

There is a special non-VM case: cloned physical servers. This is how I'd run a mid-sized or large web-farm.
This requires careful design, a substantial toolset, competent Admins and a resilient Network design. Layer 4-7 switches are mandatory in this environment.

There are 3 system components of interest:

• The base Platform: CPU, RAM, motherboard, interfaces, etc
• Local high-speed persistent storage. i.e. SSD's in a RAID configuration.
• Large-scale common storage. Network attached storage with filesystem, not block-level, access.

Note that complex, expensive SAN's and their associated disk-arrays are no longer economic. Any speed advantage is dissolved by locally attached SSD's, leaving only complexity, resilience/recovery issues and price.
Consequentially, "Fibre Channel over Ethernet" with its inherent contradictions and problems, is unnecessary.

Designing individual service configurations  can be broken down into steps:

• select the appropriate CPU config per service component
• specify the size/performance of local SSD per CPU-type.
• architect the supporting network(s)
• specify common network storage elements and rate of storage consumption/growth.

Capacity Planning and Performance Analysis is mandatory in this world.

As a professional, you're looking to provide "bang-for-buck" for someone else who's writing the cheques. Over-dimensioning is as much a 'sin' as running out of capacity. Nobody ever got fired for spending just enough, hence maximising profits.

Getting it right as often as possible is the central professional engineering problem.
Followed by, limiting the impact of Faults, Failures and Errors - including under-capacity.

The quintessential advantage to professionals in developing standard, reproducible designs is the flexibility to respond to unanticipated load/demands and the speed with which new equipment can be brought on-line, and the converse, retired and removed.

Security architectures and choice of O/S + Cloud management software is outside the scope of this piece.

There are many multi-processing architectures, each best suited to particular workloads.
They are outside the scope of this piece, but locally attached GPU's are about to become standard options.
Most servers will acquire what were known as vector processors and applications using this capacity will start to become common. This trend may need their own Design Rule(s).

Different, though potentially similar design rules apply for small to mid-size Beowulf clusters, depending on their workload and cost constraints.
Large-scale or high-performance compute clusters or storage farms, such as the IBM 120 Petabyte system, need careful design by experienced specialists. With any technology, "pushing the envelope" requires special attention by the best people you have,  to even have a chance of success.

Not unsurprisingly, this organisation looks a lot like the current fad, "Cloud Computing" and the last fad, "Services Oriented Architecture".

---

Google and Amazon dominated their industry segments partly because they figured out the technical side of their business early on. They understood how to design and deploy datacentres suitable for their workload, how to manage Performance and balance Capacity and Cost.

Their "workloads", and hence server designs, are very different:

• Google serves pure web-pages, with almost no coupling/communication between servers.
• Amazon has front-end web-servers is backed by complex database systems.

Dell is now selling a range of "Cloud Servers" purportedly based on the systems they supply to large Internet companies.

---

An App too far? Can Windows-8 gain enough traction.

Summary:
"last to market" worked as a strategy in the past for Microsoft.
But "everything is a PC" is probably false and theyll be sidelined in the new Mobile Devices world.

Why Apple won't add peer-peer to iCloud

The sister post to this speculates that Apple could add peer-peer protocols/functionality to its iCloud service, and the benefits that would flow.

I'm firmly of the opinion that Apple won't go there, not soon, not in-a-while, not ever.

They have too firmly entrenched attitudes about constructing and maintaining "full-control" over the euphemistically named, "User Experience".

Apple don't do "collaboration", sharing their technology or allowing the mere User to tinker with its Gorgeous Stuff. It'd no longer be "their Design" and they anathema to Apple.

Apple are into "control", which in itself is not a bad thing, but severely limits their software and system design decisions and implementations.

This isn't some simple "we know best" thing, but much deeper, intimately tied to their focus on High Concept Design and a finely crafted "User Experience". Which also means controlled experience.

Apple could make huge inroads into the PC market by licensing OS/X - something it could've done anytime in the last 10 years. Now that "classic" computers are under 25% of its business, Apple could let go of its stranglehold on its computer hardware and light the fires of innovation: "let a thousand roses bloom".  But they cannot and won't.

This translates to iCloud in two ways:

• they haven't thought of the idea themselves, and
• they probably couldn't model the response times of torrent-like service and would baulk at any service which is in the least unpredictable, perhaps sometimes not-quite-perfect.

Apple need to control its "User Experience", which means they can't let other players on-board and can't adopt "radical" or "unproven" solutions. (ie. "not invented here").

So they will build and run some very large datacenters to run iCloud.

The trouble with this approach is they are leaving the field of Innovation open to their competitors.
We know Microsoft won't embrace it, but Google and Android will and do.

Even Great Design can be copied and tweaked, sometimes even improved.That the British lost it home-grown motorcycle and motor car industries, know for radical and innovative design, to the Japanese and their "continuous improvement/refinement cycle" demonstrates this thesis.

In 10 years, will the "iPhone 15" be a patch on Android and the gazillion services/applications it runs?
I suspect not. The most amazing and usable devices are unlikely to come from the standalone corporations.

It could be Google and Android, it could be something completely new.

It just won't be Apple at the front, again.
Think how they blew their advantage of the Apple II. They've got form and the same fixed, rigid mindset is still rampant. That's good for Bold New Steps, poor for continuous stepwise refinement.

Apple iCloud and peer-peer (Torrents)

Will Apple add a torrent like ability to its iCloud offering??

iCloud is a remote filesystem with a lot of metadata and does 4 things:

• provides "second copy of my precious data" (for files I've generated)
• allows synchronisation of those files across the multiple devices/platforms a user connects. This is the aspect Apple 'sells': email, contact and calendar sync and restore/recover.
• mediates the enforcement of copyright and content distribution
• does Internet-Scale data de-duplication.
  
  • By data volume, the Internet is a 'Viewing Platform'.
  • 1 upload == zillions downloads [write once, download ~infinite]

Apple could create an Internet-Scale "Content Delivery Network" with iCloud if ran a peer-peer network, something like the hugely successful bit-torrent protocol/service.

Because you've got authorised content and validated entities/logins in a vendor controlled environment, there isn't a direct copyright or leakage problem, just the ever-present and non-removable "analogue hole".
There is scope for scanning never-before-seen files to see if they are recodings, subsets or 'analogue rerecordings' of know files.
What action then? Automatically remove the file, "Bill the User" or send a Summons?

'Backups' of already known files take the time to transfer and compare the checksum/identifier. That's a incredible compression ratio/speed-up. Those checksum/identifiers also are the natural keys for both the 'torrent' and backing-store key.

Storing the per-machine file/directory structure is another layer and doesn't yield to the same de-duplication/compression techniques.
If I were implementing the local filesystem, I'd do two things:

• calculate and store checksums on-the-fly and store in the metadata.
• make sure part of the metadata was as whole-file checksum or UUID-type identifier.

Possibly also calculate and store large-chunk (8-64Mb) checksums.

This enables two services usually only seen at Enterprise scale:

• Document-Management-System like controls, searches, functionality.
• user collaboration: tagging, comments, hilighting, edits/recuts + mashups, annotation, linking, etc.

As bit-torrent shows, using distributed {storage, net-bandwidth, CPU} scales, amplifies 'Server Effectiveness' and gives apparent 100% uptime/availability to services.
It's really cheap and easy for the content provider, though causes more traffic on the ISP network.

BTW, this isn't limited to Desktops and mobile devices.
It scales to IP-TV and on-demand content services.
Your Apple-TV box effectively has infinite & perfect storage...
Internode has announced 'fetchTv' - so these services are on the radar for ISP's.

It also has significant consequences for Ozzie customers who pay per Gb.
You really don't want your iPad on a 1Gb wireless 3G plan acting as a torrent server. A nasty $2000 bill surprise!

There are serious performance issues with local congestion (POP, ISP, backhaul, main-site) inter-network links  and dealing with ADSL bandwidth characteristics.

The NBN is going to be a Layer 2 network, (2-level VLANs or 802.11 "Q in Q").
The presumption is ISP's will offer PPPoE to begin with, as for the current ADSL services.

PPoE is not well suited to distributed data/torrents:

•  your source device puts packets onto your LAN,
•  your firewall/router fields those packets and pushes them to your ADSL modem which encapsulates packets into PPPoE, then puts these new packets onto the wire
• the bytes go down your line to the DSLAM
• are routed down the 'backhaul' link to the ISP's nearest site
• into the 'Access Concentrator' to become a public IP addr
• then routed towards the destination public IP address, which could be on the same Concentrator, the same POP, the same ISP, a shared 'Interconnect', or an upstream provider
• into the destination Access Concentrator
• down the backhaul, DSLAM, ADSL model, firewall/router and eventually appear on the destination LAN to the receving device.

Which gives you so many single-point-of-failure/congestion/saturation that it isn't funny...

If the other person is across the hall or across the road, this incurs a massive and needless overhead, not to mention delays and multiple local resource contention.

The wikipedia PPPoE article discusses problems and current solutions

So, if iCloud becomes a torrent-like service, will it overload the NBN??