SteveJ-on-IT

Smart Internet+Smart-Grid: making money and reducing carbon footprint

2011-12-17T15:29:00.002+11:00

Two technology/commercial trends are coming our way in Australia:

the Internet everywhere (3G wireless or NBN), means "smart controllers" will be cheap, easy and everywhere. They will be able to trivially hook into 3-7 day local forecasts, especially useful for air-conditioning units.
"Smart Power" metering will start to charge power at different prices during the day, rather than the disconnected traditional pricing of "a single price whenever you use it".
You can see real-time wholesale electricity prices on-line and a Pretty graph in 30-min periods.
Yesterday (12/12/11), the 30-min price varied from $52/MWhr @ 4PM to $16/MWhr @ 2AM. In 5-min periods, the price ranged from $95 to $16.

There's a bit of background you may or may not know about Power Generators: they over-build capacity to meet any and all demands placed on them. There aren't just no incentives for Power Generators or their customers to reduce either aggregate or instantaneous demand, but the reverse: significant economic disincentives to reducing demand, and hence to lowered income and profit. This is a perverse economic outcome costing us a lot of money and burning carbon unnecessarily.

From "An EnergySmart Plan. Positioning Queensland for a Diversified Energy Future 2010 - 2050" (Nov 2010 report for Queensland Government):

Ergon and ENERGEX will each spend $6 billion (that is $12B combined) in capital expenditure over the next five years to cope with extraordinary consumption during a fraction of the year, rather than the average consumption over the course of the year.

To put this into perspective, ENERGEX has over $900M in assets that are only used for approximately 3.5 days per year. (Mark Paterson, ENERGEX, The SPRA Standard).

That's not good business, but how can a solution be converted into useful products that make a profit?

1. "Trigeneration" (being implemented in Sydney, and already very big in UK/Europe)
results in some Zero-carbon footprint services.

Burn methane/natural-gas in "gas turbines" (really jet engines).
they are very high efficiency, start quickly and are responsive to load, unlike big coal-fired units which take days to start and cannot easily adjust to extra demand.
extra money is saved by locating at point-of-use.
There are no "network" losses, avoiding very expensive network upgrades because of the separation of generator and load: the current cause of huge price rises in NSW over the last few years.
Around 66% (yes more than half!) the energy released in burning coal/gas is wasted: just thrown away as heat into the local environment. "Tri-gen" captures a bunch of this in hot-water.
The "Tri" in "Tri-gen" is

electricity,
heating and
cooling.

Hot water can be used to drive "chillers" in the same way that old Kero Fridges and modern caravan LPG fridges work.
For cities, avoiding using grid-electricity to drive skyscraper air-conditioning is a massive win. The heating and air-conditioning is "carbon free" - created free from the gas turbine waste heat, with the power running lighting etc.

This is the sort of thing that could be retrofitted to high-end apartments, where the owners collectively have capital to invest, the willingness to "spend money to make money" and possibly a concern for the environment. There is no additional gas or electricity distribution network to be built: they will already have gas connected for hot-water heating.

There must be ways to down-scale this technology to suburban residential, [or small shopping centres who all need refrigeration, air-conditioning and power] In time, Governments might allow such power to be sold back to the grid by individuals. Right now, if current large generators owned and controlled the equipment, they could use it for highly profitable peak-load generation.

These systems could be built under "lease-back" arrangements. The property owners raise the money for the installation, then the asset is leased back to the power generation company under a standard commercial contract. The power generators get to save carbon offsets and can sell those on the open market, over time a useful revenue source.

This approach would be especially useful for regional country towns (Cairns, Mt Isa) that could locally source methane/natural gas. "Methane Digesters", running on agricultural are cheap and well known.

2. Large shopping centres/malls in hot climates save BIG slabs of money (both Capital and Operating costs) by making ice overnight, storing it in tanks, then using that ice to provide cooling, especially in the hottest part of the day (late afternoon). It saves money and energy in many ways:

the ice-plant is maybe 25% capacity of the "chillers" normally needed. Much cheaper to build and service.
the power to drive the ice-plant is bought at the cheapest rates possible, as against the guaranteed most expensive rate when you buy-when-you-use.
the work required to make 1 tonne of ice is the least possible: being done at the coolest time of day, working against the "least resistance" of the air temperature.
The ice making plant always works at optimum efficiency. It is run just long enough each day to make whatever is needed, then stops.
Controlling large electric motors on heavy machinery, like the compressors in "chillers", is difficult and expensive. Mostly the are run like domestic refrigerators, in "stop-start" mode. This causes a lot of extra wear-and-tear on the mechanical parts as well as massive spikes in load: electric motors take a big whack of current to get them going. (more than 10 times operating current). This is why modern domestic split-systems are mostly "inverters" - the compressor can be adjusted to run continuously at a fraction of full-power.
the plant to chill the air is simpler, smaller and cheaper because 2-5,000 times (by volume) less cold water is needed:
A small pump brings very cold water from a tank through a "radiator" (heat-exchanger) that cools the air. To cool more/less just change the speed of the pump, versus starting/stopping the large compressors. Much simpler/easier/cheaper than running massive chillers in "make-when-you-use" mode.
the tanks to store ice are cheap and robust. Easy to build extra storage (if you know you've got a heat-wave coming). When big enough, you don't even have to insulate them.
One cubic meter of ice, around one tonne, stores approx. 3.5kWatt-hours of input energy, around three hours use of a small domestic unit. Not sure of "large plant" sizes, but I think 200-400 tonnes of ice in a day would be "large". Olympic swimming pools are 2,500 cubic meters in comparison
plant breakdown under very high loads are more unlikely. If there isn't enough ice, the A/C air just gets a little warmer.

Can this be applied to suburban residential (houses or town-houses) - individually or in groups?
It certainly can be applied to high-end apartment complexes: saving a bunch of money, being more reliable and "being green".

There's a simple A/C technique used in large buildings that I've never seen in houses.

evaporative cooling for the "chiller". (Those water cooling towers you see).
In normally hot-dry areas (inland) this is a really big win for domestic A/C plant.
In coastal areas (Cairns, Brisbane), it provides some benefit, even on humid days.
BUT, would really only become economic/cheap when serving multiple dwellings.

3. Instead of building large, expensive generators out in the country-side that get used 3-4 days a year, store power in batteries (buy overnight, sell at peak, for approx 3:1 difference).

If such a simple/obvious idea was "economic", then the big generators would already be doing it, aside from the perverse economic incentives to build and sell more power...

such a system, like Mobile Phone networks, only works when run by one big "provider", but ideally you'd have thousands of small-scale units (homes etc) embedded in the network, which by definition is all those householders who've put up Solar Panels.
They already done 80% of the investment and pre-qualified themselves as having an interest in "thinking differently" and shown they have the necessary financial resources.
Problem:
currently domestic solar power installations selling electricity back to the grid are banned from having batteries.
The large generators could lobby for this additional function for individuals, but as long as they own and control in-network power storage, there should be no regulatory problems.
It's a really good investment compared to $900M sitting idle for 360 days/year.
The CSIRO have developed a hybrid "ultra battery", suitable for many applications and already in testing for power storage.
Some questions and options:

How to make money in small scale, when the big players are owning/running it all?

Scouting for sites, signing them up and organising installation. [cf. people who buy rights to "air space" for mobile phone antennas]

Householders already with Solar PV could add power-storage under a "lease-back" deal. But who gets to keep the gear at the end? The householder or lessee (power company)?
Installation/upgrade is:
controller/power-meter, charger, batteries.
Does a reseller/installer need to become an Electricity Provider to do this?
Who is responsible for installation and site maintenance and on-going replacements? (the batteries especially).

The idea of addressing the perverse economic incentives of Power Generators isn't new.
Amory Lovins, author and co-creator of the Rocky Mountain Institute, created the "negawatt" concept in 1989:
increase efficient power use by consumers and claim-back the avoided consumption as power generated.
The "mere technical difficulty" is measuring (and accounting for) product/service not used.

The underlying economic drivers are simple:

There is no nett difference between a watt avoided and a watt generated, there is exactly the same nett economic benefit derived from the consumption of the derived service.
the capital cost of avoiding generating a watt is many times lower than building plant to generate a watt.
the on-going operational costs of an avoided-watt are precisely zero, and it is available "24x7", unlike some popular renewal resources (solar PV, wind, waves).

There can be hidden costs like additional maintenance, but often new technologies, such as compact fluorescent bulbs, reduce maintenance/replacement costs by extending component useful life by 2-10 times.

Microsoft Troubles XIII: "Business Insider" articles

2011-12-03T11:26:00.001+11:00

A couple of articles discussing Microsoft's future, directly and indirectly.
Others are starting to forecast the potential for a collapse and "Microsoft 2012 == IBM 1990".

Steve Jobs Was Right: Google IS Turning Into Microsoft
(reminding us that Microsoft has tried to get into TV, cable, music and a bunch of other things)

(Larry Pages asked Steve Jobs for advice.)
Jobs told him to focus on fewer things and do them really well.

Jobs later recounted the conversation to his biographer Walter Isaacson.
"Figure out what Google wants to be when it grows up. It's now all over the map. What are the five products you want to focus on? Get rid of the rest because they're dragging you down. They're turning you into Microsoft"

Search is like Windows: an 800-pound gorilla
Display advertising is like Office: piggybacking on success
Android is like Xbox: a surprise success in a brand new business area (but not a big money maker)
Google+ is like Bing: a reaction to a fearsome new competitor
Google Music is like Zune: a me-too attempt to compete with Apple
Google Apps is like SQL Server: a cheap alternative that nobody pays much attention to...yet
Chrome is like Internet Explorer: a giveaway that pushes the company's agenda on the Web
Google TV is like WebTV: an attempt to make TV more like the Internet
The Kansas City Fiber experiment is like Microsoft's huge cable investments in the 1990s
Google is buying companies like crazy, just like Microsoft used to do

So maybe Steve Jobs was right. So what?
Jobs meant the Microsoft crack as an insult, but maybe it's not so bad.

But it's still immensely profitable, garners $70 billion a year in sales, and is growing around 10% per year.
Nobody can stay on top forever, but if Google wants to have a long ride, it could do worse than being like Microsoft.

STEVE BALLMER'S NIGHTMARE: How Microsoft's Business Actually Could Collapse

The iPad eats the consumer PC market.
Employees gradually switch away from using Windows PCs for work.
Windows 8 fails to stop the iPad.
Loyal developers start to leave the Microsoft platform.
Windows Phone gets no traction despite the Nokia deal and RIM's collapse.
Office loses relevance.
Microsoft's other business applications start to erode.
The platform business collapses.
The Xbox was never going to make up the slack, and Microsoft can no longer afford to keep investing in it.
Microsoft suffers a huge quarterly loss. Ballmer retires to play golf.

Apple needs to invent 'The Brick': screenless high-performance CPU, Graphics and Storage

2011-11-30T11:40:00.001+11:00

Two new things appeared in my world recently:

A friend showed me Google 'Jump Desktop', connecting his iPad to his Desktop. [Apple have their own 'Remote Desktop' product which I haven't researched. Not sure if it works over the wider Internet as well.]
Joe Wilcox's piece (Jan-2011) on Apple dumping desktops sometime (soon) as they become less relevant.

Wilcox wonders what will happen to "Power Users" if Apple move on Desktops, as they've moved on from the rack-mount X-serve line. Customers needing servers now have the choice of a Mac Mini Server or a "Mac Pro" (tower).

Can Apple afford to cut-adrift and ignore the needs of good folk who rely on their product for their business/livelihood, some of whom may have used Macs for 20+ years?? Would seem a Bad Idea to alienate such core and influential users.

Clearly Apple look to the future, and like the floppy drive they expunged long ago in favour of Optical drives (now also obsolete), Desktops as we know them are disappearing from mainstream appeal and usefulness.

I think there are two markets that Apple needs to consider:

One they haven't won yet: Corporate Desktops, and
One that's been part of their core business for decades: High-end Graphics/Media

Thunderbolt on laptops means big, even dual, monitors are simple for Corporate Desktops, addressing a large part of the demand/needs. While Apple retain the Mac Mini line, they have a viable PC Desktop replacement for those organisations that like the "modular PC" model, especially those that don't want laptops walking out the door.

The simplicity and elegance of Just One Plug of the iMac makes it unbeatable in certain niche applications, such as public use PC's in Libraries or battery workstations in call centres.

Can Apple produce a "power" laptop with the processing, graphics and storage size/performance that meets the needs of High-end Media folk?
A: No, never. Because the fastest, most-powerful CPU's, GPU's, most RAM and largest/fastest storage only ever come with high-power and big footprint: you need a big box with a big power supply: The definition of a Desktop or Workstation.

One solution would be to licence OS/X to "tier 1" PC vendors like Dell or HP for use on certified systems. But that's not going to happen, Apple is a hardware/manufacturing company - they will never go there.

Hence "the Brick" that is mainly accessed via "Remote Desktop".
My suggestions are a modular design, not dissimilar to the NGEN's expandable 'slices':

CPU's and RAM in a housing with capacity to gang together for scale-up.
GPU's in a PCI-slot chassis, with Thunderbolt available for physical displays.
Local storage via e-SATA, SAS or Thunderbolt.
remote bulk storage over the network
External power-supply, or part of a base-unit (CPU, RAM, PCI-slot, network, Thunderbolt).

The point of "the Brick" is ComputePower-on-Demand and Universal-Workspace-View, not unlike SUN's 1993 "Starfire video" prototype.
It can live in a (locked) cupboard, or many can be hosted on a server cluster as one of many Virtual Machines. For even a modest operation, high-power servers running VMware makes operational and economic sense. VM's mean another licensing deal. Perhaps VMware, part of EMC, might have the clout to do a deal like this with Apple. Or not.

Jim Gray authored a paper in 2004, "TerraServer Bricks" as an alternative architecture. The concept is not new/original and more than the usual low-power appliances.

An aside on "Jump Desktop", it uses well established (and secure) remote desktop protocols (RDP, VNC). But for Unix/Linux users interested in security and control, this is important:

Jump also supports SSH tunneling for RDP and VNC connections which also adds a layer of encryption but this must be configured manually.

Surprises reading up on RAID and Disk Storage

2011-11-10T11:45:00.000+11:00

Researching the history of Disk Storage and RAID since Patterson et al's 1988 paper has given me some surprises. Strictly personal viewpoint, YMMV.

Aerodynamic drag of (disk) platters is ∝ ω³ r⁵ (RPM^3 * radius^5)
- If you double the RPM of a drive, spindle drive power consumption is cubed. All that power is put into moving the air, which in a closed system, heats it.
  Ergo, 15K drives run hot!
- If you halve the size of a platter, spindle drive power consumption is reduced by the fifth-power. This is why 2½ inch drives use under 5W (and can be powered by USB bus).
There's a 2003 paper by Gurumurthi on using this effect to dynamically vary drive RPM and save power. Same author in 2008 suggests disks benefit from 2 or 4 sets of heads/actuators. Either to increase streaming rate or seek time, or reduce RPM and maintain seek times.

The Dynamic RPM paper to be the genesis of the current lines of "Green" drives. Western Digital quote RPM as "intellidrive", but class these as 7,200RPM drives. Access time just got harder to predict.

This is also the reason that 2.5" and 3.5" 15K drives use the same size platters.

In Patterson's 1988 RAID paper. They compare 3 different drives and invented the term "SLED" - Single Large Expensive Disk to describe the IBM mainframe drives of the time.

Name	Capacity	Drive Size	Mb per Rack Unit	Platter Size	Power	Specific Power

IBM 3380	7.5Gb	whole cabinet	180Mb/RU	14"	6.6kW	0.9 W/Mb

Fujitsu Super Eagle	600Mb	6RU, 610mm deep	60Mb/RU	10.5"	600W	1.0 W/Mb

Conner CP-3100	100Mb	4in x 1.63in, 150-250mm deep	350Mb/RU	3.5"	6-10W	0.1 W/Mb

And two smaller surprises, all these drives had 30-50,000 MTBF and the two non-SLED drives were both SCSI, capable of 7 devices per bus.
8 or 9 3.5" drives could be fitted vertically in 3RU, or horizontally, 4 per RU.
Because of the SCSI bus 7-device limit, and the need for 'check disks' in RAID, a natural organisation would be 7-active+1-spare in 2RU.

2½ inch drives aren't all the same thickness! Standard is ~ 70mmx100mm x 7-15mm
- 9.5mm thick drives are currently 'standard' for laptops (2 platters)
- 7mm drives, single platter, are used by many netbooks.
- 7mm can also be form-factor of SSD's
- "Enterprise" drives can be 12.5mm (½ inch) or 15mm (more common)
- Upshot is, drives ain't drives. You probably can't put a high-spec Enterprise drive into your laptop.
IBM invented the disk drive (RAMAC) in 1956. 50 platters of 100KB (= 5Mb). Platters loaded singly to read/write station.
IBM introduced it's last SLED line, the 3390, in 1989. The last version, "Model 9" 34Gb, was introduced in 1993. Last production date not listed by IBM.
IBM introduced the 9341/9345 Disk Array, a 3390 "compatible", in 1991.
When Chen, Patterson et al published their follow-up RAID paper in 1994, they'd already spawned a whole industry and caused the demise of the SLED.
IBM sold its Disk Storage division to Hitachi in 2003 after creating the field and leading it for 4 decades.
RAID-6 was initially named "RAID P+Q" in the 1994 Chen, Patterson et al paper.
The two parity blocks must be calculated differently to support any two drive failures, they aren't simply two copies of "XOR".
Coming up with alternate parity schemes, the 'Q', is tricky - they can be computationally intensive.
Meaning RAID-6 is not only the slowest type of RAID because of extra disk accesses (notionally, 3 physical writes per logical-block update), but it also consumes the most CPU resource.
IBM didn't invent the Compact-Flash format "microdrive", but did lead its development and adoption. The most curious use was the 4Gb microdrive in the Apple iPod mini.
In 2000, the largest Compact Flash was the 1Gb microdrive.
By 2006, Hitachi, after acquiring from IBM in 2003, had increased the capacity to 8Gb, its last evolution.
According to wikipedia, by 2009, development of 1.3", 1" and 0.85" drives was abandoned by all manufacturers.
Leventhal in 2009 pointed out if capacities kept doubling every 2 years, then by 2020 (5 doublings or *32), then RAID would need to adopt triple-parity (and suggests "RAID-7").
What I found disturbing is that the 1993 RAID-5 and 2009 RAID-6 calculations for the probability of a successful RAID rebuild after a single drive failure is 99.2%.

I find an almost 1% chance of a RAID rebuild failing rather disturbing.
No wonder Google invented it's own way of providing Data Protection!
The UER (Unrecoverable Error Reading) quoted for SSD's is "1 sector in 10^15".
We know that flash memory is organised as blocks, typically 64kB, so how can they only lose a single sector? Or do they really mean "lose 128 sectors every 10^17 reads"?
Disk specs now have"load/unload cycles" quoted (60-600,000).
Disk platters these days have a plastic unload ramp at the edge of the disk, and the drive will retract the heads there after a period of inactivity.
Linux servers with domestic SATA drives apparently have a reputation for exceeding load/unload cycles. Cycles are reported by S.M.A.R.T., if you're concerned.
Rebuild times of current RAID sets are 5hours to over 24 hours.
Part of this is due to the large size of "groups", ~50. In 1988, Patterson et al expected 10-20 drives per group.
As well, the time-to-scan a single drive has risen from ~100 seconds to ~6,000 seconds.
One of the related problems with disks is archiving data. Drives have a 3-5 year service life.
A vendor claims to have a writeable DVD-variant with a "1,000 year life".
They use a carbon-layer (also called "synthetic stone") instead of a dye layer.
There is also speculation that flash-memory used as 'write-once' might be a good archival medium. Keep those flash drives!

Update 11-Nov-2011:

Something new I learnt last night:
 The 1.8" disk format is very much alive and well.
 they're used in mobile appliances.
 I wonder if we'll see them "move up" into laptops, desktops or servers?
 Already I've seen a 2.5" SSD which is a 1.8" module in a carrier...

Another factoid:
 For the last 2 years, HP has only shipped servers with 2.5" internal
drives.

Apple lead the desktop world twice in this fashion:
  Mac's skipped 5.25" floppies, only ever 3.5".
  Mac removed floppy drives well before PC's.

Does the 'Air' w/o optical drive count too?
The Mac Classic used SCSI devices, which seemed like a very good idea at the time. But not great for consumer-level devices and they've gone to SATA now.
Apple did invent Firewire (IEEE 1394 a.k.a. "iLink"), which took off in the video market, and I believe still support it on most devices.

Articles

"Triple-Parity RAID and Beyond", ACM Queue
Adam Leventhal (SUN), December 17, 2009

"Calculating Mean Time To Data Loss (and probability of silent data corruption)"
Jeff Whitehead, Zetta, June 10, 2009

"A Better RAID Strategy for High Capacity Drives in Mainframe Storage" [PDF],
ORACLE Corporation, Sept 2010.

"Comparison Test: Storage Vendor Drive Rebuild Times and Application Performance Implications"
Dennis Martin, Feb 18, 2009

"Considerations for RAID-6 Availability and Format/Rebuild Performance on the DS5000" [PDF]
IBM, March 2010.

"Your Useable Capacity May Vary ..."
Chuck Hollis, EMC Corp, August 28, 2008.

"Five ways to control RAID rebuild times" [requires login. Only intro read]
George Crump. July, 2011 ???

In a recent test we conducted, a RAID 5 array with five 500 GB SATA drives took approximately 24 hours to rebuild.

With nine 500 GB drives and almost the exact same data set, it took fewer than eight hours.

"DRPM: Dynamic Speed Control for Power Management in Server Class Disks", Gurumurthi, Sivasubramaniam, Kandemir, Franke, 2003, International Symposium on Computer Architecture (ISCA).

"Intra-Disk Parallelism: An Idea Whose Time Has Come", Sankar, Gurumurthi, Mircea R. Stan, ISCA, 2008.

The importance of Design Rules

2011-11-06T15:03:00.000+11:00

This started with an aside in "Crypto", Stephen Levy (2000), about Rivest's first attempt at creating an RSA Crypto chip failing because whilst the design worked perfectly on the simulator, it didn't work when fabricated.

[p134] Alderman blames the failure on their overreliance on Carver Mead's publications...

Carver Mead and Lynn Conway at CalTech revolutionised VLSI design and production around 1980, publishing "Introduction to VLSI System Design" and providing access to fabrication lines for students and academics. This has been widely written about:
e.g. in "The Power of Modularity", a short piece on the birth of the microchip from Longview Institute, and a 2007 Computerworld piece on the importance of Mead and Conway's work.

David A. Patterson wrote of a further, related, effect in Scientific American, September 1995, p63, "Microprocessors in 2020"

Every 18 months microprocessors double in speed. Within 25 years, one computer will be as powerful as all those in Silicon Valley today

Most recently, microprocessors have become more powerful, thanks to a change in the design approach.
Following the lead of researchers at universities and laboratories across the U.S., commercial chip designers now take a quantitative approach to computer architecture.
Careful experiments precede hardware development, and engineers use sensible metrics to judge their success.
Computer companies acted in concert to adopt this design strategy during the 1980s, and as a result, the rate of improvement in microprocessor technology has risen from 35 percent a year only a decade ago to its current high of approximately 55 percent a year, or almost 4 percent each month.
Processors are now three times faster than had been predicted in the early 1980s;
it is as if our wish was granted, and we now have machines from the year 2000.

Copyright 1995 Scientific American, Inc.

The important points are:

These acts, capturing expert knowledge in formal Design Rules, were intentional and deliberate.
These rules weren't an arbitrary collection just thrown together, they were a three-part approach, 1) the dimensionless scalable design rules, 2) the partitioning of tasks and 3) system integration and testing activities.
The impact, through a compounding rate effect, has been immense e.g. through Moore's Law doubling time, bringing CPU improvements forward 20 years.
The Design Rules have become embedded in software design and simulation tools, allowing new silicon devices to be designed much faster, with more complexity and with orders fewer errors and faults.
It's a very successful model that's been replicated in other areas of I.T.

So I'm wondering why vendors don't push this model in other areas?
Does it not work, not scale or is not considered 'useful' or 'necessary'?

There are some tools that contain embedded expert knowledge, e.g. for server storage configuration. But they are tightly tied to particular vendors and product families.

Update 13-Nov-2011: What makes/defines a Design Rule (DR)?

Design Rules fall in the middle ground between "Rules-of-Thumb" used in Art/Craft of Practice and the authoritative, abstract models/equations of Science.

They define the middle ground of Engineering:
more formal than R-o-T's but more general and directly applicable than the theories models and equations of pure Science, suitable for creating and costing Engineering designs.

This "The Design Rule for I.T./Computing" approach is modelled after the VLSI technique used for many decades, but is not a slavish derivation of it.

Every well understood field of Engineering has one definitive/authoritative "XXX Engineering Handbook" publication that covers all the sub-fields/specialities, recites all the formal Knowledge, Equations, Models, Relationships and Techniques, provides Case Studies, Tutorials, necessary Tables/Charts and worked examples. Plus basic material of ancillary, related or supporting fields.

The object of these "Engineering Handbooks" is that any capable, competent, certified Engineer in a field can rely on its material to solve problems, projects or designs that come their way. They have a reference they can rely upon for their field.

Quantifying specific costs and materials/constraints comes from vendor/product specifications and contracts or price lists. These numbers are used for the detailed calculations and pricing using the techniques/models/equations given in The Engineering Handbook.

A collection of "Design Rules for I.T. and Computing" may serve the same need.

What are the requirements of a DR?:

Explicitly list aspects covered and not covered by the DR:
eg. Persistent Data Storage vs Permanent Archival Storage
Constraints and Limits of the DR:
What's the largest, smallest or complex system applicable.
Complete: all Engineering factors named and quantified.
Inputs and Outputs: Power, Heat, Air/Water, ...
Scalable: How to scale the DR up and down.
Accounting costs: Whole of Life, CapEx and Opex models.
Environmental Requirements:
Availability and Serviceability:
Contamination/Pollution: Production, Supply and Operation.
Waste generation and disposal.
Consumables, Maintenance, Operation and Administration
Training, Staffing, User education.
Deployment, Installation/Cutover, Removal/Replacement.
Compatibility with systems, components and people.
Optimisable in multiple dimensions. Covers all the aspects traded off in Engineering decisions:
- Cost: per unit, 'specific metric' (eg $$/Gb),
- Speed/Performance: how it's defined, measured, reported and compared.
- 'Space' (Speed and 'Space' in the sense of Algorithmn trade-off)
- Size, Weight, and other Physical characteristics
- 'Quality' (of design and execution, not the simplistic "fault/error rate")
- Product compliance to specification, repeatability of 'performance'. (manufacturing defects, variance, problems, ...)
- Usability
- Safety/Security
- Reliability/Recovery
other factors will be needed to achieve a model/rule that is:
{Correct, Consistent, Complete,Canonical (ie min size)}

QUPSRSTCO: Software Design has more dimensions than 'Functionality'

2011-09-16T10:27:00.001+10:00

Summary: There are multiple Essential Dimensions of Software Design besides "Functionality".

There are three Essential External Dimensions, {Function, Time, Money} and multiple Internal Dimensions.
I'm ot sure where/how "Real-Time" is covered, it isn't just "Performance": the necessary concurrency (not just "parallelism") and asynchronous events/processing require 10-100 times the cognitive capacity to deal with, and problems scale-up extraordinarily (faster than exponential) due to this added complexity. This is why Operating Systems and embedded critical systems (health/medicine, aerospace control, nuclear, Telecomms, Routers/Switches, Storage Devices, ...) are so difficult and expensive.

Not understanding and enumerating these multiple Dimensions whilst seemingly teaching Functionality only is perhaps currently the single biggest single failure of the discipline of Software Engineering.

The Necessary or Essential Dimensions of the further Software phases of Software Construction, Software Deployment and Software Maintenance besides the meta-processes of Software Project Management and I.T. Operations are beyond the scope of this piece.

This non-exhaustive taxonomy implies that there are additional Essential Dimensions, such as Maintainability and Manageability, elsewhere in the Computing/I.T. milieu.

My apologies in advance that this piece is in itself a first pass and not yet definitive.

+++ Need to deal with "Documentation" vs "Literate Programming" vs "Slices & tools"
+++ Dev - Ops. Infrastructure is part of the deliverable. Scripts on PRD/DEV/TST must be same. Software Config Mgt and Migration/Fail-back/Fail-over are different and essential/necessary

Software Design:
I'm using Software Design in an unconventional sense:
everything that precedes and defines Coding and Construction.

While noting that Software Design and Construction are closely intertwined and inter-dependent and that all Software Projects are iterative, especially after notional Deployment and during Software Maintenance.

The acts of coding and testing uncover/reveal failings, errors, assumptions, blind-spots and omissions in the Design and its underlying models and concepts.

Where do the various Testing activities belong?
Wherever your Process or Project Methodology define them to be.
Many Software Design problems are revealed when first attempting to construct tests and later in performing them. Thus creating feedback, corrections and additional requirements/constraints.

What's an "Essential Dimension"?
In Formal Logic and Maths, there's the notion of "necessary and sufficient conditions" for a relationship or dependency to hold.
It is in this sense that I'm defining an "Essential Dimension" of elements or phases in the Software process, that they individually be Necessary and together be Sufficient for a complete solution/result.

A Dimension is Essential if it's removal, omission or non-performance results in Defective, Incomplete, Ineffective, Non-Performing or Non-Compliant Software and Systems.
Or more positively, a Dimension is Essential if it must be performed to achieve the desired/specified process and product outputs and outcomes.

A marker of an Essential is

Defective, or colloquially "Buggy", Software, has many aspects, not just "Erroneous, Invalid or Inconsistent Results".

The term is meant to be parsed against each of the Essential Design Dimensions for specific meanings, such as "Hacked or Compromised" (Security), "Failure to Proceed or Complete" (i.e. crash or infinite loop: Quality), "Too Slow" (Performance), "Corrupt or Lose Data" (Quality), "Unmaintainable" (Quality) and "Maxed out" (Scalability).

Initial candidate Essential Dimensions.
From my experience and observations of the full Software cycle and I.T. Operations, a first cut, not in order of importance:

F - Functionality
Q - Quality
U - Usability
P - Performance
S - Security/Safety
R - Reliability/Recovery
S - Scalability
T - Testability
C - Concurrency/Asynchronousity
O - Operability/Manageability

Relative Importance of the Design Dimensions
Which Dimension is most important?
All and None: it depends on the specific project or task and its goals, constraints and requirements.

An essential outcome of the Specification phase of Software Design is to precisely define:

The criteria for each Essential Design Dimensions for the Product, Project, all Tasks and every Component.
The relative importance of the Dimensions.
How to assess final compliance to these criteria in both Business and Technical realms.

The one universally applicable Design Dimension is Quality.

Which of its many aspects are critical for any project, sub-system, task, phase or component, and how they will be monitored, controlled and confirmed, must be defined by your meta-processes or derived through the execution of your Methodology.

Minimally, any Professionally produced Software component or product must be shown to conform both to the Zeroth Law requirements (keep running, terminate, Do no Damage and produce results) and its written Functional Requirements/Specifications.

Quality

Zeroth Law requirements (keep running, terminate, Do no Damage and produce results)

From "The quality of software", Hoare, Software-Practice and Experience Vol 2, 1972 p103-5

Hoare's Software Quality Criteria:

(1) Clear definition of purpose

(2) Simplicity of use
(3) Ruggedness
(4) Early availability
(5) Reliability

(6) Extensibility and improvability in light of experience

(7) Adaptability and easy extension to different configurations

(8) Suitability to each individual configuration of the range
(9) Brevity
(10) Efficiency (speed)

(11) Operating ease

(12) Adaptability to wide range of applications
(13) Coherence and consistency with other programs
(14) Minimum cost to develop
(15) Conformity to national and international standards

(16) Early and valid sales documentation

(17) Clear accurate and precise user’s documents

Security/Safety

Performance

Usability

Reliability/Recovery

Scalability

Testability

Functional Testing or Specification Compliance Testing?
Load Testing
Regression Testing, post-Release esp.
Acceptance Testing. Commercial Compliance?
Others?

Concurrency/Asynchronousity

Operability/Manageability

A new inflection point? Definitive Commodity Server Organisation/Design Rules

2011-09-14T15:00:00.001+10:00

Summary:

For the delivery of general purpose and wide-scale Compute/Internet Services there now seems to be a definitive hardware organisation for servers, typified by the E-bay "pod" contract.

For decades there have been well documented "Design Rules" for producing Silicon devices using specific technologies/fabrication techniques. This is an attempt to capture some rules for current server farms. [Update 06-Nov-11: "Design Rules" are important: Patterson in a Sept. 1995 Scientific American article notes that the adoption of a quantitative design approach in the 1980's led to an improvement in microprocessor speedup from 35%pa to 55%pa. After a decade, processors were 3 times faster than forecast.]

Commodity Servers have exactly three possible CPU configurations, based on "scale-up" factors:

single CPU, with no coupling/coherency between App instances. e.g. pure static web-server.
dual CPU, with moderate coupling/coherency. e.g. web-servers with dynamic content from local databases. [LAMP-style].
multi-CPU, with high coupling/coherency. e.g. "Enterprise" databases with complex queries.

If you're not running your Applications and Databases in Virtual Machines, why not?
[Update 06-Nov-11: Because Oracle insists some feature sets must run on raw hardware. Sometimes vendors won't support your (preferred) VM solution.]

VM products are close to free and offer incontestable Admin and Management advantages, like 'teleportation' or live-migration of running instances and local storage.

There is a special non-VM case: cloned physical servers. This is how I'd run a mid-sized or large web-farm.
This requires careful design, a substantial toolset, competent Admins and a resilient Network design. Layer 4-7 switches are mandatory in this environment.

There are 3 system components of interest:

The base Platform: CPU, RAM, motherboard, interfaces, etc
Local high-speed persistent storage. i.e. SSD's in a RAID configuration.
Large-scale common storage. Network attached storage with filesystem, not block-level, access.

Note that complex, expensive SAN's and their associated disk-arrays are no longer economic. Any speed advantage is dissolved by locally attached SSD's, leaving only complexity, resilience/recovery issues and price.
Consequentially, "Fibre Channel over Ethernet" with its inherent contradictions and problems, is unnecessary.

Designing individual service configurations can be broken down into steps:

select the appropriate CPU config per service component
specify the size/performance of local SSD per CPU-type.
architect the supporting network(s)
specify common network storage elements and rate of storage consumption/growth.

Capacity Planning and Performance Analysis is mandatory in this world.

As a professional, you're looking to provide "bang-for-buck" for someone else who's writing the cheques. Over-dimensioning is as much a 'sin' as running out of capacity. Nobody ever got fired for spending just enough, hence maximising profits.

Getting it right as often as possible is the central professional engineering problem.
Followed by, limiting the impact of Faults, Failures and Errors - including under-capacity.

The quintessential advantage to professionals in developing standard, reproducible designs is the flexibility to respond to unanticipated load/demands and the speed with which new equipment can be brought on-line, and the converse, retired and removed.

Security architectures and choice of O/S + Cloud management software is outside the scope of this piece.

There are many multi-processing architectures, each best suited to particular workloads.
They are outside the scope of this piece, but locally attached GPU's are about to become standard options.
Most servers will acquire what were known as vector processors and applications using this capacity will start to become common. This trend may need their own Design Rule(s).

Different, though potentially similar design rules apply for small to mid-size Beowulf clusters, depending on their workload and cost constraints.
Large-scale or high-performance compute clusters or storage farms, such as the IBM 120 Petabyte system, need careful design by experienced specialists. With any technology, "pushing the envelope" requires special attention by the best people you have, to even have a chance of success.

Not unsurprisingly, this organisation looks a lot like the current fad, "Cloud Computing" and the last fad, "Services Oriented Architecture".

Google and Amazon dominated their industry segments partly because they figured out the technical side of their business early on. They understood how to design and deploy datacentres suitable for their workload, how to manage Performance and balance Capacity and Cost.

Their "workloads", and hence server designs, are very different:

Google serves pure web-pages, with almost no coupling/communication between servers.
Amazon has front-end web-servers is backed by complex database systems.

Dell is now selling a range of "Cloud Servers" purportedly based on the systems they supply to large Internet companies.

An App too far? Can Windows-8 gain enough traction.

2011-09-14T13:16:00.000+10:00

Summary:
"last to market" worked as a strategy in the past for Microsoft.
But "everything is a PC" is probably false and theyll be sidelined in the new Mobile Devices world.

A response from a friend, this page by John Gruber on Daring Fireball, is worth reading.

Gruber points out that Apple and Microsoft have radically different marketing strategies.
Apple shows The Real Deal when it's ready, Microsoft pre-announces a long way out.

Neither is "better" in itself, they're just different.

I'm wondering if Microsoft's "Windows 8" is too little, too late.
Yep, they've finally got an Apps Store, Cloud Storage and are trying to
engage App Developers.

With Smartphones, people buy Apps, not O/S features.
This works against Microsoft.

Win-7 didn't take off, why should this latest attempt?
Remember that launch parade by MS with a coffin, 'cause they were going
to bury the iPhone... How did that go for them?

If I was a rusted-on Microsoft ISV, then I'd appreciate the API I already knew and get stuck in and deliver 'new product'.

But will these iconoclasts produce really interesting or useful Apps?
At a price the market will bear?
Can they adjust their Big-Footprint mindset into the nimble, lightweight World of Mobile?

You'd think it'd be a stretch.

For established App providers, supporting 2 environments (iOS, Android) is a necessary evil. But a third???
They won't pick it up in droves until they *need* to, and customers won't buy the platform until it has all the Good Apps.

It's the chicken-and-egg problem that ironically has allowed Microsoft to maintain dominance of the Desktop market for a couple of decades.

Companies like Woolworths have a dual-platform App to drive grocery sales, the App is a loss-leader. They'll be early adopters, because they must. The App is given away and all development costs are absorbed.
That business model can't work for software-only enterprises.

Nokia has committed to Windows 8, but who else?
Samsung "fondleslab" were given away at the 'BUILD' conference, so there are obviously some.

Microsoft is initially targeting 'tablets', but is that enough of a market for them?

The Win-8 memory footprint is half the size of "Windows 7", so kudos to them.
But is it enough?
Saying that you've improved fuel-economy from 3 miles-per-gallon to 6 mpg is irrelevant when all your competitors are giving 30mpg.

Mobile Platforms are about usability - which critically depends on battery life.
Platforms that spent special attention to power-management did well (eg. PSION 3, the prototypical 'smart device').

It's more than just memory footprint, it's code complexity and CPU efficiency - areas where Microsoft has traditionally been lacking or profligate.

Talking CPU's, solidly supporting ARM is potential threat to their long-standing partnership with Intel.
Will that lead anywhere? [Probably not]

Counter-intuitively, Microsoft's exceptionally long release cycle (3 years) isn't a "drop-dead" for mobile devices because upgrades are overwhelmingly by replacement, not in-place reimaging.

As iOS and Android release new features and better usability every 6-12 months, vendors tied to Microsoft will be very displeased and the majority of users will be unimpressed at the "old and boring" interface.

Microsoft can only be a follower, and a distant one, in this market, unless it radically changes its development process/pipeline.
Something it has either been unable or unwilling to do for the last 20years...
Which doesn't bode well for them in this niche market.

Will Microsoft follow its X-Box strategy and attempt to buy Market Share with direct hardware subsidies?
Not sure how you make a profit that way...
Which might (finally) lead to a shareholder revolt.

Or will Microsoft come up with another surprise?

Microsoft became the behemoth it is through exceptional marketing and a strategy, intentional or not, of "last to market" and it certainly is last to bring out a competitive mobile platform. [and then, it's avoiding phones, only selling into the smaller tablet market.]

This has worked well for them so far, but are mobile devices a "disruptive change" or is Microsoft really right in "Everything is a PC"?
They are taking a "Bet The Company" gamble on this one.

With the global PC market going off-the-boil, the next step is the Japanese route and into a serious decline, suggesting that Microsoft has to make the transition to the mobile platform or face dwindling revenues - soemthing it's never had to face before.

Windows has a strangehold on the Enterprise Desktop and, IMO, deservedly so:

Active Directory created a solid, even definitive, framework for the management and admin of Enterprise PC's.

Microsoft and its partners have leveraged this framework to good effect.
This is a business with a bright, if not exciting, future.
Personally, if I could invest in just this line of business, I would be sorely tempted.

What's next for Microsoft?
We'll know within 3 years if this play works or not.

Links:

The Register report on the BUILD conference.

This quote from the AllthingsD D9 conference, June 2011, is prescient:

When Sinofsky took the stage at D9 yesterday, Walt Mossberg began by
asking how he felt being left out of the "Gang of Four" which are
running the Internet (Google, Apple, Amazon, and Facebook).

Sinofsky replied, "You know, I'm watching it and feeling like the guy
who's in the race and not winning it." In addition, nothing that starts
as a "Gang of Four" ends well, either."

Another piece on the announcement.

Why Apple won't add peer-peer to iCloud

2011-06-12T12:07:00.001+10:00

The sister post to this speculates that Apple could add peer-peer protocols/functionality to its iCloud service, and the benefits that would flow.

I'm firmly of the opinion that Apple won't go there, not soon, not in-a-while, not ever.

They have too firmly entrenched attitudes about constructing and maintaining "full-control" over the euphemistically named, "User Experience".

Apple don't do "collaboration", sharing their technology or allowing the mere User to tinker with its Gorgeous Stuff. It'd no longer be "their Design" and they anathema to Apple.

Apple are into "control", which in itself is not a bad thing, but severely limits their software and system design decisions and implementations.

This isn't some simple "we know best" thing, but much deeper, intimately tied to their focus on High Concept Design and a finely crafted "User Experience". Which also means controlled experience.

Apple could make huge inroads into the PC market by licensing OS/X - something it could've done anytime in the last 10 years. Now that "classic" computers are under 25% of its business, Apple could let go of its stranglehold on its computer hardware and light the fires of innovation: "let a thousand roses bloom". But they cannot and won't.

This translates to iCloud in two ways:

they haven't thought of the idea themselves, and
they probably couldn't model the response times of torrent-like service and would baulk at any service which is in the least unpredictable, perhaps sometimes not-quite-perfect.

Apple need to control its "User Experience", which means they can't let other players on-board and can't adopt "radical" or "unproven" solutions. (ie. "not invented here").

So they will build and run some very large datacenters to run iCloud.

The trouble with this approach is they are leaving the field of Innovation open to their competitors.
We know Microsoft won't embrace it, but Google and Android will and do.

Even Great Design can be copied and tweaked, sometimes even improved.That the British lost it home-grown motorcycle and motor car industries, know for radical and innovative design, to the Japanese and their "continuous improvement/refinement cycle" demonstrates this thesis.

In 10 years, will the "iPhone 15" be a patch on Android and the gazillion services/applications it runs?
I suspect not. The most amazing and usable devices are unlikely to come from the standalone corporations.

It could be Google and Android, it could be something completely new.

It just won't be Apple at the front, again.
Think how they blew their advantage of the Apple II. They've got form and the same fixed, rigid mindset is still rampant. That's good for Bold New Steps, poor for continuous stepwise refinement.

Apple iCloud and peer-peer (Torrents)

2011-06-12T11:38:00.000+10:00

Will Apple add a torrent like ability to its iCloud offering??

iCloud is a remote filesystem with a lot of metadata and does 4 things:

provides "second copy of my precious data" (for files I've generated)
allows synchronisation of those files across the multiple devices/platforms a user connects. This is the aspect Apple 'sells': email, contact and calendar sync and restore/recover.
mediates the enforcement of copyright and content distribution
does Internet-Scale data de-duplication.
- By data volume, the Internet is a 'Viewing Platform'.
- 1 upload == zillions downloads [write once, download ~infinite]

Apple could create an Internet-Scale "Content Delivery Network" with iCloud if ran a peer-peer network, something like the hugely successful bit-torrent protocol/service.

Because you've got authorised content and validated entities/logins in a vendor controlled environment, there isn't a direct copyright or leakage problem, just the ever-present and non-removable "analogue hole".
There is scope for scanning never-before-seen files to see if they are recodings, subsets or 'analogue rerecordings' of know files.
What action then? Automatically remove the file, "Bill the User" or send a Summons?

'Backups' of already known files take the time to transfer and compare the checksum/identifier. That's a incredible compression ratio/speed-up. Those checksum/identifiers also are the natural keys for both the 'torrent' and backing-store key.

Storing the per-machine file/directory structure is another layer and doesn't yield to the same de-duplication/compression techniques.
If I were implementing the local filesystem, I'd do two things:

calculate and store checksums on-the-fly and store in the metadata.
make sure part of the metadata was as whole-file checksum or UUID-type identifier.

Possibly also calculate and store large-chunk (8-64Mb) checksums.

This enables two services usually only seen at Enterprise scale:

Document-Management-System like controls, searches, functionality.
user collaboration: tagging, comments, hilighting, edits/recuts + mashups, annotation, linking, etc.

As bit-torrent shows, using distributed {storage, net-bandwidth, CPU} scales, amplifies 'Server Effectiveness' and gives apparent 100% uptime/availability to services.
It's really cheap and easy for the content provider, though causes more traffic on the ISP network.

BTW, this isn't limited to Desktops and mobile devices.
It scales to IP-TV and on-demand content services.
Your Apple-TV box effectively has infinite & perfect storage...
Internode has announced 'fetchTv' - so these services are on the radar for ISP's.

It also has significant consequences for Ozzie customers who pay per Gb.
You really don't want your iPad on a 1Gb wireless 3G plan acting as a torrent server. A nasty $2000 bill surprise!

There are serious performance issues with local congestion (POP, ISP, backhaul, main-site) inter-network links and dealing with ADSL bandwidth characteristics.

The NBN is going to be a Layer 2 network, (2-level VLANs or 802.11 "Q in Q").
The presumption is ISP's will offer PPPoE to begin with, as for the current ADSL services.

PPoE is not well suited to distributed data/torrents:

your source device puts packets onto your LAN,
your firewall/router fields those packets and pushes them to your ADSL modem which encapsulates packets into PPPoE, then puts these new packets onto the wire
the bytes go down your line to the DSLAM
are routed down the 'backhaul' link to the ISP's nearest site
into the 'Access Concentrator' to become a public IP addr
then routed towards the destination public IP address, which could be on the same Concentrator, the same POP, the same ISP, a shared 'Interconnect', or an upstream provider
into the destination Access Concentrator
down the backhaul, DSLAM, ADSL model, firewall/router and eventually appear on the destination LAN to the receving device.

Which gives you so many single-point-of-failure/congestion/saturation that it isn't funny...

If the other person is across the hall or across the road, this incurs a massive and needless overhead, not to mention delays and multiple local resource contention.

The wikipedia PPPoE article discusses problems and current solutions

So, if iCloud becomes a torrent-like service, will it overload the NBN??

A tablet from Microsoft? It won't save the company.

2011-05-28T16:24:00.002+10:00

Update 1: Mary Jo Foley of ZDnet tips a launch of Windows 8 next week at AllThingsSD.

From a Forbes blog:

Microsoft’s Windows OS For Tablets Coming Next Week?
May. 27 2011 - 12:11 pm, Agustino Fontevecchia

A tablet-optimized, Windows OS could be ready to see the light of day as early as next week, according to recent market chatter and media reports. (designed for System on a Chip)

... at the Ira Sohn Conference, Einhorn, president of Greenlight Capital, said that it is “hard to envision success in mobile” for Microsoft, despite its recent partnership with Nokia.

Why was Microsoft so slow to enter the tablet market?
It has to do with Microsoft’s dominance of the PC operating system market.
As the world leader in PC/Client platform space, with Windows as one of the most expensive components in a PC (along with the CPU), the company receives “anywhere between $40 and $100 for a Windows OS license with 85% contribution margin when someone buys a new PC,” explained Aggarwal.
Lighter and cheaper tablets push down prices for PC, putting pressure on Windows pricing. The same thing goes for smartphones, which lower users’ dependency on PCs.
The incentives to lead the charge into cheaper products are not in place.

But now, with tablets eating their way into the market, Microsoft is being forced to adapt.

Reuters reported a Citigroup forecast that Microsoft could garner a 'meaningful' share of the Tablet market by 2013.
In two years, what's the iPad going to be like, and how will have Android manufacturers responded?
Unlike the latest Windows Mobile release meant to "bury the iPhone", Microsoft can't just be "me too", but must be substantially better than than all competitors.
It has to play in a field it has no demonstrated leadership in: Usability.

The big lesson from Apple's iPod, iPhone and iPad is they've understood the price-point in this market, as well as the importance of usability and design, on top of a strict requirement for Content (via iTunes).
Microsoft in the PC world has competed on its "universality" (ie. near monopoly) and if that fails (Internet Explorer vs Netscape) on price, even resorting to "free" or worse.

Neither of those strategies is going to work in the smartphone and Tablet market.

What I love about the Forbes piece is the optimism and its refusal to remember inconvenient facts like:

"everything is a PC" according to Ballmer this year.
So why aren't you shipping iPhone killers with Windows Vista or 7?
Microsoft invented the smartphone market with Windows CE, at least a decade before the iPhone, yet they've become essentially irrelevant.
So what went wrong? Microsoft definitively had the First Mover Advantage.

Will Microsoft repeat it's X-Box strategy of subsidising the hardware? (a retail price below cost of manufacture).
That's high on my list: they will try to buy market-share without thought of impact on profits or longevity. Microsoft focus on market-share rather than the more important, profitability.

And like the X-box, will be able to tell themselves for years they are doing "great things" while bleeding red-ink.

The benchmark is Apple: when did their new products, even less successful ones like Apple TV, make losses?

I agree with Einhorn that Ballmer is the reason that Microsoft shares are undervalued.
But I don't think there's an opportunity there, for the same reason I've declined Telstra shares.

Even at a %Yield of 5-10% versus the current 2.6%, I'd need 10-20 years to recover my investment.
The way the company's stock performance is going (dropping stone-like), I'd never get my money back.

i.e. If you bought stock 10 years ago and sold it today, the loss is more than the dividends paid, so you've made a nett loss.

I expect the stock price to keep falling by more than Dividends paid, and cant see a change until Microsoft manages to compete successfully in a New Internet Market (search, game consoles, smartphone, ...) or comes up with a better revenue model plus methods to drastically cut overheads, while maintaining "Software Development" efforts and good product (code) quality.

Microsoft could be a profitable company for many decades to come by concentrating just on its "mature" and market dominant products, "Windows" and "Office". It can only do this by being a much smaller, more focussed and better run company.

To be a profitable PC-focused company, it can't be a $70B/year company with a 75+% gross margin.
It has to be a $5-20B/year company with a 40% gross margin, which a consequentially much reduced market capitalisation.

There is no way the current Board, CEO and major stockholders, Gates and Ballmer included, could countenance such an outcome. Ballmer is a self-made "10 Billion-dollar Man". He'd rather lose all that money than wilfully backtrack. His ego won't let him admit things have changed.

So Microsoft and all their shareholders are stuck in a death spiral, where every turn inspires riskier plays, more squandering of its wealth and less capacity to change.
("lets buy Skype!" "Why?" "Because it will save us!" "How?" "Because it will!")
(or "lets do a Tablet O/S!" "Huh?" "We'll kill everyone!" "Huh?!" "We always have before!")

So I respectfully disagree with Mr Einhorn and maintain that Microsoft is a bad investment at any price.
It can't stay big and manage its way forward in The Age of the Internet, it's had 10+ years trying, and if it transitions to a profitable well-managed 'small' PC company, it's share price must collapse.

Microsoft Troubles XII: IBM market cap re-overtakes MSFT

2011-05-24T08:21:00.003+10:00

Update 1. Influential hedge-fund manager, David Einhorn president of Greenlight Capital, calls for Ballmer to stand aside.

Update 2. The MSFT board stands behind Ballmer, rejects David Einhorn's call to stand aside.

Einhorn has 9M MSFT shares (0.011%). He's bought because he thinks they're undervalued.
This could just be a media beat-up by him to make some money - the share price has increased.

Whatever the cause, this is a significant milestone.
The MSFT board has had to consciously and publicly defend their continued choice of Ballmer as CEO.

Original:
From Reuters, Bill Rigby, SEATTLE, Mon May 23, 2011

IBM passes Microsoft's market cap after 15 years

Since the Internet technology bubble burst in 2000, the tables have been reversed and Microsoft's stock has been stagnant as investors doubt its ability to move beyond its Windows operating system and Office suite of software

An investor putting $100,000 into both stocks 10 years ago would now have about $143,000 in IBM stock and about $69,000 in Microsoft stock.

According to Reuters data, Apple's market value stood at $309.2 billion on Monday, IBM at $203.8 billion and Microsoft at $203.7 billion.

IBM is now ranked fourth in terms of market value in the United States, behind oil giant Exxon Mobil Corp at $397.4 billion, Apple, and industrial and finance conglomerate General Electric Co at $205.6 billion.

The piece seems to give two different valuations of APPL, $309.2B vs $205.6B, but it is poor syntax in the 2nd para.

My takeaway from this piece:

MSFT (senior) management has failed woefully since 2000 to keep up with its Industry & Sector
The MSFT board has been "captured" by that management, and
the company is not being run in the best interests of the majority of stock-holders, large or small.

Look at MSFT's "financials", they are rapidly approaching "product maturity" (as seen by a growth rate closer to the S&P 500 than their Industry & Sector), but their Gross Margin is way, way too high.

                                       Company    Industry    Sector    S&P 500
Sales - 5 Yr. Growth Rate 9.45    11.99   16.93      7.63
Gross Margin - 5 Yr. Avg.       80.31    68.90       39.98    31.07
Operating Margin - 5 Yr. Avg. 36.92    25.93   16.81    15.00
Revenue/Employee (TTM)   770,955    5,007,815    5,391,381    731,155
Net Income/Employee (TTM)    244,876    572,253      222,783      89,500

They are setting themselves up to be slammed by the first good "substitute" product.

The Internet completely changes the market dynamics of intangible goods - notably Software.
Not only does it reduce the "Cost of Sales" to a few cents, it prompts companies to adopt entirely new Revenue Models and be brave enough to let go of it's star performers.
ie. "what got us here won't get us there, to the next level".

If MSFT is forced to reduce it's Gross Margin to 40% (not even ~30%), it's ~$70B yearly Revenue collapses, and their amazing Profits become outstanding and irrecoverable losses.

APPL might have had a vision with new hardware types (like iPod, iPhone), but it had to embrace an entirely different sphere of operations to enable this: without iTunes, the APPL i-Revolution could not have happened.
APPL had the insight, people want content, the device is just a means to an end.

But most importantly, APPL has been prepared to let their own (new) products compete with its "core business". Computers are now under 25% of APPL's business, and falling. (But their market share in that line of business is growing).

GOOG demonstrates the power of the Internet marketing model, "bits are free" and that 'alternative' revenue models don't just work, they can be very profitable.
That it wasn't a fluke is proven by the demise of all the other search engine companies, including the front-runners and "first movers". GOOG was nowhere 15 years ago, now it's King of the Search.

Microsoft Troubles XI: APPL more profitable than MSFT

2011-05-11T16:30:00.000+10:00

Adam Harthung at Forbes wrote Why Not All Earnings Are Equal; Microsoft Has the Wal-Mart Disease, byline is May3, 2011.

Read the article, says more than I can, with more (economic) facts and more eloquently.

It's only ever been about the company's economic performance.
Poor products and ignoring your customers only ever have one outcome.
Sad for the employees and shareholders, though.

Please note, I am specifically saying that Microsoft products are NOT doomed, just the company.

Winders on the Corporate Desktop isn't going away anytime soon (2-3 decades to run at least).
Too much invested, too many careers tied to it and the Lemmings Rush hasn't turned elsewhere yet...

The next two Big Questions for Microsoft, the company:

How soon before the Board notices and removes Ballmer?
Who will be the eventual purchaser of the profitable lines-of-business - Windows and Corporate solutions?

Why Microsoft is doomed to Commercial Failure.

2011-01-25T09:06:00.000+11:00

Microsoft doesn't understand Software, or at least how to build it at-scale and the people who create it.

All software and systems have a design life. As the world and platforms change, what was once a good design degrades. The classic example of "bit-rot" was "Y2K". The code and applications didn't change, but time showed up fundamental design limitations or flaws. Some organisations resolved the issue by setting the date back 28 years and "screen-scraping". It's a temporary solution that amplifies the point.

On top of that, complexity is your enemy:
a lot of "undergrowth" develops over time and has to be cleared or it chokes the forest.

Which means that at some point we all have to start again...
[Hint: Apple did that with OS/X]

A neat demonstration/proof comes from the world of Windows protocols:
SAMBA, a portable reimplementation of the Windows SMB protocol for "File and Print".

The SAMBA team is up to its fourth complete redesign in 20 years.
Their work is not only faster and more robust than Microsoft's code, they follow the specifications exactly and their testing regimes have yielded multiple serious faults in Microsoft's implementations.

In the title, Commercial is the key word.
Like IBM's OS/360 (and MS-DOS) I expect Windows code of many flavours to be still running after 40-50 years.

I'm not saying Windows software will go away any time soon, but that Microsoft will become Commercially unviable.

Profits are the lifeblood of all companies and the difference between income and expenses.
Elsewhere I've said I expect sales of Windows to erode and for some (not all) market segments, collapse.

I'm asserting two things in this piece:

Windows will become less competitive, and hence lose income, in an increasing number of market segments as the importance and use of "standard services" increases. The cause: an old codebase and design.
Expenses for maintenance and developing new "features" will rise exponentially as programmers fight with a lot of old "cruft" in the code and try to adapt an inappropriate design to provide new services.

In time, the two lines will cross and Windows at least, will become commercially unviable.

We know from IBM's experience in the late 1980's that while many customers demonstrate near "inelastic demand" and prices can be increased year-on-year for much longer than you might think.
At some point they "snap" and look to replacements (product substitutes).

Unless Microsoft follows Apple and "Resets" its entire codebase, it is doomed commercially.

Up until now the Corporation and its leaders have shown extreme unwillingness to even consider this, apparently because:

their self-image of "World's Best Software Company" doesn't allow them to admit their code could be less than perfect.
they are "packrats". They can't bear to remove old code and designs.

The last 10 years have seen the Microsoft share price halve in one year after the "Dot Boom" became the "Dot Bust", and since "gone sideways" while it's competitors have gone from strength to strength.

The next 10 years are crunch time for Microsoft. It's market domination in "PC's" is eroding and more important market segments are developing where it doesn't have a presence and it's usual approach of "bloatware" will be rejected in the marketplace. Think iPod and iPad.

Microsoft Troubles X: Ballmer as CEO being questioned

2011-01-15T13:17:00.003+11:00

Richard Waters published a piece, "Ballmer's opportunity to prove his worth", on 12-Jan-2011 in the Financial Times. It's been picked up and reprinted - I become aware of this through an investment newsletter. Microsoft's performance is now a concern/topic for mainstream investors. That can't be good.

Waters starts:

Some members of Microsoft’s board of directors feel the need to keep their chief executive under closer scrutiny.

That does not mean that Steve Ballmer’s job is on the line – at least, not yet. But it does show that some on the board are getting restless about the company’s performance under his 11-year watch. And there may be few catalysts that will make things look any better in the short term to make his position any more comfortable.

Please respect FT.com's ts&cs and copyright policy which allow you to: share links; copy content for personal use; & redistribute limited extracts. Email ftsales.support@ft.com to buy additional rights or use this link to reference the article

Five years ago this article would've been unthinkable, Ten years ago (2000), talking about the worlds' leading software and hardware companies without mentioning Microsoft was inconceivable.

Ballmer insists "Everything is a PC".

Nope. He's got it upside-down and inside out.
Not just a logic error, but a fault in his world view - you might call it "blinkered".

"The Internet Changes Everything" and "Everything can be a Computer" seem axiomatic and behind the current (whatever it is) revolution that's given us the iPhone and iPad - and their many competitors. These new devices rely on ubiquitous (and affordable) communications and "The Cloud" for storage and services.

(Microsoft) PC's do Internet and are computers - but to reverse the thought is wrong.
I.e. "If it connects to the Internet and has a CPU it must be PC" fails Propositional Logic 101.

The original (1980) vision of Paul Allen (the co-founder moved on by Bill Gates and Steve Ballmer after becoming ill) "A computer on every desk" was achieved by 2000. We're now in new territory.

A recent Endgadget piece commenting on "Windows on ARM" gave me a couple of fresh insights:

MSFT's long release cycle is a strength for both it's Corporate Customers and Developers
Corporate clients, now that Windows has 'sufficient' capabilities, want/value stability over everything: a decade between releases would suit them just fine.

There are two worlds in software now: 'fast' and 'slow' release cycle.

'fast', typified by Apple and Open Software, does major releases at least yearly and with continuous (at least monthly) updates. On the Supply Side, Developers need to keep up, especially if they track all the new features. On the Demand Side, Users/Consumers get lots of new stuff.
'slow', typified by Microsoft's 3+ year cycle, allows the Supply Side to focus on their code and only rarely deal with platform upgrades. On the Demand Side, businesses can maximise their ROI and make maximise the efficiency of their processes.

In 2002, Windows XP became Microsoft's first fully competent universal Operating System by combining the Industrial-Strength features of Windows 2000 and supported the popular legacy consumer line derived from "Windows 3.1".

It was an accident of history that Longhorn was cancelled and it took 5 years to release "Vista", which both consumers and corporations resisted. [Proof is the rapid take-up of "Windows 7".]

By accident, not design, Microsoft gave Home and Corporate customers a taste of a Good-Enough platform which they didn't tinker with and do their usual "forced upgrade" - and met with overwhelming approval.

The management challenge for Microsoft now is to figure out how to deal in conflicting or opposite worlds:

monetize their slow-release, stable platforms and
partake in the rapid innovation, high-release Internet World.
OR, chose one and own it.

Ballmer isn't cutting it or understanding the world has changed and the existing 17-layer bureaucratic model won't support or allow the free-wheeling, low-inertia, high-speed culture needed.

Proof is the XBox (Entertainment & Devices) division:
Released at the start of 2002, it was 6 and a half years before they announced their first profitable year.
Their strategy was to give away the hardware for long enough to buy enough market share.

And they've had a parade of failed products: Zune, "Project Pink" (Kin), and the market collapse of "Windows CE" and the associated phones, spring to mind. [I'm not a Microsoft watcher.]
With an associated trail of fired executives... [And more by the day, see this piece by Joe Wilcox.]

The Microsoft Board should be able to understand these notions and know enough History of Technology to see this pattern has been repeated often.

The answer for Ballmer and the Board is simple: "You can't get there from here".

Where can you get?
Either broke and irrelevant or leaner, meaner and "owning" the Corporate Desktop.

Do one thing and do it well, you cannot be All Things to All Customers.

A piece of mine from a year ago, "Why Microsoft is being left behind" describes them as "being trapped by success" and suffering massive Management/Leadership failure in these areas:

Lack of vision and competence in Management,
Rigid and bureaucratic hierarchy,
Dysfunctional culture, and
Incompetence in their technical speciality: writing software.

What I didn't say at the time, Microsoft is addicted to insanely high gross profit margins: over 90%.
Long-term gross-margins can only be 20-40%, or your customers will leave and your competitors take over.
This extraordinary profitability has led directly to very poor management practices. Winding back profit margins will create a lot of angst and upheaval, creating further organisational challenges.

For more on the mechanics, see this piece: "Death by Success". Victor Cook points out that Microsoft is breaking the rules of Economics: spending more for Market Share than the revenue returned.

Microsoft management, and you'd guess the Board, have come to believe that a) 90% gross margin is normal and b) that "small" markets are beneath their aspirations.
After the 2008 Global Financial Meltdown, every adult on the planet should understand that 20-year Bull Markets aren't normal or sustainable. In the same way, neither are extraordinary profits.

Every market, even for PC's, start small. The iPod in its first year wasn't a world-beater.
I'm sure, despite the external hype, the Execs at Apple weren't sure of the success of the iPhone before release. They may have been hopeful and optimistic, even confident. But not sure...

That's the nature of real-world Business: it's unsure and risky. Sometimes it takes years to become "an overnight success".
But you have to start somewhere, and small is easy and limits your losses.

After 10-years at the helm, Ballmer is completely responsible for the state of the company.
There is never a business reason for "viscous internecine wars", only personal/political.
Over time, that alone will prevent success and/or destroy the organisation.
Part of the root cause of Microsoft's miserable stock market performance is a lack of focus and a lack of insight into their Strengths, Abilities and Market:

Ballmer can't make this transition on his own and now, being free from the shadow of Bill Gates for 3 years, will the Board give him the impetus to make Microsoft his own?

Is Past Performance an indication or guarantee to Future Performance?
Not necessarily, but Microsoft under Ballmer has only made plays from their usual Play Book.
So why would anyone, especially investors and The Board, expect anything to change?

Ballmer is not a Software guy like Bill is/was. Despite his claims, Longhorn and Vista shows he and his Big Band of managers don't know understand Software, the people that do it, nor how to manage large-scale development.

Nor does Ballmer have Gate's ability to "turn on a dime" and take the whole company with him: the 1995 "Internet Tidal wave" memo when they dumped their whole proprietary network.

There are two factors that, if left unchecked, will see a rapid decline in Microsoft's fortunes:

Lack of strong, insightful Leadership, and
a culture of incompetent management.
"Fat, Dumb and Happy" seems to describe their overblown bureaucracy.

I can see three outcomes:

Ballmer "gets it" and transforms himself and then the company. [I think most unlikely given his past performance.]
Ballmer does "more of the same" and spends a fortune trying to buy Market Share, or
the board finally gives up on him and tries to find someone to take on the lame duck...

The only scenario that sees Microsoft getting back to high earnings is a Ballmer 'transformation'.
The two Business As Usual approaches will either see them burning cash at a remarkable rate or Some New Guy taking a giant axe to the company, which will ultimately destroy it.

A weak Microsoft would make a great acquisition for someone like Computer Associates or Fujitsu: business that exploit dependence on once must-have Software and Systems.

OR, my favourite, Bill Gates might take it private and recreate it. I don't think he'd give up his family to do that.

The startling thing for I.T. outsiders is how quickly technologies can be superseded, modulo the few in the "long tail" for whom Change is All Too Hard.
The rate of change-over is exponential.
By the time "Management" and Shareholders become aware there is a problem, the situation is irreversible, compounded by the lead-time to create new products.
The I.T. landscape is littered with the bones of Dinosaurs - once great and powerful companies who dominated their field and were seen as "permanent".

From the first sign of problems in the Balance Sheet to "Close the Doors" seems to be less than 5 years. [Sorry, guesstimate, haven't found the hard data yet.] The usual path is for the brands and assets to be sold to specialist "End-of-Life" companies.

5 years ago I started predicting Microsoft would have a "financial pothole" around now (+/- 1 year).
It may still, but through a different route than I expected - over-spending, rather than through product substitution.

Clocks running...

Why new Secure Internet solutions are technically Hard

2010-10-13T08:49:00.000+11:00

Information Security is both very hard and very easy at the same time.

Not only are Internet Nasties a nuisance, or worse, they prevent the new, useful Applications and Networks like e-Commerce, i-EDI, e-Health, e-Banking, e-Government and other business/commercial transactions systems.

Perfect Security isn't possible: ask any bank.

Defenders need to be 100.00% correct, every minute of every day.
Attackers need just one weakness for a moment to get in.

Not all compromises/breaches are equal: from nothing of consequence, up to being in full control with system owners not being aware of it.

All 'Security Systems' can only be "good enough" for their role, which depends on many factors.
How long do you need to keep your secrets? Minutes or Decades?

Building a system isn't an end-point:

"Information Security is a journey, not a destination" (Schneier)

Security has two aspects:

creation and
operation or patrolling.

Not only do you have to build it safe, you have to work to keep it safe and have real-time Intrusion Alarms being monitored: just like in the real-world.

Best efforts are needed to provide the mechanisms to keep Information Confidential + Correct, but equal attention has to be paid to detecting and tracing breaches and exploits.

The usual acronyms used for Information Security are "CIA" and "AAA":

Confidentiality (secrets stay secret)
Integrity (data isn't changed, added, deleted unless authorised)
Availability, ("denial of service". Can't get to your data == No system)

and

Authentication, (proof of Identity)
Authorisation, (what the ID can do) and
Accounting/Auditing (what was done)

If this were all there were to it, then the Internet wouldn't be riddled with malware, keyloggers, worms, SPAM, etc.

There's more...

First, all systems have to be "hardened":
be difficult to break into.
Second is Administrative Control:
every system and application change can only made by a few competent, diligent people authorised to do so.

Once unauthorised systems changes can occur, the system is untrustworthy and can't be used to store, handle, transmit confidential information.
Ever.
User-controlled systems, not just poor systems and Apps, are the source of the woeful state of the Internet.

This is why all efforts to create general Internet Security solutions are doomed before they start.

There are thousands of Secure Internets running in Government, Defence and Finance/Banking.
They use a simple technique:

strong security boundaries, or "air gaps".

Within these networks, with their 'military grade' encrypted links, there is no call for "secure protocols". They run secure webservers with plain HTTP, not https/SSL.

Once you have hardened systems, nailed down links, controlled systems and strong identification + authentication of users, there's no need for complex protocols or multiple levels of encryption.

Under these conditions, even Windows is good enough and doesn't need virus/malware scanning to keep it clean.
A serious and determined hacker always targets the weakest points, so stronger systems are used for the most secret or sensitive networks.

The approach is called "defense in depth" versus "hard shell, soft centre".
Getting through the first/outer defence shouldn't give attackers the keys of the kingdom.

The first commercial computers were built and sold in 1950, by 1968 there was enough concern with failing projects that NATO sponsored the first conference on "Software Engineering".

1969 saw the birth of both the Internet and Unix (and 'C', the universal programming language)
1970 saw the enumeration and formalisation of Computer Security principles.
1977 saw the invention of usable "Public Key" encryption systems.

By 1980, everything that was needed to prevent and control all the evils running around unchecked on the Internet were known and being practised somewhere.

The problem isn't one of Knowledge, but of implementation.

The specific problem is endemic in I.T.:

"reinventing the wheel"

Unless you've a PhD and a decade or two of "industry" experience, you really don't want to be reinventing 'Computer Security'. If you do need to invent something new, you need to go through a public and extended examination by experts in the field.
Just as the US DoD did when developing AES (Advanced Encryption Standard).

Even the encryption algorithm dreamed up by the ITU, the world body for telephone standards, for GSM phones (A4) was quickly cracked when exposed to the professionals in the field...

"Security by obscurity" has been proven repeatedly to be a fools' paradise.
The algorithmns and processes need to be public and transparent, and well examined. Operational security comes does to "key management" and diligence.

So what might solutions look like?

PKI isn't "the answer":
Single point of attack - too much trust in an external entity without "skin in the game".
Hijacked connections and wrongly granted certificates are easy exploits.
Nor are VPN's "the answer":
They don't identify people and are useless without hardened/controlled systems everywhere.

There are a whole raft of systems out there on the Wild Wild Web that are both (very) high-value targets and have to be cheap and effective: servers run by ISP's and hosting providers.
They do get compromised occasionally, but not in large numbers or for long. Nothing like the 'botnets' of millions, yes, millions, of Windows machines.

Simple tools that work and are used by those that care...

SSH: 1995. 1997 SSH-2 Internet-Draft
PGP: 1991
POSIX (~1990) [Linux, Unix, Solaris, AIX, OS/X] and SELinux (2003) for the truly paranoid.

PGP, "Pretty Good Privacy", is much stronger than it might sound. (Computer Nerds often indulge in puns and understatement). The US Government regarded it as so good, it's creator, Phil Zimmerman, was charged with exporting Military "munitions". It took around 5 years for the case to be dropped.

Eugene Kaspersky, the principal of security company " Kaspersky", is advocating for Global Internet Passport for everything:
http://www.theregister.co.uk/2009/10/16/kaspersky_rebukes_net_anonymity/

"Everyone should and must have an identification, or internet passport," he was quoted as saying. "The internet was designed not for public use, but for American scientists and the US military. Then it was introduced to the public and it was wrong...to introduce it in the same way."

Elsewhere he notes that smartphones partly solve the Identity issue. Each mobile phone has a unique identifier: an IMEI. He suggests that can be associated with an individual, in the same way that phone numbers are. Whilst a start, there is a basic flaw, who's using the device?

Any Global system smacking of "Big Brother" just isn't going to happen. Revolutions have been caused by less.

There is also a more subtle conflation between two different needs/worlds:

when you need trusted, secure systems and
when the freewheeling, anarchic Wild Wild Web is just right.

But the notions behind Kaspersky's proposal are good and fundamental to any Secure Internet solution:

registered and identified devices, like mobile phones
strong user identification and forced authentication
off-line processes associating people (and how to find them) with digital identities
explicit registration or authorisation of a device and user onto a service
with the inverse, repudiation of compromised, fake/forged/stolen/lost identities.
administrative control of all devices on a network.

These are already what is done globally on mobile phone networks.
It's a model that's known to work, scales to immense numbers, is locally administered and socially acceptable

Quality and Excellence: Two sides of the same coin

2010-09-20T06:07:00.000+10:00

Quality is predicated on Caring.
High Performance, also called "Excellence", first requires people to Care about their results.

They are related through the Feedback Loop of Continuous Improvement, also known as O-O-D-A (Observe, Orient, Decide, Act) and Plan-Do-Check-Act (from W. Edwards Deming).

The Military take OODA another level with After-Action-Reviews or After-Action-Reports (AAR's), a structured approach to acquiring "Lessons Learned".

High Performance has two aspects: work-rate and consistency.
It's not enough to produce identical/consistent goods or results everytime, but you have to do it with speed.

There's an inviolate Quality Dictum:
You can't Check your own work.

For Organisations, this Dictum becomes:
Objective assessment requires an Independent Expert Body.

From which follows the necessity for an External Auditor:
Only Independent persons/bodies can check an Organisation and its people/processes for compliance and performance.

For around 80 years, Aviation has separated the roles of Investigation, or Root Cause Analysis, from Regulation, Compliance and Consequences. In the USA the NTSB Investigates and the FAA Regulates. This has led to consistent, demonstrable improvement in both Safety and Performance. Profitability is linked to Marketing, Financial Management and Administration, not just Performance.

All of which leads to the basic Professional Test for individuals:
"Never Repeat, or allow to be repeated, Known Errors, Faults and Failures".

And the Raison d'être of Professional Associations or Bodies:
To collect, preserve and disseminate Professional Learnings of Successes, Failures, Discovery and Invention.

Barry Boehm neatly summaries the importance of the Historical Perspective as:

Santayana half-truth: “Those who cannot remember the past are condemned to repeat it”

Don’t remember failures?
Likely to repeat them
Don’t remember successes?
Not likely to repeat them

All these statements are about Organisations as Adaptive Control Systems.

To effect change/improvement, there has to be reliable, objective measures of outputs and the means to effect change: Authority, the Right to Direct and Control, the ability to adjust Inputs or Direct work.

Which points the way as to why Outsourcing is often problematic:
The Feeback Loop is broken because the hirer gives up Control of the Process.

Most Organisations that Outsource critical functions, like I.T., completely divest themselves of all technical capability and, from a multitude of stories, don't contract for effective Quality, Performance or Improvement processes.

They give up both the capability to properly assess Outputs and Processes and Control mechanisms to effect change. Monthly "management reports" aren't quite enough...

Business Metrics and "I.T. Event Horizons"

2010-09-12T00:35:00.029+10:00

Is there any reason the "Public Service", as we call paid Government Administration in Australia, isn't the benchmark for good Management and Governance??

Summary: This piece proposes 5 simple metrics that reflect, but are not in themselves pay or performance measures for, management effectiveness and competence:

Meeting efficiency and effectiveness,
Time Planning/Use and Task Prioritisation,
Typing Speed,
Tool/I.T. Competence: speed and skill in basic PC, Office Tools and Internet tools and tasks, and
E-mail use (sent, read, completed, in-progress, pending, never resolved, personal, social, other).

As a taxpayer, in a world of White Collar Desktop Automation, I'd expect some quantitative metrics for "efficiency" and "effectiveness" as required of Agency heads in s44 of the FMAA (Financial Management and Accountability Act, 1997), not just some hand-waving, bland reassurance by those Heads and the Audit Office that "we're World's Best Practice, Trust Us".

We know that "what you measure is what you get" (or is maximised) and that career bureaucrats are:

risk adverse (C.Y.A. at all times),
"very exact", they follow black-letter rules/regulations to the precise letter, and
very adept at re-interpreting rules to their advantage.

Unanticipated outcomes abound, the least of which is using reasonable rules to fire or move-on challenging and difficult people, such as "whistle-blowers", innovators, high-performers (showing up others is "career suicide") or those naively enquiring "why is this so?".

These "challenging" behaviours are exactly those required under s44 of the FMAA to achieve:
"Efficient, Effective and Ethical use of Commonwealth Resources",
yet they are almost universally considered anathema by successful bureaucrats.

This bureaucratic behaviour also extinguishes and punishes exactly the elements of "Star Performers" identified in the research of Robert E. Kelley.
Done in the mid-90's, the lack of take-up, or penetration, of this research in the Public Service leads to other questions: Why Not?

In the Bureaucratic world, asking for one thing gets precisely the opposite result.
Something is missing, wrong or perverted... But this has been the essential nature of large bureaucracies since the Roman Empire.

There's a double- or triple-failure going on here:

The desired outcomes are not being achieved,
This isn't being detected by the Reporting and Review mechanism's: Agency Annual Reports or Audit Office reports, and
a culture of non-performance is built, reinforced and locked-in.

An aside:
If any Agency is truly managed to the minimum standards required by the FMAA, the three-E's, how could there ever be any whistle-blowing??

That there are whistle-blowers such as Andrew Wilke, is proof of systemic, perhaps systematic, failures, and worse, at many levels.

Simple mindedly imposing minimum "standards" across-the-board would not only be a waste of time, but would be massively counter-productive within this environment.

So what might work??

"Events Horizons" in the world of Information Technology may point the way.

Between 1990 and 1995, using {Intel 486, 256Mbit RAM chips, twisted-pair Ethernet LANs, and Windows Desktop plus File and Print Servers}, PC Desktops made their way onto the bulk of clerical desktops. Usage was mainly "Productivity Applications", mainframe access and a little in-house messaging. Cheaper and Faster paper-and-pencils plus zero-delay transfer-to-next-in-process: Automated Manual processing, with simple fall-back to actual manual processes.

From 1995 to 2000, Internet use took off and Office Desktops went from being expensive/Leading Edge items to low-end commodity units. Usage focused more on e-mail, Intranets and some office tools. "Client-Server" became the buzz-word. New processes arose that weren't just Automated Manual processing.

After 2000, and the forced upgrades for Y2K, there was a 3-5 year technology plateau/recovery followed by an up-tick in both penetration, usage and integration of I.T. tools and facilities.
File/Print and Office Tools "on the Network" are now taken "as a given", as is high-speed Internet links, good Security, "Standard PC images" and centralised I.T. purchasing, admin and support.
E-mail and Web access are ubiquitous and inter-system compatibility is a necessity.

From 1990 to the present, Government Agencies have moved from having all backend processing automated, to the majority of front-end processes and work tasks being dependent on I.T. Automation: Desktops, Networks and Services.

Telephony systems are increasingly being moved to the common network, becoming less robust and less reliable in the process. We are yet to see the full impact of this trend and it's reversal for critical services.

Now, when a large Agency has a major computer room malfunction or upgrade glitch, most or all office staff are sent home until all critical systems are restored.

This didn't happen only 10 years ago:
the loss or slowing of back-end systems didn't halt all Agency work, an effect unremarked and unreported by both Agencies and their oversight organisations, Finance and the Audit Office.

There are real End Service-Delivery implications of this current Event Horizon and they aren't being addressed or even acknowledged. Nor do these avoidable costs and employee time losses constitute efficient or effective management.

We've passed the Event Horizon of Dependence of Front Office Operations on I.T. [The next is complete dependence, there after "invisibly dependent", like water, gas and electricity.]

The bell can't be "unrung", we now have to manage our efforts in this context. Wanting to go back to "simpler times" is not an option for many and complex reasons. Even if it were possible or desirable...

Can we use this insight to define some universal metrics related to the use ("efficiency") and outputs ("effectiveness") of the technology and whose measurement has insignificant staff impact?

An aside:

Measuring values and analysing data, "instrumentation", always costs: money, time, complexity.
This process/proposition is not free or trivial.
Nor is the valid interpretation of read-outs always simple and obvious.

You wouldn't think of flying a 747 without instrumentation and that wall of switches-and-displays are the minimum needed for "Safe and Effective" aviation (we know this from the constantly improving safety/performance metrics from the ATSB etc).
Why do Managers and Boards think the larger, more complex, more costly machines we call "organisations" need little or no instrumentation?

Some Metrics

Business tasks, perhaps as well the stock-in-trade of Managers, "decisions", have four major internal dimensions:

Degree-of-Difficulty
Size
Urgency
Importance (and/or Security Classification)

And four related external dimensions:

Time: Deadline, timeliness or elapsed time
Minimum Acceptable "Quality"
Input Effort (work-time or staff-days)
Cost

All tasks, projects and process have Inputs as well:
Resources, Plant, Equipment, Tools, Information, Energy, Licences, etc
and, for well-defined tasks/projects, defined measurable Outputs.

When measuring Inputs vs tasks completed, actions taken or messages communicated, classification by Internal dimensions is necessary for "Like-with-Like" ("really-same", not "about-same") comparisons.

Are Urgent tasks/enquiries dealt with as appropriate?
Are Important tasks/projects completed? On time, On Budget, To Spec?
How many tasks of what size/difficulty are reasonable to expect?
Do staff find the work rewarding and motivating or not?
Are they engaged, stimulated and developed through the work or demotivated, unproductive and either leaving or Time-Serving until they can ("golden handcuffs" of superannuation or other benefits).

With E-mail, both the inputs and outputs are available to be analysed.
Individual and Group Outputs can be assessed according to the External dimensions.

Were budgets (costs, deadline, effort, resources) met for a matrix of different task classes (urgency, importance, size, difficulty)?
Were Quality targets met per task matrix?
Where were Staff Effort and Resources expended?
Was this optimal or desired?

Measuring Realised Benefits and "Expectations vs Outcomes", the very heart of the matter, is beyond the scope of this piece.

Having generic I.T. tools available on every desktop and used for every task implies three metrics related to mastery of basic tools and skills:

Typing Speed,
Tool/I.T. Competence: speed and skill in basic PC, Office Tools and Internet tools and tasks, and
E-mail use (sent, read, completed, in-progress, pending, never resolved, personal, social, other).

There are two basic work-practice skills, where the I.T. systems gather data necessary for analysis:

Meeting efficiency and effectiveness
Time Planning/Use and Task Prioritisation

After 5 decades of availability of definitive solutions in, and widespread training and consulting firms offering services, these basic and critical Business Processes, there is no excuse for poor meeting skills or undue waste of staff-time in meetings.

Nor of incompetent time management/task prioritisation and the associated waste of staff-time, idleness and under- or non-achievement of goals.

"Meetings, The Practical Alternative to Work" (or now, "Email, T-P-A-t-W"), is not "that's how it is here" or just amusing, it is an indictment of poor and ineffective management and failed governance systems.

So we hit an inherent contradiction:
We need to measure basic performance metrics to affect improvement, but if we try to use those metrics to achieve improvement, we can only create the opposite effect.

If tying Performance Metrics to pay, bonuses or "Consequences" isn't useful, why measure and report?

Jerry Landsbaum in his 1992 book, "Measuring and Motivating Maintenance Programmers" definitively answered this question.

Just by measuring and publicly posting individual resource usage, he was able to achieve radical changes in habits (and costs) without imposing penalties or instuiting any formal process.

Reasonable people given timely, objective feedback will modify their behaviour appropriately.
Landsbaum went on to provide a suite of tools to his staff providing various code metrics.
Without direction, they consistently and deliberately improved the measured "quality" of their code.

As a side-effect, Landsbaum was able to quantify for his management considerable savings and productivity improvements. Most importantly, in a language and form they could understand:
an Annual Report with year-on-year financial comparisons.

This approach is backed up by Dr. Brent James, Executive Director of Intermountain Health Care in Salt Lake City, Utah, described in "Minimising Harm to Patients in Hospitals", broadcast in October 2000.

Dr James and his team spent time discovering exactly what caused the most harm to patients under their care, then prioritising and addressing those areas.

The major cause of "adverse events" (harm to patients) wasn't Human Error, but injuries due to Systems failures, by a factor of 80:1 (eighty times).

Charles Perrow calls these "Normal Accidents", whilst James T. Reason, author of "Human Error" and the "Swiss Cheese Model" of accidents, calls them "Organisational Accidents".

Perrow and Reason's work is the underpinning of the last 5 decades improvement in Aviation and Nuclear safety. It's based a sound theory that works in practice, based on real, verifiable evidence.

Dr James said the approach: "could save as much as 15% to 25% of our total cost of operations" whilst delivering much reduced Adverse Events and better, more consistent, patient outcomes.

An unanticipated benefit of Dr James work was identifying the occasional "bad physicians and nurses":
"If we see active criminal behaviour, if we see patterns of negligence or malfeasance, we will react.".

(Because there was) "less noise in the system. It’s easier to see them.
And I have to tell you that was startling when we first encountered that.
We knew we needed to go after the 95% of the system’s failures
but as we started to take down those rates we also knew that there were some bad physicians,
it was just hard to find them,
and suddenly, there they were,
and we were able to take appropriate action."

Bundaberg 2005: Dr Jayant Patel

In Dr James' hospitals, Jayant Patel would have been quickly noticed and dealt with.

If you lived in Bundaberg, you might be asking why their systems didn't detect Patel: 5 years after Dr James' public broadcast in Australia?
Is there an excuse the management at Bundaberg ignored Dr James proven, effective, methods?
Especially as he'd documented substantial savings as well as fewer injuries and better patient outcomes from his approach. All the self-described goals of every Health system in the country.

Queensland Health's performance fails the basic Professional Test:
"Repeat, or allow to be repeated, Known Errors, Faults and Failures".
And seemingly without timely, direct or personal consequences to anyone.

Whilst Patel is seen to be "the one bad apple", his being charged and held to account is not timely nor will it improve the standard of care for others, or cause lasting change where "it can't happen again".

Just what did those lives lost and needlessly destroyed, and the ruining of Patel buy the community?
Seemingly, very little.
Retribution leaves ashes in the mouth, and "playing the Blame Game" only increases workplace fear and risk-adverse management decisions. None of which drives useful or lasting organisaitonal change.

In Bundaberg, the culprit is "The System" that let Patel firstly pratice at all, then get away with bad performances for an extended period. I won't go into the poor treatment of the nurses that tried to address the situation and who eventually managed to get media attention.
It's all the same Organisational Failure.

Other 2005 events: Lockhart River aircrash and the sinking of the Malu Sara.

Where is the routine investigation and transparent, public reporting by an independent expert body akin to CASA/FAA, as for the 2005 Lockhard River crash that killed 15 and led to the demise or deregistration of two companies. This same crash led to a Senate Inquiry critical of the oversight bodies: the coronial inquest and CASA. "Who watches the Watchers?" The Senate, for one.

Patel was linked to 87 deaths, six times more that the crash, though only convicted of the manslaughter of 3. In spite of the size of this "oops" and the overwhelming evidence of the power and effectiveness of the NTSB/FAA system in Aviation, there are no moves to effect this sort of change.

This isn't an isolated organisational condition or limited to any one level of Government or area of practice.

Consider the 2005 deaths of all five on-board the "Malu Sara", specified, purchased and operated by the Department of Immigration. Sinking 6 weeks after going into service.
The 2008 Coroner's Report, is cited by a 2010 SBS programme, around 12 months after the Coronial Inquest, as saying:

... Queensland's coroner ruled it was a totally avoidable disaster, caused by the incompetence and indolence of senior Immigration official ...

The SBS programme claims:

"No charges were laid after a 2007 police investigation."
The senior Immigration official "avoided departmental disciplinary proceedings by retiring from immigration - with his full entitlements."
"The federal work place regulator ... is prosecuting Immigration over the deaths.
The maximum penalty - a $240,000 fine."
"So far, all the families have received from authorities is an apology from Immigration and, in January, the department named two rooms in its Canberra headquarters after the deceased (departmental) officers."

The formal words of the Coroner, "incompetence and indolence", should alarm anyone reading them, especially those with oversight of the Department or responsible for Good Governance.

This behaviour is never justifiable in a well managed and is completely inconsistent with the Three-E's required of Agency Heads. That one senior officer failed their basic performance requirements and was either undetected or known and allowed to continue, is a failure of Governance and oversight.

One major event like this is complete proof of failing under s44 of the FMAA.

The Audit Office has not investigated the matter, nor has Finance, the administrators of the FMAA, taken an interest.

A Senate question was asked in May 2007 about the investigation:
In July/August 2006, more Senate questions were asked in relation to AusSAR.

Is the Department’s report on the Malu Sara incident a document that the Committee can have access to? If not, why?
Answer:
The Department’s report was provided to the Coroner at the directions hearing on 15 February 2007. The Coroner, on his own motion, made an order that prohibits the publication of the report other than to the formal parties to the proceeding.

There was an independent ATSB inquiry and report released in May 2006 (No 222) and a supplemental report (MO-2009-007) released in September 2009, reopening the investigation after the Coroners Report.

In late 2009, ComCare issued a media release saying they would be launching court action against the Department and the boat builder.

The Head of Immigration in the 2008-9 Annual Report commented:

The department has since made changes and improvements to its procedures to ensure that such a tragedy could never occur again,...

It seems all Agencies involved in the matter are unaware of the Quality dictum:
You cannot check your own work.

Organisationally, this equates to:
Performance and Compliance can only be assessed by an Independent Expert Body.

Organisations can't investigate their own problems, nor categorically state, as Immigration has, "We fixed it, it can't happen again. Trust Us".

Since the 1926 formation in the US of the dual-body model used in Aviation, one to investigate causes (NTSB) and another to form and enforce regulations and issue non-compliance penalties (FAA), has shown itself to be an effective, possibly a definitive solution to Organisation Safety and Quality improvement.

From the steady improvement in aviation performance figures, an unintended effect of the dual-body system is that it may also improve Performance and Efficiency/Effectiveness for free.

Why is there this blindspot, especially as the NTSB/FAA model is so well known and respected throughout the commercial and public sectors, and in political and judicial circles?

Wrapping up Performance Metrics

Putting real numbers out in Public enables good people to lift their game while exposing poor performers and worse (malfeasance, misfeasance, nonfeasance, negligence, incompetence, indolence, ...)

Formalising the measurement of basic management outcome metrics and tying them to rewards and punishments can only result in disaster. Mangers will devote themselves to doing whatever it takes to get promotion or reward, not achieving their mission: good taxpayer outcomes and good governance.

Providing good data to taxpayers and their proxies, journalists and researchers, will provide enough leverage to see real, lasting change.

But this approach is not "sexy", big or expensive - and certainly can't be used as a punitive political weapon, either for career bureaucrats or their political masters.

Why wouldn't you do this if the whole Output of Government was dependent on the use of I.T. and you cared about "the efficient, effective and ethical use and management of public money and public property"?

So who will champion this idea?

Who has something to gain from real change? [Not the major Political Parties, not incumbent Bureaucrats, not existing oversight bodies: the status quo works for all them.]

Who has the Motivation, Authority and Will to make real change happen?
That's the real question here...

Internetworking protocols

2010-08-29T17:45:00.000+10:00

Placemarker for a piece on Internetworking protocols and problems with IPv4 (security and facilities) and IPv6 (overheads, availability).

"The Internet changes everything" - the Web 2.0 world we have is very different to where we started in 1996, the break-through year of 'The Internet' with IPv4.

But it is creaking and groaning.
Around 90% of all email sent is SPAM (Symantec quarterly intelligence report).

And since 2004 when the "Hackers Turned Pro", Organised Crime makes the Internet a very dangerous place for most people.

IPv6 protocols have been around for some time, but like Group 4 Fax before them, are a Great Idea, but nobody is interested...

What are the problems?
What shape could solutions have?
Are there (general) solutions to all problems?

Systems Design

Are these new sorts of systems possible with current commercial or FOSS systems?
What Design and Implementation changes might be needed?

How do they interact with the other 'Computing Challenges' in this series?

Flexible, Adaptable Hardware Organisations

2010-08-29T17:35:00.001+10:00

Placemarker for a piece on flexible hardware designs.

I'd like to be able to buy a CPU 'brick' at home for on-demand compute-intensive work, like Spreadsheets.
I'd like be able to easily transfer an application, then bring it back again.

Secondly, if my laptop has enough CPU grunt, it won't have the Graphics processing or Displays (type, size, number) needed for some work... I'd like to be able to 'dock' my laptop and happily get on with it.
The current regime is to transfer files and have separate environments that operate independently and I have to go through that long login-start-apps-setup-environment cycle.

I prefer KDE (and other X-11 Windows Managers) to Aqua on Snow Leopard (OS/X 10.6) because they remember what was running in a login 'session', and recreate it when I login again.

In 1995, I first used HP's CDE (IIRC) on X-11, that provided multiple work-spaces. This was mature technology then.

It was only this year, 15 years on, that Apple provided "Spaces" for their uses.
Huh??

We already have good flexible storage options for most types of sites.
Cheap NAS appliances are available for home use, up to high-end SAN solutions for large Enterprises.

For micro- and portable-devices, the main uses are "transactional" web-based.
These scale well already, and little, if nothing, can be done to improve this.

Systems Design

What flows from this 'wish list' is that no current Operating System design will support it well.
The closest, "Plan 9", developed around 1990, allows for users to connect different elements to a common network and Authentication Domain:

(graphic) Terminals
Storage
CPU

The design doesn't support the live migration of applications.

Neither do the current designs of Virtual Machines (migrate the whole machine) or 'threads' and multi-processors.

Datacentre Hardware organisation

2010-08-29T15:56:00.006+10:00

Senior Google staffers wrote The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, which I thought showed break-through thinking.

The importance of this piece is it wasn't theoretical, but a report of What Works in practice, particularly at 'scale'.

Anything that Google, one of the star performers of the Internet Revolution, does differently is worthy of close examination. What do they know that the rest of us don't get?

While the book is an extra-ordinary blueprint, I couldn't but help asking a few questions:

Why do they stick with generic-design 1RU servers when they buy enough for custom designs?
How could 19-inch racks, designed for mechanical telephone exchanges a century ago, still be a good, let alone best, packaging choice when you build Wharehouse sized datacentres?
Telecommunications sites use DC power and batteries. Why take AC, convert to DC, back to AC, distribute AC to every server with inefficient, over-dimensioned power-supplies?

Part of the management problem with datacentres is minimising input costs whilst maximising 'performance' (throughput and latency).

There are three major costs, in my naive view:

real-estate and construction costs
power costs - direct and ancillary/support (especially HVAC).
server and related hardware costs

Software, Licensing and "Operation, Administration and Maintenance" (OAM) costs may also be 'material' in the Accounting sense, I don't have that information or sources describing them.
[HVAC: Heating, Ventilation, Air Conditioning]

The usual figure-of-merit for Datacentre Power Costs is "Power Use Effectiveness"(PUE), the proportion of power that actually ends up being consumed by the Data Processing (DP) systems.
Google and specialist hosting firms get very close to the Green IT "Holy Grail" of a PUE of One. Most commercial small-medium Datacentres have PUE's of 3-5, according to web sources.

IBM Zurich, in partnership with ETH has done work water-cooling servers that beats a PUE of one...
2007, 2008 and 2009 press releases on their Zero Emission Datacentre.

Their critical insight is that the cooling fluid can start hot, so the waste ('rejected') heat can be used elsewhere. Minimally for hot water, maybe heating buildings in cold climates. This approach depends on nearby consumers for low-grade heat, either residential or commercial/manufacturing demands.

Water is 3,500 times more dense than air. It's much more efficient to use water rather than air as the working fluid in heat exchangers. The cost (plus noise) of moving tons of air around, both in capital costs, space and operation/maintenance is very high. Fans are a major contributor to wasted energy, consuming around 30% of total input power (according to web sources).

But in all that, where's the measure of:

DP Power productively applied (% effectiveness, Η (capital Eta)), and
DP Power used vs needed to service demand (% efficiency, η).

%-Effective, Η, is about power-use of hardware unavailable for production, poor architecture, poor OAM practices or duplicated-but-not-contributing 'mirrors' as in traditional active/passive redundancy. Network Appliance (rightly) make a lot of their active/active configuration.

Pay for two, use two not one. A better than 50% reduction is capital and operational costs right there, because these things scale super-linearly.

%-Efficiency, η, is the proportion of production DP power on-line used to serve the demand. Not dissimilar to "% CPU" for a single system, but needs to include power for storage and networking components, as well as CPU+RAM.

One of the critical deficiencies of PUE is it doesn't account for power-provision losses.
Of each 10MW that's charged at the sub-station, what proportion arrives at the DP-element connector?
[Motherboard, Disk Drive, switch, ...] There are many other issues with Datacentre design, here's a 2010 Top Ten.

When is a Wharehouse not a Wharehouse?

Google coined the term, Warehouse Scale Computing, so why aren't modern wharehousing techniques being applied, especially in "Greenfield" sites?

Why would anyone use raised flooring (vs solid or concrete) in a new datacentre?
One of the unwanted side-effects of raised floors is "zinc whiskers". Over time, the soft metal edging on the tiles suffers tiny growths, 'whiskers'. These get dislodged and when they end up in power supplies, create serious problems.
Raised floors make very good sense for small server rooms in office buildings, not wharehouse sized Datacentres.

Automated wharehousing technologies with tall racking and robotic "pick and place", with small aisles, seem to be quite applicable to Datacentres. Or at least trundling large sub-assemblies around with fork-lifts. These sub-assemblies don't need to be bolted in, saving further install and remove/refit time.

Which goes to the question of racking systems.
Servers are built to an archaic standard - both in width and height.
Modern industrial and commercial racking systems are very different - and because they are in mass-production, available immediately.

Which doesn't exactly go against the trend to use shipping containers to house DP elements, but suggests that real Wharehouses might not choose container-based solutions.

As the cost of Power, and even its availability, changes with economic and political decisions, it's efficient and effective management and use in Datacentres will increase in importance.

Can a Datacentre store power at off-peak times and, like other very large power consumers, go off-grid, or at least reduce power-use in these days of the "smart grid"?
Could those "standby" generators be economically used during times of excess power demand? It creates a way to turn a dead investment into an income producing asset.

A useful result of treating a Wharehouse Datacentre as a true wharehouse is that modulo the additional power and cooling, they are conventional wharehouses. Allowing either existing buildings to be bought and used with minimal conversion, or old/surplus Datacentre buildings to be sold and used "as-is".

Datacentre Cooling

Whilst "Free Cooling" (using environmental air directly, without any cooling plant) is a very useful and under-utilised Datacentre technique applicable to all scales of installation, it's not the only technique available.

One of the HVAC technologies that's common in big shopping malls, but never mentioned in relation to Datacentres is "Thermal Energy Storage", or Off-peak Ice production and Storage.

Overnight, when power is cheap and conditions favour it, you make ice.
During your peak demand period (afternoon), your A/C plant mainly uses the stored ice.
This has the dual benefit of using cheap power and of significantly reducing the size cooling plant required.
In hot climates requiring very large ice storage, they don't even have to insulate the tanks. The ratio of volume to surface area for large tanks means the losses for daily use are small.

"Hot Aisle" and related techniques are well known and in common use. Their use is assumed here.

Datacentre Power Provisioning

Reticulating 110-240V AC to each server requires multiple levels of inefficient redundancy:

UPS's are run in N+1 configuration, to allow a single failure. Additional units are installed, but off-line, to cover failures and increased demand.
Ditto for backup Generators.
Every critical DP-element (not all servers are 'critical', but all routers and Storage Arrays are), requires redundant power-supplies to cater for either PSU or supply failure. You need to draw a line once between what's protected and what isn't, a major gamble or risk. Retooling PSU's in installed servers isn't economic, and mostly impossible.

That's three levels of N+1 redundancy, plus UPS's and generators that must be regularly live-load tested, which in itself is hugely problematic.

Plus you've got a complex AC power distribution system, with at least dual-feeds to each rack, to run, test and maintain. Especially interesting if you need to upgrade capacity. I've never seen a large installation come back without significant problems after major AC power maintenance.

The Telecommunications standard is dual 48V DC supplies to each rack and a number of battery rooms sized for extended outages, with backup generators to charge the batteries.
Low voltage DC has problems, significantly the power-loss (voltage drop) in long bus-bars. Minimising power-losses without incurring huge copper conductor costs is an optimisation challenge.

Lead-acid batteries can deliver twice the power for a little under half the time (they 'derate' at higher loads), so maintenance activities are simple affairs with little installed excess capacity (implied over-capitalisation waste) because a single battery bank can easily supply multiples of it's charging current.

There are three wins from delivering low-volatage DC to servers:

Power feeds can be joined with simple diodes and used to feed redundant PSU's. Cheap, simple and very reliable. Operational cost is a 0.7V or 1.4V supply-drop through the diode, so higher voltages are more efficient.
Internal fans are not needed, less heat is dissipated (according to web sources). I'm unsure of the efficiency of DC-DC converters at well below optimal load, if you over-specify PSU's.
This suggests sharing redundant PSU's between multiple servers and matching loads to PSUs.
the inherent "headroom" of batteries means high start-up currents, or short-term "excessive" power demand are easily covered.

The biggest win for a DC Datacentre supply is the integrated power control we see in every laptop:
the battery, PSU and HVAC systems can dynamically interact with the DP-elements to either minimise power use (and implied heat production) or maximise power to productive DP-elements.

The 3 dimensional optimisation problem of Input Power, Load-based Resource Control and Cooling Capacity can be properly solved. That's got to be worth a lot...

The "standard" 1RU Server

The problems I've got with typical 1RU servers (and by extension, with the usual 'blades') are:

There are 6 different types of heat load/producers in a server. Providing a single airflow for cooling means over-servicing lower-power devices and critical placement modelling/design:

CPU, ~100W per unit. Actively cooled.
RAM, 1-10W per unit. Some need heatsinks
Hard Disks, 2-20W. Some active cooling required.
motherboard 'glue' chips, low-power, passive cooling, no heat sinks.
GPU's, network and PCI cards. Some active cooling required.
PSU. Own active cooling, trickier with isolation cage.

1.75", the Rack Unit, is either too tall or too short for everything.
It's not close to optimal packing density.
Neither is the "standard height rack" of 42RU optimised for large, single-floor Wharehouse buildings. It's also too tall for standard shipping containers with raised flooring.
The fans and heat-sinks needed to cool high-performance CPU's within the 1.7" available are tricky to design and difficult to provide redundantly. Blowers are less efficient at the higher air speed caused by the restricted form-factor. Maintenance and wear is increased as well.
1Gbps ethernet has been around for a decade or more and is far from the fastest commodity networking option available. SAS cards are cheap and available, using standard SATA chips running at 3 or 6Gbps down each of 4 'lanes' with the common SFF 8470 external connector. At $50 extra for 12-24Gbps, why aren't these in common use? At least until 10Gbps copper Ethernet becomes affordable for rack-scale switching.
HDDs, (disks) are a very bad fit for the 1RU form factor, both for space-efficiency and cooling.
3.5" drives are 1" thick. They need to lie flat, unstacked, but then take considerable internal real-estate, especially if more drive bays are allocated than used.
2.5" drives still have to lie flat, but at 9.5-12.5mm thick, 2-3 can be stacked.
3.5" drives running at 10-15,000 RPM consume ~20W. They need good airflow or will fail.
Low-power 2.5" drives (2-5W) resolve some of those issues, but need to be away from the hot CPU exhaust.

One technology solution that starts to address the "blade cooling problem" is the sealed immersed-board liquid-coolant system from the UK startup, "Icetope", also in this referring article.

Sealing a motherboard puts all the liquid-tolerant components in the same environment, providing you've chosen your "liquid" well.
The IBM "ZED" approach above is directed at just the major heat load, the CPU's and uses plain water.
IBM is taking the stance that simple coolant loops, air-cooling of low-power devices and simple field-maintenance are important factors.

Which approach is better for Wharehouse-scale computing? Real-life production data is needed.
As both are new, we have to wait 3-5 years.

The obvious component separations are:

PSU's, separated from the motherboard and shared between multiple motherboards. PSU capacity can be very closely matched to their optimal load and cooling capacity and all DP-elements can benefit from full-path redundancy.
Disk drives can be housed in space- and cooling-optimised drawers.

There are already commercial solutions that pack 42 * 3.5" drives into 4RU drawers, either SATA or SAS. It's cheap and simple to provide a bulk connector, then run cabling to adjacent motherboards.

This shows my antipathy to the usual commercial architecture of large, shared storage-arrays with expensive and fragile Fibre Channel SAN's. They are expensive to buy and run, suffer multiple reliability and performance limitations, can't be optimised for any one service and can't be selectively powered down.
Google chooses not to use them, a very big hint for dynamic, scalable loads.

What could Google Do?

Very specifically, I'm suggesting Wharehouse-scale Datacentres like Google's with dynamic, scalable loads could achieve significant PUE and η gains (especially in a Greenfield site) by:

purpose designed tall racking, possibly robot-only aisles/access
DC power distribution
shared in-rack redundant DC-DC PSU's
liquid-cooled motherboards
packed-drawer disks, and
maybe non-ethernet interconnection fabric.

I suspect the total power demand for a notionally 20MW site might be reduced 25-50%.

Some other links to 'The Hot Aisle" blog by Steve O'Donnell:

What can you learn from a self-proclaimed "World's Greatest"?

2010-07-30T09:29:00.001+10:00

Note: This document is copyright© Steve Jenkin 1998-2010. It may not be
reproduced, modified or distributed in any way without the explicit
permission of the author. [Which you can expect to be given.]

Lessons from the Worlds' Greatest Sys Admin - July 1998

Presented at SAGE-AU Conference, July 1998

Contents
Introduction
Background
Principles of System Admin
Some WGSA Attributes
About The WGSA
Sayings of the WGSA.
Some Sound Management Laws
So What?
How do you work with a "World's Greatest ..."
Some "Good Stuff" I learnt from friends.
Some of the WGSA's work
Summary

Introduction

Why

(2005) The most frequent comment I received on the 'WGSA' talk was:
"So you think you are the World's Best Sys Admin?"

Answer: No, I am not the "WGSA".

This paper is about someone, who is really an amalgamn of a number of people, who regarded themselves as "The World's Best Sys Admin". They never verbalised this opinion - they just lived it.

Unfortunately, as is the case with all self-appointed 'guru's I've met, they had limited raw talent and an arrogance that prevented them admitting less-than-perfect performance, taking on-board any useful criticism or correction or learning new tools, techniques, processes, andorganisations from others they didn't consider an authority.

I apologise in advance to the reader that the paper is mostly about "negative" learning,
or What Not To Do...

I included a section on Good Things I've Learned to show that I wasn't totallypreoccupied with the negative :-) But there were just too many good stories, and I really had to let steam off over this...

Do you think you know the identity of "The WGSA"?

You don't. For those that may even think it's them - No, it's is not you.

The observations and opinions here came over a considerable period of time. It's not a single person - and it's not just Administrators. I've met "WG Programmer", Architect, Designer, Tester, Integrator, Networker, Technical Manager and CIO.

So - onto the main game - the paper as presented to SAGE-AU in Old Parliment House.
If you were there - did you catch any of the lollies I threw, or even a chocolate egg?

Feedback is something I'm interested in.

Drop me a line if you have your own stories, can add more useful models, or if you're late to this and found it useful. Suggestions for improvement gratefully accepted and acknowledged.

Why?

To codify and inform.

Once a problem is recognised and named, you can start to understand and address it.

Audience

Junior Sys Admins: - If you work for one.
Senior Sys Admins: - If you work with one.
Managers: - If you have one working for you.

Format

Talk
War stories
Feedback
And lots of Opinion.

The BIG Questions

So What?
How do you work with one?

Background

I spent a year in 96/97 contracting in Sydney for what should've been a large, prosperous Australian multinational. They hadn't paid a dividend since 1990 and were taken over by a Dutch company at the end of 1996.

The I.T. group was appalling.
Staff turnover in the Unix Support and Networking areas was high - close to 100% in 12-18 months! The company spent under 1% of turnover on I.T. - against the industry average of 5+%.

It felt like we were doing the impossible - and we were.

They'd outsourced their mainframe, embraced 'open systems', installed a large scale WAN, gone client-server, were developing GUI and O-O applications and had an Internet presence.

They'd also radically downsized in two steps:- from 200+ to 30-40 staff in 18 months.

They were fully Buzzword Compliant, but were going nowhere.

I was privileged to meet two people - both ex-telecoms technicians who had moved into computing. One, the self-proclaimed WGSA, had been responsible for setting up the Unix environment, and it's associated X.25 network, and been the Unix support manager for a couple of years, until finally taking a job in 'Technology Planning' - but just doing more of the same.

The other ex-tech I've remained good friends with. He moved into Networking after a career in Civil Aviation, then a TAFE. He had enough PC, Unix, and Internet knowledge 'to be dangerous'. He'd left behind at the TAFE an environment where just 2 of them supported and ran the
whole state TAFE network, +1 for Unix, 1 for printers and passwords, and 2 on the HelpDesk. When the lot was outsourced and a crack systems company took over - they boast they can cut 10%-20% from any operation - they ended up having to spend more.

The contrast was stark and savage - one had left behind a legacy of chaos and disorder, the other was undoing the damage and providing real business productivity.

This talk is about that experience and what I've learned.

Principles of System Admin

These are my values and principles. Your mileage may vary.

Know why you're there - To statisfy others business
needs.
Know what you Know, Know what you Don't Know, and don't be afraid to get assistance.
Obey Sound Management Laws.
Learn, Develop, and Stay Current.
We learn through Invention, Discovery, and Failure
"That which isn't growing, is dying"
Give Value for Money.

Actively seek ways to put yourself out of work.
Minimise recurrent costs - wages, maintenance/support/rental charges
Maximise Reusability, Flexibility, Functionality, Reliability/Robustness.

Provide what's needed, not apparently wanted.
Listen and communicate with your users.

Provide Solutions.
Focus on Outcomes.

Some WGSA Attributes

They don't exist outside fertile ground. They have to be allowed
and encouraged by management and peers.
Only Dysfunctional people thrive and rise in dysfunctional
environments.
Good people leave broken places - possibly after fighting for a time.
The only other alternative is to withdraw and retreat into minimal
performance.
People are the ONLY asset of I.T. Organisations
- Hardware: $1M to zero in 3 years
- Software: $100k to zero in 3 minutes
- Network: $250/point to zero in 3 seconds
Indicators:

High staff turnover
High contractor ratio
N.I.H. - Resistance to Change
Lack of "Professional" work habits - Defined Processes,
Designated Responsibilities, Delegated Authority
Lack of History, Documentation, Policy, Procedures, Config Mgt,
Version Control, Handovers, Induction
Chaos and Frenzy. Apparently understaffed and overworked - definitely unorganised
"No time to fix problems, too busy fixing faults."
Nobody tasked with automating jobs or passing work back to level 1 support.
Every install project goes into crash mode.
No standard, fast, system builds.
Maintenance frenzy - never seems to get any better.
Lack of fault analysis, reviews, Post Mortems, Post Implementation Reviews, capacity planning.
Single source of innovation and improvement - The "Guru"
The "BIG BANG FIX" is coming [Or the"Silver Bullet"]
Nothing can be done, because "someone" [WGSA or friend] is creating "the Solution to All Our Problems".
(Senior) Management "Swooping" is allowed and tolerated.
" Don't show me problems, Show me solutions"
Mentoring and skills transfer absent
Constant reactive, not proactive, administration.
No organisation accountability - fail to do any task - routine or project - with impunity.
Blaming and Recrimination normal. No attempt to perform 'root cause analysis' and rectify faults.
No recognition or rewards for work well done.
Few Diagnostic, debugging, or troubleshooting Tools - even for common failure modes.
No Communication - up or down.
No Performance Indicators or Measurement/Assessment

About The WGSA

Of course he was the best. He had read every single 'white paper' from the vendor, and with his photographic memory, could recite it all back. All he needed to know was in those papers, and the manuals he'd read.

He didn't need to meet and talk with his peers, he had none anyway! He had no need of professional organisations or finding out what had worked, or not, for other people.

If he didn't have time to do something himself, he would get in a contractor, create a project, or hire a consultant. Funnily, these people were always only of very modest ability. The projects mostly ran
out of money in "phase 1", when only the basic work was being done and well before the real benefits were to accrue.

He'd written 25,000 lines of shell script to provide a "common" menuing and execution environment. It was a most flexible, adaptable, and configurable environment - and surprisingly similar to that run by
his previous employer. Just the thing to control 12 machines... It was a real engineering triumph - for 1982! He'd built and deployed all this with no version control, configuration management, or documented release and maintenance procedures - and certainly no review.

His crowning glory, "Xferutility", 7,000 lines in a single script, heavily utilised 'comes from' control files [they just appeared places, with no trace of whence they came], and could use 'rcp', 'ftp',
and e-mail to achieve the functionality of uucp. Plus, it was the transfer mechanism, the interactive menu, the scheduler, and the status reporter. All things to all people bar those left to maintain it.

Having not apparently done "Programming 1A", he'd not been introduced to the concepts of "coupling and cohesion" - put together everything that belongs together, separate unrelated concerns -
and least necessary complexity.

To go from the login prompt to the first displayed menu, over a dozen files or scripts were executed - often in perverse order. The system drive defaults would overwrite the local definitions!

He also seemed unaware of basic capacity planning issues - like tracking the number of systems in the machine room and providing adequate rack space and cabling. Backups were another story entirely.

The I.T. department policy was to have separate small systems for every division, no two the same. In 12 months it went from 12 systems in the machine room, to 23. And then to 35+ in the next 6 months.

Having labelled me "a cannonball contractor who won't be around in the long term", he resigned the week he penned it, took an overseas holiday [run in the same flexible fashion], and rejoined his previous employer, through a services company, performing Network Management.

Sayings of the WGSA.

A few of these are paraphrases.

What I find myself saying often is :-
Why would you want it any other way?, and
(2005)Would you expect any less?.

The answers to these questions are usally: Yes, any other way, and "NO!".

Sayings and tactics of WGSA and friends:

A basic tactic: Plan, Plan, Plan - and produce massive documents everyone else has to review.
Nothing will actually get done.
Another basic tactic: Reality is at Fault, Adjust your Perceptions.
"It's worked that way for 3 years - it couldn't be broken now." [A basic tactic. You obviously have got the nature of the fault wrong. Ignorance and Rigidity are a powerful combination.]
Another basic tactic: Concentrate on the trivial, the Big Issues will fix themselves.
"You don't understand the full range of issues or complexities." [I know, you don't.]
"It works/worked fine for me..." [Hasn't told you or Reality is at fault.]
"Read the documentation I wrote." [But hasn't told you about.]
"You have to fully document that." [An attempt to divert, stall, or put you off.]
"The client doesn't want that." [Were they ever asked? Were they ever given options?]
"They [the clients] never asked." [Deflection. Clients are expected to be technical experts.]
"It's UnAuthorised." [But where is the Policy on that?]
"It's not Standard" [It's free. We have to pay heaps or the other boys will think we're not cool.]
"We can't afford that." [May be true, but unlikely based on the money chucked around on junkets/trinkets for the favoured few.]
"It's freeware. It's not supported." [Often said without a hint of irony in response to 'costs too much'.]
"If you can Cost Justify that..." [A stalling tactic. Nothing you put up will ever get approved.]
"You Just ..." [Makes you out to be a fool/incompetent, even though there is no way you could've known.]
"Why haven't you ... <;said angrily>" [So how would you know to do that, when you haven't been told about it?]
"It's really flexible/efficient/configurable/Easy when you use it... " [Defending a wildly over-complicated script]
"We need it because ...or We have to do it that way." [Of course there is nothing written to back it up. The WGSA wrote it, so it's going to stay.]
"We won't discuss that [now]." [No argument if there is no discussion.]
"That's not the way we do it around here." [No change is possible. Of course, nothing is written down and there is no Policy to back that up.]
"You can't do/say that." [Controlling.]
"What is the Vendor's policy on replacing that?" [Deflect and control. Of course the vendor doesn't have a written policy on when something is broken.]
"The Vendor's White Paper/Documentation says ..." [Appeal to another Authority. Stifle argument. Don't let facts or prior experience get in the way.]
"The Consultant's Report says ..." [Appeal to another Authority.]

Remember, there are rules for him and another set for you.
He will ignore e-mail, talk about you behind your back, set impossible deadlines [for you], and not keep his promises. Don't expect to be told about important stuff that impacts you, or that you happen to be expert in. You won't get invited to meetings, see reports, or be involved in
the 'discussions' held before major decisions are announced.

Rumour, disinformation, and 'Need to Know' are powerful tools for the WGSA.

He will casually drop bombshells, regularly spring 'surprises' on you, and practices 'Divide and Conquer' extremely well. He allocates work, but will never help or clarify what he wants. And of course, won't follow up on it. He may fly into a 'justifiable rage' if he comes back in a month and something hasn't been done to his satisfaction... It's not easy being so perfect and all-knowing all the time.

Rational argument won't work with the WGSA. What matters is that he thought it up, he's important, and the bosses [his mates], think he is an absolute Guru on everything.

And if you ever get close to criticising him or winning an argument - slander and libel work just fine for him.

Some Sound Management Laws

(2005) Note:I don't try to come up with any principles or Laws for The "WGSA" follows.
There is probably only one: "Seize Ever Opportunity". Which isn't a bad dictum, if it respects other people, fulfills your business's needs and goals and isn't only about advancing your personal agenda.

My version of "Sound Management Laws" are presented for you to consider and understand where I am "Coming From":

Delegate Authority with Responsibility and Accountability.
Follow up, Follow through, Be Consistent.
Value and Empower your staff: People are your only asset.
Do It NOW!
Follow The Quality Circle: Plan, Act, Evaluate.
Encourage and Reward Professional Behaviour, deal quickly with repeats of poor behaviour.
Lead by Example.
Forge, maintain, and support Teams.

In I.T. there are special management considerations:

Users come first

Satisfy Business Needs
Actively sell your successes and services to your users.
Constantly set and manage users expectations.
Inform, advise, consult
Be Honest and forthright - especially about your mistakes and failures.
Take care to explain Why it won't happen again.
Be Proactive. You get to drive the technology, they drive the business operations.

Know Yourself, Your Staff, Your Tools.
Never take on a job you cannot do.
Don't give others jobs they can't do.
Risk Management, Reviews, and 'Performance Audits' are your chief tools in establishing a Learning organisation.

Good working relationships between management and staff take time and effort to develop. They proceed through the following stages and are fragile. The whole lot, years of work, can be destroyed in an instant with a lie.

What management want are people they trust, work very hard, and consistently produce quality work. People who hold the company's best interests to heart.

Development Stages of People and Teams:

Honesty, Integrity, Openness, Frankness, Consistency
TRUST
RESPECT
LOYALTY
COMMITMENT, CARING

So What?

Since the advent of the 486 in ~91, cheap LAN's in ~94, and the Net in ~96, I.T. systems and infrastructure have become essential and critical for all business operations. Systems Administration, Networking, Help Desk, and Database Admin are the glue that holds it all together from
day to day.

There is a myth that software doesn't wear out like machinery.
The bits don't change, so it must be OK! By implication, you don't need to "maintain" systems and software, like you do machines.

So why aren't we all running 286's and DOS 3.3?

It's called 'bit rot'. The software doesn't change, but the environment does - which gives the same net effect. Year 2000 isn't a problem until your clock says 01/01/00.

My argument is that company profitability is related directly to, ignoring management and leadership issues, staff efficiency [$ cost / $ sales] and new product evolution. These are driven directly by I.T.
capability, which requires systems be constantly upgraded and enhanced - just to stay where you are! Similarly, I.T. operations staff must be continually increasing their own efficiency just to keep up.

(2005) See the 2003 Harvard Business Review article
"I.T. Doesn't Matter" by Nicholas G. Carr.

Effective Systems Administration is the single greatest point of leverage in the I.T. infrastructure - which is itself the single greatest point of leverage in an organisation. It amplifies and extends the
thinking, analysis, and decision making ability of the people in the organisation. Even sometimes the managers. It can even provide some corporate memory - a prerequisite for Knowledge.

It's obvious the software in airplanes, spaceships, nuclear reactors, medical instruments, weapons systems, banks, and ATM's has to be correct, robust, and dependable or there are disastrous, often
immediate, consequences. People die or billions goes missing. [Roll on NT - reactor control!]

What's not obvious is the long, lingering decline and demise of businesses - large and small.

The cost to Australia of losing a multi-billion dollar multinational company is incalculable. Well managed and well lead, it could still be a potent force on the global stage. Instead we have lost profits,
destroyed assets, and put a few thousand people out of work.

(2005) On May 28 2001, Australia's fourth largest telco, One-Tel, ceased trading on the ASX.
The Packer and Murdoch families, who control the media conglomerates PBL and News Corporation, lost about A$1Billion in the debacle. A major factor in the failure was uncollected "receivables". The computer billing system was faulty.

One.Tel closely followed the failure of HIH Insurance and Impulse Airlines.

That's a disaster 10 times bigger than TWA-800 going down outside New York just after take-off in 97, and they are still fishing out pieces. Just because it is in glorious slow motion - taking a decade, not a minute to unfold - doesn't mean we shouldn't still be as concerned with businesses going down as with aircraft crashes. People lives are destroyed and assets lost just as thoroughly in both types of crashes.

The government and professional bodies should be just as concerned with these outcomes and ensuring they can never happen again.

How do you work with a "World's Greatest ..."

I don't have an answer.

My style has been described as "Straight Up the Middle, with lots of smoke and noise."

My only response is to recognise an intractable situation early and leave as quickly as you can. A luxury I can afford, having no dependants and a low level of debt.

I Need an answer and would like - Your Feedback.

Some "Good Stuff" I learnt from friends.

Know what's important. Focus on that, ignore the trivia.
Practice - Order, Discipline, Rigour.
The job isn't done until your records are up to date.
Professionals do for $100k what anyone can do for $1M.
Remember Good Ways to do things when you see them.
ASK other people - what works? What doesn't.
STANDARDISE. Make it so they is just one way things happen.
Be prepared to work odd hours to not impact your users.
Hit your deadlines.
Clean up as you go.
The details are important.
DON'T accept a job you can't do.
Be Proactive, not Reactive.
Practice 'Root Cause Analysis' - fix faults and processes, not just
symptoms.
You have to stay on the leading edge. This takes lots of time and experimentation.
There is NO substitute for ability, experience, and general knowledge.
Aim for 100% reliability. Know what you have to do to achieve it.
If you make Rules, apply them without exception.
You may get called The Network Nazi, but it will all work and you will be respected.
Be personally flexible when dealing with users. Meet their needs, not just their expressed desire. This may involve some education.
Let users know what's happening.
Protect your staff from the vicissitudes of Management.
Freeware is FINE. If it meets the need, use it.
Know and Explore your tools.

Some of the WGSA's work

Here is a [longish] list of some of the wonderful technical and process problems I came across. Remember this was a largish, not huge, enterprise. There were only 75 Unix hosts, a thousand or so users [total], and a network that went to less than 100 sites.

Many of the systems were front-ends to the mainframe or a production system for the business.
The Unix support team was mostly 3 people, sometimes with a manager, sometimes with people doing performance analysis/reporting, or 'implementations' - such as HP Openview [I.T. Operations].

Common Environment: 25,000 lines of Shell Script. A good technology for 1982, not 1997.
Very poorly written. Basic programming rules of 'Coupling and Cohesion' violated.
All actions implemented as shell functions, but merged with interactive menu system. Extremely heavy reliance on Environment variables, with perverse re-mapping of names.
'Standard Operating Environment'. More shell script! No concept of standard builds, current patch levels, consistent program versions, or automatic software updates. 12 or more months of wasted effort. [Sold to the management initially on the great results from HP's internal network.
With 100,000 PC's and 23,000 Unix hosts spread over 660 sites, they saved US$200M/year in support costs alone by adopting a 'Common Operating Environment'. That was based on keeping all systems up to the same versions of software and config files.]
Xferutility: 7,000 lines of shell script, doing a subset of uucp's functionality.
Insidious bugs like:-
- Using the (local) return code of 'rsh' and thinking it was all working.
- Using rcp and not checking for a previous aborted transfer.
- [Destination file ends up with zero modes. Not writable by owner. Copy aborts, but script keeps chugging along.]
HP-UX 10 'bug'. #!/bin/ksh missing. Default '/sbin/sh' used with surprising results - 'exit' doesn't work.
/usr/local/bin banned. All executables and tools to reside in admin's home directories.
Common Admin logons banned. But 'essential utilities', like Xferutility, used a common account with .rhosts trusted all over the place, and even privileged access possible with sudo.
Common User Home directories basic to functioning of 'Common Environment' scripts. Ran ~/.profile to start menu, which [eventually] ran ~/$LOGNAME.profile.
NO master passwd file. No unique UID's, but notionally unique LOGNAME's.
NO mechanism to add or remove users from multiple machines.
NO shadow password files.
NO password aging.
NO retiring of unused accounts. No checking for intrusions.
Default password of LOGNAME. Never checked and never reset.
Help Desk's 'Password reset' function broken on most machines. No corrective action taken.
Crack broke 80% of the passwords on the central admin hub. [Including that of WGSA]. Nothing was done.
WGSA login setup on all systems, with .rhosts back to the admin hub, and 'sudo' access to 'mv' and 'cp'. WGSA had two passwords, family member names + digit. These were well publicised to all admins, and others.
NO definitive list of managed hosts.
DNS control files rebuilt every time from a 'hosts' file with 'host_to_named'.
NO alternate DNS primary.
A single central machine contained all the network services - DNS, e-mail, dial-in access, administration, master copies of scripts and system config files, root
passwords for all machines. This 'admin hub' was trusted, and could access all other systems. There was no fail-over system or contingency plan for massive failure.
Crippled DNS secondaries.
This was for 'security'. There was NO IP access control in the network. A user with only a little knowledge could navigate the entire network. There was an IP path back to the central DNS, and the IP number were allocated in an orderly fashion.
Internal domain left at: XXXXX.com. Even where a firewall was installed with the domain of XXXXX.com.au!
Even with over 2000 device entries in the DNS, and a strong numbering plan initiated by Networks, running sub-domains was firmly and frequently rejected.
Win-NT and DHCP posed no problem for the DNS. Permanent number leases were granted.
10 or more IP address ranges in use. Including a Class-B [the company owned], and other cute addresses like, 150.150.x.x [Wells Fargo's!]
IP over X.25 was chosen in 1994.
Routers were 'too expensive'. By March 1996 there were massive network failures - morning and afternoon - due to overload of the $250k X.25 switches.
Expensive terminal servers were deployed widely, 'because they handle IP over X.25'. Most production support problems related to config mgt, Network, printers, or terminal servers.
Untested backup tapes. In spite of a failure resulting in almost total loss of backup tapes for a system, no testing of readability of backups was performed.
Configuration Managements consisted of copies of scripts in WGSA home directory.
NO mechanism for rolling out fixes to faults as found.
Version Control consisted of block comments at the start of the scripts.
Common Code duplicated across 'menus'.
Hard coded 'user types' in Common Environment scripts.
File names not distinguished by hostname. All called 'AdminMenu' for the 'Admin' user.
nonStandard capitalisation of file Names and environment variables.
Very early version of 'sudo' used and modified. Non-standard config files. No repository of config files. No version control. [And WGSA didn't believe me when I found a long standing bug in his code.]
No reviews of code, scripts, systems.
Little testing of new code. Try it live!
No documented procedures for standard tasks.
No records of faults fixed.
No regular analysis or reporting of production faults.
No running sheets on production faults.
No weekly section meeting. No dissemination of information, plans.
No standard machine builds. [Complex and long procedures to build the production systems - with many variants.]
No capability to track or report critical file changes on production systems.
Network Naming Standard defined [but not for Printers and print queues]:
ux div 2 loc nr : 11 chars. Accepted by hostname, not by uname
- ux = Unix,
- div = Division 3 letter code,
- 2 = 1st digit of state postcode,
- loc = 3 letter code for town/suburb, arbitrarily assigned,
- nr = 2 digit machine number
WGSA Response: Set hostname to the long name, and uname to the old short
name! [So what's the standard??]
X.400 was chosen as the 'Standard external E-mail system'.
HP Openview [@$100k ?], was chosen as the corporate mail system - 'because it could make an address a program'.
External E-mail addresses were:- Firstname_Lastname@XXXXX.com.au
It took a long and bloody fight to get a script into production that used the Net standard of 'First.Last@XXXXX.com.au', plus generate all the usual abbreviations, and allow specific people to be included/excluded. This of course was removed a week or so after I left... [Only for them to hurriedly fall back to a manual list once they found a mail-loop problem.]
There were over 10 printing mechanisms, no map of network printers, and no naming standard for printers. [There was a printer called 'printer', and more than one called 'laser'.] Of course, nothing was documented on how it all worked, what got changed, or subtle faults found.
No disaster recovery or contingency plans existed. Hardware in the old AIX boxes occasionally died and caused not inconsiderable panic to the new admins.
The machine room had no sensible layout - even though it was newly installed in 96. There was a single ethernet for all the production, development, accounting, and maintenance systems.
Disk Layouts were recorded nowhere.
There was no consistency or standard way to way out Logical Volumes on disks.
The Journalling Filesystem [Veritas], was supposedly 'banned' from all HP-UX 10 systems. The defrag and on-the-fly extend utilities were an extra [pay for] package, so the 'free' part couldn't be used.

The watchword for the I.T. branch was 'CHEAP'.
[Do you think that was in any way related to the company dying?]

Summary

There are some people out there that don't just think, but know, they are the best.

They are dangerous.

Left unchecked they will not only make life a misery for everyone around, they help bring companies,
even very large ones, down.

What singles them out is their inability to take input from others.

Typical behaviours are:

Rigidity. Nothing can be changed.
Control. They have to say how everything is done.
Fixation. Things have to be done their way or not at all.
Discipline, rigour, defined processes. Usually absent. Always perverted.
Favoured Few. There is always an inner sanctum who control everything.

If they are well settled and well regarded, the organisation is dysfunctional. It will be soul destroying staying.

The only defense I know against them, once entrenched, is to leave.

And thank you all for your patience. I hope you have taken something away from all this...

Questions and Comments, please.

Page Last Updated:

Fri 30 Jul 2010 09:19:47 EST (to blogger)
Wed Feb 1 19:17:47 EST 2006
02-Jul-98 (first version)

Microsoft Troubles - IX, the story unfolds with Apple closing in on Microsoft size.

2010-05-09T09:54:00.001+10:00

Three pieces in the trade press showing how things are unfolding.

Om Malik points out that Intel and Microsoft fortunes are closely intertwined.
Jean-Louis Gassée suggests that "Personal Computing" (on those pesky Personal Computers) is downsizing and changing.
Joe Wilcox analyses Microsoft latest results and contrasts a little with

Whatever the state of play is, Steve Ballmer has been at the helm of Microsoft for a decade.
These are his results. He's taken an effective monopoly with a guaranteed cash-flow and a mountain of cash and turned it into "not the coolest place to work".

Will Microsoft go out with a Whimper, not a Bang?
A fierce Open Source advocate friend now "Just doesn't care" about Microsoft.
Is apathy the marker for a company's decline?

"Why Intel Will Be a Mobile Loser" by Om Malik

Atom would be competitive until 2011.
Too bad Intel sold its StrongARM technology to Marvell.

"Very Personal Computing" - Edited by Jean-Louis Gassée

The center of financial gravity in the computing world—the Center of Money—has shifted.

"Microsoft Q3 2010 by the numbers: Beats the Street, but Apple closes in" by Joe Wilcox

Apple announced fiscal 2010 Q2 results: $13.5 billion revenue and net profit of $3.07 billion, or $3.33 a share.

In 2009 Q2, the year before, Microsoft reported $4.4 billion operating income, $2.98 billion net income or 33 cents a share.
Apple: $9.08 billion revenue and $1.62 billion net income or $1.79 earnings per share.

Apple made enormous revenue and earnings gains against Microsoft in just one year.
The question now isn't so much if Apple might catch or surpass Microsoft but when.

Everything Old is New Again: Cray's CPU design

2010-05-03T16:12:00.000+10:00

I found myself writing, during a commentary on the evolution of SSD's in servers, that large-slow-memory like Seymour Cray used (not cache), would affect the design of Operating Systems. The new scheduling paradigm:

Allocate a thread to a core, let it run until it finishes and waits for (network) input, or it needs to read/write to the network.

This leads into how Seymour Cray dealt with Multi-Processing, he used multi-level CPU's:

There were Application processors, many bits, many complex features like Floating Point and other fancy stuff, but had no kernel mode features or access to protected regions of hardware or memory, and
Peripheral Processors (PP's), really a single very simple, very high-speed processor, multiplexed to look like 10 small, slower processors that performed all kernel functions and controlled the operation of the Application Processors (AP's)

Not only did this organisation result in very fast systems (Cray's designs were the fastest in the world for around 2 decades), but very robust and secure ones as well: the NSA and other TLA's used them extensively.

The common received wisdom is that interrupt-handling is the definitive way to interface unpredictable hardware events with the O/S and rest of the system. That polling devices, the old-way, is inefficient and expensive.

Creating a fixed overhead scheme is more expensive in compute cycles than an on-demand, or queuing, system, until the utilisation rate is very high. Then the cost of all the flexibility (or Variety in W. Ross Ashby's Cybernetics term) comes home to roost.

Piers Lauder of Sydney University and Bell Labs improved total system throughput of a VAX-11/780 running Unix V8 under continuous full (student/teaching) load by 30% by changing the serial-line device driver from 'interrupt handling' to polling.

All those expensive context-switches went away, to be replaced by a predictable, fixed overhead.
Yes, when the system was idle or low-load, it spent a little more time polling, but marginal.
And if the system isn't flat-out, what's the meaning of an efficiency metric?

Dr Neil J Gunther has written about this effect extensively with his Universal Scaling Law and other articles showing the equivalence of the seemingly disparate approaches of Vector Processing and SMP systems in the limit of their performance.

My comment about big, slow memory changing Operating System scheduling can be combined with the Cray PP/AP organisation.

In the modern world of CMOS, micro-electronics and multi-core chips, we are still facing the same Engineering problem Seymour Cray was attempting to address/find an optimal solution to:

For a given technology, how do you balance maximum performance with the Power/Heat Wall?

More power gives you more speed, this creates more Heat, which results in self-destruction, the "Halt and Catch Fire" problem. Silicon junctions/transistors are subject to thermal run-away, as they get hotter, they consume more power and get hotter still. At some point that becomes a viscous cycle (positive feedback loop) and its game over. Good chip/system designs balance on just the right side of this knife edge.

How could the Cray PP/AP organisation be applied to current multi-core chip designs?

Separate the CPU designs for kernel-mode and Application Processors.
A single chip needs only have a single kernel-mode CPU controlling a number of Application CPU's. With its constant overhead cost already "paid for", scaling of Application performance is going to be very close to linear right up until the limit.
Application CPU's don't have forced context switches. They roar along as fast as they can for as long as they can, or the kernel scheduler decides they've had their fair share.
System Performance and Security both improve by using different instruction sets and processor architectures for different applications. While a virus/malware might be able to compromise an Application, it can't migrate into the kernel unless it's buggy. The Security Boundary and Partitioning Model is very strong.
There doesn't have to be competition between the kernel-mode CPU and the AP's for cache memory 'lines'. In fact, the same memory cell designs/organisations used for L1/L2 cache can be provided as small (1-2MB) amounts of very fast direct access memory. The modern equivalent of "all register" memory.
Because the kernel-mode CPU and AP's don't contend for cache lines, each will benefit hugely in raw performance.
Another, more subtle, benefit is the kernel can avoid both the 'snoopy cache' (shared between all CPU's) and VM systems. It means a much simpler, much faster and smaller (= cooler) design.
The instruction set for the kernel-mode CPU will be optimised for speed, simplicity and minimal transistor count. You can forget about speculative execution and other really heavy-weight solutions necessary in the AP world.
The AP instruction set must be fixed and well-know, while the kernel-mode CPU instruction set can be tweaked or entirely changed for each hardware/fabrication iteration. The kernel-mode CPU runs what we'd now call either a hypervisor or a micro-kernel. Very small, very fast and with just enough capability. A side effect is that the chip manufacturers can do what they do best - fiddle with the internals - and provide a standard hypervisor for other O/S vendors to build upon.

Cheaper, Faster, Cooler, more robust and Secure and able to scale better.

What's not to like in this organisation?

A Good Question: When will Computer Design 'stabilise'?

2010-05-03T14:10:00.000+10:00

The other night I was talking to my non-Geek friend about computers and he formulated what I thought was A Good Question:

When will they stop changing??

This was in reaction to me talking about my experience in suggesting a Network Appliance, a high-end Enterprise Storage device, as shared storage for a website used by a small research group.
It comes with a 5 year warranty, which leads to the obvious question:

will it be useful, relevant or 'what we usually do' in 5 years?

I think most of the elements in current systems are here to stay, at least for the evolution of Silicon/Magnetic recording. We are staring at 'the final countdown', i.e. hitting physical limits of these technologies, not necessarily their design limits. Engineers can be very clever.

The server market has already fractioned into "budget", "value" and "premium" species.
The desktop/laptop market continues to redefine itself - and more 'other' devices arise. The 100M+ iPhones, in particular, already out there demonstrate this.

There's a new major step in server evolution just breaking:

Flash memory for large-volume working and/or persistent storage.
What now may be called internal or local disk.

This implies a major re-organisation of even low-end server installations:

Fast local storage and large slow network storage - shared and reliable.

When the working set of Application data in databases and/or files will fit on (affordable) local flash memory, response times improve dramatically because all that latency is removed. By definition, data outside the working set isn't a rate limiting step, so its latency only slightly affects system response time. However, throughput, the other side of the Performance Coin, has to match or beat that of the local storage, or it will become the system bottleneck.

An interesting side question:
How will Near-Zero-Latency local storage impact system 'performance', both response times (a.k.a. latency) and throughput.

I conjecture that both system latency and throughput will improve markedly, possibly super-linearly, because one of the bug-bears of Operating Systems, the context switch, will be removed. Systems have to expend significant effort/overhead in 'saving their place', deciding what to do next, then when the data is finally ready/available, to stop what they were doing and start again where they left off.

The new processing model, especially for multi-core CPU's, will be:

Allocate a thread to a core, let it run until it finishes and waits for (network) input, or it needs to read/write to the network.

Near zero-latency storage removes the need for complex scheduling algorithms and associated queuing. It improves both latency and throughput by removing a bottleneck.
It would seem that Operating Systems might benefit from significant redesign to exploit this effect, in much the same way that RAM is now large and cheap enough that system 'swap space' is now either an anachronism or unused.

The evolution of USB flash drives saw prices/Gb halving every year. I've recently seen 4Gb SDHC cards at the supermarket for ~$15, whereas in 2008, I paid ~$60 for USB 4Gb.

Rough server pricing for RAM in 2010 is A$65/Gb ±$15.
List prices by Tier 1/2 vendors for 64Gb SSD is $750-$1000 (around 2-4 times cheaper from 'white box' suppliers).
I've seen this firmware limited to 50Gb to improve performance and reliability comparable to current production HDD specs.
This is $12-$20/Gb, depending on what base size and prices used.

Disk drives are ~A$125 for 7200rpm SATA and $275-$450 for 15K SAS drives.
With 2.5" drives priced in-between.
Ie. $0.125/Gb for 'big slow' disks and $1 per GB for fast SAS disks.

Roll forward 5 years to 2015 and 'SSD' might've doubled in size three times, plus seen the unit price drop. Hard disks will likely follow the same trend of 2-3 doublings.
Say SSD 400Gb for $300: $1.25/Gb
2.5" drives might be up to 2-4Tb in 2015 (from 500Gb in 2010) and cost $200: $0.05-0.10/Gb
RAM might be down to $15-$30/Gb.

A caveat with disk storage pricing: 10 years ago RAID 5 became necessary for production servers to avoid permanent data loss.
We've now passed another event horizon: Dual-parity, as a minimum, is required on production RAID sets.

On production servers, price of storage has to factor in the multiple overheads of building high-reliability storage (redundant {disks, controllers, connections}, parity and hot-swap disks and even fully mirrored RAID volumes plus software, licenses and their Operations, Admin and Maintenance) from unreliable parts. A problem solved by electronics engineers 50+ years ago with N+1 redundancy.

Multiple Parity is now needed because in the time taken to recreate a failed drive, there's a significant chance of a second drive failure and total data loss. [Something NetApp has been pointing out and addressing for some years.] The reason for this is simple: the time to read/write a whole drive has steadily increased since ~1980. Recording density (bits per inch) times areal density (tracks per inch) have increased faster than read/write speeds, roughly multiplying recording density times rotational speed.

Which makes running triple-mirrors a much easier entry point, or some bright spark has to invent a cheap-and-cheerful N-way data replication system. Like a general use Google File System.

Another issue is that current SSD offerings don't impress me.

They make great local disk or non-volatile buffers in storage array, but are not yet, in my opinion, quite ready for 'prime time'.

I'd like to see 2 things changed:

RAID-3 organisation with field-replaceable mini-drives. hot-swap preferred.
PCI, not SAS or SATA connection. I.e. they appear as directly addressable memory.

This way the hardware can access flash as large, slow memory and the Operating System can fabricate that into a filesystem if it chooses - plus if it has some knowledge of the on-chip flash memory controller, it can work much better with it. It saves multiple sets of interfaces and protocol conversions.

Direct access flash memory will be always be cheaper and faster than SATA or SAS pseudo-drives.

We would then see following hierarchy of memory in servers:

Internal to server

L1/2/3 cache on-chip
RAM
Flash persistent storage
optional local disk (RAID-dual parity or triple mirrored)

External and site-local

network connected storage array, optimised for size, reliability, streaming IO rate and price not IO/sec. Hot swap disks and in-place/live expansion with extra controllers or shelves are taken as a given.
network connected near-line archival storage (MAID - Massive Array of Idle Disks)

External and off-site

off-site snapshots, backups and archives.
Which implies a new type of business similar to Amazon's Storage Cloud.

The local network/LAN is going to be ethernet (1Gbps or 10Gbps Ethernet, a.k.a 10GE), or Infiniband if 10GE remains very expensive. Infiniband delivers 3-6Gbps over short distances on copper, external SAS currently uses the "multi-lane" connector to deliver four channels per cable. This is exactly right for use in a single rack.

I can't see a role for Fibre Channel outside storage arrays, and these will go if Infiniband speed and pricing continues to drop. Storage Arrays have used SCSI/SAS drives with internal copper wiring and external Fibre interfaces for a decade or more.
Already the premium network vendors, like CISCO, are selling "Fibre Channel over Ethernet" switches (FCoE using 10GE).

Nary a tape to be seen. (Hooray!)

Servers should tend to be 1RU either full-width or half-width, though there will still be 3-4 styles of servers:

budget: mostly 1-chip
value: 1 and 2-chip systems
lower power value systems: 65W/CPU-chip, not 80-90W.
premium SMP: fast CPU's, large RAM and many CPU's (90-130W ea)

If you want removable backups, stick 3+ drives in a RAID enclosure and choose between USB, firewire/IEEE 1394, e-SATA or SAS.

Being normally powered down, you'd expect extended lifetimes for disks and electronics.
But they'll need regular (3-6-12 months) read/check/rewrite cycling or the data will degrade and be permanently lost. Random 'bit-flipping' due to thermal activity, cosmic rays/particles and stray magnetic fields is the price we pay for very high density on magnetic media.
Which is easy to do if they are kept in a remote access device, not unlike "tape robots" of old.
Keeping archival storage "on a shelf" implies manual processes for data checking/refresh, and that is problematic to say the least.

3-5 2.5" drives will make a nice 'brick' for these removable backup packs.
Hopefully commodity vendors like Vantec will start selling multiple-interface RAID devices in the near future. Using current commodity interfaces should ensure they are readable at least a decade into the future. I'm not a fan of hardware RAID controllers in this application because if it breaks, you need to find a replacement - which may be impossible at a future date. (fails 'single point of failure' test).

Which presents another question using a software RAID and filesystem layout: Will it still be available in your O/S of the future?
You're keeping copies of your applications, O/S, licences and hardware to recover/access archived data, aren't you? So this won't be a question... If you don't intend to keep the environment and infrastructure necessary to access archived data, you need to rethink what you're doing.

These enclosures won't be expensive, but shan't be cheap and cheerful:

Just what is your data worth to you?

If it has little value, then why are you spending money on keeping it?
If it is a valuable asset, potentially irreplaceable, then you must be prepared to pay for its upkeep in time, space and dollars. Just like packing old files into archive boxes and shipping them to a safe off-site facility cost money, it isn't over once they are out of your sight.

Electronic storage is mostly cheaper than paper, but it isn't free and comes with its own limits and problems.

Summary:

SSD's are best suited and positioned as local or internal 'disks', not in storage arrays.
Flash memory is better presented to an Operating System as directly accessible memory.
Like disk arrays and RAM, flash memory needs to seamlessly cater for failure of bits and whole devices.
Hard disks have evolved to need multiple parity drives to keep the risk of total data loss acceptably low in production environments.
Throughput of storage arrays, not latency, will become their defining performance metric.
New 'figures of merit' will be:

Volumetric: Gb per cubic-inch
Power: Watts per Gb
Throughput: Gb per second per read/write-stream
Bandwidth: Total Gb per second
Connections: Number simultaneous connections.
Price: $ per Gb available and $ per Gb/sec per server and total
Reliability: probability of 1 byte lost per year per Gb
Archive and Recovery features: snapshots, backups, archives and Mean-Time-to-Restore
Expansion and Scalability: maximum size (Gb, controllers, units, I/O rate) and incremental pricing
Off-site and removable storage: RAID-5 disk-packs with multiple interfaces are needed.

Near Zero-latency storage implies reorganising and simplifying Operating Systems and their scheduling/multi-processing algorithms. Special CPU support may be needed, like for Virtualisation.
Separating networks {external access, storage/database, admin, backups} becomes mandatory for performance, reliability, scaling and security.
Pushing large-scale persistent storage onto the network requires a commodity network faster than 1Gbps ethernet. This will either be 10Gbps ethernet or multi-lane 3-6Gbps Infiniband.

Which leads to another question:

What might Desktops look like in 5 years?

Other Reading:
For a definitive theoretical treatment of aspects of storage hierarchies, Dr. Neil J Gunther, ex-Xerox PARC, now Performance Dynamics, has been writing about "The Virtualization Spectrum" for some time.

Footnote 1:
Is this idea of multi-speed memory (small/fast and big/slow) new or original?
No: Seymour Cray, the designer of the world's fastest computers for ~2 decades, based his designs on it. It appears to me to be a old idea whose time has come again.

From a 1995 interview with the Smithsonian:

SC: Memory was the dominant consideration. How to use new memory parts as they appeared at that point in time. There were, as there are today large dynamic memory parts and relatively slow and much faster smaller static parts. The compromise between using those types of memory remains the challenge today to equipment designers. There's a factor of four in terms of memory size between the slower part and the faster part. Its not at all obvious which is the better choice until one talks about specific applications. As you design a machine you're generally not able to talk about specific applications because you don't know enough about how the machine will be used to do that.

There is also a great PPT presentation on Seymour Cray by Gordon Bell entitled "A Seymour Cray Perspective", probably written as a tribute after Cray's untimely death in an auto accident.

Footnote 2:
The notion of "all files on the network" and invisible multi-level caches was built in 1990 at Bell Labs in their Unix successor, "Plan 9" (named for one of the worst movies of all time).
Wikipedia has a useful intro/commentary, though the original on-line docs are pretty accessible.

Ken Thompson and co built Plan 9 around 3 elements:

A single protocol (9P) of around 14 elements (read, write, seek, close, clone, cd, ...)
The Network connects everything.
Four types of device: terminals, CPU servers, Storage servers and the Authentication server.

Ken's original storage server had 3 levels of transparent storage (in sizes unheard of at the time):

1Gb of RAM (more?)
100Gb of disk (in an age where 1Gb drives where very large and exotic)
1Tb of WORM storage (write-once optical disk. Unheard of in a single device)

The usual comment was, "you can go away for the weekend and all your files are still in either memory or disk cache".

They also pioneered permanent point-in-time archives on disk in something appearing to the user as similar to NetApp's 'snapshots' (though they didn't replicate inode tables and super-blocks).

My observations in this piece can be paraphrased as:

re-embrace Cray's multiple-memory model, and
embrace commercially the Plan 9 "network storage" model.

Promises and Appraising Work Capability and Proficiency

2010-05-03T10:46:00.002+10:00

Max Wideman, PMI Distinguished Contributor and Person of the Year and Canadian author of several Project Management books plus a slew of published papers, not only responded to, and published, some comments and conversations of between us, he then edited up some more emails into a Guest Article of his site.

Many thanks to you Max for all your fine work and for seeing something useful in what I penned.

Australia and the Researchers' Workbench

2010-04-18T10:44:00.000+10:00

This is a pitch for something new: the "Researchers' Workbench".
Australia has the wealth and inventiveness to do it, but most probably, not the political will.
Chalk that up to "the Cultural Cringe".

Pitch.
I.T./Computers are "Cognitive Amplifiers".

They aren't just a 'good' fit to Research, but a Perfect 'fit':

Researchers are the definitive "Knowledge Workers".

This idea isn't new or exclusive, here's what's gone before.

Theme 1: Augmenting Human Intellect.
In 1968, Doug Englebart of Stanford/SRI and his Augmenting Human Intellect Research Centre did a demo (available on-line) of their work in the area.
40 years on, this initial work has languished.
Could the ANU or CSIRO repeat the demo, even given some time? I doubt it...
http://en.wikipedia.org/wiki/The_Mother_of_All_Demos
http://en.wikipedia.org/wiki/Douglas_Engelbart
http://en.wikipedia.org/wiki/Augmentation_Research_Center

Theme 2: Bush's Memex.
Couple this with what Vannevar Bush actually proposed in his 1945 article, "As we may think", the "Memex".

This is much more than simple "Hypertext": It was a way to distil 'threads' of knowledge into accessible units, index and search them, and give/swap with others.
http://en.wikipedia.org/wiki/Memex

There is a huge obstacle in the way of implementing true "Memex" capabilities these days:

Copyright and Digital Library access.

These obstacles are trivially solvable for people working within a single institution which has paid for collective access. But much more is needed to expand the scope to a general solution for swapable "threads" incorporating copyright, digital material.

For instance, why does every University and Research organisation, or Libraries, in Australia separately license access to the many digital resources on-line?

Especially when DEST (Dept of Education, Science and Training), and ARC (Australian Research Council), who fund the vast bulk of work, could copy the "Pharmaceutical Benefits Scheme" (PBS) and negotiate single licenses for all Australian Institutions.

Theme 3: Researchers Workbench.
The world of Programming was transformed around 1973 by Bell Labs and its "Programmers' Workbench".
It consisted of a few tools, now considered staples by the Open Source project community.
The power of the work was distilling essential processes into fewer than a dozen tools, applicable to many fields.
http://en.wikipedia.org/wiki/PWB/UNIX

Where is the "Researchers' Workbench" or the studies identifying what's needed or optimal at the individual, group/team, departmental, Institutional and Subject Area level??

Theme 4: Research into Research.
Lastly, the organisation and execution of Researcher work, time and processes needs definitive investigation.
Mihály Csíkszentmihályi suggested, after studying what made Nobel Laureates different, "Flow" as a critical difference.
Is it? i.e. Has that finding been tested and proven or refuted?
Are there others?
If "Flow" is real, why isn't it well known and practised as a matter of course in Academia?
http://en.wikipedia.org/wiki/Flow_(psychology)

There have to be some simple disciplines that underpin the output of the best, most prolific Researchers.
Something that allows them to use the equivalent of Einstein's "most powerful force in the Universe": compound interest.

What techniques, disciplines or processes would allow Researchers to improve their output/performance by an significant fraction every year?

I posit that there are only a very few tools needed for a Researchers' Workbench, such as:

Shared Annotated and Ranked Bibliographies
Mind-Map representations of Knowledge Areas
a (single) definitive searchable document document repository (docs as PDF's) for selected Bodies of Knowledge. i.e. "Google Books" for academic books/journals/papers.
PDF annotation, link/reference tools, and
appropriate search and information/knowledge organisation tools & representations.

Gestalt.
Applying these approaches together (Englebart's Augmentation, Bush's Memex, the Workbench and Research into Research) would be transformative in the Academic/Research world.
Also surprisingly cheap.

Here's a simple test/question:

Is there any evidence that Microsoft Word is even a suitable, let alone effective, tool for creating Academic Papers?

If not, why it is used almost exclusively through major Universities?
What are the characteristics needed of "Perfect Academic Writing Tools"?
Why aren't these needs well known?

The world of Academia and Research is an "Elite Sport": highly competitive, intense on-going activity requiring the best/most effective training techniques of the most suited and most capable individuals. With only occasional "crystallisation" points: where performance/outcomes are unequivocally judged. (Not unlike the Olympics).

The A.I.S. (Australian Institute of Sport) proved two things:

"It takes a village to raise a child": or a large co-operative/coordinated organisation to create Great Sportspeople, and
the process can be replicated. [other countries now do it]

The A.I.S. result wasn't accidental or unplanned: They applied the "Scientific Method" to their task/goal.
What matters in producing "winning performances"? e.g. {Selection, Coaching, Technique, Training/Learning, Training Regime and Work-load, Nutrition, Mental/psych Factors, Recuperation and Recovery from Injuries}
"What Works" in each of the areas?
How do we measure and improve each area?

After around 25 years of outstanding and obvious achievement of the A.I.S., I don't understand the seemingly complete blindness of Academia/Research to applying their own methods/principles to their own work. If Human Performances can be systematically improved in Sports by applying 'Science', then isn't that enough evidence to at least trial the idea on Academia and Research itself?

Where is the Academic Discipline of "Research into Research"?
Where is the "Office for Improving Research" in each and every University?
Clearly, the simple, coarse metrics used now to characterise "Researcher Outputs" for DEST funding are not sufficient for intensive study of the Science of Research. One of the priority research areas of the A.I.S. initially was breaking down performances and the factors contributing.

Alan Kay, a noted Computing pioneer at XEROX PARC, commented "Point of View is worth 80 IQ points".
PARC managed to redefine the world of Computing/Networking in the early '70's with a small team and some very powerful organisational ideas and approaches.

I suggest that applying the "I.T. as a Cognitive Amplifier" Point of View to Academic Research in a disciplined, organised way would significantly raise the Collective I.Q. of Australian Research, and in certain fields really produce that "80 IQ point" advantage Kay suggests.

This is equivalent of Roger Bannister in 1954 not just breaking the record by 2 seconds and achieving the first "four minute mile", but beating Sebastian Coe's time of 30 years later.

Australia could "steal a march" on the rest of the world.
Once ahead, the work should feed on itself and push our Researchers further ahead.
A secondary effect is The Best Centres attract the best, brightest and most ambitious. A virtuous circle.

The A.I.S. experience shows the cost of creating "super-stars" is calculable and within the reach of a wealthy small nation, such as Australia.
It also shows there is a very strong "First Mover" advantage, that can be maintained and extended for quite some time.

The obvious starting place is Canberra, home of the Australian National University, the same place as the A.I.S.
Like the A.I.S., the program has to become trans-national, with decentralised areas of excellence and expertise.

Why now and where's the urgency?

The USA's National Science Foundation, via it's Office of Integrative Activities, is now funding projects for a new program:

Cyber-Enabled Discovery and Innovation (CDI).
... is NSF’s bold five-year initiative to create revolutionary science and engineering research outcomes made possible by innovations and advances in computational thinking.

Australians have always been very inventive and occasionally this has translated into "innovation".
We have shown we "punch above our weight" in areas where the Research Outputs can be directly applied, as suggested here...

We get one shot at being "The First" to Augment and Amplify Researcher Intelligence.
We can choose to be Leaders or Followers and enjoy the on-going consequences of either.

Death by Success II

2010-04-04T13:34:00.000+10:00

There is another, much more frequent "Death by Success" cause, first introduced to me by Jerry Weinberg and Wayne Strider and Elaine Cline (Strider and Cline).

It's the same process that some herbicides use: unconstrained growth.
Monsanto's flagship herbicide Round Up is exactly this sort of agent.

If you are very good at what you do and much sought after, this can lead directly to massive Failure - personally and in business.

Growth is Good, but too much, too fast is a Killer.

The only protection is awareness.
As Virginia Satir pointed out, "We can't see inside other people's heads, nor can we see ourselves as others see us" (courtesy again of Jerry and "Strider and Cline".)

Typically you need objective, external help is recognising this condition.
Once you have restored Situational Awareness, you can choose your response. Which may be "I'm outa here", Denial or something in between.

There is an alternative form of "Death by Success", which again we see in the Plant Kingdom.

Your initial approach, solution or technique may not Scale-Up or have a fixed Upper-Bound.
E.g. if you sell "factory seconds", there is a limited supply that sets your maximum turnover.
Or selling fragments of the Berlin Wall - at some point the Genuine Article is all gone...

The example in the Plant Kingdom are when tree seedlings 'set' in unsuitable places, like a small pot or within a bottle. Down the road, they will become "root bound", which slows growth, then they'll consume all the nutrients and having converted 'everything' into plant material, die.

That's it for that plant - all of one resource has been exhausted and it's Game Over.

Death by Success

2010-04-04T13:06:00.000+10:00

The things you do in the beginning, when you're the minnow-against-the-giants, to start and build a business may not work well when you're successful, when you've become The Giant.

Exactly what leads to Success can eventually lead to your downfall.

You become very good at the things that have gained and seemingly maintained Success. Every problem and challenge you've met have been solved with your brilliance and individual style.

Why would you ever want or need to vary that approach?

Until something new comes along and it all goes wrong:
Inevitably in Business and Life, things change (perturbations arise in Control Systems terms).
Responding with "More of the Same", as in the past, will, at some point, not work.
If you've grown large, it will take time to fail, you'll have notice "things aren't great".
Many companies only ever do "More of the Same", often amping-it-up as results don't appear.
The results are as predictable are throwing oil on a fire.

Often I mention Sydney Finkelstein's book, "Why Smart Executives Fail" in which Finkelstein describes the results of 6 years of research. He self-describes as "Steven Roth Professor of Management at the Tuck School of Business at Dartmouth College, where I teach courses on Leadership and Strategy".

In Smart Executives, Finkelstein and his team documents a whole slew of companies (50) that burned bright and collapsed. This book was published in 2003, covering a turbulent period of US and global business, as well as some famous cases going back decades.

The subjects of the research were chosen precisely because they were wildly successful and suffered a notable collapse. Enron and Worldcom are on the list, plus many I.T. companies such as Wang Computers. The common thread is the collapse was avoidable and predictable.

Would the conclusions, Lessons Learned and "Early Warning Signs" be different post the 2008 GFC (Global Financial Crisis)? I think not...

Finkelstein lists 7 naive causes of failure:

The Executive were Stupid.
The Executives couldn't have known What was Coming.
It was a Failure to Execute.
The Executives weren't trying Hard Enough.
The Executives lacked Leadership Ability.
The Company lacked the Necessary Resources.
The Executives were simply a Bunch of Crooks.

and comments in a para entitled "Failure to understand Failure":

All seven of these standard explanations for why executives fail are clearly insufficient. (Because the companies had demonstrated excellence in becoming highly successful.)

The next 300 pages are his answer. Part I describes "Great Corporate Failures" and Part II their Causes.
This research ends with a positive message, Part III is "Learning from Mistakes":

Predicting the Future, Early Warning Signs.
How Smart Executives Learn, Living and Surviving in a World of Mistakes.

His "Seven Habits of Spectacularly Unsuccessful People" are worth reiterating:

They see themselves and their companies as dominating their environments.
They identify so completely with the company that there is no clear boundary between their personal interests and their corporation's interests.
They think they have All the Answers.
They ruthlessly eliminate anyone who isn't 100% behind them.
They are consummate company spokespersons, obsessed with the company image.
They underestimate major obstacles.
They stubbornly rely on what worked for them in the past.

Each of the 11 chapters has 30-50 references. Although written and published for the general market, this isn't any "Puff piece".

MMC - the Microsoft death blow for non-Enterprise markets

2010-03-07T21:19:00.001+11:00

MMC, "Mostly Macintosh Compatible", the equivalent for OS/X of WINE for Windows, doesn't yet exist, that I'm aware of.

This comes from a personal need: on my Intel linux systems, I'd like to be able to load and run the Mac software out there, especially for things like printer drivers.

Why would MMC be a Microsoft killer?

And why only for non-Enterprise markets?

What's needed to build MMC for Linux?
What might Apple have to say?

The average "Joe User" who's price sensitive, will want to buy a commodity Intel PC, more commonly a laptop or netbook these days. If they weren't price sensitive, they would buy an Apple product and this discussion is moot. Enterprises have a very different cost-structure and set of concerns. Anyone who's locked into Microsoft will first consider moving to Apple products, not Linux. Only organisations who need a "poverty pack" solution will consider Linux+MMC. By their nature, they are difficult customers and any solution is likely to be judged "expensive".

Windows and OS/X have a slam-dunk advantage for home users: you can buy software and support for it.
Linux often fails because ISV's only make their product available for Windows or OS/X platforms - hence the interest in WINE and the commercial variant, CrossOver.
MMC would make a large number of Apps sold or available on OS/X to be used simply on Linux.
This is the potential death blow for Windows in the home: It Just Works. (with only a little configuration).

MMC is a way for ISV's to generate one package and satisfy multiple markets.
They are an obvious point of leverage and potential partners.
Would Adobe and Oracle rather try to support the myriad Linux & Unix variants out there, or meet their needs with just one (OS/X) package plus an emulator supported by someone else.

MMC would target the commercial software deficit from the OS/X, rather than Windows side.
This is significantly easier because OS/X and Linux are both POSIX-like systems underneath and the raw OS/X System calls and libraries are public via "Darwin".

OS/X also uses common software for tasks like printing (CUPS). Extracting and translating the corresponding files and executables should be straightforward.

The requirements to emulate an environment include, but aren't limited to:

CPU, support chips, hardware drivers, 32 or 64-bit, endian memory layout,
System calls,
standard libraries,
file system and package/distribution file formats,
file system and directory layout and naming,
networking,
packaging and distribution system, scripting languages, assumed tools and support libraries,
GUI calls, tools and functions.

With Windows, OS/X and Linux all running commodity IA32 hardware, many of these problems are solved.
Please be clear: MMC will not be trivial to create, nor can it ever be a perfect replica of OS/X.
Often, close, not 'perfect', is all a user wants or needs.
People who want full OS/X and are not prepared to pay Apple for a properly licensed copy, won't find an alternative with Mostly Mac Compatible. That argument is one for Apple and those people to have.

One of the hardest low-level facilities of Windows to emulate elsewhere is file I/O and networking because of the very different approaches and structures used.
Linux and OS/X both share a common heritage , POSIX, making this an already solved "porting problem".

Filesystem and package file formats have proven to be difficult for Windows to Linux.
OS/X ".dmg" files are compressed HFS+ filesystem images - already supported "read-only" in the Linux kernel. I've not looked into OS/X "packages" or ".mpkg" format.

Googling suggests there is an OS/X (Cocoa or Aqua?) emulator for Linux.

Most of the necessary components might already exist.

But Apple is litigious. This is a show-stopper. They are very protective of what they see as their competitive advantage...

How to sell MMC to them?
There's no way MMC can ever run applications faster or completely without problems on Linux. People who would've bought a Mac won't buy Linux+MMC instead.
MMC can only ever be a stop-gap or compatibility measure.

The real plus for Apple is that there will be a proportion of MMC users that will decide to "upgrade" to a Mac. That will steal sales directly from Microsoft at the high-end, where it hurts most.

Two Open Source companies, RedHat and Canonical, have the technical and marketing ability to negotiate with Apple and potentially agree licensing deals with Apple.

Apple would be a lot happier and supportive of any project that both paid them money and brought them new clients.

So, what next?? I'm not sure...

Why Microsoft is being left behind

2010-02-28T13:56:00.001+11:00

Paul Budde recently questioned, "Will Microsoft be able to make the jump?"
[04-Apr-2010] For other comments see my pieces "Death by Success" and "Death by Success II".

He quotes the marketing "S-curve" and Summer Players by Carol Velthuis describing company performance and market maturity in seasons of the year.

This is a response to Budde's query: "I have always found it difficult to understand why Microsoft has left it so late to fully embrace the new environment -i n relation to both the Internet in general and mobile broadband."

It's because they are trapped by their success.
They are not alone nor the first, as definitively documented in Why Smart Executives Fail, by Sydney Finkelstei, a Professor of Management at Dartmouth's Tuck School of Business.

Specifically:

Lack of vision and competence in Management,
Rigid and bureaucratic heirarchy,
Dysfunctional culture, and
Incompetence in their technical speciality: writing software.

Many of the problems with Microsoft's culture and bureaucracy are in my response to the Joe Wilcox piece, "Microsoft Office is obsolete, or soon will be".
17+ levels of 'management' and viscous internecine wars

On 'vision' and management competence, contrast Microsoft to the companies it sees as it major competitors: Apple and Google.

Google is actively contributing back to the community in many ways and expanding directly and indirectly into other businesses: Power, gigabit networks, community wireless networks.

Not to mention 'Android' and their phone.

Google's "Summer of Code" actively seeks new and talented people and pays them to contribute to Open Source projects in many ways.

"Google Ventures" and "Google Foundation" are some of the indirect means Google is experimenting and expanding.

Definitive evidence of the failure of Microsoft management was the money mountain (~ $65Bn) it sat on for many years. Other commentators have addressed this much better than I, so I'll be brief...

It comes down to Internal Rate of Return (IRR) - successful companies can always get a much better ROI on capital by investing in their own business than lending at market rates. How could an entire senior management team and board not think of strong new investments for even 10% of their cash reserves? That's my evidence of "lack of vision" and "poor management".

An aside: Paul Allen had the vision of "a computer on every desk". Gates and Ballmer eased him out around 1983 after Paul contracted Hodgkin's Disease. Microsoft has gone on to fulfil that vision, but has been unable to create another strong and profitable line of business.

Microsoft has very poor Software Engineering skills, going back a long way.
How it gained market dominance was boldness and great marketing.

Here's my evidence:

abandonment of multi-platform support of Windows NT,
the 2004 abandonment of "Longhorn" (the 'reset'),
the 2002 "Trustworthy Computing" initiative, and
3 yearly update cycle, not yearly.

The Longhorn 'Reset' threw away 3 years work by a team of 10,000. That was an extraordinary business action. It was also an admission that the work, the direct result of poor practice, was not salvageable. How was it ever allowed to get to that??

Apple took a big gamble in the late 1990's to abandon it's in-house developed Operating System and embrace another platform from NeXT - now called 'Darwin'. In 2001, it ran both desktops and servers.

That one codebase now supports four platforms: PowerPC, ARM, Intel IA32 and IA64.
And many product lines: desktop, server, iPhone/iPad, Apple TV and more to come.

Microsoft has different code bases for everything. The only commonality is their "Windows" branding.
They have even struggled with producing a server without a GUI. Currently, they still haven't released a fully functional server that is command-line only.

The Xbox may be a success, but the division has not regularly returned profits. Is that a line of business, or a charity?

Microsoft had a public "embracing" of Security, and continues to trumpet it success and commitment therein. But for critical code, reliability and security are fundamentals. Security to Operating Systems is like Safe Operation is to Passenger Aircraft (or ships). It's not optional.

Since the late 1980's, every vendor of Unix systems has run on a yearly update cycle, with possibly a three yearly 'major release', including those times they've changed platform.
That's been the industry standard and should be known to any competent Software Engineer and Manager.

SUN, IBM, HP and Apple have all quietly shipped new releases each and every year for the past 15 years.

So why can't Microsoft? Because they are poor at Software Engineering, as opposed to mere programming.

A final word:
Steve Ballmer has been at the helm for the last 10 years, and sole leader since 2006 (?).

Gone are the days when Bill Gates could issue a memo and have the company 'spin on a dime' - the famous Internet reversal.

ICT Productivity and the Failure of Australian Management

2010-02-27T16:18:00.000+11:00

Prior Related Posts:
Quantifying the Business Benefits of I.T. Operations
The Triple Whammy - the true cost of I.T. Waste
Force Multipliers - Tools as Physical and Cognitive Amplifiers
I.T. in context

Alan Kohler and Robert Gottleibsen have been writing in "Business Spectator" about the relationship between jobs and Economic Productivity.

They note that the USA has improved productivity in the last year while in Australia it has declined (+4% and -3% respectively). My take on this is: a gross Failure of Australian Management.

There is solid research/evidence that "ICT" is the single largest contributor to both partial and multi-factor Productivity, and is expected to be so for the next 20 years. This is an big issue.

Kohler ties this back to a Telstra survey, now in its second year.

The 2009 ACIL Tasman whitepaper notes:

While investment in ICT has boosted the productivity of workers, Australian firms, industries and the national economy, productivity levels in Australia remain below those of many OECD countries.

Kohler's big takeaway is the apparent decline in Australian CEO's ability or interest in measuring their firms' "productivity".

The current Federal Government is opening itself up to on-going criticism from senior media commentators at two levels, so others may follow:

Australian Economic performance
Public Service Administration

Whilst the use and measuring/reporting of private sector "Productivity" can only be indirectly influenced by Government (e.g. through mandating reporting in "Standard Financial Reports"), the administration of the Public Service is directly under its control.

There aren't any good reasons for Government not to address known systemic problems and/or deficiencies of its paid administrative staff. The two Telstra reports surveyed Government managers and noted no difference.

Even s44 of the FMAA makes an attempt at enshrining this.

Ignoring mounting evidence for around 10 years of the most critical factor determining output is not "poor" management, but much worse. Ignorance can't be used as a justification, these were Government reports. Which implies deliberate, wilful non-action - much worse than mere "negligence".

As one of the reports notes:

The business adage of "you can't manage what you don't measure"...

So, will the Government of the Day, act? I very much doubt it.

There is always a Great Political Crisis... period.
Waiting for "a good time" to raise important, but "unsexy", issues means deferring them infinitely.

There is a world of difference between "Urgent" and "Important" issues, well known in Management and Time-Management circles.

To achieve continuing success, a Government needs to act on important issues before they develop into Crises and are picked up by the media and used by the Opposition to undermine them. That's walking a tightrope.

What would such a crisis look like? You'd have to be a Politician to answer that...

Links and quotes:
Jan 2009 ACIL Tasman Whitepaper
http://www.telstraenterprise.com/SiteCollectionDocuments/Whitepapers/Productivity_Whitepaper.pdf

Telstra Productivity Indicator (TPI) 2010
http://www.telstraenterprise.com/RESEARCHINSIGHTS/Pages/TelstraProductivityIndicator.aspx

Telstra Productivity Indicator (TPI) 2009
http://www.telstraenterprise.com/researchinsights/Pages/TelstraProductivityIndicator2009.aspx

Business Spectator articles:

"Australia's productivity gap wider in 2010: report", 23 Feb 2010
http://www.businessspectator.com.au/bs.nsf/Article/Telstra-pd20100222-2W6KF?OpenDocument

"Jobs and debt for everyone", Alan Kohler, 24 Feb 2010
http://www.businessspectator.com.au/bs.nsf/Article/Jobs-and-debt-for-everyone-pd20100224-2XQZP?OpenDocument

"Time to halt the productivity slide", Robert Gottliebsen, 23 Feb 2010
http://www.businessspectator.com.au/bs.nsf/Article/Not-made-to-measure-CEOs-pd20100222-2W6AM?OpenDocument

"Three Kinds of Productivity", Robert Gottliebsen, 3 Feb 2009
http://www.businessspectator.com.au/bs.nsf/Article/Three-kinds-of-productivity-$pd20090203-NVUS5?OpenDocument

Other:

DCITA (now DBCDE), "Forecasting productivity growth: 2004 to 2024", Mar 2006

There is a broad consensus emerging in the economic literature that ICT has been a major driver of productivity growth in developed countries in the last couple of decades. Recent overseas economic studies that highlight the central importance of ICT in driving productivity growth include David (2000), Schreyer (2000), Stiroh (2001), Colecchia and Schreyer (2001), Jorgenson et al (2002), Oliner and Sichel (2002), Bassanini and Scarpetti (2002), Gordon (2003), Inklaar et al (2003) and OECD (2004a).

Australian macro-productivity studies that point to the same conclusion include DeLong (2000), the Ovum Report (2002), Diewert and Lawrence (2005a and 2005b), NOIE (2004) and DCITA (2005a and 2005b). Parham (2004a) agrees that ICT had a major influence on productivity growth but places greater emphasis on micro-economic reform. Parham (2004b) recognises more clearly the importance of ICT but adopts a ‘semi-sceptical’ position.

Judging by recent technological futures studies, ICT will remain a major driver of productivity growth in advanced industrialised countries in the next two decades. An important contribution of ICT to productivity growth will be in improving automation in virtually all areas of the economy including offices, factories, shops, warehouses, schools, hospitals, farms and mines and in facilitating information search and flow.

"ICT Use and Productivity: A Synthesis from Studies of Australian Firms", Productivity Commission Research Paper, 2004.
Parham, D., Roberts, P. and Sun, H.

Executive Summary: Scope and purpose
The role of information and communications technology (ICT) in promoting productivity growth is an issue that has attracted particular attention not only in Australia, but also overseas, in the context of the ‘new economy’ debate.

Whilst it is now generally agreed that the use of ICT has a positive influence on productivity growth — at least in industries that use ICT intensively — the observed differences across countries in the extent of ICT uptake and related productivity effects have continued to be a puzzle.

Summary
There has been strong growth in investment in IT in Australia since the mid-1990s. IT investment reached over 15 per cent of total annual investment in the market sector in 1999-00.

The concomitant increase in the productive IT capital stock made a sizeable contribution to output and labour productivity growth over the 1990s.
•    IT contributed nearly 40 per cent of market sector output growth (averaging 3.4 per cent a year) from 1989-90 to 1999-00.
•    IT capital deepening contributed around 35 per cent of the record 3.1 per cent a year growth in labour productivity from 1993-94 to 1999-00.
•    IT capital deepening contributed about 45 per cent of the 1.1 percentage point acceleration in labour productivity growth in the 1990s. However, this was offset
by negative capital deepening based on other types of capital. With no net contribution from overall capital deepening, faster MFP growth accounted for virtually all the faster labour productivity growth.

Compared with the US, Australia showed:
• a higher labour productivity and MFP acceleration; and
• a similar contribution to labour productivity growth from IT capital deepening, even without making allowance for the absence of communications equipment in the Australian estimates.

Australia has generated MFP gains from IT use and/or from non-ICT related factors that, in comparison with the US, have more than offset the absence of MFP gains from an ICT production sector.

There is no clear, strong relationship between ICT use and MFP growth across industries. In Finance and Insurance, the relationship is clearly positive. Whilst the relationships are not as obvious in other industries, they may nevertheless be there.

Evidence from other studies suggest that the direct gains from IT use are only part of the story. Complementary changes in organisation, management and work practices can be important in tapping the full efficiency gains that IT can offer over the long term. Non-IT factors can, of course, also be at work.

Microsoft Troubles - VIII, MS-Office challenged

2010-02-11T13:11:00.215+11:00

"Microsoft Office is obsolete, or soon will be" By Joe Wilcox.

I hadn't picked this trend, it's quite important.
It squeezes their 2nd "birthright" (the other is the PC Operating System, I'd focussed on.)

Microsoft achieves around 90% "Gross Margin" on two product lines (O/S and Office), and these lines carry the rest of the business. There is also the "Enterprise" division: they leverage these two product lines and add server software like O/S, Exchange and SQL Server.

Perhaps Office 2007 was the zenith, even for Enterprises?
Microsoft has never embraced the idea that products can be complete.

To go forward, you can't add more frills.
You have to completely reinvent/redefine the product or the field - like the iPhone.

I can't do better than Wilcox's article - I suggest you read the link.
For a quicker read, I've picked out what I see as the most salient points.
There are two very good paras at the start and a cute last one. [Italicised]

To draw out the financial problem with "Gross Margin" - successful companies attain 20-30% in the long term - here's a small calculation using the Microsoft 6-month results for Q2-2010 (to 31-Dec-2009).

You'll note we aren't given per-product break-downs, only Divisional (Microsoft uses the word "Segment" or "Channel".). From a straight business perspective, 3 divisions account for 80+% of the Revenue, and 34% of the Expenses. If Microsoft killed the other Divisions, they'd improve their bottom line 11-fold. Which you'd expect them to do if pressed hard.

Obviously they have important divisions, (Research and Development, Corporate Admin) that aren't tied to specific Revenue streams. They would be subject to cost-cutting as well.

Division	Revenue	Expenses	Profit	Gross Margin

Windows & Windows Live Division	$9,528M	$2,674M	$6,854M	72%
Server and Tools	$7,278M	$4,511M	$2,767M	38%
Microsoft Business Division	$9,149M	$3,282M	$5,867M	64%

Sub-Total	$25,955M	$10,467M	$15,488M	60%

Consolidated	$31,942M	$30,643M	$1,299M	4%

At 90% Gross Margin (GM), each $100 of Revenue is matched with $10 of Expenses, yielding a Profit of $90. At 20% GM, each $100 Revenue costs $80 to raise.

If Inputs (Expenses) for the same Sales Volume remain the same, but the Gross Margin drops from 90% to 20%, then $80 in Inputs that once yielded $800 Revenue, now only yields $100 Revenue and $20 of profits, not $720. Ouch!

So what would that look like in the Income Statement?

Division	Revenue	Expenses	Profit	Gross Margin

Windows & Windows Live Division	$3,450M	$2,674M	$776.3M	22.5%
Server and Tools	$5,639M	$4,511M	$1,128M	20%
Microsoft Business Division	$4,376M	$3,282M	$1,094M	25%

Sub-Total	$13,465M	$10,467M	$2,998M	22.26%

Consolidated	$19,412M	$30,643M	-$11,230M	-37%

Reflecting on that scenario, it might take 5-10 years to slide down that far.
What's more likely in the next few years? The Business Division and "Servers and Tools" are going to slide a bit, maybe Windows will halve it's gross margin:

Division	Revenue	Expenses	Profit	Gross Margin

Windows & Windows Live Division	$4,178M	$2,674M	$1,504M	36%
Server and Tools	$6,444M	$4,511M	$1,933M	30%
Microsoft Business Division	$3,282M	$3,282M	$3,282M	50%

Sub-Total	$17,031M	$10,467M	$6,719M	38.6%

Consolidated	$23,018M	$30,643M	-$7,625M	-33%

To remember that these things happen quickly and with high impact, here's what the NY Times wrote in early 1993 about IBM, their crisis still to run for a year:

For the quarter, the company reported a loss of $5.46 billion on revenues of $19.56 billion. In the quarter a year ago, I.B.M. posted a loss of $1.46 billion on revenues of $21.97 billion.

For the year, the company reported a loss of $4.97 billion on revenues of $64.5 billion. In 1991, I.B.M. posted a loss of $2.86 billion on revenues of $64.8 billion.

The full-year loss of $4.97 billion by I.B.M. is the largest in American corporate history.

Before they controlled the haemorrhage, IBM had lost $20Bn between 1991 and 1993.
What finally saved the company was the board replacing John Ackers with an industry outsider, Lou Gerstner.

Unisys has shown that large, badly managed I.T. companies riven with internal politics and dysfunction can survive on their own for decades.

Declining Revenues isn't necessarily the end of iconic brands.

This exercise in Gross Margin suggests that if Microsoft maintains the same Sales Volume and cost structures and ditches unprofitable Divisions, it's Revenues might more than halve, and it could increase it's profitability.

But it has to concentrate on its core business or face failure.
Unisys and IBM provide salutary lessons on what a board has to do and not to do.

Extracts:

This month's Office 2010 retail pricing announcement and ongoing discounts for Office 2007 Home and Student are Microsoft's tacit acknowledgment that the productivity suite isn't as valuable as it once was.

Office is tracking a course of unplanned obsolescence and the inevitable end shared by oh-so many other products: Commoditization.

Additionally, Microsoft faces a fundamental shift in what content people create and where.

Commoditization and the emerging mobile device-to-cloud services applications stack are Office killers.

I'll ask upfront: Do you really need Microsoft Office on a daily basis?
Is Office vital to your work day?
Do you use it at home?
If you use it at work, how often?
If you use it at home or for college, how often?
But in a Web-connected world, Office's value diminishes.
The pressing question: How low can Office's value go how soon?

For years, Microsoft has kept Office pricing fairly stable, mainly because of monopoly's power.
Since Office vanquished WordPerfect in the mid-1990s, no productivity suite could compete. [and competent alternatives have arisen since. SJ]

Word processing reached commodity status years ago, as more applications incorporated the basic formatting features most people use more than 90 percent of the time.

Be honest, how much of the writing you regularly do requires a dedicated wordprocessor?

In a January 2008 Microsoft Watch blog post I [Joel] asked "if Office 2007 would be a one-hit wonder"?

Reasoning: With the Ribbon interface, v2007 will be good enough for most businesses to skip next Office version or the one after.

The new applications stack, which is outside of Microsoft monopolies, is mobile device to cloud service.
[on content types] ... Blogs, photos, videos, tweets and social network postings, among others.
These content types have little or nothing to do with wordprocessing, spreadsheet or presentation applications.

The mobile handset market dwarfs the PC market by 4.6 times.

IDC asserts that the mobile Internet is only 450 million users, which is expected to top 1 billion by 2013.

PEW Internet predicts that by the end of the decade cell phones will replace PCs as primary Internet devices.

As the mobile Internet install base increases, natural user interfaces
will further supplant productivity suite functionality.

For me, Microsoft Office already is obsolete.
The question remains: When will Office be obsolete for you?

Microsoft Troubles - VII, An Insiders View

2010-02-06T10:12:00.001+11:00

A friend sent me this link to a New York Times Op-Ed 'contribution'.
Huge news...

February 4, 2010
Op-Ed Contributor
Microsoft’s Creative Destruction
By DICK BRASS
Dick Brass was a vice president at Microsoft from 1997 to 2004.

This guy was a VP in the glory years - either side of Y2K, and before the 2004/5 Longhorn 'reset'.
The failure to build the successor to XP was a breaking-point: the forced upgrade cycle was gone.

He's likely to have a bunch of stock, or options, and a vested interest in the company's success/survival. His comments are likely to be both informed and as positive as they can be...

My summary:

Microsoft, he says baldly, "is struggling", "And yet it is failing, even as it reports record earnings."

"Microsoft has become a clumsy, uncompetitive innovator."

Despite many efforts, fine engineers, huge R&D and many "visionaries", it hasn't created anything substantial to replace it's two franchises/cash-cows (MS-Windows & MS-Office) - designs dating back over 2 decades, "venerable (old) products".

"Unlike other companies, Microsoft never developed a true system for innovation".
On 'ClearType' a major innovation of his, "a decade passed before ... (it) finally made it into Windows."

Microsoft, he notes, is beset by "internecine warfare".

Marketing has failed because of "understandable caution" and their "Timing has also been poor".

"Internal competition ... (has become) uncontrolled and destructive. At Microsoft, it has created a dysfunctional corporate culture..."

The best people have been driven out - "It’s not an accident that almost all the executives in charge of .... over the past decade have left." and "There has been a steady exit of its best and brightest."

"Perhaps worst of all, Microsoft is no longer considered the cool or cutting-edge place to work."

Commentary

Microsoft suffers failed Management, ineffective Marketing, dysfunctional and strife torn 'culture' and is milking decades old cash-cows, too afraid to take risks or really innovate. It can't Execute and is stifled by bureaucracy and internal politics.

The 'crush all opposition' culture/mindset has rebounded and been transferred from external to internal 'competitors'. An aggressive approach that grew the company to "an accidental monopoly", has turned inwards and now devours itself. Ugly and sad...

When someone both informed and sympathetic is mourning their decline and fall, the prospects can't be good...

Ballmer is celebrating a decade as CEO (since Jan 2000). If he was able to recognise and address the problems, he would've done so by now.

Bill Gates issued his "Internet Tidal Wave" memo in May 1995, and within six months it launched MSN and all Microsoft products were "Internet enabled".
Fifteen years ago they were agile and able to Execute, capable of fast and radical change.

The "flat decade" and stagnation/implosion of the organisational culture since 2000 can be laid solely at Ballmer's feet...

The questions left hanging are:

Will firing Ballmer be enough?
Can a strong Microsoft be salvaged from its Death Spiral?

Unfortunately, there is probably a perfect example of what's in store for Microsoft:
Unisys, the 1986 merger of Burroughs and Sperry/Univac (numbers 2 and 3 behind IBM), has declined four-fold, from $10.5B revenue and120,000 employees to $5Bn and 30,000 in 2010.

IBM expanded its 1986 revenues from $51B to $100+B in 2009 - roughly in-line with inflation, whilst surviving 2 years of massive losses in 1991/92.

I worked for Unisys on a contract in the mid-90's. A critical business system of a major Government Department was in deep trouble and they were threatening to throw them out - they were on "Worldwide Alert". It reminded me of a line from the film "Broken Arrow", it's not that has happened, but that you have a term for it (i.e. many times).

The decline was apparent then and the reasons blindingly obvious: Failed management and intense 'internal politics'.

Conclusion

Please be clear: I am not saying "Microsoft will suddenly collapse".
Like Unisys, I expect the brand, if not the company to linger on for many decades yet.

In 2006 I started saying Microsoft would "hit a financial pothole around 2010".
I believe their bottom line, profit not revenues, will collapse.
When that happens the 'golden glow' around them will be shattered, herd effects they've previously gained from ("Everyone is using Microsoft"), will work against them. The herd will rush to embrace the Next Big Thing, so the cycle will repeat.

What happens next is strictly up to Microsoft Management.
The company can be destroyed, but will it be?
It depends on what happens inside Microsoft, how 'loyal' (change resistant) customers are and how aggressively the competition responds.

The brand and products will survive: There are enough successful I.T. companies whose business model is to buy 'mature products' at the end of their lifecycle, then raise prices and do the absolute minimum of support. Customers are leaving anyway, how can they become more disgruntled? If people are locked-into your product and unwilling to change, you're 'providing a valuable service' by keeping their necessary software alive... Financial blackmail? No, 'just good business'.

I do think Microsoft can still be technically innovative and could come back and be competitive.
I just don't think their mindset allows them to do what's necessary: follow Apple's lead and embrace software standards. This, and 'Great Design', is Apple's 'secret sauce', not 'doing hardware'.

Murdoch, Google and Microsoft

2009-11-24T10:38:00.000+11:00

Rupert Murdoch anti-Google strategy seems to becoming clear. Previously, I was puzzled at his public stance, it seems to make no sense.

This Register piece, and there are many other sources, starts with:

Rupert Murdoch is in talks with Microsoft over his plans to delist his newspaper websites from Google.

Classic "My enemy's enemy is my Friend" thinking.
It's not really so, they are only a temporary ally at best with a fragile common cause.

The danger lies in your "friend" deciding they don't need you anymore or worse, turning on you once the main enemy is gone or the battle is lost.

I still think Murdoch's strategy is seriously off the mark.

I.T. Failure == Corporate Failure

2009-11-20T08:49:00.000+11:00

Stephen Bartholomeusz writing in Business Spectator, 18 Nov 2009, on the ASIC court case over the collapse of One.Tel.

Bartholomeusz neatly summarises the root cause of the failure:

Unhappily, its billing systems didn’t work, so it piled up debtors, while its competitors responded to the cut-price strategies.

He goes on to say:

While professing publicly that the group was on-track to be cash-positive..., internally One.Tel appears to have had little control or understanding of its cash flows or the mounting issues created by its billing system

and finishes:

Whatever Rich might claim, One.Tel wasn't a successful company, unless success is measured by revenue, not cash flows or execution.

This is the first case I've noticed where the immediate cause of failure of a large, public company has been it's I.T. systems. The root cause is poor management with an inability to execute - or to understand and control it's I.T.

The field of "Software Engineering" is 40 years old now.
How could this foreseeable and preventable failure have happened with competent professionals, especially if Software Engineering had achieved it's aims?

There is a multiple tragedy hidden here:

Software Engineering has failed to impress it's primary market: Business Management.
Educators and Researchers are not, as a matter of course, going to analyse this failure and use it as a case study. Compare the 1974 explosion at Flixborough or the 1970 collapse of the Westgate bridge during construction.
IT practitioners aren't going to be informed by their Professional Societies of the causes and preventing a recurrence.
Business Management and I.T. practice remains "Consequence Free".

If a Billion Dollar Failure isn't a notable event and worthy of preventing recurrence, then what is?
Why are ASIC, the ASX and the Federal and State Governments silent on this point?
If not their job, then whose?

Imagine if QANTAS had a fire at a maintenance facility and lost $1B of buildings, plant and equipment. You know absolutely the company, multiple regulators and all the professional bodies would actively investigate the matter.

They would be looking for "root causes" of this event, other problems, ways to fix the system, processes & procedures to prevent or early-detect this class of problem again and co-incidentally if any individuals were responsible. Not just front-line grunts, but if anyone in management (up to the CEO) was culpable, negligent, incompetent or asleep-at-the-wheel.

The absolute tragedy here is not the loss to these investors (employees, vendors, customers, ...) but that nothing is going to change, that this massive loss bought nothing.

What is more galling to me is that nobody in the Press, Government, ASX, Investment bodies, Judicary or Regulators thinks anything more could or should be done...

Microsoft Troubles - VI, First Words

2009-11-20T07:40:00.000+11:00

"On the latter, Microsoft is hoping Windows 7 will pull it out of a financial hole" by Charles Arthur, Sydney Morning Herald, Nov 5, 2009. First time I've seen in popular press the actual words "financial hole" w.r.t. Microsoft.

Previous piece - "Microsoft Troubles V"

Why Yet Another ReOrganisation won't improve the Public Service

2009-09-05T13:17:00.005+10:00

The Rt. Hon. Ken Rudd PM has suggested on the News that he'll be seeking to improve the Federal Public Service. There's talk of a special Centre at the ANU to train people up too.

Rudd might end up with a bunch of tests, metrics and new programs & processes, but I can guarantee it won't amount to a hill 'o beans. The one thing known about Bureaucracies is their ability to Resit Change.

Read C. N. Parkinson ("Parkinsons Law" etc) for a view from the 1950's and some definitive economic analysis of the ultimate Bureaucracy: The UK's Ministry of Defence. After WWI, ships and fighting men - the essence of the Navy - declined dramatically. The Bureaucracy 'running' them increased overwhelmingly...

Why? Because the primary purpose of Bureaucracies is themselves, not producing outcomes.

The 38 episodes of the BBC's "Yes Minister" from the 1980's are a timeless tutorial for budding bureaucrats, Public or Private sector.

Wind the clock back a 100 years when Gilbreth and Taylor pioneered "Scientific Management", or in todays' language "Evidence Based Management": sound, practical and carefully researched approaches that produced outstanding results while treating employees as Real People. It is notable for both not being embraced and being mischaracterised as 'evil'. The book is on-line and a revelation to anyone willing to read a little.

Parkinson stated his First Law in 1957 as:

Work expands so as to fill the time available for its completion.

His 1960 Second Law stated:

Expenditure rises to meet income.

Which might be an interesting topic for Dr Ken Henry, Secretary of Treasury, to address in his Review of the Taxation System...

Have either of these hypotheses been disproved in the last 5 decades?
No.

In 1970, Parkinson published a 10-year follow-up on his work. The book had sold impressively, but had anything changed? Not discernibly.

Has Rudd proposed anything that recognises or addresses the central problems?
Not in public...

The people who rise to the top of Bureaucracies, Public or Private sector, are very good at what they do. This is exactly why we had the Enron, Global Crossing,e tc collapses around 2000 and within the decade, Sub-Prime Meltdown and GFC. Senior Bureaucrats overwhelmingly look after themselves and their own positions, not their stakeholders. It's the inverse of Fiducary Duty.

The trouble is What they do, not finding & training highly competent people.
There's over a century of ignored Management Science to prove the point.

What would work?

Consequences, direct and personal, for poor performance.
This needs two separate parts: an expert, independent investigator and a powerful compliance & enforcement body. Behind this needs to be a repository of Known Faults, Failures and Errors. Repeating a Known Problem without Consequence is the antithesis of Good Governance.

The Government knows very well how to do this and exactly what would be required: it already does precisely this for Aviation. The ATSB (formerly BASI) and CASA (formerly CAA).

It'd cost about the same to run as the National Audit Office, indeed, the Investigator would be the ANAO.
The compliance and enforcement powers, and the body to enact them, already exist in the FMAA - Financial Management and Accountability Act. s44 states:

(1) A Chief Executive must manage the affairs of the Agency in a way that promotes proper use of the Commonwealth resources for which the Chief Executive is responsible.

"proper use" means efficient, effective and ethical use that is not inconsistent with the policies of the Commonwealth.

All the machinery is there...

Will politicians actually hold their Senior Bureaucrats to Account?

No.

To do so would deprive pollies of their core modus operandi:

arbitrary and capricious decisions and policy changes.

There cannot be lasting Bureaucratic Reform without Political Reform.

But we'll have a lot of "colour and movement", many press releases & back-thumping and spend a lot of money achieving precisely nothing...

Here's something from the end of 2007 which has never been acknowledged...

When 'ping' fails

2009-08-25T08:27:00.002+10:00

In Networking, the 'performance objects' are links, usually end-to-end, consisting of many elements - like ethernet segments, switches, routers/firewalls, long-distance circuits and security scanner devices, laid on top of Telco/backbone services that provide dynamic and asymmetrical routes.

The most frequently used measure is 'ping' time - ICMP “echo request” packets, often 64 bytes long.
Firewalls, routers and underlying networks filter/block classes of traffic, implement traffic & flow rules and attempt "Quality of Service". There are many common rulesets in use:

blocking 'ping' outright for 'security reasons'. Stops trivial network scanning.
QoS settings for different traffic classes. VoIP/SIP gets high priority, FTP traffic is low, your guess on HTTP, DNS, time (NTP) and command line access - SSH, Telnet, ...
traffic profiling on IP type (ICMP, UDP, TCP) and packet size (vs link MTU).
traffic prioritisation based on source/destination.

Real-time voice needs many small packets, would like low latency/delay, and no jitter, and can stand some packet loss or data errors. FTP relies on TCP to detect packet loss & retransmit them. It likes big packets and attempts to increase bandwidth consumed through TCP 'fast-start' etc.

The only time 'ping' is accurate is within a simple ethernet segment - no rules, no firewalls, no 'traffic engineering', no link losses, no collisions, ...
Otherwise, it's a dangerous and potentially very misleading measure.

'Time of Flight' for UDP & ICMP packets is only measurable when you control both ends of the link and can match headers. Not something most people can or want to do.

TCP packets can be accurately timed - sessions are well identified, packets can be uniquely identified and they individually acknowledged. It is possible to accurately and definitively monitor & measure round-trip times and effective throughput (raw & corrected) of links and connections at any point TCP packets are inspected - host, router, firewall, MPLS end-point, ...
I'm not aware of this being widely used in practice, but that's a lack of knowledge on my part.

This is not a tutorial in Networking or TCP/IP.
Neither am I a Networking expert. I'm demonstrating that even with my knowledge, "tools ain't tools" (meaning not all tools and methods are equal),
and that using just 2 metrics, 'bandwidth' & 'latency' to characterise links is simplistic and fraught. As professionals, we have to be able to "dig under the covers" to diagnose & fix subtle faults.

Consider the case of TCP/IP over a noisy satellite phone link, the type you buy from Intelsat for ships or remote areas. The service notionally delivers a data channel of 64kbps, but is optimised for a digital voice circuit, not IP packets. The end-end link has per-bit low latency on/off the link, long transmission delays, nil jitter and limited error-correction (forward-error-correction (FEC), no retransmit) - nice for telephony. These links also have buckets of errors - which voice, especially simple PCM, easily tolerates & can even be smoothed or interpolated out with simple equipment - which will be there to handle echo cancellation.

Say you're on a ship at the end of one of these links.
People are complaining that email 'takes forever' and they can't download web pages.
You run a ping test out to various points - and everything is Just Fine.

What next?

The most usual response is 'Blame the Victim' and declare the link to be fine and 'the problem' to be too many people sending large messages, 'clogging up the link', and too much web surfing. You might set quotas and delete large emails. That might work, or at least improve things marginally.

Radio links, especially back-to-back ones, each crossing 36,000km to geostationary satellites, are noisy.
If the BER is 1:100000 (1:10^5, under the hoped for 1:10^6) and you're using the default ethernet MTU of 1500 bytes, you'll get an error every 1-2 seconds. No worries, eh?
1500 bytes = 12,000 bits = 5.3 packets/second. Or 1 in 10 packets. Hardly noticeable.
TCP/IP has a minimum overhead of 40bytes/packet (less with Van Jacobson compression).
The data payload per packet is 1460 bytes for the ethernet default MTU.

Sending a 1.25Mb file (10Mbit), that's ~856 raw sent packets and 103 errors, or ~12% retransmissions. Of those 103 resends, 12 get errors as well and are resent, and 1.5 errors of those go onto a 3d round...
Or ~115 errored packets, or 14% errors on raw packets. Just a minor problem.
There's a probability of >1 error per packet, but I don't have the maths to solve for that.

The effective bandwidth (throughput) of the link, using 970 * 1500-by packets to move 1Mb in 182 seconds, is 55kpbs. Quite acceptable.

What if there's a corroded connector or tired tracking gimbal and you get a 3db change in SNR and the BER doubles? (That's a guess, not science.)
856 raw packets and 250 1st round errors, 62 2nd, 15 3d, 4 4-th, 1 5th = 333 resends. An almost 50% increase in total number of packets needed to send the file. 223 seconds and 45kbps.
Doubling the BER again (4:10^5 or 0.004%) increases the 1st round error rate to 400 packets, or 47% - 750 retransmits in 10 rounds. 301 seconds and 33.1kpbs. Half-speed.

Back to the original BER, if you were running 'jumbo frames' (9000 by) locally & these went down the link as is, you get 0.8 packets/sec and have a 72% chance of an error in a packet. One in four of the packets would get through unscathed. 140 'jumbo frames' are sent raw, 350 packets are needed with 16 rounds of retransmission.
The file takes 400 seconds at 25kbps - a hefty penalty for forgetting to configure a switch.

The problem is that packet size amplifies error rates.
A change in BER of 0.001% to 0.004%, undetectable by the human ear, halves the throughput fo TCP/IP.
Using an MTU size of 168 (128 data + 40 TCP/IP overhead) gives good performance at a BER of 1:10^4, trading 25% protocol overhead for link robustness.

'ping', using default 512 bit packets, won't detect the error.
But who'd think the MTU was a problem when standard diagnostics were reporting 'all clear'?

Summary: In difficult conditions, the BER doesn't have to drop much for link throughput to significantly degrade.

This example is about simple-minded tools and drawing incorrect conclusions.
The Latency of the link was constant, but the Effective Bandwidth, throughput, changed because of noise or link errors.

Surely that proves the Myth: Latency & Bandwidth are unrelated.

Nope, it proves that link speed and throughput bear a complex relationship to each other.

If you had been measuring TCP/IP statistics (throughput & round-trip-time) at the outbound router, or using 'ping' with MTU of 1500, you'd have seen the average latency rising as throughput dropped. All those link-layer errors & subsequent retransmits were causing packets to take longer.

But a simple low-bandwidth radio link isn't the whole story.
It's a "thin, long pipe" in some parlances.
What was special about that link was:

no link contention, data rate was guaranteed transfer rate.
synchronous data

Rupert Murdoch - Fool or Genius?

2009-08-07T13:37:00.010+10:00

Does Rupert Murdoch know something the rest of us don't?
The recent news is that News Ltd would start charging for on-line access to its newspapers.

Not a good idea.
Experienced Journalist & commentator, Alan Kohler also thinks so...

First, Rupert is not in the business of selling 'news', quality journalism or not.
He sells Advertising.
Just like Google and friends. But apparently not nearly as well as they do on-line.

There are people who sell 'news', and they are going strong.
Organisations like Reuters, Associated Press, Bloomberg, AAP, ... The wire-services.
The same ones that sell to Google, businesses, TV and Mr Murdoch's newspapers.

News Ltd doesn't sell journalistic content (news): like every major newspaper, it has always given away its content. Exactly the same as Free-to-Air radio and TV.

The "value proposition" to most newspaper customers, News & Stories, is a Free Good.
Major papers actually cost their publishers to sell. Newsagents typically keep the full "cover price" of the local major papers. Perhaps this is why Fairfax Ltd lists "Newsprint and Ink" as its single biggest expense.

Publishers make their money from the advertising they sell (Classified and 'Display' or general).
They set their advertising rates on the estimated number of readers - not copies sold/distributed. (There's a whole industry 'auditing' circulation & readership).

Small advertisers will always be 'price-takers', while the large regular advertisers can negotiate.

On-line breaks many/all the Newspaper assumptions:

no intermediaries with good 'passing trade' to find customers
exact counts, not estimates, or readership
exact counts of advertiser hit-rates (count links followed)
targeted/niche audiences, not "broad spectrum" mass market

Browsing the 1200+ entries for 'newspapers' in the Australian Yellow Pages, these groupings seem apparent:

Business, Trade & Industry, Lifestyle, Sports, Political, Special Interest, Community/Local, Regional & Rural, Ethnic, Language and Religious,

and versions of the "Trading Post".
A newspaper without content, pure advertising, the ideal for the business side of newspaper.

Second, a long time ago newspapers were the source for capital-N News - timely, important, factual.
They broke stories, 'scooped' one another, had many editions during the day and dealt in "the facts m'aam, just the facts" as Joe Friday might say.
The sort of thing shown in 1930's Black and White movies.

By the 1970's, newspapers had comprehensively lost the race as the first news source.
"Watergate" showed they could still 'scoop' other media with investigate journalism, but the Vietnam War played out on the nightly TV news.

When Ted Turner started CNN, the game changed - Free-to-Air was usurped.
The 1991 Gulf War had CNN "reporting live from Baghdad" and assumed the mantle of "first news source".

These days it is a tussle between Cable News and on-line services to be "first".
And that race has always led to problems with accuracy and false/fabricated stories.

An editor who is under pressure "to be first" can be manipulated into publishing without good fact checking. When there was considerable effort & expense in rolling the presses, the downside ensured more caution. In the on-line world, nearly all barriers to production are eliminated alongside "instant" publication. An editorial mistake is much more likely and potentially much more damaging to a large publisher, the Drudge Report non-withstanding.

Newspapers have been providing Opinion & Analysis for a couple of decades.
Any pretence they are cutting edge or breaking stories in real-time is a "fools paradise" and delusional.

News Ltd has great content produced by many great people and serves a faithful cohort of consumers. It just isn't 'news' they are selling.

Third, there are many good free alternatives for news, on-line and not, to newspapers.

Google pays for wire-services and gives away the content.
Publicly funded media - radio, TV and on-line - have a mandate to provide services with public monies. The BBC and Australian ABC have large news rooms and international reporters.
The ABC alone has 700 people in its News Division providing current content for all its outlets.

How does a newspaper, with at best 300 journalists, compete with a better resourced competitor who's content is free?

Not on news - only with other types of content and other incentives - like DVD's and special offers...

Fourth, there are just 3 workable Revenue Models.
Revenue options are:
subscription/donations and cover-price + advertising.
(pre-paid vs 2-part charging)

There are only 4 Revenue Models possible in this scheme.
Fully free can't self-support itself, so there are only 3 workable models.

Murdoch is complaining the Revenue Model that has worked well in the physical world for approaching a century doesn't work on the Internet. Who'd have thought?!

The wire-services thrive and on-line advertising is booming.
The only people out of step are the firms running newspapers.

They could have acted in 1995 to move their advertising on-line, but didn't.
Were they blinkered or lacked 'vision'?
Was it a sound business decision based in part on not canabalising their main cash flows?

Things are how they are...
There seems little to be gained from now analysing the reasons for non-action.

There are other issues that have to be resolved when moving to on-line services.

Paper is simple and always "Just Works', modulo getting wet.
Attempts to stream printed news electronically have been widely successful outside of offices. Radio serves the travelling public well. Printed media is cheap, available and can be forgotten without dire consequences. Some section of the population may read the news on their Kindle or iPhone on their morning commute, but it won't be a large audience.
Neither will there be much call for $10 newspapers...
Serving "The Diaspora": An important function of newspapers is allowing non-resident locals to "keep in touch with home". Australia shows that people may permanently emmigrate and never return home, but still identify strongly with their country of origin. This fuels our strong ethnic newspapers. For people who've only moved towns, a daily or weekly "fix" of their hometown newspapers fills a strong need. They even pay a premium.
Niche buyers. Most buyers throw most of a newspaper away. They are very specific & selective in their needs and uses of the massive content provided. There are better ways to serve many of those niches on-line. Like classified advertising is better served by e-bay and 'trading post'. It's fast, current and cheap - plus very efficient for the reader. The service does the searching and the reader can be contacting a seller within minutes of loading the site.
Network effects and the tipping point. When a product has reached around 40% market penetration, it 'suddenly' becomes popular and quickly saturates the market. This happened in 1984 with Group-3 fax and then around 1996 with The Internet/World Wide Web. Newspapers need to be keenly aware of their competitors - when the end comes, it may be frightenly fast.
Copyright and Libel Laws. The journalists union has spent a very long time negotiating what rights the publisher & content-creator have. This all has to be done again in an on-line world. The other side of the coin is commercial protection of journalists against Libel or defamation actions. The publisher wears the risk once the editor decides to print. Those named know that a newspaper can afford to and will defend itself. If journalists are personally exposed to litigation, justified or not, they will sensibly withhold contensious pieces. Why wreck your life for a decade or more, as happened to Chris Masters over the "Moonlight State" and other pieces? For many, the price is too high.

Lastly, What would work?

This is an argument in three parts: as a society we need 'quality journalism', news rooms aren't cheap, and are there models we could follow?

The media as "The Fifth Estate" is an important and necessary part of any Democratic government. A Free Press is a necessary part of Open and Transparent government.

But whither Investigative Journalism. There is a lot of TV reportage of politicians doing 'door stops' or in stage-managed events. And a lot of 'tabloid journalism' on TV.

Would Woodward and Bernstein now be funded for their lengthy Watergate investigations?
Would any editor allow it to be published these days?
I think Watergate is less likely to be reported these days for many reasons and the Drudge Report and other gossip sources do not fill the gap.

"Quality Journalism" has to be nutured & supported for us to have stable, prosperous societies.

The ABC states it has 700 people in its News Division.
On-line sources suggest major newspapers have ~300 journalists in their newsrooms. [This information isn't in the Annual Reports I scanned.]

What would it cost to run such a news room?
$30M a year in wages, $10M in wire-services, $10-15M for bluidings and systems.
Marketing & Sales probably $30M. Accounting and collecting subscriptions: $5-10M.
Publishing on-line would add another $20-30M, with an overall 30% Gross Margin required to fund upgrades, depreciation and dividends.

Perhaps $150M/year in revenue, or $3M/week.
The Sydney Morning Herald has an audited circulation of 210-360,000 and readership from 850,000-1,100,000 [without SunHerald, $1.80 and 480,000/1.25M]
Previous comments: "Internet Changes Everything: Newspapers".

Even if you achieved 500,000 individual subscriptions, a weekly price of $5+, versus the $1.40/weekday and $2.40 Saturday for the paper version.
I believe that's far beyond the consumer 'price point' for a single publication.

So what models are out there that might work?
'Cable TV' provides content aggregation, common marketing services & subscription and billing.

It has severly impacted Free-to-Air TV over the last 4 decades for many reasons.
One of the big factors I believe is allowing content providers to focus on their strengths and the Cable Service Provider (Foxtel in Australia) to focus on the technical and retail/customer relations and support business.

The public are offered content aggregated into affordable and desirable 'packages'.
They can decide the utility to them of each package and compare the cost to other forms of entertainment. A$30-$50 seems to be the price point.

Cable TV for content providers removes barriers to entry and avoids competition between technical delivery methods. The customer wants the service and isn't interested in the technology per se. This model allows & promotes small, new entrants with serving specialist or highly targeted niches.

The revenue returned to content providers is unknown to me. Large studios are not 'price takers' and have significant negotiating power as 'headline products'.
Presumably small niche providers get a return based on consumer views.

This shared infrastructure and 'content packages' seems ideally suited to on-line delivery of paid content - which doesn't have to be limited to news or 'quality journalism', but certainly includes them. Plus we have the natural providers already operating with large, high-quality customer lists: Cable Service Providers.

The technical implementation for an "on-line Channel Service" is simple. Though the ABC iView experience suggests that collaborating with ISP's and allowing unmetered content is necessary. [Australian ISP's impose download quotas on broadband].

Customers already have some sort of PC (Windows, Mac, Linux, ...) and look after their own broadband connection.
A controlled, universal 'player' is required - happily companies like VMware already provide, free, a basic product that work across all platforms, the "VMware Player", which can run pre-built systems with embedded applications, "Virtual Appliances".

The only necessary work is tailoring a VPN or similar and distributing the required registration/connection keys. Foxtel already has the infrastructure in place to source, distrubute, service and support hardware & devices.

That's not a big leap...
And one where services can be packaged in a series of packages with many different price-points.
All those free Community papers, plus a Major Metro Papers: $5/month?
Add a speciality or trade paper, all the Major Metros, a financial services 'feed' and an alert service like 'Media Monitors' for a company of 75 people: $lots.

How does this proposal site with the Newspaper assumptions:

intermediaries with good 'passing trade'
Exactly what the Cable TV companies do.
exact counts, not estimates, or readership
Page counts and the precise subscriber unequivilacly identified & grouped.
Near-perfect marketing information, and a perfect, undisputed source of revenue figures for 'page hits' revenue scheme.
exact counts of advertiser hit-rates (count links followed)
You don't have to sell advertising, and many content providers would not,
But if, like the Trading Post, you did... Trivial.
targeted/niche audiences, not "broad spectrum" mass market
Tailored content per source, niche & specialist sources, remembered preferences and interests... Near perfect for subscribers and providers alike.
Normal print-media space restrictions are lifted: Content providers can provide additional "in-depth" material easily & cheaply.
Individual content providers can use the Channel Service to provide archive, search and print-on-demand services. Leaving each party doing what they do best.
'Leakage' of content can be controlled with the Virtual Appliance.
It may be configured to only allow 10 pages a day to be printed... With extras purchased.
Anyone interested in expensive periodicals, Academic Journals or hard-to-find books?
With controlled access and clear charging regimes in place, there is no issue about denying or destroying copyright.
In another day, this might have been called "Your Local Library".

On top of this, additional options allowing subscribers to pre-pay to view or print normally inaccessible content. The Channel Service Provider doesn't become a credit provider - in fact gains by holding the prepaid money - which it never need return and might even expire, like pre-paid mobiles.

Importantly, 'micro-payments' are avoided. They are very, very hard to get right and consequentially expensive. That's why we've never seen Visa and Mastercard move on this market.

But moving 1cent 'funny money' from your pre-paid balance to a vendor - very cheap.
It's the basis of prepaid mobiles.
With the business-friendly upside of all the unused payments that expire - a tidy 5-10% profit.

In summary: Do I think Murdoch is wrong-headed in charging for access to his newspapers?
Absolutely.

Do I think an on-line service offering this facility effectively, efficiently and profitably can be constructed?
Absolutely.

Will anyone read & respond to this piece and the proposal?
Who Knows :-)

Internet Changes Everything: Newspapers

2009-07-02T12:21:00.005+10:00

The News Broadsheet was a pivotal element of the 1776 American Revolution, eventually becoming enshrined in the First Amendment to the US Constitution.

Newspapers were integral to the Twentieth Century rise and evolution of Western Democracies. Without "frank and fearless" reporting (and an engaged electorate), governments can quickly spiral out of control.

We're now 15+ years into the "Internet Revolution", so where are Newspapers in their journey on-line? Significantly, nobody seems to have discovered a "secret sauce" to generally monetise News and the work of Journalists in the way that Amazon and Google etc have monetised books and on-line ads. Those who make money from writing seem to do so from direct subscriptions - the on-line form of "newsletters".

In my first job, there was a direct and obvious connection between the 5g +/- 0.01g of sugar being analysed and 'the business': Our analyses determined payments for 500 or 1,000 tonne lots. Getting it wrong wasn't an option. My place in the scheme of things was self evident.

The world of Software and I.T., even after 60 years, still doesn't have that direct & obvious link.
I.T. shares many traits with Journalism:

Both are "Performance Disciplines", like Music, Art, Surgery and Gymnastics.
"Effortless Performances" (as in 'making it look easy') take a lot of skill and experience. The public and more often now, management, have little appreciation of the process & skills.
Input Effort and Results bear no discernible relationship.
Quality is Everything, but seems impossible to measure.
Although both are central and necessary to the businesses they support, they are managed as "Cost Centres", with seemingly no attempts at connecting outputs with Profit.
Both deal in intangible and invisible "stuff" - information. Often with tight deadlines and very fast decay in product "usefulness".
Both share a central problem: Effort/Inputs are decoupled from Income/Results.

To research this, I chose "Fairfax Media" (ASX:FXJ), a major Australian player, because some of their "Mastheads" are over 100 years and "The Sydney Morning Herald" used to be legendary - known worldwide for its "Rivers of Gold".

Getting even rough numbers to judge demand and price-points seems very hard.
For any on-line business to succeed, revenues have to support costs. While serving bit-streams may be considerably cheaper than printing & shipping paper, what will people pay for it and how do you get money off them? How do you draw in more subscribers - what are your Marketing & Sales channels?

Newsagents keep the full "cover price" of most newspapers. For magazines and most other products, the publisher gets half with the distributor & newspaper splitting the rest.

The content of newspaper, News etc, is why people buy newspapers.
The price people are willing to pay for the whole paper indicates the "ultility" they get.
But how do you arrive at a value-in-use of just the News component?
Are there synergist effects in operation?

The output of Journalists, "News", is essentially a "Free Good" to both consumers and the business. To the publisher, each newspaper printed is only a cost - which they actively attempt to minimise, whilst revenues are from advertising and independent of pages printed. The connection is the "rate card" - people will pay more in advertising for wider circulation & readership. The connection is anything but direct and immediate. Meanwhile, "content" (& "classified" adverts) brings in readers, but the supply-demand curve is generally unknown.

Makes the economics of "free" community publications more obvious. The distribution costs are only slightly more than to newsagents and the claimed readership is maximised.

It also says why businesses are so very happy with "advertising only" publications, like "Trading Post". Pure profit and no content producers to wrangle! These business translate on-line very well - but lose all "display advertising" to corporates like retailers looking for mass market advertising.

Audited Circulation figures aren't too hard to come by and "The Press Council" publish a good snapshot of the whole Print Media scene, but there are only tangential references to journalist "head counts". Fairfax journalists feel pressured by staff cuts and are taking action.

To come up with viable business models for on-line News, especially when competing with "Free" on-line services like Google or Free-To-Air broadcasts (radio & TV), you need both sides of the Accounting Equation:

Profit = Revenues - Expenses.

Other marketing data is needed to determine whether Size Matters (only one global Google and Amazon) or Little & Local works or some combination in between...

Whatever the result, there is one guaranteed loser: newsagents.
They are the traditional Marketing & Sales Channel for newspaper.
What becomes of them in an on-line world?
Can their access to "passing trade" and existing business relationships be leveraged?

What we do know is that people are very happy with "Free" on-line content: using search engines to point to unbilled on-line newspaper articles.
Newspapers initially tried to force reader registration, even though content was "free" people went elsewhere. Fees were charged to access "archives" or additional material made available to paid subscribers.
News Ltd's upcoming experiment in charging for on-line content will be watched very carefully throughout the industry.

What would it cost to run an adequate newsroom? Do the 15 journos of brisbanetimes.com.au reported by the Press Council provide adequate local coverage? Various sources suggest major metropolitan news have 200-300 journalists, while the national broadcaster, the ABC, have 700 people in their "News Division" (which is how many journalists?).

Any 24/7 operation requires shift-work. To provide a minimum staffing of 2 people, needs a team of ~12. Numbers build very quickly as more sections & coverage is needed.

Fairfax's Annual Report (2007) says they made a profit of ~A$500M ($447M EBITDA) on A$2.3B sales.
Around one third of sales were from New Zealand. Overseas and non-print revenues were unclear.
The Australian Digital operation made A$37M on sales of $137M.

Their ~10,000 employees were their largest single expense: A$700M.
Paper and ink came in second at A$270M.
Sales and Promotions were A$88M.
Communications: A$17.5M
I.T.: A$15M and
News services: A$12M

Fairfax reports $8B of Assets - $6B of which is "intangibles" - "goodwill" and "value of mastheads". They've around A$850M in Property, Plant & Equipment - the physical assets needed to produce newspapers.

But how many journalists were there and what does a single "high-quality" newsroom cost to run? That's not going to change for an on-line News service.

What will an average subscriber pay for an on-line News service?
What additional content are needed, and what synergies exist?
How many on-line subscribers will sign-up for each different offering?
Can the existing subscribers be converted to on-line subscribers? What would help in the transition?
What are the different attributes that subscribers value and what premiums will they pay?

This says that monetising on-line News services may be hard and their general lack says solutions are still not obvious.

Other comments, observations and relevant factoids:

Newspapers serve important subsidiary functions, like being "the paper of record" for Births, Deaths and Marriages as well as public events, political speeches and disasters, crises and more.
"Public Notices" of many types are published - from bankrupts to probate on wills to personals.

Newspapers have come to be the definitive textual mass-communication device.
There appears to not yet be any on-line equivalent.
In the USA, recent failures of long-running mastheads says there isn't that much time left to find a good answer.

Circulations:
The SMH sells ~212,000 copies Mon-Fri (@ $1.40), 364,000 on Sat ($2.40) and [Sun-Herald] 505,000 on Sun. ($1.80). It has increased circulation in recent years. No figures available for pages of advertising sold.
The Melbourne Age has comparable sales figures, though Sunday sales are roughly half the SMH.

'Readership' of the SMH is estimated at 853,000 Mon-Fri and 1,116,000 Sat.
Papers are shared around.

These figures suggest that newsagents & supermarkets make around $1.5M/week for Mon-Fri sales and the same again on weekends. Removing that income stream would leave a very unhappy sales channel...

Payments and Subscriptions types:

Pre-paid or billed subscriptions. Like Home Delivery
On-demand use: Ad-hoc or occasional purchase
Prepaid access to articles.
Business or family sharing. Limited copies shared between many.
Public access - libraries.
"Media Monitors" - clipping services across many sources relevant to a business.

What do Newspapers do?

Inform
Educate
Entertain, and
Surprise and Delight.

Newspapers tell you things you need to know, especially things you didn't realise you needed to know.

What comprises 'stories'?

facts
expert analysis
commentary
opinion, and
interviews

Professional journalism brings brevity and precision. Stories are {complete, correct, concise} and hopefully {consistent}.

The value to the reader is collection and pre-screening: reducing a mountain of data and facts to quickly and easily accessible information. Journalists classify and prioritise stories, letting the reader minimise time used and maximise information found.

This "Usability" principle extends to presentation, layout and story structure.
The fonts and columns widths are chosen to best suit human factors.
Modern printing added pictures to text - an important aid to readers and writers alike.
The use of white space, headlines and graphics/pictures increase readability & accessibility.
Stories follow the "inverted pyramid" - the most important facts first, tailing off. It means readers can quickly scan articles and find the most relevant/useful to them, and sub-editors can easily trim articles to fit by removing trailing paragraphs.

It is instructive to read a 100-yo paper without these modern features - they look dense and impenetrable. The small number of pages would've been a blessing.

Professional Journalists bring special factors to collecting stories:

access to people & organisations, like police, politicians, CEO's, ...
funding for travel, communication and fees - like FOI requests or corporate data.
Researchers, archives and access to expensive subscription services.

The users of News services look for many different benefits and uses:

text, images, audio and video
Mobile access
Alerts and "Instantaneous news items" (the latest stories)
Distraction, Relaxation and passing-time (as in commuting)
Social settings - Cafe and Brunch
Search and information: Browsing classified and focused searches

On-line challenges:

distribution format. PDF's or HTML?
How do I read it on the train?
Does it read well on a laptop or iPhone?
enforcing DRM. How to limit copying? What to do if copied illegally?
on-going subscription rights:
If I've bought a "paper" once, do I have permanent access to it, even after my subscription ends?
Competing with Free-To-Net services. Seems hard, have to provide additional value.
Catering for local community news
Accepting input from the general public.
The Blogosphere and non-professional writers. They can't be held to journalistic standards and ethics, can't be reprimanded or censured and will publish/repeat unsubstantiated gossip and rumour.

"non-News" Newspaper functions that Online services may need to duplicate:

"paper of record" function: Public Notices, Births-Deaths-Marriages
Mass-market display advertising.
Marketing and Sales channels. How to grow new business?
Access to Printed copies - eg. weekly summaries.
Searchable service provider directories and classified adverts.
Implies RSS-style alerts & monitoring.

On-line service challenges:

Monetisation. Advertising, subscription or sponsor based? Other?
Niche marketing. By location, interest, community, employment sector.
Tiered Subscriptions: free, basic, premium, target area, search tools, ...
Profits can be maximised by segmenting services with multiple price-points.
Organisational access.

Telco pricing and market 'price elasticity'

2009-05-16T12:44:00.004+10:00

There's a counter-intuitive effect with marginal cost of Production Factors, like energy (and Teleco services) - using the factor more efficiently, consumes more of the resource. Because you make more profit, lower prices, produce more and demand for the resource increases. The Khazzoom-Brookes Postulate/Jevons Paradox:

"energy efficiency improvements that, on the broadest considerations, are economically justified at the microlevel, lead to higher levels of energy consumption at the macrolevel."

The structural reason is simple: the market is highly price elastic, so decreasing prices a little lifts total sales considerably. In economics, this is a well solved problem for non-monopoly markets, "Profit Maximisation" occurs when MR = MR (Marginal Revenue equals Marginal Costs). [For monopolies, MR = 2*MC, IIRC.]

In the 70's & 80's at O.T.C., we made record profits each and every year - by exceptional marketing and sales strategy, which included dropping prices every year. [The TV adverts series, like the 'Memories' campaign, won many awards.]

This was driven by the technology: Moore's Law drove the per unit cost of services in both cable and satellite down exponentially.

Profits margins increased because full cost reductions weren't passed on. By the mid-80's it was cheaper from the East Coast to call London or New York than use Telstra to call Perth ($1/min).

This is a lesson Telstra Management never learnt and was obviously lost when O.T.C. was subsumed into 'the Borg' in 1992: passing on part of the Moore's Law savings, Revenue and Profits both increase.

Economic theory is very clear on this point:
Traditional (Premium) Telco Pricing isn't just 'bastardry', it kills your profits which will eventually kill your company.

People love to talk/communicate, it is one of the defining characteristics of Homo Sapien.
This underpins the profitability and constantly growing demand for Telco services... This, the consumer demand, is the real 'crown jewels' (or 'birth-right') of Telco's, not their networks and technology.

Nobody has yet plumbed the limits of the price elasticity curve for human communications.
Setting the marginal cost of phone calls to zero doesn't kill your profits as shown by the US 'free call' areas.
It does lead to changed behaviours - more calls, many shorter.

Conclusion:
The incumbent Australian phone companies, Telstra, Optus & friends, don't understand the fundamentals of their business:

Moore's Law has been driving down their input costs exponentially for 4-5 decades, yet they haven't passed on those savings. The differential is now 100-10,000 (guesstimate).
The demand for human communication is close to infinite.
The Comms market has close to infinite price elasticity.
Profit Maximisation is simple and 'Premium Pricing' (what-the-market-will-bear) is exactly wrong.

The challenge for the Telcos now, especially Telstra, is unwinding from their 'Premium Pricing' to a rational 'Profit Maximisation' model. How do they explain price reduction of 10-100 times?

The consumer backlash, if not managed, will be savage and punitative. The original 1990's duopoly Telco's, Telstra and Optus, will suffer greatly - to the point of being endangered.

"You can fool some of the people all of the time" - but woe betide you when those you've fooled understand they've been gouged/conned and have alternatives.

Aus High Speed Broadband: Barriers and Challenges

2009-05-10T15:33:00.007+10:00

A response to the "Public Sphere" run by Senator Lundy on "High Bandwidth for Australia".
Previous comments on the National Broadband Network - about excess costs, poor design and addressing lost strategic opportunities.

Future posts will be linked from here.

Roger Clarke finished his presentation with a comment something like "the upside is obvious, the downside needs to talked about".

Without bringing down the tone, optimism and vision of the event, enumerating the Barriers and Challenges associated with roll-out and adoption of 'universal' High Speed Broadband (HSB) in Australia allows for generating better designs/options, avoiding 'surprises' and anticipating routes around and out-of "bear pits", "Forewarned is forearmed". Who knows, new insights may emerge from this viewpoint.

Alan Kay in his 2003 Turning Award speech identified the central importance of ICT:

"our field is a field that's the next great 500-year idea after the printing press"

Getting the transition into High-Speed right enough is important:
• There is no way back.

Setting user expectations is very important:
• Servers, not the network, will become the new bottleneck.

The central technical challenges are:
• Usability, Performance, Security, Reliability/Robustness.
Scaling-up and scaling-out current solutions won't be easy, obvious or simple. Probably not cheap or quick either.

The "drop-dead" challenges are non-technical:
• Social, Legal/Political, Regulatory, Cost.

User acceptance:
• Social, Legal/Political, Regulatory, Cost.

Australia has a history of slow take-up of new technologies. It took until 1975 for Colour TV to arrive in Australia. This is counter-intuitive because Australians are well known innovators, inventors and scientists. Individually we embrace change, collectively reject it.

Some of the central problems & differences with The Net are:

universal & global access
Uncertain Identities - "nobody knows you're a dog"
non-physical, untraceable digital goods
The Permanent Digital Record

Topics:

Libraries
Copyright, copying digital docs (c.f. photocopies & royalties).
Content: DRM, Protection, Enforcement, "free for domestic use", "Free for Mash-Ups"?
Creative Commons & shared content/resources
e-Commerce and Monetising Content:
- Royalties: Registration, Recognition, Collection, Distribution
- Definition, Usage Rights, Nr Plays, Time-shifting, Time-to-live & View-Time
- 'Sharing'
Costs
- $: up-front/install, on-going & operations, upgrade/variation, usage (Volume/Time/access), 'surprise' charges: excess-data, reverse-charging, malware induced
- Damage and OH&S issues: e.g. RSI, eye-strain, ...
- Availability: Regional, Speed, Filtering
- User Time: Amplifying User Effectiveness, time cost of {Admin, Tools, machines}, Training, PC maintenance/modification/upgrade, Home Network {design, install, maint, upgrade}
- Server Access: Free local on-net access, free home servers, Caching {Cost, access, copyright}
- Security & Privacy: Admin, monitoring, repair
- Time Shift Streaming:
"Acceptable Use Policies", Netiquette, defining & enforcing "Rules of Conduct" {email, IM/twitter, blogs, web pages}
Digital Divide: Socio-economic, Cultural, Geographic, Age, Disability
- Digital Natives vs Gen X vs Baby Boomers vs Pre-boomers
- Electoral Disenfranchisement through link/computer speed, proprietary file formats/browsers/utilities and codecs
Smart Networks vs "Too smart by half" - limiting and inflexible implementations/options
Power (Electricity), Mobility, Screen Sizes (iPhone, Kindle, netbook, laptop, desktop, Video)
"All About the Data": Guaranteed Open Standards and Open Access to Public Content. {Audit, enforcement, appeal}
Unintended Consequences: Recognition, Identification, Redress, Enforce/Change, Legal suits
Privacy and malicious content: Erasing 'leaks' and the untrue/misleading from The Permanent Digital Record.
High Performance networking: Video, Adult Content and Gambling.
SPAM, phising, malware, etc:
- Cyberspace is a unique and different Legal Jurisdiction. Global approaches, like "The Law of the Sea", Intellectual Property and 'Post and Telegraph' are needed. A critical enabler is physical identification and tracking of maldoers and universal extradition: "nowhere to hide" has to be both a mantra and a reality.
  Bringing recalcitrant regimes into line shouldn't be hard: disconnect or rate-limit Internet to their country.
- Digital Organised Crime and Digital Conspiracy are serious, global offences which are in dire need of addressing. Underpinning legal and properly funded policing frameworks are needed. Unlike the US "War on Drugs" and "on Terror", th
- Identity Theft is a major crime & needs one international policing body, with an associated court and realistic penalties.
- Falsified E-mail Identity, the basis of SPAM, is as illegal and serious as Mail Fraud. It needs to be treated so, with efficient detection and rapid response.
- Theft of resources: bandwidth, CPU or storage, is as much theft as stealing Fax paper. There was a landmark case in the UK where adverts using unsolicited Faxes was successfully prosecuted. Internationally, the same precedent is urgently needed to control 'botnets' and other malware.
- Stealing bank login details and so money is exactly Digital Bank Robbery. This is crime, pure and simple, already recognised by every jurisdiction. Why there seems to be no Political will to identify and rapidly bring-to-book these criminals I find astounding and indefensible. The mischief and damage possible through Digital Robbery is enormous - if ever fully realised, 'shock and awe' will be appropriate.
Government
- The Net is a Distance and Time shifter: it changes the nature and style of public interactions with their elected representatives. Importantly, it changes representative accountability.
- A single/integrated approach for all levels of Government (Local, State, Federal) is required for best results.
- Public access to appropriate Bureaucrats requires both a role/positional Directory and 'ticket handling' systems like Helpdesks. The public have a reasonable expectation to prompt, efficient resolution of requests, complaint etc with Open and Transparent reporting of performances.
- On-line anonymous venues are the perfect mechanism for whistle-blowers and other public-interest disclosures. Handling problems like the Rockhampton "Dr. Death" allegations or the "Butcher of Bega" sexual attacks, while avoiding false or malicious rumour is the key test.
Addressing Human Limits on Input & Output
- John Mashey, creator of the MIPS chip, observed the disparate bit-rates for Human Input & Output. Typing is incredibly slow (30-40bps) speaking & listening relatively fast (10-32kbps), but we excel at inputting vision (10-100Mbps).
more...

The issue of seeding and supporting early adopters and leveraging their experience/skills into the general populace is central to the wide-spread adoption and use of the new technologies. "Who you gonna call?" when you need to know how to do setup/do something is a critical support mechanism.

Alan Kay has some insight in this problem as applied to Education, with additional comments on applying to adoption of new Computing paradigms. A feasible approach would seem small, focused communities of interest who support & reinforce one-another and who 'spread the word'. Something like the CRC program for Research.

Why are we doing this?

"Augmentation", an important concept, was mentioned by a later speaker. (James Delow?)

Doug Englebart, one of the early giants in Computing, ran SRI's Augmentation Research Centre.
In 1968 he and his group gave an extraordinary demonstration for the ACM - it included hypertext, interactive video and teleconferencing. Implementing this same demo forty years on would still be a challenge.

In 1945 Vannevar Bush, the inventor of Hypertext, designed a system, "The Memex", to augment/amplify a researchers ability.
Key to its use and value was the ability to share 'Associative trails'. It allowed "researchers" to identify and exchange exact lines of research - all the material in one place.

Do our current regimes of copyright and DRM support or hinder the widespread use of technologies like this?

Homo sapien has a need for freedom and meaning.
What marks us as unique is our thirst for knowledge, the need to 'understand' and to communicate. "Solitary Confinement" is an especial punishment.

We cherish stories and story-telling. Maslow even identified "self actualisation" as our highest need. We don't just learn, we record History and identify its Lessons - what works, what doesn't. We advance because we "stand on the shoulders of Giants".

High Speed Broadband is about Humans limitless need to Learn and Communicate.
It is the lastest, but not the last, technological means to facilitate our many needs.
Not the least, our daily work, trade and commerce.

Which segues to our individual and collective 'darkside'.
Creating a new tool/mechanism means creating effective non-draconian ways to keep it safe.
Taking the stance this brave new frontier should be Free and Unfettered is amazingly naive and self-serving. If it were this simple, there'd be no SPAM or malware. Much worse is waiting out there.

Alongside the technical and organisational challenges, we have to consider the Regulatory and Legal environment of what is acceptable behaviour and 'words'. Then figure out how to police and enforce those limits, without destroying the very thing we are trying to foster.

Universal access high-speed broadband isn't "just faster", it's a game changer in exactly the same way that motor vehicles redefined our societies. Megatropolis's are not possible without high efficient, high performance transport networks. Our advanced technologies aren't possible without those large conurbations, either.

"More of the same" thinking isn't adequate for this new technology.

A useful beginning is to ask the questions:

What do we need to do MORE of and
LESS of?
What do we need to STOP doing,
START doing, or
CHANGE/MODIFY what we are doing?

It's a Brave New World. We can try to get it off to a good start.

Broadband Lost Opportunities: Gas and Broadband

2009-05-06T09:21:00.004+10:00

1999/2000 saw the laying of the "Eastern Gas Pipeline" (EGP) - 795km from Longford in Victoria to Horsley Park in Sydney, at around $450M. It carries natural gas from Gippsland Basin (and Bass Strait?) and provides additional supply security to NSW.

The mainline route is:

close to the towns of Bairnsdale, Orbost and the Cann River, before turning north past Bombala and Cooma. From the eastside of the Snowy Mountain region, it travels northeast to the coast through Nowra, Port Kembla, Wilton and on to Horsley Park to its termination point on the outskirts of Sydney.

Country Energy Gas is the NSW regional retailer. The Energy Networks Australia State Summary say this about the NSW country towns connected to Natural Gas:

There are 24 210 km of reticulation mains serving 901 000 customers in the State’s major urban areas, and its large and small regional centres.

Wagga Wagga, Tumut, Adelong, Cooma, Bombala and Gundagai, Albury
Nowra, Bomaderry and Queanbeyan.

Some of NSW's gas pipeline "Laterals" are listed in this NCC PDF file:

the Mainline from Moomba to Wilton;
the Dalton to Canberra lateral;
the lateral pipeline from Young to Cootamundra and Wagga Wagga;
lateral pipelines from Young to Bathurst, Orange, Lithgow and Oberon; and
the lateral pipeline from Burnt Creek to Junee, Griffith, Leeton and Narrandera;

The Sydney Hydraulic Power Company (now part of KONE Elelvators) pipe network was shutdown in 1975 after nearly a century of operation. The buried high-pressure water pipe through the CBD remained a valuable asset. In the last 1990's it was purportedly acquired for Telecommunications use (no references found).

Utility companies have been mindful, since the deregulation of Telecommunications in Australia, of their unique access to households/businesses and the commercial possibilities flowing from it. PowerTel, formed in 1998, was 31% owned by the "Downtown Utilities" consortium, EnergyAustralia, Citipower and Energex. They provided the local loop right-of-way to a potential 2.5M of their customers.

Transact, in 1999/2000, also leveraged this access from powerlines to roll-out its network.

Rail corridors have also been used (extensively?) to lay optical fibre cheaply. A very brief sample:

In 2001, Leighton Holdings announced the 1800km Queensland "Reef Network" costing A$85M installed in the QR corridor. ($45/m)
QR Networks is currently selling spare capacity, dark fibre and site/corridor access.
VicTrack also has a Telecommunications arm.

Reef Networks' $45/m cost is well under the NBN's estimated ducting only costs of $60-$150/m.

The various Gas Networks should've been aware of the these options and had the opportunity to leverage their extensive pipe network upgrades and additions into both long-distance and local-loop Telecommunications networks. Only in one town, Cooma, did a third party partner with them to lay empty conduit/pipe when the reticulation network was constructed. The marginal cost of laying another pipe in the same trench was minor.

This oversight has massive implications now with the (A$43Bn) NBN roll-out.
Not only would those utilities have an additional on-going revenue renting duct access, it would've made the NBN roll-out much cheaper, easier and faster. Nobody wants their street dug up one more time.

The shareholders can't be happy with their boards either. The real loss was in National strategic and commercial competitiveness. A lack of vision and execution by the responsible regulators has cost the country dearly.

Phone Companies vs Broadband

2009-05-05T10:15:00.002+10:00

Traditional Telcos, "phone companies", are exactly the wrong people to own and operate consumer Broadband networks, and their close relative, Cable TV.

There is an inherent conflict:

Telco Engineering is very different to Computer Data Network Engineering. You cannot be good at both or build networks that are good at both.
The network & equipment designs and tradeoffs are antithetical and antagonistic.
Telcos make money from phone services, which are relatively low bit-rate. It is against their commercial interests to sell high-quality, high-speed data networks at affordable prices.
To expect them to cannibalise their most profitable line of business is asking for incompetent & irresponsible management, which puts them in jeopardy from corporate regulators and malpractice suits from shareholders.

There has always been a fundamental conflict of interest in any Telco rolling out Internet broadband services: cost-per-bit charging either makes telephony effectively free or streaming TV monumentally expensive.

In Australia after Telco deregulation, the initial duopoly (Telstra & Optus) were allowed to roll-out both Broadband and Cable TV. The early 1990's Cable TV debacle with 80+% duplication showed both the inherent commercial conflict and that both Telcos were aware of it and would act, aggressively, to maintain their commercial interest.

The current parlous state of play with National broadband service underlines the argument.
The design of the new NBN by Telcos is similarly flawed. They need to make it expensive and complex to protect their phone networks for as long as possible.

That our legislators and regulators have failed to understand this inherent conflict, to see the massive market distortions/failures is an amazing oversight. There are either blind to it or accepting of it and in the thrall of the large Telcos. Is it ignorance, incompetence, arrogance, apathy or something worse?

The solution is clear and demonstrably effective:

Issue separate licences for telephony and data networking, with the same cross-ownership restrictions/regulations already applied to the Media.
Papers, radio and TV need to compete with one another.
Regulations prohibit a single marketplace being dominated by a sole player.

Explaining why Australian consumers aren't served by a protective communications regime is becoming increasingly hard...

Telco Engineering vs Network Engineering

2009-05-03T16:30:00.009+10:00

For many years I had the uneasy feeling that the Telco Engineers I worked with/for over most of a decade at O.T.C. did not actually understand Computer Data Networks.

Here I attempt to formalise and explain that thought.
The implications/ramifications won't be examined in this piece.

History

Telephone/Telco Engineering dates back to between ~1880 with the formation of the first phone companies and 1915 with the first US transcontinental (long-distance) calls. Dial phones were introduced in the US circa 1919, but automatic exchanges had been invented well before. The profession of Telco Enginering has around 125 years of tradition and practice.

As a profession, they have been very good at doing what they do - creating reliable point-to-point voice communication. They have extended into high-availability point-to-point digital services.

Modern Computer Data Network Engineering dates from around 1982 with cheap mini-computers and the IEEE 802.3 standard for Ethernet. Cheap PC's, affordable Interface cards and UTP (10Base-T), standardised in 1990, set the stage for current LAN's - desktop and server. This gives the modern discipline around a 25-year history.

The Universal Network Glue, IP (Internet Protocol(s)), dates from 1969 with the interconnection of the first two systems. Early Telco Data Networks go back to Telegraph (morse code) and Telex and ended with X.25.

For completeness, the World Wide Web, created by Tim Berners-Lee and Robert Cailliau at CERN, and now popularly referred to as 'The Internet', leveraged in 1989 PC's, ethernet and TCP/IP.

1996, the birth of the Modern widespread Internet, was marked by Microsoft abandoning it's proprietory MSN network & protocols and adopting Internet Everywhere.

Differences in Networks and Approach.

Telco networks started with patent wars, bleeding-edge technology and on-going & increasing requirements for large capital investments. That 'sunk cost' became huge as phone access was rolled out almost universally, at least in the 'First' and 'Second' world, providing a considerable barrier-to-entry for new players. After some decades, incumbents could easily kill new competitors by under-pricing them - they had paid for their networks and with great Free Cash Flow, could upgrade & extend their cable plant out solely from operating revenues.

To compete in the Telco world required huge investments in cable plant and switching equipment, and extensive, preferably full, network coverage. "Metcalfe's Law" states the value of a network increases with the square of the number of connections. A provider with a slightly better coverage in an area, all else being equal, quickly gained an economic advantage. Achieving better cash flow & profits either through higher charges or more subscribers. This could pay for faster expansion of the network, increasing their advantage. A virtuous circle.

Subscribers could not, even if they wanted, install & run their own cable plant & subscriber equipment. Initially, it was too expensive and patent-protected, then precluded by technical compliance requirements, then by legislation and regulation.

Telco operating principles became:

effective monopolies per region
hub-and-spoke design
extensive overbuild to cater for projected demand
high-availability, high-cost central equipment and interconnects
simple subscriber equipment and complex exchanges and transmission systems
'Premium Pricing' model ("what the market will bear", vs "cost plus")

Telco networks are a classic Cost Accounting study: almost all costs are Fixed and Indirect, often dominated by Financing costs. There are almost no Direct or Variable costs.

E.g. the only marginal cost for any phone call is the cost of electricity: ~1 watt per phone. Around 20 milli-cents per hour. Capturing & processing billing data is 100-1000 times more expensive.

Telephony and Data Networks differ in almost all details, "Let me count the ways":

Factor	Telco	Data
connection	Circuit	Packets
speed	fixed	variable
Noise	ignored	error-correction
echoes	cancellation	n/a
multiplexing	external systems	inherent
Model	Central Switching	Distributed switching
Topology	Hub-and-Spoke	Buses, self-healing loops distributed equipment
High-Availability	component redundancy	whole switch duplication
congestion	no circuit, call fails	slower transfers
lost data	noise & dropouts	retransmit
switch design	non-blocking, continuous connection	queues, dropped packets, retransmit
lost data	noise & dropouts	retransmit
variable delay, jitter	highly sensitive	tolerant, retransmit
multiple connections	more links	increase link speed
Encryption	External, expensive	Embedded, Extensible
Intelligiblity and 'Quality of Service'	good, guaranteed	variable, no general QoS
Local loop Scalability	Rebuild, reinstall	In-place link upgrade
Upgrade	Long-range forecasting, Initial overbuild	In-place upgrade, incorp Technology advances
Equipment source	Specialised, expensive	generic, commodity
Financing	Large CapEx hidden in monthly rental	Prepaid install and Customer owned
Billing	Post-paid, unlimited Credit, Itemised bills	Prepaid with limits
Capacity	megabit range	HD video capable, tens of gigabits
Multicast	Only single-cast	multicast capable

What must be made clear:
there are some services that IP Data Networks do not currently deliver as well as the Telco networks. Those requiring low-latency, low-jitter, and low-noise. I.e. a guaranteed (high) Quality of Service. Even the traditional consumers of these services, radio and TV, are changed their work practices and moving to in-house IP networks or general Internet delivery.

Until 1999/2000 overseas telephony dominated those trunks. Since then, direct internet traffic has kept growing (exponentially - what doubling period?) and now swamps all other service demands.

As an example of the capacity differences: the ~10M landline services (at 32kbps) and 21M mobile services (at 9.6kpbs) represent a maximum of ~500Gbps demand, possible in a single, albeit large, router. The usual demand is around 2-5% of the maximum (10Gbps) - now well within the capability of low-cost routers.

Telco Engineering applied to Public Data Networks.

Large modern corporate Data Networks are "Pure Internet" networks. Networks such as the Department of Defence, cover most parts of Australia and extends overseas. It provides for 100,000+ desktops, a larger telephone networks, audio & video broadcast and secure services. They compete in size, service range and complexity with normal "common carrier" (Telco) networks.

If 'traditional' Telco Engineering approaches were more cost-effective or provided better availability & reliability, then they would be in use. The usual arguments for not delivering "Pure Internet" (commodity links/equipment, symmetrical upload/download) to households is population density. Defence faces the same distribution problems across its many, large campuses - and still run "Pure Internet". When cabling/trenching costs dominate, it still makes sense to run small copper or optical fibre cables with switching systems distributed through the network.

The costs of running individual copper or optical fibre from a central exchange to households increase dramatically over a commodity Ethernet/Internet solution:

total physical copper or fibre required is 10-100 fold more.
large cables (eg 200-pair) are expensive to buy, install, join and repair/maintain
many joints, each a failure point, are required to each customer premises, versus a single clear run to a local access point
duct sizes near the centre get very large, compounding the costs, complexity and maintenance/upgrade problems
with copper, cross-talk & interference problems compound with increasing circuits
generic, commodity equipment cannot be used in either the Central Office or subscriber
link speeds are fixed unless a major equipment upgrade is performed.

The Transact and the HFC Cable TV networks utilised this 'network embedded switching equipment' approach, resulting in per-house-passed costs of under $2,000 vs the $5,500 of the proposed NBN.

Perhaps the most convincing argument is what the Telcos now use for their backbone networks: Pure Internet. Many of their new services offerings are managed services derived from this internal IP network.

Or ask what companies are best positioned to offer "Triple Play" (TV, Data/Internet, Phone).
People with high-bandwidth backbones and upgradeable local-loops.

NBN - Powerplay or 'for real'?

2009-05-02T16:57:00.011+10:00

During the week I had a fault on my phone line and got to talk to the Telstra tech afterwards.

His view was naturally Telstra-centric, but contained wisdom & insight.
It would make sense to have just one local access loop with just one maintenance organisation to which all Telco's have equal access. And it couldn't be owned by any one or two commercial players, that distorts the market.

He described an asymmetry: how new Telco entrants can build their own infrastructure and deny access to all others, but Telstra was obliged to provide access to 'their' copper network to everyone. He sited the sad case of a pensioner needing a phone and waiting for quite sometime (and paying a big fee) while Telstra dug in a cable that the developer should've installed... All the time there was physical cable to the premises owned by another Telco, but inaccessible.

The 1994/5 HFC cable TV rollout by Optus & Telstra (80-90% duplication) shows the insanity of Telecomms commercial arrangements and regulation in Australia. Behaviour that you wouldn't tolerate in school children.

That it was never a commercial decision is shown by the subsequent huge write-offs by both players. If they'd be ordered to "play nice" and construct a single infrastructure with bilateral access, the face of Australian Telecomms would be fundamentally different today. Cable TV would be a real force - possibly covering 80% of houses and making real profits.

The Telco regulators have allowed the same pattern to be repeated with mobile phone operators. In the USA & Europe, operators allow competitors access to their networks and make good income from it. It's called "roaming" and is supported by all the standards, hardware and handsets...

Everyone benefits, it's a positive-sum game. It doesn't stop operators extending their networks when they know they'll make more money by building their own infrastructure. [They have hard data on their customer call patterns.] It also means new entrants are 'born global' and have time to build-out their network and manage their cash-flow and CapEx. It encourages and enables real competition, which again benefits everyone.

Only it doesn't happen in Australia. There are 4 or more independent networks, everywhere, and no roaming agreements. It's not about commerce or service, but sheer bloody mindedness. All these Telco's have roaming arrangements with overseas operators. They have both the technical and commercial knowledge to do local roaming. [There is some for mobile wireless internet.] Everybody loses - it's a negative-sum game.

Regulators should not allow "Coverage" to be a marketing differentiator.
Failure to cross-connect/access should lead to heavy fines and eventual revocation of a Telco license. Telso Licenses, like Banking, are granted so commercial entities to primarily provide a public service. In return, a limited monopoly is granted.

It's all about providing public services, not about the profits of licensees.

Back to the NBN.

What's the long-term impact on Telstra if Kevin Rudd & Co forcefully construct "The One True Local Access Network"? If I were Rudd and was forced down this path, I'd deny Telstra access for forcing the issue in the first place.

Everybody loses, comms prices are high, services are limited and national economic competitiveness declines.

There is a 'critical point' of market share at which Telstra cannot continue to maintain and operate a parallel, full-coverage network - even with more limited services. As their market share declines, their profits reduce, putting pressure on maintenance, operations and customer service. This leads to more limited offerings, poorer service and unhappy customers, leading to yet smaller market share, decreasing profits and an inevitable "death spiral" that can only be broken by massive capital injection or embracing the NBN.

Telstra embracing the enemy seems very unlikely. Over several decades, the Telsra Board and Management have demonstrated they will not work with others, sometimes even after ACCC action and court directions. The have proven to be obstinate and recalcitrant.

There is also the competitive services problem: if the NBN provides desirable services that Telstra cannot deliver, then an increasing number of people leave Telstra, leading to the 'death spiral' by another route. Telstra can buy market share by dropping prices, but at the price of longevity - they destroy their profits and the ability to fund growth & new services.

The Federal Government and its regulators must know these things.

Are they taking Telstra on head-on with the intention of putting it out of business? (It can only be a 'take no prisoners' struggle to be won by the deepest pockets or a change of political direction.) The last thing Australia needs is the NBN being privatised before it is the dominant player. Without unlimited financial backing, Telstra would win, even if mortally wounded itself.

Or is it just a Power Play to force the Telstra Board and Management to wise up and 'play nice'?

We'll only know in hindsight.
Australia shouldn't have to bear either the massive cost of duplicating the local access loop or of Telstra failing.

Update 1. Sunday May 3, 2009. Senator Kate Lundy points to this piece by Richard Alston (Minister for Communications etc 1996-2003). Alston notes that Telstra aggressively duplicated the Optus Cable TV roll-out. He doesn't say that as the responsible Minister that he could've acted to prevent or change that.

Update 2. Tuesday May 12, 2009. The Australian reports :

The federal Government will offer Telstra the chance to buy up to 49 per cent of its national broadband network, if it agrees to voluntarily hive off its wholesale arm.

Telstra undertakes "structural separation" - into Wholesale & Retail arms. In return for its current Fibre Network, gets 20% of NBC (National Broadband c/o).

With both a new CEO and a new Chair (now Donald McGauchie is gone), Telstra may be able to resile from its "never, ever separate" position.

That values Telstra's 'Fibre Network' at $8.6Bn and allows them another $12.5Bn investment.
After the original $4.7Bn for NBN 1.0, the Govt. needs an additional $12.4Bn - half public, half private.

Guesstimates for NBN components:

Domestic subs, 9M @ 100Mbps ($1,500/house): $13.5Bn
Exchanges, 1,000 @ ($500/sub + routers/uplink): $5-8Bn
Backhaul/Interstate upgrades (10+Tbps scale): $5Bn
International cables - 10Tbps (1Mbps/household): $5-$10Bn
Peering & ISP interconnects (20-100): $2Bn
Rural/Remote radio/Satellite, 1M @ 12Mbps: $5-10Bn
Content provider feed network: $2Bn
High bandwidth subs: Business, Schools, Hospitals, Govt: $5Bn
Network Operations, Maintenance & Test spares/depots/equip, Training: $5Bn
Billing, Call record & traffic analysis, Line management & other IT systems: $2Bn

Update 3. Friday May 15, 2009. Stephen Bartholomeusz in "Business Spectator" reports:

Lindsay Tanner has confirmed what was suspected. In arriving at its estimate that the cost of the revised national fibre-to-the-premises broadband network would cost $43 billion, the Rudd government essentially dreamed up a big number and then added to it. ....

The evaluation that should have preceded the commitment will now occur, with the government commissioning an "implementation study", which one assumes will consider the complex economic issues involved and come to a conclusion whether the NBN is a viable commercial proposition.

If Telstra agrees to "structural separation" & buys in, the economics change.
If not, there's enough money in the bucket to do this thing "right".

Mircosoft can't write O/S code

2009-04-27T15:33:00.003+10:00

This cartoon lampoons Windows 7.0 Beta. Eerie.

It underlines for me that Microsoft is crap at writing Operating Systems code.
O/S's need to be correct, secure, robust (resilient to errors internal & hardware) first and foremost. Only after that look to features and 'performance'.

How come the Great Unwashed, the home & business users and the media seem blinded in the same way the fanboi's are?

Wikipedia's "Time Line of Operating Systems" underscores this point - compare Microsoft's offerings since 1999: Win-2000, Win-XP, Longhorn: abandoned, Vista and 'soon', Win-7.

Compare to Annual or Biennial shipping of new releases OS/X, AIX, Solaris, HP/UX, Ubuntu, Red-Hat & Fedora, SuSE, ...

Microsoft is the exception, not the rule. Everyone else in their class does better.

Apologists for MSFT may add Win-98 SE, Win-ME, Win-server 2003/8, Media Centre, Home Server, Small Business Server and even Win-CE - plus the many 'Service Packs'.
This is all confabulation. The code differs mainly in licensing restrictions and add-ons.

The 10,000 strong Microsoft 'team' put around 25,000 man-years into Longhorn before Jim Allchin decided to call it quits and start again with what became Vista...

Jim Allchin, who had overall responsibility for the development and delivery of Windows, explained how development of Longhorn was "crashing into the ground" due in large part to the haphazard methods by which features were introduced and integrated into the core of the operating system, without a clear focus on an end-product.

Producing commercial grade O/S's require discipline, design and defined processes - Software Engineering 101 - to pump out regular updates & upgrades. The professional challenge isn't in producing a good piece of code just once, but like the Space Shuttle - doing it again and again to Schedule, Cost, and near flawlessly.

That's also basic Project Management: Plan, Schedule, Control.
Decide what you're going to do, plan how & when you're going to execute the project, manage deviations/problems/surprises during execution.

Operating Systems are vitally important - they sit under every other software layer.
They set the upper-bound for Security, Reliability, Usability and Performance of all applications on the platform. Nothing can be more robust than the layers it sits atop.

Hardware Engineers might take issue. It's long been understood how to fabricate ultra-reliable hardware systems from redundant components. Nobody has paralleled that work in the O/S and Applications arena, so my assertions stand.

Don't get my criticism wrong. Microsoft and Windows have been important in creating the world of commodity computing as we know it. I'd even rate them good at producing some Applications, such as Office. (But the software does suffer from 'featurism' , another challenge.)

The first Microsoft O/S's were adequate and, supported with unrivalled marketing, brought in the age of ubiquitous, mass market computing.
Microsoft have leveraged their natural monopoly (read 'proprietory lock-in') better than any company I know of, in any field.

Their Marketing is an example to be emulated, unfortunately their production of O/S software isn't.

High-Speed Broadband: Excess Costs and Opportunity Losses

2009-04-23T11:38:00.006+10:00

The National Broadband Network, NBN, is a chance to correct regulatory deficiencies, address broadband network design problems & high costs and ameliorate past strategic mis-steps.

Gas network providers not leveraging their investment in 'holes in the ground' and high consumer connectivity was a huge strategic blunder.
Allowing the 80% duplication of cable TV network was a massive failure of Government oversight. It would've been obvious within 2-3 months and should've resulted in quick, decisive action. As was not forcing 'roaming' between mobile service providers.
The NBN design reflects that Telco Engineers don't do Data Networks well... Exactly the same way you don't build a water, electricity or gas network as if its a phone network (hub-and-spoke).
The technology choices & network design seem wrong-headed to me. They are not using off-the-shelf commodity hardware, but very expensive 'Carrier grade' offerings with limited scope for upgrade & expansion.

What seems missing in the designs I've seen is redundancy and self-healing...

Optical Fibre is very good for running in 'loops', and the simplest arrangement being 'counter rotating rings'. If one segment is broken, traffic can still reach the other side by going the other way around the ring/loop.

The NBN install will be the main broadband infrastructure and "local-loop" for the next 30-50 years. It's an opportunity that can't be let slip. And one that the public deserves good value for its $43Bn.

Costs of NBN - National Broadband Network, Fibre-to-the-Home

2009-04-22T16:42:00.012+10:00

Connecting Australia with broadband/fibre systems is closer to implementing Cable TV than a new Phone network. The $43B estimate is 3-5 times too high based on Cable TV rollouts.

That difference completely changes the economics of the project.

The $43B NBN estimate is for 90% houses connected, or ~$5,500 per house passed.

Stewart Fist estimated in an article, 'Pipe Dreams', in 1996, that it would cost $10Bn to roll-out cable TV to 66% of houses. That's not the 90% of the NBN, but indicates a factor of three difference in price.

Queensland Government in Project Vista ($550M) was looking at around $3,000/house, as was ConnectSEQ ($300M for '3 million people' - for perhaps 1M houses)

The Victorian Governments "Project Aurora" was estimating $1500/lot in a new subdivision, confirming the QLD ~$3000/house.

The Bis Shrapnel report to the ACCC in 2001 on "Telecommunications Infrastructure in Australia" cites:

TransActs' costs for fibre (phone & TV) and DSL services to 100,000 homes in Canberra as $80M - under $1,000 per house passed.
Optus HFC cable rollout (95-97) at $3,000M for 2.25M houses passed, and
Telstra's cable (94-97) $4,000M for 2.5M houses passed (underground cabling)

Circa 1995, costs were under $2,000/house for either aerial (Optus) or underground (Telstra).

In the Telstra/Optus roll-out, there's an 80% duplication factor. Optus, who started second, used a duplication factor of 10-15% in their Business Case. The inference I draw is they did not cause the massive, irrational and uneconomic duplication.

Elsewhere, a government report notes that years later Telstra wrote-down its cable network by ~$960M and Optus by $1.4B. There was also considerable litigation by Optus is gaining access to Foxtel content on its network. (Telstra had originally gained an exclusive distribution agreement.)

Moore's Law, mass manufacture and advances in lasers & optical fibre/jointing/laying since 1995 should've brought down the equipment & install costs 2-5 times in real terms.
Widespread use & availability of horizontal boring should've decreased underground laying costs by a large margin. In 1994, it was considered a novel technique in Australia.

Neighbourhood Cable offers cable TV and 50Mbps in Mildura, Ballarat and Geelong. One article claims it cost $65M for 32,000 homes in Ballarat: $2,000/house.

Wikipedia lists a number of existing fibre-to-the-home deployments in Australia.

Summary:
The economics have to work if there are already companies providing FTTH services.

Historical costs of large scale cable rollouts are seriously at odds (by factor of 3-4) with the estimated $43Bn.

Something is wrong, but nobody is saying why the NBN estimates are so high.
What is being bought for the additional $25-$30B apparent overspend??

Canberra Region initiatives.

Around 2000, Adrian Blake ran an ISP in Cooma (2,600 houses, 300 businesses). He paid AGL to include 80km of plastic pipe in the trenches when gas was laid. (Fibre was to be later pulled in.) Promised funding failed to materialise when the 'dot boom' turned into the 'dot bust' and the ISP went into liquidation. The empty pipe asset was finally purchased by Telstra but not used to deliver domestic services.

TransACT rolled out a Fibre-to-the-node cable TV, broadband and telephone network over the aerial Electricity poles owned by its parent company, ACTEW (later merged with AGL). The key point of difference was 50Mbps VDSL over the short copper tails to support streaming TV and broadband.
In 1998, TransACT started a trial in the suburb of Aranda and was to fully deploy its network around 2000. It too was caught up by funding problems related to the 'dot bust'. It seems the rollout stopped in 2003.
Robin Eckermann was the original TransACT Chief Architect

In 1995, Telstra announced it would spend $20-30M laying 'fibre-to-the-kerb' in the new town of Gungahlin. Anthony Goonan, General Manager for Strategic Marketing for Telstra in Canberra, was interviewed by the ABC about the project at the time. The initiative seems to have sunk without trace after the announcement...

The ACT Government in 2004 ran a major project upgrading its internal network to 200 sites around the Territory over a private fibre network. Later upgrades included moving its 12,000 telephones from Telstra to VoIP running on this backbone.

ICON (Intra Government Communications Network) provides point-to-point dark fibre between Australian Government agencies around Canberra. ICON started in the mid-90's by DFAT (Foreign Affairs and Trade) and Defence for links between their systems. Management has moved from DFAT to Dept. Finance and Administration.

Death by Success

2009-04-19T13:59:00.006+10:00

Being too successful leads to failure unless you are aware of the problem and carefully monitor and protect against it. Every large company faces this problem - in I.T. for example, Google and Microsoft.

The rule applies at all levels - individuals, projects/teams, companies and countries. Even potentially our species and the whole living planet.

I was reminded again of this yesterday by Victor Cook in "The New Battle for Your Desktop" where he analyses the financials of Google and Microsoft (GOOG & MSFT). He describes a problem with Microsoft:

The company actually captured 77.5% of combined revenues in the quarter ended June 30, 2007.

Sometimes it's great to be the market leader. But in this case it's not. Microsoft shelled out $539 million for that 77th share point. And it was worth only $172 million after total costs. Not a good thing. To maximize earnings (by optimizing total costs) the company should have captured only 60.3% of the market. That's the point where the marginal cost and earnings per share point were exactly equal.
Microsoft's earnings were $3.991 billion at its actual revenue share of 77.5%. At the optimum level of competitive spending Microsoft's revenue share should be 60.3%, generating earnings of $6.271 billion. The difference is the $2.3 billion in theoretical earnings that management threw down the drain in the 2nd quarter of 2007.

Cook analyses Google in the next piece in the series and concludes they'd been under-spending into 2007, but were correcting it. (Piece is "Blue vs. Red Ocean Earnings Productivity")

Economics 101 - or What's behind this apparent contradiction?

If Microsoft had 100% of the market, a pure monopoly, wouldn't that maximise income and hence profits?

There are three issues to address in there:

income isn't profit. (Profit = Income - Expenses)
Max income may cause a loss if the expenses in earning it are too high.
Maximum profit occurs when the cost of the last dollar of income earned equals the cost of gaining that dollar (as in sales/marketing/production&distribution).
MR = MC. (Marginal Revenue = Marginal Cost)
Monopolies have a different problem, as there are no 'substitutes' for their products and not other players in their market.
Do they charge a single-price or not?
Effectively this creates a series of markets which compete at the change-over price.

People won't pay 'anything' for a good that's not necessary to life: there is a Price/Demand curve. A small number of people will pay a high price, and everyone will have it 'for free'. The trick is knowing the shape of the Demand Curve (because that's external) and matching that to your Supply/Cost curve (you should know this, its internal).

Wikipedia has good introductory pieces on these topics:
Marginal Revenue, Marginal Cost, Profit Maximisation, Price Elasticity of Demand and Mirco-Economics. A piece on Monopolies discusses the "The single price monopoly profit maximization problem" - which states that price should be twice the production cost (MR = 2 * MC).
It also introduces the notions of "Consumer Surplus", "Producer Surplus" and "Deadweight Loss".

Note that while the Profit Maximisation rule is invariant - it applies to all operating companies - it is just one area needing close attention/management in very large, successful companies.

Microsoft has been operating below its Optimum Profit level for a time - ignoring this ecomonic law will lead to financial disaster. The only question is "how long before they notice and correct?"

Google likewise has to attend to this same problem. It's in a better position - under-spending - but by how much can they afford to under-perform?

References for "Death by Success"

I haven't found a book that documents the causes of the 2007/8 Global Financial Crisis in these terms. The financial manipulation at the root - knowingly making bad loans (sub-prime mortgages) and passing them off to others - is obviously a Ponzi or Pyramid scheme if phrased this way. Kept small, it would've been a blip... Allowed to become mega-successful, it has come close to bringing down the global financial system

Jarrod Diamonds' book, "Collapse: How Societies Choose to Fail or Succeed"
documents this for societies.

Sydney Finkelstein, in "Why Smart Executives Fail: And What You Can Learn from Their Mistakes" does it for companies and their CEO's.

Jerry Weinberg teaches the rule at the Project/Team level in simulations within "Problem Solving Leadership" and is probably in his extensive writing. "Becoming a Technical Leader: An Organic Problem-Solving Approach" is the PSL textbook.

There's enough media coverage of self-destructing celebrities, sports stars and the newly rich to illustrate the idea.
Waldroop and Butler's "The 12 Bad Habits That Hold Good People Back: Overcoming the Behavior Patterns That Keep You From Getting Ahead" makes a good start for self study.

SteveJ-on-IT

Smart Internet+Smart-Grid: making money and reducing carbon footprint

Microsoft Troubles XIII: "Business Insider" articles

Apple needs to invent 'The Brick': screenless high-performance CPU, Graphics and Storage

Surprises reading up on RAID and Disk Storage

Articles

The importance of Design Rules

QUPSRSTCO: Software Design has more dimensions than 'Functionality'

A new inflection point? Definitive Commodity Server Organisation/Design Rules

An App too far? Can Windows-8 gain enough traction.

Why Apple won't add peer-peer to iCloud

Apple iCloud and peer-peer (Torrents)

A tablet from Microsoft? It won't save the company.

Microsoft Troubles XII: IBM market cap re-overtakes MSFT

Microsoft Troubles XI: APPL more profitable than MSFT

Why Microsoft is doomed to Commercial Failure.

Microsoft Troubles X: Ballmer as CEO being questioned

Why new Secure Internet solutions are technically Hard

Quality and Excellence: Two sides of the same coin

Business Metrics and "I.T. Event Horizons"

Top Computing Problems

Internetworking protocols

Flexible, Adaptable Hardware Organisations

Datacentre Hardware organisation

What can you learn from a self-proclaimed "World's Greatest"?

Introduction

Why

Why?

Audience

Format

The BIG Questions

Background

Principles of System Admin

Some WGSA Attributes

About The WGSA

Sayings of the WGSA.

Some Sound Management Laws

So What?

How do you work with a "World's Greatest ..."

Some "Good Stuff" I learnt from friends.

Some of the WGSA's work

Summary

Questions and Comments, please.

Fri 30 Jul 2010 09:19:47 EST (to blogger)Wed Feb 1 19:17:47 EST 200602-Jul-98 (first version)

Microsoft Troubles - IX, the story unfolds with Apple closing in on Microsoft size.

Everything Old is New Again: Cray's CPU design

A Good Question: When will Computer Design 'stabilise'?

Promises and Appraising Work Capability and Proficiency

Australia and the Researchers' Workbench

Death by Success II

Death by Success

MMC - the Microsoft death blow for non-Enterprise markets

Why Microsoft is being left behind

ICT Productivity and the Failure of Australian Management

Microsoft Troubles - VIII, MS-Office challenged

Extracts:

Microsoft Troubles - VII, An Insiders View

My summary:

Commentary

Conclusion

Murdoch, Google and Microsoft

I.T. Failure == Corporate Failure

Microsoft Troubles - VI, First Words

Why Yet Another ReOrganisation won't improve the Public Service

When 'ping' fails

Rupert Murdoch - Fool or Genius?

Internet Changes Everything: Newspapers

Telco pricing and market 'price elasticity'

Aus High Speed Broadband: Barriers and Challenges

Why are we doing this?

Broadband Lost Opportunities: Gas and Broadband

Phone Companies vs Broadband

Telco Engineering vs Network Engineering

History

Differences in Networks and Approach.

Telco Engineering applied to Public Data Networks.

NBN - Powerplay or 'for real'?

Mircosoft can't write O/S code

High-Speed Broadband: Excess Costs and Opportunity Losses

Costs of NBN - National Broadband Network, Fibre-to-the-Home

Fri 30 Jul 2010 09:19:47 EST (to blogger)
Wed Feb 1 19:17:47 EST 2006
02-Jul-98 (first version)