What can you learn from a self-proclaimed "World's Greatest"?

Note: This document is copyright© Steve Jenkin 1998-2010. It may not be
reproduced, modified or distributed in any way without the explicit
permission of the author. [Which you can expect to be given.]

Lessons from the Worlds' Greatest Sys Admin - July 1998

Presented at SAGE-AU Conference, July 1998

Contents
Introduction
Background
Principles of System Admin
Some WGSA Attributes
About The WGSA
Sayings of the WGSA.
Some Sound Management Laws
So What?
How do you work with a "World's Greatest ..."
Some "Good Stuff" I learnt from friends.
Some of the WGSA's work
Summary

Microsoft Troubles - IX, the story unfolds with Apple closing in on Microsoft size.

Three pieces in the trade press showing how things are unfolding.

Om Malik points out that Intel and Microsoft fortunes are closely intertwined.
Jean-Louis Gassée suggests that "Personal Computing" (on those pesky Personal Computers) is downsizing and changing.
Joe Wilcox analyses Microsoft latest results and contrasts a little with Apple.

Everything Old is New Again: Cray's CPU design

I found myself writing, during a commentary on the evolution of SSD's in servers, that  large-slow-memory like Seymour Cray used (not cache), would affect the design of Operating Systems. The new scheduling paradigm:

Allocate a thread to a core, let it run until it finishes and waits for (network) input, or it needs to read/write to the network.

This leads into how Seymour Cray dealt with Multi-Processing, he used multi-level CPU's:

• There were Application processors, many bits, many complex features like Floating Point and other fancy stuff, but had no kernel mode features or access to protected regions of hardware or memory, and
• Peripheral Processors (PP's), really a single very simple, very high-speed processor, multiplexed to look like 10 small, slower processors that performed all kernel functions and controlled the operation of the Application Processors (AP's)

Not only did this organisation result in very fast systems (Cray's designs were the fastest in the world for around 2 decades), but very robust and secure ones as well: the NSA and other TLA's used them extensively.

The common received wisdom is that interrupt-handling is the definitive way to interface unpredictable hardware events with the O/S and rest of the system. That polling devices, the old-way, is inefficient and expensive.

Creating a fixed overhead scheme is more expensive in compute cycles than an on-demand, or queuing, system, until the utilisation rate is very high. Then the cost of all the flexibility (or Variety in W. Ross Ashby's Cybernetics term) comes home to roost.

Piers Lauder of Sydney University and Bell Labs improved total system throughput of a VAX-11/780 running Unix V8 under continuous full (student/teaching) load by 30% by changing the serial-line device driver from 'interrupt handling' to polling.

All those expensive context-switches went away, to be replaced by a predictable, fixed overhead.
Yes, when the system was idle or low-load, it spent a little more time polling, but marginal.
And if the system isn't flat-out, what's the meaning of an efficiency metric?

Dr Neil J Gunther has written about this effect extensively with his Universal Scaling Law and other articles showing the equivalence of the seemingly disparate approaches of Vector Processing and SMP systems in the limit of their performance.

My comment about big, slow memory changing Operating System scheduling can be combined with the Cray PP/AP organisation.

In the modern world of CMOS, micro-electronics and multi-core chips, we are still facing the same Engineering problem Seymour Cray was attempting to address/find an optimal solution to:

For a given technology, how do you balance maximum performance with the Power/Heat Wall?

More power gives you more speed, this creates more Heat, which results in self-destruction, the "Halt and Catch Fire" problem. Silicon junctions/transistors are subject to thermal run-away, as they get hotter, they consume more power and get hotter still. At some point that becomes a viscous cycle (positive feedback loop) and its game over. Good chip/system designs balance on just the right side of this knife edge.

How could the Cray PP/AP organisation be applied to current multi-core chip designs?

1. Separate the CPU designs for kernel-mode and Application Processors.
   A single chip needs only have a single kernel-mode CPU controlling a number of Application CPU's. With its constant overhead cost already "paid for", scaling of Application performance is going to be very close to linear right up until the limit.
   
2. Application CPU's don't have forced context switches. They roar along as fast as they can for as long as they can, or the kernel scheduler decides they've had their fair share.
   
3. System Performance and Security both improve by using different instruction sets and processor architectures for different applications. While a virus/malware might be able to compromise an Application, it can't migrate into the kernel unless it's buggy. The Security Boundary and Partitioning Model is very strong.
   
4. There doesn't have to be competition between the kernel-mode CPU and the AP's for cache memory 'lines'. In fact, the same memory cell designs/organisations used for L1/L2 cache can be provided as small (1-2MB) amounts of very fast direct access memory. The modern equivalent of "all register" memory.
   
5. Because the kernel-mode CPU and AP's don't contend for cache lines, each will benefit hugely in raw performance.
   Another, more subtle, benefit is the kernel can avoid both the 'snoopy cache' (shared between all CPU's) and VM systems. It means a much simpler, much faster and smaller (= cooler) design.
   
6. The instruction set for the kernel-mode CPU will be optimised for speed, simplicity and minimal transistor count. You can forget about speculative execution and other really heavy-weight solutions necessary in the AP world.
   
7. The AP instruction set must be fixed and well-know, while the kernel-mode CPU instruction set can be tweaked or entirely changed for each hardware/fabrication iteration. The kernel-mode CPU runs what we'd now call either a hypervisor or a micro-kernel. Very small, very fast and with just enough capability. A side effect is that the chip manufacturers can do what they do best - fiddle with the internals - and provide a standard hypervisor for other O/S vendors to build upon.

Cheaper, Faster, Cooler, more robust and Secure and able to scale better.

What's not to like in this organisation?

A Good Question: When will Computer Design 'stabilise'?

The other night I was talking to my non-Geek friend about computers and he formulated what I thought was A Good Question:

When will they stop changing??

This was in reaction to me talking about my experience in suggesting a Network Appliance, a high-end Enterprise Storage device, as shared storage for a website used by a small research group.
It comes with a 5 year warranty, which leads to the obvious question:

will it be useful, relevant or 'what we usually do' in 5 years?

I think most of the elements in current systems are here to stay, at least for the evolution of Silicon/Magnetic recording. We are staring at 'the final countdown', i.e. hitting physical limits of these technologies, not necessarily their design limits. Engineers can be very clever.

The server market has already fractioned into "budget", "value" and "premium" species.
The desktop/laptop market continues to redefine itself - and more 'other' devices arise. The 100M+ iPhones, in particular, already out there demonstrate this.

There's a new major step in server evolution just breaking:

Flash memory for large-volume working and/or persistent storage.
What now may be called internal or local disk.

This implies a major re-organisation of even low-end server installations:

Fast local storage and large slow network storage - shared and reliable.

When the working set of Application data in databases and/or files will fit on (affordable) local flash memory, response times improve dramatically because all that latency is removed. By definition, data outside the working set isn't a rate limiting step, so its latency only slightly affects system response time. However, throughput, the other side of the Performance Coin, has to match or beat that of the local storage, or it will become the system bottleneck.

An interesting side question:
 How will Near-Zero-Latency local storage impact system 'performance', both response times (a.k.a. latency) and throughput.

I conjecture that both system latency and throughput will improve markedly, possibly super-linearly, because one of the bug-bears of Operating Systems, the context switch, will be removed. Systems have to expend significant effort/overhead in 'saving their place', deciding what to do next, then when the data is finally ready/available, to stop what they were doing and start again where they left off.

The new processing model, especially for multi-core CPU's, will be:

Allocate a thread to a core, let it run until it finishes and waits for (network) input, or it needs to read/write to the network.

Near zero-latency storage removes the need for complex scheduling algorithms and associated queuing. It improves both latency and throughput by removing a bottleneck.
It would seem that Operating Systems might benefit from significant redesign to exploit this effect, in much the same way that RAM is now large and cheap enough that system 'swap space' is now either an anachronism or unused.

The evolution of USB flash drives saw prices/Gb halving every year. I've recently seen 4Gb SDHC cards at the supermarket for ~$15, whereas in 2008, I paid ~$60 for USB 4Gb.

Rough server pricing for RAM in 2010 is A$65/Gb ±$15.
List prices by Tier 1/2 vendors for 64Gb SSD is $750-$1000 (around 2-4 times cheaper from 'white box' suppliers).
I've seen this firmware limited to 50Gb to improve performance and reliability comparable to current production HDD specs.
This is $12-$20/Gb, depending on what base size and prices used.

Disk drives are ~A$125 for 7200rpm SATA and $275-$450 for 15K SAS drives.
With 2.5" drives priced in-between.
Ie. $0.125/Gb for 'big slow' disks and $1 per GB for fast SAS disks.

Roll forward 5 years to 2015 and 'SSD' might've doubled in size three times, plus seen the unit price drop. Hard disks will likely follow the same trend of 2-3 doublings.
Say SSD 400Gb for $300: $1.25/Gb
2.5" drives might be up to 2-4Tb in 2015 (from 500Gb in 2010) and cost $200: $0.05-0.10/Gb
RAM might be down to $15-$30/Gb.

A caveat with disk storage pricing: 10 years ago RAID 5 became necessary for production servers to avoid permanent data loss.
We've now passed another event horizon: Dual-parity, as a minimum, is required on production RAID sets.

On production servers, price of storage has to factor in the multiple overheads of building high-reliability storage (redundant {disks, controllers, connections}, parity and hot-swap disks and even fully mirrored RAID volumes plus software, licenses and their Operations, Admin and Maintenance) from unreliable parts. A problem solved by electronics engineers 50+ years ago with N+1 redundancy.

Multiple Parity is now needed because in the time taken to recreate a failed drive, there's a significant chance of a second drive failure and total data loss. [Something NetApp has been pointing out and addressing for some years.] The reason for this is simple: the time to read/write a whole drive has steadily increased since ~1980. Recording density (bits per inch) times areal density (tracks per inch) have increased faster than read/write speeds, roughly multiplying recording density times rotational speed.

Which makes running triple-mirrors a much easier entry point, or some bright spark has to invent a cheap-and-cheerful N-way data replication system. Like a general use Google File System.

Another issue is that current SSD offerings don't impress me.

They make great local disk or non-volatile buffers in storage array, but are not yet, in my opinion, quite ready for 'prime time'.

I'd like to see 2 things changed:

• RAID-3 organisation with field-replaceable mini-drives. hot-swap preferred.
• PCI, not SAS or SATA connection. I.e. they appear as directly addressable memory.

This way the hardware can access flash as large, slow memory and the Operating System can fabricate that into a filesystem if it chooses - plus if it has some knowledge of the on-chip flash memory controller, it can work much better with it. It saves multiple sets of interfaces and protocol conversions.

Direct access flash memory will be always be cheaper and faster than SATA or SAS pseudo-drives.

We would then see following hierarchy of memory in servers:

• Internal to server
• L1/2/3 cache on-chip
• RAM
• Flash persistent storage
• optional local disk (RAID-dual parity or triple mirrored)
• External and site-local
• network connected storage array, optimised for size, reliability, streaming IO rate and price not IO/sec. Hot swap disks and in-place/live expansion with extra controllers or shelves are taken as a given.
• network connected near-line archival storage (MAID - Massive Array of Idle Disks)
• External and off-site
• off-site snapshots, backups and archives.
  Which implies a new type of business similar to Amazon's Storage Cloud.

The local network/LAN is going to be ethernet (1Gbps or 10Gbps Ethernet, a.k.a 10GE), or Infiniband if 10GE remains very expensive. Infiniband delivers 3-6Gbps over short distances on copper, external SAS currently uses the "multi-lane" connector to deliver four channels per cable. This is exactly right for use in a single rack.

I can't see a role for Fibre Channel outside storage arrays, and these will go if Infiniband speed and pricing continues to drop. Storage Arrays have used SCSI/SAS drives with internal copper wiring and external Fibre interfaces for a decade or more.
Already the premium network vendors, like CISCO, are selling "Fibre Channel over Ethernet" switches (FCoE using 10GE).

Nary a tape to be seen. (Hooray!)

Servers should tend to be 1RU either full-width or half-width, though there will still be 3-4 styles of servers:

• budget: mostly 1-chip
• value: 1 and 2-chip systems
• lower power value systems: 65W/CPU-chip, not 80-90W.
• premium SMP: fast CPU's, large RAM and many CPU's (90-130W ea)

If you want removable backups, stick 3+ drives in a RAID enclosure and choose between USB, firewire/IEEE 1394, e-SATA or SAS.

Being normally powered down, you'd expect extended lifetimes for disks and electronics.
But they'll need regular (3-6-12 months) read/check/rewrite cycling or the data will degrade and be permanently lost. Random 'bit-flipping' due to thermal activity, cosmic rays/particles and stray magnetic fields is the price we pay for very high density on magnetic media.
Which is easy to do if they are kept in a remote access device, not unlike "tape robots" of old.
Keeping archival storage "on a shelf" implies manual processes for data checking/refresh, and that is problematic to say the least.

3-5 2.5" drives will make a nice 'brick' for these removable backup packs.
Hopefully commodity vendors like Vantec will start selling multiple-interface RAID devices in the near future. Using current commodity interfaces should ensure they are readable at least a decade into the future. I'm not a fan of hardware RAID controllers in this application because if it breaks, you need to find a replacement - which may be impossible at a future date. (fails 'single point of failure' test).

Which presents another question using a software RAID and filesystem layout: Will it still be available in your O/S of the future?
You're keeping copies of your applications, O/S, licences and hardware to recover/access archived data, aren't you? So this won't be a question... If you don't intend to keep the environment and infrastructure necessary to access archived data, you need to rethink what you're doing.

These enclosures won't be expensive, but shan't be cheap and cheerful:

Just what is your data worth to you?

If it has little value, then why are you spending money on keeping it?
If it is a valuable asset, potentially irreplaceable, then you must be prepared to pay for its upkeep in time, space and dollars. Just like packing old files into archive boxes and shipping them to a safe off-site facility cost money, it isn't over once they are out of your sight.

Electronic storage is mostly cheaper than paper, but it isn't free and comes with its own limits and problems.

Summary:

• SSD's are best suited and positioned as local or internal 'disks', not in storage arrays.
• Flash memory is better presented to an Operating System as directly accessible memory.
• Like disk arrays and RAM, flash memory needs to seamlessly cater for failure of bits and whole devices.
• Hard disks have evolved to need multiple parity drives to keep the risk of total data loss acceptably low in production environments.
• Throughput of storage arrays, not latency, will become their defining performance metric.
  New 'figures of merit' will be:
• Volumetric: Gb per cubic-inch
• Power: Watts per Gb
• Throughput: Gb per second per read/write-stream
• Bandwidth: Total Gb per second
• Connections:  Number simultaneous connections.
• Price: $ per Gb available and $ per Gb/sec per server and total
• Reliability: probability of 1 byte lost per year per Gb
• Archive and Recovery features: snapshots, backups, archives and Mean-Time-to-Restore
• Expansion and Scalability: maximum size (Gb, controllers, units, I/O rate) and incremental pricing
• Off-site and removable storage: RAID-5 disk-packs with multiple interfaces are needed.
• Near Zero-latency storage implies reorganising and simplifying Operating Systems and their scheduling/multi-processing algorithms. Special CPU support may be needed, like for Virtualisation.
• Separating networks {external access, storage/database, admin, backups} becomes mandatory for performance, reliability, scaling and security.
• Pushing large-scale persistent storage onto the network requires a commodity network faster than 1Gbps ethernet. This will either be 10Gbps ethernet or multi-lane 3-6Gbps Infiniband.

Which leads to another question:

What might Desktops look like in 5 years?

---

Other Reading:
For a definitive theoretical treatment of aspects of storage hierarchies, Dr. Neil J Gunther, ex-Xerox PARC, now Performance Dynamics, has been writing about "The Virtualization Spectrum" for some time.

Footnote 1:
Is this idea of multi-speed memory (small/fast and big/slow) new or original?
No: Seymour Cray, the designer of the world's fastest computers for ~2 decades, based his designs on it. It appears to me to be a old idea whose time has come again.

From a 1995 interview with the Smithsonian:

SC: Memory was the dominant consideration. How to use new memory parts as they appeared at that point in time. There were, as there are today large dynamic memory parts and relatively slow and much faster smaller static parts. The compromise between using those types of memory remains the challenge today to equipment designers. There's a factor of four in terms of memory size between the slower part and the faster part. Its not at all obvious which is the better choice until one talks about specific applications. As you design a machine you're generally not able to talk about specific applications because you don't know enough about how the machine will be used to do that.

There is also a great PPT presentation on Seymour Cray by Gordon Bell entitled "A Seymour Cray Perspective", probably written as a tribute after Cray's untimely death in an auto accident.

Footnote 2:
The notion of "all files on the network" and invisible multi-level caches was built in 1990 at Bell Labs in their Unix successor, "Plan 9" (named for one of the worst movies of all time).
Wikipedia has a useful intro/commentary, though the original on-line docs are pretty accessible.

Ken Thompson and co built Plan 9 around 3 elements:

• A single protocol (9P) of around 14 elements (read, write, seek, close, clone, cd, ...)
• The Network connects everything.
• Four types of device: terminals, CPU servers, Storage servers and the Authentication server.

Ken's original storage server had 3 levels of transparent storage (in sizes unheard of at the time):

• 1Gb of RAM (more?)
• 100Gb of disk (in an age where 1Gb drives where very large and exotic)
• 1Tb of WORM storage (write-once optical disk. Unheard of in a single device)

The usual comment was, "you can go away for the weekend and all your files are still in either memory or disk cache".

They also pioneered permanent point-in-time archives on disk in something appearing to the user as similar to NetApp's 'snapshots' (though they didn't replicate inode tables and super-blocks).

 My observations in this piece can be paraphrased as:

• re-embrace Cray's multiple-memory model, and
• embrace commercially the Plan 9 "network storage" model.

Promises and Appraising Work Capability and Proficiency

Max Wideman, PMI Distinguished Contributor and Person of the Year and Canadian author of several Project Management books plus a slew of published papers, not only responded to, and published, some comments and conversations of between us, he then edited up some more emails into a Guest Article of his site.

Many thanks to you Max for all your fine work and for seeing something useful in what I penned.

Australia and the Researchers' Workbench

This is a pitch for something new: the "Researchers' Workbench".
Australia has the wealth and inventiveness to do it, but most probably, not the political will.
Chalk that up to "the Cultural Cringe".

Death by Success II

There is another, much more frequent "Death by Success" cause, first introduced to me by Jerry Weinberg and Wayne Strider and Elaine Cline (Strider and Cline).

It's the same process that some herbicides use: unconstrained growth.
Monsanto's flagship herbicide Round Up is exactly this sort of agent.

If you are very good at what you do and much sought after, this can lead directly to massive Failure - personally and in business.

Growth is Good, but too much, too fast is a Killer.

The only protection is awareness.
As  Virginia Satir pointed out, "We can't see inside other people's heads, nor can we see ourselves as others see us" (courtesy again of Jerry and "Strider and Cline".)

Typically you need objective, external help is recognising this condition.
Once you have restored Situational Awareness, you can choose your response. Which may be "I'm outa here", Denial or something in between.

There is an alternative form of "Death by Success", which again we see in the Plant Kingdom.

Your initial approach, solution or technique may not Scale-Up or have a fixed Upper-Bound.
E.g. if you sell "factory seconds", there is a limited supply that sets your maximum turnover.
Or selling fragments of the Berlin Wall - at some point the Genuine Article is all gone...

The example in the Plant Kingdom are when tree seedlings 'set' in unsuitable places, like a small pot or within a bottle. Down the road, they will become "root bound", which slows growth, then they'll consume all the nutrients and having converted 'everything' into plant material, die.

That's it for that plant - all of one resource has been exhausted and it's Game Over.

Death by Success

The things you do in the beginning, when you're the minnow-against-the-giants, to start and build a business may not work well when you're successful, when you've become The Giant.

Exactly what leads to Success can eventually lead to your downfall.

You become very good at the things that have gained and seemingly maintained Success.  Every problem and challenge you've met have been solved with your brilliance and individual style.

Why would you ever want or need to vary that approach?

Until something new comes along and it all goes wrong:
  Inevitably in Business and Life, things change (perturbations arise in Control Systems terms).
  Responding with "More of the Same", as in the past, will, at some point, not work.
  If you've grown large, it will take time to fail, you'll have notice "things aren't great".
  Many companies only ever do "More of the Same",  often amping-it-up as results don't appear.
  The results are as predictable are throwing oil on a fire.

Often I mention Sydney Finkelstein's book, "Why Smart Executives Fail" in which Finkelstein describes the results of 6 years of research.  He self-describes as "Steven Roth Professor of Management at the Tuck School of Business at Dartmouth College, where I teach courses on Leadership and Strategy".

In Smart Executives, Finkelstein and his team documents a whole slew of companies (50) that burned bright and collapsed. This book was published in 2003, covering a turbulent period of US and global business, as well as some famous cases going back decades.

The subjects of the research were chosen precisely because they were wildly successful and suffered a notable collapse. Enron and Worldcom are on the list, plus many I.T. companies such as Wang Computers.  The common thread is the collapse was avoidable and predictable.

Would the conclusions, Lessons Learned and "Early Warning Signs" be different post the 2008 GFC (Global Financial Crisis)?  I think not...

Finkelstein lists 7 naive causes of failure:

1. The Executive were Stupid.
2. The Executives couldn't have known What was Coming.
3. It was a Failure to Execute.
4. The Executives weren't trying Hard Enough.
5. The Executives lacked Leadership Ability.
6. The Company lacked the Necessary Resources.
7. The Executives were simply a Bunch of Crooks.

and comments in a para entitled "Failure to understand Failure":

All seven of these standard explanations for why executives fail are clearly insufficient. (Because the companies had demonstrated excellence in becoming highly successful.)

The next 300 pages are his answer. Part I describes "Great Corporate Failures" and Part II their Causes.
This research ends with a positive message, Part III is "Learning from Mistakes":

• Predicting the Future, Early Warning Signs.
• How Smart Executives Learn, Living and Surviving in a World of Mistakes.

His "Seven Habits of Spectacularly Unsuccessful People"  are worth reiterating:

1. They see themselves and their companies as dominating their environments.
2. They identify so completely with the company that there is no clear boundary between their personal interests and their corporation's interests.
3. They think they have All the Answers.
4. They ruthlessly eliminate anyone who isn't 100% behind them.
5. They are consummate company spokespersons, obsessed with the company image.
6. They underestimate major obstacles.
7. They stubbornly rely on what worked for them in the past.

Each of the 11 chapters has 30-50 references.  Although written and published for the general market, this isn't any "Puff piece".

MMC - the Microsoft death blow for non-Enterprise markets

MMC, "Mostly Macintosh Compatible", the equivalent for OS/X of WINE for Windows, doesn't yet exist, that I'm aware of.

Why Microsoft is being left behind

Paul Budde recently questioned, "Will Microsoft be able to make the jump?"
[04-Apr-2010] For other comments see my pieces "Death by Success" and "Death by Success II".

He quotes the marketing "S-curve" and Summer Players by Carol Velthuis describing company performance and market maturity in seasons of the year.

ICT Productivity and the Failure of Australian Management

Prior Related Posts:
Quantifying the Business Benefits of I.T. Operations
The Triple Whammy - the true cost of I.T. Waste
Force Multipliers - Tools as Physical and Cognitive Amplifiers
I.T. in context

Alan Kohler and Robert Gottleibsen have been writing in "Business Spectator" about the relationship between jobs and Economic Productivity.

They note that the USA has improved productivity in the last year while in Australia it has declined (+4% and -3% respectively).  My take on this is: a gross Failure of Australian Management.

There is solid research/evidence that "ICT" is the single largest contributor to both partial and multi-factor Productivity, and is expected to be so for the next 20 years.  This is an big issue.

Microsoft Troubles - VIII, MS-Office challenged

"Microsoft Office is obsolete, or soon will be" By Joe Wilcox.

I hadn't picked this trend, it's quite important.
It squeezes their 2nd "birthright" (the other is the PC Operating System, I'd focussed on.)

Microsoft Troubles - VII, An Insiders View

A friend sent me this link to a New York Times Op-Ed 'contribution'.
Huge news...

February 4, 2010
Op-Ed Contributor
Microsoft’s Creative Destruction
By DICK BRASS
Dick Brass was a vice president at Microsoft from 1997 to 2004.

This guy was a VP in the glory years - either side of Y2K, and before the 2004/5 Longhorn 'reset'.
The failure to build the successor to XP was a breaking-point: the forced upgrade cycle was gone.

He's likely to have a bunch of stock, or options, and a vested interest in the company's success/survival. His comments are likely to be both informed and as positive as they can be...

Murdoch, Google and Microsoft

Rupert Murdoch anti-Google strategy seems to becoming clear. Previously, I was puzzled at his public stance, it seems to make no sense.

This Register piece, and there are many other sources, starts with:

Rupert Murdoch is in talks with Microsoft over his plans to delist his newspaper websites from Google.

Classic "My enemy's enemy is my Friend" thinking.
It's not really so, they are only a temporary ally at best with a fragile common cause.

The danger lies in your "friend" deciding they don't need you anymore or worse, turning on you once the main enemy is gone or the battle is lost.

I still think Murdoch's strategy is seriously off the mark.

I.T. Failure == Corporate Failure

Stephen Bartholomeusz writing in Business Spectator, 18 Nov 2009, on the ASIC court case over the collapse of One.Tel.

Bartholomeusz neatly summarises the root cause of the failure:

Unhappily, its billing systems didn’t work, so it piled up debtors, while its competitors responded to the cut-price strategies.

He goes on to say:

While professing publicly that the group was on-track to be cash-positive..., internally One.Tel appears to have had little control or understanding of its cash flows or the mounting issues created by its billing system

and finishes:

Whatever Rich might claim, One.Tel wasn't a successful company, unless success is measured by revenue, not cash flows or execution.

This is the first case I've noticed where the immediate cause of failure of a large, public company  has been it's I.T. systems. The root cause is poor management with an inability to execute - or to understand and control it's I.T.

The field of "Software Engineering" is 40 years old now.
How could this foreseeable and preventable failure have happened with competent professionals, especially if Software Engineering had achieved it's aims?

There is a multiple tragedy hidden here:

• Software Engineering has failed to impress it's primary market: Business Management.
• Educators and Researchers are not, as a matter of course, going to analyse this failure and use it as a case study. Compare the 1974 explosion at Flixborough or the 1970 collapse of the Westgate bridge during construction.
• IT practitioners aren't going to be informed by their Professional Societies of the causes and preventing a recurrence.
• Business Management and I.T. practice remains "Consequence Free".

If a Billion Dollar Failure isn't a notable event and worthy of preventing recurrence, then what is?
Why are ASIC, the ASX and the Federal and State Governments silent on this point?
If not their job, then whose?

Imagine if QANTAS had a fire at a maintenance facility and lost $1B of buildings, plant and equipment. You know absolutely the company, multiple regulators and all the professional bodies would actively investigate the matter.

They would be looking for "root causes" of this event, other problems, ways to fix the system, processes & procedures to prevent or early-detect this class of problem again and co-incidentally if any individuals were responsible. Not just front-line grunts, but if anyone in management  (up to the CEO) was culpable, negligent, incompetent or asleep-at-the-wheel.

The absolute tragedy here is not the loss to these investors (employees, vendors, customers, ...) but that nothing is going to change, that this massive loss bought nothing.

What is more galling to me is that nobody in the Press, Government, ASX, Investment bodies, Judicary or Regulators thinks anything more could or should be done...

Microsoft Troubles - VI, First Words

"On the latter, Microsoft is hoping Windows 7 will pull it out of a financial hole" by Charles Arthur, Sydney Morning Herald, Nov 5, 2009. First time I've seen in popular press the actual words "financial hole" w.r.t. Microsoft.

Previous piece - "Microsoft Troubles V"

Why Yet Another ReOrganisation won't improve the Public Service

The Rt. Hon. Ken Rudd PM has suggested on the News that he'll be seeking to improve the Federal Public Service. There's talk of a special Centre at the ANU to train people up too.

Rudd might end up with a bunch of tests, metrics and new programs & processes, but I can guarantee it won't amount to a hill 'o beans. The one thing known about Bureaucracies is their ability to Resit Change.

Read C. N. Parkinson ("Parkinsons Law" etc) for a view from the 1950's and some definitive economic analysis of the ultimate Bureaucracy: The UK's Ministry of Defence. After WWI, ships and fighting men - the essence of the Navy - declined dramatically. The Bureaucracy 'running' them increased overwhelmingly...

Why? Because the primary purpose of Bureaucracies is themselves, not producing outcomes.

When 'ping' fails

In Networking, the 'performance objects' are links, usually end-to-end, consisting of many elements - like ethernet segments, switches, routers/firewalls, long-distance circuits and security scanner devices, laid on top of Telco/backbone services that provide dynamic and asymmetrical routes.

The most frequently used measure is 'ping' time - ICMP “echo request” packets, often 64 bytes long.
Firewalls, routers and underlying networks filter/block classes of traffic, implement traffic & flow rules and attempt "Quality of Service". There are many common rulesets in use:

• blocking 'ping' outright for 'security reasons'. Stops trivial network scanning.
• QoS settings for different traffic classes. VoIP/SIP gets high priority, FTP traffic is low, your guess on HTTP, DNS, time (NTP) and command line access - SSH, Telnet, ...
• traffic profiling on IP type (ICMP, UDP, TCP) and packet size (vs link MTU).
• traffic prioritisation based on source/destination.
  

Real-time voice needs many small packets, would like low latency/delay, and no jitter, and can stand some packet loss or data errors. FTP relies on TCP to detect packet loss & retransmit them. It likes big packets and attempts to increase bandwidth consumed through TCP 'fast-start' etc.

The only time 'ping' is accurate is within a simple ethernet segment - no rules, no firewalls, no 'traffic engineering', no link losses, no collisions, ...
Otherwise, it's a dangerous and potentially very misleading measure.

'Time of Flight' for UDP & ICMP packets is only measurable when you control both ends of the link and can match headers. Not something most people can or want to do.

TCP packets can be accurately timed - sessions are well identified, packets can be uniquely identified and they individually acknowledged. It is possible to accurately and definitively monitor & measure round-trip times and effective throughput (raw & corrected) of links and connections at any point TCP packets are inspected - host, router, firewall, MPLS end-point, ...
I'm not aware of this being widely used in practice, but that's a lack of knowledge on my part.

This is not a tutorial in Networking or TCP/IP.
Neither am I a Networking expert. I'm demonstrating that even with my knowledge, "tools ain't tools" (meaning not all tools and methods are equal),
and that using just 2 metrics, 'bandwidth' & 'latency' to characterise links is simplistic and fraught. As professionals, we have to be able to "dig under the covers" to diagnose & fix subtle faults.

Consider the case of TCP/IP over a noisy satellite phone link, the type you buy from Intelsat for ships or remote areas. The service notionally delivers a data channel of 64kbps, but is optimised for a digital voice circuit, not IP packets. The end-end link has per-bit low latency on/off the link, long transmission delays, nil jitter and limited error-correction (forward-error-correction (FEC), no retransmit) - nice for telephony. These links also have buckets of errors - which voice, especially simple PCM, easily tolerates & can even be smoothed or interpolated out with simple equipment - which will be there to handle echo cancellation.

Say you're on a ship at the end of one of these links.
People are complaining that email 'takes forever' and they can't download web pages.
You run a ping test out to various points - and everything is Just Fine.

What next?

The most usual response is 'Blame the Victim' and declare the link to be fine and 'the problem' to be too many people sending large messages, 'clogging up the link', and too much web surfing. You might set quotas and delete large emails. That might work, or at least improve things marginally.

Radio links, especially back-to-back ones, each crossing 36,000km to geostationary satellites, are noisy.
If the BER is 1:100000 (1:10^5, under the hoped for 1:10^6) and you're using the default ethernet MTU of 1500 bytes, you'll get an error every 1-2 seconds. No worries, eh?
1500 bytes = 12,000 bits = 5.3 packets/second. Or 1 in 10 packets. Hardly noticeable.
TCP/IP has a minimum overhead of 40bytes/packet (less with Van Jacobson compression).
The data payload per packet is 1460 bytes for the ethernet default MTU.

Sending a 1.25Mb file (10Mbit), that's ~856 raw sent packets and 103 errors, or ~12% retransmissions. Of those 103 resends, 12 get errors as well and are resent, and 1.5 errors of those go onto a 3d round...
Or ~115 errored packets, or 14% errors on raw packets. Just a minor problem.
There's a probability of >1 error per packet, but I don't have the maths to solve for that.

The effective bandwidth (throughput) of the link, using 970 * 1500-by packets to move 1Mb in 182 seconds, is 55kpbs. Quite acceptable.

What if there's a corroded connector or tired tracking gimbal and you get a 3db change in SNR and the BER doubles? (That's a guess, not science.)
856 raw packets and 250 1st round errors, 62 2nd, 15 3d, 4 4-th, 1 5th = 333 resends. An almost 50% increase in total number of packets needed to send the file. 223 seconds and 45kbps.
Doubling the BER again (4:10^5 or 0.004%) increases the 1st round error rate to 400 packets, or 47% - 750 retransmits in 10 rounds. 301 seconds and 33.1kpbs. Half-speed.

Back to the original BER, if you were running 'jumbo frames' (9000 by) locally & these went down the link as is, you get 0.8 packets/sec and have a 72% chance of an error in a packet. One in four of the packets would get through unscathed. 140 'jumbo frames' are sent raw, 350 packets are needed with 16 rounds of retransmission.
The file takes 400 seconds at 25kbps - a hefty penalty for forgetting to configure a switch.

The problem is that packet size amplifies error rates.
A change in BER of 0.001% to 0.004%, undetectable by the human ear, halves the throughput fo TCP/IP.
Using an MTU size of 168 (128 data + 40 TCP/IP overhead) gives good performance at a BER of 1:10^4, trading 25% protocol overhead for link robustness.

'ping', using default 512 bit packets, won't detect the error.
But who'd think the MTU was a problem when standard diagnostics were reporting 'all clear'?

Summary: In difficult conditions, the BER doesn't have to drop much for link throughput to significantly degrade.

This example is about simple-minded tools and drawing incorrect conclusions.
The Latency of the link was constant, but the Effective Bandwidth, throughput, changed because of noise or link errors.

Surely that proves the Myth: Latency & Bandwidth are unrelated.

Nope, it proves that link speed and throughput bear a complex relationship to each other.

If you had been measuring TCP/IP statistics (throughput & round-trip-time) at the outbound router, or using 'ping' with MTU of 1500, you'd have seen the average latency rising as throughput dropped. All those link-layer errors & subsequent retransmits were causing packets to take longer.

But a simple low-bandwidth radio link isn't the whole story.
It's a "thin, long pipe" in some parlances.
What was special about that link was:

• no link contention, data rate was guaranteed transfer rate.
  
• synchronous data

Rupert Murdoch - Fool or Genius?

Does Rupert Murdoch know something the rest of us don't?
The recent news is that News Ltd would start charging for on-line access to its newspapers.

Not a good idea.
Experienced Journalist & commentator, Alan Kohler also thinks so...

---

First, Rupert is not in the business of selling 'news', quality journalism or not.
He sells Advertising.
Just like Google and friends. But apparently not nearly as well as they do on-line.

There are people who sell 'news', and they are going strong.
Organisations like Reuters, Associated Press, Bloomberg, AAP, ... The wire-services.
The same ones that sell to Google, businesses, TV and Mr Murdoch's newspapers.

News Ltd doesn't sell journalistic content (news): like every major newspaper, it has always given away its content. Exactly the same as Free-to-Air radio and TV.

The "value proposition" to most newspaper customers, News & Stories, is a Free Good.
Major papers actually cost their publishers to sell. Newsagents typically keep the full "cover price" of the local major papers. Perhaps this is why Fairfax Ltd lists "Newsprint and Ink" as its single biggest expense.

Publishers make their money from the advertising they sell (Classified and 'Display' or general).
They set their advertising rates on the estimated number of readers - not copies sold/distributed. (There's a whole industry 'auditing' circulation & readership).

Small advertisers will always be 'price-takers', while the large regular advertisers can negotiate.

On-line breaks many/all the Newspaper assumptions:

• no intermediaries with good 'passing trade' to find customers
  
• exact counts, not estimates, or readership
• exact counts of advertiser hit-rates (count links followed)
  
• targeted/niche audiences, not "broad spectrum" mass market
  

Browsing the 1200+ entries for 'newspapers' in the Australian Yellow Pages, these groupings seem apparent:

Business, Trade & Industry, Lifestyle, Sports, Political, Special Interest, Community/Local, Regional & Rural, Ethnic, Language and Religious,

and versions of the "Trading Post".
A newspaper without content, pure advertising, the ideal for the business side of newspaper.

---

Second, a long time ago newspapers were the source for capital-N News - timely, important, factual.
They broke stories, 'scooped' one another, had many editions during the day and dealt in "the facts m'aam, just the facts" as Joe Friday might say.
The sort of thing shown in 1930's Black and White movies.

By the 1970's, newspapers had comprehensively lost the race as the first news source.
"Watergate" showed they could still 'scoop' other media with investigate journalism, but the Vietnam War played out on the nightly TV news.

When Ted Turner started CNN, the game changed - Free-to-Air was usurped.
The 1991 Gulf War had CNN "reporting live from Baghdad" and assumed the mantle of "first news source".

These days it is a tussle between Cable News and on-line services to be "first".
And that race has always led to problems with accuracy and false/fabricated stories.

An editor who is under pressure "to be first" can be manipulated into publishing without good fact checking. When there was considerable effort & expense in rolling the presses, the downside ensured more caution. In the on-line world, nearly all barriers to production are eliminated alongside "instant" publication. An editorial mistake is much more likely and potentially much more damaging to a large publisher, the Drudge Report non-withstanding.

Newspapers have been providing Opinion & Analysis for a couple of decades.
Any pretence they are cutting edge or breaking stories in real-time is a "fools paradise" and delusional.

News Ltd has great content produced by many great people and serves a faithful cohort of consumers. It just isn't 'news' they are selling.

---

Third, there are many good free alternatives for news, on-line and not, to newspapers.

Google pays for wire-services and gives away the content.
Publicly funded media - radio, TV and on-line - have a mandate to provide services with public monies. The BBC and Australian ABC have large news rooms and international reporters.
The ABC alone has 700 people in its News Division providing current content for all its outlets.

How does a newspaper, with at best 300 journalists, compete with a better resourced competitor who's content is free?

Not on news - only with other types of content and other incentives - like DVD's and special offers...

---

Fourth, there are just 3 workable Revenue Models.
Revenue options are:
subscription/donations and cover-price + advertising.
(pre-paid vs 2-part charging)

There are only 4 Revenue Models possible in this scheme.
Fully free can't self-support itself, so there are only 3 workable models.

Murdoch is complaining the Revenue Model that has worked well in the physical world for approaching a century doesn't work on the Internet. Who'd have thought?!

The wire-services thrive and on-line advertising is booming.
The only people out of step are the firms running newspapers.

They could have acted in 1995 to move their advertising on-line, but didn't.
Were they blinkered or lacked 'vision'?
Was it a sound business decision based in part on not canabalising their main cash flows?

Things are how they are...
There seems little to be gained from now analysing the reasons for non-action.

---

There are other issues that have to be resolved when moving to on-line services.

• Paper is simple and always "Just Works', modulo getting wet.
  Attempts to stream printed news electronically have been widely successful outside of offices. Radio serves the travelling public well. Printed media is cheap, available and can be forgotten without dire consequences. Some section of the population may read the news on their Kindle or iPhone on their morning commute, but it won't be a large audience.
  Neither will there be much call for $10 newspapers...
  
  
• Serving "The Diaspora": An important function of newspapers is allowing non-resident locals to "keep in touch with home". Australia shows that people may permanently emmigrate and never return home, but still identify strongly with their country of origin. This fuels our strong ethnic newspapers. For people who've only moved towns, a daily or weekly "fix" of their hometown newspapers fills a strong need. They even pay a premium.
  
  
• Niche buyers. Most buyers throw most of a newspaper away. They are very specific & selective in their needs and uses of the massive content provided. There are better ways to serve many of those niches on-line. Like classified advertising is better served by e-bay and 'trading post'. It's fast, current and cheap - plus very efficient for the reader. The service does the searching and the reader can be contacting a seller within minutes of loading the site.
  
  
• Network effects and the tipping point. When a product has reached around 40% market penetration, it 'suddenly' becomes popular and quickly saturates the market. This happened in 1984 with Group-3 fax and then around 1996 with The Internet/World Wide Web. Newspapers need to be keenly aware of their competitors - when the end comes, it may be frightenly fast.
  
  
• Copyright and Libel Laws. The journalists union has spent a very long time negotiating what rights the publisher & content-creator have. This all has to be done again in an on-line world. The other side of the coin is commercial protection of journalists against Libel or defamation actions. The publisher wears the risk once the editor decides to print. Those named know that a newspaper can afford to and will defend itself. If journalists are personally exposed to litigation, justified or not, they will sensibly withhold contensious pieces. Why wreck your life for a decade or more, as happened to Chris Masters over the "Moonlight State" and other pieces? For many, the price is too high.
  

---

Lastly, What would work?

This is an argument in three parts: as a society we need 'quality journalism', news rooms aren't cheap, and are there models we could follow?

The media as "The Fifth Estate" is an important and necessary part of any Democratic government. A Free Press is a necessary part of Open and Transparent government.

But whither Investigative Journalism. There is a lot of TV reportage of politicians doing 'door stops' or in stage-managed events. And a lot of 'tabloid journalism' on TV.

Would Woodward and Bernstein now be funded for their lengthy Watergate investigations?
Would any editor allow it to be published these days?
I think Watergate is less likely to be reported these days for many reasons and the Drudge Report and other gossip sources do not fill the gap.

"Quality Journalism" has to be nutured & supported for us to have stable, prosperous societies.

The ABC states it has 700 people in its News Division.
On-line sources suggest major newspapers have ~300 journalists in their newsrooms. [This information isn't in the Annual Reports I scanned.]

What would it cost to run such a news room?
$30M a year in wages, $10M in wire-services, $10-15M for bluidings and systems.
Marketing & Sales probably $30M. Accounting and collecting subscriptions: $5-10M.
Publishing on-line would add another $20-30M, with an overall 30% Gross Margin required to fund upgrades, depreciation and dividends.

Perhaps $150M/year in revenue, or $3M/week.
The Sydney Morning Herald has an audited circulation of 210-360,000 and readership from 850,000-1,100,000 [without SunHerald, $1.80 and 480,000/1.25M]
Previous comments: "Internet Changes Everything: Newspapers".

Even if you achieved 500,000 individual subscriptions, a weekly price of $5+, versus the $1.40/weekday and $2.40 Saturday for the paper version.
I believe that's far beyond the consumer 'price point' for a single publication.

So what models are out there that might work?
'Cable TV' provides content aggregation, common marketing services & subscription and billing.

It has severly impacted Free-to-Air TV over the last 4 decades for many reasons.
One of the big factors I believe is allowing content providers to focus on their strengths and the Cable Service Provider (Foxtel in Australia) to focus on the technical and retail/customer relations and support business.

The public are offered content aggregated into affordable and desirable 'packages'.
They can decide the utility to them of each package and compare the cost to other forms of entertainment. A$30-$50 seems to be the price point.

Cable TV for content providers removes barriers to entry and avoids competition between technical delivery methods. The customer wants the service and isn't interested in the technology per se. This model allows & promotes small, new entrants with serving specialist or highly targeted niches.

The revenue returned to content providers is unknown to me. Large studios are not 'price takers' and have significant negotiating power as 'headline products'.
Presumably small niche providers get a return based on consumer views.

This shared infrastructure and 'content packages' seems ideally suited to on-line delivery of paid content - which doesn't have to be limited to news or 'quality journalism', but certainly includes them. Plus we have the natural providers already operating with large, high-quality customer lists: Cable Service Providers.

The technical implementation for an "on-line Channel Service" is simple. Though the ABC iView experience suggests that collaborating with ISP's and allowing unmetered content is necessary. [Australian ISP's impose download quotas on broadband].

Customers already have some sort of PC (Windows, Mac, Linux, ...) and look after their own broadband connection.
A controlled, universal 'player' is required - happily companies like VMware already provide, free, a basic product that work across all platforms, the "VMware Player", which can run pre-built systems with embedded applications, "Virtual Appliances".

The only necessary work is tailoring a VPN or similar and distributing the required registration/connection keys. Foxtel already has the infrastructure in place to source, distrubute, service and support hardware & devices.

That's not a big leap...
And one where services can be packaged in a series of packages with many different price-points.
All those free Community papers, plus a Major Metro Papers: $5/month?
Add a speciality or trade paper, all the Major Metros, a financial services 'feed' and an alert service like 'Media Monitors' for a company of 75 people: $lots.

How does this proposal site with the Newspaper assumptions:

• intermediaries with good 'passing trade'
  Exactly what the Cable TV companies do.
  
  
• exact counts, not estimates, or readership
  Page counts and the precise subscriber unequivilacly identified & grouped.
  Near-perfect marketing information, and a perfect, undisputed source of revenue figures for 'page hits' revenue scheme.
  
  
• exact counts of advertiser hit-rates (count links followed)
  You don't have to sell advertising, and many content providers would not,
  But if, like the Trading Post, you did... Trivial.
  
  
• targeted/niche audiences, not "broad spectrum" mass market
  Tailored content per source, niche & specialist sources, remembered preferences and interests... Near perfect for subscribers and providers alike.
  
  
• Normal print-media space restrictions are lifted: Content providers can provide additional "in-depth" material easily & cheaply.
  
  
• Individual content providers can use the Channel Service to provide archive, search and print-on-demand services. Leaving each party doing what they do best.
  
  
• 'Leakage' of content can be controlled with the Virtual Appliance.
  It may be configured to only allow 10 pages a day to be printed... With extras purchased.
  
  
• Anyone interested in expensive periodicals, Academic Journals or hard-to-find books?
  With controlled access and clear charging regimes in place, there is no issue about denying or destroying copyright.
  In another day, this might have been called "Your Local Library".
  

On top of this, additional options allowing subscribers to pre-pay to view or print normally inaccessible content. The Channel Service Provider doesn't become a credit provider - in fact gains by holding the prepaid money - which it never need return and might even expire, like pre-paid mobiles.

Importantly, 'micro-payments' are avoided. They are very, very hard to get right and consequentially expensive. That's why we've never seen Visa and Mastercard move on this market.

But moving 1cent 'funny money' from your pre-paid balance to a vendor - very cheap.
It's the basis of prepaid mobiles.
With the business-friendly upside of all the unused payments that expire - a tidy 5-10% profit.

---

In summary: Do I think Murdoch is wrong-headed in charging for access to his newspapers?
Absolutely.

Do I think an on-line service offering this facility effectively, efficiently and profitably can be constructed?
Absolutely.

Will anyone read & respond to this piece and the proposal?
Who Knows :-)

Internet Changes Everything: Newspapers

The News Broadsheet was a pivotal element of the 1776 American Revolution, eventually becoming enshrined in the First Amendment to the US Constitution.

Newspapers were integral to the Twentieth Century rise and evolution of Western Democracies. Without "frank and fearless" reporting (and an engaged electorate), governments can quickly spiral out of control.

We're now 15+ years into the "Internet Revolution", so where are Newspapers in their journey on-line? Significantly, nobody seems to have discovered a "secret sauce" to generally monetise News and the work of Journalists in the way that Amazon and Google etc have monetised books and on-line ads. Those who make money from writing seem to do so from direct subscriptions - the on-line form of "newsletters".

In my first job, there was a direct and obvious connection between the 5g +/- 0.01g of sugar being analysed and 'the business': Our analyses determined payments for 500 or 1,000 tonne lots. Getting it wrong wasn't an option. My place in the scheme of things was self evident.

The world of Software and I.T., even after 60 years, still doesn't have that direct & obvious link.
I.T. shares many traits with Journalism:

• Both are "Performance Disciplines", like Music, Art, Surgery and Gymnastics.
• "Effortless Performances" (as in 'making it look easy') take a lot of skill and experience. The public and more often now, management, have little appreciation of the process & skills.
  
• Input Effort and Results bear no discernible relationship.
  
• Quality is Everything, but seems impossible to measure.
• Although both are central and necessary to the businesses they support, they are managed as "Cost Centres", with seemingly no attempts at connecting outputs with Profit.
• Both deal in intangible and invisible "stuff" - information. Often with tight deadlines and very fast decay in product "usefulness".
  
• Both share a central problem: Effort/Inputs are decoupled from Income/Results.

To research this, I chose "Fairfax Media" (ASX:FXJ), a major Australian player, because some of their "Mastheads" are over 100 years and "The Sydney Morning Herald" used to be legendary - known worldwide for its "Rivers of Gold".

Getting even rough numbers to judge demand and price-points seems very hard.
For any on-line business to succeed, revenues have to support costs. While serving bit-streams may be considerably cheaper than printing & shipping paper, what will people pay for it and how do you get money off them? How do you draw in more subscribers - what are your Marketing & Sales channels?

Newsagents keep the full "cover price" of most newspapers. For magazines and most other products, the publisher gets half with the distributor & newspaper splitting the rest.

The content of newspaper, News etc, is why people buy newspapers.
The price people are willing to pay for the whole paper indicates the "ultility" they get.
But how do you arrive at a value-in-use of just the News component?
Are there synergist effects in operation?

The output of Journalists, "News", is essentially a "Free Good" to both consumers and the business. To the publisher, each newspaper printed is only a cost - which they actively attempt to minimise, whilst revenues are from advertising and independent of pages printed. The connection is the "rate card" - people will pay more in advertising for wider circulation & readership. The connection is anything but direct and immediate. Meanwhile, "content" (& "classified" adverts) brings in readers, but the supply-demand curve is generally unknown.

Makes the economics of "free" community publications more obvious. The distribution costs are only slightly more than to newsagents and the claimed readership is maximised.

It also says why businesses are so very happy with "advertising only" publications, like "Trading Post". Pure profit and no content producers to wrangle! These business translate on-line very well - but lose all "display advertising" to corporates like retailers looking for mass market advertising.

Audited Circulation figures aren't too hard to come by and "The Press Council" publish a good snapshot of the whole Print Media scene, but there are only tangential references to journalist "head counts". Fairfax journalists feel pressured by staff cuts and are taking action.

To come up with viable business models for on-line News, especially when competing with "Free" on-line services like Google or Free-To-Air broadcasts (radio & TV), you need both sides of the Accounting Equation:

Profit = Revenues - Expenses.

Other marketing data is needed to determine whether Size Matters (only one global Google and Amazon) or Little & Local works or some combination in between...

Whatever the result, there is one guaranteed loser: newsagents.
They are the traditional Marketing & Sales Channel for newspaper.
What becomes of them in an on-line world?
Can their access to "passing trade" and existing business relationships be leveraged?

What we do know is that people are very happy with "Free" on-line content: using search engines to point to unbilled on-line newspaper articles.
Newspapers initially tried to force reader registration, even though content was "free" people went elsewhere. Fees were charged to access "archives" or additional material made available to paid subscribers.
News Ltd's upcoming experiment in charging for on-line content will be watched very carefully throughout the industry.

What would it cost to run an adequate newsroom? Do the 15 journos of brisbanetimes.com.au reported by the Press Council provide adequate local coverage? Various sources suggest major metropolitan news have 200-300 journalists, while the national broadcaster, the ABC, have 700 people in their "News Division" (which is how many journalists?).

Any 24/7 operation requires shift-work. To provide a minimum staffing of 2 people, needs a team of ~12. Numbers build very quickly as more sections & coverage is needed.

Fairfax's Annual Report (2007) says they made a profit of ~A$500M ($447M EBITDA) on A$2.3B sales.
Around one third of sales were from New Zealand. Overseas and non-print revenues were unclear.
The Australian Digital operation made A$37M on sales of $137M.

Their ~10,000 employees were their largest single expense: A$700M.
Paper and ink came in second at A$270M.
Sales and Promotions were A$88M.
Communications: A$17.5M
I.T.: A$15M and
News services: A$12M

Fairfax reports $8B of Assets - $6B of which is "intangibles" - "goodwill" and "value of mastheads". They've around A$850M in Property, Plant & Equipment - the physical assets needed to produce newspapers.

But how many journalists were there and what does a single "high-quality" newsroom cost to run? That's not going to change for an on-line News service.

What will an average subscriber pay for an on-line News service?
What additional content are needed, and what synergies exist?
How many on-line subscribers will sign-up for each different offering?
Can the existing subscribers be converted to on-line subscribers? What would help in the transition?
What are the different attributes that subscribers value and what premiums will they pay?

This says that monetising on-line News services may be hard and their general lack says solutions are still not obvious.

---

Other comments, observations and relevant factoids:

Newspapers serve important subsidiary functions, like being "the paper of record" for Births, Deaths and Marriages as well as public events, political speeches and disasters, crises and more.
"Public Notices" of many types are published - from bankrupts to probate on wills to personals.

Newspapers have come to be the definitive textual mass-communication device.
There appears to not yet be any on-line equivalent.
In the USA, recent failures of long-running mastheads says there isn't that much time left to find a good answer.

Circulations:
The SMH sells ~212,000 copies Mon-Fri (@ $1.40), 364,000 on Sat ($2.40) and [Sun-Herald] 505,000 on Sun. ($1.80). It has increased circulation in recent years. No figures available for pages of advertising sold.
The Melbourne Age has comparable sales figures, though Sunday sales are roughly half the SMH.

'Readership' of the SMH is estimated at 853,000 Mon-Fri and 1,116,000 Sat.
Papers are shared around.

These figures suggest that newsagents & supermarkets make around $1.5M/week for Mon-Fri sales and the same again on weekends. Removing that income stream would leave a very unhappy sales channel...

Payments and Subscriptions types:

• Pre-paid or billed subscriptions. Like Home Delivery
• On-demand use: Ad-hoc or occasional purchase
  
• Prepaid access to articles.
  
• Business or family sharing. Limited copies shared between many.
• Public access - libraries.
  
• "Media Monitors" - clipping services across many sources relevant to a business.
  

What do Newspapers do?

• Inform
• Educate
• Entertain, and
• Surprise and Delight.

Newspapers tell you things you need to know, especially things you didn't realise you needed to know.

What comprises 'stories'?

• facts
• expert analysis
• commentary
• opinion, and
• interviews

Professional journalism brings brevity and precision. Stories are {complete, correct, concise} and hopefully {consistent}.

The value to the reader is collection and pre-screening: reducing a mountain of data and facts to quickly and easily accessible information. Journalists classify and prioritise stories, letting the reader minimise time used and maximise information found.

This "Usability" principle extends to presentation, layout and story structure.
The fonts and columns widths are chosen to best suit human factors.
Modern printing added pictures to text - an important aid to readers and writers alike.
The use of white space, headlines and graphics/pictures increase readability & accessibility.
Stories follow the "inverted pyramid" - the most important facts first, tailing off. It means readers can quickly scan articles and find the most relevant/useful to them, and sub-editors can easily trim articles to fit by removing trailing paragraphs.

It is instructive to read a 100-yo paper without these modern features - they look dense and impenetrable. The small number of pages would've been a blessing.

Professional Journalists bring special factors to collecting stories:

• access to people & organisations, like police, politicians, CEO's, ...
  
• funding for travel, communication and fees - like FOI requests or corporate data.
  
• Researchers, archives and access to expensive subscription services.

The users of News services look for many different benefits and uses:

• text, images, audio and video
• Mobile access
• Alerts and "Instantaneous news items" (the latest stories)
• Distraction, Relaxation and passing-time (as in commuting)
• Social settings - Cafe and Brunch
• Search and information: Browsing classified and focused searches

On-line challenges:

• distribution format. PDF's or HTML?
  How do I read it on the train?
  Does it read well on a laptop or iPhone?
  
• enforcing DRM. How to limit copying? What to do if copied illegally?
  
• on-going subscription rights:
  If I've bought a "paper" once, do I have permanent access to it, even after my subscription ends?
  
• Competing with Free-To-Net services. Seems hard, have to provide additional value.
  
• Catering for local community news
• Accepting input from the general public.
• The Blogosphere and non-professional writers. They can't be held to journalistic standards and ethics, can't be reprimanded or censured and will publish/repeat unsubstantiated gossip and rumour.
  

"non-News" Newspaper functions that Online services may need to duplicate:

• "paper of record" function: Public Notices, Births-Deaths-Marriages
• Mass-market display advertising.
• Marketing and Sales channels. How to grow new business?
• Access to Printed copies - eg. weekly summaries.
• Searchable service provider directories and classified adverts.
  Implies RSS-style alerts & monitoring.
  

On-line service challenges:

• Monetisation. Advertising, subscription or sponsor based? Other?
  
• Niche marketing. By location, interest, community, employment sector.
• Tiered Subscriptions: free, basic, premium, target area, search tools, ...
  Profits can be maximised by segmenting services with multiple price-points.
• Organisational access.