SteveJ-on-IT

2013/03/14

NBN: CommsDay vs ABC.

Media Watch pointed to the CommsDay "riposte" to the ABC's definitive article listing the issues around NBN. [Part 2 of article]. Written 14-march-2013.

Here's my analysis of Grahame Lynch's "50 problems". Commsday comments in this colour.
[18-Mar: Apologies to Grahame for misspelling his name]

For completeness, some disclosure: I don't work for, or hold shares in, a Telco, NBN Co or one of their suppliers or contractors, nor am I a member or supporter of any Political Party. Neither do I support the Labor NBN plan over the Coalition's unannounced NBN. I hold the position that either Policy with deliver better broadband access. My interest is for a clear and informed debate on what I consider the defining Technology of the 21st Century for national Business Productivity and Competitiveness.

Unsolicited Advise to a classmate

I've never been a politician, managed a large budget or run even a moderate sized project, so why do I have the hubris to offer some unsolicited advice to a newly minted I.T. Minister, to whom by accident, I was once briefly a classmate?

Rejecting these thoughts "because nobody else is doing them" is an option, though not a great reason.

Rejecting them "because they're too expensive" is a judgement call, but has to be measured against "Compared to What?".
Doing nothing will cost you a whole bunch, you've already have the report on that.

The core arguments in support of my observations are:

How important to the current and future BackOffice and FrontOffice operations of your Government is I.T.? I suggest that the machinery of Government cannot operate without it I.T. systems, not simply ineffectively, but like airlines, not at all.
There is an internal consistency in what I propose, derived from one of the toughest businesses around. The challenge is not "will this work", but "how can it be made to work for us".

National Security: Prevention and Strengthening Defences missing from Gillard Strategy

The Gillard government has released a new National Security strategy specifically including Cyber-Security. It updates a 2009 strategy released by the Rudd government:

Strong and Secure: A Strategy for Australia's National Security

The strategy is strong, competent and wrong...
Because what is outlined is incomplete:

They have failed to address the root cause of cyber-attacks: vulnerable and error-filled Operating Systems and poor Application Software. Fix the weakness, stop the compromises before they happen, spend the money on where it can do good, not support "Business As Usual".

Cleaning up the mess and containing damage after the fact is exactly wrong: it's attempting to catch the horse after its has bolted.

Security: Computer Security for Business Continuity in Healthcare

If you run a Healthcare-realted Business, things changed in the last 6 months...
Ransomware is set to boom [0] and cyber-security is now part of our National Security Plan.
Businesses now have to secure their computers and data just as they secure their premises and goods.

It's not optional, fail to do so and you will go out of business, just when is the question.
Ask yourself this: "If my computers were destroyed, how long could I continue the business? At reduced capacity or at all?", then act accordingly.
i.e. Does anyone around the world see you as a high-value, exploitable target?
Especially those in low-income countries with employment problems: poverty corrupts, not just power or the love of money.

The Internet is defined by its explosive growth: A few For-Profit hackers have noticed Business Ransomware is an ideal way to monetise remote computer attacks & exploits.
The numbers of these attacks will now double every few months as word gets around, new "toolkits" are sold to them and they ramp up their activities.

Every business that can raise $5,000 and relies on its systems and data for daily operations is now in their sights. These people have no morals, ethics or compassion in their work: they want your money and don't care about the damage they cause or the impact of their actions. Appeals to them will fall on deaf ears. Neither believe that a single ransom payment will be the last you'll hear of them. Why would you trust the word of criminals who've already broken in and callously damaged your systems?

Security: Healthcare systems are "soft-targets": the Next Big Exploit

Previous pieces on Security:

I'd been racking my brains as to how Cybercriminals can "monetise" e-Health Records and writing to someone else, think I've understood it finally after a "Top of the News" report by the Security for Professionals: SANS.

There are two ways to monetise e-Health Records:

Identity Theft. Huge amount of high-quality info. Medicare Cards are worth 'points' as Govt. ID's.
Ransomware: healthcare can't operate without its data and they print money by the truckload.

Storage: Smaller is better in Hard Disks

I upgraded my TimeMachine ® drive two days ago on my Apple Desktop [2009 Intel Mac Mini, OS/X Snow Leopard].

I went to a local office supply chain-store and bought a Seagate USB3 drive for $90, replacing my older 'Buffalo' USB2 drive, maybe 12-18 months old. While the Mac Mini only has USB2 ports and I can realise the higher bus speed, it will work on newer computers as well.

My new drive is 1TB, 2.5", covered by the statutory 12mth warranty. Better to replace without pressure, than when things are broken... The old drive can go on the shelf as a long-term roll-back.

This is a trick I learnt from a friend with a number of Macbooks and iMacs:

permanently attach an external drive for TimeMachine... Cheap and Effective!

2.5" is important.
The consumer drive variants (not 'enterprise', designed for servers) are built to be mobile (laptops & portable devices) and so are more robust, shock-resilient, etc.

But that's not the 'secret sauce': it's power consumption, and its relative, heat dissipation.

2.5" drives can be fully powered by USB 2, in the normal/correct specification [not the 10W+ that the iPad demands, but well under 5W, even 2.5W]. They run cool, quietly, on low-power.

Not needing a "wall-wart" is really important: it can never be lost, never blow up, never "leak mains power" or cause earth-loop problems (my Western Digital 3.5" drive sparks against the frame when I plug it in. Scary.)

There's a reason based in physics for 2.5" drives using a lot less power:

the aerodynamic drag of the platters, the main power consumer that is turned into heat, is affected by 3 factors: number of platters, rotational rate, size of platters.

Firstly, they only have 1 or 2 platters, versus 4 (max 5?) you'll find in 3.5" drives.
"low performance" 2.5" drives at 5400rpm spin slower than 3.5" drives, mostly7200rpm - a ratio of 1.33:1.
Aerodynamic drag increases with the cube (third power) of speed/rotational rate. The cube of 1.33 is 2.37.

Just these two factors reduce potential power demand by about five times, albeit for one-quarter the space compared to a 4TB 3.5" drive. But 1TB is more than enough to maintain snapshots of my 300GB drive and is a good fit for me.

For the next factor:

Disk Drive form-factors are related - each size has platters half the area of the larger size, so the diameter/radius ratio is 0.7/1.4 times, the square root...

The 3rd factor overshadows all others:

The drag is proportional to the fifth (5th) power of diameter.
This ratio for a diameter ratio 0.7/1.41 is 0.18/5.66.

Just halving platter area, and halving capacity/plater, reduces power needs by another five times.

Without trying, 2.5" drives use 20 times less power, for only 1/2 to 1/4 the capacity.
For the same capacity but slower spin-rate, they use 10 (ten) times less power,

Four 2.5" drives use just a little less space than a single 3.5" drive [102mm x 140mm x 19-25.4mm] vs [102mm x 147mm x 26.1mm], ignoring connectors.

The Small Form Factor Committee very nicely designed the footprint of each device size to be the same ratio meaning the length of the next smaller size is the width of the current size. If you've ever noticed, you can lay a 2.5" drive sideways across a 3.5" drive. Four 2.5" drives, in 2 stacks of 2, will fit within a box that would hold a single 3.5" drive. Adding connectors is trickier - they are at right-angles to each other in the two form factors.

A side benefit you have 4 sets of independent electronics, buffers and heads working for you.

You get better aggregate throughput and seek performance from 4 smaller drives, OR you can sacrifice 33% of capacity for radically improved fault-resilience and setup RAID-5 over 4 drives.

But 2.5" drives cost around 35-100% more per GB, depending on models.
It depends what value you put on your time and your data...
I think $90/TB vs $75/TB is insignificant compared to the benefits, even for home use.

If its business or professional-use data for a modest drive count, the economics of 2.5" drives are a slam-dunk (or "no-brainer"). If you have petabytes to store, you got other problems to solve.

At this point you might be wondering "If small is good, and much smaller is much better, why don't we have smaller Form Factors in common use?"

We do.

1.75 inch drives are manufactured in volume, but not sold in "computers", only embedded in mobile devices. They are tricky to manufacture with the same problems as making watches.
The Small Form Factor Committee never specified a standard, especially for thickness, of these drives.

There have been previous credible attempts at micro-minature HDD's: around 2000, IBM built the largest capacity "Compact Flash" drive available. It was a hard disk. While they were both highest capacity and best price/MB, they weren't popular with working Professional Photographers - they were too fragile... One of the extreme advantages of solid-state memory, like "Flash", is its robustness, especially when turned off. If you keep static electricity away from them, they can be nearly indestructible. The data will fade away ("Flash" EPROM is not permanent) long before they'll succumb to mechanical damage...

But within a couple of years the market collapsed because real Flash memory overtook them in capacity, price/MB and transfer rate.

Not long after RAID-5 was proposed in 1989, there was a good academic paper that suggested by 2000 we'd have a new type of storage: many single platter 1 inch drives soldered onto boards.

The economics of very small form-factor disk drives turned out never to be compelling.

If nobody is building the drives in volume, the price cannot be competitive with substitutes, like Flash for smaller units and 2.5" and 3.5" drives for larger units.
If there is no demand for a product, then no manufacturer will invest in the R&D and manufacturing facilities to ramp-up to volume production.

It sounds like a chicken-and-egg problem: nobody builds small drives so nobody designs products based on them so there is no reason to build small drives.

But that's not the whole story: drive manufacturers can do the Maths and know their production economics and capabilities very well.
If they thought they could produce rugged, small drives that were cheaper and higher-capacity compared to Flash, then one of them would've tried it. If they'd have made a profit, everyone would've followed, just like the 5.25 inch to 3.5 inch and 3.5 inch to 2.5 inch changeovers.

The stumbling block is per-device manufacturing cost.

Disks are complex mechanical devices, assembled individually to very fine tolerances. Their production costs are dominated by that, not raw materials and the process doesn't scale well with volume. While disk capacity has doubled every year for 15 years, the minimum unit price has either remained the same or risen.

Very small format drives, even in high volume, would, per unit, cost a sizeable fraction of 2.5" or 3.5" drives.

Compact Flash chips, whilst small capacity (1-32Gb) compared to even 2.5 inch drives (1000Gb), are cents per chip to produce and the technology to scale volume production of them is very well known and researched, but more importantly lends itself to scaling up.

Flash current sells for $1-$10/Gb and 32Gb is $10-20 retail, or less. SSD's are more expensive per GB than SD cards and USB Flash memory devices for a number of reasons related to speed, reliability/durability and wear-rate.

No 2.5" drive can be smaller than mid-capacity Flash memory or SSD: no user would buy one.
Or, the smallest 2.5" drives you can sell for $50-$60 are 250GB, or $0.20/GB.

If you could make a very small format disk drive for as little as $30 retail, it has to compete with Flash.
It'd have to be over 128GB to sell, or at least half the capacity of a small 2.5 inch drive or one-eight the size of high-capacity 2.5 inch drives.

Which gives a smallest economic form-factor of 1.25 inch, this year.
Next year, when Flash memory reduces another 30%, it's marginal and the year after, when Flash cost/GB has halved, completely uneconomic. A two year product life is well below the payback period.

There haven't been any discoveries or developments in magnetic disks that will change these economies within 10 years: we won't ever see high volume production of computer drives smaller than 2.5 inch.

Drive Dimensions:
2.5": 70mm x 102mm x 9mm [thickness varies, 7mm to 12.7mm. 15mm for 'enterprise']
3.5": 102mm x 147mm x 26.1mm [std "1-inch" thickness]

Sources:
[http://www.sffcommittee.com/ie/index.html]
[ftp://ftp.seagate.com/sff/8000_PRJ.HTM]
[ftp://ftp.seagate.com/sff/SFF-8200.PDF] 2.5"
[ftp://ftp.seagate.com/sff/SFF-8300.PDF] 3.5"

2012/12/26

Content: What's wrong with the Wikipedia Knowledge Model

The Register had a recent piece on Wikipedia: they're rolling in cash and the lucky few are enjoying that gravy train. Talking about this led to a discussion on the strengths and failings of the Wikipedia Content Model vs Classical Encyclopaedias, like Britannica.

Wikipedia has a simplistic Model of Knowledge: There is But One Truth.

If you are an engineer or a programmer, then this is as axiomatic as binary data and logic: true/false, 0/1, just one correct answer to every equation or calculation/test, there is but one truth.

Which is exactly where Wikipedia excels: if you need to look up a technical topic, especially in computing/communications and some engineering, where there is a well defined specification, then you find very good articles, often concise, complete and well explained.

But this is at best an undergraduate view of engineering: incomplete, naive, simplistic and completely without nuance. It's deeply wrong and misleading.

What does Encyclopaedia Britannica provide?

Experts. With a comprehensive knowledge of the field, its history, the current "State of the Art and Practice" and the currently accepted theories with competing ideas and approaches to problems at 'the edge'. They understand nuance and the many theories nature of the development of science. But more importantly, they bring judgement and an ability to explain their field to others.

Even in objective technical fields, like computing and electronic engineering, there are many competing theories about the field, each with their benefits and limitations, all argued for strongly by proponents and all, even those appearing 'weak' or later proven wrong or limited, offer insight and understanding.

Knowing the whole story, what works, whats been tried, whats been discarded - and why, is as important as knowing the current best theory. And that's for objective fields of knowledge: 'truth' isn't fixed and immutable, but constantly and unpredictably evolving and changing.

The proof of this is the 1902 statement by the newly appointed Commissioner of Patents in the US:

In my opinion, all previous advances in the various lines of invention will appear totally insignificant when compared with those which the present century will witness. I almost wish that I might live my life over again to see the wonders which are at the threshold.

He was proven right, beyond his wildest dreams... The nature of Science is change, "truth" evolves over time.

For more subjective fields, the 'soft sciences', even history and philosophy, the many viewpoints approach of science is vitally important: the discourse is more important than "the" answer.
Or fields requiring judgement, in answering questions about "best", ethics or belief and emotions: the stuff of real life.

Within this Many Theories and Viewpoints model of knowledge, articles with single identified authors are important. Readers may wish to read more by the author or disagree with their viewpoint and avoid their writing.

Contributions from Domain Experts in Encyclopaedias offer the same promise as Undergraduate courses for professionals:

Complete coverage of the field: an A-Z guide of the current state of the art, influenced by the experts viewpoint and approach.

This is the weakness of the "autodidact", or the self-taught: they can easily miss large and important areas of a field. They don't know what they don't know, whilst a "good" undergraduate course carries the promise of knowing what you know and what you don't know: formal education is about removing the "Unknown Unknowns". The self-taught, without feedback from others, can never be certain they've learnt the whole field. [Because they aren't hampered by preconceptions, they can sometimes arrive at startling new answers, but mostly they fall into known fallacies and logic traps.]

To evolve, grow and thrive, Wikipedia needs to embrace single identified authorship, multiple competing views on the same subject and good histories of the subject, reflecting the cut-and-thrust of all lively, developing areas of academic debate.

For Wikipedia to be more than the naive "font of all undergraduate wisdom", it has to embrace the complex, many faceted view of knowledge and discourse in world of graduates and researchers.

2012/12/19

Recruiting FAIL: The Non-Response of ITCRA.

I'm not impressed with the ITCRA grievance process.

As a mere 'candidate', they were happy to accept a 'grievance' from me, using that term, to communicate with me on the matter over many months and to put me to some trouble to provide them 'evidence'. When I raised concerns about maintaining the confidentiality of my private emails, they were brushed aside: Our Way or the Highway was the message.

After too many months, today I finally have a letter saying the matter is 'resolved' and now closed.
I am none the wiser about the facts of my complaints or if ITCRA viewed any/all/some of them as proven.
Did they think I was a reliable witness and weigh my evidence highly or not? I've No Idea.

ITCRA was also unhappy that I should dare to write-up a DEIDENTIFIED version of the story.

As an organisation they seem to have difficulty with the notions of Freedom of Speech, Open and Transparent Governance Processes and Natural Justice.
Nor did anyone have the decency to offer me an apology.
From that, do I infer they found NO FAULT on the part of the Agency and Agent? That I've been a jerk, wasting their time and making a nuisance of myself? I've No Idea.

If anyone has a problem with an ITCRA Member Company, my advice now is:

Don't waste your time.

They'll only jerk you aroud, waste your time and leave you very unsatisfied.

Previous posts on the matter:

http://stevej-on-it.blogspot.com/2012/09/recruiting-fail-how-to-foul-up-employee.html

http://stevej-on-it.blogspot.com/2012/09/recruiting-fail-part-3-itcra-complaint.html

DRAFT Complaint: 10-Sept-2012.

Mr. AAA BBB,
ITCRA Member Company Representative
XXX Recruitment
YYY Street
Melbourne VIC 3000

Dear AAA,

I am in receipt of a grievance lodged by a candidate Mr. Steve Jenkin, against a XXX consultant Mr. ZZZ ZZZ. He has certified that the information provided is correct to the best of his knowledge and that he has not provided any false or misleading claims against another company or individual and has provided additional email evidence to support the claims made. In summary the issues are:

Background (commencing in August)

• The issue is focused on a *permanent* position for a senior IT staffer and Mr Jenkin was required to move interstate.
• Agency approached Mr Jenkin, no job application ever made by Mr Jenkin.
• Job never advertised with a start date, no immediacy ever stated. Comment was made that it was expected to take 2 months to fill position [by October]

Grievances

• Misleading statements:
1. Confused an email from consultant (ZZ) to work seeker (SJ) with a written offer from the Client. Insisted this was a "Job Offer ", implying a binding contract.

2. Promised help finding accommodation for relocating interstate and none was provided. Having local accommodation was always a condition of accepting the position.

• Harassment or cyber-stalking:

1. One incident of a dozen calls/texts in 30mins by (ZZ)

2. (SJ) asked, in an email, to desist. (ZZ) repeated again later with more very inappropriate and abusive statements.

• Privacy Act breach.
1. (ZZ) used referee contact details for purposes data not supplied for. When work seeker (SJ) would not answer consultant’s (ZZ) calls, consultant (ZZ) rang referee to complain and spoke inappropriately.

• Additional Issues

1. When SJ had failed to receive a contract 10 days from first proposed start date, informed ZZ of inability to start due to personal circumstances. ZZ then harassed and browbeat SJ and seriously overstepped by continuing to demand exactly what the personal circumstances were. This was justified by saying that there was a need to explain to the Client.

2. Misspelt SJ name on contract.

3. ZZ demanded a written apology from SJ to Client for pointing this out.

4. ZZ Sent 7 emails in 15 minute period informing work SJ of Job Offer. ZZ Kept issuing recall emails for incorrect offers sent.

5. ZZ commented that his manager was continually chasing him to comply with company requirements and document all communications.

6. SJ requested 3 times in email that ZZ ask Client if 9-day fortnight [with RDO] would be acceptable during first 3 months while transitioning to Sydney. ZZ never asked Client [confirmed]. Client was happy to comply when asked by SJ.

7. ZZ deliberately misled SJ by stating the job was full-time only and RDO’s were not possible.

Formal Letter of Response: 19-Dec-2012

Mr Steve Jenkin,

Dear Steve,,

The ITCRA Board of Directors has asked me to write and thank you for raising the professional practice matter against ZZZ ZZZ of XXX Recruitment. The complaints process allows us, as an Association, to address issues and to ensure that recruitment practices achieve best practice benchmarks.

The Conduct Committee reviewed the evidence provided by both parties and asked further questions of XXX Recruitment, as the Member of ITCRA. The findings and recommendations of the Conduct Committee were discussed by the ITRCRA Directors and the following outcomes have resulted:

• XXX Recruitment were required to agree to a number of actions as a result of the Committee's recommendations and these have been committed to, with some already completed and assessed by an external quality auditor.

• The ITCRA Directors accepted the Committee's findings and the undertakings by XXX with respect to process review and consultant training.

• The ITCRA Directors raised concerns with respect to the discussions of this matter in the public domain via your blog as this could have jeopardized the process if the Member company felt the Committee had been unduly influenced prior to the matter being heard.

The purpose of the ITCR's Code of Conduct to provide a channel for complaints to be addressed and to provide improvement strategies for professional and business conduct of Members.

As explained to yo, the resolution of the matter was between ITRCA and the Member company and, as such, is confidential and cannot be made public - although the content can be used as a case study, with anonymity, for industry improvement. You would have the right to review any such case study as protection of your rights is also a priority.

The Directors wish you well in your future endeavours and hope that your experience with recruiters, in the future, is not as frustrating.

Have an excellent festive season,

Yours sincerely,
.... CEO

Email to President of ITCRA on their unsatisfactory response.

To: Russell MacDonald, President ITCRA
Subject: ITRCA complaint against XXX
Date: Wed, 19 Dec 2012 13:10:36 +1100

Russell,

After many months of waiting, today I finally received a letter telling me my complaint against XXX Recruiting on multiple grounds had been resolved.

The letter is best described as "content free".

How was my complaint resolved? I've no idea, just that you've closed the book on it.

Nothing in this process has answered my complaints, nor told me what, if anything, XXX had done wrong, or if any of their malfeasance *I* perceived was supported or not.

All I can gather is they were given a good talking to and they promised not to get caught doing "it" again.

I'm a party to these proceedings, not a third party or 'the public':
these actions were visited *upon me* and were personally quite devastating and financially disastrous for me AND I personally raised the issue to get some satisfaction from you.

Releasing to me matters that *I* raised, even under a Confidentiality Agreement, doesn't involve any abuse of process or publicly exposes anyone. I already know the names of the players, I was one of them.

If you knew *all along* that I would never be told the outcome of my complaint WHY DIDN'T YOU JUST SAY THAT in September???
That would have been reasonable, but your organisation has jerked me around for months on end...

That's beyond inconsiderate: it's deliberate and intentional misleading behaviour.

At the very least, I would've expected a response to the complaint I raised that comprised two parts:

- the findings of fact on each individual issue/point raised.
i.e. "we found a breach of the privacy act, but not harassment"
Note: I am NOT party to the internal actions, remedies and disciplinary process.

- if the Agency, or their agent, was found to have acted in Bad Faith or outside acceptable limits in your Professional Code, then an apology to me, at least, is in order.

Where is a letter to me from the management of XXX Recruiting saying "We're sorry, we've looked to fixing things"?????

I'm not just disappointed in the process and this unsatisfactory result, but that you don't understand the concepts behind of Natural Justice. If you asked me to provide a *lot* of personal information as evidence as part of your process, so why cannot I know the determinations you made?

*That* is the reason I approached you with a complaint: to have you make a determination and tell me what it was.

I wish that you:
- write me a response detailed the determination, not actions taken, on each point of complaint I made.
- direct XXX Recruitment to write (or phone) an appropriate formal apology to me.

regards
steve jenkin

2012/12/13

Security: The Massive hole in the PCEHR system

In the last few days, three computer security stories have hit the news:

These may seem small, incidental stories, but they are signs of something much darker. At the end of 2004 the Hackers Turned Pro [and a 2007 piece]: now they're after the money, not publicity nor headlines. In fact, rather the reverse, like special tactical units, military or police, they now want to go completely undetected - to avoid detection, to be completely stealthy.

NBN: The Economics of FTTH vs FTTN

Do you know a Bargain when you see one??
Is either NBN Policy proposal offering a bargain or the opposite, a "lemon"?

NBN: After CommsDay, Questions for Mr Turnbull

In his CommsDay Conference 2012 address, Mr Turnbull came as close as ever to detailing the Coalition NBN plan, seemingly centred around a 25Mbps fixed-line FTTN.

These are some reasonable questions Mr Turnbull should answer to inform the electorate of the Coalition NBN Policy he is prosecuting. I can't see how he can meet any, let alone all, of the Coalition NBN promises - Cheaper, Sooner, More Affordable:

Given it took until early this year, 15 years since the start of the Domestic Internet Revolution, for the Coalition to grudgingly adopt a Fast Broadband Policy, What guarantee do we, the electorate, have that the Coalition will fulfil its promises, not revert to its previous position and nationally deliver everyone The Gungahlin Experience?

The exact details matter in Technology Projects like the NBN and Mr Turnbull has shown his poor grasp of critical details, or willingness to disingenuously spin the facts. Sanity checks against previous deployments (Transact) and FTTN proposals (Telstra 2005) of Coalition claims of lower FTTN costs and comparing FTTN and Fibre construction costs suggests that in achieving the Coalition promise "Better Broadband, Cheaper, Sooner and More Affordable", something different, and not yet disclosed, will need to be deployed:

Does the Coalition guarantee to match the current fixed-line coverage of 93% premises?
The Coalition demanded detailed, precise costings for the current NBN, it is their turn to provide them well, the sooner the better to allow a full discussion. Will Mr Turnbull release costings to the nearest $100MM in the near future?
The GPON NBN has a design life of 30 years starting with new cable and equipment, while the FTTN is End-of-Lifecycle technology using End-of-Life cables: Will Mr Turnbull release detailed comparisons of the Total Cost of Ownership of both proposals?
Mr Turnbull has implied the Coalition FTTN equipment will be field upgradable to Fibre.

Won't building a copper network to throw away be more expensive?
If the Coalition intends to deploy a Fibre solution, why not start with it?
Isn't adhoc, on-demand laying of fibre and digging trenches the most expensive and inefficient method of rolling out Fibre?

FTTN technology is a very poor fit for low-population density areas, such as the regional towns and rural areas. Costs escalate exponentially: copper-run distances increase, pushing down access rates and ports per node decrease geometrically, raising the per port costs. Will Mr Turnbull release early the detail of how he will guarantee metropolitan access rates for Country users of the Coalition NBN? Will he guarantee not to rely on expensive, congestion-prone Fixed Wireless (3/4G) for country areas?

For some Party Political questions, I've included two recent sets included in Ministerial media releases. Duplicated questions are coloured. While detailed, they didn't address many questions that I, as a voter, wanted answered.

Senator Conroy: Questions for Mr Turnbull, 4-Sep-2012

POLICY (OWNERSHIP)

Does he accept that his FTTN network is a government monopoly network?
Will Mr Turnbull’s network be on budget or off budget? How much will his policy cost the budget?
Is Mr Turnbull really going to buy back the deteriorating copper network and its expensive maintenance costs? Will he also buy the Telstra ducts? Does he stand by his media release of 17 May 2011 that acquiring use of the copper would be “more billions out the door”?
Will he guarantee the structural separation of Telstra? Has it been agreed by shadow cabinet?

CAPACITY AND TECHNOLOGY

What upload and download capacity will Mr Turnbull guarantee?
Which of his previous statements does Mr Turnbull stand by when it comes to what speeds Australians need:

In August 2010, he said he could do everything he needed with 3.5 Mbps download[iv]
In October 2010, he said 12 Mbps is enough for anybody[v], and
In May 2012 he said residential customers need no more than 25 Mbps.[vi]

Will FTTN be built in areas where there is HFC? Who will pay to make the HFC open access, enter multi-dwelling units or provide a business grade service? Does Mr Turnbull accept that the upload capacity of HFC is limited to 2 Mbps? Will Telstra be required to divest the HFC assets?
How many FTTN nodes does he plan to build? What percentage of premises connected to each cabinet will be able to benefit from speeds of 80 Mbps?
How many more premises will be connected using wireless than under the Government’s NBN plans?

PRICES

What price will be charged in country areas without the cross subsidy? What will regional users be charged before Mr Turnbull’s on budget “vouchers”? How much will the vouchers be and how many will be issued?
Has shadow cabinet formally rejected the National Party policy that fibre to the home should be built to at least 50% of premises in regional Australia?
Does Mr Turnbull stand by his claim that his FTTN network will be required to generate a 7% return as claimed in his Op Ed in the Tele on 23 August?

HOW LONG IT WILL TAKE

How will he select the private network providers for the few areas he plans to build any new infrastructure?
Will he guarantee his new broadband policy will start within 12 months, despite his promise of a Productivity Committee review and tender for a private sector network provider?

Senator Conroy: Questions for Mr Turnbull, 10-Oct-2012

COST

Will MrTurnbull's network be on budget or off budget? How much will his policy cost the budget?
Is Mr Turnbull really going to buy back the deteriorating copper network and its expensive maintenance costs? Will he also buy the Telstra ducts? Does he stand by his media release of 17 May 2011 that acquiring use of the copper would be "more billions out the door"?
Will he guarantee the structural separation of Telstra? Has it been agreed by shadow cabinet?

CAPACITY AND TECHNOLOGY

What upload and download capacity will Mr Turnbull guarantee?
Does he accept that his policy only commits to providing a 25 Mbps service?[iii]
Has shadow cabinet formally rejected the National Party policy that fibre to the home should be built to at least 50% of premises in regional Australia?
Will FTTN be built in areas where there is HFC? Who will pay to make the HFC open access, enter multi-dwelling units or provide a business grade service? Does Mr Turnbull accept that the upload capacity of HFC is limited to 2 Mbps? Will Telstra be required to divest the HFC assets?
How many FTTN nodes does he plan to build?

PRICES

What price will be charged in country areas without the cross subsidy? What will regional users be charged before Mr Turnbull's on budget "vouchers"? How much will the vouchers be and how many will be issued?
Does Mr Turnbull stand by his claim that his FTTN network will be required to generate a 7% return as claimed in his Op Ed in the Tele on 23 August?

HOW LONG IT WILL TAKE

When would his FTTN plan be finished?
How will he select the private network providers for the few areas he plans to build any new infrastructure?
Will he guarantee his new broadband policy will start within 12 months, despite his promise of a Productivity Committee review and tender for a private sector network provider? Will his "thorough inquiry into the management and governance of the NBN Co" be conducted at the same time as the Productivity Commission review of the project?

OWNERSHIP/POLICY

Does he accept that his FTTN network is a government monopoly network?
Will he commit to maintaining the network in government ownership till it is fully built?

NBN: FTTN in Australia - the Gungahlin Experiment

In 1995, Telstra announced that the new Town Centre in Northern Canberra, Gungahlin, would be a testbed for "broadband", with a $20-$30MM project announced laying "Fibre to the Kerb", which many people like me interpreted as "Fibre to the Home".

What Telstra deployed was RIM's, Remote Integrated Multiplexors, designed initially as an 'integrated' extension of the local Telephone Exchange. ADSL broadband was not supported. Instead of a world-class demonstration of the application and utility of fast broadband, Gungahlin residents were locked in a battle for even minimal access speeds.

For the rest of Australia, we have very clear demonstration of two things from this nearly two decade 'experiment':

Fibre-to-the-Node, even 15 years ago when it was a much, much cheaper technology than Fibre-to-the-Premises, is fraught for consumers.

Even when the subscribers thought they were out of the woods with high-speed ADSL 2+ available, the sting-in-the-tail of FTTN became apparent: congestion on the node "backhaul" makes line access rate irrelevant.

The Coalition, during the Howard government from 1996 to 2007, did not prioritise, nor apparently see the need for, a National Broadband Network.

It is only since the Telstra SSU agreements early 2012 that the Coalition has become committed to an NBN of any flavour.

NBN: Costing the Coalition FTTN. It's not Good News.

Mr Turnbull, in his CommsDay address, seemed to be proposing the Paul Fletcher, "Wired Brown Land" FTTN:

25Mbps guaranteed, VDSL2
800m max wiring (approx. 500m road distance)
we can infer 40-60 subs per node, consistent with the 160 subs/node of the first Telstra FTTN.
The "golden screwdriver" is "Multi-Service Nodes": field upgradable from copper to Fibre.

So, what's it going to cost to roll this out to 12M subs, the 93% of premises addressed by NBN Co?

Will Mr Turnbull achieve the Coalition tagline, "Better Broadband, Cheaper, Sooner, More Affordable"?

MSFT: The 'Surface' may be irrelevant, but is this the Beginning of the End for Microsoft?

Is the Microsoft Surface "too little, too late" or another Big Play that will crush the competition?
It is only, as Android devices are, similar to the iPad in design, features, construction and price, not a "must have" device.

Security: The Desktop Wars are over

Comments back from a friend on an idea I was toying with for security on PC's.

It seems the Desktop Wars are over and I need to embrace the New World Order:
Smartphones, Tablets, Mobile-Devices have changed the problems and our thinking.

TLS: Sol Trujillo's Telstra legacy

Two views of the same events, how Telstra performed under Sol Trujillo and the three friends he brought with him: Greg Winn, Bill Stewart and Phil Burgess.

Sol's own biography describes his triumph at Telstra (ASX:TLS):

Sol served most recently as CEO of Telstra Corporation, Australia’s largest media-communications enterprise, where he completed the privatization of a previously government-owned monopoly and led the transformation of a traditional telecommunications utility into an integrated media-communications company – including telephone, CATV, wireless, directories, advertising, online trading, and the world’s largest, fastest, and most advanced mobile internet.

And summarises his achievements at Telstra as:

Privatized a previously government-owned telecommunications ultility.
Built an integrated media-communications company with CATV, Internet, directories, phone, and satellite.
Built the world’s largest, fastest, most advanced mobile Internet – a high-speed wireless 3G network.
Achieved highest user and revenue growth in the industry.
Grew average revenue per user (ARPU) by approximately 15 percent and data ARPU by more than 200 percent.
Created Hong Kong’s largest mobile business.
Changed the company’s focus from products to the customer experience.

Sol's wikipedia entry reports his time at Telstra as:

He was appointed Telstra's chief executive officer on July 1, 2005.

During the period of Trujillo's tenure, Telstra's share price underperformed the market by around twenty percent, losing over $25 billion in value while customer complaints rose 300 percent.

Major factors in the company's share price decline were the global financial crisis of 2008–2009 and being disqualified for submitting a non-compliant bid to the National Broadband Network tender issued by the Rudd Government.

On February 25, 2009, Trujillo announced he would stand down as Telstra's CEO on June 30, and return to the United States with his family. On May 19, 2009, Trujillo left Telstra and shortly after returned to the US.

'Crikey' described Sol's legacy as 'a shambles' as ambitious projects didn't deliver promised benefits, revenues fell short of projections and Telstra acquired a reputation for abysmal customer-service:

Exactly why Trujillo was paid so much to deliver so little remains a mystery to all except the Telstra Remuneration Committee. Trujillo was already a very rich man before arriving in Australia, having been paid a US$72 million golden handshake by US West after leading the company into an ill-fated merger with Qwest.

It isn’t only be Telstra shareholders who have rights to be aggrieved at Trujillo’s remuneration. Former CEO, Ziggy Switkowski who presided over a company which made similar profits and employed far more people than Trujillo’s Telstra was paid only $1.65 million in 2000, rising to $6.3 million in 2005 (which included a $2 million termination payment). Trujillo’s local replacement, David Thodey, is receiving fixed remuneration of $2 million, 33% less than Trujillo’s base pay.

Hiring expensive, celebrity CEOs never seems to deliver solid returns for shareholders. It’s a shame no one told the Telstra board in 2005.

How the Stock Market valued Telstra:

Sol doesn't mention the share price collapsing under his reign and that slide continuing to nearly half his starting price ($2.60). Signing the NBN agreements has consolidated a share price recovery this year.

28-Nov-1997: $3.50 [$3.30 issue price for 'T1']
01-Oct-1999: $8.00 [$7.40 issue price for 'T2']
17-Nov-2006: $3.75 [$3.60 issue price for 'T3']
01-Feb-1999: $9.15 [peak]
07-May-1999: $7.36 [low for 1999]
20-Jul-1999: $8.95 [peak]
31-Dec-1997: $4.11
31-Dec-1998: $7.63
31-Dec-1999: $8.28
05-Jan-2000: $8.00
01-Jan-2001: $6.73
07-Jan-2002: $5.52
01-Jan-2003: $4.41
05-Jan-2004: $4.84
03-Jan-2005: $4.91
04-Jan-2006: $4.06
02-Jan-2007: $4.12
03-Jan-2008: $4.71
05-Jan-2009: $3.70
04-Jan-2010: $3.45
05-Jan-2011: $2.79
16-Mar-2011: $2.60 [low]
02-Jan-2012: $3.33
03-Aug-2012, $4.02 [peak in 2012]
05-Oct-2012: $3.95 [last close]
01-Jul-2005: $5.02 [Trujillo becomes CEO]
26-Jun-2009: $3.82 [Trujillo leaves]
28-Feb-2012: $3.26 [ACCC approves Structural Separation Understanding]
07-Mar-2012: $3.23 [Definitive Agreements in force]

2012/10/05

Unringing the Bell: Impact of IT systems on Large Business Survival

I've posited that Telstra will be severely challenged within 15 years due to Structural Change within their Industry. They aren't fast or agile enough to adapt to the new world..

What are the factors that will prevent them from adapting? My top two:

Management Culture
IT Infrastructure

An underlying problem is that we don't have language or metrics to describe and quantify the many important aspects of management. In Physics and Engineering we have many concepts and terms that are measurable to great precision. The importance of those is shown by the materials revolution over the last 100 years. For a tour de force on the topic, a book that's been in print for 45 years: The New Science of Strong Materials.

Whilst "Management" and descriptions and theories about it has become an increasingly large field of study, we don't have a "Science of Management" with precisely defined and measurable terms.

This is crucially important when ownership (shareholders) and control (managers) is separated. The owners have no nuanced, standardised measures to evaluate the most critical part of the business: management. All we've got is the Accounting Standard Reports provided in Annual Reports. This is far from enough to make informed decisions on a business' future prospects.

I can't detail or quantify the many problems of Telstra's Management Culture, just waive my hands and say "it's the vibe". E.g. they not only don't do Customer Service well, they prioritise short-term cost-savings above good service and seem to go to great lengths to not resolve customer faults, at least in some areas.

Management is about doing what's important consistently well, doing what has to be done well enough and not doing at all the things that don't need to be done. [And avoiding entirely the things that should never be done.]

My Professional expertise is in IT Infrastructure. I hold a contrary view to the mainstream Management view of "IT is a Cost Centre". IT provides automated Business Processes, like employees directly responsible for all the Business Revenues: IT is a Profit Centre, not Cost Centre.

We've already seen businesses failure due to their failed I.T. systems: One.Tel is a shining example.

IT Infrastructure has, for the last two decades at least, constrained business mergers. If I recall correctly, the St. George-Westpac merger was cancelled twice due to "incompatible IT systems". Not sure how they solved that in the end.

Westpac itself is notorious for a decade long project, CS90, that was cancelled in December 1990. It was meant to be the ultimate Banking System (an 'ERP') that IBM would resell around the world for them. It was consuming 5-10% of Westpac's operational revenue.

The $10B Telstra "IT Transformation" under Greg Winn ran for around 5 years with the intention of reducing 1500 systems to 300. It failed to meet its targets and went live in 2009 with the 30% most valuable customers not migrated [from my Case Study evidence, now full of migration errors.]

The point: large businesses are inextricably intertwined with their IT Systems - they are part of the Business DNA and essential to the Business Differentiation: What we do differently that people value.

Too often IT Systems are allowed to "grow like topsy" and are never rationalised or reorganised, presumably because no immediate savings or value can be demonstrated, but mostly because nobody is responsible for everything and ensuring IT Systems are well maintained and suitable.

Leading to "Big Bang" projects like Telstra embracing of off-the-shelf ERP and CRM systems to resolve the mess. Inevitably they find that things are much more complex and intertwined than they knew, not the least because they have no correct, current System Maps and the whole was never designed or planned, it just happened. All of which suggests Management asleep at the wheel.

It will probably take Telstra 10 years to get staff trained, most, not all, data corrected in their new systems and workaround established to cater for what the new systems don't do.

But what will it be left with then? Will those systems be nimble, quick and responsive or big, cumbersome and so hard to change as to be effectively frozen?

Young, small companies start small and add functions as needed: the I.T. equivalent of "greenfields".

They don't carry of a legacy or mindset of "we have to cope with everyone and everything".

This is the commercial advantage small ISP's had over Telstra: no past, no baggage, just simple effective systems.

This will be the problem that Telstra will have to face again in 2020 as Retail Providers using the NBN challenge it. Telstra knows the pain, cost and delay in redoing their I.T. Systems, they won't be going there again anytime soon.

This is a generic and on-going challenge for all successful businesses, including those small, nimble Retail Providers: how to keep I.T. Systems from degrading into an unchangeable morass?

When Data hardens in Organisational Arteries and structures/processes ossify, a major Cardiac event will follow... More of the Same cannot fix the problems, radical rethinking is needed.

In an increasingly automated world, a problem looms for every large business: what happens to the IT Systems when you downsize?

You get to drag the big, bloated corpse of yesterdays organisation along with you. It never gets better with age...

It's easy to lay-off staff and "reorganise", but I've never heard of any organisation looking to make commensurate simplifications to their I.T. systems.

I suspect that Large Business who aren't consciously and deliberately cleaning-up and refreshing their I.T. Systems will ultimately fail due to the complexity, inflexibility and inadequacy of their Legacy Systems.

You can lay off Staff and cut whole Departments, but where do you start with the weeds that permeate your whole organisation and choke the life out of it?

Once built, it seems you can't "Unring the Bell" of legacy I.T. systems, you're stuck with them and they define what you can do, while smaller companies whizz past you on their way to Market Domination and being strangled by their Legacy systems.

Jerry Gregoire as CIO of Dell Computers in 1999 talked about how he tackled this problem - and won.
When he joined Dell, there was a massive ERP project underway, "One System To Rule Them All". It was late, over-budget and failing. His first action was to cancel the project and front the board...

Instead, he moved Dell to a new architecture dubbed "G2", based around a message broker.
It reduces the N*N-1 or N-factorial system interface problem to one...

All every system needs to interact with every other system is one 'message broker' interface.
It comes with a cost - you need infrastructure and rule sets to switch the messages. But at least that's a known, computable cost.

Dell Business Strategy Secrets
An ERP Package for You...and You...and You...and Even You

Telco Customer Service Madness and the NBN

Will Telstra, as it is now, survive to see the NBN contracts end in 35 years?
My view: It won't, not in its current form.

In 2006 I started to write about my concerns for Microsoft's future, giving them 5-6 years before major cracks appeared. They haven't collapsed yet, but Horace Dediu (asymco) has produced a graph that unequivocally shows their rapid decline in whole-sector market share: the quantitative support for my hypothesis.

I didn't understand that when "the pie is rapidly growing", as in technology, companies can survive, even increase sales, whilst their market-share falls off a cliff. The model is IBM post-1980, not Unisys post the 1986 merger of Burroughs and Sperry, with their revenues shrinking by around 10 times.

Telecomms Industry Structural Changes

There are three structural economic changes that Telstra has to master to survive another 15, let alone 35 years:

Sector Economic Reform through "Digital Convergence".
Open Access wholesale networks competing for all services except mobile 3G/4G.
Good Customer Service and Cost-Effective Operations.

Quality, Performance Improvement and Profitability are inter-dependent, all are based on Reviews and deliberate Corporate Learning and Adaption.

Telstra have regarded the vertically integrated network as their "birthright", operated under the "Traditional" Telco Business Model ("what the market will bear" not "cost plus margin") and relied on captive markets. Once they could claim "Engineering Excellence" as a counter to outrageous Customer Service, but not so for the last 2 decades. Their management mindset must change to accept current conditions, or with the NBN, they will "lose the farm".

All these "pillars" of their business are being shattered, first by Internet Everywhere, by the NBN removing their customer lock-in and an increasing number of Technology Businesses that "get" Apple's insight and innovation: The User/Customer is all important for your Business.

What Telstra should be doing to undo the resulting Brand Damage

Below is a case study that Telstra should deeply investigate as it encapsulates most of their challenges/deficiencies and could be used as an on-going Reference for Change, but why would they?

In an ideal world, the centre of the study would have these outcomes:

A personal meeting with the Head of Telstra for the State.
An apology from him, a guarantee it would never happen again and his personal phone number if further problems arose.
An audit of all records for their services and accounts to correct all errors.
A written account of:

Exactly what went wrong,
Why it couldn't be fixed, and
Why it won't recur.

An offer of compensation for the non-supply of service, for the hours of customer time wasted on the phone and waiting and an ex-gratia payment for the "pain and suffering" caused.

Does any of that sound "over the top" to you?

Consider for a moment, "What would Richard Branson do?". If he was in the country, he'd personally see them, otherwise it would be someone very senior and it would be done very quickly. Stories about his interventions are legion, this is not wild speculation.

If you think Branson and his Virgin Empire are "off with the pixies" and not in the real-world of Big Bureaucracies, mass workforces and challenging business environment, consider the page, "Turn Complaining Customers into Advocates" by The Royal Mail, one of the oldest communications companies on the planet, working under one of the most demanding Industrial Relations systems, riven by Unions and staffed by British Workers, renowned for their lack of customer empathy and poor work-ethic.

If The Royal Mail management understands Customer Complaints are opportunities to both fix your business processes and to convert a hostile customer who'll damage your brand into a strong Brand Ambassador for you, then why don't Telstra?
Don't they understand the rules of Customer Service or read the same well known management books?

What Telstra says it does

In the 2005 Telstra presentation, released to the ASX, supporting their 20,000 node 12Mbps ADSL2 network, I was very impressed with their guiding principles (p3), but those are nowhere to be seen in this case:

Principle #1: Do it once

Right first time, every time
Simplify, standardise, focus
Less of everything – fewer products, platforms, applications, processes, vendors
Capture the benefits of scale through focus

Principle #2: Do it right for the customer

Invest against the things customers value

Principle #3: Do it in an integrated way

One Factory
End to end approach
Whole greater than the parts

Principle #4: Do it at the lowest unit cost

Scalable
Costs grow slower than revenues and volumes
Limited manual intervention

Case Study

The facts of the case study and an analysis of causative Systemic Failures are in a previous post.

Psychological Dimension: Stirring strong customer abreactions

Feelings of Frustration, Powerlessness and Agitation in response to poor Customer Service aren't a "minor annoyance" or idiosyncratic: there is some very deep human psychology involved.

The positive effects of Goal Attainment means the inverse, preventing people from achieving goals, is devastating, more so for high-performing individuals as here. If intermixed with multiple events setting up false hopes and then dashing them, the customer response is even more profound.

Treating customers badly, especially when you know about it, is really bad for business. The cumulative Brand Damage may not be curable. It will cause massive customer revolt and backlash when they have reasonable service substitutes available, such as from the NBN.

Business Consequences

This whole episode was preventable: it was clearly an internal fault within Telstra systems.
It wasn't a user-error (the Client did nothing), it wasn't a hardware, connection, patching or line fault nor a an accounting or software error.

My speculation is that Telstra has significant service database errors since the $10B 2005 (1st phase live in 2007) "IT Transformation Project" led by Greg Winn, one of Sol's "Three Amigos" whom returned to the USA with full saddle-bags. It seems complex, high-value customers like the Client were never catered for, from the 2009 article on the project over-running by $200MM (2%):

Thodey said 9.2 million customers have moved onto Telstra's new billing and CRM systems, which represented over 70 percent of the carrier's customer base.

The final thirty percent were "multi-product holding customers" he said - referring to those Telstra customers that use more than one of the carrier's services.

The fault had something to do with a modem attachment being incorrectly setup in the database, possibly by an automatic provisioning system attached to the order/fulfilment system.

The ARPU for the single service is $60-$80/month. Total revenue on this account, would be $1-2,000/month. Gross Margin must be 30-60%, Net Margin more like 10-15%??

This whole episode put at risk $10-15,000/year on-going revenue for a $10/month Net Margin. I'm sure Telstra won't bother to detail and account for the cost of the event. Why would they? The fault is fixed.

Over the six week period, there must have been:

30-50 phone calls
20 staff directly involved and 10-20 indirectly or in 'backroom'.
50-100 hours of phone calls [$50/hour?]
4-6 site visits, each 1 hour or more [$150+/hour]
10-40 hours of marketing and engineering effort [$100+/hour]

This preventable error has cost Telstra close to $10,000 for under a $100/year return. They can never make their money back. They are also quite likely to lose all of the Clients business, forever, if they don't directly attempt to follow The Royal Mail's approach and win them back.

As a shareholder, the Client was frustrated that the business was wasting money so prolifically, yet the organisation resisted all efforts to hear this news.

The worst aspect is that Telstra seems oblivious to any need to learn from this affair and follow their own Principles espoused in 2005, "Do it once, Do it right for the customer and Do it in an integrated way".

Lessons

If faults aren't covered by the (telephony) Customer Service Guarantee, then Telstra behave very poorly towards Customers.
The Telstra Customer Service and Complaints fails dismally with complex issues.

There appears to be no recognition of "process faults" or identification of "not previously seen" faults.
There appears to be no fault escalation process.

Shareholders are not treated better than anyone else. A marketing opportunity to improve shareholder relations going begging.
Busy people's time is worth a lot to them, yet Telstra fail to acknowledge this nor provide ways to bring more certainty to site visits. Telstra could help itself and customers by:

Having registered 'home sitters' that customers could use to allow them to carry on with their lives, or
Telstra could charge extra for shorter attendance windows (2 hours, 1 hours, 30 mins). If this is allowed by the ACCC and Telco Regulations, it would earn them considerable money and by only reordering technician visits within a single day, not affect service calls.

Telstra seems not to have a culture of Review Incidents, Learn from Mistakes to intentionally Improve Service, Profits and Productivity.

Telstra has a major improvement opportunity here and seems to be deliberately discarding it, being intent on destroying customer goodwill and shareholder value.

Prognosis

Within 10 years, NBN-Co has planned to displace most of Telstra's wholesale copper network with fibre.

By then, all Retail Providers that can provide good Customer Service will beat Telstra in the marketplace. We are likely to see many small retailers who can offer good, local service, like ISP's, as well as a few large existing companies that compete solely on price,

The NBN seems to be the "magic bullet" that will allow customers to change and release decades of pent-up frustration with Telstra and their oligarchy and monopolistic mindset. The Internet and smartphones/mobile devices have changed the rules of the Telecommunications forever.

If Telstra doesn't learn the lessons of Great Customer Service practiced by the likes of Richard Branson, their only competitive asset will be their 4G mobile network. Which, because they haven't allowed competitors open access via third-party roaming, and forced them to overbuild networks (like HFC Cable TV) is of very little value.

Unlike Microsoft, Telstra is in a Mature Market with moderate, but non-zero, barriers to entry. In a low-growth market like Australia, its revenues will be "eaten" by others, it will follow the Unisys path downhill, but like Unisys, is likely to remain as a brand or engineering operation, though nothing like they are now.

In the same way that I viewed Microsoft as entering a challenging period, I think Telstra is as well, though I don't have a way to estimate or forecast the timeline.

My father spent his entire working life within PMG/ABC/Telecomm/Telstra and it was one of Australia's finest achievements for many decades. I doubt he would be proud of what they've become and I am saddened at their fall from grace.

2012/10/04

Telco Customer Service Madness: Case Study

Will Telstra, as it is now, survive to see the NBN contracts end in 35 years?
My view: It won't, not in its current form because of multiple failures within the Organisation.

Below is a case study that Telstra should deeply investigate as it encapsulates most of their challenges/deficiencies and could be used as an on-going Reference for Change, but why would they?

In an ideal world, the centre of the study would have these outcomes:

A personal meeting with the Head of Telstra for the State.
An apology from him, a guarantee it would never happen again and his personal phone number if further problems arose.
A desk audit of all records for their services and accounts to correct all errors.
A written account of:

Exactly what went wrong,
Why it couldn't be fixed, and
Why it won't recur.

An offer of compensation for the non-supply of service, for the hours of customer time wasted on the phone and waiting and an ex-gratia payment for the "pain and suffering" caused.

Case Study

The facts of the case study are:

Customer, 'A', has on their account multiple individuals, multiple service addresses, and multiple services for each individual and service address (mobiles, landlines, ADSL, Cable TV, Cable Internet, ...).

Whilst these are all domestic services, Telstra regularly deals with this complexity for SME's.
They are a "high-value" Telstra customer. This seemed irrelevant in the process.
Unsure if all individuals and services are billed together or by separate, linked accounts.

'A' is also a Telstra shareholder, which seems to have been irrelevant in the process.
'A' is highly educated, has run businesses and is well conversant with modern PC's and networking, relying on it for work and private life.

There are multiple family members who are quite I.T. literate and provide in-home I.T. support and troubleshooting.

A new Cable Internet service was ordered by 'C' in June. (date?)

The modem was never delivered.
When queried at the Telstra shop, customers were advised "the order had been cancelled".
The customers had not cancelled the order, nor been advised of that action.

'A' had a working Cable Internet service that then became intermittent. It met their needs and wasn't reported as a fault due to very poor past customer experiences.

"Not wholly broken, don't tempt fate" was the reasoning.

'B', another of the service holders, took it on themselves to report the fault to Telstra.
The first technician attended on 23rd-August, intending to change the cable modem.

They were unable to rectify the fault, did not replace the cable modem as it was serviceable and left saying "there is an error", which at some point changed to "an activation error".
The replacement cable modem was left on-site, unconnected.
'A' was told the install failed because of "Error Code CCP0012", and Tech suggested that the system “thought” there was already a modem on order.
Technician advised 'A' to call the general BigPond Enquires number (137 663), quote the Error Code, and the fault would be fixed.

Multiple technician attendances were booked:

Technician did not attend, did not phone customer. More than once? (date?)
Technician sent to wrong address, an old service address on the account. (17-Sep-2012).
Technician 'M' attended (19-Sep-2012), gave customer personal contact number and spent considerable time on-site and continued to work at resolving the fault.

Possibly instrumental and worthy of commendation.

'M' followed-up a week later (25-Sep-2012) saying:

TRG (Technical Response Group?) were aware of the problem,
other customers (in the area, state, nationally?) were affected and
TRG didn't know when or if the Error Code could/would be cleared.

There were a large number of unsatisfactory and long (1-4 hour) calls to the "Help Desk". e.g. 18-Sep-2012 following Technician no-show.

'A' was repeatedly shunted between departments (Accounting, Technical, ...), with no-one taking responsibility. The call finally dropped whilst 'on-hold'.
No evidence on subsequent calls of any knowledge of previous calls. Every call was a return to the "pass the parcel" with no person/department taking responsibility.

A Telstra complaint was lodged (04-Sep-2012), 'A' was given a "trouble ticket" number and told to contact Technical Support (number supplied). [[Two people assigned to the case (?), with promises to call-back within 24 hours.]]

Tech Support called (11-Sep-2012), on-site visit booked for following week (17-Sep).
Being able to speak to someone with "English as a First Language" had been an immense relief to 'A'. Finally their concerns were noted and seemed to be taken seriously.
Neither person called 'A' back within 24 hours.
When contacted, the complaints folk said they'd tried to contact 'A' using an incorrect phone number, one 'A' had never held. No apology was made for this. The complaints people could not correct the database error.

Having the number corrected took a good deal of time and effort in itself. Multiple departments claimed "can't do it" or "not my area".

'A's mobile phone number has been registered with Telstra as their primary contact point for more than a decade. Why were any of the databases incorrect?

After this (mid-late Sep-2012?) a very confident Telstra employee rang and identified themselves as "Level 3 support" and embarked on a very long and trying support call. They reassured 'A' that they could and would fix the fault.

Under instruction, the replacement modem was connected by 'A' and failed to work.
When the original modem was reconnected, it failed to work as well.

The service was now non-operational and the support person left it that way.
No apology or explanation was offered.
The "support" person did not book a recall or ever call back.

'A' was nonplussed: Telstra had oversold their competency and destroyed a usable service without progressing resolution of the fault.

'A' visited a local Telstra Shop (26-Sep-2012). Wished:

a credit for the time the service was not provided, and
to cancel the cable internet service.
'A' was told that because of the technician visit arranged for the next day, the service could not be cancelled. The Telstra Shop staff were not interested that the fault had not been fixed in a month.
'A' had wished to speak, as a shareholder, to someone senior about costs to the business for the fault. The manager was not present, no meeting was organised.
'A' had wished to request checking and correct all related account and service records. This was not organised either.

'A' purchased a Telstra prepaid wireless modem from Australia Post (26 or 27-Sep-2012), unable to get working after spending time with Call Centre. Device returned. (date?)

'A' bought a Vodafone prepaid wireless modem from Australia Post (26 or 27-Sep-2012) and after a few false starts, got it working and regained their Internet service.
Telstra Complaints officer called next day (27-Sep-2012) to say "we're working on it".
Telstra sent a standard e-mail survey following up on the prepaid wireless modem (bought 27-Sep-2012).

'A' detailed their disappointment in Telstra service and invited them to call.

(02-Oct-2012) A Melbourne based Customer Service rep.. 'R', called 'A' about the wireless modem and the on-going fault. 'R' said they would ring the next day.
(02-Oct-2012) The Teltra Complaints Officier assigned to 'A' called saying another person in their section would contact 'A' later that morning.

No call was received.
'A' left messages that afternoon and the next morning. These were not returned.

(03-Oct-2012) 'R' rang 'A' in a conference call including a technician , 'J' in Melbourne. 'R' had to leave the call early, with 'J' spending an hour on the phone with 'A', attempting "a manual override" of the Error Code. This required long waits and providing the hardware address of the original modem. This had to be read by 'A', 'J' did not seem to have this on record.

This over-ride appeared successul at the time.
The connection failed overnight.
It seems to be working today.
How will 'A' know the fault has been cleared?

They currently believe the fault is rectified.

Why wasn't this done on, or just after, the first site visit, six weeks earlier?

Why the long wait and run-around?

After the apparent resolution, 'A' had multiple calls from people within Telstra, all very excited the fault had been fixed.

None offered an apology or any compensation, some seemed to claim direct credit.
None offered an explanation of either the Technical fault within their systems, nor what had gone wrong with internal Telstra processes and automatic systems to cause the multiple faults suffered.
None offered a "magic phrase" to be repeated to Technicians and Help Desk about the fault should it recur.
No on-going "trouble ticket" number was given to 'A', should the fault recur.
No-one offered a shortcut for service if the fault recurred shortly.

After the overnight service disruption, it wasn't clear if one of the other Telstra personnel had undone the "manual override" with an individual attempt to rectify the fault they'd claimed.

There was no evidence of good co-ordination amongst the various Telstra "Silos".

'A' had concluded on 3 October, that Error Code CCP0012, is not a technical problem, nor is it an accounting problem, but an Activation error problem and simply a code that needs to be removed from the Telstra system to allow the modem to connect and activate.
'A' raised a complaint with the TIO (Telecommunications Industry Ombudsman) (03-Oct-2012) sending their records of the incident.
'A' had been originally told "there is construction work in your area, a cable may have been cut". This seems to have been a deliberate, misleading statement.
Telstra did not give any hint that after a month:

That the fault had been escalated
That for failing to provide the service, they would rebate 'A' the service charge.
No offer was made to supply a temporary service, such as a 3G USB modem.

'A' had had to cancel a number of important business and personal meetings to wait aimlessly for a Telstra technician to attend on multiple occasions.

No option for an increased priority owing to the long-standing nature and difficultly of the fault was offered.
Telstra would never offer better than a 4-hour window for any attendance. They never scheduled 'A' at the beginning of the window, always near the end, or didn't attend.

Psychological Dimension: Induced Pain and Suffering

It's also worth noting that completely out of character, 'A' suffered extreme agitation, frustation and desperation at both the impenetrable wall of "service" and the inability to be heard, treated respectfully and to get a resolution to a service that had become necessary for conducting their business and life.

This isn't a "minor annoyance" or idiosyncratic: there is some very deep human psychology involved.

There's a branch of psychological therapy that relies on our limbic systems' (the cingulate nucleus) response to attaining goals: pre- and post-goal attainment happiness, two very distinct and important phases.

For all humans, striving and overcoming challenges is innate and core to our psychological well-being.

Consistently setting goals and achieving them isn't just "nice", but necessary, for our continued happiness and psychological well-being. Goal Attainment forms the basis of various powerful approaches addressing Depression and other conditions.

Knowingly and uncaringly forcing people into powerlessness and frustration would, in an OH&S workplace setting, be illegal: employers are required in Australia to provide a Safe Workplace. Deliberately causing employees harm, physical, emotional or psychological, is illegal and attracts civil penalties, as well as curative support for those affected.

I'm not sure if current OH&S law can be extended to customers. If so, companies like Telstra which seemingly have a policy and strategy of blocking communications and frustrating customers, would face considerable penalties...

The positive effects of Goal Attainment means the inverse, preventing people from achieving goals, is devastating, more so for high-performing individuals like 'A'. It can be categorised as "cruel and unusual" treatment, especially if intermixed with multiple events setting up false hopes and then dashing them. The human response in this case is even more profound and damaging.

Systemic Failures within Telstra

This whole adventure was unnecessary and presumably preventable: some automatic system failed when a new Cable Modem was ordered and incorrect configuration data uploaded to an operational system, without detection, audit or correction. Who has been charged with finding the root cause?

The final fix, a "manual override", should at worst, have been done the next day by the technician 'R' in Melbourne, prompted by the call from 'A'.

If Telstra's fault resolution system had worked properly, the fault would have been automatically passed to 'R's section as soon as the first technician recorded the Error Code.

Telstra has no fault escalation procedures, conclusively demonstrated here.

After any fault has been open/unresolved for two weeks, it should have been escalated to the Head of Operations for the State.
Any fault that is due to an internal process failure, like this, should be immediately escalated to senior officers with full cross-organisational authority and access to diagnose the root cause and initiate permanent prevention measures.
Own-goal process faults like these need to be reviewed, tracked and addressed by the CEO and their Senior Management Team.

They threaten the viability of the whole business and sufficient Responsibility and Authority only comes together at the top of all Silos, the CEO and their team.

High-value multi-service clients are treated worse than low-value customers: there are demonstrated errors in Service and CRM databases.

There is a major deficiency within Telstra: nobody is checking and correcting these service records.

After the "IT Transformation" project, it was known that high-value customers could not be automatically transferred.
To have stale data in multiple locations (service address, customer contact) says the database is seriously compromised, leading to many costly preventable errors.

Who is responsible and accountable for Data Quality, and do they have the Authority, will and budget to force records to be corrected?

This appears to be a major organisational failing and oversight.

Correcting faults in Telstra records is onerous and time consuming for Customers, it should be simple and easy.

It cannot be done in real-time whilst speaking to Service Reps, or
Service Reps are poorly trained or refuse to execute their tasks.

Telstra shareholders are treated no better than anyone else.

This is a marketing opportunity going begging to create engaged and supportive shareholders. Telstra has one of the largest 'Mom and Pop' share registers.

Service discounts, special offers and loyalty bonuses are possible.
Special service and access arrangements for shareholders would encourage them to give all their business and that of their immediate families to Telstra.

Telstra BigPond seems to offer nothing like the Telephony Customer Service Guarantee (CSG).

The Internet is a vital lifeline personally, professionally and in business for almost all Telstra customers now.
Decent Service Guarantees would match Consumer expectations and usage, as well as provide Product Differentiation.

Telstra's Offshore "Help Desk" with ESL speakers are counter-productive, especially for complex, long-running faults.

Whilst possibly tolerable for simple tasks and "script driven" data acquisition, they are a Nett Negative Value in this situation and many others. Saving money on Help Desks may be illusory and detrimental to the whole business.

Allow Customers to choose more expensive support options:

Like Airlines, offer multiple levels of pay-for-service, allowing the business to maximise profits by offering multiple price-points. (No "money left on the table").
Higher cost support could be automatically included as 'upgrades' for high-value customers, as Banks do.

After two calls on the same fault, automatically direct the Customer to a specialist Held Desk with a single person assigned and responsible for achieving Customer Satisfaction.

Instead of measuring "time to finish or transfer call", complex calls need to measure "Time until Customer is fully satisfied". Only the Customer can close complex faults.

The Help Desk practice of "pass the parcel" is frustrating to Customers and Counter-productive as it causes significant Brand Damage.

Somebody within Telstra needs to be responsible for detecting, monitoring/reporting and preventing this situation.
Ideally, the phone system should track customers who are passed around and offer them a "circuit breaker".

Making Customers wait on Help Desk queues for hours serves no purpose other than weeding out those with better things to do, prompting them to look for alternate service providers.

Long Help Desk delays are an invitation for Customers to Choose Another Carrier, a tactic which would not impress shareholders one iota.
Under provisioning Help Desk service staff only serves to reinforce the stereotypical image provided by Lily Tomlin in her "We're the Phone Company" sketches. This is against the best interests of the Business.
This reinforces Telstra as a Toxic Brand to Customers. Whilst Customers have no better place to go, they will tolerate it. Given the choice, they will flee, never to return.
This is known, preventable Brand Damage at its worst.

Complex faults are slow and difficult to solve. This isn't simply a Customer Service and Brand Damage issue, but very expensive to the organisation.

This fault cost $5-10,000 more than it should have.
We know that this wasn't a one-off and that other Cable Internet subscribers were affected, but their faults weren't resolved.

The initial problem, the non-supply of an ordered service, was never addressed.

How much business can an organisation deliberately throw away and survive?
It seems nobody is directly responsible or accountable for this lost revenue.

Telstra, if it wants to engage and retain customers, must never internally cancel a service order without contacting the customer, explaining the situation and offering alternatives, It's an opportunity to "upsell" the client.

There is a major fault with the Technician ticket system: The first on-site Technician should not have been able to pass a known, unresolvable fault back to general enquiries.

Telstra confirmed that "TRG":

knew of the "Error Code CCP0012" fault,
that it affected multiple customers,
presumably they had no idea of the immediate or root cause, and
they had no idea of when they would be able to fix it.

In ITIL-speak, this was a Severity One Major Problem, but wasn't classified as such.

An appropriate organisational response would've been to establish a war-room comprising the State Heads of Branches and Senior Line-of-Business Managers.
Following the successful work-around, all other faults associated with the Problem should've been corrected.
Investigations initiated as to the root causes (automatic systems and processes) responsible for the error and means of detecting recurrences and costs of prevention measures.

Telstra's Problem Management is either deficient or non-existent.

Problems are not Faults, but the cause of one or more Faults.
To have a Known Problem not detected by the Fault Ticket Handling system is a major Professional failure that should be explicitly investigated and reviewed.
All the Help Desk and Technical Systems should've found an outstanding Problem with "Error Code CCP0012", with a known workaround ("manual over-ride").

The systems and processes need Review and correction.
A major internal inquiry is needed to uncover the root causes of this meta-failure.

Multiple Telstra employees were contacting the Client unbeknownst to one another.

This is the compelling reason for a CRM, a "single Client Communication Flow".
This definitively failed, either because the CRM was faulty, or it was bypassed or procedures ignored.

All of these are cause for deep concern and deserve an inquiry.

The inability of the Complaints Officer to progress the issue, or identify it was a Known Unresolved Problem, means their systems and/or processes are deficient or faulty.

This is a major problem deserving immediate attention.

The fault was only resolved accidentally when a Customer Survey person became involved and somehow was able to refer the fault to a diligent, competent and active Technician.

The unidentified "level 3" support person that caused the service to fail entirely should be found and castigated, as should the many service personnel who failed to follow-through on the fault.

These actions are consistent with a widespread attitude of "Care Factor: Zero", inimical to resolving faults or preventing faults through good Problem Management.
The one Technician who persevered should be found and commended.

The consistent lack of apology to the Client, the lack of any offers to provide an alternate service until the service was restored or anyone offering the statutory minimum (under the TPA/CCA) of rebating service charges show a systemic failure in even adequate, not good, Customer Service training and knowledge of legal requirements.

At a minimum, this is a systemic Training failure.
It indicates that nobody is monitoring, measuring and reporting on general levels of Customer Service and evaluating adequacy of Training.

The most critical and over-arching and pervasive Failure is the identification and investigation/analysis of massive cost over-runs on service faults etc:

This fault cost the organisation an unnecessary $5-10,000, more than the $100/year service margin could ever return.
This will not be an isolated occurrence, many of these will be eating away at Profits and turning away Customers, particularly high-value long-term Clients.
This only got resolved through an accidental interaction, the Customer Survey person who bothered to follow-up on the feedback. This bodes very poorly for the future performance on the organisation.
By rights, Telstra should have standing reports with automatic escalation:

Identify full internal cost to resolve faults and other service issues.
Report and escalate excessive fault resolution costs to Senior Management.
Mandate Root Cause Analysis (RCA) of the "Top Ten" faults found in the RCA's.
Require the CEO and their Senior Management Team to track all "Top Ten" issues and regularly report to the Board on progress and problems identified.

The irony is that I shouldn't be writing this analysis at all. None of this should've happened in a well-run organisation that cared sufficiently for its Customers.

The tragedy is that Telstra will probably continue "Fat, Dumb and Happy" for the next 10-15 years in blissful ignorance of this piece and then wonder why they are "suddenly" losing Customers, disproportionately their most valuable, at an accelerating rate.