If you run a Healthcare-realted Business, things changed in the last 6 months...
Ransomware is set to boom [0] and cyber-security is now part of our National Security Plan.
Businesses now have to secure their computers and data just as they secure their premises and goods.
It's not optional, fail to do so and you will go out of business, just when is the question.
Ask yourself this: "If my computers were destroyed, how long could I continue the business? At reduced capacity or at all?", then act accordingly.
i.e. Does anyone around the world see you as a high-value, exploitable target?
Especially those in low-income countries with employment problems: poverty corrupts, not just power or the love of money.
The Internet is defined by its explosive growth: A few For-Profit hackers have noticed Business Ransomware is an ideal way to monetise remote computer attacks & exploits.
The numbers of these attacks will now double every few months as word gets around, new "toolkits" are sold to them and they ramp up their activities.
Every business that can raise $5,000 and relies on its systems and data for daily operations is now in their sights. These people have no morals, ethics or compassion in their work: they want your money and don't care about the damage they cause or the impact of their actions. Appeals to them will fall on deaf ears. Neither believe that a single ransom payment will be the last you'll hear of them. Why would you trust the word of criminals who've already broken in and callously damaged your systems?
Showing posts with label IT Service Management. Show all posts
Showing posts with label IT Service Management. Show all posts
2013/01/26
2011/09/14
A new inflection point? Definitive Commodity Server Organisation/Design Rules
Summary:
For the delivery of general purpose and wide-scale Compute/Internet Services there now seems to be a definitive hardware organisation for servers, typified by the E-bay "pod" contract.
For decades there have been well documented "Design Rules" for producing Silicon devices using specific technologies/fabrication techniques. This is an attempt to capture some rules for current server farms. [Update 06-Nov-11: "Design Rules" are important: Patterson in a Sept. 1995 Scientific American article notes that the adoption of a quantitative design approach in the 1980's led to an improvement in microprocessor speedup from 35%pa to 55%pa. After a decade, processors were 3 times faster than forecast.]
Commodity Servers have exactly three possible CPU configurations, based on "scale-up" factors:
[Update 06-Nov-11: Because Oracle insists some feature sets must run on raw hardware. Sometimes vendors won't support your (preferred) VM solution.]
VM products are close to free and offer incontestable Admin and Management advantages, like 'teleportation' or live-migration of running instances and local storage.
There is a special non-VM case: cloned physical servers. This is how I'd run a mid-sized or large web-farm.
This requires careful design, a substantial toolset, competent Admins and a resilient Network design. Layer 4-7 switches are mandatory in this environment.
There are 3 system components of interest:
Consequentially, "Fibre Channel over Ethernet" with its inherent contradictions and problems, is unnecessary.
Designing individual service configurations can be broken down into steps:
As a professional, you're looking to provide "bang-for-buck" for someone else who's writing the cheques. Over-dimensioning is as much a 'sin' as running out of capacity. Nobody ever got fired for spending just enough, hence maximising profits.
Getting it right as often as possible is the central professional engineering problem.
Followed by, limiting the impact of Faults, Failures and Errors - including under-capacity.
The quintessential advantage to professionals in developing standard, reproducible designs is the flexibility to respond to unanticipated load/demands and the speed with which new equipment can be brought on-line, and the converse, retired and removed.
Security architectures and choice of O/S + Cloud management software is outside the scope of this piece.
There are many multi-processing architectures, each best suited to particular workloads.
They are outside the scope of this piece, but locally attached GPU's are about to become standard options.
Most servers will acquire what were known as vector processors and applications using this capacity will start to become common. This trend may need their own Design Rule(s).
Different, though potentially similar design rules apply for small to mid-size Beowulf clusters, depending on their workload and cost constraints.
Large-scale or high-performance compute clusters or storage farms, such as the IBM 120 Petabyte system, need careful design by experienced specialists. With any technology, "pushing the envelope" requires special attention by the best people you have, to even have a chance of success.
Not unsurprisingly, this organisation looks a lot like the current fad, "Cloud Computing" and the last fad, "Services Oriented Architecture".
Google and Amazon dominated their industry segments partly because they figured out the technical side of their business early on. They understood how to design and deploy datacentres suitable for their workload, how to manage Performance and balance Capacity and Cost.
Their "workloads", and hence server designs, are very different:
For the delivery of general purpose and wide-scale Compute/Internet Services there now seems to be a definitive hardware organisation for servers, typified by the E-bay "pod" contract.
For decades there have been well documented "Design Rules" for producing Silicon devices using specific technologies/fabrication techniques. This is an attempt to capture some rules for current server farms. [Update 06-Nov-11: "Design Rules" are important: Patterson in a Sept. 1995 Scientific American article notes that the adoption of a quantitative design approach in the 1980's led to an improvement in microprocessor speedup from 35%pa to 55%pa. After a decade, processors were 3 times faster than forecast.]
Commodity Servers have exactly three possible CPU configurations, based on "scale-up" factors:
- single CPU, with no coupling/coherency between App instances. e.g. pure static web-server.
- dual CPU, with moderate coupling/coherency. e.g. web-servers with dynamic content from local databases. [LAMP-style].
- multi-CPU, with high coupling/coherency. e.g. "Enterprise" databases with complex queries.
[Update 06-Nov-11: Because Oracle insists some feature sets must run on raw hardware. Sometimes vendors won't support your (preferred) VM solution.]
VM products are close to free and offer incontestable Admin and Management advantages, like 'teleportation' or live-migration of running instances and local storage.
There is a special non-VM case: cloned physical servers. This is how I'd run a mid-sized or large web-farm.
This requires careful design, a substantial toolset, competent Admins and a resilient Network design. Layer 4-7 switches are mandatory in this environment.
There are 3 system components of interest:
- The base Platform: CPU, RAM, motherboard, interfaces, etc
- Local high-speed persistent storage. i.e. SSD's in a RAID configuration.
- Large-scale common storage. Network attached storage with filesystem, not block-level, access.
Consequentially, "Fibre Channel over Ethernet" with its inherent contradictions and problems, is unnecessary.
Designing individual service configurations can be broken down into steps:
- select the appropriate CPU config per service component
- specify the size/performance of local SSD per CPU-type.
- architect the supporting network(s)
- specify common network storage elements and rate of storage consumption/growth.
As a professional, you're looking to provide "bang-for-buck" for someone else who's writing the cheques. Over-dimensioning is as much a 'sin' as running out of capacity. Nobody ever got fired for spending just enough, hence maximising profits.
Getting it right as often as possible is the central professional engineering problem.
Followed by, limiting the impact of Faults, Failures and Errors - including under-capacity.
The quintessential advantage to professionals in developing standard, reproducible designs is the flexibility to respond to unanticipated load/demands and the speed with which new equipment can be brought on-line, and the converse, retired and removed.
Security architectures and choice of O/S + Cloud management software is outside the scope of this piece.
There are many multi-processing architectures, each best suited to particular workloads.
They are outside the scope of this piece, but locally attached GPU's are about to become standard options.
Most servers will acquire what were known as vector processors and applications using this capacity will start to become common. This trend may need their own Design Rule(s).
Different, though potentially similar design rules apply for small to mid-size Beowulf clusters, depending on their workload and cost constraints.
Large-scale or high-performance compute clusters or storage farms, such as the IBM 120 Petabyte system, need careful design by experienced specialists. With any technology, "pushing the envelope" requires special attention by the best people you have, to even have a chance of success.
Not unsurprisingly, this organisation looks a lot like the current fad, "Cloud Computing" and the last fad, "Services Oriented Architecture".
Google and Amazon dominated their industry segments partly because they figured out the technical side of their business early on. They understood how to design and deploy datacentres suitable for their workload, how to manage Performance and balance Capacity and Cost.
Their "workloads", and hence server designs, are very different:
- Google serves pure web-pages, with almost no coupling/communication between servers.
- Amazon has front-end web-servers is backed by complex database systems.
2009/11/20
I.T. Failure == Corporate Failure
Stephen Bartholomeusz writing in Business Spectator, 18 Nov 2009, on the ASIC court case over the collapse of One.Tel.
Bartholomeusz neatly summarises the root cause of the failure:
This is the first case I've noticed where the immediate cause of failure of a large, public company has been it's I.T. systems. The root cause is poor management with an inability to execute - or to understand and control it's I.T.
The field of "Software Engineering" is 40 years old now.
How could this foreseeable and preventable failure have happened with competent professionals, especially if Software Engineering had achieved it's aims?
There is a multiple tragedy hidden here:
Why are ASIC, the ASX and the Federal and State Governments silent on this point?
If not their job, then whose?
Imagine if QANTAS had a fire at a maintenance facility and lost $1B of buildings, plant and equipment. You know absolutely the company, multiple regulators and all the professional bodies would actively investigate the matter.
They would be looking for "root causes" of this event, other problems, ways to fix the system, processes & procedures to prevent or early-detect this class of problem again and co-incidentally if any individuals were responsible. Not just front-line grunts, but if anyone in management (up to the CEO) was culpable, negligent, incompetent or asleep-at-the-wheel.
The absolute tragedy here is not the loss to these investors (employees, vendors, customers, ...) but that nothing is going to change, that this massive loss bought nothing.
What is more galling to me is that nobody in the Press, Government, ASX, Investment bodies, Judicary or Regulators thinks anything more could or should be done...
Bartholomeusz neatly summarises the root cause of the failure:
Unhappily, its billing systems didn’t work, so it piled up debtors, while its competitors responded to the cut-price strategies.He goes on to say:
While professing publicly that the group was on-track to be cash-positive..., internally One.Tel appears to have had little control or understanding of its cash flows or the mounting issues created by its billing systemand finishes:
Whatever Rich might claim, One.Tel wasn't a successful company, unless success is measured by revenue, not cash flows or execution.
This is the first case I've noticed where the immediate cause of failure of a large, public company has been it's I.T. systems. The root cause is poor management with an inability to execute - or to understand and control it's I.T.
The field of "Software Engineering" is 40 years old now.
How could this foreseeable and preventable failure have happened with competent professionals, especially if Software Engineering had achieved it's aims?
There is a multiple tragedy hidden here:
- Software Engineering has failed to impress it's primary market: Business Management.
- Educators and Researchers are not, as a matter of course, going to analyse this failure and use it as a case study. Compare the 1974 explosion at Flixborough or the 1970 collapse of the Westgate bridge during construction.
- IT practitioners aren't going to be informed by their Professional Societies of the causes and preventing a recurrence.
- Business Management and I.T. practice remains "Consequence Free".
Why are ASIC, the ASX and the Federal and State Governments silent on this point?
If not their job, then whose?
Imagine if QANTAS had a fire at a maintenance facility and lost $1B of buildings, plant and equipment. You know absolutely the company, multiple regulators and all the professional bodies would actively investigate the matter.
They would be looking for "root causes" of this event, other problems, ways to fix the system, processes & procedures to prevent or early-detect this class of problem again and co-incidentally if any individuals were responsible. Not just front-line grunts, but if anyone in management (up to the CEO) was culpable, negligent, incompetent or asleep-at-the-wheel.
The absolute tragedy here is not the loss to these investors (employees, vendors, customers, ...) but that nothing is going to change, that this massive loss bought nothing.
What is more galling to me is that nobody in the Press, Government, ASX, Investment bodies, Judicary or Regulators thinks anything more could or should be done...
2008/03/07
Service Desk and Politician e-mail
Over the last year I've penned 6+ e-mails to various Labor Party politicians - including one of my local representatives who've I dealt with for ~10 years.
And not one reply. Zero, Zip, Nada...
Rang the Good Person's electoral office today - and got various run-around responses. "Oh, I've been on holiday", "Oh, can they call you" and "they are booked solid for a month".
Yeah, right.
I first contacted my rep. last December saying "this can wait until after the School Holidays". January came and went, no reply... A follow-up email yielded nothing... A note to the support staff was replied to: "I've moved. XXX is responsible".
What I originally wanted to talk about was 3 emails I'd sent various members without even getting acknowledged. Which is strange, because in the media I've seen reports that Political Parties are now tracking every contact from a voter. Putting together, apparently, impressive profiles - and all completely legit under the Privacy Laws.
For a new Government this seems a pretty poor response, doubly so for one that prides itself on 'listening'.
The solution that I wanted to put forward to my Rep:
Use HelpDesk Software to manage constituent contacts.
Not just piecemeal, but an integrated system for all participating elected members.
Not all that hard.
It scales. It goes across the whole Party. It covers both 'aph.gov.au' contacts and via other email addresses. It copes with email, phone, fax, mail and personal contacts - and the worst of all "voice prompt systems".
The software is well known, there are many vendors and trained consultants and the marketplace is competitive. As consumers and office workers, most of us are used to the concepts and who these systems all work.
It creates a definite process - with self-imposed rules & priorities that are checked and enforced.
AND it ensures that little people like me don't just fall between the cracks.
Or if some 'critical person' falls down - work queues can get given to those who can best deal with them.
Imagine getting a tracking number back from your local Pollie, and being able to automatically check where it is up to - and just when you should expect an answer. Wow! Just like they worked for us and were trying to use the technology responsibly...
It would do a service for our erstwhile representatives - you know, the ones we pay to work for us:
The Internet Changes Everything - but Politicans and their ways.
And not one reply. Zero, Zip, Nada...
Rang the Good Person's electoral office today - and got various run-around responses. "Oh, I've been on holiday", "Oh, can they call you" and "they are booked solid for a month".
Yeah, right.
I first contacted my rep. last December saying "this can wait until after the School Holidays". January came and went, no reply... A follow-up email yielded nothing... A note to the support staff was replied to: "I've moved. XXX is responsible".
What I originally wanted to talk about was 3 emails I'd sent various members without even getting acknowledged. Which is strange, because in the media I've seen reports that Political Parties are now tracking every contact from a voter. Putting together, apparently, impressive profiles - and all completely legit under the Privacy Laws.
For a new Government this seems a pretty poor response, doubly so for one that prides itself on 'listening'.
The solution that I wanted to put forward to my Rep:
Use HelpDesk Software to manage constituent contacts.
Not just piecemeal, but an integrated system for all participating elected members.
Not all that hard.
It scales. It goes across the whole Party. It covers both 'aph.gov.au' contacts and via other email addresses. It copes with email, phone, fax, mail and personal contacts - and the worst of all "voice prompt systems".
The software is well known, there are many vendors and trained consultants and the marketplace is competitive. As consumers and office workers, most of us are used to the concepts and who these systems all work.
It creates a definite process - with self-imposed rules & priorities that are checked and enforced.
AND it ensures that little people like me don't just fall between the cracks.
Or if some 'critical person' falls down - work queues can get given to those who can best deal with them.
Imagine getting a tracking number back from your local Pollie, and being able to automatically check where it is up to - and just when you should expect an answer. Wow! Just like they worked for us and were trying to use the technology responsibly...
It would do a service for our erstwhile representatives - you know, the ones we pay to work for us:
- They could become more efficient - by delegating work, not needing to deal with "whatever happened to" requests, and identifying common themes and selecting the most efficient way to respond.
- They could make a very exact case for additional clerical support from the Parliament - or even have a pool of paid staff doing the grunt work.
The Internet Changes Everything - but Politicans and their ways.
2007/12/29
IBM, Outsourcing and the IT Profession
This is a reaction to Robert X. Cringely's "Pulpit" of 28-Dec-2007:
Leaner and Meaner Still: IBM's U.S. operations continue to shrivel.
There are 3 parts to my comments:
They are interlinked. Lou Gertsner set IBM on the road on "Services" and away from Mainframes. It looked promising.
IT Services look very appealing on the Balance Sheet - nearly no investment (no tangible assets) and what seem to be good profits from turnover. The ROA and ROI (Return on Assets and Return on Investment) look great - until you take some other factors into account.
Quality is not 'gold-plating' - it is central to improving productivity, reducing waste and fulfilling customer expectations. These are the drivers for growth, profitability and sustainability - not penny-pinching and cost-cutting.
IT Services companies cannot, and will not, pursue Excellence & Quality if they are not driven to it.
It is only their Clients who can hold them accountable and force a change.
Concurrently, IT has to evolve from an Industry to a Profession so that managers can realistically evaluate the performances of different practitioners. It's not hard to win new business and make good profits if your employees are 10 times more productive than your competitions.
Answering the poll question: Will IBM survive?
Lou Gertsner turned IBM around, starting 1993.
It took an outsider to do it - and the board knew that.
His legacy, after leaving in 2002, should've been a company with a solid future. Five years on, it appears not so - that can only be "Corporate Culture".
IBM is far too important to be let fail and broken up in a firesale.
But we have a perfect model for the future of Cringely's "lumbering giant": Unisys.
In 1986, Numbers 2&3 in the market (Burroughs & Sperry Univac) combined and produced a dud. It's still alive, but failing. Because enough people use their mainframes (2200's and A-series), they can't be allowed to die. Slowly withering on the vine seems to be fine.
Fujitsu is the perfect vacuum-cleaner to buy the hardware business in the final break-up.
IBM GSA and the other 'Tier 1' outsourcers operate from the same playbook - a version of 'bait and switch'. Also known as "The Value Prevention Society".
I've worked with and for all the major outsourcers in Australia. They all bid low to win contracts and adopt a dual strategy of "controlling costs" and price gouging for "variations".
'Controlling costs' is reducing staff, replacing competent staff with 'cheap and cheerful' newbies, not performing maintenance and avoiding capital investment.
What's wrong with a 5-10 year-old system? Nothing if you don't have to suffer the performance and other problems!
They routinely ignore contract provisions - like scheduled roll-outs of new desktops, upgrades and system performance targets.
The problems are at least three-fold:
- inequality of parties (Outsourcer vs Client)
- internal 'manager' performance has no upside, only downside
- no impartial umpire and effective 'stick' to enforce system performance targets
Inequality
Every company that signs an IT Outsourcing agreement signs just one. The outsourcers has done this many, many times before.
Clients also don't factor in the increased staff and reporting costs - each side needs additional staff for 'contract management'.
The Client thinks it has stitched up an iron-clad contract and they forecast a bountiful harvest... Which doesn't happen.
Service degrades, minor works become hugely expensive, major works take forever and often don't get implemented.
The business people give-up and adapt around it.
In Australia, all the major EDS contracts let around 10 years ago are now being re-tendered - with EDS getting very little of the new work.
Are they the worst? Hard to say...
Aligning internal rewards with Client Needs
Outsourcer 'managers' can only be assessed on monetary performance. With fixed price contracts, base income is fixed.
If a manager reduces costs 5% one year, this becomes the expectation for every following year - it is not seen as a 'one-off'. Without significant staff training and capital expenditure, this quickly becomes impossible without sacrificing service quality. Commercial systems are quite reliable these days. For existing stable systems, 'Do nothing' is good for at least 3 years - then you are in deep trouble.
The only ways to increase profits are to reduce expenses or increase non-base income.
Every service request is deemed a 'change' and subject to the full, heavyweight, project evaluation methodology. No project, not even buying a simple standalone appliance, takes under 4 man-weeks ($20-50,000). For the client, this stifles change/innovation (or forces it underground) and these additional costs overshadow most systems costs.
Capital expenditures are worse. Payback has to be within 12-18 months - and it has to beat 'do nothing'.
Since the 2003 slowdown in Moores' Law for CPU speed, the problem has compounded.
Take a 5 year-old file server that is now close to saturated most of the day. It is not yet 'end of life' and maintenance costs still low.
Because file open/close, read/write performance is not specified and the system is "available" during work hours, the Client cannot complain.
The Operating System (O/S) may be old and need constant attention, updates and reboots - but they are part of the normal admin workload, so not an 'additional' cost. Salaried staff as 'professionals' must work any unpaid overtime that is demanded.
Any proposal to replace the server or upgrade it has to pass a simple, and reasonable, test:
'Do nothing' is the benchmark - for zero capital expenditure and a few extra unpaid admin hours, a service is provided that brings in the service full revenue - and will continue to do so. That's a very tough argument to beat.
Only when the client funds the replacement, hardware maintenance costs are high enough, an O/S upgrade is required for security or compatibility or qualified admin staff move on will the system be upgraded. And then it will begin the same inevitable slide into entropy and uselessness.
Finding solutions that benefit the customer and reduce operating expenses are career suicide for outsourcing staff in a culture focussed on increasing billables.
For example: a major Australian bank replaced all the local file servers with small Network Appliance NAS's. These are the most expensive product per Gb available. The outsourcer had charged ~$2,500/month to 'administer' these systems. The bank paid for the change in under a year, increased availability and performance and solving many other issues to boot.
If the client gives all its IT staff to the outsourcer, who is going to seek out, design and implement new cost saving technology/systems?
Not the outsourcer - it's not in the contract and not in its (short term) interests.
The client has no IT staff - so it cannot and doesn't happen.
Audits and an Impartial Umpire
Who reports to the Client on the performance of their systems?
Who has the training/qualifications to check and asses the metrics and reports?
Who maintains & audits the basis of payments - the asset register?
Only the Outsourcer.
What are the downsides to the Outsourcer of a major failure in Prime Time?
A small number of 'service credits'.
Meanwhile, the Client suffers real costs and potentially large losses.
The Client wears all the business and financial risk with only minor penalties to the Outsourcer.
We are yet to see a corporate collapse due to an outsourcers IT failures - but it is only a matter of time.
There is a clear conflict of interest, or an real Agency Theory problem.
The outsourcer is Judge, Jury and Executioner...
There is no way to hold them to account or dispute their figures.
The huge (100+:1) variability in individual competence and the inability to measure it is one of the worst problems in our industry.
IT is not a 'Profession'. It, like 'Management', fail a very simple test:
Mostly it is "fire/blame the innocent, promote the guilty". The exact inverse of what you'd want.
People may trump technology and process, but Politics trumps everything...
And our Professional Bodies don't help.
The only real research into the causes of Project Failure are by consultancies - who are driven by the ability to sell their products, not what will benefit the Profession.
The ACM, IEEE, IFIP and friends have abrogated their responsibilities. We on the firing line, get to suffer their inaction.
Managers have to go with what they can quantify and inspect. Good managers will see through the B/S - but mostly too little, too late. Mostly, office politics, influence and self-promotion rule.
The adversarial nature of Outsourcing and the seemingly universal decline in code and service Quality stems from this failure of IT as a Profession.
Steve Jenkin 29-December-2007
Leaner and Meaner Still: IBM's U.S. operations continue to shrivel.
There are 3 parts to my comments:
- Will IBM Survice?
- Outsourcing
- IT as a Profession
They are interlinked. Lou Gertsner set IBM on the road on "Services" and away from Mainframes. It looked promising.
IT Services look very appealing on the Balance Sheet - nearly no investment (no tangible assets) and what seem to be good profits from turnover. The ROA and ROI (Return on Assets and Return on Investment) look great - until you take some other factors into account.
- Barriers to Entry for competitors are low.
EDS under Ross Perrot came from nowhere to define and dominate the field - so can the next giant in the field.
If your business model is "hire cattle and drive them till they drop" - you have no market differentiation.
Same cattle, same drivers, same pay - same 'ol, same 'ol... The cattle aren't loyal, motivated or engaged.
Writing new contracts is a matter of perception, influence and contacts.
There is so much feeling against IT Outsourcers in business at the moment, the first company to come along and tell a better story will take the field.
The change won't be overnight, but fast enough that the incumbents won't notice until too late.
- Whilst only tangible assets appear on the Balance Sheet, IT Services are driven by your Human Capital and some Intellectual Capital embodied in your processes, branding and IP, such as trademarks and patents.
What value is let in the offices when everybody has gone home? Very, very little.
What is the business risk of a large, sudden exodus of your staff? A competitor may deliberately poach enough to put you in trouble.
It's a failing of the Board not understand this and institute appropriate metrics, accounting and management rewards.
- Profit based on Operations turnover are very fragile/volatile.
Income and Expenses are very large numbers with a small difference. Expenses are mainly employees - which you may not be able to shed as quickly as service contracts expire.
Tendering for new contracts implies you have, or can quickly get, the resources to fulfill the contract. That's an extreme business risk.
The key figures-of-merit are Income/Employee and Profit/Employee.
We don't see those reported or obviously managed.
- IT Services work is Knowledge Work - it is mostly invisible and intangible.
Driving IT staff like unskilled labourers with threats/punishment to lift performance is anti-productive.
Unhappy staff withdraw and pushback. At best they aren't engaged or motivated. They 'do the minimum' - a grudging compliance.
They stop caring about their work, the customer and their employer. And if you are lucky, it stops there.
Hiring bright, capable people doing intangible work and treating them badly is not just a recipe for disaster, it is foolishness writ large.
- IT is a cognitive amplifier and this can be leveraged both within the business and internally in IT.
The only sustainable strategy to deliver improved profits is through investment. - Automating tasks.
Applying our own technology to our jobs to make tasks, not jobs, redundant.
Investing in tools and hardware to increase the both Quality of work
- Building Human Capital.
Investing in the people at the work-face to build their capability and performance.
The SEI's Barry Boehem created COCOMO - a quantitative model for estimating Software costs.
Experienced, competent practitioners not only produce better work, fewer defects, faster - they are cheaper.
- Actively reducing Errors.
Consciously reducing waste, rework and wrong work.
Quality is not about 'doing the minimum', it's a mindset where Errors are allowed, but their repetition is anathema.
High Quality performances are only achieved with deliberate, focussed intention. Not blaming and denial.
Quality Systems only goal is to make it difficult for good people to make mistakes.
Deming said it all with "Plan-Do-Check-Act", or in new-speak: "Preparation - Execution - Review & Evaluation - Improvement" - Learning is central to improving Quality, Performance, Security & Safety and Usability.
Learning systems, processes and procedures takes an investment of time, tools and technology.
Failing to build teams and their capability will decrease expenses in the short-run and will increase them in the long-run.
- Resiling from the classic adversarial stance of IT Outsourcing.
IT is a business enabler. It is now central to normal business operations. It is still where 80% of efficiency improvements arise.
Every act that hurts the client will turn-around and hurt the provider, but more.
The client is earning the income that pays for the IT.
More income, more IT, more outsourcing revenue and profits. A simple equation that seems lost on Outsourcing managers.
Outsourcing contracts need to align the internal management rewards with improving business outcomes for the Client.
What's anathema to current management - reducing Client costs - must be aggressively pursued to create a long-term Outsourcing business.
Quality is not 'gold-plating' - it is central to improving productivity, reducing waste and fulfilling customer expectations. These are the drivers for growth, profitability and sustainability - not penny-pinching and cost-cutting.
IT Services companies cannot, and will not, pursue Excellence & Quality if they are not driven to it.
It is only their Clients who can hold them accountable and force a change.
Concurrently, IT has to evolve from an Industry to a Profession so that managers can realistically evaluate the performances of different practitioners. It's not hard to win new business and make good profits if your employees are 10 times more productive than your competitions.
Will IBM Survive?
Answering the poll question: Will IBM survive?
Lou Gertsner turned IBM around, starting 1993.
It took an outsider to do it - and the board knew that.
His legacy, after leaving in 2002, should've been a company with a solid future. Five years on, it appears not so - that can only be "Corporate Culture".
IBM is far too important to be let fail and broken up in a firesale.
But we have a perfect model for the future of Cringely's "lumbering giant": Unisys.
In 1986, Numbers 2&3 in the market (Burroughs & Sperry Univac) combined and produced a dud. It's still alive, but failing. Because enough people use their mainframes (2200's and A-series), they can't be allowed to die. Slowly withering on the vine seems to be fine.
Fujitsu is the perfect vacuum-cleaner to buy the hardware business in the final break-up.
Outsourcing
IBM GSA and the other 'Tier 1' outsourcers operate from the same playbook - a version of 'bait and switch'. Also known as "The Value Prevention Society".
I've worked with and for all the major outsourcers in Australia. They all bid low to win contracts and adopt a dual strategy of "controlling costs" and price gouging for "variations".
'Controlling costs' is reducing staff, replacing competent staff with 'cheap and cheerful' newbies, not performing maintenance and avoiding capital investment.
What's wrong with a 5-10 year-old system? Nothing if you don't have to suffer the performance and other problems!
They routinely ignore contract provisions - like scheduled roll-outs of new desktops, upgrades and system performance targets.
The problems are at least three-fold:
- inequality of parties (Outsourcer vs Client)
- internal 'manager' performance has no upside, only downside
- no impartial umpire and effective 'stick' to enforce system performance targets
Inequality
Every company that signs an IT Outsourcing agreement signs just one. The outsourcers has done this many, many times before.
Clients also don't factor in the increased staff and reporting costs - each side needs additional staff for 'contract management'.
The Client thinks it has stitched up an iron-clad contract and they forecast a bountiful harvest... Which doesn't happen.
Service degrades, minor works become hugely expensive, major works take forever and often don't get implemented.
The business people give-up and adapt around it.
In Australia, all the major EDS contracts let around 10 years ago are now being re-tendered - with EDS getting very little of the new work.
Are they the worst? Hard to say...
Aligning internal rewards with Client Needs
Outsourcer 'managers' can only be assessed on monetary performance. With fixed price contracts, base income is fixed.
If a manager reduces costs 5% one year, this becomes the expectation for every following year - it is not seen as a 'one-off'. Without significant staff training and capital expenditure, this quickly becomes impossible without sacrificing service quality. Commercial systems are quite reliable these days. For existing stable systems, 'Do nothing' is good for at least 3 years - then you are in deep trouble.
The only ways to increase profits are to reduce expenses or increase non-base income.
Every service request is deemed a 'change' and subject to the full, heavyweight, project evaluation methodology. No project, not even buying a simple standalone appliance, takes under 4 man-weeks ($20-50,000). For the client, this stifles change/innovation (or forces it underground) and these additional costs overshadow most systems costs.
Capital expenditures are worse. Payback has to be within 12-18 months - and it has to beat 'do nothing'.
Since the 2003 slowdown in Moores' Law for CPU speed, the problem has compounded.
Take a 5 year-old file server that is now close to saturated most of the day. It is not yet 'end of life' and maintenance costs still low.
Because file open/close, read/write performance is not specified and the system is "available" during work hours, the Client cannot complain.
The Operating System (O/S) may be old and need constant attention, updates and reboots - but they are part of the normal admin workload, so not an 'additional' cost. Salaried staff as 'professionals' must work any unpaid overtime that is demanded.
Any proposal to replace the server or upgrade it has to pass a simple, and reasonable, test:
How much extra revenue will we make? How long will the payback period be?
'Do nothing' is the benchmark - for zero capital expenditure and a few extra unpaid admin hours, a service is provided that brings in the service full revenue - and will continue to do so. That's a very tough argument to beat.
Only when the client funds the replacement, hardware maintenance costs are high enough, an O/S upgrade is required for security or compatibility or qualified admin staff move on will the system be upgraded. And then it will begin the same inevitable slide into entropy and uselessness.
Finding solutions that benefit the customer and reduce operating expenses are career suicide for outsourcing staff in a culture focussed on increasing billables.
For example: a major Australian bank replaced all the local file servers with small Network Appliance NAS's. These are the most expensive product per Gb available. The outsourcer had charged ~$2,500/month to 'administer' these systems. The bank paid for the change in under a year, increased availability and performance and solving many other issues to boot.
If the client gives all its IT staff to the outsourcer, who is going to seek out, design and implement new cost saving technology/systems?
Not the outsourcer - it's not in the contract and not in its (short term) interests.
The client has no IT staff - so it cannot and doesn't happen.
Audits and an Impartial Umpire
Who reports to the Client on the performance of their systems?
Who has the training/qualifications to check and asses the metrics and reports?
Who maintains & audits the basis of payments - the asset register?
Only the Outsourcer.
What are the downsides to the Outsourcer of a major failure in Prime Time?
A small number of 'service credits'.
Meanwhile, the Client suffers real costs and potentially large losses.
The Client wears all the business and financial risk with only minor penalties to the Outsourcer.
We are yet to see a corporate collapse due to an outsourcers IT failures - but it is only a matter of time.
There is a clear conflict of interest, or an real Agency Theory problem.
The outsourcer is Judge, Jury and Executioner...
There is no way to hold them to account or dispute their figures.
The Profession of IT
Contributors Michael Ellis, BJ, Kevin James, Richard Steven Hack,... started a thread about the 'value'/competency of individual IT practitioners.The huge (100+:1) variability in individual competence and the inability to measure it is one of the worst problems in our industry.
IT is not a 'Profession'. It, like 'Management', fail a very simple test:
What are the personal and organisational consequences of repeating, or allowing to be repeated, a known error, fault or failure??
[Do your mistakes have clear 'consequences' professionally?']
[Do your mistakes have clear 'consequences' professionally?']
Mostly it is "fire/blame the innocent, promote the guilty". The exact inverse of what you'd want.
People may trump technology and process, but Politics trumps everything...
And our Professional Bodies don't help.
The only real research into the causes of Project Failure are by consultancies - who are driven by the ability to sell their products, not what will benefit the Profession.
The ACM, IEEE, IFIP and friends have abrogated their responsibilities. We on the firing line, get to suffer their inaction.
Managers have to go with what they can quantify and inspect. Good managers will see through the B/S - but mostly too little, too late. Mostly, office politics, influence and self-promotion rule.
The adversarial nature of Outsourcing and the seemingly universal decline in code and service Quality stems from this failure of IT as a Profession.
Steve Jenkin 29-December-2007
2007/05/02
Defining I.T. Service Management
Objectives (The What)
Having begun around 1950, the world of Commercial I.T. is now mature in many ways. "Fields of Work" and professional taxonomies are starting to become standardised. Professional "Best Practices" are being documented and international standards agreed in some areas.
For the first time, audits of one of the most pragmatic I.T. disciplines, "Service Management", are possible with ISO 20,000. Business managers can now get an independent , objective opinion on the state of their I.T. operations - or of their outsourcers.
Being "documented common sense", ITIL and the related ISO 20,000 are good professional guides, but not underpinned by theory. Are there any gaps in the standard? How does Service Management interface with other IT Fields of Work? and What changes in those other disciplines are necessary to support the new audited practice?
Analysis of the full impact of I.T. Service Management, creation of a full taxonomy and definitions of "I.T. Maturity" are beyond the scope of a small "single researcher" project.
Approach (The How)
ITIL Version 2 and 3 and ISO 20,000, as published documents, form the basis of the project.Prior work in the field has yet to be identified. Secondary research will be the first step.
Each of the models will be codified and uniformly described, then a 3-way comparison performed. A Gap Analysis done of the 3 models, and a formal model built describing "I.T. Service Management" and its interfaces built and each of the existing approaches mapped to it.
Importance/Value (The Why)
The global economy, especially businesses in the "Western Industrialised World" are increasingly dependent on I.S./I.T. and their continued efficient operation. Corporate failures partially due to I.S./I.T. failure have occurred. Improving delivery of I.T. Services and the business management and use of them is important to reduce those failures in the future.The advent of ubiquitous and universal computing requires concomitant development of business management.
There assertions are considered axioms in this context:
- Organisations these days are dependent on their I.T. Operations.
- I.T. cuts across all segments of current organisations.
- I.T. defines the business processes and hence productivity of the whole organisation.
- What you don't measure you can't manage and improve.
- Improving the effectiveness of I.T. Operations requires auditable processes.
- Common I.T. Audit and Reporting Standards, like the Accounting Standards, are necessary to contrast and compare the efficiency and effectiveness of I.T. Operations across different organisations or different units within a single organisation.
For simple, repetitive cognitive tasks, computers are 1-5,000 times cheaper than people in western countries.
From this amplification effect, computers still provide the greatest single point of leverage for organisations. The underpin the requirement to "do more with the same", improving productivity and increasing profitability.
The few studies of "IT Efficiency" that are available show that IT effectiveness is highly variable and unrelated to expenditure.
The value-add to business of a complete I.T. Service Management model is two-fold:
- manage down the input costs of the I.T. infrastructure and Operations and,
- audit assurance for the board and management of the continued good performance of I.T. Operations.
[A 1990 HBS or MIT study into "White Collar Productivity" - reported a decrease in the first decade of PC's]
Previous Work (What else)
There is much opinion in the area, without substantive evidence: e.g. Nick Carr and "Does IT Matter?" The McKinsey report/book on European Manufacturers and their I.T. expenditure versus financial performance shows there is no co-relation between effort (expenditure) and effect (financial performance)."Commonsense" IT Practitioner approaches, SOX, ITIL and COBIT and others, do not address the measuring and managing of I.T. outputs and interfaces and their business effects, utiliation and effectiveness.
Jerrry Landsbaum's 1992 work included examples of their regular business reports - quantifiable and repeatable metrics of I.T. Operations phrased in business terms.
Hope to find (The Wherefore)
- Create a formal model for I.T. Operations and its performance within and across similar organisations.
- From the model, generate a standard set of I.T. performance metrics.
- Generate a set of useful I.T. Operations Business Impact metrics.
Report Outline
- Coded process models of ITIL version 2, 3 and ISO 20,000.
- 3-way comparison of ITIL version 2, 3 and ISO 20,000.
- Gap Analysis of ITIL version 2, 3 and ISO 20,000 models.
- Formal I.T. Service Management model.
- Common I.T. Service Management internal metrics and Business Impact
metrics flowing from the model. - Interfaces to other I.T. and business areas and changes necessary to support audits of I.T. Service Management.
- Further Work and Research Questions
Execution Phases
- Learn ITIL Version 2 - Service Managers Certificate course [complete]
- Learn ISO 20,000 - IT Consultants training [in process]
- Acquire and learn ITIL Version 3 [depends on OGC availability. mid/late 2007]
- Create/identify process codification.
- Codify ITIL version 2, 3 and ISO 20,000
- Compare and contrast coded descriptions. Report.
- Create/adapt process description calculus for formal model.
- Create formal I.T. Service Management model.
- Derive interfaces to business and other I.T. processes
- Derive internal metrics, role KPI's and business impact metrics
- Finalise report.
Subscribe to:
Posts (Atom)