2009/08/25

When 'ping' fails

In Networking, the 'performance objects' are links, usually end-to-end, consisting of many elements - like ethernet segments, switches, routers/firewalls, long-distance circuits and security scanner devices, laid on top of Telco/backbone services that provide dynamic and asymmetrical routes.

The most frequently used measure is 'ping' time - ICMP “echo request” packets, often 64 bytes long.
Firewalls, routers and underlying networks filter/block classes of traffic, implement traffic & flow rules and attempt "Quality of Service". There are many common rulesets in use:
  • blocking 'ping' outright for 'security reasons'. Stops trivial network scanning.
  • QoS settings for different traffic classes. VoIP/SIP gets high priority, FTP traffic is low, your guess on HTTP, DNS, time (NTP) and command line access - SSH, Telnet, ...
  • traffic profiling on IP type (ICMP, UDP, TCP) and packet size (vs link MTU).
  • traffic prioritisation based on source/destination.
Real-time voice needs many small packets, would like low latency/delay, and no jitter, and can stand some packet loss or data errors. FTP relies on TCP to detect packet loss & retransmit them. It likes big packets and attempts to increase bandwidth consumed through TCP 'fast-start' etc.

The only time 'ping' is accurate is within a simple ethernet segment - no rules, no firewalls, no 'traffic engineering', no link losses, no collisions, ...
Otherwise, it's a dangerous and potentially very misleading measure.

'Time of Flight' for UDP & ICMP packets is only measurable when you control both ends of the link and can match headers. Not something most people can or want to do.

TCP packets can be accurately timed - sessions are well identified, packets can be uniquely identified and they individually acknowledged. It is possible to accurately and definitively monitor & measure round-trip times and effective throughput (raw & corrected) of links and connections at any point TCP packets are inspected - host, router, firewall, MPLS end-point, ...
I'm not aware of this being widely used in practice, but that's a lack of knowledge on my part.

This is not a tutorial in Networking or TCP/IP.
Neither am I a Networking expert. I'm demonstrating that even with my knowledge, "tools ain't tools" (meaning not all tools and methods are equal),
and that using just 2 metrics, 'bandwidth' & 'latency' to characterise links is simplistic and fraught. As professionals, we have to be able to "dig under the covers" to diagnose & fix subtle faults.

Consider the case of TCP/IP over a noisy satellite phone link, the type you buy from Intelsat for ships or remote areas. The service notionally delivers a data channel of 64kbps, but is optimised for a digital voice circuit, not IP packets. The end-end link has per-bit low latency on/off the link, long transmission delays, nil jitter and limited error-correction (forward-error-correction (FEC), no retransmit) - nice for telephony. These links also have buckets of errors - which voice, especially simple PCM, easily tolerates & can even be smoothed or interpolated out with simple equipment - which will be there to handle echo cancellation.

Say you're on a ship at the end of one of these links.
People are complaining that email 'takes forever' and they can't download web pages.
You run a ping test out to various points - and everything is Just Fine.

What next?

The most usual response is 'Blame the Victim' and declare the link to be fine and 'the problem' to be too many people sending large messages, 'clogging up the link', and too much web surfing. You might set quotas and delete large emails. That might work, or at least improve things marginally.

Radio links, especially back-to-back ones, each crossing 36,000km to geostationary satellites, are noisy.
If the BER is 1:100000 (1:10^5, under the hoped for 1:10^6) and you're using the default ethernet MTU of 1500 bytes, you'll get an error every 1-2 seconds. No worries, eh?
1500 bytes = 12,000 bits = 5.3 packets/second. Or 1 in 10 packets. Hardly noticeable.
TCP/IP has a minimum overhead of 40bytes/packet (less with Van Jacobson compression).
The data payload per packet is 1460 bytes for the ethernet default MTU.

Sending a 1.25Mb file (10Mbit), that's ~856 raw sent packets and 103 errors, or ~12% retransmissions. Of those 103 resends, 12 get errors as well and are resent, and 1.5 errors of those go onto a 3d round...
Or ~115 errored packets, or 14% errors on raw packets. Just a minor problem.
There's a probability of >1 error per packet, but I don't have the maths to solve for that.

The effective bandwidth (throughput) of the link, using 970 * 1500-by packets to move 1Mb in 182 seconds, is 55kpbs. Quite acceptable.

What if there's a corroded connector or tired tracking gimbal and you get a 3db change in SNR and the BER doubles? (That's a guess, not science.)
856 raw packets and 250 1st round errors, 62 2nd, 15 3d, 4 4-th, 1 5th = 333 resends. An almost 50% increase in total number of packets needed to send the file. 223 seconds and 45kbps.
Doubling the BER again (4:10^5 or 0.004%) increases the 1st round error rate to 400 packets, or 47% - 750 retransmits in 10 rounds. 301 seconds and 33.1kpbs. Half-speed.

Back to the original BER, if you were running 'jumbo frames' (9000 by) locally & these went down the link as is, you get 0.8 packets/sec and have a 72% chance of an error in a packet. One in four of the packets would get through unscathed. 140 'jumbo frames' are sent raw, 350 packets are needed with 16 rounds of retransmission.
The file takes 400 seconds at 25kbps - a hefty penalty for forgetting to configure a switch.

The problem is that packet size amplifies error rates.
A change in BER of 0.001% to 0.004%, undetectable by the human ear, halves the throughput fo TCP/IP.
Using an MTU size of 168 (128 data + 40 TCP/IP overhead) gives good performance at a BER of 1:10^4, trading 25% protocol overhead for link robustness.

'ping', using default 512 bit packets, won't detect the error.
But who'd think the MTU was a problem when standard diagnostics were reporting 'all clear'?

Summary: In difficult conditions, the BER doesn't have to drop much for link throughput to significantly degrade.

This example is about simple-minded tools and drawing incorrect conclusions.
The Latency of the link was constant, but the Effective Bandwidth, throughput, changed because of noise or link errors.

Surely that proves the Myth: Latency & Bandwidth are unrelated.

Nope, it proves that link speed and throughput bear a complex relationship to each other.

If you had been measuring TCP/IP statistics (throughput & round-trip-time) at the outbound router, or using 'ping' with MTU of 1500, you'd have seen the average latency rising as throughput dropped. All those link-layer errors & subsequent retransmits were causing packets to take longer.

But a simple low-bandwidth radio link isn't the whole story.
It's a "thin, long pipe" in some parlances.
What was special about that link was:
  • no link contention, data rate was guaranteed transfer rate.
  • synchronous data

2009/08/07

Rupert Murdoch - Fool or Genius?

Does Rupert Murdoch know something the rest of us don't?
The recent news is that News Ltd would start charging for on-line access to its newspapers.

Not a good idea.
Experienced Journalist & commentator, Alan Kohler also thinks so...



First, Rupert is not in the business of selling 'news', quality journalism or not.
He sells Advertising.
Just like Google and friends. But apparently not nearly as well as they do on-line.

There are people who sell 'news', and they are going strong.
Organisations like Reuters, Associated Press, Bloomberg, AAP, ... The wire-services.
The same ones that sell to Google, businesses, TV and Mr Murdoch's newspapers.

News Ltd doesn't sell journalistic content (news): like every major newspaper, it has always given away its content. Exactly the same as Free-to-Air radio and TV.

The "value proposition" to most newspaper customers, News & Stories, is a Free Good.
Major papers actually cost their publishers to sell. Newsagents typically keep the full "cover price" of the local major papers. Perhaps this is why Fairfax Ltd lists "Newsprint and Ink" as its single biggest expense.

Publishers make their money from the advertising they sell (Classified and 'Display' or general).
They set their advertising rates on the estimated number of readers - not copies sold/distributed. (There's a whole industry 'auditing' circulation & readership).

Small advertisers will always be 'price-takers', while the large regular advertisers can negotiate.

On-line breaks many/all the Newspaper assumptions:
  • no intermediaries with good 'passing trade' to find customers
  • exact counts, not estimates, or readership
  • exact counts of advertiser hit-rates (count links followed)
  • targeted/niche audiences, not "broad spectrum" mass market

Browsing the 1200+ entries for 'newspapers' in the Australian Yellow Pages, these groupings seem apparent:
Business, Trade & Industry, Lifestyle, Sports, Political, Special Interest, Community/Local, Regional & Rural, Ethnic, Language and Religious,
and versions of the "Trading Post".
A newspaper without content, pure advertising, the ideal for the business side of newspaper.



Second, a long time ago newspapers were the source for capital-N News - timely, important, factual.
They broke stories, 'scooped' one another, had many editions during the day and dealt in "the facts m'aam, just the facts" as Joe Friday might say.
The sort of thing shown in 1930's Black and White movies.

By the 1970's, newspapers had comprehensively lost the race as the first news source.
"Watergate" showed they could still 'scoop' other media with investigate journalism, but the Vietnam War played out on the nightly TV news.

When Ted Turner started CNN, the game changed - Free-to-Air was usurped.
The 1991 Gulf War had CNN "reporting live from Baghdad" and assumed the mantle of "first news source".

These days it is a tussle between Cable News and on-line services to be "first".
And that race has always led to problems with accuracy and false/fabricated stories.

An editor who is under pressure "to be first" can be manipulated into publishing without good fact checking. When there was considerable effort & expense in rolling the presses, the downside ensured more caution. In the on-line world, nearly all barriers to production are eliminated alongside "instant" publication. An editorial mistake is much more likely and potentially much more damaging to a large publisher, the Drudge Report non-withstanding.

Newspapers have been providing Opinion & Analysis for a couple of decades.
Any pretence they are cutting edge or breaking stories in real-time is a "fools paradise" and delusional.

News Ltd has great content produced by many great people and serves a faithful cohort of consumers. It just isn't 'news' they are selling.





Third, there are many good free alternatives for news, on-line and not, to newspapers.

Google pays for wire-services and gives away the content.
Publicly funded media - radio, TV and on-line - have a mandate to provide services with public monies. The BBC and Australian ABC have large news rooms and international reporters.
The ABC alone has 700 people in its News Division providing current content for all its outlets.

How does a newspaper, with at best 300 journalists, compete with a better resourced competitor who's content is free?

Not on news - only with other types of content and other incentives - like DVD's and special offers...




Fourth, there are just 3 workable Revenue Models.
Revenue options are:
subscription/donations and cover-price + advertising.
(pre-paid vs 2-part charging)

There are only 4 Revenue Models possible in this scheme.
Fully free can't self-support itself, so there are only 3 workable models.

Murdoch is complaining the Revenue Model that has worked well in the physical world for approaching a century doesn't work on the Internet. Who'd have thought?!

The wire-services thrive and on-line advertising is booming.
The only people out of step are the firms running newspapers.

They could have acted in 1995 to move their advertising on-line, but didn't.
Were they blinkered or lacked 'vision'?
Was it a sound business decision based in part on not canabalising their main cash flows?

Things are how they are...
There seems little to be gained from now analysing the reasons for non-action.



There are other issues that have to be resolved when moving to on-line services.
  • Paper is simple and always "Just Works', modulo getting wet.
    Attempts to stream printed news electronically have been widely successful outside of offices. Radio serves the travelling public well. Printed media is cheap, available and can be forgotten without dire consequences. Some section of the population may read the news on their Kindle or iPhone on their morning commute, but it won't be a large audience.
    Neither will there be much call for $10 newspapers...

  • Serving "The Diaspora": An important function of newspapers is allowing non-resident locals to "keep in touch with home". Australia shows that people may permanently emmigrate and never return home, but still identify strongly with their country of origin. This fuels our strong ethnic newspapers. For people who've only moved towns, a daily or weekly "fix" of their hometown newspapers fills a strong need. They even pay a premium.

  • Niche buyers. Most buyers throw most of a newspaper away. They are very specific & selective in their needs and uses of the massive content provided. There are better ways to serve many of those niches on-line. Like classified advertising is better served by e-bay and 'trading post'. It's fast, current and cheap - plus very efficient for the reader. The service does the searching and the reader can be contacting a seller within minutes of loading the site.

  • Network effects and the tipping point. When a product has reached around 40% market penetration, it 'suddenly' becomes popular and quickly saturates the market. This happened in 1984 with Group-3 fax and then around 1996 with The Internet/World Wide Web. Newspapers need to be keenly aware of their competitors - when the end comes, it may be frightenly fast.

  • Copyright and Libel Laws. The journalists union has spent a very long time negotiating what rights the publisher & content-creator have. This all has to be done again in an on-line world. The other side of the coin is commercial protection of journalists against Libel or defamation actions. The publisher wears the risk once the editor decides to print. Those named know that a newspaper can afford to and will defend itself. If journalists are personally exposed to litigation, justified or not, they will sensibly withhold contensious pieces. Why wreck your life for a decade or more, as happened to Chris Masters over the "Moonlight State" and other pieces? For many, the price is too high.



Lastly, What would work?

This is an argument in three parts: as a society we need 'quality journalism', news rooms aren't cheap, and are there models we could follow?

The media as "The Fifth Estate" is an important and necessary part of any Democratic government. A Free Press is a necessary part of Open and Transparent government.

But whither Investigative Journalism. There is a lot of TV reportage of politicians doing 'door stops' or in stage-managed events. And a lot of 'tabloid journalism' on TV.

Would Woodward and Bernstein now be funded for their lengthy Watergate investigations?
Would any editor allow it to be published these days?
I think Watergate is less likely to be reported these days for many reasons and the Drudge Report and other gossip sources do not fill the gap.

"Quality Journalism" has to be nutured & supported for us to have stable, prosperous societies.

The ABC states it has 700 people in its News Division.
On-line sources suggest major newspapers have ~300 journalists in their newsrooms. [This information isn't in the Annual Reports I scanned.]

What would it cost to run such a news room?
$30M a year in wages, $10M in wire-services, $10-15M for bluidings and systems.
Marketing & Sales probably $30M. Accounting and collecting subscriptions: $5-10M.
Publishing on-line would add another $20-30M, with an overall 30% Gross Margin required to fund upgrades, depreciation and dividends.

Perhaps $150M/year in revenue, or $3M/week.
The Sydney Morning Herald has an audited circulation of 210-360,000 and readership from 850,000-1,100,000 [without SunHerald, $1.80 and 480,000/1.25M]
Previous comments: "Internet Changes Everything: Newspapers".

Even if you achieved 500,000 individual subscriptions, a weekly price of $5+, versus the $1.40/weekday and $2.40 Saturday for the paper version.
I believe that's far beyond the consumer 'price point' for a single publication.

So what models are out there that might work?
'Cable TV' provides content aggregation, common marketing services & subscription and billing.

It has severly impacted Free-to-Air TV over the last 4 decades for many reasons.
One of the big factors I believe is allowing content providers to focus on their strengths and the Cable Service Provider (Foxtel in Australia) to focus on the technical and retail/customer relations and support business.

The public are offered content aggregated into affordable and desirable 'packages'.
They can decide the utility to them of each package and compare the cost to other forms of entertainment. A$30-$50 seems to be the price point.

Cable TV for content providers removes barriers to entry and avoids competition between technical delivery methods. The customer wants the service and isn't interested in the technology per se. This model allows & promotes small, new entrants with serving specialist or highly targeted niches.

The revenue returned to content providers is unknown to me. Large studios are not 'price takers' and have significant negotiating power as 'headline products'.
Presumably small niche providers get a return based on consumer views.

This shared infrastructure and 'content packages' seems ideally suited to on-line delivery of paid content - which doesn't have to be limited to news or 'quality journalism', but certainly includes them. Plus we have the natural providers already operating with large, high-quality customer lists: Cable Service Providers.

The technical implementation for an "on-line Channel Service" is simple. Though the ABC iView experience suggests that collaborating with ISP's and allowing unmetered content is necessary. [Australian ISP's impose download quotas on broadband].

Customers already have some sort of PC (Windows, Mac, Linux, ...) and look after their own broadband connection.
A controlled, universal 'player' is required - happily companies like VMware already provide, free, a basic product that work across all platforms, the "VMware Player", which can run pre-built systems with embedded applications, "Virtual Appliances".

The only necessary work is tailoring a VPN or similar and distributing the required registration/connection keys. Foxtel already has the infrastructure in place to source, distrubute, service and support hardware & devices.

That's not a big leap...
And one where services can be packaged in a series of packages with many different price-points.
All those free Community papers, plus a Major Metro Papers: $5/month?
Add a speciality or trade paper, all the Major Metros, a financial services 'feed' and an alert service like 'Media Monitors' for a company of 75 people: $lots.

How does this proposal site with the Newspaper assumptions:
  • intermediaries with good 'passing trade'
    Exactly what the Cable TV companies do.

  • exact counts, not estimates, or readership
    Page counts and the precise subscriber unequivilacly identified & grouped.
    Near-perfect marketing information, and a perfect, undisputed source of revenue figures for 'page hits' revenue scheme.

  • exact counts of advertiser hit-rates (count links followed)
    You don't have to sell advertising, and many content providers would not,
    But if, like the Trading Post, you did... Trivial.

  • targeted/niche audiences, not "broad spectrum" mass market
    Tailored content per source, niche & specialist sources, remembered preferences and interests... Near perfect for subscribers and providers alike.

  • Normal print-media space restrictions are lifted: Content providers can provide additional "in-depth" material easily & cheaply.

  • Individual content providers can use the Channel Service to provide archive, search and print-on-demand services. Leaving each party doing what they do best.

  • 'Leakage' of content can be controlled with the Virtual Appliance.
    It may be configured to only allow 10 pages a day to be printed... With extras purchased.

  • Anyone interested in expensive periodicals, Academic Journals or hard-to-find books?
    With controlled access and clear charging regimes in place, there is no issue about denying or destroying copyright.
    In another day, this might have been called "Your Local Library".

On top of this, additional options allowing subscribers to pre-pay to view or print normally inaccessible content. The Channel Service Provider doesn't become a credit provider - in fact gains by holding the prepaid money - which it never need return and might even expire, like pre-paid mobiles.

Importantly, 'micro-payments' are avoided. They are very, very hard to get right and consequentially expensive. That's why we've never seen Visa and Mastercard move on this market.

But moving 1cent 'funny money' from your pre-paid balance to a vendor - very cheap.
It's the basis of prepaid mobiles.
With the business-friendly upside of all the unused payments that expire - a tidy 5-10% profit.



In summary: Do I think Murdoch is wrong-headed in charging for access to his newspapers?
Absolutely.

Do I think an on-line service offering this facility effectively, efficiently and profitably can be constructed?
Absolutely.

Will anyone read & respond to this piece and the proposal?
Who Knows :-)