In an incident yesterday in Reading, a downed router on BT’s network took out 9,000 homes. If you believe the press release.
If you actually check however, you’ll find that BT has once again lied. Plus.net’s user graph (remember that Plus.net is a BT subsiduary?) alone shows a drop of nearly 25,000 users.
How does a nationwide broadband and telephony provider in this day and age allowed to get away with such a crap non-resilient network design and still be handed the sole country-wide monopoly?
One of the telecoms engineers in my workplace spoke to a source inside BT and found out the true story behind the two outages. Last Friday was not a power cut. A card in a router locked up, and rather than replacing it BT took the decision to simply reboot it and hope it doesn’t happen again.
This goes against general practices in any large datacentre where you have a multitude of people relying on a single device to be working properly 100% of the time – when something that critical fails, you replace it immediately. It also means they really don’t have any redundancy as it appears there was no second card or router to take over when the first failed, and that means a 2-6 hour outage while engineers are gotten out of their beds, travel to site, diagnose and fix.
So it seems I was right, given that this morning once again the entire of Northern Ireland, Scotland and parts of England all lost their broadband connections again. Except this time BT is blaming an unnamed hardware vendor for the issue, again at the Edinburgh datacentre.
Imagine the broadband access for the entire of California, of even just a large city like New York going out for hours on end. There would be a senate hearing on the cause, loss of manhours for businesses and public outrage. In the UK… our government just gives the incumbent telco some more money and a tells them to try and not do it again.
There was an overnight outage where BT’s Edinburgh datacentre lost power. Now bearing in mind that the BRAS’s in this datacentre service pretty much the entire of Northern Ireland and Scotland, and parts of the North of England, why would they claim that only 20,000 homes were affected?
My workplace specializes in providing DSL to businesses in Northern Ireland, and I know for a fact that every single one of them was down aside from a small of handful of lines that we have in England. These are lines spread across the entire province, from Belfast to Derry. None of my neighbors had working broadband either, except for the two that were using Sky LLU. My Dad didn’t and he’s on Plusnet, many miles from where I live.
20,000 homes is about an 8th of the numbers of homes in Belfast alone, never mind across Northern Ireland, Scotland, or the North of England. Why won’t BT admit to the real number of homes affected? Because then people might realize how utterly incompetent they are to design such a widely used and relied upon system to be able to be take down nearly half the country because of a supposed power outage in a single location (which in a properly built datacentre, should be nearly impossible).
BT – Liars and idiots. Can someone remind me again why we gave these people the telecoms monopoly?
There’s this great debate in the IPv6 world about how to chop up your allocation into assignments for your customers. Typically, most ISPs are being handed a /32, and general guidelines say to allow for a /48 per DSL/leased line/cable customer.
However a lot of people are asking, why not a /64? Quoted below is the sort of answer you’re likely to receive on NANOG, by one Mark Smith:
There are a variety of scenarios where customers, including residential, will benefit from having multiple subnets. They may wish to separate the wired and wireless segments, to prevent multicast IPTV from degrading wireless performance. They may wish to segregate the children/family PC from the adult PC network or SOHO network, allowing the subnet boundary to be an additional Internet access policy enforcement point. They’ll need separate subnets if they wish to use a different link layer technology, such as LoWPAN. They may wish to setup a separate subnet to act as a DMZ for Internet facing devices, such as a local web server for sharing photos with relatives. Game consoles may be put in a separate subnet to ensure file transfers don’t interfere with game traffic latency, using the subnet ID as a QoS classifier.
This answer is quite simply unrealistic. It’s the answer of a typical geek with no sense of perspective as to what the average consumer wants. It’s the opinion of what Mark Smith the network engineer and geek would want.
In the real world, most consumers of domestic internet services have absolutely no concept of IP addresses let alone subnetting, VLANs, segregation or quality of service. Most domestic networks are a single flat subnet with NAT to a single IP address and no servers that would require port forwarding, and rarely an IPTV system, but those are usually setup to use special triple-play routers configured by the ISP. Most domestic users just want to be able to plug stuff in and have it work.
Now, people will argue that there are more IPv6 addresses than there are atoms in the world. However that argument isn’t as good when you are assigning 1,208,925,819,614,629,500,000,000 IP addresses for just 2 or 3 devices. It’s a grossly inefficient waste no matter what you say. Not to mention that if you’re one of the big cable or DSL providers with millions of customers, it makes much more sense. Each barely used /48 that you throw out contains 256 /64’s.
As such, I personally am inclined to go for the default of a /64 per customer, but allow for a /48 should they need it. There is absolutely no point in issuing a /48 subnet to someone who is never ever going to use it… it’s just laziness, which is what got us into the current situation with IPv4 in the first place.
They tried to requote us for their 21CN broadband platform, assuming we take it in London as they wanted to charge us 50p per meter all the way from Manchester to Belfast (totaling some £250k). After carefully examining our current installation they decided that we should replace our pair of 34Mb pipes with a single pipe containing:
When I entered the correct figures into their shitty little price sheet, added in all the things they forgot, it came to a whopping £26 per user before any profit margin is added.
By comparison, Be/Fluidata is charging a non-recurring £3k to setup a simple crossconnect in any London Telehouse, and then all we pay are simple line charges depending on the product used, the average one of which is £16 per month.
It’s quite clear that BT Wholesale is not interested in providing any sort of service to other service providers. The ridiculous ordering/faults system, the outright denial of clear area-wide faults and now these ridiculous and quite arbitrary charges for access to their so-called 21st Century Network that still doesn’t properly support IPv6 are all very telling.
A new level of fail from our friends at BT Wholesale. They have actually willingly provided proof that they do not read fault reports the first time around:
Yes.. that is a grand total of 43 seconds from reporting the fault to BT Wholesale rejecting it. This was even reported via KBD, which lets you confirm that the user has already attempted to replace his router, cables, filters and even tried from the test socket. 43 seconds is not enough time for most people to type that long-winded reply about SFI appointments, let alone for BT to run the necessary diagnostics to determine if there is a fault or not.
At my place of work we have suspected that BT was doing this for a long time as all too often, and 9 times out of 10 blatently obvious faults are rejected with the message “not due to a network fault”. Now I have a handful of faults, some where it was customer some, but some where there was genuine faults such as the DSLAM being faulty where BT has denied anything being wrong and cleared the fault in less than a minute.
Customer complains of a fault. I diagnose and determine that the customer is doing nothing wrong, and it’s probably a line fault. BT’s response is near instant, lambasting me for reporting what is clearly not a BTW (BT Wholesale) fault. An entire day later, they update it to say that there’s a fault with the DSLAM. This is of course long after I’ve told the customer there must be something wrong on their end as BT says there’s no fault.
Is it any wonder that people think their ISPs are incompetent and BT can do no wrong? It’s because when BT has a fault they have the ability to actually go and look at the DSLAM and see for themselves. When anyone else has a fault, they have to rely on the BT Wholesale broadband diagnostics systems which are quite frankly, half assed and go down more often than a thai hooker.
This is what happens when someone has a monopoly but are mandated to supply wholesaler services to others to make it look like they don’t have a monopoly. Their own service is perfect and can do no wrong, while all those wholesalers can scream and shout and cry about the godawful service they’re being given.
I hate these articles. They’re always written from the perspective of a consumer who knows some of the buzzwords but doesn’t actually bother to take the time to understand any of it or the technology.
Bandwidth Throttling
One oft-protested behavior of various ISP’s is the throttling – that is, limiting – of bandwith at certain times or for certain uses.
Yes, some ISPs shape the bandwidth supplied to the customer. There’s good reason for this however. Residential broadband connections are contended services. That means that you share that bandwidth along with a number of other people. In the UK, the standard contention ratio for residential users is 50:1. If bandwidth is not shaped, then just one customer abusing Bittorrent to download pirate movies or games or music can use up all the bandwidth of 50, leaving those other customers shit out of luck. You’ll also note that the only people who ever complain about shaping are those pirating content.
Imagine that the bandwidth is a 3 lane motorway. If everyone behaves, we can all drive down the motorway at reasonable speeds, occasionally going faster where possible. However, if one driver in an articulated lorry starts swerving all over the road, everyone else has to slow down and be late. Bandwidth shaping is done to preserve the use of the road for everyone.
Deceptive Speed Claims
Examine the fine print on most ISP commercials, and you will likely find that the promised Internet speed (say, 10MBPS) has the words “up to” in front of it. As it turns out, this is often a clever means of dodging the truth about the actual speeds you are likely to receive.
Again, the service is contended. If you have an 8Mb 50:1 service, you are sharing that 8Mb with potentially up to 50 other people. There’s also the technology involved; ADSL is distance limited so the further away from the telephone exchange you are the lower it will connect at. “Upto 8Mb” covers all of this in two words. You get what you pay for and at £20-25 a month, no ISP could guarantee what this writer is demanding. Let me break it down:
Never mind the infrastructure required to deliver such a service, servers for providing DNS and email, datacentre space, cooling and electricity and so on – but to give you your 8Mb ADSL without it being shared with anyone would cost somewhere in the region of £6,500 per month to have their own dedicated ISP not shared with anyone else. NOT £25. It’s only by sharing infrastrucure and bandwidth with other customers that cheap broadband actually becomes economical. If you want a dedicated guaranteed 8Mb circuit, feel free to go talk to your ISP – they’ll quote you a figure probably somewhere between £10,000 to £12,000 a year, plus probably a £15,000 install.
Targeted Advertising
Increasingly, some of the most passionate complaints against ISPs have involved privacy concerns. A case in point is Charter’s decision in 2008 to begin tracking its users’ search behavior and using them to insert ads into their results.
Never mind that the example screenshot given is from Google, highlighting Google’s own advertising which has absoloutely nothing to do with whatever ISP you are using to connect with; the article in question successfully gives a single example of a single large ISP abusing the Phorm advertising system. As far as I am aware the only other ISP to consider using this system is BT in the UK, and they were smacked down for being in breach of privacy laws.
This is tarring all ISPs with the same brush for the sake of a one or two bad apples.
ISP Wiretapping
2007’s Communications Assistance for Law Enforcement Act mandated that all ISPs enable the feds to “wiretap” Internet transmissions in much the same way they do phone calls.
Note the word “mandated”. That’s not your ISP screwing you over – that’s your government. Most ISPs have this capability anyway in order to comply with court orders or police investigations. Do you really want to be responsible for some 3 year old getting raped by a pedophile because it was made illegal for ISPs to help catch sick fucks just so you could download your pirate movies without having to worry that someone might be watching you because the police told them to?
Ad-Filled “Website Not Found” Pages
Always on the lookout for new sources of revenue (however small), some ISPs have taken to displaying ads in their error pages.
Some ISPs do this, however the good ones will give you a source of opting out, and as this is usually DNS based if you don’t like it you can always either setup your own DNS server or use OpenDNS.
Deep Packet Inspection
Another serious gripe privacy advocates have with ISPs is what is known as “deep packet inspection.”
That would because privacy advocates don’t actually understand DPI, which looks for patterns in order to recognize traffic types. The actual content cannot usually be observed, but it can also log when someone is for example, using Bittorrent. These boxes are usually used to packet shape your traffic (see “bandwidth throttling” above), but are also extraordinarily expensive – usually only affordable by quite large ISPs. But wait! There’s more…
However, it is also been used by ISPs to police copyright infringement by detecting when someone is or may be downloading songs or movies – and some ISPs go a step further by turning this information over to inquiring record labels.
And so we get to the real crux of the issue that the author has with DPI – pirating movies and music! Of course, if you’re not doing anything illegal then DPI really isn’t something you have to worry about. Damn you pesky ISPs! Conforming with the letter of the law and trying to prevent yourselves from being used to commit illegal activties! Grr and much fist-shaking and so on.
Packet Spoofing/Forgery
Comcast engaged in what is known as “packet spoofing” (or packet forgery) by interrupting file transfers with bogus packets that killed any P2P downloads a user happened to be engaging in.
One ISP does something that contravenes the way that TCP/IP is supposed to work and we all get tarred with the same brush again. I can assure you that few if any ISPs that have any sort of technical savvy would even consider doing this. Comcast are the only ISP known to have deployed this system as it far too heavy handed – affecting both legitimate and illegitimate traffic. It also opens the ISP up to a certain amount of liability for having demonstrated that they can block certain traffic types, for then not blocking other traffic types such as viruses or spam, which happily leads me to the next point in this ignorant article.
Inadequate Virus/Spyware Protection
ISP’s have also come under fire for charging high subscriber fees without adequately protecting consumers from spyware, viruses and other forms of online fraud.
As I think I’ve already demonstrated, most ISPs are not charging “high subscriber fees”, and in one sentence the author of this article has demonstrated his complete ignorance of any of the previously mentioned technologies. If you want your ISP to prevent you from idiotically downloading a virus and running it, they have to install what is known as an IDP, or Intrusion-Detection-Prevention device. What is an IDP? Basically, it’s a Deep-Packet Inspection device configured to look for viruses, trojans, spyware and known hacks. It would then have to use Packet Spoofing to block your attempt to download that virus. So the author wants us to protect him from viruses without actually using any of the known technologies to do so. Does he want us to send someone around to his house to operate his computer for him or what?
Generally speaking, service agreements between you and your ISP indemnify them from responsibility for any damage or losses caused by spyware or viruses you get infected with on their network.
…because we all know that it’s the pesky ISP forcing viruses and spyware onto your computer. Out in reality-world (as opposed to this crack-smoking monkey of an author’s fantasy world), 9 times out of 10 virus and spyware infections are because the user actively downloaded that cute new screensaver of the puppies doing barrel rolls and installed it, and that screensaver was actually a shell for a massive spyware infection. Or the user received an email from King Mambatu who wanted his help to move $9,843,699 dollars out of the Bank of Nambia and needed him to open this harmless attachment to get the process started. Yup, all the ISPs fault that is. This couldn’t possibly be the reason why ISPs have had to indemnify themselves against protecting you from viruses and spyware because a certain section of society wouldn’t sue them into oblivion with frivolous lawsuits over their own stupidity or that when they do offer antivirus or antispam services on email, that the technology cannot guarantee that it will catch everything.
Sneaky Fees
MSNBC reports on a telling example back in 2006, when a a $2-$3 per month federal tax on DSL users was taken off the books. But rather than lowering its subscriber fees by $2-$3, Verizon thought better of it and kept fees the same by adding a “supplier surcharge” fee.
It’s hardly a sneaky fee if it’s listed on the bill. It might be sneaky for the muppets who don’t bother to read their bills, which is rather disturbing seeing as this is written by someone calling himself “Bill Shrink Guy”.
I won’t deny that doing such a thing isn’t disreputable, but the real issue here is to make sure you read all your bills and ensure you understand what you’re paying for. If your gas supplier suddenly adds a “Boiler Maintenance” surcharge to your bill and you pay it without question, you’re an idiot.
Bandwidth Throttling
