Category Archives: Internet
Internet Access Data Shaping
Internet access data shaping, also known as “packet shaping” and “traffic shaping” is the method used by some Internet Service Providers providers to limit certain types of internet data traffic. The internet provider may impose data shaping for several reasons, the most common being insufficient network capacity; nearly always a result of the “profit motive” – ISPs oversell capacity and try to load as many paying customers on to a network as possible, to grab extra profit from the limited resources.
Internet service providers will seldom admit their sole reason for data shaping is profit, they will usually tell clients they regulate network data transfer rates to assure a certain level of performance or quality of service (QoS) for all clients. Whether or not their data shaping adversely affects these same clients is of little consequence except in a highly competitive market.
Msnbot – Ignorant spider or Deliberate Rule Breaker
Msnbot – (Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) – from search.msn.com has become a pest, not only for my websites, but for many others as well.
First of all, Microsoft sees fit to send multiple bots at the same time; as many as 17 have been reported crawling at once. I regularly have up to 12 on-site simultaneously. This number of bots crawling at the same time is effectively a dDOS (Distributed Denial of Service) attack, possibly locking the server up and preventing normal traffic.
Secondly, the Msnbot often ignores robots.txt disallow rules and crawls prohibited folders and paths, e.g. /js/ folders and Java files (.js). The numerous iterations of msnbot don’t seem to talk to each other – one bot will GET robots.txt, while the rest don’t bother. A good bot, e.g. Googlebot reads robots.txt regularly, and obeys the directives. Good bots also share that information!
Banning the Bad Hosts
I’m a no-compromise banner. It doesn’t take much effort to get an IP banned from my websites. A single bad event will normally be enough to block access to my sites from an IP address. Several attempts from a range of IP’s with a common service provider will get the entire IP range banned, the hostname or domain banned.
Currently there are about 700 entries in the banned list – representing millions of IPs, and the list gets longer daily. I cannot recall a day this year when at least one new bad IP was not added to the list.
Sharing the Bad IP Info
Mostly these IP’s were simply denied access, and no record was kept about the reason for the ban. At one time I started keeping a record, then lost interest and lacked time to continue. So I decided to start again, this time publishing the info where I can get to it, and other bloggers can also find the details. So now it’s published as a page on this blog…
The Challenges of Telecommuting Internationally
Modern information and communication technology introduced a new era in the way we work. It promised a new era where we could work from anywhere, no matter where in the world our office is located, or where in the world our customers came from. For some of us at least the this new era has arrived.
For others, the promise of telecommuting is only partly realised.
DoDaddy Server Network Crashed by DDos Attack
GoDaddy Hosting’s entire server network was taken down Monday Sept 10, 2012 by a Distributed Denial of Service (DDos) attack by a hacker. Most, if not all, websites hosted by GoDaddy were down for several hours. E-mail and telephone services were also affected
Exactly how many sites were affected. GoDaddy is one of the world’s largest hosting providers and domain registrar with more than 5 million websites affected by the outage, and peak downtime estimates suggest as many as 48 million websites were affected by the attack
Mail Online (dailymail.co.uk) reports:
- Web hosting giant hacked and all of the websites run through GoDaddy were shut down temporarily as a result of Monday’s attack
- Service was eventually restored for the bulk of customers by 5:43pm (GMT)
GoDaddy spokeswoman Elizabeth Driscoll said:
MWEB IPs used by Spammers and Hackers
Checking an IP record for 18.104.22.168 after noticing a minor offence this morning – the ubiquitous and quite stupid practice of adding “/undefined” to the end of actual URLs – brought up a list of IPs in the neighbourhood. All the IP’s included below belong to MWEB. (whois.domaintools.com IP lookup records.)
MWEB, a South African Internet Service Provider, has previously had IP’s under their control listed in several databases as a source of spam e-mails. According to Project Honeypot a range of IP’s managed by MWEB is (or was) used by Spammers and Dictionary Attackers.
False Valuation by Pansee.com
Ever had a site valuation by pansee.com – I got sent a mail informing me ‘someone’ had conducted a valuation of my website graphicline.co.za using pansee.com valuation tools, with a link to the valuation report. Interested to see what the report contained, I checked if Google had any information about malware on the site, then visited the page.
The valuation report had some interesting data. From the country where most of the website traffic is derived from, to number of daily visitors. And a claim to the value of advertising on the front page.
France is the Biggest Source of Traffic
This amused me… According to pansee.com, 12.2 percent of my traffic comes from France, while the USA only accounts for 8.1%
What is Markmonitor.com?
Markmonitor.com is a company providing brand protection to (mainly) global brands. Markmonitor monitors the Internet (supposedly) looking for brand-piracy, domain name hijacking and counterfeiting (of branded goods) among it’s range of client services. The company must use search spiders to trawl websites looking for this information.
They also have another side of business, as a domain registrar, and a number of large corporations includng Apple.com have their domains under their ambit.
The IP’s seen so far are part of Google’s network, specifically Google Plus1 shares IPs in the Range 174.125.
So far so good. Most of us will agree the (claimed) services offered by Markmonitor.com are desirable, at least by international corporations trying to protect their brands. So why have I banned their spiders and their website IP addresses?
Data Bundle Overload: End to Free Mobile Data
It looks like mobile data users at the Southern tip of Africa are not the only ones looking at paying high prices for data bundles (or buckets as they are known in some locales). The USA is joining the trend of charging for mobile data. Down South we have never known the privilege of free mobile data, having some of the highest data cost per GB in the world – at least when compared to first world civilisation.
The Data Crunch
The end of free mobile data is no real surprise. The RF frequencies used to carry this traffic are limited in capacity, new compression technology can only go so far to reduce the load on these channels. Urguably more frequencies could be made available, however these would have to be in higher frequency bands than are currently used. Increasing RF frequency brings it’s own problems – higher frequencies (into the mid and high Ghz region) suffer more loss from inclement weather signal absorption from structures, components cost more, and we have to consider the human health related dangers of microwave radiation.