Bing and MSN Bots Are Banned
I have banned Bing, Yahoo and MSN search engine spiders from my sites! I’m tired of the constant rule breaking and over-crawling by Bing and MSN search bots.
Bing is a Rule Breaker
Microsoft claims Bing honours robots.txt rules. In my experience that is a blatant lie. Bingbot / msnbot simply ignore robots.txt rules and crawl whatever they want. Some of the specific rules broken include;
- crawling system folders
- crawling image folders (msn-media bot). Image folders and extensions jpg, png,gif, bmp are disallowed
- crawling RSS feeds. All RSS feeds are disallowed; rss.xml, /feed/, etc
- crawling comment forms; DOMAIN/comment/184 – the path /comment/ is disallowed in robots.txt
The last straw was today. 2 days ago I added Bing and MSN user agent strings to disallowed bots in robots.txt across all my sites; this morning I see these bots read robot.txt then ignored it totally, and crawled the sites anyway.
Bingbot and MSN bots are Rulebreakers
I’m tired of watching Bingbot and msnbot breaking rules and crawling disallowed files, folders and paths. Microsoft say their bots obey robots.txt rules – They don’t. Bingbot/msnbot occasionaly read the robots.txt file, then immediately afterwards continue on to crawl items specifically listed e.g;
1st Rule Breaker Example – Comments
Comment paths are disallowed:
- Disallow: /comment/
- Disallow: /*/comment/
- Disallow: /comment/reply/
And the result, Bing crawls these paths
- 188.8.131.52 /comment/169 2004/07/13 14:24
- 184.108.40.206 /comment/179 2004/07/13 14:25
- 220.127.116.11 /comment/201 2004/07/13 14:26
Microsoft Security Essentials Under Microscope
A look at Microsoft Security Essentials, the free anti-virus application from Microsoft. Is Microsoft Security Essentials any good? Will it protect a Windows PC from most common threats? Can MSE compete with commercial security applications?
Over the next few months we will see.
Annoyed with Commercial AV Software.
I have become increasingly annoyed with commercial anti-virus applications. They have become overpriced, use too much system resources, interfere with other applications, or slow down internet access. Worse still, none of the apps are able to detect every virus or malware… An example is Trojan Generic 24, which seems to be only detected by AVG (but doesn’t stop or remove it). Trend Micro Titanium and Norton AV don’t find all versions of this dangerous trojan.
Open Source Becoming Mainstream
“Open source is a development method for software that harnesses the power of distributed peer review and transparency of process” (Open Source Initiative)
Considering a poll being conducted on eWEEKeurope open source software is gaining penetration with business users.
47% of respondents significantly favour open source applications, and a further 15% are trying it out.
Slightly more tha 63% of responders make use of OpenSource to some extent.
A further 8% state they might use it but are locked into propretory software.
Only 13% said they they do not trust it.
Refering to a Softpedia article, Linux makes up 40.6% of the operating systems in use, while Microsoft Windows still holds a narrowing lead of 57.5%
Open Source in South African Public Sector
As far back as 2003 open source software started gaining support at official levels in South Africa. In 2003 the CSIR (Council for Scientific and Indsustrial Research) Launched an project to assist in the development of open source in the region (SouthAfrica.info).
A number of Government departments today rely on opensource, including The Northern Cape Provincial Government, Department of Land Affairs, and SA Revenue Services (Linux and Oracle). Education also makes extensive use of open source applications for in-school computer learning programs.
Facebook and Twitter Opensource.
I wonder how many of the approximate 800 million Facebook users realise the worlds most used social media network is based on open source technology.
Then there are the more than 600 million users of Twitter, another major open source appplication. How many of the 13% who ‘do not trust’ open source are regular tweeters, and spend hours every week on Facebook?