Server Resource Overload


My Domain is too Busy – Server Overloaded

server resource overload iconI never though I would be saying this – at least not within 12 months of switching my site(s) to CMS. It’s only 12 months – nearly twelve months, since I installed Drupal for my main website, less since adding two WordPress sub-domain sites. Now my the shared server is having to push out more than 80 000 pages a month – or so the server stats show. And it’s causing problems!

It’s causing problems because my domain is exhausting available server resources. The most common resource exceeding limits is CPU – often hitting 100% utilisation. Memory is often going over the available share of 1GB – that’s right 1000MB – available for the domain. While maximum entry processes are averaging between 4 and 6, there have been several instances where the 20 limit was exceeded.

And I’ve been wondering why I’ve been battling to edit and publish content for the past 6 weeks.

server resources used by graphicline.co.za graphic chart

Server Resources Usage Chart for graphicline.co.za

Pages Served Double in a Month

Well, nearly. In June 47000 pages were served, in July 82000. And it’s since July I’ve been having a lot of hassle working on the sites. As today was a day for looking at statistics, I produced a stats forecast for page served increase over the next six month. Without any major other factors, the trend indicates 120 000 pages per month in January 2013. But that’s only stats, and I didn’t weight the result for the usual reduction in traffic in January and February.

pages served by graphicline chart

Pages Served by Graphicline Websites, and Forecast

However I must consider I have at least another 180 pages to publish on the shop site, as well as my regular articles and posts. It’s not impossible the pages served will hit 150 000 early in 2013. If that happens I may have to put the 3 busy sites into maintenance mode just to publish or edit content, not a really great idea.

And to top it all, I have another commercial site in development I need to add to the domain, as well as one more personal interest site… And I’ve already removed both my live testing sites in the last month…

So What to Do Going Forward

It seems to me there are only a few options.

  • The first solution is a big upgrade in hosting, and the associated cost – which I just cannot afford!. Ideally the site needs a dedicated server, or at least one I can share with a limited number of other websites. Or it needs hosting on a distributed hosting solution. Both are way out of my budget.
  • I guess I could always stop adding content, maybe even ban Google bot (Ha ha) That would definitely work – 99% of my traffic comes from Google
  • I could even put the domain up for sale! Which may not be a terrible idea after all. Then I could go fishing, or even think about finally making the much desired move to Mozambique or even New Zealand! Well, I’m open to offers!

Excluding the Banned Visitors

hits denied access to site

Each hit here represents a bad action

The stats used exclude 16 000 hits from banned IPs in July, and already this month there have been 14 000. The graph shows an exponential increase in 403 forbidden status messages in the logs. My sites have been heavily targeted by hackers (mainly bots) and comment spammers (also mostly bots). The number of IPs banned gets largely daily. Often entire ranges are denied access – several hundred thousand IPs. I have started banning these ranges by blocks if they come from some geographic regions, mostly the far east, as I get no worthwhile traffic from there anyway…

I deny access to most search bots except Google and Bing. Any bot breaking rules gets banned. Hotlinkers now get banned completely.

And that statistic is scary: The past 3 months have seen an exponential increase in bad visits… If this trend continues the break point will come long before 2013

Impacting Page Loads

Worse than the hassle to publish contnet and manage the sites is the negative impact this server overload is having on page loads. Google crawl page load has jumped on average 30% – that is 30 percent slower. Google Analytics shows page download time to visitors has worsened by as much as 50% – from 3oooms to 4000ms, with one particularly bad day when pages were taking more than 60 seconds – but that had more to do with a broken SAT3 internet cable than the server….

Interestingly, my Drupal site still serves page significantly faster than either WordPress site…

Now if only all my sites were hosted on WordPress.com

me on google plus+Mike Otgaar

Advertisements

About Mike

Web Developer and Techno-geek Saltwater fishing nut Blogger

Posted on August 17, 2012, in General News, Websites and tagged , , , , . Bookmark the permalink. Leave a comment.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: