Sunday, 23 February 2014

Morning all... except the server

She's gone again, and I'm in touch with support already.  I've also reminded them about the extra memory, so please wait out.  Now.  Breakfast...

EDIT: Sometime before 11:30am we were back - I forgot to take a note of the time.  I closed the forum briefly to do some repairs, and it seems to be working fine now.  Someone had tried to post a topic 5 or 6 times, and they were all broken, so had to be deleted, and there were some small errors in the database that needed tidying up.

Mike.

Saturday, 22 February 2014

What was I saying? Oh yeah... memory!

We've been talking to our service provider about ways of increasing the stability of the server, and they're of the opinion that given that we have over 1.2m posts and a database that exceeds 2gb (which I get the impression is a bit big), we should really have more RAM on our server to give it some breathing room when it gets busy.  We typically have in the region 400+ people browsing the forum at any one time during the daylight hours, and we shift some sizeable chunks of data around.  The last time I looked at the stats (also the first time!) we have an average of over 8 million page impressions each month, with almost 46 million hits from 675,000 visits.  That's a lot, and it takes its toll on our server.

We're currently looking into installing extra memory and are waiting for quotes from our hosts before proceeding.  The server will need to be powered off to install the RAM once we've decided how much to put in (as much as we can afford), but we'll post up a warning, and try to make it happen during the European night, where the largest percentage of our membership resides.  It should be pretty quick to do once the technician gets to our box in the datacentre, as it's just a case of opening the box, taking out the old RAM (if necessary) and slotting in the new DIMM(s), then closing up the box and pushing the power switch.  They may run some diagnostics on the new configuration in order to check it works, but we shouldn't be down too long unless there's a problem.

Mike.

Thursday, 20 February 2014

You snooze, you lose

I woke up to a down server and a number of messages on my phone from friends that help out with keeping me informed when I'm away from my PC.  I couldn't even get in via the text based SSH interface, so Support were called, and they're working on it.  I'm a bit late posting this as I was busily sanding away at the intakes of the L-29 Delfin I'm building during the down-time.  Sorry 'bout that! :$

EDIT:  As of 15:41 we're back :)  A reboot was done after some investigation, so the forum is back and running.

EDIT: 11:29 21st Feb 2014 and it's off again.  We got a quick response from support and they're looking into it.  Sorry for the interruption folks, but we'll resume normal service as soon as we're able :)

EDIT: 12:50 and some database errors that weren't immediately evident have cropped up now we're using it again, so I've set a repair process in motion.  The server will be unresponsive or slow until this has finished, so bear with us.

Mike.

Saturday, 25 January 2014

Meh!

We're off again, and I've been onto support both at our hosts and at Invision, who supply the forum software.  Each time the server goes off, we get it back up and running again reasonably quickly (providing I'm not asleep or elsewhere), but we never quite get to the bottom of the cause.  This is what I'm going to try & do this time around.  Wish me luck!

Quick update 5:30pm Sat 25 Jan 2014: Support are going to get the actual box in the data centre on screen to see what info they can glean from the box before trying another reboot.  We're working on it, so bear with us :)

Late Update: I've disabled some 3rd party hooks (think add-on apps), and the last one to be disabled brought the load right down.  I'm leaving it at that for now, and have passed some information to Invision for them to study.  Thanks for bearing with us ;)