Right. The tempoary server has now been moved to Amsterdam and is running fine. Backups are now stable, and the mailing function is now operational. Everything seems to be back to normal, except the pages load a heck of a lot faster now. I am going to be looking at the attachment side of things just now, as we have over 1gb of files, a lot of which appear to be abandoned. I'll need to figure out what the best option is for keeping attachments. Anyway, onwards and upwards!
Nick
Previous Update Wednesday 07/08/2013 at 2200Hrs.
Expect some issues at present as I iron out bugs...
Right. I've had a chance to do some work on the forums, and got everything running smooth enough to write this stuff down.
First off, I'm sorry for the downtime over the past few days. I'm also sorry for the forum performance over the last while, which includes pages taking a while to load (in some cases, up to 60 seconds) and pages refusing to load after making posts. We've all been there and theres nothing more frustrating than crafting a nice reply and seeing it fail with a white screen. Sorry!
To explain what happened. I received an email from a forum moderator letting me know about the site being down. He described it as
Quite possibly the truest statement I've ever read when a site goes down.lying face-down in a puddle of sick since Saturday evening.
Before I used dreamhost.com for the forum hosting. I've been with them since 2007 and never had any issues until recently. I'm guessing within the last year they started putting more and more people on the same server and it unfortunately had a negative effect on the performance of the forum. They do, however, have an awesome status blog where if theres any issues at all, they post them there. This means that if the forum appears down, some of the folks on here will have a look, and see if theres any issues so I know if I have to report it. On this occasion, there were no status updates.
Checking within the dreamhost site itself (after logging in) I was greeted with a reason for the site being down. In short, a RAID card had failed (which controls the hard drives) and they were replacing the raid card. Because of the way the card failed, they had to restore the hard drives from a backup.
The RAID cards were replaced about 24hours after the initial fault.
They then ran a script to restore all the files form backup, but still had the data on the old hard drives so it could be accessed. Because of the amount of work the server was under (Serving hundreds of web sites AND copying and replacing TB's of data), the server began acting really slow. Not ideal, but at least the site was up... Sort of.
Because the server was so busy (called how much load the server was under) certain tasks on the server couldn't be completed. Like showing a web page. This started causing 503 error messages (Service unavailable).
Now this is where I share some blame. Due to the sheer size of 406oc.co.uk, my latest backup was about 2 weeks out of date. I didn't really fancy loosing anything, so I used a little bit of trickery and managed to backup the database. From there, I've set up shop with a new host (who I had been testing with - the mods were assisting) and have transferred the database over.
I've now managed to transfer over almost 1.5Gb of files relating to this forum alone (Wow. Just wow. There will be some cleaning up going on!) and as a result, everything *should* be back to normal.
So. Thats you guys and gals up to date with whats happened so far.
As for whats going to happen.
I still have a few behind the scenes things to work on regarding the forum. I've set it up currently in a San Francisco data centre. It will be moved to a Data Centre in Amsterdam (closer to the UK and within the EU for marginal speed increases) sometime within the next week. The site will be closed with a message advising of when this will be. Because it will involve DNS changes, it *could* take up to 24 hours to fully change over.
I also have to sort of the email address for the forum. At the moment, there are NO emails being sent out. You need emails to register, receive topic reply notifications, receive PM notifications, etc.
This is my priority just now. It may be a gmail address the forum sends email from as a temporary measure. Again, I'll update here when I know.
Because the server effectively went dark, I've seen a few messages where people didn't know where to go to let me know, and confusion over the official 406oc.co.uk Facebook page. As a result, I've set up this Facebook here. Anyone fancy making a bigger version of our logo? ;)
If you have any questions about whats happened, feel free to leave me a message below and I'll try and give you an answer!
Below I've left the original message from this post.
Hi folks.
As some of you have realised, 406oc.co.uk is not working as well as it should. The site has been suffering from hardware issues which the host has been trying to solve.
Unfortunately, attempting to fix one thing has made it more complicated (See this here. One of the most apt images I have ever seen.).As a result, 406oc.co.uk has been down for over 48 hours, and I have had no way to contact you.
My immediate plan is to move the database and web server to a new host, and get 406oc.co.uk up and running within the next 24 hours.
If your reading this, the DNS changes have propagated, and the site is now set up (albeit hastily) on a new server.
If you have any questions or comments, please post a message below.
Sorry for 406oc.co.uk being down.
Thanks,
Nick, the Admin & Moderator teams for 406oc.co.uk
Just FYI folks. I'll post a full explanation of whats happened either tomorrow or Wednesday. There will be some more downtime, but it will be scheduled.
Thanks, Nick