406oc.co.uk Downtime - Whats happened and whats happening!

We tell you what we're doing, you tell us what we're doing good or bad.

Moderator: Moderators

User avatar
nick
Site Admin
Posts: 340
Joined: Wed Jul 27, 2005 8:59 pm
Location: Glasgow, Scotland, UK
Contact:

406oc.co.uk Downtime - Whats happened and whats happening!

Post by nick »

Last updated Tuesday 27/08/2013 at 1030Hrs.
Right. The tempoary server has now been moved to Amsterdam and is running fine. Backups are now stable, and the mailing function is now operational. Everything seems to be back to normal, except the pages load a heck of a lot faster now. I am going to be looking at the attachment side of things just now, as we have over 1gb of files, a lot of which appear to be abandoned. I'll need to figure out what the best option is for keeping attachments. Anyway, onwards and upwards!

Nick



Previous Update Wednesday 07/08/2013 at 2200Hrs.

Expect some issues at present as I iron out bugs...

Right. I've had a chance to do some work on the forums, and got everything running smooth enough to write this stuff down.

First off, I'm sorry for the downtime over the past few days. I'm also sorry for the forum performance over the last while, which includes pages taking a while to load (in some cases, up to 60 seconds) and pages refusing to load after making posts. We've all been there and theres nothing more frustrating than crafting a nice reply and seeing it fail with a white screen. Sorry!

To explain what happened. I received an email from a forum moderator letting me know about the site being down. He described it as
lying face-down in a puddle of sick since Saturday evening.
Quite possibly the truest statement I've ever read when a site goes down.

Before I used dreamhost.com for the forum hosting. I've been with them since 2007 and never had any issues until recently. I'm guessing within the last year they started putting more and more people on the same server and it unfortunately had a negative effect on the performance of the forum. They do, however, have an awesome status blog where if theres any issues at all, they post them there. This means that if the forum appears down, some of the folks on here will have a look, and see if theres any issues so I know if I have to report it. On this occasion, there were no status updates.

Checking within the dreamhost site itself (after logging in) I was greeted with a reason for the site being down. In short, a RAID card had failed (which controls the hard drives) and they were replacing the raid card. Because of the way the card failed, they had to restore the hard drives from a backup.

The RAID cards were replaced about 24hours after the initial fault.

They then ran a script to restore all the files form backup, but still had the data on the old hard drives so it could be accessed. Because of the amount of work the server was under (Serving hundreds of web sites AND copying and replacing TB's of data), the server began acting really slow. Not ideal, but at least the site was up... Sort of.

Because the server was so busy (called how much load the server was under) certain tasks on the server couldn't be completed. Like showing a web page. This started causing 503 error messages (Service unavailable).

Now this is where I share some blame. Due to the sheer size of 406oc.co.uk, my latest backup was about 2 weeks out of date. I didn't really fancy loosing anything, so I used a little bit of trickery and managed to backup the database. From there, I've set up shop with a new host (who I had been testing with - the mods were assisting) and have transferred the database over.

I've now managed to transfer over almost 1.5Gb of files relating to this forum alone (Wow. Just wow. There will be some cleaning up going on!) and as a result, everything *should* be back to normal.

So. Thats you guys and gals up to date with whats happened so far.

As for whats going to happen.

I still have a few behind the scenes things to work on regarding the forum. I've set it up currently in a San Francisco data centre. It will be moved to a Data Centre in Amsterdam (closer to the UK and within the EU for marginal speed increases) sometime within the next week. The site will be closed with a message advising of when this will be. Because it will involve DNS changes, it *could* take up to 24 hours to fully change over.

I also have to sort of the email address for the forum. At the moment, there are NO emails being sent out. You need emails to register, receive topic reply notifications, receive PM notifications, etc.
This is my priority just now. It may be a gmail address the forum sends email from as a temporary measure. Again, I'll update here when I know.

Because the server effectively went dark, I've seen a few messages where people didn't know where to go to let me know, and confusion over the official 406oc.co.uk Facebook page. As a result, I've set up this Facebook here. Anyone fancy making a bigger version of our logo? ;)

If you have any questions about whats happened, feel free to leave me a message below and I'll try and give you an answer!

Below I've left the original message from this post.
Hi folks.

As some of you have realised, 406oc.co.uk is not working as well as it should. The site has been suffering from hardware issues which the host has been trying to solve.

Unfortunately, attempting to fix one thing has made it more complicated (See this here. One of the most apt images I have ever seen.).As a result, 406oc.co.uk has been down for over 48 hours, and I have had no way to contact you.

My immediate plan is to move the database and web server to a new host, and get 406oc.co.uk up and running within the next 24 hours.

If your reading this, the DNS changes have propagated, and the site is now set up (albeit hastily) on a new server.

If you have any questions or comments, please post a message below.

Sorry for 406oc.co.uk being down.

Thanks,

Nick, the Admin & Moderator teams for 406oc.co.uk

Just FYI folks. I'll post a full explanation of whats happened either tomorrow or Wednesday. There will be some more downtime, but it will be scheduled.
Thanks, Nick
Last edited by nick on Mon Aug 18, 2014 4:14 pm, edited 1 time in total.
Reason: Remove from Global list. No longer required to be sticky. Nick
Thanks,
Nick
DaiRees wrote:I got the "premium" standard fit system in mine, 10 speakers, changer, JBL amp. I'm quite happy with it for listening to the radio / CDs or my "teach yourself Welsh" stuff on the way to work :lol:
User avatar
nick
Site Admin
Posts: 340
Joined: Wed Jul 27, 2005 8:59 pm
Location: Glasgow, Scotland, UK
Contact:

Re: transfer of host.

Post by nick »

***
Reserved for future updates
***
Thanks,
Nick
DaiRees wrote:I got the "premium" standard fit system in mine, 10 speakers, changer, JBL amp. I'm quite happy with it for listening to the radio / CDs or my "teach yourself Welsh" stuff on the way to work :lol:
GingerMagic
3.0 24v
Posts: 3577
Joined: Fri Apr 06, 2012 9:02 pm
Location: Bournemouth

Re: transfer of host.

Post by GingerMagic »

Good work fella, seems to be up and running.
I thought I had been banned or something.... :shock: :cry:
2003 2.2hdi estate - mine
1998 Volvo 940 auto estate - also mine
2019 Citroen C3 something - the wife's
PP2000 user, can help with faults / diagnostics in the Bournemouth area.
User avatar
Ares
3.0 24v
Posts: 673
Joined: Mon Apr 18, 2011 7:36 am
Location: Istra,Croatia

Re: transfer of host.

Post by Ares »

haha me too.
Of all the forum i have been,this has been the best so far,i dont mind if something further happens,we shall wait.
I know there is a lot of work on this forums to keep them alive so i respect that.
cheers
dirtydirtydiesel
3.0 24v
Posts: 1521
Joined: Sun Apr 28, 2013 11:02 pm
Location: Coalville, Leics.

Re: transfer of host.

Post by dirtydirtydiesel »

Keep up the good work Nick 8) , without you & this forum I would be lost in a sea of 406 electrical woe's :cry:
2002 110 HDi estate, Rapier in monaco blue! ( found quite a bit more power :cheesy: ) SOLD I've sorted the airbag light :cheesy:
1998 3.0 V6 SE coupe in Diablo Red ( my baby ) sold
2006 206 1.4 16v sport in Aegean blue ( wife's shopping trolley :roll: )

2006 BMW e61 535d m-sport :supafrisk: run's for cover :lol:
User avatar
steve_earwig
Moderator
Posts: 19811
Joined: Thu Oct 04, 2007 6:09 pm
Location: Jastrebarsko, Croatia http://www.jastrebarsko.hr/lokacija/

Re: transfer of host.

Post by steve_earwig »

Well done Nick 8) The site's looking 150% from here.

Btw thanks for getting back to me yesterday, I didn't reply because I knew you were in the thick of it. :oops:
GingerMagic wrote:I thought I had been banned or something.... :shock: :cry:
I was pretty sure it was going to be the host as they weren't the most reliable. I managed to get the site to load a couple of times Sunday and I could see that nobody had posted since Saturday evening, so I knew we were all in the same boat. Sad thing was I wanted to post something somewhere to say not to panic but I couldn't work out which Farcebook page was current :(
Unskilled meddling sin©e 2007

The submitted form was invalid. Try submitting again.
Sonia406
3.0 24v
Posts: 1738
Joined: Thu Feb 14, 2013 4:35 pm
Location: Somewhere in the UK

Re: transfer of host.

Post by Sonia406 »

I must admit I presumed it was server issues as well!
Glad you're back up and running - I've missed you all :D

I know it's still early days after all this change over, but do you plan on getting us onto Auto Guide, to enable us to reply from our mobile phones, instead of having to open a brower? :mrgreen:
Miss Pug 2001 - 2023.
User avatar
Doggy
Mod with a 2.2 HDi, De-Fapped!
Posts: 10710
Joined: Mon Oct 13, 2008 11:49 pm
Location: Northants

Re: transfer of host.

Post by Doggy »

Great work Nick. Looking good so far. 8)
2002 HDi 2.2 Exec Estate, (2008-12) (wonderful)
2003 HDi 2.2 6-speed Exec Estate (2012-19) (also a gem)
2009 Citroen C5 2.0 HDi VTR+ Estate (godawful heap)
2008 BMW E91 330i touring (great fun - murdered by a reversing SUV)
2007 BMW E91 325i touring (slower smoother quieter)
User avatar
rwb
3.0 24v
Posts: 2612
Joined: Thu Jun 17, 2010 7:53 pm
Location: Yorkshireman exiled in Salop
Contact:

Re: transfer of host.

Post by rwb »

Glad to see we're back :D

But while your back was turned I did try some other forums :oops:

Current: 407 2.2 HDi 170 & C6 2.7 HDi.
Former: 406 1.9 TD; 406 HDi 90; 407 2.2 160; 307cc 180; 508 HDi 140.
Map of PeugeotForums users offering PP2k
Sonia406
3.0 24v
Posts: 1738
Joined: Thu Feb 14, 2013 4:35 pm
Location: Somewhere in the UK

Re: transfer of host.

Post by Sonia406 »

rwb wrote:Glad to see we're back :D

But while your back was turned I did try some other forums :oops:

You and me both............. Where did you go? :lol:
Miss Pug 2001 - 2023.
User avatar
rwb
3.0 24v
Posts: 2612
Joined: Thu Jun 17, 2010 7:53 pm
Location: Yorkshireman exiled in Salop
Contact:

Re: transfer of host.

Post by rwb »

I've been to peugeotforums.com where I saw trem, and maybe someone else :supafrisk: and Sam talked me into joining tdoc which I'm quite glad about -- I'll enjoy reading that.

Who've you been cheating with? :wink:

Current: 407 2.2 HDi 170 & C6 2.7 HDi.
Former: 406 1.9 TD; 406 HDi 90; 407 2.2 160; 307cc 180; 508 HDi 140.
Map of PeugeotForums users offering PP2k
trem1
3.0 24v
Posts: 1842
Joined: Sat Dec 31, 2005 2:32 pm
Location: south shields,tyneside

Re: transfer of host.

Post by trem1 »

rwb wrote:I've been to peugeotforums.com where I saw trem, and maybe someone else :supafrisk: and Sam talked me into joining tdoc which I'm quite glad about -- I'll enjoy reading that.

Who've you been cheating with? :wink:
yep i use peugeotforums quite a lot as do some others, dont we mrs406 :roll: i joined tdoc years ago but never bother with it


2004 Iron Grey 407se 136bhp......Written Off
2006 Moonstone blue 407 se 136 bhp.....Written off
2006 silver 407 SW..........replacement
Sonia406
3.0 24v
Posts: 1738
Joined: Thu Feb 14, 2013 4:35 pm
Location: Somewhere in the UK

Re: transfer of host.

Post by Sonia406 »

trem1 wrote:
rwb wrote:I've been to peugeotforums.com where I saw trem, and maybe someone else :supafrisk: and Sam talked me into joining tdoc which I'm quite glad about -- I'll enjoy reading that.

Who've you been cheating with? :wink:
yep i use peugeotforums quite a lot as do some others, dont we mrs406 :roll: i joined tdoc years ago but never bother with it

What is tdoc? I am guessing turbo diesel owner club? :oops:

Yeah, Pug forums is okay... it's no 406oc I grant you that much :lol:
Miss Pug 2001 - 2023.
dirtydirtydiesel
3.0 24v
Posts: 1521
Joined: Sun Apr 28, 2013 11:02 pm
Location: Coalville, Leics.

Re: transfer of host.

Post by dirtydirtydiesel »

[quote="Sonia406"]


What is tdoc? I am guessing turbo diesel owner club? :oops:

It's a very busy forum for info on tip's 'n' tricks on making turbo diesel's faster :twisted: handle better ect: :wink:
2002 110 HDi estate, Rapier in monaco blue! ( found quite a bit more power :cheesy: ) SOLD I've sorted the airbag light :cheesy:
1998 3.0 V6 SE coupe in Diablo Red ( my baby ) sold
2006 206 1.4 16v sport in Aegean blue ( wife's shopping trolley :roll: )

2006 BMW e61 535d m-sport :supafrisk: run's for cover :lol:
Sonia406
3.0 24v
Posts: 1738
Joined: Thu Feb 14, 2013 4:35 pm
Location: Somewhere in the UK

Re: transfer of host.

Post by Sonia406 »

dirtydirtydiesel wrote:
Sonia406 wrote:

What is tdoc? I am guessing turbo diesel owner club? :oops:

It's a very busy forum for info on tip's 'n' tricks on making turbo diesel's faster :twisted: handle better ect: :wink:

Just as I thought... diesel related... absolutely no point in me joining that one :lol: :oops:
Miss Pug 2001 - 2023.
Post Reply