Who? Where? What? When? Why?

Well here it is, the post to end all speculation and before all the rumours start up.

I've been pondering at work the most interesting way to write this, I was going to include a video clip and everything - but I've just got back from the bar, and not in the mood anymore

Anyway, suffice to say, I'm sure you all noticed all was not quite right from Thursday evening, and I'm sure you all also want an explaination.

Well, someone absolutely awful happened, a server crash so unimaginably horrific that it took 3 whole days to fix.....

Except it didn't.... what actually happened was this...

23/03 - 15:25 GMT - I board the nightbus to Kyoto
23/03 - 16:34 GMT - My phone battery dies
23/03 - 21:45 GMT - mySQL Service has memory leak and crashes
23/03 - 21:46 GMT - Email alert sent to my mobile..... (see the problem here)
26/03 - 13:12 GMT - I finally find a shop that sells a phone charger, charge phone...
26/03 - 14:35 GMT - Turn on phone to get alert warning me mySQL service is down....! Leave ancient temple in Nara where I currently am and get ride to nearest internet cafe.
26/03 - 14:45 GMT - Remote login to FF server
26/03 - 14:46 GMT - Click: My Computer > Manage > Services > mySQL > Start - FF back up.....

So there you have it.... all exciting... NO, just so damn annoying... all that needed to be clicked was the start button....

Technically also, the server was never down Just the mySQL bit.... otherwise you wouldnt have got the "error" screen of course... even more interestingly, everytime that error screen is shown, i get sent an email warning me of an issue - this morning when I logged onto outlook i had no less that 38,732 emails waiting for me!!?!?!

As for what can be done to make sure this doesn't happen again, I've taken up a new contract with the datacentre to receive alerts on my behalf - so they can then act accordingly....

So thanks to Sammi and H for sorting out the temporary home, and sorry again to everyone for the inconvenience I must have caused you

 

[AppleSei]

i hope you are gonna be a good boss and reply to evryone of those emails

glad to see nothing major

 

[donaldmaclennan]

I sure it was more of inconvenience for you then for us!

sorry you lost your weekend

 

[Babysei]

LOL just think of it as flattery Ben We all missed FiatForum that much that we hounded you til the end

Just shows though what a massive impact FF has on peoples lives and that you've created something that you must be really proud of

 

[Stu DemonD]

Next time, we will make it 60,000 emails :slayer:

 

[custard]

just set opera to refresh every 5 seconds

 

[Jai]

38732 e-mails, you're gonna have bleeding fingers replying to that lot......

 

[Giggz]

bet all those emails was a select all and delete lol

 

[gaz.d]

glad we back online, dave is sorry for ruining your trip

 

[charlieboy]

38732 e-mails, you're gonna have bleeding fingers replying to that lot......

copy and paste

 

[styles]

Next time, we will make it 60,000 emails :slayer:

Hmmmm sounds to much of a even number I feel bad now that i didnt send a email or two i feel left out .

 

[Helz]

Thanks Ben for spending your free time getting us back up and running We missed yooooooooooooooou!

 

[Fiat Fan]

i had nothing to do on my computer, so i had to go on ebay and spend money

 

[Liam]

Was the 38 thousand emails only users trying to access the site or bots and users?

Liam

 

[Morritt]

could you not set the service to auto restart on failure? if the service had actially crashed completely (rather than just hung) it should have been up in a minute or so.

 

[ben]

Indeed so. I usually dont like doing this, as I am normally right near a machine, so like to look at actually why - rather than just hash it back up.

I've now found the actual reason for the crash in the first place, which is good news - dodgy coding by another script basically just overloaded the server .... it's since been taken offline for coding review!

 

[Koa]

That's really weird because I found two other forums getting the same error over those days the FF was down, but they were more random, sometimes I could get on and some not but always the same error message.

 

[Morritt]

That's really weird because I found two other forums getting the same error over those days the FF was down, but they were more random, sometimes I could get on and some not but always the same error message.

probably overactivity from all the regulars here hitting them

 

[ben]

Update on this I forgot to post.

The FF server has a number of other sites running on it, one of which had an old version of PHPBB running. This got exploited and the upshot was a massive attack was made on the mySQL database - which then crashed.

The offending site has since been updated - and as I said, a maintenence company now keep eyes on the server as a whole too