Regarding Downtime over the Past Day (9/12/10)

[LiNuX]

Our datacenter did a routing hardware upgrade on the entire datacenter, and in order for that to happen, they had to shut down, remove, and replace all routers. Unfortunately, the smart people at our DC decided to just go for it without testing the new routers. They figured out they were incompatible so to fix the issue, it took them about 16-17 hours in total with several hours of outages in between.

I just want to let you know that nothing on this forum has been lost. Our server was never shut off, the network router was just replaced which made it inaccessible.

I would have notified all of you ahead of time if I knew but I found out just as it was happening. Sorry for those that didn't know what was happening but the few that came into the IRC channel knew what was happening as I told a few mods about it last night and this morning.

~LiNuX

[jango]

/me passes Linux a beer

[LemonRising]

Haha was wondering what was going on. Didn't cause too much inconvenience.
It was funny though, it would start working again, and just as I'm about to post, BAM offline. haha

[LiNuX]

Haha was wondering what was going on. Didn't cause too much inconvenience.
It was funny though, it would start working again, and just as I'm about to post, BAM offline. haha

Yeah, I was literally yelling at them for now knowing how to properly upgrade routing hardware. I mean...who wouldn't test it on a select few servers before installing it on the entire datacenter? Their entire node was down full of 1000s of servers.

[Jaykub]

Thanks for letting us know!

I remember a long time ago it seemed like we were down every other day.

[Knibbler]

Thanks for the info Linux!

[LemonRising]

lmfao. I can't even imagine you yelling. Your robot/human voice ablazin~

At least it's done with now.

[jango]

lmfao. I can't even imagine you yelling. Your robot/human voice ablazin~

Well, just to give you an idea .. here's what he was like after 2 hours of downtime ..

http://www.youtube.com/watch?v=ROPEICInVPY

[LiNuX]

Thanks for letting us know!

I remember a long time ago it seemed like we were down every other day.

yeah, that was a few years ago, we didn't have our own server - had to rent space from someone else's server. We weren't big enough and weren't making enough money to have our own server and it was like 2-3 months of constant downtime every other day, annoyed me.

I know the host owner, he's a good friend of mine so I never blamed him, the DC he was with had crappy techs. He eventually moved to a new DC which helped. We stayed with him until GF got bigger and it was causing issues in a shared environment and I had to get my own server then.

It was like 2 years ago...time flies.

Well, just to give you an idea .. here's what he was like after 2 hours of downtime ..

http://www.youtube.com/watch?v=ROPEICInVPY

+++++rep

[LemonRising]

LMFAO JANGO

perfect clip.
perfect.

+rep as well... err... i would but i need to spread it around.

D: +spiritrep