Page: 1 2
Thor
Head Administrator
Runboard developer
Registered: 01-2003
Location: RB machine room
Posts: 4995
Karma: 232 (+275/-43)
|
Reply | Quote
|
|
A Hellish day
It all started when I wake up this morning too late to think about a breakfast, quick shower and riding way too fast to work on my scooter (not sure what it is called in USA, kind of a small bike, mine is pretty small and goes up to 80 km/h, cheap stuff, bought it a couple months ago, second hand).
Anyway, it was a 3 days weekend (14th july is national holiday in France) and it has been a great weekend.
I walk into the office, my coworkers tell me that everything is crashing, etc. A good start. Turns out that the firewall is behaving badly, and after quick investigation, it's doesn't look like I'm going to fix it quickly. While doing minor changes to the our DNS system, I make an error and break the DNS for our production web servers, which are running poorly coded Java applications, who of course crash when DNS is not working. Fix it after a while.
It's so hot in Paris people around me are almost passing out, so do I.
To the end of the day, a about 7pm actually, our master name server crashes totally. The computer itself is down. It's located a few miles from my office, but remember it's 7 pm in Paris. Means the traffic jams are at its highest. I try to call up the ISP (Worldcom), to get them to reboot the computer, but quickly remember how they are trying to "restructure" and "cut the costs", so after dialing the french support number, I somehow land in Zurich, Switzerland, speaking to people who barely can tell a computer from a basket of eggs, let alone reboot anything in *France*.
Swearing and cursing, I jump on my scooter and start riding to the colocation facility where our prod servers are hosted at. When finally breaking out of Paris, I ride through Neuilly, on a fast lane, like I've been doing it since I bought the scooter. You must drive fast there, since everyone else is driving fast, and I must drive in the left lane because otherwise I won't be able to switch lanes fast enough to take a left turn. Switching lanes driving a scooter where everyone drives 1.5 as fast as you is dangerous, so I just take the left lane.
Suddenly a big white powerful car closes in from behind, and gets real close to me, like maybe 5 meters, which is nonsense when driving at this speed (about 80 km/h, top speed for my scooter with, what, 12 inch wheels) I think speed limit there is 70 km/h. I touch my brakes to warn him he's too close, but he continues. I touch my brakes a couple times again, he goes mad and passes me, and steers back to my lane and starts slowing down and down, effectively slowing me down - he knows well I can't change lanes now, since people drive at 100 km/h+ there and it would be pure suicide. I decide that I'll follow him and rip his head off for trying to kill me. Or at least break something on his car. Happened to me before. After slowing me down to like 40 km/h, making me take great risks since a car coming from behind might not see me fast enough to slow down, he accelerates and disappears very fast, slaloming between cars.
"Whatever", I think, and continue my trip to the colocation facility, and finally get there with no casualties.
I've been there about once per 2 weeks for the last 4 years. But starting a few days ago, you can't just give them your ID card and go check your servers. You must now open a ticket with customer support (the Switzerland one) and then maybe, if it works, they let you in. After wasting close to an hour, I finally get to see my computers there, taking mental note to move all the computers (like 20) to a competitor as soon as possible.
I reboot the DNS server, which seems to have issues with its hard drive, think about how great is to have backups in these times, fix it all, and find out that my cell phone isn't working, so I can't call back to office to tell them it's working now and they can start starting the Java crap. No biggie, I just connect to the office network, log on our main server, and send them a message. Before logging out I quickly, simple habit, check the kernel logs out and whammo, one of the disks is dead, so instead of calmly driving home I have to drive back to office and replace it in the RAID array. It's all right, though, sh*t happens.
So I start driving back, too fast for my security really, but again, must drive fast on this road. I get almost ran over by a big truck who wasn't paying too much attention, I think.
This is when the engine dies. I think the engine is just plain broken, the sound of it makes me beleive it (I'm not good at mechanical stuff but have some very basic knowledge). So basicly, my scooter is dead, replacing the engine will be more expensive than what I paid for the scooter itself. I paid 180 (200 $ or so), euro for the technical revision of the scooter FIVE days ago. Anyway. It can't get much worse now, can't it?
This is when the clouds start to be really black. It has been a very hot and sunny week in Paris, and now, just NOW, the wind is starting to be strong and the clouds are clearly those of a bit storm. I park the dead scooter near a store, jump in the subway, and get back to office. When I get out of the subway, the rain is REALLY strong, everyone on the street is running for cover. Spending over 10 seconds has the same effect as spending a minute in the shower, fully clothed.
I know the server is dying in the office, and even though I have good daily backups of it, losing this server means a couple of days of pain and suffering to me and my coworkers. I just rush through the rain and get to the office as wet as it can get.
As I'm typing this, my throat is getting sore.
Oh well. At least I'm alive and akheva didn't crash
|
7/15/2003, 8:47 pm
|
Link to post
Email Thor
PM Thor
ICQ
Yahoo
Blog
|
Iansgirl
Registered: 03-2003
Location: Mi Vida Loca
Posts: 1794
Karma: 26 (+41/-15)
|
Reply | Quote
|
|
Re: A Hellish day
Sounds like you have A LOT going on..Sorry it is going so bad, Go home have a drink go to bed and try again tomorrow
|
7/15/2003, 9:00 pm
|
Link to post
PM Iansgirl
AIM
MSN
Yahoo
Blog
|
kriptonic
Heavy poster
Registered: 07-2003
Location: Frankfort Ohio
Posts: 623
Karma: 1 (+1/-0)
|
Reply | Quote
|
|
Re: A Hellish day
Yes, sorry it was so bad. If there's anything we can do Thor to make you happy let us know. I doubt there's much we can do, but we'll try.
--- - Kriptonic -
|
7/15/2003, 9:42 pm
|
Link to post
Email kriptonic
PM kriptonic
|
Thor
Head Administrator
Runboard developer
Registered: 01-2003
Location: RB machine room
Posts: 4995
Karma: 232 (+275/-43)
|
Reply | Quote
|
|
Re: A Hellish day
Haha, nothing, I just had to share
|
7/15/2003, 10:24 pm
|
Link to post
Email Thor
PM Thor
ICQ
Yahoo
Blog
|
Museum
Turkish Goddess
Registered: 05-2003
Posts: 2029
Karma: 79 (+104/-25)
|
Reply | Quote
|
|
Re: A Hellish day
Sorry to hear all that Thor,
Actually my couple of days have not been the greatest either. After having a weekend of and then back into work yesterday the first news I get from a line manager is that a young colleague whom had been in hospital had died all of a sudden on Saturday.
The poor guy was just in his early 20's a very nice friendly smiling healthy guy. He died over a kidney failure and today just as I was leaving for the day I find out his funeral is on Monday coming.
Looks like alot of us possibly about 10 maybe more will be going to pay our respects....
What a nasty few days it has been
|
7/15/2003, 11:17 pm
|
Link to post
PM Museum
|
Danta
MAN WITH THE GOLDEN CHALICE
Registered: 06-2003
Location: In my cave
Posts: 1179
Karma: 3 (+14/-11)
|
Reply | Quote
|
|
Re: A Hellish day
there are a lot of you guys who had a bad day. srry for you guys.
--- quote: Danta that I've been seeing u around in many great boards,u're kind of famous
|
7/16/2003, 6:11 am
|
Link to post
Email Danta
PM Danta
Blog
|
Jamie Lauren
Heavy poster
Registered: 02-2003
Location: United States
Posts: 293
Karma: 0 (+2/-2)
|
Reply | Quote
|
|
Re: A Hellish day
{{{hugs}}} Thor and Museum
--- Jamie Lauren: "I can write, you can write too!
|
7/16/2003, 6:51 am
|
Link to post
Email Jamie Lauren
PM Jamie Lauren
|
Museum
Turkish Goddess
Registered: 05-2003
Posts: 2029
Karma: 79 (+104/-25)
|
Reply | Quote
|
|
Re: A Hellish day
That is sweet thank you
|
7/16/2003, 11:04 am
|
Link to post
PM Museum
|
kellyosbourneuk
Regular poster
Registered: 05-2003
Location: Liverpool
Posts: 158
Karma: 0 (+0/-0)
|
Reply | Quote
|
|
Re: A Hellish day
hope u will be on museum and hopw u have a better day Thor
|
7/16/2003, 11:41 am
|
Link to post
Email kellyosbourneuk
PM kellyosbourneuk
|
kriptonic
Heavy poster
Registered: 07-2003
Location: Frankfort Ohio
Posts: 623
Karma: 1 (+1/-0)
|
Reply | Quote
|
|
Re: A Hellish day
I am very sorry to hear that mom. (musuem) If you're ever having problems and need to talk hit me up in PM.
--- - Kriptonic -
|
7/16/2003, 3:20 pm
|
Link to post
Email kriptonic
PM kriptonic
|
Add a reply
Page: 1 2
You are not logged in ( login)
|