MySql Bug Caused A Brief Transmission Interruption

Hey Blippers,

 

This is just a quick notice to let you know what caused the downtime today. We encountered a couple little problems in quick succession that led to a site outage of somewhere between 30min to 1hr, depending on how often you hit refresh. Everything is up and running again, sorry to take away your music and force you to focus on work on a Friday. It was all Yan’s fault though, he walked in this morning with a jolly excitement and exclaimed, “Happy Friday!” completely jinxing the day. Thanks, Yan, really smooth.

 

So here’s what went down. First we had unusually heavy traffic on one of our web servers, causing it to run out of memory and lock itself in an infinite death spiral. Luckily it was caught by our awesome support staff at Contegix (best supported hosting evar) before it became a real problem and they quickly reconfigured the server. Threat averted, right? Not quite.

 

All of a sudden the main database started locking up and wouldn’t allow any new connections, which means no blipping, no giving props, no logging in if you hadn’t been already. In order to troubleshoot that problem we swapped the master database to the understudy, the database in the wings, and under normal circumstances this would go unnoticed by anyone except the people back stage. But this time the same thing started happening to the backup database. Crap.

 

Fast forward through checking logs for DoS (no), checking the data layer logic (fine), find another site causing the same problem on a different database (it’s not Blip), scratch head, Google, MySql support forum, repeat, find bug, find fix, reboot. Breathe…

 

So there you go, that was our exciting Friday morning. There’s still a little residual database stuff to take care of but everything should be just about back to normal. In the meantime some of you finally got to see our special little error page. Yes it’s just for fun, Blip.fm wasn’t really rebooting right on your screen, but thanks for the emails!

 

Have a great weekend everyone, I think it’s almost beer o’ clock around here. If you see me out tonight, I’ll buy you a cold man soda (or woman soda) for depriving you of Blip.fm for what I’m sure felt like an eternity.

 

Update: There’s still some problems with the data on one of the databases. We’re going to seed it from the main database now. There might be some slowness throughout today (Saturday) but should all be cleared up in a couple of hours. Also, there’s been reports of props-to-give being lower than expected. I will recalculate the props-to-give once the above is completed, so don’t worry, you’ll get them all back!

 

Update: Things have been stable for the past few hours. I just recalculated the props-to-give for all blippers and gave back any that were missing. If you still see any residual strangeness please email support@blip.fm and we’ll help you get it figured out.

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • bodytext
  • Sphinn
  • del.icio.us
  • Mixx
  • Pownce
  • Reddit
  • SphereIt
  • StumbleUpon
  • TwitThis
  • Facebook

23 Comments

melodyofyourlife  on January 23rd, 2009

I got a funky screen and then a black basic ibm reboot screen, some real funny text. then a major file error. hope that helps.

Last night I was up to 1,900+ props to give and now I only have 142? I lost 1, 750+ props to give out after the reboot. is there any chance to get them back?

GoGoSunshine  on January 23rd, 2009

That’s crazy because today my blibs keep disappearing and so does my props. They’ll show back up but then disappear again.

GoGoSunshine  on January 23rd, 2009

sorry I meant blips

ThePhillyman  on January 23rd, 2009

The site may be working but the missing props (I lost almost all of ‘em) are having a major impact on my ability to pass out some love to my fellow blippers.

melodyofyourlife  on January 23rd, 2009

@Aluciel had 1962 Props to give out as of this afternoon…now she has 58? hook her back up to .

here is her reply to me
Aluciel @melodyofyourlife Obviously, you’ve stolen my PTGO AND someone’s replies. Shame on you! :P (I had 1962 PTGO as of this afternoon…58 now)

GR8FL  on January 24th, 2009

Blip is still out of sorts.

1. Won’t accept blips no matter how many times I hit play (or maybe it does, but since the refresh is not working, I wouldn’t know).

2. Refresh not refreshing… timelines are off. Trying to get “home” again goes to hours ago or another day.

3. See #2 - same thing happens on “replies”, “props” and also when selecting another user name.

4. Please explain how people have over 1000 props to give out… I give out at least 300/day and get tons back (not as many as I give out, but that’s not my point). What is wrong with my math that I only have 290 or so to give out.

markmac  on January 24th, 2009

Yeah, everything I blipped yesterday and the props I received disappear sometimes when I click home or refresh. Other times the blips and props are up-to-date. I haven’t noticed a problem with the number of props I have to give out, though.

thephillyman  on January 24th, 2009

Blippers!! Why on earth do you have more than a few hundred props in your PTGO? Give them away, people! What are you saving them up for? They ain’t nickels. Share the love and pay ‘em forward!!

melodyofyourlife  on January 24th, 2009

have a rough time staying logged in today. I keep having to sign back in and then I blip a song hit send then I have to sign back in. my avatar was reverted to an old one. I changed it and it changed back to the old one again. Weird.

as for @thephillyman if you get 6-8 props a lot on tunes its hard to give them all out. Play better music get more props I guess! :-P just as an FYI I give out tons I just get more back!

Miss Bella  on January 26th, 2009

I almost died! that felt like forever!

suelifms  on January 28th, 2009

As null and create another with te same adress

Milas  on January 29th, 2009

Blippers of the world, unite! I notice today that I CANNOT exist in the web world without BLIP FM since the day I got into it! What am I gonna do from now on? What garantees do I have that I will have BLIP with my dayly net cruisin…please, please, please…make it last forever!

god  on January 30th, 2009

why dont you sites put a notice on the front page telling people of this problem? or is that impossible? have you have a shed load of emails and messages about blip not working? then why not put it on the front page? i dont know much about computers so I dont know if this is do-able- just seems like a labour saving idea…common sense

jebus  on January 30th, 2009

nothing plays or works- pages do not exist…the world is crumbling around me

memori@  on January 30th, 2009

Please, nothing plays and works (pages and blips), not exist, error 404 not exist http://blip.fm/memoria

miguesme  on January 30th, 2009

Please, nothing plays and works, (pages,blips,profiles).. error 404 not exist http://blip.fm/miguesme.

Thanks

Ivonne  on January 31st, 2009

Ya van varias ocasiones que me pasa esto de no poder accesar a su página, no sé por qué estando en otra computadora que no es la de mi casa, no puedo accesar a su página y con la clave de la dirección del trabajo en estos momentos tampoco, quisiera repararan el error y ver si van a hacerlo, si no para cancelar mis cuentas y no seguirlos recomendando con mis amistades.
Gracias.

pedronunesnomundo  on February 8th, 2009

affraid somethings still not working

cant even make a simple search

whats wrong guys?…

Danielwoods  on September 26th, 2009

This is the message I got and I took a photo of it. http://digg.com/d315dlT

Proopureaum  on October 11th, 2009

Hello, it really interesting, thanks

FernandaW  on January 29th, 2011

well since yestarday i can’t hear my own station, my blips, nothing, and nobody can give me props, makes reblip of my blips…it s is very strange, cause my husband use the same computer, and doesnt has any roblems.I sent emails but nobody answered to me

Jessie  on June 12th, 2011

The forum is a brtigehr place thanks to your posts. Thanks!

hogan outlet online  on March 27th, 2012

Thanks for the post. I like this.

Leave a Comment