PipelineDeals Update – What went on this week?

For those of you who do not know, it has been a rough week for PipelineDeals as we have had two outages totaling 34 minutes. First and foremost, we apologize for the problems. Second, we want to address any questions that people may have and update you on what we are doing to fix the problem. We received this email from one of our customers yesterday and thought it would be a good example of the frustration some of our customers may be experiencing. Please feel free to read through the email from our customer Jennifer and our response below to see how we are addressing the problem.

Customer Email:

Hi,

We have been having quite a bit of trouble lately with the site going down frequently.  I am trying to get in right now and it is down again.  I wanted to express my frustration – even though it seems to only go down temporarily, it is happening frequently and it is extremely inconvenient.  Is this something your company is aware of and planning to address?

Thanks,
Jennifer

Response to Jennifer’s email:

The short answer is YES. We are *very* aware this is happening and we are very very sorry that we have had outages this week. We had a 12 minute outage earlier this week (March 28) and a 22 minute outage yesterday (March 31). I personally apologize – this should not happen and we are taking measures to fix the problems as we speak.

The reason for the outage is we are a victim of our own success. Admittedly this is not a good reason. We have grown and the requirements on our database and our technical infrastructure have increased as well. Over the last 5 years we have experienced growth problems like this in a stair step fashion. The PipelineDeals application will power along smoothly for a period of time (sometimes up to 2 years even) and then we hit some problems. The reasons for the problems are usually the same (growth = good) but the outcomes, as in this case, can impact our users (downtime = bad). Our job is to do our best to make sure these matters never impact you – something we have not done well this week.

Please know that our email, mobile phones and online monitoring tools light up like a christmas tree when even the smallest blip occurs. Our team is immediately on the case and we work feverishly to fix the problem and bring the application back online. The good news is we have never lost any data, but, as you know, we have lost the ability for our customers to access the application. This is what you experienced today and for which we are sorry.

Here is what we have done and are doing to fix the problem:

1) On March 1st we hired another developer to focus solely on our infrastructure. Before March our Lead Developer, Grant, was able to handle the development operations tasks himself but we saw the writing on the wall (arguably not quite early enough) and hired Michael. Michael is a quick study joining us from the Human Genome project at Baylor University and has been feverishly working to get things settled down. I am confident we have the right person on the job.

2) Our development team launched a new infrastructure today when the site was down that is considerably more powerful. We were going to wait to launch this solution until this weekend, but when we were offline, our Lead Developer made the call (and I believe the right one) to roll the fix in. This new infrastructure is running now. We believe this new solution will remedy the issues we have all been unfortunately experiencing on and off this week. Admittedly we may experience a couple more hiccups in the next few days but they should be minimal if they occur at all. We are doing our best to make sure that we have no more outages. (Note: If any customers would like a more technically gory look into what went wrong, please let us know in the comments and our developers will provide a post-mortem next week when they get some breathing room.)

3) Finally, I don’t know if you are on Twitter, but when we do go down we always update status on our Twitter feed located here:

http://twitter.com/#!/pipelinedeals
 

(End email response) 

To all of our customers, please feel free to ask any follow up questions in the comments below. We will continue to keep our customers updated as situations warrant. I know I speak for everyone here at PipelineDeals when I say we are sorry and we thankour customers for being understanding. We are fortunate to have found such a great lot of people who choose us as thier sales application. We will get these problems fixed as soon as possible and keep you informed as to our progress.