Server Issues 16 April, 8:00 PM EST

The TweetLater.com server is currently not responding.

We’re working on getting it back online as soon as possible.

If you happen to get to the login form, and it says your credentials are invalid, it’s because the database is down.

We will update this message when service has been restored.

Update 9:24 PM EST — The faulty drive is being replaced on the server. Unfortunately it was the primary drive, which holds the operating system. The OS needs to be reloaded, which they tell us will take between 6 and 8 hours.

Update 10:51 PM EST — The server rebuild is being started now. The database has been backed up, so there should not be any data loss. Follower processing and scheduled tweets are currently in mid-air suspension. When we reactivate the batch jobs once the server is rebuilt, follower processing will resume. All unpublished scheduled tweets and DMs that should have been published during the downtime will be published at that point. You may experience a sudden publishing of several tweets at the same time in your timeline if you had several tweets that should have been published during the downtime.

Update 11:04 PM EST –The hosting folks say that the server rebuild will take 6 to 8 hours. We’re going to assume a horizontal position for a few hours. When we’re awake and fresh, we’ll check that everything has been correctly configured on the server before we open TweetLater.com for logins. Will update here again in the morning.

Update 5:08 AM EST — The server has been rebuilt, and the backups are currently being restored.

Update 6:55 AM EST — Backups have been restored. There has been no data loss, apart from a few Keyword Alert archives that were corrupted beyond repair. It shouldn’t be an issue because those archives are only kept for 24 hours anyway. There are a few more configuration tasks left to do. It will probably be another hour, maybe two.

Update 7:02 AM EST — After doing an analysis of the scheduled tweets that should have been published during the downtime, we felt it would be prudent to put those tweets into an error condition so that they do not publish, and that you can decide whether you want to delete them or reschedule them. There are many Twitter accounts that will have a small avalanche of tweets published on them all at once if we just let the system publish the tweets. We’ll make a separate blog post regarding these tweets as soon as the system goes live.

Update 8:33 AM EST — It’s still going to be a while. Apache is still recompiling. Once that’s done, there are still a few configuration tasks to perform.

Update 9:14 AM EST — Apache recompile has finished. Now doing the last configuration tasks. It won’t be much longer.

Update 9:55 AM EST — Nearly there. Some of the batch jobs are already running, so you will see scheduled tweets being published. The site is still down because MEMCACHE is not yet running, and without it the site simply unceremonially crashes.

Update 11:01 AM EST — The hosting provider technicians forgot to compile several essential libraries into PHP. Just waiting for them to do that, then the site should come back online.

Update 12:01 PM EST — Just learned an important lesson. HostGator works on the tickets that have the oldest customer reply time first. When you keep submitting replies, such as “Please work on this, it is really important,” you bump your own ticket just further down the queue because the reply is new. So we learn a new thing every day.

Update 12:30 PM EST — And…….

WE ARE BACK!!!!

30 thoughts on “Server Issues 16 April, 8:00 PM EST”

  5. Jan

    hey guys, just wanted to say thanks. you do great work and is greatly appreciated. hopefully, you can all take a breather soon and enjoy your friday.

  6. Scott Bradley

    When it rains it pours… hummingbird has similar problems so does twitterfeed and twitter itself the next button when you try to follow more folks is not working… Not a fun situation for all concerned. Keep your chin up it will get better. Great job Tweetlater!

  9. Rebecca

    Service has been intermittent all week I've noticed, so hopefully this will help you develop a solid back up plan…Sometimes the biggest challenges reveal the biggest rewards…(redundancy is always a good thing too)Thanks for the proactive updates…hope to see the service up and healthy soon, I've been recommending to followers and friends : )

  10. Dewald Pretorius

    Not sure what you mean by the service has been intermittent all week. It has actually been running quite smoothly up to the disk failure.Complex redundancy solutions are definitely the ideal. However, one has to do the best you can do within your budget. Up until February 2009, when I launched TweetLater Professional, I provided the free service basically from my own groceries budget. Now that the site is beginning to generate revenue, I am able to expand the redundancy as far as funds allow.

  11. Tim

    We share your pain, we had an scheduled downtime of 30 minutes go to 13 hours 🙂 ouch. We moved to a 64bit OS and things are far more stable now.All the best, hope all goes well.CheersTim

  13. Rebecca

    Hi There,thanks for the response. As a professional, paying user, I assure you that the service has been intermittent…I assumed you were aware. for the past week the majority of @ posts for my twitter accounts have gone unnoticed by my Tweetlater account. I have found numerous postings on my own when reading indidivual tweets on updates of people I follow, RT or @ myself. I have confirmed this with other professional Tweetlater users including @shannoncherry who introduced me (and many others) to the service and has experienced the same, especially this past week.I understand that you may have started this as a free service prior to now, but when you are charging $30/mo ($360 a year) there is a certain level of expectation for delivery of service…which would include back ups or server monitoring that may have clued you in that your servers were about to bail on you.Redundancy doesn't have to be complex. Heck, even nightly backups on a hosted server or a virtual server VPS in a datacenter may have averted the impact of your server failure. Hosting can be very affordable (I'm sure you have many customers and tweeps that could offer or recommend a hosted solution with redundancy or back up).It sounds like this is being hosted on premises for you, perhaps in your home or office? Which also makes one wonder about security as you collect credit card numbers…but that's another topic.May have saved you from all the pain this has caused you and your paying subscribers. I really don't meant to be harsh, I love the service and have tweeted about it. Even became an affiliate (dare I ask if that is working properly?) however when I do so it is in essense an endorsement and my folks expect me to have done due diligence, therefore my detailed response.I do hope you get it all figured out (quickly : )…please keep us posted!Thanks!

  14. johngrovers

    Dewald – good work sir – nearly there! Not sure if this is what Rebecca means but my twitter page is showing no DM's sent since 9th April yet tweetlater was saying that there were – dont know if this is the same as Rebecca and it may be a twitter problem not yours of course 😉

  15. Dewald Pretorius

    Rebecca,I am running on one of the largest servers available at HostGator. So, no, the server is not sitting under my desk in my basement, and it is not my old unused 286 Intel PC. LOLI have redundant backups. That's why there was no data loss.When the primary disk that holds the operating system fails, it means the server has to be rebuilt, unless you have very expensive and complex mirroring in place. I am not there yet, unfortunately.And, I do not store any credit card or other personal financial information on my server. All payments are through PayPal. They hold all your financial information on their servers.

  16. Dewald Pretorius

    And as far as your missing @ replies go, I can only give what Twitter Search gives to my system when it does a search for your @replies. If Twitter Search for some reason misses some of your @replies, then they will be missing in the @replies I show you. I can give you no more and no less than what Search returns.

  19. Rebecca

    THANK YOU for your detailed responses…they are enlightening and very assuring! The explanation of how Twitter search plays into your @ filter is also helpful.Like I said, wasn't trying to be harsh, just seemed like the service was acting up before this and when you have lots of folks relying on the service (a tribute to the service you created) it becomes even more important when they are paying for the service…that's all.

  20. iSenne

    Nice update,Glad things are almost done :).Maybe a strange question, but i am currently using my week trial form the pro version of tweetlater. Is it possible that i can get an extra day?

  21. Dewald Pretorius

    Yes, absolutely. I will extend everyone's free trial by one additional day. As soon as I can get HostGator to correctly configure PHP. They have missed a few essential libraries, which means my software is currently crashing.

  22. Scott

    Dewald, you are a true trooper. You have not hidden in time of distress. Many other services that might be in your not enviable position in the future would not have gone to the trouble you have with keeping all of us Tweepers in the loop. Hats off– As the old saying goes “Stuff Happens!” You have dealt with your stuff in a very professional and courteous manner. I am a pro customer and give my following “explosion” credit to your software and the ability to schedule multiple tweets in the future.Good Job,Scott

  23. JulReDesigns

    Thanks for your thorough updates. As always they are much appreciated. Love your service.

  24. ShamnaLachmo

    Hi, will the service dm and autofollow all the people that have followed me during the down time ?

  29. Nemo

    Thanks for the constant updates and getting this back so quickly. Really an amazing job at, not only fixing it, but keeping your users informed.Cheers

Comments are closed.