Purge Your Direct Message Inbox with TweetLater

Tired of manually deleting hundreds of direct messages from your DM Inbox?

No need to do that anymore.

With TweetLater you can now run a purge on your DM Inbox, and delete all DMs, DMs older than 7 days, or DMs older than 30 days.

The feature is available to TweetLater Free and TweetLater Professional users.

To access it, login to TweetLater, click the blue DM menu tab in the top menu bar, and then the Purge DMs sub-menu tab.

TweetLater Adds Authenticated Bit.ly URL Shortening

Up to now we have used several URL shortening services that provided you with no means to track the clicks on the links in your scheduled tweets and DMs.

We have now added bit.ly as a URL shortening option. All you need to do is add your bit.ly login and API key to your TweetLater profile. From that point forward we will shorten all your links using your bit.ly account. We won’t use any of the other URL shortening services.

This means you can track the clicks on all your links in your bit.ly account.

To activate bit.ly on your TweetLater profile, do the following:

  1. Login to your TweetLater account.
  2. Click the “Edit details” button in the sidebar on the right.
  3. Enter your bit.ly login and API key in the appropriate fields.
  4. Click the Save button.

It’s as easy as that. From that point forward all your links will be shortened using bit.ly.

This will also soon include also includes links on tweets created by your Blog Feeds, if you are a TweetLater Professional user. We have not yet now added it to the blog feeds processor.

The same two bit.ly fields are also available on the registration page, for folks who are registering new TweetLater accounts.

If you don’t have a bit.ly account, it’s super easy to register a free account at bit.ly.

If you don’t know where to get your bit.ly API key, login to your bit.ly account and click the “account” link in the top right corner of the page. Your API key is displayed on that page. Your “login” is the username that you use to login to bit.ly.

Why TweetLater Doesn't Offer Catch-Up Or Friends/Follower Sync Features

We often get requests from users who want to know if we can run a script on their Twitter accounts that will go back in time and follow everyone that follows them, and unfollow everyone that does not follow them. In essence, sync-up their Friends with their Followers.

Here’s why we don’t offer that feature. It’s important to read this so that you don’t get your Twitter account into trouble.

I recently discovered that our @tweetlater account was in “restricted” status in the Twitter system. Everything appeared to be working normally on the account, except that its tweets were not being indexed by Twitter Search. I thought it was a bug in Twitter Search, and when I submitted an API support ticket, I was informed that it was because @tweetlater was in “restricted” status.

Until recently @tweetlater was not following anyone, except @spam. Folks were raising questions about why @tweetlater was not following anyone. So, I decided to run a sync script that simply took the list of existing followers, and followed everyone of them back. In other words, @tweetlater’s Friends count went from 1 to 6,000+ in a matter of 30 minutes or so. This was before Twitter implemented the “you can only follow 1,000 new people per day” rule.

At no time during that sync process did @tweetlater follow anyone that was not already a follower, and at no time did the Friends count exceed the Followers count.

And yet, this appeared to have flagged the @tweetlater account for “aggressive follow behavior”.

I might be wrong, but in my opinion this was a completely legitimate exercise and action on my @tweetlater account. Apparently the Twitter system did not think so.

The moral of this story is the following.

Think two or three times before you do any kind of sync or catch-up on your Twitter account. It’s more than likely to put your Twitter account in jeopardy.

I have been holding back on adding a sync feature to TweetLater because a few weeks ago I sent an email to Twitter asking whether it would cause trouble. The last thing I want to do is offer a feature that will get people’s Twitter accounts suspended.

I still have not received a reply to that email, but this issue with @tweetlater has basically confirmed my suspicions and vindicated my caution.

^DP

Update April 22 9:00 PM EST — This matter has now been resolved. I want to express my gratitude and thanks to the Twitter team. I am truly amazed at their responsiveness and helpfulness, despite the fact that their workload must warrant 48 hours in a day.

Problems Getting Data From Twitter

Friday, April 17, 7:44 PM EST.

If you’re seeing a lot of entries in TweetCockpit that say the Twitter API returned zero tweets, or if you have trouble with the Vetting page, or your tweets are not being published on time or not at all, or if you’re getting a message on the Twitter account add or edit page about invalid Twitter credentials, please note the following.

Both the Twitter API and the Twitter Search API seem to be very unresponsive at the moment.

We assume it is a fallout of the new Mikeyy worm attack that struck today.

It is not a problem in the TweetLater system. It’s just that we’re not getting data back from the two APIs.

We’re even seeing it on the batch jobs that run on two different servers. They have all slowed down to an absolute crawl, and are getting tons of fail whale errors back from Twitter.

Update April 17 9:14 PM EST — Just heard back from Alex Payne at Twitter. The Twitter system is completely overloaded at the moment. It was featured on Oprah earlier today, and is right now on Larry King. That’s why the APIs are not responding.

Update April 18 6:42 AM EST — The APIs are still extremely unresponsive. Some of the API calls are going through, but the majority are still getting fail whales.

Update April 18 12:22 PM EST — As noted in the Comments, I am waiting to hear back from Twitter. Earlier this morning I have asked them to check whether their spam team has perhaps put a block on my IP address. There is a remote possibility that someone could have scheduled tweets that the spam team considers as spam. They are not supposed to block my server IP address in that case, because it does not make sense to inconvenience 40,000+ people because of some stupid thing one person did. However, I have asked them to check whether that is the case. Personally I do not think this is the case, because some of the API calls are going through. There is not a complete blackout of calls from my server IP address. With a spam block the blackout is usually 100%, i.e., no API calls get through. Twitter is on PST, so I expect to hear back from them later this afternoon.

Update April 18 7:20 PM EST — Have not heard anything back from Twitter. It is extremely frustrating. However, let us remember that it is weekend.

Update April 18 8:27 PM EST — It appears as if the Twitter folks are still combating the new Mikeyy worm. Alex Payne and Matt Sanford of the API team are usually very responsive to my requests for assistance. I’m sure I will hear back from them the moment they are able to stop working though code and can check their emails. I’m convinced that in the Twitter offices combating the worm takes priority over everything else.

Update April 19 6:56 AM EST — The problem still persists, and Twitter has not responded yet.

Update April 19 10:36 AM EST — My head is now really very sore from repeatedly banging it against the wall. We should probably all resign ourselves that this issue will only be resolved on Monday during PST business hours. My sincere apologies to more than 44,000+ of my users who have been inconvenienced by this issue. If there were anything more I could do to have it fixed I would have done it days ago. ^DP

Update April 19 7:00 PM EST — Nothing new to report yet, except that the bruise on my head is now turning purple and black around the edges. For a third-party application like TweetLater to have an API issue, which essentially cripples the service for several days, with no support, is really not an acceptable situation. Even though it’s not done maliciously, it is an issue that Twitter will have to address. ^DP

Update April 19 9:05 PM EST — I saw Alex Payne, head of the Twitter API team, online and tweeted him. He has just replied with, “Got ’em [my emails]. We should have it sorted for you tomorrow [Monday].”

Update April 19 10:11 PM EST — You may notice a slight improvement in TweetCockpit, adding and editing Twitter accounts, and Vetting. The API issue still exists, but I thought of a crude work-around, which I’ve now implemented. It may still be spotty, meaning it may work only intermittently. At least it’s better than nothing.

Update April 20 3:36 PM EST — The issue has now been resolved. TweetLater was unfortunately caught in the cross-fire between Twitter and spammers. With the latest round of spammers and worms that have hit Twitter, they have severely tightened their API behavior analysis algorithms. Due to the very high volume of Twitter API calls that TweetLater makes around the clock (we make several calls per second), one of the spam traps in the tightened algorithms was unfortunately triggered. As they say, **it happens. I have added additional code on my side to try and prevent this from again happening in the future. I will extend trial periods before end of day today. Follower processing will, over the next 8 hours, resume where it left off on Friday. ^DP

Update April 20 5:16 PM EST — A further update regarding follower processing. The normal 8-hour cycle of follower processing has resumed. Your account will be processed within the next 8 hours, and all the followers that followed you since Friday will be picked up. However, I’m seeing that the API is slow, which could be just because it is very busy or because the unblock is still filtering through all of Twitter’s networking equipment. The API usually speeds up late at night when the load is not so heavy. I’m expecting that things will return to normal within the next 24 hours. By then the TweetLater system should have caught up with the backlog that has built up. ^DP

The problems you’re having with TweetLater will automatically go away the moment that the Twitter API and Twitter Search API again start responding.

The trial periods of folks using the TweetLater Professional trial will be extended as soon as the issue is resolved. These extensions will be have been done between 5:20 PM EST and 5:50 PM EST.

Update April 20 9:57 PM EST — This issue is now officially resolved and closed. Follower processing has a HUGE backlog and will take 24 to 48 hours to catch up. At the time of writing there were 1.7 million unprocessed API calls on the processing queue, and the queue is still growing every second.

I want to express my sincere thanks to the Twitter API team, and to Matt Sanford in particular, for the efficient and professional help in getting this issue resolved.

Update April 21 10:38 AM EST — I have now employed a second server, which will process new API calls from the top of the processing queue. Hence, you may find that new followers of today are processed before folks who followed you during the outage. The outage has created a backlog of nearly 4 million API calls and it will take the system a few days to work through that backlog. ^DP

Update April 21 5:33 PM EST — Working through the backlog is going smoothly. There are now only 2.3 million API calls left. Piece of cake. ^DP

Update April 22 10:30 AM EST — It is official. The system has now finished working through the backlog of nearly 4 million API calls. We are now fully caught up, and just normal processing will take place from now on. At various points during the catch-up process, we were making in excess of 30 calls per second to the Twitter API, and the API handled it without blinking an eye.

Server Downtime: The Impact

The TweetLater website was unavailable for roughly 20.5 hours, from around 4:00 PM EST on April 16, 2009 to 12:30 PM EST on April 17, 2009.

Scheduled Tweets

All scheduled tweets that were supposed to have been published during the downtime have been placed into an error condition. They will not be automatically published. You need to review each one and decide if you want to reschedule it, or if you want to delete it.

Follower Processing

Follower processing has resumed where it stopped when the server went down. In other words, followers who have followed you during the downtime will be processed. It will take roughly 8 hours from 12:30 PM EST to cycle through all the Twitter accounts.

Profile Banners

We have already sent credits to the two people whose profile banners were affected by the downtime.

TweetLater Professional Subscriptions

All payment information sent by PayPal to our system during the downtime have been manually applied to the user accounts.

If any TweetLater Professional customer requires a partial refund for the downtime, we will be more than happy to oblige. On payments of $19.97 the credit is $0.57, and on payments of $29.97 the credit is $0.85. Please submit a support ticket and let us know if you want a credit.

TweetLater Professional Trial Members

We added two additional days to all TweetLater Professional trials that were active when the server went down.

Lessons Learned

  1. Communication under these circumstances is vital. We’re blown away by the understanding and support that folks expressed on the other blog post that gave status updates during the downtime.
  2. Hosting your blog and support system on a different server is very important, so that they are available even when your main site is down. This is something we put in place a little more than a week ago (thankfully).
  3. We’re just again reminded that we’re working with a machine that can break. These server disks normally have a life span of 50,000 power on hours. However, just like any other piece of machinery they can break.
  4. An attribute of crap is that it usually hits the fan at the most inconvenient times.

Server Issues 16 April, 8:00 PM EST

The TweetLater.com server is currently not responding.

We’re working on getting it back online as soon as possible.

If you happen to get to the login form, and it says your credentials are invalid, it’s because the database is down.

We will update this message when service has been restored.

Update 9:24 PM EST — The faulty drive is being replaced on the server. Unfortunately it was the primary drive, which holds the operating system. The OS needs to be reloaded, which they tell us will take between 6 and 8 hours.

Update 10:51 PM EST — The server rebuild is being started now. The database has been backed up, so there should not be any data loss. Follower processing and scheduled tweets are currently in mid-air suspension. When we reactivate the batch jobs once the server is rebuilt, follower processing will resume. All unpublished scheduled tweets and DMs that should have been published during the downtime will be published at that point. You may experience a sudden publishing of several tweets at the same time in your timeline if you had several tweets that should have been published during the downtime.

Update 11:04 PM EST –The hosting folks say that the server rebuild will take 6 to 8 hours. We’re going to assume a horizontal position for a few hours. When we’re awake and fresh, we’ll check that everything has been correctly configured on the server before we open TweetLater.com for logins. Will update here again in the morning.

Update 5:08 AM EST — The server has been rebuilt, and the backups are currently being restored.

Update 6:55 AM EST — Backups have been restored. There has been no data loss, apart from a few Keyword Alert archives that were corrupted beyond repair. It shouldn’t be an issue because those archives are only kept for 24 hours anyway. There are a few more configuration tasks left to do. It will probably be another hour, maybe two.

Update 7:02 AM EST — After doing an analysis of the scheduled tweets that should have been published during the downtime, we felt it would be prudent to put those tweets into an error condition so that they do not publish, and that you can decide whether you want to delete them or reschedule them. There are many Twitter accounts that will have a small avalanche of tweets published on them all at once if we just let the system publish the tweets. We’ll make a separate blog post regarding these tweets as soon as the system goes live.

Update 8:33 AM EST — It’s still going to be a while. Apache is still recompiling. Once that’s done, there are still a few configuration tasks to perform.

Update 9:14 AM EST — Apache recompile has finished. Now doing the last configuration tasks. It won’t be much longer.

Update 9:55 AM EST — Nearly there. Some of the batch jobs are already running, so you will see scheduled tweets being published. The site is still down because MEMCACHE is not yet running, and without it the site simply unceremonially crashes.

Update 11:01 AM EST — The hosting provider technicians forgot to compile several essential libraries into PHP. Just waiting for them to do that, then the site should come back online.

Update 12:01 PM EST — Just learned an important lesson. HostGator works on the tickets that have the oldest customer reply time first. When you keep submitting replies, such as “Please work on this, it is really important,” you bump your own ticket just further down the queue because the reply is new. So we learn a new thing every day.

Update 12:30 PM EST — And…….

WE ARE BACK!!!!

Modified TweetLater Site Header Graphic

The big site header graphic at the top of each TweetLater page has been bugging us for a while, because it was taking up so much screen real-estate.

The site header has now been reduced in height, which allows you to see more above the fold on every page of the site.

The font used in the header has also been changed. The new font, we believe, looks a little more “professional” than the old one.

Rookie Mistake Amuses Village Children

Last night a real rookie mistake with regex (regular expressions) caused some of our follower snapshots to become corrupted. Really badly corrupted.

The system is currently rebuilding all follower snapshots from scratch (April 10th, 2009 9:00 AM EST). This process will run until 12:00 PM EST.

The nett effect is that folks that followed you before 12:00 PM EST will not be processed. In other words, they will not be auto-followed and they will not receive your DM.

After 12:00 PM follower processing will resume as usual.

The rookie has been sent to stand in the village square so that all the children can ridicule him.

Friends/Followers Counts Disabled

For the time being we have disabled the Friends/Followers counts that were displayed for each Twitter account on the List Accounts page.

The Twitter API was returning 0 Friends for certain accounts, which made the display rather useless, and the API has been extremely slow the past few days.

The slowness in the API resulted in the List Accounts page being very slow to load.

Once the API is back to its usual speedy self, we will re-enable the display of those counts.