Friday, April 17, 7:44 PM EST.

If you’re seeing a lot of entries in TweetCockpit that say the Twitter API returned zero tweets, or if you have trouble with the Vetting page, or your tweets are not being published on time or not at all, or if you’re getting a message on the Twitter account add or edit page about invalid Twitter credentials, please note the following.

Both the Twitter API and the Twitter Search API seem to be very unresponsive at the moment.

We assume it is a fallout of the new Mikeyy worm attack that struck today.

It is not a problem in the TweetLater system. It’s just that we’re not getting data back from the two APIs.

We’re even seeing it on the batch jobs that run on two different servers. They have all slowed down to an absolute crawl, and are getting tons of fail whale errors back from Twitter.

Update April 17 9:14 PM EST — Just heard back from Alex Payne at Twitter. The Twitter system is completely overloaded at the moment. It was featured on Oprah earlier today, and is right now on Larry King. That’s why the APIs are not responding.

Update April 18 6:42 AM EST — The APIs are still extremely unresponsive. Some of the API calls are going through, but the majority are still getting fail whales.

Update April 18 12:22 PM EST — As noted in the Comments, I am waiting to hear back from Twitter. Earlier this morning I have asked them to check whether their spam team has perhaps put a block on my IP address. There is a remote possibility that someone could have scheduled tweets that the spam team considers as spam. They are not supposed to block my server IP address in that case, because it does not make sense to inconvenience 40,000+ people because of some stupid thing one person did. However, I have asked them to check whether that is the case. Personally I do not think this is the case, because some of the API calls are going through. There is not a complete blackout of calls from my server IP address. With a spam block the blackout is usually 100%, i.e., no API calls get through. Twitter is on PST, so I expect to hear back from them later this afternoon.

Update April 18 7:20 PM EST — Have not heard anything back from Twitter. It is extremely frustrating. However, let us remember that it is weekend.

Update April 18 8:27 PM EST — It appears as if the Twitter folks are still combating the new Mikeyy worm. Alex Payne and Matt Sanford of the API team are usually very responsive to my requests for assistance. I’m sure I will hear back from them the moment they are able to stop working though code and can check their emails. I’m convinced that in the Twitter offices combating the worm takes priority over everything else.

Update April 19 6:56 AM EST — The problem still persists, and Twitter has not responded yet.

Update April 19 10:36 AM EST — My head is now really very sore from repeatedly banging it against the wall. We should probably all resign ourselves that this issue will only be resolved on Monday during PST business hours. My sincere apologies to more than 44,000+ of my users who have been inconvenienced by this issue. If there were anything more I could do to have it fixed I would have done it days ago. ^DP

Update April 19 7:00 PM EST — Nothing new to report yet, except that the bruise on my head is now turning purple and black around the edges. For a third-party application like TweetLater to have an API issue, which essentially cripples the service for several days, with no support, is really not an acceptable situation. Even though it’s not done maliciously, it is an issue that Twitter will have to address. ^DP

Update April 19 9:05 PM EST — I saw Alex Payne, head of the Twitter API team, online and tweeted him. He has just replied with, “Got ’em [my emails]. We should have it sorted for you tomorrow [Monday].”

Update April 19 10:11 PM EST — You may notice a slight improvement in TweetCockpit, adding and editing Twitter accounts, and Vetting. The API issue still exists, but I thought of a crude work-around, which I’ve now implemented. It may still be spotty, meaning it may work only intermittently. At least it’s better than nothing.

Update April 20 3:36 PM EST — The issue has now been resolved. TweetLater was unfortunately caught in the cross-fire between Twitter and spammers. With the latest round of spammers and worms that have hit Twitter, they have severely tightened their API behavior analysis algorithms. Due to the very high volume of Twitter API calls that TweetLater makes around the clock (we make several calls per second), one of the spam traps in the tightened algorithms was unfortunately triggered. As they say, **it happens. I have added additional code on my side to try and prevent this from again happening in the future. I will extend trial periods before end of day today. Follower processing will, over the next 8 hours, resume where it left off on Friday. ^DP

Update April 20 5:16 PM EST — A further update regarding follower processing. The normal 8-hour cycle of follower processing has resumed. Your account will be processed within the next 8 hours, and all the followers that followed you since Friday will be picked up. However, I’m seeing that the API is slow, which could be just because it is very busy or because the unblock is still filtering through all of Twitter’s networking equipment. The API usually speeds up late at night when the load is not so heavy. I’m expecting that things will return to normal within the next 24 hours. By then the TweetLater system should have caught up with the backlog that has built up. ^DP

The problems you’re having with TweetLater will automatically go away the moment that the Twitter API and Twitter Search API again start responding.

The trial periods of folks using the TweetLater Professional trial will be extended as soon as the issue is resolved. These extensions will be have been done between 5:20 PM EST and 5:50 PM EST.

Update April 20 9:57 PM EST — This issue is now officially resolved and closed. Follower processing has a HUGE backlog and will take 24 to 48 hours to catch up. At the time of writing there were 1.7 million unprocessed API calls on the processing queue, and the queue is still growing every second.

I want to express my sincere thanks to the Twitter API team, and to Matt Sanford in particular, for the efficient and professional help in getting this issue resolved.

Update April 21 10:38 AM EST — I have now employed a second server, which will process new API calls from the top of the processing queue. Hence, you may find that new followers of today are processed before folks who followed you during the outage. The outage has created a backlog of nearly 4 million API calls and it will take the system a few days to work through that backlog. ^DP

Update April 21 5:33 PM EST — Working through the backlog is going smoothly. There are now only 2.3 million API calls left. Piece of cake. ^DP

Update April 22 10:30 AM EST — It is official. The system has now finished working through the backlog of nearly 4 million API calls. We are now fully caught up, and just normal processing will take place from now on. At various points during the catch-up process, we were making in excess of 30 calls per second to the Twitter API, and the API handled it without blinking an eye.