Friday, April 17, 7:44 PM EST.
If you’re seeing a lot of entries in TweetCockpit that say the Twitter API returned zero tweets, or if you have trouble with the Vetting page, or your tweets are not being published on time or not at all, or if you’re getting a message on the Twitter account add or edit page about invalid Twitter credentials, please note the following.
Both the Twitter API and the Twitter Search API seem to be very unresponsive at the moment.
We assume it is a fallout of the new Mikeyy worm attack that struck today.
It is not a problem in the TweetLater system. It’s just that we’re not getting data back from the two APIs.
We’re even seeing it on the batch jobs that run on two different servers. They have all slowed down to an absolute crawl, and are getting tons of fail whale errors back from Twitter.
Update April 17 9:14 PM EST — Just heard back from Alex Payne at Twitter. The Twitter system is completely overloaded at the moment. It was featured on Oprah earlier today, and is right now on Larry King. That’s why the APIs are not responding.
Update April 18 6:42 AM EST — The APIs are still extremely unresponsive. Some of the API calls are going through, but the majority are still getting fail whales.
Update April 18 12:22 PM EST — As noted in the Comments, I am waiting to hear back from Twitter. Earlier this morning I have asked them to check whether their spam team has perhaps put a block on my IP address. There is a remote possibility that someone could have scheduled tweets that the spam team considers as spam. They are not supposed to block my server IP address in that case, because it does not make sense to inconvenience 40,000+ people because of some stupid thing one person did. However, I have asked them to check whether that is the case. Personally I do not think this is the case, because some of the API calls are going through. There is not a complete blackout of calls from my server IP address. With a spam block the blackout is usually 100%, i.e., no API calls get through. Twitter is on PST, so I expect to hear back from them later this afternoon.
Update April 18 7:20 PM EST — Have not heard anything back from Twitter. It is extremely frustrating. However, let us remember that it is weekend.
Update April 18 8:27 PM EST — It appears as if the Twitter folks are still combating the new Mikeyy worm. Alex Payne and Matt Sanford of the API team are usually very responsive to my requests for assistance. I’m sure I will hear back from them the moment they are able to stop working though code and can check their emails. I’m convinced that in the Twitter offices combating the worm takes priority over everything else.
Update April 19 6:56 AM EST — The problem still persists, and Twitter has not responded yet.
Update April 19 10:36 AM EST — My head is now really very sore from repeatedly banging it against the wall. We should probably all resign ourselves that this issue will only be resolved on Monday during PST business hours. My sincere apologies to more than 44,000+ of my users who have been inconvenienced by this issue. If there were anything more I could do to have it fixed I would have done it days ago. ^DP
Update April 19 7:00 PM EST — Nothing new to report yet, except that the bruise on my head is now turning purple and black around the edges. For a third-party application like TweetLater to have an API issue, which essentially cripples the service for several days, with no support, is really not an acceptable situation. Even though it’s not done maliciously, it is an issue that Twitter will have to address. ^DP
Update April 19 9:05 PM EST — I saw Alex Payne, head of the Twitter API team, online and tweeted him. He has just replied with, “Got ’em [my emails]. We should have it sorted for you tomorrow [Monday].”
Update April 19 10:11 PM EST — You may notice a slight improvement in TweetCockpit, adding and editing Twitter accounts, and Vetting. The API issue still exists, but I thought of a crude work-around, which I’ve now implemented. It may still be spotty, meaning it may work only intermittently. At least it’s better than nothing.
Update April 20 3:36 PM EST — The issue has now been resolved. TweetLater was unfortunately caught in the cross-fire between Twitter and spammers. With the latest round of spammers and worms that have hit Twitter, they have severely tightened their API behavior analysis algorithms. Due to the very high volume of Twitter API calls that TweetLater makes around the clock (we make several calls per second), one of the spam traps in the tightened algorithms was unfortunately triggered. As they say, **it happens. I have added additional code on my side to try and prevent this from again happening in the future. I will extend trial periods before end of day today. Follower processing will, over the next 8 hours, resume where it left off on Friday. ^DP
Update April 20 5:16 PM EST — A further update regarding follower processing. The normal 8-hour cycle of follower processing has resumed. Your account will be processed within the next 8 hours, and all the followers that followed you since Friday will be picked up. However, I’m seeing that the API is slow, which could be just because it is very busy or because the unblock is still filtering through all of Twitter’s networking equipment. The API usually speeds up late at night when the load is not so heavy. I’m expecting that things will return to normal within the next 24 hours. By then the TweetLater system should have caught up with the backlog that has built up. ^DP
The problems you’re having with TweetLater will automatically go away the moment that the Twitter API and Twitter Search API again start responding.
The trial periods of folks using the TweetLater Professional trial will be extended as soon as the issue is resolved. These extensions will be have been done between 5:20 PM EST and 5:50 PM EST.
Update April 20 9:57 PM EST — This issue is now officially resolved and closed. Follower processing has a HUGE backlog and will take 24 to 48 hours to catch up. At the time of writing there were 1.7 million unprocessed API calls on the processing queue, and the queue is still growing every second.
I want to express my sincere thanks to the Twitter API team, and to Matt Sanford in particular, for the efficient and professional help in getting this issue resolved.
Update April 21 10:38 AM EST — I have now employed a second server, which will process new API calls from the top of the processing queue. Hence, you may find that new followers of today are processed before folks who followed you during the outage. The outage has created a backlog of nearly 4 million API calls and it will take the system a few days to work through that backlog. ^DP
Update April 21 5:33 PM EST — Working through the backlog is going smoothly. There are now only 2.3 million API calls left. Piece of cake. ^DP
Update April 22 10:30 AM EST — It is official. The system has now finished working through the backlog of nearly 4 million API calls. We are now fully caught up, and just normal processing will take place from now on. At various points during the catch-up process, we were making in excess of 30 calls per second to the Twitter API, and the API handled it without blinking an eye.
SocialOomph Blog 
This is really annoying. Almost every day Twitter is overloaded. If people there won't address this issue soon, a lot of people might leave…
Any current time frame as to when the system may clear as the API's still seem to be down.
Can we have an update please ?Tweetdeck , twhirl , tweetie are all working .Presumably you all use the same Twitter API ?StevenTweetlater Professional User
The problems you’re having with TweetLater will automatically go away the moment that the Twitter API and Twitter Search API again start responding.ANY IDEA ON THE UPDATED TIME FRAME?
I am waiting to hear back from Twitter. Earlier this morning I have asked them to check whether their spam team has perhaps put a block on my IP address. There is a remote possibility that someone could have scheduled tweets that the spam team considers as spam. They are not supposed to block my server IP address in that case, because it does not make sense to inconvenience 40,000+ people because of some stupid thing one person did. However, I have asked them to check whether that is the case.Twitter is on PST, so I expect to hear back from them later this afternoon.
Unfortunately I do not know. The issue is on the Twitter side of the fence.
Hi DewaldMany thanks for your timely reply ..Tweetlater is an amazing service which I am happy to promote on twitter andto the networks I belong to.I went Pro , before I found the problem , I now have a second account for@betternet which will make extensive use of tweetlater services.I was already impressed by the services I have been using.RegardsSteven2009/4/18 Disqus <>
Shitty. I subscribed to the newsletter. Maybe you can email us when this issue is resolved. I have been unable to add my twitter account.
Please monitor this post for updates. I wouldn't want to send service announcements to the newsletter subscribers. It will annoy a good percentage of the people.
Hi Dewald,Thanks for the updates. Hoping this problem is resolved today as I am anxiously awaiting to begin using this. How will we know when it is finally back up and when we can begin to use it?~Kimberly
I will update this post when the issue has been resolved.
Thanks, I'll be ready when you are ;)~Kimberly
Thanks for your updates. The scheduled tweets post, but the vetting is totally down. I have quite a list of followers that is backing up. Hope Twitter gets it fixed this afternoon. You know what the squeaky wheel does…….
I think what will frustrate most people is that TweetDeck and some other apps are working. But from what I have read elsewhere it may be the Mickeyy Worm that is causing these problems.
See Twitter's blog on their status: http://status.twitter.com/
I notice they posted an update on the status blog today. What that tells me is the Twitter folks are probably too busy with the worm issue to even look at their emails.
We love TweetLater. We're with ya!
I am using Twitter API to update my Twitter from my blog brilliantspot.com. The API was working all day and night. My guess is that Twitter is overloaded and it is just cannot digest bulk updates.
I can't add my twitter account at allWill come back later
Latest news was 6 hours ago. Still waiting to add my account.
I'm trying to add my twitter account and always get the message that the username/password are inccorrect. is it because of the problem you've described?
Yes, it is because of this problem. The system cannot connect to the API in the background to validate your credentials.
I just found out about this and still trying to add my twitter account. Maybe because it's sunday. I'l probably try back again tomorrow and hopefully i should get some joy.
Cheers for the valuable updates
Thanks for keeping us posted. In same boat trying to register new account and having no success. This often feels like user error, but for once, it's someone else's fault. 😉 Well keep watching the blog. ~Chris
I was starting to wonder I was only told about this site yesterday and haven't been able to log in as a new account – I thought I would snoop around to find out what was so – glad I found this thread
Dewald, Awesome work and thanks for your updates . No problem here. I would suggest any new users to hold tight , Tweetlater is the cream of the crop. Your Twitter life will much better and easier with Tweetlater than without Dave
Dave, thank you very much. I deeply appreciate your support.
Hi the autoreply feature is not working. Is this the same api problem?
I have had to change my password on twitter; now tweetlater will not work with the new password when i try to update my auto tweets. Is that also a part of the current unpleasantness ?
Yes, it's the same problem.
Yes indeed.
I'm confused – does Twitter have multiple APIs? My code base is able to send tweets accross the pipe with my credentials. Just curious?
Twitter has only one API. I suspect my IP addresses got caught in a spam block because one of my users did something stupid. Haven't confirmed that yet, but that's what I think it is.
It's not your fault!
Thanks for explaining this. I'll be patient.
Hang in there. This can't last forever. Thanks for your hard work. Stop banning your head man!
Curious – how does your auto-message system work? Do you pull down a users followers based on some timestamp? Or do you maintain a db or users that got DMed from a user to know who got a Dm already?
I use autofollower – and I have many pending the fix. So many that if they kick in I'll surpass the 10% rule of being limited to following a total of 10% more that follow you (if you are over 2000 followers). However I also have many that I am following who, vice versa, will probably be following me automatically also.Any idea of what will happen? Or is it purely a statistical luck of the draw? Thanks for a great service and your obvious care for your users – both paying and non-paying.BESTESTDavid
Unfortunately that's proprietary information, which I do not disclose.
When you exceed the Twitter limits, Twitter simply prevents you from following more people until such time that you are within the limits again.
Thanks for keep us up to date with the latest status.
Understood – does that mean that the autofollower will not work once that limit hits – and that I will have to do the follows manually, if I choose?Thanks again!
When you hit the limit you cannot do any follows, automatically or manually.
I agree. They really need to scale their hardware infrastructure. In fact, knowing that they were going to be on Oprah and Larry King along with the whole @aplusk they should have already increased their systems in anticipation of increased load. If something doesn't improve soon (we've been having fail whales for way too long), then it's going to be prime for a competitor to sweep up the market.
I know Twitter was recently furiously busy with adding more hardware. It’s really tough to fully gear up for an avalanche of traffic, because you never know how much is enough.
Twitter is so far ahead of any potential competitor in terms of marketshare, scaling, architecture, etc., that I don’t think they have anything to worry about for a long time. It will take serious investment and a lot of time from someone else to even see Twitter’s dust in the road ahead of them.
Thanks – what I am wondering, is that when I go below the limit again (due to more following me or I stop following some) will the auto-follow play catch-up with those of my followers who I am not following in the interem, or do I have to follow them manually? In other words, does it act similar to when I first signed up that it only follows automatically those from that time on, or as I was registered it knows to auto-follow those it missed due to the limit?Sorry for the convaluted question! And thanks again for your gracious help and explanation!BESTESTDavid
Just wanna say thanks to the Tweetlater team for your tireless effort in trying to make things work for us. You're awesome!
Thank you. I appreciate your thanks and feedback.
Congratulations for the calm and professional way you dealt with this (though I suspect there was a bit of the swan thing going on – paddling madlyunder water whilst appearing serene on the surface!)
DITTO! Your hard work and professionalism are much appreciated!BESTESTDavid