Twitter Arbitrarily “Filters” Search: Big Brother is Here

by Karoli on July 13, 2009 · 42 comments

At Friday’s Techcrunch Real Time Conference (crunchup), there was a panel on real time search. Several of the commentators argued that real time search was useless without filters and algorithms to refine the search. Philosophically, that approach is similar to Dick Cheney saying it’s okay to wiretap ordinary citizens without any accountability or disclosure because Dick Cheney said so. What these panelists were arguing for was subjective filtering of what people post to social networks based upon what they view as important, relevant, and worth including.

Twitter seems to be leading the charge on this approach. Over the past few weeks, many of the people I follow closely have been unpleasantly surprised when they try a search for posts made from their username (e.g. FROM:karoli). Today was appalling. Many people I consider to be interesting and genuine were removed from search. Actually, according to @delbius, the Twitter spam monitor and public voice, they were ‘filtered’.

liz ditz profileOne of those ‘filtered’ was my friend Liz Ditz. She was subsequently restored to search, but as you’ll see, the restoration was only for posts after she appealed to Twitter for reinstatement. All of her old messages are gone, evidently for good. Here’s screen captures illustrating the point:

Note that Liz has over 5140 posts to twitter. She follows less than follow her, and in any event the follow ratio is well within acceptable bounds. Here are her search results, captured 10 minutes ago:

That’s it. That’s all of her search results. Over 5,000 tweets and one single page of results. Yes, Twitter “restored” her account, but nothing before 4pm can be accessed via search.

Here’s another example, even more chilling. @zorinsmom tweets information about Iran. It is her mission, as stated on her Twitter profile. She has posted over 3,000 updates. She is focused on the Iran election, and nearly everything she posts is an effort to relay information that promotes a free Iran. A noble mission, right? Have a look at her search result:

That’s right. No results. If you search on the hashtag #IranElection and zorinsmom, you get updates directed TO her but nothing FROM her.

Zorinsmom has been ‘filtered’. Despite her best efforts to get information out onto the network, she has been caught in the net that is Twitter’s subjective algorithm for filtering search results in such a way that certain people are simply….gone.

Imagine the power plays here. Imagine the ability to shape search by purely subjective standards to shape the flow of information. Go ahead and accuse me of being paranoid, and then look at Zorinsmom’s timeline. Or Liz Ditz’s timeline. Both of them are concerned citizens using Twitter to get a message out. In Liz’s case, it’s about autism. In ZorinsMom’s case, it’s about the Iran elections. In BOTH cases, they are legitimate, concerned, engaged users who are hardly spammers. One of my most trusted favorites, RoadKillRefugee was also ‘filtered’ (only 3 pages of search results for an account that has some of the most reliable and interesting links around the political scene there are…thanks, twitter.).

By comparison, try a search on “400 free followers” and take a look at the results. I picked the top search result to see whether or not she came up in search. She did.

So Twitter has decided that spammy promos for ‘free followers’ (as if anyone actually pays for them?) is fine to leave in search, but legitimate, concerned citizen activists are not? It’s been clear for awhile that Twitter’s business model involves high profile celebrity, including the requisite appearances on Oprah, Conan O’Brien and other shows, but the decision-making process at work here is simply outrageous.

Further, they don’t seem to understand what community is on Twitter. They don’t understand the value of trustworthy links being spread through the network. They obviously have no concern for the value of the one-to-many model when it comes to sharing relevant and important information about really important issues. Twitter’s concern is whether Ashton Kutcher has a million followers or meeting Warren Buffet.

What they have done is take an inherently democratic model and turned it into an autocracy. The standard response is that a link has been posted that Google has identified as ‘malware’. I clicked through as many archival links as I could on these accounts and could find exactly zero links that Google flagged as malware. Not even one. But even if I did find one, shouldn’t users receive some sort of warning that they’ve posted a link rather than simply being ERASED, obliterated, and sent to Twitter prison?

This is a serious enough issue to me that I am seriously considering posting from identi.ca and bridging to twitter. I see absolutely no excuse for wielding power over people like they are in such an arbitrary and capricious fashion, with no appeal and no recourse for past updates. They’re just gone. Links and information that are of value. I’m not talking about the “I had fish for lunch” updates. I’m talking about updates that really ARE valuable. Roadkillrefugee had posted a series of tweets reporting the contents of the newly released report on Cheney’s involvement with illegal wiretapping. Liz Ditz was posting updates with important autism activism information. Zorinsmom was fighting for Iran and fair elections. Yet, in each case, Twitter arbitrarily ‘filtered’ their voice.

We need tools so we can get rid of asshats posting crap about free followers, not so we can silence voices using these tools for good, not evil. More importantly, decisions about what is filtered should not come from the top down; rather, they should come from us, the user, deciding what WE think is relevant in a search.

Shoq (who has been ‘filtered’ more than once) has written a complete step-by-step procedure for checking your own Twitter result and then taking steps to fix it if you’ve been the victim of their ‘filter’. I highly recommend bookmarking it. Chances are, you’ll need it.

Getting unbanned from Twitter search

  • http://cynematic.wordpress.com/ Cynematic

    I just used the “get out of jail” procedure @Shoq outlined myself.

    Wonder if there are discernible patterns, such as partisan lines, to the “flltering”? Starting to feel a wee tad paranoid.

  • http://www.drumsnwhistles.com/ Karoli

    When they let you out of jail, all of your previous tweets will be gone. Erased. Not recoverable. How much does that suck?

    Yes, I'm feeling very paranoid indeed. Too many people that I follow have suddenly been “filtered” (read, silenced, erased, deleted). Paranoia is generally the result when arbitrary, capricious decisions to exclude voices and speech are made.

  • Pingback: Karoli (karoli) 's status on Monday, 13-Jul-09 07:04:28 UTC - Identi.ca

  • Pingback: Rebecca MacKinnon (rmack) 's status on Monday, 13-Jul-09 08:07:43 UTC - Identi.ca

  • http://www.facebook.com/tonymaro Tony Maro

    Given Twitter's technical prowess – meaning their inability to keep the servers afloat under any reasonable load, and the weeks during which random users couldn't change their profile backgrounds and … well you get the picture. I tend to think this is more likely a result of a technical glitch and less likely something Evil™

  • http://www.drumsnwhistles.com/ Karoli

    Tony, by their own admission it's intentional. Delbius spends a lot of time telling users who actually figure out how to complain that they either posted the same post multiple times or posted a link to a Google-flagged malware link. I find no evidence that it's true, but they are clearly owning the fact that it's intentional.

    Twitter search was Summize, remember, which was an incredibly efficient and accurate tool before they acquired it.

  • http://www.facebook.com/liz.ditz Liz Ditz

    Karoli — I am figuring this stuff out as I go.

    My tweets from this AM have not shown up in search.

  • http://www.drumsnwhistles.com/ Karoli

    Sure enough. I would be inclined to say that Tweetdeck might need to merge data in or something…maybe we should give them some time to update the indexes, but they certainly ought to appear.

  • http://www.facebook.com/liz.ditz Liz Ditz

    Tested: posted a couple of tweets in Twitter, not Tweetdeck. Still don't appear. Sent messages to as suggested by Shoq.

  • http://www.facebook.com/liz.ditz Liz Ditz

    Oh, and I did tweet a lot, from Tweetdeck, using #iranelection tag.

  • http://thoughtbythought.net Tre

    This is invaluable information. Very appalled for all the reasons you've identified.
    Why pour out what we want as useful information and attempt to build community this way.
    Please keep me/us posted on your indentica effort….
    Also makes me realize the need for finding other tools..you use friend feed a ton…any filtering going on there?

    Thanks so much for this.

  • Pingback: StevieB’s Shared Items – July 14, 2009 at Lost in Cyberspace

  • http://twitter.com/Lena_ Lena Cardell

    Looks like they're penalizing heavy RT usage and possibly also folks who break the change to @replies (with a leading . or other character) by pulling them out of search.

    To some people that is spammy and/or non-contributive.

  • http://www.drumsnwhistles.com/ Karoli

    …Although @delbius steadfastly insists that RTs are not penalized
    whatsoever.

  • http://www.seoaly.com SEOAly

    As someone who just found out today that I was also filtered out of Twitter Search, I empathize with all who have suffered the same fate. To my knowledge, I have never violated any of Twitter's published “rules” or TOS. The issue has since been resolved, but with well over 15,000 updates in my Twitterstream, only six – yes…you read that right – SIX of them now appear in Twitter Search. Nice. Color me DISPLEASED. :(

    And, for those with questions about possible TweetDeck issues, I use it exclusively and as soon as the filter was removed from my account my tweets immediately began appearing in Twitter Search again.

  • http://saltydroid.wordpress.com SaltyDroid

    I was filtered out of Twitter search for 48 hours back in March.

    http://www.saltydroid.info/twitter-search-not-f

    I write an anti-spammer blog :: And I use Twitter aggressively to try and disrupt the businesses of various web ConMen. Once I write a post about a bad guy … I will send the link as an @reply joke to anyone who talks to that ConMan. The tweets are always original … but the link is always the same. This violates the letter, but not the spirit, of the “same link” rule. I was also reported as Spam simultaneously by a large network of, ironically, spammers. That was enough to trip the filter … and my account went silent.

    I immediately put in a ticket with Twitter Support. 48 hours later I was back in full … only tweets during the filter were lost to search … the rest were restored. Support never replied to my ticket over 9 weeks with three resubmissions.

    However :: I've since become even more aggressive … and the Spam complaints have likely been more frequent … and it's never happened again. I think it's likely that one human override of the spam-filter-algorithm prevents future occurrences. So maybe that's a small upside to being unfairly silenced?

    It's too bad that they are so terrible at blocking actual spammers :: Who are clearly running riot.

  • http://www.drumsnwhistles.com/ Karoli

    Hi Tre,

    Friendfeed isn't filtering at all. I'm using Friendfeed as my archival tool for tweets – it's quite useful that way. And if I keep getting spammers popping up in my twitter timeline, I'm going to use it exclusively. Twitter is not coming at this problem in the right way, and it's starting to really aggravate me.

  • Pingback: » I Have Made The Twitter Gods Angry - Wired For Noise

  • Pingback: Stop Twitter Spam » Poll Results: Twitter Moderation of Trending Topics

  • http://www.facebook.com/avivagabriel Aviva Gabriel

    Karoli, this is an extremely well-written article about an extremely serious “Twitter censorship” issue.

    I recently discovered my #healthcare, #hcr, and #iranelection tweets were “filtered out” of Twitter search results. Further investigation shocked me: ALL my tweets were “filtered out” of Twitter search.

    After 5 weeks, I was finally restored to visibility. But NONE of my prior tweets will EVER appear in Twitter search results. Even tweets that were never “filtered out,” tweets that I posted BEFORE being filtered out, have DISAPPEARED!

    This is true censorship. There's no other word for it. Twitter CENSORS valid, useful tweets ~ and facilitates invalid, spammy commercial tweets.

    They're deciding that Twitter readers want to see commercials and ads, and don't want to see posts on healthcare, the Iran election, jazz, and so on.

    That's CENSORSHIP… as you so clearly point out in this phenomenal blogpost!

  • http://www.facebook.com/avivagabriel Aviva Gabriel

    Perhaps the problem is this (as SaltyDroid says):

    Tweets with significantly varying text ~ containing the SAME link ~ trigger Twitter's filter.

  • avivagabriel

    I have permanently lost 4300+ tweets …they will NEVER again appear in Twitter Search results. I was wrongfully filtered from Twitter Search results.

    I was restored (after 5 weeks filled with multiple (unsuccessful) attempts to get help, and growing anxiety).

    But NONE of my prior tweets were restored. TWO YEARS of my tweets have VANISHED FOREVER…altho' I've only been filtered for 3 months.

    This is NOT acceptable to me…and shouldn't be to any Twitter user.

    You (and anyone) may suddenly be censored without warning, simply because your tweet habits triggered a mindless, ineffective Twitter filter.

    Spammers abound on Twitter. Why aren't THEY being filtered out of Twitter search results?

  • avivagabriel

    I decided to bypass Twitter Search (which only goes back 55 days), and use Google + Bing to find my 4300+ tweets.

    Google found them ALL!

    Bing found NONE!

    I submitted my URL ( http://twitter.com/avivagabriel ) to Bing. We'll see if they eventually index my tweets…or not.

    Wonder why Bing doesn't “see” tweets by @avivagabriel ~ but Google DOES?

    Weird.

  • Pingback: » Banned But BadAss

  • pjs

    I am new to twitter. What is it and why is it causing so much trouble. I haven't used it and with what i have read afraid to. Can you tell me what it is supposed to do please? thanks

  • http://www.drumsnwhistles.com/ Karoli

    Twitter is an amazingly effective communication platform. Unlike instant messenger, it allows you to broadcast a message out to many, and follow the thoughts of many who interest you. There's nothing to be afraid of, really — there are lots of controls for you to protect your privacy and still participate. Because Twitter is used by many to broadcast a message out to many, search results and inclusion are important. After all, If I say something or link to something I think is interesting to those who follow me, I should be able to find it later. That was really the point of my post here…by filtering search results based on arbitrary rules results could be unfairly skewed. But for users who just want to communicate with a select number of people, or follow some folks for information purposes, that's really not an issue and shouldn't stop you from using their service.

  • http://twitter.com/avivagabriel avivagabriel

    @pjs, No need to be afraid of using Twitter! Here's the story on one of the little glitches that can occur (it's a fixable glitch):

    if you unknowingly violate any of Twitter's Terms of Service, a filtering algorithm in the Twitter Search Engine may be automatically triggered. The result is that your tweets (your name, the content of your messages, and any #hashtags you may have used) will be filtered out of Twitter Search results. Your name and your tweets will not appear at all when someone searches by your name, a keyword, or a #hashtag.

    For example, let's say you posted a tweet to someone, encouraging them to read an article on healthcare reform. Let's say you also tagged the message with the hashtag #healthcare so other people could easily find the article. Let's also say that you were curious to see if your tweet showed up in Twitter Search, and you did a quick Twitter Search using the term #healthcare. Your tweet would never appear in Twitter Search results. You would basically be invisible.

    If someone tried to find your Twitter profile (web page) by searching on your name, so they could “follow” you and your tweets ~ or just read your tweets ~ they would arrive at a page telling them that there was no Twitter member by that name. You would essentially “disappear.”

    The good news?

    (1) Anyone could still access your Twitter profile page by typing your Twitter name after the standard Twitter URL into their browser's address bar. For example, if someone wanted to go to my Twitter profile, they'd enter the following link (URL) into the browser's address bar (just like any other URL): http://www.twitter.com/avivagabriel

    (2) Twitter WILL help you clear up any problems (like being wrongly banned from Twitter Search results). Once I identified the right tech support person, my problem was fixed in a few hours. I now appear in Twitter Search results.

    There's nothing to be afraid of. However, it's a good idea to read the Terms of Service. And it's a great idea to use forums like this to get answers and advice (as you've already figured out)!

    Twitter ~ Terms of Service and Rules: http://twitter.zendesk.com/forums/26257/entries

    (To access Twitter Help ~ once you've registered for an account ~ just click on the link at the top right-hand side of your Twitter profile page).

  • http://twitter.com/avivagabriel avivagabriel

    I would “second” what @Karoli wrote: Twitter is an amazing service, and there's no need to be afraid of using it. In fact, I should also add that my problem with being “filtered out” of Twitter search results was cleared up promptly (once I reached the correct tech support person). The benefits of Twitter far outweigh annoyances.

    @pjs, I sent a reply to your concerns about using Twitter via Disqus (email) on Saturday, August 1. For some reason, it didn't appear here. I reposted it manually (see above).

  • avivagabriel

    Getting “banned” from Twitter Search results: I want folks to know the good news. Once I identified the right tech support person, my problem was fixed within a few hours.

    I was successfully reinstated, and all my tweets (and profile) now appear in Twitter Search results.

    Twitter Search apparently only goes back 55 days. Although none of my 2 years of tweets (prior to reinstatement) will appear in Twitter Search results any longer ~ only 55 days of tweets have actually been “lost.”

  • avivagabriel

    What Twitter “Rules” must you “violate” to get “banned” from Twitter Search results?

    Twitter Best Practices: http://help.twitter.com/forums/10713/entries/42646

    We've included some tips below to keep your content relevant (and listed in Twitter search.) We suggest that you stay away from:

    * Posting duplicate content (links or tweets)
    * Misuse of hashtags (words followed by the '#' sign)
    * Sending automated tweets or replies
    * Using bots or applications to post similar messages based on keywords
    * Posting similar messages over several accounts
    * Abusing trending topics
    * Aggressively following and un-following people (a common spam tactic)
    * Re-posting public content without attribution to the author

    Twitter may automatically remove accounts [from Twitter Search results] engaging in these behaviors from search in order to ensure the best experience for everyone.

  • avivagabriel

    I would “second” what @Karoli wrote: Twitter is an amazing service, and
    there's no need to be afraid of using it. In fact, I should also add that
    my problem with being “filtered out” of Twitter search results was cleared
    up promptly (once I reached the correct tech support person). The benefits
    of Twitter far outweigh annoyances.
    ……………………………….
    Aviva Gabriel, MFA, MSc
    802.649.7033
    ……………………………….

  • http://www.drumsnwhistles.com/ Karoli

    They're pretty vague, but they're here: http://twitter.zendesk.com/forums/26257/entries

  • Pingback: Hippy Steve (exador23) 's status on Tuesday, 04-Aug-09 17:53:10 UTC - Identi.ca

  • Pingback: Børge (forteller) 's status on Friday, 07-Aug-09 09:25:13 UTC - Identi.ca

  • Pingback: Twitted by ingematland

  • Pingback: The Web2Marketer » Blog Archive » weekly (weekly)

  • Mary Goldwater

    Oh…thank you! What a nice article. I have been so busy with the business in Iran that I haven't spent much time looking at otyher things! I happen across this when I was googling myself to find references to our current Project Greenlight to Light up the Night while Ahamadinijad is in NYC for the UN General assembly (Sept 18 – 25). I am trying to get the Empire State building, hotels and restaurants (or any other businesses) to light up green! poster http://twitpic.com/gk3c5. Coincidently, I was banned again for a day due to my promotion efforts…lol I will tweet your article to #iranelection :) BTW @delbius has fixed my account several times, as recently as yeaterday.

  • Mary Goldwater

    Oh…thank you! What a nice article. I have been so busy with the business in Iran that I haven't spent much time looking at otyher things! I happen across this when I was googling myself to find references to our current Project Greenlight to Light up the Night while Ahamadinijad is in NYC for the UN General assembly (Sept 18 – 25). I am trying to get the Empire State building, hotels and restaurants (or any other businesses) to light up green! poster http://twitpic.com/gk3c5. Coincidently, I was banned again for a day due to my promotion efforts…lol I will tweet your article to #iranelection :) BTW @delbius has fixed my account several times, as recently as yeaterday.

  • http://divolk.blogspot.com/ DiVolk

    http://socialsupport.pbworks.com/Shoq's-Twitter… (Getting unbanned from Twitter search) – available :(

  • http://www.uc2.blogspot.com Parris Boyd

    Excellent article. I've been active on Twitter for several months – very critical of Toyota – and have encountered censorship issues from Twitter and Yahoo.

  • http://www.petitiononline.com/TMC2003/petition.html toyotasludge

    Twitter censorship…

    Twitter isn't the only entity censoring comments made. Since being involved in the Toyota engine oil sludge debacle which started well before Toyota's announcement of a sludge policy in 2002, my comments have been filtered at various sites. Edmund's Town Hall (now CarSpace) has active manipulation of comments and will BAN a poster if there is consumer organization involved! YouTube comments are blocked by ToyotaUSA, too, if the message pertains to real-world Toyota vehicle defects! Ahhhh…freedom of speech doesn't exist in Toyota's “Real-World!” The reading public has NO IDEA!

Previous post:

Next post: