utilities/twitter-hack: twitter crawler
utilities/twitter-preprocessor/script/twitter-product.py: naive language detector using UTF-8 code range
utilities/twitter-preprocessor/LuceneIndexBuilder: Use Lucene to build the index
utilities/twitter-preprocessor/Preprocessor: filter out retweets, extract URLs, timestamps and hashtags, eliminate @ tag

Please direct all questions and suggestions to our mailing list: twitter-semantic-search@lists.cs.columbia.edu