web2express.org

I notice a perfect case study from today’s news:  “Continental airlines incident” was emerging at about 8:29am (PST) on the twitter daily new topic list on this http://web2express.org website. it was probably at the same time as other major news outlets broke the news, maybe even a litter bit earlier.  Surprisingly, this top news did not show up on Twitter.com’s trending topics list, nor on google trends at all for the whole day.

This case study shows the difference between various trending applications. I think the underlying technology is the key. Calais is the semantic analysis core of my real time trending system. It seems Calais does pretty good job as it promises.

more digging:

I just found the flight schedule from NYT.com report:  The flight, Flight 61, took off at 9:54 a.m. in Brussels (3:54 a.m. Eastern time), according to Continental’s Web site. It touched down at 11:47 a.m., earlier than its scheduled noon landing, at Gate C123 at Newark Liberty International Airport.

Something is amazing if you compare the time carefully. The incident showed up as new hot twitter topics at 8:29am PST while the flight was still in the air.  where did the tweets come from so early? did someone tweet in the airplane? or people on the ground in Europe got the information early and tweet? Anyway, news travels on twitter fast, very fast!

In the past several days, web2express digest site did not receive data from twitter api. This was because twitter was still trying to fix the problem with its timeline feed for data mining. Earlier today, twitter api team announced the retirement of the data mining timeline feed. But, good news is that twiiter has something better for developers: the streaming api.

I implemented streaming api for web2express digest web site yesterday. It has been running smoothly. The latest version of Twitter4j library released a couple of days supports streaming api. I just plug it in and it works beautifully.

New feature: a new feature is also added to allow visitors to add twitter user as friend while browsing the hot topics on the digest web site.  I think this feature will make the free web site more useful to twitter users.

-aj

After several months of development, I’m happy to announce twitter digest web site is now available for the public. Web2express.org web site’s default home page is switched to this “Digest” web app.

Thanks to twitter’s api and the api team, the data feed for data mining is just wonderful. Web2express Digest is a real time system that takes in the feeds as they become available, and does some NLP analysis on the tweets using open tools like Open Calais and openNLP.  The results are freely available on http://web2express.org/.  Using this twiiter web app, you can spot   daily hot topics and for each hot topic, quickly find the top contributing twitter users. I hope this real time information will
help users to understand the popular topics at any given moment and easily identify who to follow.

I have run the system internally since the beginning of this year. Tweets from millions of users have been analyzed. Each day, large amounts of public tweets pass through the system, from which hot topics are identified by semantic analysis and ranked by popularity. By comparing hot topics in current 24-hour period to the previous 24-hours, new topics are selected.

Please let me know if you have any comment.

AJ Chen