A Department-Based Twitter Aggregator in Google App Engine

We have a department twitter account here at CELT. The idea of that account is it’s a place to share edtech and ed design info without forcing faculty members to sort through my political rants, Jenny’s comments about beer and biking, etc.

But obviously it’s makes very little sense for us to log into that account whenever we have a new edtech insight or retweet. We’d like to just stay in our own spaces, and let the @kcelt feed pick up the pertinent stuff.

So I built this little aggregator to compile all of our tweets that use the kcelt keyword.

http://twaggart.appspot.com/?g=kcelt&a=raganmd,holden,gobman,judybrophy,scastriotta

where basically param ‘g’ is the keyword you choose to mark stuff you want to go to the group acct, and ‘a’ is the set of twitter accounts you want to look in for that key word.

Once you get to that point (a clean RSS 2 feed) you just have twitterfeed check it every half-hour and do the required postings. You can also embed the RSS in your blog, or Pipe it to a thousand other uses.

To make sure the call stays under the 30 sec limit for App Engine threads, it makes only one call to twitter (searching for the key term) then goes through the results filtering out those authors that are not approved.

Here’s the code (Python in App Engine’s wsgi env):

http://docs.google.com/View?id=dcb2rjpn_16f9cxvknp

Here’s how you might post to it if you are using the params we set

twag1

And here’s how that comes out after Twitterfeed posts from the aggregated feed:

twag2

Obviously, you would likely use a different keyword and different accounts. But it’s App Engine, so feel free to use it if you want, and let others know about it.

You can see our newly rejuvenated kcelt feed, with everyone’s contributions, here.

About these ads

6 Comments on “A Department-Based Twitter Aggregator in Google App Engine”

  1. Mike says:

    I’ll add one more thing — I lost a bunch of time because when I set the HTTP response headers to application/rss+xml I ended up with duplicate headers, one of them printing AFTER the xml, which of course blows validity up, among other things.

    After some research and help from @psychemedia and my brother, I found that basically you can’t use print statements as your output if you want to mess with the default headers — it looks like “print” flushes the response buffer immediately (which results in your headers being delivered before your modifications take effect).

    So if you are messing with header in App Engine, try the verbose but header-friendly

    self.response.out.write(‘foo’)

    instead of

    print ‘foo’

  2. Tony Hirst says:

    So you’re using app engine to aggregate the feeds,

    Not having used app engine – is it straightforward setting an app up?

  3. Mike says:

    Yeah, pretty much. It’s not a purely cloud development system — you test on 127.0.0.1:808x and upload, but the system makes it easy.

    There’s also the wsgi object model, the bane of those that want a quick script fix, but if you are a hacker you just throw all your code into the main wsgi handler and voila it works.

  4. Mike says:

    Nice. Cool setup. For GAE, are you talking about the timout for external calls? You know this is configurable, right?

    You should try proxifeed.com, a more advanced tool than Twitterfeed.

  5. Mike says:

    @Mike — no I did not know it was configurable — do tell!

  6. Cruz says:

    App Engine is in its infancy. As with all their Beta projects there are problems. The main problem is how they are dealing with the problems. In short they are overwhelmed. People are asking for PHP, and their favorite python projects to be supported. They made the mistake of claiming that most python frameworks will run on it without putting up the proper CAUTION signs. It is possible to get Zope to run on it with some work.
    You can read this article:-
    http://bygsoft.wordpress.com/2010/01/09/cloudy-combo-google-app-engine-and-amazon-s3-combo-pack/


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 130 other followers