Weekends for me are usually spent doing one of two things, working or hacking. Hacking is learning and usually I do it to inform decisions I need to make for real projects. In this case a client asked me to build a website aggregator in WordPress. He wanted something more like a news portal in the vein of the HuffingtonPost, but a lot of the features he was requesting reminded me of TechCrunch. So to help me figure out the best way to accomplish doing this client project, I decided to build my own aggregator using WordPress.
Step 1 – PostRank
Techmeme is a website that uses algorithms to determine early trending stories related to technology. The algorithms used to accomplish this have to be pretty sophisticated, looking at comments, in and outbound links, re-tweets as well as other things like credible tips and story leads. They have minimal human input and let their tecch do most of the work. I wanted to do the same. Understanding how much of a pain this type of programming can be due to my work with SwiftRiver, I knew I wasn’t about to attempt to come up with any custom algorithms for a ‘toy’ project like this. Instead I decided to leverage a platform that already does this, PostRank.
PostRank offers a number of free and paid web apis that can be used to process content to determine influence and popularity online. They started out about three years ago as AidRSS, and have since grown to become a pretty awesome platform. They also have a hosted platform that lets individual users add feeds that they then begin ranking for social influence. It took about an hour to add about 25 feeds and get them indexed.
PostRank then returns a new feed of ‘ranked’ content. Best, Great, Good and All are the four methods with which you can filter your feeds. As the titles suggest they filter based on how popular the sites content has been online. For example, best means that a blog post has had a lot of delicious.com favoirtes, a lot of Twitter retweets and a lot of comments. The others would be less. Each item in the feed has to be given a score and these scores are used to filter out other content that don’t meet certain criteria as determined by PostRank.
Now I had 25 feeds, all optimized to only allow the best content through.
Step 2 – Yahoo Pipes
I didn’t want to pay PostRank for access to their advanced API (which would allow me to get one feed back from all of the ones I added in the previous step). So, instead I took each of those PostRank’d feeds and added them to YahooPipes.
Is a free service for creating web mashups with minimal programing skills necessary. My pipe (which can be found here) is pretty simple. It takes several of the ranked feeds, mashes them together and orders them chronologically, re-orders them in descending order by popularity, strips the ‘PostRank’ number from the title and outputs a new feed.
This new feed will consist of only the most recent popular content.
Step 3 – Feedburner
Feedburner is a Google service for RSS feed processing and optimization. Yahoo Pipes is great but for some reason in the past three years, Yahoo hasn’t figured out, or bothered to figure out how to output feeds that validate. Also, Yahoo Pipes doesn’t offer any sort of analytics, branding or advanced syndication options. So I took my newly created Yahoo Pipe feed and added it to Feedburner.
Here I was able to make sure the feed validates, rebrand it with a title and description, set-up auto-mated email blasts and ad ‘flare’ options for sharing content via Delicious and Facebook.
- Akismet – spam filtering, never press without it
- FD Feedburner – for redirecting subscribers to my Feedburner feed
- Feed WordPress – advanced aggregation for WordPress
- Feed WordPress Duplicate Post Filter – all aggregators need to filter out duplicates
- Shorten2Ping – shorturls for all your posts (ex. t.co, is.gd, bit.ly)
- U-Cron – Cron Jobs (automated tasks) made simple for WordPress
Step 4 – Yourls
I wanted my aggregator to have it’s own short url. I decided to go with a.fricame.me (Africa Meme) because my aggregator showcases trending topics related to African tech. With Yourl you simply buy and configure your domain name then setup the script in the root folder. It’s just like setting up WordPress. Dead simple, took all of five minutes. You can get it here.
Once I had Yourl configured, I used the Shorten2Ping plugin to automatically create a short URL for all my posts.
Step 6 – Twitter and Ping.fm
Finally, I had a completely automated aggregator that was only finding the most popular trends about African tech. Awesome. Next I needed to automate the process of broadcasting. I setup a new Twitter account @africameme and a new Ping.fm account. Ping.fm is a service that ‘pings’ multiple social networks with the same updates at once. If I wanted to blast Facebook, LinkedIn, Twitter, Email and a plethora of other services at once, I could use it do so. I opted for just having it ping Twitter whenever new content is aggregated. (I only needed Ping.fm because the Twitter updates for Shorten2Ping are no longer working due to Twitter moving to Oauth only authentication.)
In six easy steps, and about as many hours you can easily set up your own Techmeme-type news site on any subject! To see it all in action follow @africameme on Twitter or visit us at http://a.fricame.me