Using Clojure to work with RSS feeds

As I mentioned in my last blog entry, I am trying to learn Clojure. I think it is an interesting challenge to internalize new thinking paradigm. So following entry/code is an attempt to learn the paradigms of the language. Drop a comment, if you have any suggestion.

I found that, while trying to learn new language, one of the challenge is to find interesting practice problem to solve. The practice problem which helps you understand the strength of the language. I think one such problem space Clojure's application would be interesting is parsing and data crunching.

Example Problem
So the problem I picked is a simple one. I heavily use google reader and for that matter most of my web reading happens as part subscribed RSS feeds. I find myself manually filtering the feed list to suite my interest at the time. This exercise creates parser for RSS feed. This parser creates Clojure records based data structure. Once you have all the feed represented in records structure you can write variety of functions to filter the feed or even apply intelligent web algorithms.

Organization of Code
I am looking for a good way to organize code in clojure. Just like Java packages, clojure has namespaces. But I have still not found a good way to organize clojure functions into files/namespaces. For now, I chose to create namespace per domain.

• rss.data.clj - Define records data structure and parser
• rss.feed.clj - Defines functions to get feeds
• rss.stat.clj - Defines functions to filter feeds (and may be some statistical algorithms)

data.clj

(ns rss.data
  (:require [clojure.xml :as xml])
  (:import  [java.io ByteArrayInputStream]))

(defrecord rss-channel [title description link lastbuilddate pubdate items])
(defrecord rss-item [title description link guid pubdate])

(defn create-rss-channel [map items]
  (let [title (:title map)
        pubDate (:pubDate map)
        description (:description map)
        link (:link map)
        lastBuildDate (:lastBuildDate map)]
    (rss-channel. title description link lastBuildDate pubDate items)))

(defn create-item [item-data]
  (let [description (:description item-data)
        pubDate (:pubDate item-data)
        guid (:guid item-data)
        link (:link item-data)
        title (:title item-data)
        ]
    (rss-item. title description link guid pubDate)))

(defn parse-item [item-data]
  (create-item (zipmap (map :tag item-data) (map :content item-data))))

(defn parse-items [items-data]
  (map parse-item items-data))

(defn parse-channel [channel-content]
  (loop [x (:content channel-content)]
    (create-rss-channel
      (into {} (filter #(not (= :item (key %))) (zipmap (map :tag x) (map :content x))))
      (parse-items (map :content (into [] (filter #(= :item (:tag %)) x)))))
      ))

(defn parse-rss [rss]
  (let [rss-data (ByteArrayInputStream. (.getBytes rss "UTF-8"))]
  (for [x (xml-seq (xml/parse rss-data))
        :when (= :rss (:tag x))]
    (map parse-channel (:content x)))))

feed.clj

(ns rss.feed
  (:require [clj-http.client :as client]))

(defn get-feed [url]
  (:body (client/get url)))

(defn get-feeds [urllist]
  (pmap get-feed urllist))

stat.clj

(ns rss.stat)

(def keywords ["Clojure" "Groovy" "Python"])

(defn find-items [rss-item-list keyword]
  (filter #(.contains (first (:title %)) keyword) (flatten rss-item-list)))

(defn find-items-keywords [rss-item-list keyword-list]
  (flatten (map #(find-items rss-item-list %) keyword-list)))

Trying Delicious RSS
Let's use RSS feed published by delicious.com. All RSS feed can found here. Let's use the feed popular which got has tag "programming". (http://feeds.delicious.com/v2/rss/popular/programming)

user=> (rss.stat/find-items-keywords (map :items (flatten (rss.data/parse-rss (rss.feed/get-feed "http://feeds.delicious.com/v2/rss/popular/programming")))) ["Clojure" "Python" "Groovy"])
(#:rss.data.rss-item{:title ["Hidden features of Python - Stack Overflow"], :description nil, :link ["http://stackoverflow.com/questions/101268/hidden-features-of-python/102062"], :guid ["http://www.delicious.com/url/16bbee33bbf4f6a03448058c5fd9f461#thaiyoshi"], :pubdate ["Sun, 23 Jan 2011 16:53:58 +0000"]} 
#:rss.data.rss-item{:title ["Writing Forwards Compatible Python Code | Armin Ronacher's Thoughts and Writings"], :description nil, :link ["http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/"], :guid ["http://www.delicious.com/url/c6fb2679bca6c193a41d92513f7edca1#papaeye"], :pubdate ["Sat, 22 Jan 2011 16:09:10 +0000"]})

This one only had few links. Let's try one which has lots of items. (http://feeds.delicious.com/v2/rss/tag/programming?count=100)

user=> (rss.stat/find-items-keywords (map :items (flatten (rss.data/parse-rss (rss.feed/get-feed "http://feeds.delicious.com/v2/rss/tag/programming?count=100")))) ["Clojure" "Python" "Groovy"])

(#:rss.data.rss-item{:title ["The Evolution of a Python Programmer : Aleks' Domain"], :description nil, :link ["http://metaleks.net/programming/the-evolution-of-a-python-programmer"], :guid ["http://www.delicious.com/url/15311aafa2207da0ffc770903091ccea#tsung"], :pubdate ["Mon, 24 Jan 2011 02:44:19 +0000"]} 
#:rss.data.rss-item{:title ["Hidden features of Python - Stack Overflow"], :description nil, :link ["http://stackoverflow.com/questions/101268/hidden-features-of-python/102062"], :guid ["http://www.delicious.com/url/16bbee33bbf4f6a03448058c5fd9f461#adharmad"], :pubdate ["Mon, 24 Jan 2011 02:43:07 +0000"]} 
#:rss.data.rss-item{:title ["PySCeS: the Python Simulator for Cellular Systems"], :description nil, :link ["http://pysces.sourceforge.net/index.html"], :guid ["http://www.delicious.com/url/40e012b894d62cd8cb9ba6b95855b868#tiguco"], :pubdate ["Mon, 24 Jan 2011 02:39:24 +0000"]} 
#:rss.data.rss-item{:title ["Hidden features of Python - Stack Overflow"], :description nil, :link ["http://stackoverflow.com/questions/101268/hidden-features-of-python/102062"], :guid ["http://www.delicious.com/url/16bbee33bbf4f6a03448058c5fd9f461#lonstile"], :pubdate ["Mon, 24 Jan 2011 02:24:25 +0000"]} 

Source Code available at GitHub (https://github.com/kartikshah/rss-feeder)

Blogged with the Flock Browser

How-To - IntelliJ Idea for Clojure/Compojure on Google App Engine

StrangeLoop conference really spiked my interest in learning Clojure. I wanted to find a way to do web development using Clojure. So this entry is primarily a how-to on setup of IntelliJ Idea to do Clojure using Compojure web framework, Leiningen build tool and Google App Engine deployment environment.

Step 1: Download Leiningen
Download Leiningen script file and follow instruction to put it in PATH.

Step 2: Install "La Clojure" and "Leiningen" plugin for IntelliJ Idea.
To do this go to Preferences -> Plugins. Go to "Available" Tab


Step 3: Setup Leiningen plugin properties
To setup Leiningen plugin properties in IntelliJ click on Preferences -> Leiningen and give the path to the executable.

Step 4: Create project template
Now with the installation complete, we are ready to create a new project. For now the project creation is done outside IntelliJ Idea using Lein on command line. This is the only step done outside the IntelliJ Idea. Choose the folder structure where your IntelliJ workspace is.

     $ lein new gae-cmpjr
     Created new project in: gae-cmpjr

You can see that Lein created project folder structure. In the project root it created a project.clj file used by Leiningen. Under src folder, it also created default namespace by the name of the project and created a core.clj file under it.

Step 5: Create project IntelliJ
Let's create IntelliJ Project, even though project structure exists pick the same folder path that you selected in step 4.

• File -> Project New... and select project from scratch
• Select both GAE and Clojure. Provide the path to the appengine sdk

Now basic setup for the project is done. You can browse the project files using IntelliJ. (Biggest benefit of La Clojure is it auto-inserts and matches brackets for very "brackety language" :-))

Step 6: Setup Leiningen to run from IntelliJ
To setup Leiningen from IntelliJ, click on Leiningen tool window. (Window -> Tool Windows -> Leiningen)

Step 7: Setup web development libraries
Setup the project to include Compojure and Ring libraries for development.
project.clj

(defproject gae-cmpjr "1.0.0-SNAPSHOT"
  :description "Example for Compojure on GAE using IntelliJ"
  :namespaces [gae-cmpjr.core] 
  :dependencies [[compojure "0.4.0-SNAPSHOT"]
                 [ring/ring-servlet "0.2.1"]]
  :dev-dependencies [[leiningen/lein-swank "1.2.0-SNAPSHOT"]]
  :compile-path "web/WEB-INF/classes"
  :library-path "web/WEB-INF/lib")

core.clj

(ns gae-cmpjr.core
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use compojure.core
        ring.util.servlet)
  (:require [compojure.route :as route]))

(defroutes hello
  (GET "/" [] "<h1>Hello World Wide Web!</h1>")
  (route/not-found "Page not found"))

(defservice hello)

Step 8:  GAE setup

web.xml

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
            http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
           version="2.5">
    <display-name>Compojure on GAE using IntelliJ</display-name>

    <servlet>
        <servlet-name>hello</servlet-name>
        <servlet-class>gae_cmpjr.core</servlet-class>
    </servlet>

    <servlet-mapping>
        <servlet-name>hello</servlet-name>
        <url-pattern>/</url-pattern>
    </servlet-mapping>
</web-app>

appengine-web.xml

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>compgae</application>
  <version>v0-2</version>
</appengine-web-app>

Step 9: Compile and Build war

From the Leiningen Tool window in Idea run following commands.

lein clean
lein deps 
lein compile 

Step 10: Run locally
To locally run click on already create "Run Configuration"

Step 11: To upload application to appengine
Tools -> Upload Appengine application

Blogged with the Flock Browser