Paging through an ATOM feed using Node.js, CoffeeScript, MongoDB and Atom Hopper

Post to Twitter

I work daily with Atom Hopper which is a Java ATOMPub server. As we add new features we need to test those features and a lot of times we end up with small scripts just like the one I’ll show you today. Atom Hopper recently was updated to allow for paging through and ATOM feed. Basically, Atom Hopper will produce Next and Previous links in the feed head that can be followed easily to get to either the tail (oldest entries) of the feed or the head (newest entries).


[ad name=”Google Adsense”]

See my previous articles or the wiki on how to get Atom Hopper setup as well as my article on configuring it to use the MongoDB data adapter. You can also use the Hibernate data adapter if you wish.

The feed head that Atom Hopper produces looks similar to this:

<feed xmlns="http://www.w3.org/2005/Atom">
  <link href="http://localhost:8080/namespace/feed/" rel="current" />
  <link href="http://localhost:8080/namespace/feed/" rel="self" />
  <id>urn:uuid:0b5061ec-94b1-4ffc-a383-803221793e2c</id>
  <title type="text">namespace/feed</title>
  <link href="http://localhost:8080/namespace/feed/?marker=urn:uuid:34ee3a9a-31dd-46e4-b39e-05bd3596966d&amp;limit=25&amp;search=&amp;direction=forward" rel="previous" />
  <link href="http://localhost:8080/namespace/feed/?marker=urn:uuid:9b2e6989-0771-4622-b0eb-49d33f4f5487&amp;limit=25&amp;search=&amp;direction=backward" rel="next" />
  <link href="http://localhost:8080/namespace/feed/?marker=urn:uuid:69128671-daf3-4bac-adbb-64e459b6701e&amp;limit=25&amp;search=&amp;direction=backward" rel="last" />
  <updated>2012-06-09T03:43:01.743Z</updated>

... Entries would be seen here ....
  
</feed>

You can see that there are several links. Since I’m at the head of the feed where the newest entries are, I can then follow the next link to move towards the tail of the feed. As I do this I could be parsing the ATOM entries contained within the returned data. As a sample I loaded 100 ATOM entries into MongoDB and I set my limit in the script to only load 25 entries per call (25 is the default and the maximum is 1000).

Using CoffeeScript I can easily follow the next links until I get to the tail. Once I reach the tail there are no more entries and therefor no next link will be produced at the tail of the feed.

You will need to install a couple NPM packages first:

$ npm install elementtree
$ npm install coffee-script

Here is the CoffeeScript:

fs = require("fs")
et = require("elementtree")
http = require("http")
url = require("url")

element = et.Element

opts =
	host: "localhost"
	port: 8080
	method: "GET"
	path: "/namespace/feed/?limit=25"

loadAtomFeed = ->
	http.get(opts, (response) ->
		returnedData = ""
		response.on "data", (chunk) ->
			returnedData += chunk
  
	  response.on "end", ->
		  etree = et.parse(returnedData)
		  nextLink = etree.find("./link[@rel=\"next\"]")
		  return if not nextLink?
		  
		  parsedUrl = url.parse(nextLink.attrib["href"])
		  console.log "Following Next Link: " + parsedUrl.href
		  
		  opts.host = parsedUrl.hostname
		  opts.path = parsedUrl.path
		  loadAtomFeed()
	).on "error", (e) ->
		console.log "Error: " + e.message

loadAtomFeed()

Note: I’m pretty sure this can be made much more efficient. As this is just a testing script it works “good enough”.

I’m using ElementTree to parse the XML. Although I’m not actually parsing out anything other than the next link you could definitely pull out anything you want from the feed.

I saved the script as app.coffee. Assuming you have CoffeeScript and Node.js installed you can run the script directly like this:

$ coffee app.coffee

My results looked like this:

Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:9b2e6989-0771-4622-b0eb-49d33f4f5487&limit=25&search=&direction=backward
Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:1ed87690-632b-4bc9-a9f1-c7c607d771bb&limit=25&search=&direction=backward
Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:82e12cc7-06d3-4b89-bedc-4f6bd4a3f4b6&limit=25&search=&direction=backward
Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:69128671-daf3-4bac-adbb-64e459b6701e&limit=25&search=&direction=backward

Post to Twitter

This entry was posted in Atom Hopper, CoffeeScript, JavaScript, Node.js, Open Source. Bookmark the permalink.