back to the future

February 2, 2011

Just a brief note to say that I haven’t abandoned all hope of getting this site going again. Even though I’m not playing MMOs at the moment, it seems a shame to just leave everything sit idle. My basic infrastructure runs without too much effort, so it is no great problem to refresh the data every couple of months.

The main obstacle is that Blizz is now serving the up-to-date data from in HTML format rather than XML. My page-scraping code needs to change to cope with that. Fortunately however the Blizz engineers are serving up valid XHTML, which means that XPath expressions can still be used to extract the data we need.

If I’ve been a good little engineer then only my XPaths need to change and nothing else…

There is a danger that the XPath paths can become more than a bit baroque because they have to navigate through all the HTML markup to get to the data nodes, although there are tricks to get the XPath engine to do a lot of the searching.

Anybody looking for inspiration on how to parse the XHTML should check out these posts by a geek blooger called Kastang.  That’s the method I’ll be using when I get back to all this.

14 Responses to “back to the future”

  1. seriallos Says:

    For the moment, I believe all the old XML feeds are still working. I think the only thing missing right now is how a player chose to reforge his/her gear.

    For my own stuff, I don’t need reforging so I’m resisting the XHTML parsing path. They’ve mentioned that they intend to friendlier feeds back up at some point so I’m just going to wait it out until those show up or they turn off the old XML altogether.

  2. zardoz Says:

    The vibe in cyberspace is that a lot of calls to the old armoury fail. Are you finding that? There are a couple of discussion threads – which Google will find – that made me pessimistic about using the old site on a large scale.

  3. seriallos Says:

    I’ve only really played around with the achievements feed recently and I haven’t noticed a ton of problems with it. Although now that I look at my code, I don’t really have any monitoring for armory errors so I’m not sure if I’ve been seeing a lot of failures or not. I’ll see if I can add some better handling of armory errors to find out if it’s stable or not.

    Only app I have right now that talks to the armory is this:

  4. zardoz Says:

    Let me know what you find – if its still up-to-date and usable I may as well stick with it.

  5. roncli Says:

    Right now, everything Armory related on my website is working, although it was really sketchy the first few days after Cataclysm hit. If you visit the old pages in IE, you see the “we’ve moved” bit, but if you view source, you’ll find the XML is still in tact.

    I have had issues with new items showing up if you go directly to item-info.xml, but searching for them appears to work.

  6. zardoz Says:

    Thanks for that. I’ll do a test run against the old armoury soon and see what happens.

  7. twiddle Says:

    The XML Feeds seem to work for all Armories (US, EU, CN, TW, KR) as far as I can tell (, although I’m using a specific User-Agent ID for my crawler). Would love to see your reports updated, greate site!

  8. Sudiin Says:

    I think I speak for countless bloggers and players when I express my profound gratitude for your efforts in taking the time to keep updating the Armory Datamine project.
    Take care!

  9. zardoz Says:

    Thanks for those kind words. I’m getting closer to a time when I may be able to do a Cataclysm update.

  10. Mercadi Says:

    Sweet, looking forward to seeing any new stuff here.

  11. Cesar Says:

    I used to datamine raid and dungeon stats on my site, I still cant figure out how to get those stats with the new armory. Sad 😦 you have any luck?

  12. zardoz Says:

    I haven’t looked very closely at the new armoury data. Still trying to free up some time for that.

  13. Johnnie Says:

    When are you going to update the information on the site? I really like your site because the information was very useful to me. The information helped me level my toons.

  14. Gorman Ghaste Says:

    Character/Guild/Arena Team APIs Preview:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: