Web Scraping.

[numnutz]

Does anyone have any experience in scraping data of a website? It currently provides a feed, but it's incomplete and after speaking to them there not going do anything about it!

I'd ideally like to scrape it, cache it, then serve it back up as json. I've limited knowledge of PHP, but always looking to learn!

Thanks

 

[numnutz]

Thanks for the suggestions. I'll need to run it on my webhosting, which at the moment is on a PHP5 setup, so that seems the logical choice. I could run the python on Google App Engine though, which would be nice as it's not my server load then! I don't have anywhere to run the Java stuff, and GAE doesn't support it apparently.

I think I'll give the Python route a go, I've always wanted to give it a go.

 

[numnutz]

IIRC correctly it started off with Java:

Regardless, good luck anyway

Google App Engine runs Java, your right there. Unfortunately , it has some restricted methods that HTMLUnit requires to run, and therefore won't run (unless patched) out of the box.

http://stackoverflow.com/questions/3854368/app-engine-support-for-htmlunit