Appropriate hosting for a custom application that needs to run every 30 minutes?

SiD the Turtle · 16 Jan 2008 at 13:30

As some of you may be aware, I run a small website that tracks Xbox Live gamerscores for people in the Console Games and Hardware sub-forum.

This is done by running a C# application I developed that grabs a web page from the Xbox site and then scrapes the information I need from it. The problem is I can only run it when I am there to press run and leaving my PC on all day is not an option. Therefore the data there becomes quickly out of date.

I was thinking I either need a host that will let me run the program on their box at a set interval or perhaps create a smaller tool in whatever language is appropriate to the server which just does the page grabbing bit every thirty minutes and FTP/emails me the files so that I could run the program once a day, but have the data for a whole day loaded in.

Any ideas on the best way of going about this at the cheapest price possible? Hiring a whole server for this is too much money for what I want to do.

uk_viper · 16 Jan 2008 at 13:32

OS? you may be able to get away with a vps

SiD the Turtle · 16 Jan 2008 at 13:34

uk_viper said:
OS? you may be able to get away with a vps

C# means .Net 2.0 and therefore Windows, but if I just want to run the bit that grabs the pages then I could go with whatever.

Edit: Hmmm what about a host that would let me run WGET, CURL or similar.

purejamie · 18 Jan 2008 at 16:07

I have a smilar service for NBA tracking statistics. This may be way off what you are looking for but the way I do it is thus; I wrote a web service which scrapes the stats and passes them back (easy to do in C#) in XML - then hosted that. On the page that presents my statistics to the user I run a query against that web service, transform it and save it locally. Everytime someone logs onto my page it uses the previously saved data to create the stats tables UNLESS the timestamp is over 10 minutes old. If the timestamp is over then the webservice is interrogated again and the data saved locally. This means the service is slightly slower for the person who hits the page when the stats need an update but not very much; this way the stat's are never more than 10 minutes old...

chesterstu · 18 Jan 2008 at 16:39

if its a web app (or can be rewritten as web app) use some kind of psuedo Cron for Windows. Set up an image on the main page as 1 pixel png at the bottom of the page but the path is refresh.aspx (or w/e), the pages just prints out the png but then have code to do the updating from your app so it will get updated for when people see it.

You can have a time stamp in file so you can set it to up date every 30 mins or whatever. Having a VPS to do this update is over kill imo but if you do insist on going down this route GoDaddy do windows VPS for about £20.

I have written similar systems so if you want more info add me MSN.

SiD the Turtle · 18 Jan 2008 at 18:06

Hmm that sounds like an interesting idea. Thanks for that I might look into it.

My host is *nix based but I don't have access to Cron on my package. I might try setting up a Linux test box and having a play.

chesterstu · 19 Jan 2008 at 01:53

That's great there is already implementations of this in PHP. You could rewrite the app as an PHP script, the best way would be to open the html page as an XML doc then XPATH to the relevant node to get the info. 2 -3 hours work at most

Pulseammo · 19 Jan 2008 at 13:41

Just pull out the business logic, wrap it or reference it in whatever classes you need to to get the the ASP.NET server to accept / execute it, build a .NET scheduler using a thread, or see if the server has an inbuilt one (http://msdn.microsoft.com/msdnmag/issues/05/03/SchedulingASPNETCode/)

In which case all you need is a server that will host .NET stuff.

(P.S I don't do .NET, but I'd say since you can run any .NET language as ASP.NET; if you have coded your app in the correct manner in the first place, it shouldn't be a trouble to port it, and use or make a scheduler.)

Edit:

http://www.angrycoder.com/article.aspx?cid=5&y=2003&m=4&d=11

Another link for grow your own scheduler.

SiD the Turtle · 19 Jan 2008 at 18:35

chesterstu said:
PHP idea

Pulseammo said:
ASP.NET idea

Both fantastic ideas chaps. One major stumbling block I'm aware of on PHP (and I'm betting in .NET as well) is execution timeouts. The processing of almost 400 pages of info completes in just a few seconds on my desktop machine but the whole process can take up to 15 minutes due to the delay requesting the almost 400 source pages from gamercard.xbox.com, which isn't the fastest site in the world.

Still, more things to look into.