Datastorage and performance

intellix

intellix

intellix

Associate
Joined
7 Dec 2007
Posts
302
Location
Derbyshire, Tibshelf
For my game, I'm going to have data about the different building: costs, requirements, names, cost multipliers and other random data about buildings, that are only ever accessed and are never likely to change...

This also goes for unit information such as costs, requirements, HP, build times yadda yadda... which are also only ever going ot be accessed by users and are also never likely to change...

At the moment these were put into excel, converted to CSV and imported into an SQL table for users to SELECT when they were about to build something... but im sure this is probably the worst way to do it... performance wise?

In comparison to SQL, which is the best way of storing data that is never likely to change and just be accessed by users?

Perhaps XML? Textfile? some other form of flatfile database? SQL? CSV? and does anyone have some sort of reference that will explain in more detail? (as this is for my uni project and I need to explain them etc)

Thanks :)
 
I don't think you need sql for that... it's a bit ott using sql.

The best way preformance wise would be a binary format perhaps with a lookup table at the start of the file saying where the data is. For data that is varible in bit size use a end charecter like \0 to say where that data stops.

i.e
[look up table]
Price Positionbytes
health Poisistioninbytes
damage Positioninbytes
[/end table]

Then move to the posistion in bytes, and read.
That what a lot game formats do.

Pretty simple really, if you don't like binary you could work as textfile but preformance will take a hit.

Any decent file i/o tuturial should teach you binary input/output.

What language you working in?

In c you used to be able to read structures in out really easily.

You also should try to read the files before the game plays and store them in ram, espically if you have blocking i/o as that will make the game jolt when you read big files, don't read files when the games being played, do the loading at the start of the game.
 
Last edited:
Web Game? Your not at wolverhampton are you,? :P

Is it turn based? Then you don't really need to worry about performance as it's only real time games you really need to avoid blocking i/o.

Just stick it in a textfile and parse it, or do the binary thing i said.
 
Last edited:
its real time based, and im not in wolverhampton :D

Need it to be able to handle any amount of users :P 1million if need be ;D
 
I should really explain myself, some of my mates at wolvo uni had to design a game in php.

How are you doing a real time game as web based?
Example?
 
Last edited:
Doesn't seem performance limited, so i think anything will do. The trick to having a website serving millions is scaling, being able to spread the application across loads of servers possibly using load balancers so unless you've got infrastructure for that I would not worry. The performance of a application does not really matter with scaling you just add servers when it gets slow. Making application perform better will only allow a few more users.

It's called scaling horizontally, scaling vertically is adding more performance to a single machine, but that has it's limits.

You will have to look into distributed systems, to make the program capable of scaling like that. A common problem is data storage(Persistence), and how you exchange data between servers, or you can just cheat and store the data in the user sessions.
 
Last edited:
cool :D thanks for the information.

Understand it better as well now, can add all this to my assignment :P Will show people my game sooner or later as well, people have helped enough to see it at least :D

The scope of my project is a little insane I think. Had to learn a language then code something like that... well I already had web and data skills to an extent, just had to learn PHP was really it... and design the game. It has been fun though :)

Apparently Travian started off with around 100 users by word of mouth as a hobby then grew into what it is now, of like 4.3million players :D
 
Pho said:
Could always use SQLite (http://www.sqlite.org/) instead of the full blown package. Probably save you hours of writing parsing routines too :p.
Click to expand...

I guessssss I could at least have a look at it, but how much more work would it be? to just implement from a CSV file with data seperated by commas and rows ending with a ; or something?

When first reading about sql lite I was a bit... blerg! pointless! but after reading that page again I can see why someone would use it :P
 
Personally, I'd either go for XML or SQL. I only skimmed this thread earlier, but I see you're using PHP, so MySQL would also be a good choice for the SQL server. You could also make use of PHP's built in XML parser.

Where are you going to store data on your game's users? The most obvious answer would be SQL, so why not make use of that connection for the rest of your game? You could even use stored procedures that get triggered every few hours to do all your game updates so you don't have to touch it as well.
 
its currently doing most of that using SQL. The users were making the updates... sooo there are 10 troop movements, everytime a user loads a page they check the troop movement table. If there are any records with a hit_target_time less than the current time, do all the updating etc.

I can imagine it would also be good for crontabs to sift through it every minute or something in the odd case that a user isn't online to make the updates.

Surely it would be quicker to check a flatfile structure such as CSV? building_ID indexed by the row number and then grabbing that information? because then there is no connection required... its only going to get read... but you know more than me :P

if you suggest using XML or SQL... out of three... Which is the quickest method for retrieving the building information?

CSV, XML, SQL. If the building information was within an XML surely it would be useful for fansites and the guide page to utilise this XML and display it elsewhere? Providing that it is the quickest way?

Also, if it is quicker than SQL having a flatfile system... how much quicker? and is that only going on the basis that the connection is always open using Apache pconnect?
Thanks :)
 
As you say, if you have a persistent connection then I imagine it would be fairly speedy anyway. SQL is designed for large amounts of data, hence the reason people moved away from flat-file databases years ago.

If you have a large list of buildings, let's say 100,000, and you want to retrieve #98,000 then using XML it would have to iterate through every row until it got to 98,000 (unless you had read them all into memory first I guess) which might be fairly slow. If you indexed the buildings table in SQL by the building id it would be able to jump straight to the row (pretty much, anyway).

It would make more sense from a relations point of view to stick it all in the database though, so a) you can define relations between all your data and b) it keeps it all in one place rather than in several locations.

I imagine you wouldn't want to give access to your main XML files though, and would create a dumbed down version for third parties, which you could easily build from SQL results instead anyway. Although written in .NET, we have a web service running at work which takes the results of an SQL query and outputs it as XML which is very quick – a PHP site requests some data, we perform calculations using several stored procedure and spit it back out as XML.

edit: I just noticed you were in Tibshelf, only about 15 miles from me :o.
 
yeah, I noticed you were from Derbyshire and decided to fill mine in :P Where abouts are you from? Did work in Alfreton Tesco, girlfriend lives in Chesterfield and im from Tibshelf :D

There will probably be about 50 units... Is it only pconnect and apache that allow the database connection to stay open? and if so... why would anyone choose any other SQL package if it allows you to do that? :P
 
Without being too specific I'm in a small village outside of Belper, I work just outside of Matlock though (even closer to you) :).

I would imagine pconnect is available on any webserver which supports PHP, i.e. Microsoft's IIS. There's even a function for Microsoft's SQL server, mssql_pconnect to allow you to establish persistant connections to that as well. Unless you really are getting hundreds of connections a second I imagine both ways would be just as quick though.
 
You must log in or register to reply here.
Back
Top Bottom