• Notice regarding image hosts and attachments. See here.

Improvement from SSD in this specific case

  • Thread starter Thread starter Seirrah
  • Start date Start date

Seirrah

Seirrah

I am batch running a piece of software up to 15,000 times in a single 'run'. Each time the software reads from a small input file (which also changes over time), and writes the output to a similar sized file. This takes a very long time. I am wondering by approximately how much using a SSD would help.

As the files are small (a few k), I'm guessing (maybe incorrectly) that the improvement in speed is down to the access time only. As the access time of an SSD is ~0.1ms and a HDD ~10ms, could I possibly see a near 100 increase in speed of the read/write part of the total operation?

Any thoughts?
 
I think there are too many variables to guess. For example, it's possible that write amplification or ineffcient firmware could make it slower on a SSD. I've noticed that unzipping lots of small files on my SSD seems quite slow (OCZ Vertex2E 60GB). Maybe that's down to the Sandforce controller doing on-the-fly compression? You'll just have to buy one and find out ;)

As a plan B I'd consider looking for RAMdisk software and batch running it from there.
 
Are you sure the drive is your bottleneck?

I can copy 20,000 small text files on my hard drive in just a few seconds (maybe some fancy pants caching and command queueing helps).

Presumably the program runs to completion then exits? If it is something like java it could say take a second fire up the jvm on each invocation. That would add 4 hours to your runtime.

I have no idea what startup overheads are for various languages but it may be worth looking into :)
 
Thanks for the replies. I have no way of speeding up the runtime of the software(.exe) that is being called thousands of times. I'm thinking that if the read and write file operation, of each iteration of the loop, is say 50% of the time taken, then maybe moving from a HDD to a SSD might help.

I think I'll try to use a RAM-disk and see what improvement I get.
 
The SSD would be faster due to two reason.

1: SSD have a much faster 4k read and write speed then HDD.
My 2TB WD drive with 64mb cache HDD does about 0.5mb/s read and 1.1mb/s write
My C300 ssd gives 28mb/s read and 50mb/s write
With Crystal Disk Mark.

2: Just like you said the access times is much faster. A HDD gives about 10ms at the fastest more like 12ms on most drive and a SSD give 0.1ms. So its about 100 times faster with accessing of the data.
 
Try it in your RAMdisk, if you get a decent improvement there, an SSD will probably help (though no reason not to just keep using the RAMdisk tbh if the total diskspace required isn't too big (30000 files at 4KB = ~120MB)

Keep an eye on your CPU usage whilst running the script too, it's quite possible that the bottleneck is elsewhere and you need to make your script more efficient. Multithreading is probably the first thing to do if it isn't already, and quite easy to do in Java iirc.
 
Last edited:
There's a fair amount of cpu overhead in what your doing.

If you could write the app your self, you would only open the output stream once, and then, depending on the implementation, flush and close it at the very end, after all your operations are completed.

They way it is now, that output stream has to be opened, closed and flushed every time you run it. Not to mention the app has to accocate and deallocate memory on each run. Thats all fairly cpu expensive. Your real issue is efficiency.

Without any other options, a RAM disk is probably as good as your going to get. I can still see you doing a fair bit of waiting tho:(
 
You must log in or register to reply here.
Back
Top Bottom