Custom build for machine learning algorithms

AndrewEdwardAda

AndrewEdwardAda

AndrewEdwardAda

Associate
Joined
9 Jan 2014
Posts
10
Hello Overclockers UK

I have recently ran into issues when running machine learning algorithms which are not down to my coding, but down to hardware. I have no access to HPC so have decided I should build a dedicated system to take over. My current system has 8gb ram, but pretty sure my windows OS and my graphics card will be eating into this significantly.

My dedicated algorithm PC will run linux and be focused entirely on efficiency for fast efficient data modeling and analysis.

My budget it 800 gbp or 1316.48 USD. I would love to see peoples recommendations with regards to this.

Many thanks AEA

Bump.
 
Last edited:
Howdy

I have no idea what you want there unless Battlefield 4 is a learning algorithm, some Pros will be along shortly to discuss your HPC ASAP
 
The devil is in the detail for scientific computing; so can you tell us a little more about the sort of machine learning algorithms you're working on and what language you're writing them in? Also, you say that you're anticipating memory issues: do you mean that you're running out of it or is it too slow?

To give you an indication, I've just been running the gamut of ML algorithms out of Kuhn & Johnson's book (written in "R", running in parallel) on my overclocked 4770K with 16Gb of 1600MHz and, although some things take a little while, I haven't been unduly perturbed by performance!
 
gregorius said:
The devil is in the detail for scientific computing; so can you tell us a little more about the sort of machine learning algorithms you're working on and what language you're writing them in? Also, you say that you're anticipating memory issues: do you mean that you're running out of it or is it too slow?

To give you an indication, I've just been running the gamut of ML algorithms out of Kuhn & Johnson's book (written in "R", running in parallel) on my overclocked 4770K with 16Gb of 1600MHz and, although some things take a little while, I haven't been unduly perturbed by performance!
Click to expand...

Thanks for the reply, I am coding in python using the scikit learn package. So far I haven't encountered speed issues, although it could always be faster and I predict it will get slower as the model gets larger. I am training multiple types of algorithms in order to best fit my model, so far I am using Random Forests, Linear Regression, Logistic Regression, Gradient Boosting Regressor, Gaussian Process and SMV. So far in my tests I have been running out of memory for both Gaussian Process and Random Forests (when I increase the number of forests at least.) Your set up would probably suffice, but I would like to get the best system possible for my budget.
 
I have provisionally designed a system, but could certainly use some pointers. I have added two small SSDs so I can operate them in RAID containing only my algorithms and data. I was wondering what sort of performance boost can be expected from operating a RAID0 configuration?

Secondly I haven't added any CPU cooling as I think it would be good to implement water cooling (I will overclock CPU) any recomendations on a water cooling system? Furthermore any comments on the general build would be appreciated.

Please see the system configuration here: http://************.com/parts/partlist/
 
My real dataset is nearly 3GB, but the one I have been testing on was only 20mb and I have been playing with the number of forests for instance it crashes with a memory error on my test dataset if I set it to 5000 forests.
 
AndrewEdwardAda said:
My real dataset is nearly 3GB, but the one I have been testing on was only 20mb and I have been playing with the number of forests for instance it crashes with a memory error on my test dataset if I set it to 5000 forests.
Click to expand...

Can you turn that 3GB & 20MB into number of observations and number of variables for me?
 
So I know next to nothing about the best kind of system for this job, but I've attempted to spec up a basket anyway as I'm guessing the below would do the job quite nicely? I'm making a pretty big assumption here that the bulk of what you do will need raw CPU grunt rather than utilising anything like CUDA? I'm also assuming it can run on multiple cores? If so I imagine the 6 core chip will give an improvement over any quad core.

YOUR BASKET
1 x Intel Xeon E5-2620v2 2.10GHz 6-Core with Hyperthreading & Turbo (Socket 2011) - Retail £319.99
1 x Gigabyte X79-UD3 Intel X79 (Socket 2011) DDR3 Motherboard £154.99
1 x TeamGroup Vulcan GOLD 16GB (2x8GB) DDR3 PC3-12800C9 1600MHz Dual Channel Kit (TLYD316G1600HC9DC01) £115.99
1 x Kingston 120GB SSDNow V300 Drive SATA 6Gb/s 3 2.5" (7mm height) Solid State Hard Drive - (SV300S37A/120G) £65.99
1 x SuperFlower Golden Green HX 450W "80 Plus Gold" Power Supply - Black £53.99
1 x Seagate Barracuda 7200RPM 1TB SATA 6Gb/s 64MB Cache - OEM (ST1000DM003) HDD £43.99
1 x MSI GeForce GT 610 SILENT Low Profile 1024MB GDDR3 PCI-Express Graphics Card £29.99
1 x BitFenix Merc Beta Gaming Case - Black £29.99
1 x Raijintek Themis Direct Contact CPU Cooler £17.99
Total : £847.01 (includes shipping : £11.75).



If you could push the budget further you could change to this CPU. This would give you an immediate boost as it's clocked considerably higher than the Xeon chip, but it's also unlocked and could be overclocked even further.

YOUR BASKET
1 x Intel 4930K 3.40GHz (Ivybridge-E) Socket LGA2011 Processor - Retail (BX80633I74930K) £439.99
1 x Gigabyte X79-UD3 Intel X79 (Socket 2011) DDR3 Motherboard £154.99
1 x TeamGroup Vulcan GOLD 16GB (2x8GB) DDR3 PC3-12800C9 1600MHz Dual Channel Kit (TLYD316G1600HC9DC01) £115.99
1 x Kingston 120GB SSDNow V300 Drive SATA 6Gb/s 3 2.5" (7mm height) Solid State Hard Drive - (SV300S37A/120G) £65.99
1 x SuperFlower Golden Green HX 450W "80 Plus Gold" Power Supply - Black £53.99
1 x Seagate Barracuda 7200RPM 1TB SATA 6Gb/s 64MB Cache - OEM (ST1000DM003) HDD £43.99
1 x MSI GeForce GT 610 SILENT Low Profile 1024MB GDDR3 PCI-Express Graphics Card £29.99
1 x BitFenix Merc Beta Gaming Case - Black £29.99
1 x Raijintek Themis Direct Contact CPU Cooler £17.99
Total : £967.01 (includes shipping : £11.75).

 
AndrewEdwardAda said:
Roughly 1,000,000 observations with 500 or so parameters
Click to expand...

OK, what I'm going to suggest you do before thinking about new hardware is to get a more quantitative grasp on the resource requirements for your problem. For the size of problem that you're interested in, it is perfectly possible that £800 simply will not do (for a random forest model), and I don't want you wasting you money!

What I suggest you do is to vary two variables independently of one another: (i) number of observations and (ii) number of trees in the forest. So cycle through 10, 50, 100, 500, 1000 trees and 100, 500, 1000, 2000, 5000 randomly chosen observations - giving you 25 observations of memory use (and anything else that you fancy, such as execution time). You should then be able to at least get an order of magnitude estimate of how much memory (and processor power) you need.

I'm not too sure of what flexibility scikit learn gives you (I've never used it), but random forest methods are amenable to splitting the problem up into parallel problems and then using model averaging, so it may be that you have to take that approach. Then if you do that, you can balance your hardware needs against how the model is split.
 
gregorius said:
OK, what I'm going to suggest you do before thinking about new hardware is to get a more quantitative grasp on the resource requirements for your problem. For the size of problem that you're interested in, it is perfectly possible that £800 simply will not do (for a random forest model), and I don't want you wasting you money!

What I suggest you do is to vary two variables independently of one another: (i) number of observations and (ii) number of trees in the forest. So cycle through 10, 50, 100, 500, 1000 trees and 100, 500, 1000, 2000, 5000 randomly chosen observations - giving you 25 observations of memory use (and anything else that you fancy, such as execution time). You should then be able to at least get an order of magnitude estimate of how much memory (and processor power) you need.

I'm not too sure of what flexibility scikit learn gives you (I've never used it), but random forest methods are amenable to splitting the problem up into parallel problems and then using model averaging, so it may be that you have to take that approach. Then if you do that, you can balance your hardware needs against how the model is split.
Click to expand...

Thanks for your reply, I know from how my model breaks down I should be able to manually break the model down into several smaller models. You mention processing power, isn't this a memory only issue? I know it could mean that it takes a week to run the analysis, but I can be patient. Its the type of model that only really needs to be ran once. I will look into some code to tell me how much memory was used.
 
AndrewEdwardAda said:
Thanks for your reply, I know from how my model breaks down I should be able to manually break the model down into several smaller models. You mention processing power, isn't this a memory only issue? I know it could mean that it takes a week to run the analysis, but I can be patient. Its the type of model that only really needs to be ran once. I will look into some code to tell me how much memory was used.
Click to expand...

I mention processing power as the FR algorithm isn't the quickest of the ML algorithms and when one takes into account the (for example) cross-validation tuning, they can run for a long time! Still, if you've got a lot of patience then for sure concentrate on the memory issue.
 
I'm a software developer I don't understand the software your programming, however do have my own artificial intelligence software that often runs 24/7. The software is multi threaded and even with 32GB can see page file access.

I can only offer this advice, you need your build to be really solid and stable, there is no point trying to get an extra few % performance to find the computer crashes and all data is lost.

I won't spec a build but offer these pointers.

1) For PSU. Add up your total system wattage, then choose a Seasonic X series that's at lease double peak power wattage.

2) Get a case thats quite, but offers good cooling. These Silverstone Fortress cases have unique cooling due to 90 degree rotation, soundproofed, plus the 3x 180mm bottom can be run at low RPM.

http://www.overclockers.co.uk/showproduct.php?prodid=CA-168-SV&groupid=2362&catid=1092

3) I would choose quality known memory. My machines are running Kingston Blu, no program crashes and this is running intense floating point for days at a time. One of my machines has it's cache memory multiplier overclocked, in performance mark memory bench it gave a good improvement and the computer stayed stable.

4) I would not run Raid 0 on SSD. I would just choose a single good SSD such as the Samsung 840 PRO. Kingston SSD's btw are known to have problems, if you google 'Kingston SSD issues'. Go with the Samsung 840 PRO.

5) Don't choose the cheapest board, get something at least mid range especially if the board will be under-load for long periods. My view PSU is still more important then motherboard however due to voltage regulators having an easier life if PSU is providing more stable power.

Hopefully above helps.
 
Last edited:
JasonM said:
I'm a software developer I don't understand the software your programming, however do have my own artificial intelligence software that often runs 24/7. The software is multi threaded and even with 32GB can see page file access.

I can only offer this advice, you need your build to be really solid and stable, there is no point trying to get an extra few % performance to find the computer crashes and all data is lost.

I won't spec a build but offer these pointers.

1) For PSU. Add up your total system wattage, then choose a Seasonic X series that's at lease double peak power wattage.

2) Get a case thats quite, but offers good cooling. These Silverstone Fortress cases have unique cooling due to 90 degree rotation, soundproofed, plus the 3x 180mm bottom can be run at low RPM.

http://www.overclockers.co.uk/showproduct.php?prodid=CA-168-SV&groupid=2362&catid=1092

3) I would choose quality known memory. My machines are running Kingston Blu, no program crashes and this is running intense floating point for days at a time. One of my machines has it's cache memory multiplier overclocked, in performance mark memory bench it gave a good improvement and the computer stayed stable.

4) I would not run Raid 0 on SSD. I would just choose a single good SSD such as the Samsung 840 PRO. Kingston SSD's btw are known to have problems, if you google 'Kingston SSD issues'. Go with the Samsung 840 PRO.

5) Don't choose the cheapest board, get something at least mid range especially if the board will be under-load for long periods. My view PSU is still more important then motherboard however due to voltage regulators having an easier life if PSU is providing more stable power.

Hopefully above helps.
Click to expand...

Thanks for the reply, yeh I can understand why long term stability is a requirement for your software, for my purposes it is simply the training of models that is important so the system may be under a heavy workload for a few days, every now and then when training a new model. Apart from that it should be relatively easy workload. I opted for configuration 0 because I will not have data to save except the results of these models, which are backed up to copy a better alternative to dropbox, this link gets you 5gb extra starting space if anyone is interested: https://copy.com?r=rUO8MU
 
You must log in or register to reply here.
Back
Top Bottom