Machine build with local ai
- Thread starter ablumski
- Start date
Soldato
- Joined
- 30 May 2007
- Posts
- 5,047
- Location
- Glasgow, Scotland
Ram ram and more ram. That allows the largest models.
CPU/gpu plays a part too in response time or output speed… but getting into that can start costing.
CPU/gpu plays a part too in response time or output speed… but getting into that can start costing.
Last edited:
Getting Github Copilot Pro for $10 a month is well worth it. Works in VS Code, Jetbrains and Xcode.
I'm working on a new build at the moment and initially it was going with mini-atx and two ram slots, so I bought 128GB.
I've put most of the bits together, but then found an ATX case that would fit where I want it, so I'm trying to decide if I should ditch the micro and go full atx so I can have 256GB ram to last me a few years and be able to dabble with local ai.
Not sure it's worth it though? Would the bump be worth it you reckon?
I've put most of the bits together, but then found an ATX case that would fit where I want it, so I'm trying to decide if I should ditch the micro and go full atx so I can have 256GB ram to last me a few years and be able to dabble with local ai.
Not sure it's worth it though? Would the bump be worth it you reckon?
You don’t need 128gb, let alone 256 to dabble with local AI. Even the 7b models are fine for messing with, and any CPU/RAM setup is going to be glacially slow compared to something that’ll fit on a GPU.
Once you’re spending that sort of money you might as well take advantage of the providers loss leading on frontier models and let them do the leg work.
Well I wanted 128GB anyway as I frequently max out my 32GB in Adobe products.
I tested Topaz a while back and they shift to using your normal ram when the video card caps out, so I figure the more of that you have the better.
I've seen vids on youtube of people with Macs and 512GB of RAM, so....there are ways to fill it up if you try! It'll be a while probably before I bother to get a 5 series graphics card though.
I already spend a lot on the frontier models every month through work.
I tested Topaz a while back and they shift to using your normal ram when the video card caps out, so I figure the more of that you have the better.
I've seen vids on youtube of people with Macs and 512GB of RAM, so....there are ways to fill it up if you try! It'll be a while probably before I bother to get a 5 series graphics card though.
I already spend a lot on the frontier models every month through work.
You end up with single digit tokens per second when running large models on CPU. The Mac Studio/Mac air aren’t bad options as the unified memory means the GPU has direct access to system memory….although to load a Mac Studio up costs megabucks (£10k+).
The nVidia DGX spark started shipping last week. Probably the best option if you are serious about training models.
Stuffing as many 4090/5090s into a PC as you can is still king for inference.
Last edited:
Bit late to this thread but FWIW any of the Strix Halo 128GB builds are pretty decent for this sort of stuff. Corsair have the top of the range (Max+ 395 128GB) box for £2k or so.
Even something simple like Lemonade server on Windows works fine - via Vulkan although you can get more out of ROCm on windows with rdna kit now.
Runs something like gpt-oss-120b-mxfp-GGUF OK although you'd be better off running that on Linux until everyone catches up with the fact that yes RDNA gpus with >64GB memory ARE now addressable above 64GB.
Its Windows and frankly I'm amazed it works as well as it does but its using the 8060S GPU (CPU usage at 5%) with Vulkan and churning out decent results on a 120W power draw. Its a monster that won't cost you a fortune to run.
Also 128GB of quad channel 8000MT DDR5 on the APU doesn't look as expensive as it did at the start of the year
Even something simple like Lemonade server on Windows works fine - via Vulkan although you can get more out of ROCm on windows with rdna kit now.
Runs something like gpt-oss-120b-mxfp-GGUF OK although you'd be better off running that on Linux until everyone catches up with the fact that yes RDNA gpus with >64GB memory ARE now addressable above 64GB.
Its Windows and frankly I'm amazed it works as well as it does but its using the 8060S GPU (CPU usage at 5%) with Vulkan and churning out decent results on a 120W power draw. Its a monster that won't cost you a fortune to run.
Also 128GB of quad channel 8000MT DDR5 on the APU doesn't look as expensive as it did at the start of the year