Nvidia Spark

[Tinek]

I know, I also get the most out of both my 5090.and 4090, pulling a combined 750w just from the GPUs in aggregate, and getting nowhere near the same level of work our of them since they can only service one request each, despite being faster.

I have usually 4 concurrent requests/agents running on a single 5090 (plus 64GB RAM) on a Windows to that (exllamav3, Qwen3.8 27B 4bpw HQ with 6bpw LM head etc., q5,5 256k context kv cache with lvl2 8gb vRAM KV cache and lvl3 SSD one (50GB). Over 200T/s in many cases, still have vram to spare but this has been tuned with my own harness and quality benchmarks compared to 8,8 KV and other variations. Point being, I don't need Spark to do that already, I don't see a reason to have one for such use. I wouldn't mind one, like I wouldn't mind 96GB vRAM for FUN, though - a hobby these days for me but can't justify it with those prices. Still, a single 5090 is already plenty good at home really IMHO. Granted, bought for MSRP as prices these days are ridiculous.

 

[safc2023]

Well mine arrived but haven’t had much chance to play due to being away for the football and next week work.
I set one up and have ssh into it so may play more next week.

Second one is still in the box and depending on my test cases it may get moved on.
Looks like sold out now and another price increase imminent!
so I’m glad I moved when I did.

 

[XeNoNF50]

I have usually 4 concurrent requests/agents running on a single 5090 (plus 64GB RAM) on a Windows to that (exllamav3, Qwen3.8 27B 4bpw HQ with 6bpw LM head etc., q5,5 256k context kv cache with lvl2 8gb vRAM KV cache and lvl3 SSD one (50GB). Over 200T/s in many cases, still have vram to spare but this has been tuned with my own harness and quality benchmarks compared to 8,8 KV and other variations. Point being, I don't need Spark to do that already, I don't see a reason to have one for such use. I wouldn't mind one, like I wouldn't mind 96GB vRAM for FUN, though - a hobby these days for me but can't justify it with those prices. Still, a single 5090 is already plenty good at home really IMHO. Granted, bought for MSRP as prices these days are ridiculous.

Fair play, it does seem you have tuned that quite well and glad it is working for you.

There are just different levels & more choice wrt models when you have multiple sparks (or even a single). I have been running GLM 5.3 Flash and used billions and billions of tokens of it. Deepseek V4.1 Flash 1017 dropped yesterday, approx GPT Sol 5.6 / Opus 5 level model that I will be running locally by the end of the week.

Speaking of ridiculous prices, since Nvidia announced 64GB DGX Spark, 128GB has had it's official reprice. Check Nvidia's direct site, now £6390. My guess is existing stock which is a lot less than this will tend up more towards that price in the not too distant future. Mad considering 2 of my 3 I got for a few hundred quid more than half that price.

 

[Freddie1980]

I'll need to watch more reviews before making up my mind on this new APU but Spark looked very underwhelming considering it's price point judging by LTT's review and he had it up against the lower end Mac's which costs less

 

[XeNoNF50]

I'll need to watch more reviews before making up my mind on this new APU but Spark looked very underwhelming considering it's price point judging by LTT's review and he had it up against the lower end Mac's which costs less

Specs are one thing. Getting the most out of it with inference engine optimisations etc is a whole other. The Sparks are very well supported, with a whole community working on this. Lower end Mac models really struggle (even my M5 Pro). Having said that, as things stand if you want hardware for the bigger AI models nothing beats an M5 Ultra Mac Studio. They are brute force specced and at least they are all available at MSRP. The best value hardware out there atm. I just opted for another Spark because they daisy chain and you get even better performance when you do so via the connectx7 port.