Computer upgrade for Photography and basic gaming also storage solutions

[Matt_Hirst]

Have posted this on a photography forum as well..

Currently I've just, when running out of space on my computer, either added a new hard drive or got a new larger hard drive and just put it in the PC, backups of things are held on a multitude of external hard drives and since I have started to do a few macro shots stacking (30 or 40 images to get one shot) it soon eats space so I have finally got to the point where I need to look at my home IT and sort it out. Speaking of macro, bulk processing of images seems to take a long time - which If I am honest is not my favourite place to be waiting on images to process.

My lightroom / image storage is 4.2 TB of data and the catalog sits on one of the NVME hard drives. Helicon (focus stacking software) cache is on one of the NVME also.
Denoising in Topaz takes approx 20 seconds per image, ok when singles but as I ma playing around with stacks and learning doing this to 50 + images - let's just say I am not the most patient when sitting in front of a computer (I work in IT 9-5 and thats enough for me).

Now my current PC is as follows:

Intel 12700k
Nvidia 1080 graphics card
32GB of DDR4 mem
3 x 1 TB NVME hard drives
1 x 4 TB ssd
1 x 8TB HD
2 * 4k monitors

Backup / PC NAS holds
1 x 8TB
1 x 4TB
1 x 3TB

2nd NAS (HP Micro Server)
4 x 2 TB RAID 5 (6 TB usable)

So to the questions for the more technical amongst you:

PC
* When batch processing or moving images around (lightroom / Photoshop / Helicon) what would you be looking at specs wise for a new PC or upgrade to the above - is it better to go AMD Threadripper (more cores lower speed) or higher single core speed with as many cores at that high speed?
* RAM, is it just a case of as much as you can get as fast as you can get?
* Local storage again is it better to just have your images in a NAS with a 10GB connection to the local machine or better to maintain a mirrored HD setup locally (NAS for copies), catalogs on NVME or go to something like an in pc NVME raid card (2tb NVME drives are fairly reasonable) or a mix or all these?
* Graphics card, I can see Helicon benefits from a faster Graphics Card - mine is getting on a bit, but the prices of them nowadays is just crazy - What is the sweet spot with Graphics Cards, is there a big benefit to go to current or wait until next gen cards or pickup a secondhand 3090? Note I haven't found a website that rates cards for photography as such to suggest by going 4080 you get x performance, 3080 y performance.

NAS 1 was looking at the new UGreen 8 bay or building a NAS from old PC in say a Jonsbo N5 case.
would then add 4 * 8TB Raid 5 (24 TB usable) and expand as necessary...probably add my old 1 Tb and 2 TB drives that I have lieing asround in here as well.

NAS 2
4 * 4TB Raid 5 1 have a HP microserver that can do this task.

Other option have 1 NAS and 1 large external hard disk as these sometimes pop up on offer for a reasonable price - how many copies of data do I really need?

Network switch
1 10 GB switch - any reccomendations?

Am I over thinking things regards everything - I usually do this - should I just accept things as they are??? I just would like to get to the point whereby I haven't got so many small 1TB / 2TB hard disks floating around and my storage is upgradeable in a more logical form as and when required.

Cost / TIme wise I am looking to move to the new NAS over time as funds allow, same for PC, but I would prioritise the NAS first I think?

Any help appreciated.

Matt

P.S. Forgot to say only game I reallly play is PUBG infrequently other than if I have say a a week off then am playing it all the time.

 

[adrichardson81]

Your cpu isn't particularly slow for what you're doing. The gpu is the obvious weak point - a 3080ti would be a good bet there.

Storage wise, you seem to have a decent amount, though 1tb drives might be a bit limiting. What's your workflow? Are you shooting raw and converting to tif before stacking? Once you've output the stacked image, are you keeping the input?

 

[Tetras]

* Graphics card, I can see Helicon benefits from a faster Graphics Card - mine is getting on a bit, but the prices of them nowadays is just crazy - What is the sweet spot with Graphics Cards, is there a big benefit to go to current or wait until next gen cards or pickup a secondhand 3090? Note I haven't found a website that rates cards for photography as such to suggest by going 4080 you get x performance, 3080 y performance.

It is tricky, you do really need to find some specific benchmarks for the app, or at least some user experiences of x versus y card.

I could advise about general price/performance, but that might be irrelevant for your use case.

* When batch processing or moving images around (lightroom / Photoshop / Helicon) what would you be looking at specs wise for a new PC or upgrade to the above - is it better to go AMD Threadripper (more cores lower speed) or higher single core speed with as many cores at that high speed?

The only website I know of that includes those kind of benches is Puget. Generally speaking, you can't get away from needing single core/thread performance and most apps do have a point of diminishing returns with cores, but it depends on the specific tasks that you're doing, as some scale with core better than others. At least, with previous versions of Adobe software and particularly Photoshop, single core/thread still used to reign supreme, but with the newer features (like the AI ones) things are more complicated and there seems to be a general shift toward greater use of the GPU in many apps, even for something which was historically considered 2D like Photography.

* RAM, is it just a case of as much as you can get as fast as you can get?

I don't think this applies to professional/server motherboards with registered memory, but with consumer platforms like AM5 and 1700, DDR5 has not been a happy bunny when using the max capacity and you lost A LOT of speed by using 4 sticks. I don't know what the current state is with 9000 series CPUs.

 

[Matt_Hirst]

Your cpu isn't particularly slow for what you're doing. The gpu is the obvious weak point - a 3080ti would be a good bet there.

Storage wise, you seem to have a decent amount, though 1tb drives might be a bit limiting. What's your workflow? Are you shooting raw and converting to tif before stacking? Once you've output the stacked image, are you keeping the input?

After writing all this I was thinking of going either 3080 or wait for the 5080 to show up and see what appears on the secondhand market.

CPU wise I may look at say a 13th or 14th gen if there are gains to be had - I can't go cutting edge as this is just a hobby for me, but would just like to have things quicker than they are.

I am keeping the originals, deleting the mid process ones and obviously keeping the output ones. I think I do just need to have it all available in some fashion on a NAS and then look at options for my local PC but just keep going down rabbit holes the more I try to research so thought ask on here.

 

[Matt_Hirst]

It is tricky, you do really need to find some specific benchmarks for the app, or at least some user experiences of x versus y card.

I could advise about general price/performance, but that might be irrelevant for your use case.


The only website I know of that includes those kind of benches is Puget. Generally speaking, you can't get away from needing single core/thread performance and most apps do have a point of diminishing returns with cores, but it depends on the specific tasks that you're doing, as some scale with core better than others. At least, with previous versions of Adobe software and particularly Photoshop, single core/thread still used to reign supreme, but with the newer features (like the AI ones) things are more complicated and there seems to be a general shift toward greater use of the GPU in many apps, even for something which was historically considered 2D like Photography.


I don't think this applies to professional/server motherboards with registered memory, but with consumer platforms like AM5 and 1700, DDR5 has not been a happy bunny when using the max capacity and you lost A LOT of speed by using 4 sticks. I don't know what the current state is with 9000 series CPUs.

They do have a benchmarks result set but they don't appear to validate things so you have a 6800xt beating a 4090, https://www.heliconsoft.com/helicon_focus_benchmark/table_view.php, hence much confusion on my part.

I think maybe the way forward is to sort out the storage situation first, then look at the computer after - the storage is the one giving me a bit of an headache esp NAS's I just want good reliable storage solution not bothered about much else but every review goes on about Docker's, VM's etc...

Thanks for the reply.

 

[Tetras]

They do have a benchmarks result set but they don't appear to validate things so you have a 6800xt beating a 4090, https://www.heliconsoft.com/helicon_focus_benchmark/table_view.php, hence much confusion on my part.

Hmm, yeah, that does not look very useful, unless I'm missing the relevance of the threads thing.

I think maybe the way forward is to sort out the storage situation first, then look at the computer after

You do have one advantage in making your assessments about everything else, in the sense that you already have a great PC that you can do most of those tasks with, so you can monitor your usage closely with software like hwinfo and Windows task manager and see where the bottlenecks are. If you can then make very specific targeted upgrades to exactly the tasks/bottlenecks that are slowing you down, that's going to be a big win compared to dumping a few £K on a threadripper in the hope it'll fix it.

 

[jigger]

Hmm, yeah, that does not look very useful, unless I'm missing the relevance of the threads thing.


You do have one advantage in making your assessments about everything else, in the sense that you already have a great PC that you can do most of those tasks with, so you can monitor your usage closely with software like hwinfo and Windows task manager and see where the bottlenecks are. If you can then make very specific targeted upgrades to exactly the tasks/bottlenecks that are slowing you down, that's going to be a big win compared to dumping a few £K on a threadripper in the hope it'll fix it.

Maybe you could improve a little on the current systems performance for a reasonable amount, but even the lower end, older generation Threadrippers would beat the i7 system into a quivering ball of snot for these types of workflows.

The question with building a Threadripper system is it worth coughing up for the latest socket.

 

[Matt_Hirst]

Maybe you could improve a little on the current systems performance for a reasonable amount, but even the lower end, older generation Threadrippers would beat the i7 system into a quivering ball of snot for these types of workflows.

The question with building a Threadripper system is it worth coughing up for the latest socket.

I don't know much about the threadripper other than people use them for their performance, always had Intel.

I did read one response regards performance and it did suggest faster core speed and as many as possible is the best way to go...as said started to go down too many rabbit holes with everything and got confused on the way to go.

 

[Matt_Hirst]

Was doing some additional testing last night:

Denoise:

CPU sits at 35%
Hd bounces up and down to around 50%
GPU 100%

In last nights test the Image takes around 9 seconds to denoise and around 4 to save to Hd afterwards.

Export from lightroom to Helicon
CPU sits around 20%
Hd 100%

Took around 2 minutes for the export of 122 images

Helicon

Was playing around with where the cache was stored - eventually had it on a SSD (note not NVME)

SSD sat at 100% cache on the SSD equated to 1TB during the processing, then deletes itself afterwards.


Now this testing got me thinking further, say I do 100 image focus stacks (not that this is needed all the time), if 4 seconds per image is lost writing each image and this may happen multiple times would it make sense to look at a SAS option for the hard drives and controller, run it in RAID 5 this would then free up the 8TB in my main pc to go into one of the NAS. the SAS drives themselves typically spin faster than your typical HD and because they're enterprise drives failiure rates are typically lower.

The other thing is running a RAID 0 NVME card with two gen 4 drives and use this as a cache drive - I know the lifespan of running the drives like this could be affected, but if say 2 x 1 TB drives could be picked up reasonably priced, does this make sense? Other option populate with 4x 2TB drives in Raid 0, download pics to Hd, copy to NVME raid 0 to do all the leg work / processing then copy back to Hd afterwards for backup / storage.

The GCard is a bottleneck, probably more so in denoise it's just trying to work out what will give an improvement and where the sweet spot is regards it.

I did look at the Helicon benchmark tables but it does seem suspect, if the graphics card was the most important part surely the newer versions i.e. 4090 would be quicker, so the chart tends to suggest it could be CPU but as no speeds are mentioned it's difficul to ascertain only other thought would be the Hard drive - which from my results suggests could be important as it's sat at 100%.

regards,

Matt

 

[Tetras]

Was doing some additional testing last night:

Cool, that looks like some handy testing!

Export from lightroom to Helicon
CPU sits around 20%
Hd 100%

Took around 2 minutes for the export of 122 images

The HD being 100% is pretty conclusive about the bottleneck I think, even if the CPU is only hitting one core it looks like you have room to scale there.

To be clear, by HD do you actually mean hard drive, or is it a SSD?

Denoise:

CPU sits at 35%
Hd bounces up and down to around 50%
GPU 100%

That's interesting and consistent with what I've heard about these newer features moving to utilising the GPU instead of the CPU.

I did look at the Helicon benchmark tables but it does seem suspect, if the graphics card was the most important part surely the newer versions i.e. 4090 would be quicker

Something very fishy with their tables, which makes it hard to draw any meaningful conclusions, BUT, with some very specific processing tasks it is true that an architecture (as a whole, e.g. 4060 to 4090) can perform oddly similar to each other if there's some kind of bottleneck in the pipeline that applies equally to all of those cards. Similarly, the same can happen when there is a need for driver optimisations to remove whatever bottleneck exists on a software level and allow that architecture to shine.

Personally, I would not consider buying anything based on the benchmark data on their website.

 

[Gray2233]

The other thing is running a RAID 0 NVME card with two gen 4 drives and use this as a cache drive - I know the lifespan of running the drives like this could be affected, but if say 2 x 1 TB drives could be picked up reasonably priced, does this make sense? Other option populate with 4x 2TB drives in Raid 0, download pics to Hd, copy to NVME raid 0 to do all the leg work / processing then copy back to Hd afterwards for backup / storage.

What drives are you running exactly? Using them as cache is very, very heavy on endurance. I'd personally go for Western Digital Red SN700 nvme, they're not the fastest but they're still quick and their rated endurance is through the roof at around 5100 TBW +.
I think given your explanation that's probably your biggest problem if I'm understanding correctly.

 

[Matt_Hirst]

What drives are you running exactly? Using them as cache is very, very heavy on endurance. I'd personally go for Western Digital Red SN700 nvme, they're not the fastest but they're still quick and their rated endurance is through the roof at around 5100 TBW +.
I think given your explanation that's probably your biggest problem if I'm understanding correctly.

I think this is the problem, but it's taken discussing on here to work things out.

currently I have

1TB Samsung 980 pro as boot disk
1TB Sabrent Rocket as D drive (games)
4TB Samsung 870- QVO (this was my old storage for my pictures but IU ran out of space, so just have email and what not on here
4TB WD Red this was a local backup of my old QVO photo drive
8TB WD Red Pro as storage for my images

The catalog and cache for Helicon is on the 4TB QVO.

I think I am going to start by:
* Replacing the 1 TB Samsung with a 2TB Gen 4 NVME.
* Replacing the 1 TB Sabrent with a 2TB Gen 4 NVME - this can become my cache / catalog drive.
* Replacing the 8TB WD Red with a 8TB NVME.

Run some performance specs on this, if there are bottlenecks I may look to add a RAID 0 cache into the mix.

Next upgrade will be the graphiocs card, I may try members market for a 2080TI first, just to see how it affects everything, then look for a 3080TI...Maybe even write a post about the performance difference it makes as there is little out there to really glean information from for people in the same situatio.

 

[Matt_Hirst]

Cool, that looks like some handy testing!


The HD being 100% is pretty conclusive about the bottleneck I think, even if the CPU is only hitting one core it looks like you have room to scale there.

To be clear, by HD do you actually mean hard drive, or is it a SSD?


That's interesting and consistent with what I've heard about these newer features moving to utilising the GPU instead of the CPU.


Something very fishy with their tables, which makes it hard to draw any meaningful conclusions, BUT, with some very specific processing tasks it is true that an architecture (as a whole, e.g. 4060 to 4090) can perform oddly similar to each other if there's some kind of bottleneck in the pipeline that applies equally to all of those cards. Similarly, the same can happen when there is a need for driver optimisations to remove whatever bottleneck exists on a software level and allow that architecture to shine.

Personally, I would not consider buying anything based on the benchmark data on their website.

Hi,

The HD being hit is the storage drive a WD Red PRO.

I've been thinking and replied to the post after yours...do my thoughts make sense or is there a better route?

 

[Tetras]

Hi,

The HD being hit is the storage drive a WD Red PRO.

I've been thinking and replied to the post after yours...do my thoughts make sense or is there a better route?

Hmm, it depends on you workflow and how easy it is to manage the drives, but just from a quick browse at your plan, it seems a bit pricey if the WD Red is the bottleneck?

Like, why replace the Samsung and the Sabrent and is it necessary to replace the whole of the 8TB with a SSD?

 

[Matt_Hirst]

Hmm, it depends on you workflow and how easy it is to manage the drives, but just from a quick browse at your plan, it seems a bit pricey if the WD Red is the bottleneck?

Like, why replace the Samsung and the Sabrent and is it necessary to replace the whole of the 8TB with a SSD?

Actually you are right, change the process, current process:

*Copy SD cards to a folder on the WD Red orderred by Date.
*Synchronise the Lightroom catalog.
*Go through deleting poor images.
*Group images based on time (ready to process)
*Select an image make mods in lightroom
*Paste settings to all images in group and then synch images (this can take a lot of time depending on whats been done to the image)
*Denoise
*Export To Helicon (2 mins ish and hits WD Red 100%)
*In Helicon, render three different images, then may re-render dependant upon output. (this hits the SSD 100% as that is where the cache is).
*Save rendered images in a subfolder of the date folder on WD Red

So if I bought another Gen 4 drive say 2TB using this as a temporary storage drive

*Copy SD cards to a folder on the new NVME orderred by Date.
*Swap catlogs to one that points to the new NVME drive.
*Synchronise the Lightroom catalog.
*Go through deleting poor images, copy ok images to the WD Red so have a copy.
*Group images based on time (ready to process)
*Select an image make mods in lightroom
*Paste settings to all images in group and then synch images (this can take a lot of time depending on whats been done to the image - should be improved as on new NVME drive)
*Denoise
*Export To Helicon (2 mins ish and hits WD Red 100% - this should improve as on new drive)
*In Helicon, render three different images, then may re-render dependant upon output. (this hits the SSD 100% as that is where the cache is - would it make sense to buy 2 new NVME and have one as cache one as temp storage, not sure if a good idea to have cache and temp storage on same drive?).
*Save rendered images in a subfolder of the date folder on WD Red

I really do appreciate your help and thoughts on this - as been musing over it all for a while.

 

[Tetras]

Actually you are right, change the process, current process:

...

I really do appreciate your help and thoughts on this - as been musing over it all for a while.

I know it's not always easy/practical to change things, but if the answer is a 8TB SSD, I tend to think "change the process"