36mp D800E Raw's only 10mb (Faster Processing - Less Storage requirements)

[An Exception]

Processing D800e raws wasn't a huge issue.. but I could definitely notice the difference compared to my D700. Everything took 3-4 times longer. So my OCD and unwavering determination kicked in, and I began to look for a solution or work around that would allow the same control that I'm used to (Jpeg was not an option), but with better performance.

My system

Now my system wasn't slow by any means, however I recently downsized my main processing rig from a 4ghz+ i7 + raid SSD's, to a retina mbp 8gb ram with SSD. I only have a 512gb SSD which would have got filled up in no time considering the size of D800 raws. Because of this, I was planning on using an external raid 1 to store all my images. Ideally I also wanted the redundancy of raid. However I'v been impressed with time machine, and daily backups to different time capsules might be a solution.

My first thought was 'more hardware' was needed. However I'm pretty skint at the moment, so that wasn't an option. I tried different camera settings such as 'lossless compressed' and 'compressed' which got file sizes down to around 45mb & 38mb. This was better but I still wasn't satisfied, so I turned to google.


I tried out the DNG converter's lossy compression, and downsizing.
Weirdly I found downsizing actually gave larger files than full res, unless you downsized to less than 12mp.

12mp = 8mb
15mp = 12mb
20mp =18mb
36mp =10mb

Workflow
In practical terms, I will be converting Nefs strait from my SD card into a folder on my desktop. My rmbp tears through the conversion process, so it doesn't actually take any longer than importing in LR. Once images are on desktop, I then use lightroom to import into the catalogue. This takes only a matter of seconds as files are small, and being read from SSD. Once in LR, I can use all my presets and make edits etc. as I would do with a normal Nef file.

Free lunch
No. Obviously the lossy compression conversion throws some information away. The question was how much, and can I live with it?
To better see any differences, I used an under exposed Nef, and applied the exact same adjustments in LR to correct the image.
My findings, were that the dng's had a little more colour noise in the darkest of shadows.

Original

DNG

NEF

Original - 100% crop 1

DNG crop 1

NEF crop 1

Original - 100% crop 2

DNG crop 2

NEF crop 2

Performance
System is more responsive and snappy, it's too early to say for sure, but I would say these 36mp files are easier to work with than 12mp D700 Nefs.
Also while there is reduced quality in the shadows, there is still far less noise and banding than there would be with a D700.

Conclusion
Am I going to convert to DNG every time?
No, if I'm shooting a landscape, I will want every last bit of quality I can squeeze from the sensor. For wedding work however, when I'm dealing with more than 1K images, this will be very useful, and I don't need the absolute best quality possible, or even the whole 36mp. However as is seems to work better by retaining the same pixel count, I will gladly take the extra crop-ability.

Edit:

I have to test further, but high ISO files seem to be around 19mb.

 

[An Exception]

Have you compared full res fine jpgs to the compressed DNGs though? Considering that there's a lot of lossy compression with the DNGs I'd imagine them to retain a similar amount of info as a full fat JPG.

Jpg is completely different. DNG is basically like a raw file. No jpg processing is applied, no blurry noise reduction is applied. I can use all the tools and adjustments that I'm used to in LR, such as camera calibration, wb balance adjustments etc...

Also I thought you hated Apple products? Why the rMBP all of a sudden?

It's OT, but I don't hate the products, just hate underhand business practices. Just happen to hate Microsofts business practices + windows 8 + PC laptops don't compare. The lesser of two evils.

 

[An Exception]

Example test files can be downloaded below.

DNG 11mb

NEF (lossless compressed) 43mb

Edit:

Just exported a Jpg from LR at maximum quality. File size was 16mb, that's more than the DNG.

 

[An Exception]

Just curious....if you shoot all your photos under exposed like that, it must look horrid on the LCD and make the culling process impossible since you can't see anything until it is processed?

I don't shoot my pictures that underexposed, I was just experimenting to see how it compared to my D7000.

I shoot 0.66 stops to the left (spot metered most of time). I then increase the brightness of my LCD on the camera which makes images look like a normal exposure. My action in Lightroom increases exposure when I import the files.
As I got bucket loads of shadow headroom relative to highlight headroom, I may as well use it.

 

[An Exception]

^^^
Yeh it is pretty good in that regard, I haven't experienced anything like it since the D7000.
If you were shooting a landscape at sunset, you wouldn't need any ND grads.

 

[An Exception]

Just a quick update, if you shoot low ISO and don't embed Jpg previews etc. file size goes down to around just over 6mb, which is pretty insane for a high quality 36mp file.
You can then use LR to render 1:1 previews as a batch which is stored in cache, you can then set LR to discard the cache after 30 days when you will likely have finished working on them, so overal storage and archival demands are incredibly low.

Below is another sample RAW.
DNG File

 

[An Exception]

There is pretty good recovery in the highlights similar to any other FF dslr, but it's the shadow recovery that is spectacular.

In the example of landscapes, there is a 2-3 stop difference from the sky exposure and the foreground exposure. So I would expose for the sky, and use the gradient tool to increase exposure in the foreground 2-3 stops.

Gain there is no free lunch, their will be 'some' reduction in quality in the shadow areas.
As a rule of thumb, if you increase the shadow areas by a stop, the quality of the shadow areas will be like shooting at 1 stop higher ISO.

For example, say you shoot a landscape and expose for the sky at ISO 100. If you need to increase the foreground exposure by 2 stops, then the shadow quality would look like ISO 400.

If you have seen what the D800e looks like at ISO 400, then the difference in quality is negligible at the worst.

 

[An Exception]

I was always under the impression it was better to overexpose as you can pull more detail from the highlights when pulled down than you can from the shadows if you underexpose and pull up. Also I thought that you got less noise by over exposing (as you see more noise in the shadows if you under expose)

A) Did I not read correctly when I read about this
B) Did I misunderstand
C) That advice was plain wrong?


Or secret answer D..... that's the best way to work with that particular body

D.. sort of. If you wasn't using a modern Nikon/Sony sensor, this technique wouldn't be possible, at least not without a much bigger file quality penalty (think banding/blotching).
To capture such high dynamic range scenes in a single frame, is very demanding on the sensor, and until relatively recently the traditional wisdom of exposing to the right, would have been the best policy in just about all cases. Where high dynamic range scenes were involved, ND filters were a necessity.

There is still technically 'some' noise related benefit with exposing to the right, but it's soo negligible, to me it's not worth the reduced headroom in the highlights.

 

[An Exception]

^^^
If your exposing to the right by increasing ISO rather than leaving the shutter open longer or using a wider aperture, then there is no benefit.

Tbh I think allot of photographers will keep ettr out of habit from using older tech, even if there is no perceptible benefit.
Also constantly monitoring histograms etc and guessing if you have blown any meaningful highlights is a slow and hindering procedure.
If your shooting landscapes or still life then it's no biggy, if your shooting people, then expect to work hard to get decent expressions from your subjects, as every time you have to monitor the lcd, your breaking connection.

 

[An Exception]

You would never increase ISO to ETTR, simply a longer shutter speed, which when you camera is sitting on top of a sturdy tripod is no issue. As I said, ETTR is only for static scenes where you can monitor the exposures and histograms.

I didn't see that when I replied, did you do a ninja edit?

 

[An Exception]

^^^
From my experience, the speed of your storage is much more significant in terms of performance if you already have a healthy amount of ram.
In LR, the slowdowns come from disk reads and processor load when rendering previews.
Also faster CPU's reduce the need for larger pools of ram.

Below is an example of what performance is like with full fat 36mp Nefs.