It's surprising how many times in the last two weeks I've wanted jot something down and the S Pen and pop up notepad would have been ideal.
Also, the camera app is much better than standard Android, I do occasionally use some of the features (like slo-mo for example)
On the flip side, Lollipop is gorgeous and so fluid. I also put more weight on speakers than most, as I tend to listen to 30 minutes or more of podcasts each morning. I actually like the extra 0.3" of screen too...
In the past I often go with the safer choice, which would be the Note 4. There wont be times missing the 0.3" will stop me doing something, or the nice loud speakers, or the slick and pretty lollipop... but there will be times I want to make a quick note and can't or film a slo-mo bit of action and I can't (granted, the slo-mo is very seldom used/needed)
But then, the Nexus 6 is just soooo nice...