I'm feeling very much the same as you. This is my first proper Samsung phone I've had in quite a while, the last being the Galaxy Nexus (built by Samsung, but not really a Samsung phone). I'm actually annoyed that I went for the Note 4, it's full of bloatware, it's laggy, and the updates are slow.
In comparison to my HTC One M7, I feel like all I've got is a bigger screen and not much else.
I wouldn't even mind rooting it to drop a custom ROM on to eliminate nearly all of my gripes, but finding ROM's for the Exynos variant that aren't full of pointless features is also providing difficult. Plus I can't even root it due to it tripping KNOX and invalidating my warranty.