I hear you. My Note 5 isn't struggling per se, its still relatively responsive its just every so often it will go into a bit of a 'tizz' (for a want of a technical term!). I guess with all the updates, etc its starting to show its age a bit. And the battery has basically given up. I need to charge the thing twice a day (by 3pm its down to 30% with little use). I used to use my stylus extensively (I'm an IT architect) so would often reach for it to show mock up designs, etc. But now its very rarely used. When the Note 7 got launched I actually held back (which is weird for me as I normally jump on the Note bandwagon with every release). Suffice to say with all the problems it experienced I gave up.
I would love to hold out for a Note 8 but my finger is currently hovering over the buy button at CPW