I'm not typical of most people. I use my phone for emergencies only, and it's normally turned off and in my backpack. I don't take it with me around town, nor do I listen to music around town. Both my mp3 player and phone are used most often when I'm out hiking or camping.
If I use all the battery of my mp3 player it's no big deal. I just have no music

Use all the charge on my phone and I need to then find somewhere I can charge it, or run the risk of not having it when I really need it (twisted ankle in the middle of nowhere, fell off cliff face, etc

)
I've also found that phones are the worst culprits for not living up to the stated battery life specs. A phone rated for 48 hours probably only does 15 max in real life usage. At least that's been my experience.
If your phone is more for recreation/luxury, and you take it everywhere with you, then yes, I can see the benefit of using to play music also. For me, right now, I'm happier keeping them separate.
I can't be the only one to take this stance, or separate mp3 players wouldn't be made anymore