I kind of get where you're coming from here. I'm not work/career driven, so I don't feel guilty at all about not working more/harder. I do however feel guilty when I do 'me' things when there's 'non-me' things that need doing.
I like to get out and about; walking, running, cycling, paddleboarding, and gyming. If I'm not feeling energetic enough, then I like to **** out for a couple of hours on the xbox.
The problem is I've got two young kids, so there's always parenting, cleaning, cooking, diy'ing on the to-do list, which makes me feel guilty if I go out and do 'me' things.
I don't think it helps that my wife doesn't really have any 'her' things she does (she's happy playing with the kids every waking minute), so that makes me feel more like guilty as I'm always the one going out leaving her at home.