I think it is about focus and attention.
I find whenever I play a proper sandbox game my attention drifts. For example in GTAV I spent an hour seeing if I could kill all of the police that came after me when I shot a civilian. After a while I got bored of that so went to see what happened when I threw myself off a mountain on a motoX bike. Then I got a bit bored of that too. Then I turned it off to go and have my tea and have not been back to it. That was ages ago, not sure exactly how long but I guess at least a few months (if anyone has me on the social club thing they could probably check?).
The story is a bit slow and less focused than most other games. This is the nature of the game but for me all that happens is I struggle to remain focused on the story and get distracted by all the other stuff I can do. This completely dilutes the effectiveness and immersion of the story and characters as by the time I get back to it I am struggling to remember what is happening. It is a bit like reading part way through a book then coming back to it 3 months later.
The problem is not necessarily with the game but with me as a player. I mainly play online FPS games so I am used to fast paced, focused play with definite objectives and more intense gameplay that lasts for 10 or 20 mins then onto the next map. When the objective is basically "do whatever you want" I feel at a bit of a loose end and whilst I don't want linear gameplay, I also don't want complete freedom because my attention span won't keep up with it.
On that front I felt that Games like Deus Ex: HR offered a good middle ground, as well as Bioshock titles. I loved Skyrim and the other TES titles, though, which is strange!