Yes, it does matter. Anyone who thinks otherwise is a fool.
Does playing an adult-rated videogame turn a child into a sex-fiend/killer? Unlikely.
Does playing an adult-rated videogame have an impact on a child's development and perception of the world? Definitely.
Things are rated a certain way for a reason. We have restrictions to protect children from things that they shouldn't be seeing until they reach an age at which they can fully comprehend and understand what they are seeing/engaging in.
No, a 17/364 year old isn't suddenly going to turn mature and ready in one day. But that argument is utterly flawed. What about a 9 year old playing games about running over prostitutes? Or stabbing people with knives?
Unfortunately it isn't feasible to assign an age rating to each person, so what happens is a universal age rating by which to protect the majority of people.
Surely anyone with more than two braincells to rub together can see that music, television, movies and videogames have a massive impact on a person, just as real-world experience does. To deny it, is to deny that which is right in front of you.