Fishing is most definitely a sport - I think it's the biggest participant sport in the world, yes, even bigger than footy. Fisherman have all sorts of competitions all over the world, and for big prize money and very lucrative sponsorship deals too.
Anyone who says fishing is not a sport doesn't know what they are talking about I'm afraid.
You can treat fishing as a hobby or pastime, same as anything else really, depends on your perspective. Some people just like dropping a line in every now and again but some people are competing practically every week.
Not physical? Ha, anyone saying that just hasn't done it, or they've been catching minnows all day. I've ached for days after fishing sessions; arms, shoulders, back, legs. Playing a good fish for an hour on whatever gear certainly takes it out of you. Doing it twenty times a session can totally wipe you out.