What is the actual wording? Is it stating the interview dress code is smart casual or that the workplace dress code is smart casual?
If workplace is smart casual then I would still wear a suit. There is a small chance this could go against you but it's a much lower risk than being marked down for not wearing a suit when the other candidates do. In an interview being underdressed is worse than being overdressed and showing that you made an effort.
If the letter states that the interview is smart casual then I would go for trousers and a shirt. I wouldn't wear a tie but I would wear a sports jacket or blazer if I had one (a sports jacket in this context is simply a more casual version of a blazer - not a leather jacket etc). If you don't have a sports jacket/blazer then remove your casual jacket before the interview or wear a v-neck jumper if it's cold. Polished leather shoes, probably in brown.
If unsure then always wear a suit (unless it's for a building site etc!).