The weather for Reykjavik in November is overnight lows of -1, not exactly cold, just be sensible, just take a few layers and you will be fine. No need to go mad buying special base layers or thermal pants, even for the northern lights tours.
OP what hiking have you planned? It depends on what you are doing really depends on what you need but it shouldn’t be that cold. The most at that time of year a cheap skiing jacket and trousers will be more than enough for an organised tourist trip. You can normally get a jacket, trousers, hat, gloves and socks for £70 from mountain warehouse. That will be good for a week in the Alps let alone a day trip in Iceland.