Perhaps you are misinterpreting what they mean by Castle.
It's an entire city based within castle walls. Within that there is the Temple Quarter and the Trade Quarter, along with Old Vizima and a big Swamp map.
Sewers, Cemeteries, various caves and crypts, and and entire section of the game taking place quite a distance from Vizima.
The game is bigger than you think!
Sure, it's not free-roaming like Oblivion/Fallout 3, but then it was never designed to be, and it's linearity is not as constricted as you think.
One of the most satisfying games I have ever played.