Well that's not true, is it? If you're on the primary entrance to a building, thinking hotels, office blocks, etc. you'd be neither ground or 1st, or 1st or 2nd, depending where in the world your particular primary entrance happened to be located. You'd be twixt the two initial floor spaces.
The first floor beyond the primary level? So the second one you encounter then?
The problem is that you can't apply the rules of Mars bars or whatever to buildings. You can have no Mars bars but you can't have no floors. Therefore you start on the ground. You cannot think of it as Mars bars/ladders/apples. It is not the same.