The crystal/diamond stuff isn't part of the building structure, the same reason things like the skulls stay on the ground and in the water just outside, the building resets everything that was in it at it's design/construction/inception but not necessarily things added afterwards.
The Doctos is affected because he's in the "memory" of the transport chamber which was there when it was started, but the TARDIS is effectively sealed separately and possibly arrived after that time so is not included in the reset.
A bit like if you make a disc image of a computer then add a card, if you use the disc image to reset the computer it doesn't have the drivers/software for the card s that was added after