iMessage is detected when you select a contacts name at the start of the message, if they have 3G or wifi switched iMessage will detect this, if they don't it will be sent as a normal sms and cost you.
iMessage waits for a predefined time before attempting to send as sms (I think there is an option to turn this off in the message app settings) so for example, if your friend has wifi on, you send an iMessage but they turn wifi off, iMessage will keep trying to send the iMessage a few times before finally sending it as a sms