Mine does this as well, even though I have 'When firefox starts' set to homepage.
But it only happens in specific circumstances...
If I browse away from my home page, and then start private browsing - then if I just exit the browser window instead of stopping private browsing, then the next time firefox starts (even if i've turned the pc off in between) then it starts with the last non-private-browsing web page. If that makes sense!