The unfortunate truth about FIRE is you do need to earn significantly above average and spend significantly less, which with kids becomes impossible. However, it is not black and white: saving earlier will make up for lower salaries, and you don't need to FIRE at 40 but maybe at 50-55, or reduce working hours etc. IMO, there are benefits to taking a light weight approach to FIRE where you try to invest, not splurge on overly expensive house or cars, and will eventually become much less stressed over Finances even if you are not independent and require a job.
You say you are directing all money is going towards housing. This is something to look at. If you already have a house but are trying to be mortgage free it is likely better to keep mortgage payments at a minimum and instead investing. If you are saving for a down payment it makes sense to carefully consider the size and location of houses and try and minimize mortgage