If you're talking about the road race, Boardman most certainly didn't call him favourite, throughout the race he was saying he's not at all known for one days. He's only ever won a single one day and I think by the way he said it, it's a smaller one without much competition or something along those lines.
Again though the track wasn't good for him, his best rides in the TdF were doing amazing climbs where he pulled away from the pack but that was still mostly against the peleton/other overall competitors there were often break aways in front of him.
Cavendish is much more known for time trials than Froome? Cavendish himself will say he can't do time trials, he's a freaking sprinter. Cav and Froome are guys who win stages or tours because their team sacrifice themselves to get them in position to do either. Without a team(like a time trial or one day) neither are anywhere near as strong.
As said, a lot of the guys focusing on the time trial were dropping out of the road race to not take anything out of themselves. Froome did okay in both because he wasn't too likely to win either. He could have dropped out of the race after 20km like others did, or gone as slowly as Cancellara did. He at no stage of either looked strong enough to win.
Also again, winning 3 or 55 TdF's means precisely nothing when it comes to the olympic single day race. You can win every TdF ever and never win a stage. TdF is about stamina, endurance and ability to survive multiple stages and do the overall faster. Being fastest over one day and being fastest over more than a week are entirely different things. In terms of a single day race compared to an entire tour, you're talking about 1600m vs marathon, one is a sprint, one is a long endurance race. Oh and if Froome is a marathon guy, and Wiggins is a 1600m specialist, Cav is the 100m guy, nothing else. He's an amazing sprinter but if you think he's going to win time trials or tours, wow. His teams get him to the final say 100-500m max then release him from their slipstream and he goes full on Bolt on top of a bike to the line.