This is true. The people who take risks to save seconds do baffle me.Yeah it can go both ways but rarely is the difference meaningful IMO - unless it is going down to the wire trying to catch a train or something like that. Though I've done some not great overtakes to get past people doing 40 MPH everywhere, even though it probably made about 2 minutes difference to my journey, because they are so annoying to sit behind.
I tend to set my cruise at an indicated 79 (75 GPS) and I fully admit that it's mostly for the psychological feeling that I'm making as much progress as I dare versus any tangible time saving.