it seems to be so random this is hard to doCould it be a DNS lookup issue.
If download/refreshing a page is quick, if it's just the initial/first time that's taking too long then that would be my first guess. Easiest way to test for that would be to try a website using the IP address instead of the URL.