I've encountered hundreds of cyclists on similar rural roads around here. Not once have I been held up for an entire mile behind one. In fact, I don't think I've ever been held up more than 30 seconds behind a cyclist, either before being able to overtake normally, or before they've pulled in to let me pass.
Compared to caravans, tractors, horse boxes, large trucks and elderly drivers, they are about the least obstructive road users I encounter. All of the above are slow and annoying, yet fast enough to justify not pulling over to let you past, so hold you up for longer and for further,. Yet it's only cyclists which get constantly derided.