Media Player will create a folder.jpg in each of your folders. It will be a hidden file, but it will be automatic.
I haven't used WMP for music in ages, but I seem to remember that when it extracted embedded album art, it would automatically resize the resulting folder.jpg to just 200x200px resolution. Worse, it had a nasty habit of overwriting the embedded art in the files themselves with the resized image without so much as a by-your-leave, which could be annoying to say the least if you've spent ages collecting and/or creating hi-res album art for your music collection.I think I'll give that a try.
The Jpegs need to be unhidden and I think someone has written a batch file that will do it.