Surly the whole point of defragging your hard drives is to improve system performance? Until the performance of your system starts to suffer and is noticeable to you, why do you need to worry about defragging at all especially individual folders?
In my experience, doing a proper defrag with a decent app (I use PerfectDisk myself) can help quite noticeably with game load times and stutter.