My 2cs,
Careful with using Shader3.0 if you are using a Nvidia card, I personally found it caused more problems then solved. The 3.0 shader is best used if you have a ATI card, still, if you have good results, then use them by all means.
TBM defaults at 1024, this of course will give faster FPS, probably best to use for general flying especially if your PC is already struggling on FPS. You will of course not be able to view or appreciate many finer detail that high res textures will give, whether they be cloud textures (if you are using higher then 1024 in REX that is) high detailed ground textures (some photoreal like NL2000v4 will need at least 2048 to fully enjoy) or HD livery textures on aircraft (like McPhat HD liveries need at least 2048, their UHDT, which I believe will be used on the upcoming PMDG737, will need 4096 to fully appreciate).
If you are using Nvidiainspector to force AA & AF, make sure you have Trilinear and AA NOT ticked in FSX, otherwise it will not force the setting from NI properly. And again, whenever you adjust or change your video driver, you need to redo your NI settings for FSX as it often resets it.
The get the best visual results in FSX (apart from addons/hardware setup) the first thing would be resolution, the higher you can run the resolution with acceptable FPS, the better FSX will look. Second (although it really ties in with the resolution) would be AA & AF settings, this will smooth out your image and textures. Thirdly, TBM, use either 1024, 2048 or 4096, depending of course if your addons/textures/liveries support it, the higher the better visuals. Lastly, your monitor, the better quality monitor/HDTV but also the better you setup your monitor (from connection HDMI/DVI/VGA etc to menu setup) the better quality visuals you will get.
That's all for now.
Cheers.
EDIT: One more thing, again, as I mentioned before. If you install FSX properly the first time around, IMO, there should be no need to reinstall it again in the future (unless something has really ****** up) but that is why I have suggested before, once you get it running properly and satisfactory, make a backup of both the install directory plus all the relevant files outside that directory. Ever since I carefully installed FSX on my new PC at the start of this year, even though I have had some annoying problems. I had NOT had the need to do a re-install. I personally would hesitate to suggest to anyone to re-install FSX unless you didn't install it properly the first time around OR if you got some major new hardware, ie an entire new PC not just a new graphic card.