Has your 360 been connected to anything else through VGA? if so you may have set the resolution to one thats not supported by the TV
If you can, connect you 360 back to whatever it was connected through (assuming it was connected through VGA to something else) and set it to the lowest res, once its set connect it your you tv and you should be able to set the res from there