That's definitely the sensible option. My wallet will thank me.There's not that much difference between a 4060 and a 6750 XT. 4070 is a bigger jump, but it is also around £200 more than a 4060 for a non-Super (£450).
Can you not give it a try first? I think you'll find the 4060 is a decent upgrade on a RX 580 (TPU's GPU database has it as 113% faster).
Fingers crossed!