Can you please show a picture of your case and its cooling configuration??
I ran an overclocked and overvolted RX470,and a overclocked GTX1080 in a mini-ITX system with different cases,and found case ventilation is an important factor. Mate had an RX580 too and it seemed OK,but they made sure the airflow was properly worked out.
So its not only the number of fans,but their airflow,the type(high pressure fans are better if the airflow is impeded),whether its a positive pressure or negative pressure fan setup,etc. Even things such as dried up thermal paste can cause issues.