Nvidia have not gone anywhere near as dense as they can on the node, GP100 is far denser than GP104. That is the main reason why they can have such high clocks. IF the density was closer to that of GP100 then GP104 would more than likely not clock as high as it can.
Now from it appearing that 36CU's may be the full P10 part, it appears that AMD have not gone as dense as they normally would with the new node, which means we may see RX480's with higher clocks than past parts. could go as high at 16-1800.
Depends how packed the transistors are together, Amd packs them tighter together from what I heard but I could be wrong.
On any given process you can design the chip differently, different density, there are different choices of metals possible, different transistor designs, some are thicker, attain higher clocks or leak less. So given a specific chip design you can apply that design to a process and achieve different things, denser, higher clocks, lower leakage, lower power, etc. As you've mentioned 1080 has a low density for the given process. But architecture also dictates clock speed, a architecture is designed for a given clock speed.
So where lets say Polaris can maybe run at either 1Ghz or 1.3Ghz base clocks based on implementing the chip differently on the process, larger die, less dense, different transistor design, another architecture, lets say Tahiti, might have options of 1.2 or 1.6Ghz using those different process implementations.
Don't forget that Maxwell ran at significantly different speeds to Kepler on the same node, while all of AMD's chips had a different design and obtained a different range of clock speeds using the same process as Nvidia.
Two things dictate clock speeds and generally there is no wrong option, higher ipc/lower clocks, lower ipc/higher clocks are equally good choices.
