You have 8 slots, max 4Gb per slot, max total of 32Gb.
Unlike DDR3 Mac Pro's in 2008/3.1 MP it has to be 800 MHz, must be FB-DIMMs (Fully Buffered) and due to design of the air tunnel all chips have to have heat sinks or the memory will run at crazy temps and become unstable. And not the flat PC heat sinks but proper "hedgehogs". Check how the raisers are positioned in the tunnel, you'll quickly figure out what design is needed.
Now, few years back the prices were quite crazy so I decided against spending small fortune on original Apple memory and instead bought three sets of 2x4Gb Samsung FB-Dimms (server spec) and a lot of 6 cheapest damaged/non functioning chips with heatsinks from old mac pro 1.1 on flebay. I then stripped the heat sinks off the old, dead chips and applied them to Samsung DIMMs with new thermal paste. It's an ugly, long and painfully DIY job as the holes and notches for the mounts and clips in Apple heat sinks don't match any regular patterns of third party memory, so you have to create your own spring clips and make sure everything is positioned correctly and tight for low temperatures while minding spacing in slots (I used some extra cable ties around heat sinks to make sure nothing moves). I'm not saying do it, as you will hate every moment of it, but once done it works fine, stable and without any issues.