I bought myself a Phantom 3 at Christmas and have barely used it.
I live in London, so you can't really fly it anywhere legally, the royal parks also have warnings about prosecution for flying drones, so even open spaces are pretty much a no-go.
I only tend to use it when I'm going travelling somewhere and will be in an area I know I could use it.
Weather of course does play a part in being able to fly, which was actually something of a downer when I actually had some time that I could fly it over christmas. You can't fly it when it's raining. The real worry though is wind, I quickly found that picking up a little hand held anemometer is useful for determining whether it's safe to fly it.
A drone like the Phantom is pretty good at holding itself in position in reasonable winds, but it's using a lot more battery power to keep itself in place. If you switch it into one of the more manual modes where it's not automatically adjusting to keep in the same spot, you can really see what the wind is doing when it quickly drifts away from you. It can then be pretty slow coming back to you if it's flying back against a head wind.
It's fun to fly, but I probably haven't justified the cost of it yet.