Aye, depending on the shape of the recess you risk the body filler coming out - you can stick a screw in with the head standing proud, which would anchor it fine but probably be just a
touch awkward if you need to drill it in the future!
I'd go with WillyNelson's idea, spend a bit of time cutting a piece of wood (or even several segments if necessary), slap wood glue over everything and knock it into place. Then you can sand it down to whatever shape you need to, and drill/chisel in the future with no problems.
Hell, you could even use Gripfill or it's equivalent to hold it in and act as a bit of a filler if your replacement wood bit isn't perfectly shaped - the pink stuff is bulletproof! Where I work we have a lot of disabled access bathrooms put in Govt. buildings, so you can just roll a wheelchair in under the shower and the floor is shaped so the water drains away in the centre.
Our Clerk of Works once went out to inspect one while it was being built - the guy was doing the floor with marine ply and using pink Gripfill to hold it together.
When the Clerk of Works questioned him about this, he got a withering look and a "Son, I've been in the trade for 40 years - if pink Gripfill wasn't the dog's danglies, half of Belfast would have fallen apart by now!"