there is nothing wrong with public breast feeding
To you! There may well be something wrong with it to some people. I'm not sure what is hard to understand about that.
are you being inconsiderate by not bowing towards his personal ideas of what is proper conduct within that establishment by removing your hat?
Of course it's inconsiderate. Imagine the alternative. To simply refuse would be to know you are upsetting/making him uncomfortable and to make a conscious decision that you do not care about that. I'm not sure how that is anything but inconsiderate. If there is some major inconvenience to taking off the hat then fair enough, but if it isn't any skin off your nose then surely it is the decent thing to do to simply consider his feelings and respect his request.