Not worried about the refs; I'm sure they'll be professional. And as you say, it's mostly (some of) the English that are stuck in the "Two world wars" mentality.
100% this, I know a lot of Germans and it always vastly amuses them that we have such an England v Germany thing in all avenues of life because its just not something they think about, they dont view us as rivals etc so they always chuckle about it being a big thing for us.