This they took our jerbs phrase is becoming so pathetic. There was someone on TV the other day saying the owner of a bus company has Polish people lined up if any English person disputes pay rises. The owner will happily get rid of them for lower pay. Does that mean this is all lies and they're racists?
No

The issue in that example is the bus company owner seeking to exploit people, decades of anti-union rhetoric, and squeezing employment rights (e.g the two year unfair dismissal rule). If the Polish weren't there ready to take the place of any sacked workers then it doesn't automatically follow that the bus company owner would suddenly start treating their staff better.
Making situations like your example about immigration shifts the focus away from the employer and allows them to get away with it.