It's not about what one government can do. It's about what happens over time - the slow erosion of rights. It's not certain to happen by any means, but leaving the EU opens the door to the possibility.
That's the basis for the opposition to Tory plans to withdraw from the HRA too. The assumption is that a replacement 'British Bill of Human Rights' may not be so robust and that it could be further eroded over time by lots of small changes that pass the majority of people by. We all know most of the UK populous have little patience for politics, and they tend to miss such small changes happening.
I'm not saying any one political party will do this, but leaving makes it possible. Look at how many small changes have been made in the past 6 years and ask yourself; would people have accepted it if those changes were made immediately?
But you could make that argument about literally anything couldn't you? what if successive governments slowly bring back the death penalty, what if successive governments slowly bring back corporal punishment in schools, What if successive governments slowly bring back slavery etc.
There's only one party who tends to think workers have too many rights, the Tories, and whilst then I understand can an immediate worry about this issue, can you really see a Corbyn led government eroding rather than increasing worker rights over its tenure, or any Labour government for that matter?
I'm more worried about an actual, demonstrable, erosion. That being our sovereignty which has massively changed since the idea of a "common market" has transformed into a pseudo-super state without any of the electorate having any say over the matter again until now. At least we are guaranteed to be able to remove an unpopular government every 5 years if they screw us over, when would we get another chance to leave an ever demanding and legislative EU? You may like the EU's social laws now, but what about in 8 or 10 years' time? Just assume they'll always do stuff that aligns with your views forever?
Also, Corbyn and McDonald's current proposals (that would massively strengthen rights) are actually illegal under EU rules so they also restrict worker rights, contrary to popular belief.
Finally, do you think TTIP, which the EU is pushing and getting signed behind everyone's back, is good for workers? I think you should be more worried about that than whether Osborne will scrap the working time directive (which we have an opt-out anyway and currently pointless).
....
ISDS
There are particular concerns about a set of controversial rules expected to be included in TTIP, which would give large corporations the ability to sue national governments in secretive offshore courts.
Within similar treaties worldwide, these investor-state dispute settlement (ISDS) rules have already been successfully used by corporations to sue governments in cases where they feel they have not had “fair and equitable” treatment.
The inclusion of these rules under TTIP, campaigners say, opens the door for corporations to sue EU governments whenever national regulations are changed in any way that impede a corporation’s “right to make profits” – and this includes labour standards.