You aren't auto set to pvp on a PVE server, It just seems like masses of people are enabling pvp for the fight. I have done about 6 invasions so far and never once been pvp enabled.
Just to post my 'cheese' strategy if like me you ended up in the middle of a truck ton of Horde on a PVP server...
Stage one - No reward given here for killing mobs so just fly above the area and wait until 100% ticks up.
Stage two - Possibility of three chances for Nethershard from the two lieutenants and the commander (think it's 5 - 10 depending on the boss) - used to buy stuff from the vendor. At this stage simply hit the NPC once, run away get out of combat and mount up. You'll still be awarded for the kill.
Stage three - Area will spawn with random demons / portals. Only the larger mobs shown on the world map with the green skull icons give good amounts of Nethershard. Strategy is same as above - wait until opposing faction engage, hit once then run away. Repeat this on all the bosses to gather the most Nethershard with minimal effort.
Stage four - final boss - again wait until engaged then hit once and run away
As you can see, this strategy is not really fun or interactive and involves lots of cowardice but saves you getting ganked lots
I've done this on multiple alts and can confirm you still get the chests / shards.
On another note, if you would like a white bonnet cosmetic item, make sure you grab the locket from within the Invasion Scenario: