I think the biggest issue is other countries validating it.
It could be forcing the Queen to formally put her weight behind it:
Her Britannic Majesty's Secretary of State Requests and requires in the Name of Her Majesty all those whom it may concern to allow the bearer to pass freely without let or hindrance, and to afford the bearer such assistance and protection as may be necessary.
Or it could be that it's a lie (see below).
What if you have both/neither?
What if you really don't feel a man even though you have a willy, or vice versa? Unfortunately life isn't that simple for some people.
It would be a lot simpler if it wasn't for the common insistence on pretending that sex and gender are the same thing when clearly they aren't. They're only vaguely and indirectly related.
I'll use myself as an example.
In my youth, my appearance was generally more feminine than masculine. When dressed in an ungendered manner, it was often assumed I was a woman. Sometimes even straight on face to face and very often from other angles. I had very long hair. My natural speaking voice is about medium pitch for an adult human, so that's not a clue to my sex. On the phone, I'm still frequently mistaken for a woman. I've heard recordings of myself on the phone and if I had to bet on it I'd put my money on it being a woman speaking.
That's all gender and roughly around the middle of the gender spectrums for those things.
I have worn makeup and perfume quite often. I've worn a skirt on occasion. Those things were gendered extremely strongly feminine in that time and place, but they're still gender.
I've always been male. That's sex. No matter how feminine I might be in gender in any way, I would not be female. Sex is physiological. Gender is social, mostly just fashion. They're not even remotely similar things, let alone the same thing. Also, gender isn't one thing. It's very common for a person's gender to be different in different aspects of life. I'd go as far as saying it's universal. I doubt if anyone is masculine<-->feminine to exactly the same degree in all things at all times. Even if the person doesn't change, fashion will. Playing video games, for example, used to be very masculine. Now it isn't.
A person can choose their gender and change it by their own actions. They can't choose or change their sex without quite major medical intervention. If I said I was more feminine than masculine, as an average of the gender spectrums for all things, I might be correct. I could easily change myself to make it correct. If I said I was female, I'd be lying or deluded (depending on whether or not I believed it).
All this bazillion gender=sex labels is wrong on a couple of levels. Gender and sex are different things and any single gender label for an individual is inaccurate because gender is a whole load of spectrums and not something that can be accurately summarised. What if, for example, someone is gendered strongly masculine in one thing and strongly feminine in another thing? What weighting should be assigned to each thing in order to decide on an overall gender? Does an overall gender make any sense anyway? I think it's like looking at an area of land containing a ridge 500m above sea level and a valley 100m above sea level and declaring that every square metre of the whole area is 300m above sea level because that's the average elevation.
Getting back to the passport thing...why not just remove the field entirely? Ignoring the pretence that sex is gender and vice versa and all the blathering that comes as a result of that misconception - is it genuinely relevant to the function of a passport to note the holder's sex on it? If not, just remove that field.