Not quite, scientifically our eyes are all made up of the same core elements. We all have 3 cones in our retina, S M and L which are short, medium and long cones that respectively pick up blue, green and red. We also all have the same identical pigment called Rhodopsin which is composed of something called opsin (basically a protein) which is linked to retinal, which is a conjugated molecule formed from vitamin A.
After that process it's simply a case of carrying that signal to your brain, which can get confused, but only within the current colour spectrum, you seeing red as most people see blue would become apparent very quickly.
If you think it could work any other way (i.e. people seeing completely different colours not in RGB) then try and think of a new colour for a few minutes and feel yourself get infuriated. Our imagination is somewhat limited by the spectrum of light.
After that process it's simply a case of carrying that signal to your brain, which can get confused, but only within the current colour spectrum, you seeing red as most people see blue would become apparent very quickly.
If you think it could work any other way (i.e. people seeing completely different colours not in RGB) then try and think of a new colour for a few minutes and feel yourself get infuriated. Our imagination is somewhat limited by the spectrum of light.