The mod adds so much compared to vanilla that I'm actually not sure. To me it's a bit like playing Skyrim without mods... except the base game leaves even more to be desired. The guy that created the mod, Litcube, spent 7 years designing and refining it.
Perhaps it is worth playing Litcube's Universe first without the Mayhem mod, but I don't know about vanilla... I actually never got to the endgame in vanilla because it didn't seem as compelling as the massive overhaul that is Litcube's Universe, which is why I recommended it in the first place.
Perhaps I can list a few core features of the game and how Litcube's Universe improves them?
In X3 you can build your own starbases and run a living economy, but each station requires power and resources (food, minerals, metal etc) to function so you have to link them together with physical tubes (that things can collide with). The issue with vanilla is that you require more and more stations linked together to make decent a profit for your empire, eventually culminating in a ton of them spammed all over a star system (all rendered in 3D when you enter the system yourself and want to build anything, of course). With Litcube's Universe, you can build structures called Saturn Complex Hubs, which function as huge starbase complexes, hence the name, but need similar resources to run as the normal starbases. That eliminates the need to have FPS-reducing separate starbases everywhere.
In vanilla you constantly have to worry about colliding with friendly units in space, which is usually deadly and extremely frustrating. The "bounce" script in Litcube's Universe mitigates this somewhat.
You can use chaff as a countermeasure in Litcube's Universe, where as in vanilla you just had to pray you could out-manoeuvre the torrent of fireworks launched in your general direction.
There's a rival corporation called Phanon in Litcube's Universe which that is attempting to beat you to the top of the food chain by building its own starbases across the universe, and it generally has a lot more tools at its disposal than the other factions to wipe you out.
There's a malevolent force called Omnicron Channel Virus (OCV) building up in a certain section of the universe. They are able to wipe out entire sectors and eventually the universe if you don't stop them. Their ships (which cannot be piloted by humanoids... or you) are massively powerful at every tier compared with other racial pilot-able ships.
Full list of features
here.