I've done everything I could think of, but I've concluded it's the CPU and possibly the motherboard, I spoke a bit too soon on the RAM, which is fine, so at least I don't have to return both things.
New CPU + old RAM, old CPU + new RAM, old and new installation with either combo, tried redoing graphic drivers with DDU, including different versions, even updated motherboard BIOS (which I did after I started getting crashes), which didn't help, and after going back to 5800X, the crashes were still happening, which was concerning at first, but that went away when I flashed the previous BIOS back, so basically I went back to the exact same CPU + motherboard BIOS version combo I was using before.
It's possible that the motherboard (Gigabyte B450-I Aorus) is picky, it's very rare that CPU is the issue, but at least I have things running stable again and I don't want to mess with that, 5800X is still decent enough so I'll stick with that for the time being (but also because with the current BIOS, 5900X and 5950X are the only upgrades I can do, 5800X3D requires newer BIOS, which gave me issues as I mentioned above).