May I ask if you followed a guide to OC your 5800x?
I also have a 5800x bit with a EK 280 AIO and wouldn't mind trying to achieve what you have also
What board are you running?
I looked up what others were getting within reasonable limits and trying to understand AMD's alien overclocking ways since it is still ingrained in my head the old school way of overclocking.
Essentially it will vary by chip but also not all are made equally in terms of being able to be cooled, I had a terrible 5600x which was actually worse than my 5800x for cooling, it ran like an absolute inferno even underclocked.
The board I use is a Gigabyte x570 Aorus Elite, nothing too special but it does the job.
The biggest wall for me is the guessing game with voltage and power limits, I absolutely hate the control that has been taken away from me, sure I could use an all core locked overclock but it won't be boosting to over 5ghz in games when the CPU takes advantage of the lightly threaded situation so whilst all core locked overclocks are better for all out straight line performance, on Ryzen it is the boost clocks that offer better gaming performance, not that the 5800x is too slow to begin with.
I don't have a real set out rule guide for how to OC a 5800x, I would first try to reach the limits of your cooler, allow your chip to throttle and work around the limitations of the cooler and at what point voltage and power limits adversely affect boosting of the CPU.
For more information on the details of Ryzen's PBO overclock mode, go here:
https://www.reddit.com/r/Amd/comments/qik4t3/zen_3_pbo_and_curve_optimizer/
I will warn you, if you use curve optimizer it will drop performance even if your overclock is high, it stops the CPU from boosting to it's full potential under certain circumstances where power draw is very high.