Eliminating too much needlessly is perhaps a concern too. Ideally you do this stuff with an IBD dietitian involved (not a nutritionist) and with some guidance from them.
Doctors/dietitians/nutritionists are all in conflict against themselves, I've spoken to numerous doctors and some are in complete agreement with me in that I should not be taking medication, and others are suggesting that I very much should be taking medication. Go figure.
If you've genuinely tried every type of diet and food group elimination to the point where you've even tried carnivore, and are still receiving symptoms,then I very much believe that, of course, you should be medicating. I just doubt many people have tried those things.
We're continued to be fed the lie that everyone needs to medicate all the time when more and more research is suggesting diet and lifestyle can be all the "medication" you need for many types of autoimmune diseases (I know the science is still unproven if IBD is autoimmune).
After 15+ years of symptoms not one doctor suggested a certain food group might be causing me to have these symptoms, not one. Medicate, medicate, medicate - that's all they care about, with the ten minutes they spend with you each year.
Treat your body like a project IMO and see what works for you. Of course, most people won't, because we're all addicted to various food groups.