Sim Swap fraud was a massive thing when I worked in telecomms retail 12+ years ago. We used to see it all the time back then, was pretty easy to shut down in store as we'd just demand a proof of ID before we'd do a sim swap, its a lot more challenging over the phone as if they know if your security details then the agent is ****ed.
My wife was a manager in a call centre a while back for a major network. The main source of failure they had with regards to account security were undeniably via overseas call centres that the company employed. She remembers one report that stated that in 200 test calls 86% of agents neglected to complete correct authentication procedures. Several agents gave account details to callers that the account holder should know (address, DOB etc).
Further investigations also found that a certain 3rd party call centre company, based in a country known for telecommunications scams, was SELLING customer data.
How does this work? Surely if doing it over the phone, the new SIM gets sent to the address on the account? I.e. your address.
You can easily obtain a SIM from anywhere. Easiest option is to pick one up for a quid or so at a local low cost supermarket and then call and give the serial to have it activated. Many operators don't even required you to call, you can activate a SIM via online portals.