@Tefal
In my opinion, they were just plain rude and no, you're not being over-sensitive.
It's not quite the same, but I had a similar situation in my youth. A branch of my family were of German origin, and at family meals we'd all chat happily in English, but there'd be periodic exchanges in quickfire German, which they knew the rest of us didn't speak. Well, it took the investment of quite a bit of time over a lengthy period, but I studied German. A couple of years later, when they did it I waited for a suitable comment, and then joined in. The chagrin was obvious on their faces, especially when I refused outright to tell then how long I'd understood what they were saying, which in reality wasn't that long, but I let their imagination roam free on just how long it may have been.
The expressions on their faces as the tried to remember what they may have previously said thinking it was private was priceless.
Learning your friends language might be a tad extreme, but a bit of effort in picking up a basic vocabulary, and then, making it clear you understood
some of what they were saying, then 'pretending' you really don't understand and, as I did, being very coy about just how much you do or don't understand, should put a stop to their antics.
The objective is not to understand them, or to convince them you do. It's to undermine their over-confidence that you don't. It's to plant seeds of doubt.
That principle, by the way, was an extremely valuable business tool for me and I'm extremely grateful to my German family for unwittingly giving me such an edge, by planting the idea. For at least 15 or 20 years, I made sure that business contacts whose language I actually knew pretty well were convinced I knew nothing more that hello or goodbye. You'd be astonished at what you can learn if you keep a blank expression, no matter what they say.
Even normally guarded and cautions people can be remarkably unguarded if they're over-confident.