You're right, early on food/diet talk can be extremely triggering. You do whatever's necessary to protect your brain: leaving the room, changing the subject, streamlining your social media, even having a conversation with repeat offenders so they understand you're not interested in that kind of discussion.
You've already moved on to stage two though, annoyance. That's good. :-) It means you're not internalizing it as much. It's becoming more aggravating than tempting. It still creates an emotional rise though, which is the not-fun part.
No worries though, because with practice you'll get to stage three. That's where your emotions are yours, other people's food and fitness is theirs, and one doesn't trigger the other. Indifference sets in. That's the stage after annoyance, indifference. It stops mattering what other people do, say, or eat because you're so happy with your way that nothing shakes you.