Interesting. Actual literal meaning of the phrase is “outside of the main work/event”, meaning the little canape thingies are not part of the main meal. So you basically called the person’s mother an irrelevance. Can’t see a problem with that. Like, it’s not like it was face to face where the sound of the words is important …