Maybe they reminded him of his sister or of a trauma perpetrator he has not told you about. I have a hard time with people who are physically built like the perpetrators of my traumas.

Maybe they said something in the past (made an off the cuff remark, even) that made him nervous or triggered a painful memory.

Maybe he has this idea that your family won't accept him if they knew his sister suicided. He may think that his sister's actions reflect on him and his family.

Maybe he thinks those people are loud and obnoxious.