I think (guess), basically, your husband's image/position in his family is very low, he is not respected by his family members from the very beginning (maybe he is the youngest one, maybe he earns less than his siblings....) Whatever u do, his family members talk and talk and talk, your husband may not feels they are talking about u but TEASING him, he is stressed from it and turn out he prefers not going with u in his family gathering.
BTW, I try to guess more (based on your message): Are u the one highly control everything and your husband has NO SAY in your family? Has he overcame for his father-in-law 俾首期買樓? Do u think he also feels upset/no position in your own family? and therefore he keeps going to his home?