I can understand your feeling and though you know what should do, you will find difficult to determine because you would care more about the feeling of your son rather than whether your husband cares the little.
I remember I had read once here from a 高人 and you may take into your consideration. Below the quotes for your reference.