How do you know it's a female colleague? You answered the phone before? Why don't you stand infront of your husband when he is on the phone with that lady? Be frank and ask him what's the matter for her to call him at the odd hours. Any urgent matter from work?
Sorry, I am quite straight forward in handling this. Better to clear things out than to suspect your husband if he is only discussing businesses with his female colleague.