if i were u, if a married man wants to ask me for dinner or frequently message, i will turn down all his requests and say no directly and firmly. i won't attend any of the one-on-one dinner /lunch regardless of whatever reasons he give. you could have done the same upfront, but you chose not to and still cared about him.