I guess you also believe in God. So trust that God will give you the strength to fight this temptation.
One suggestion, every time you see this man, just say this to yourself (in your heart), " I belong to my husband, I belong to Jesus, go away temptation."
I am sure you will get the power from God to help you.