Dear ohbasan2000,
I had been there before so I know how you are feeling.
I'd like to share my thoughts with you.
The first one is if your husband is Cantonese then you've got a good tutor.
The second one is if you don't mind that people think that you are from Mainland China, you can still speak Mandarin because most people today in Hong Kong can speak Mandarin. The third one is some Universities open two kinds of Cantonese classes for foreigners, one is for English speakers and the other one is for Chinese speakers. For instant, Chinese University has both of them, and she only opens one class for English speakers, because she can’t find enough students(8) to open the other one.
Hang in there, I know you will get through it in no time and good luck.