I have similar problem but my OB said it is ok cause it is a hormone problem only. She said we may have m back as soon as 5 to 6 weeks.
Some BF mommy said they get m back later but my mother-in-law said she also BF my husband and got m back around a month. So it varies from body to body.
But it should be ok. Don't worry too much cause anxiety will affect milk supply too.