According to the law, it's illegal to ask your maid to work at other address, but I know many people are doing so. The better way to do, if possible, is to ask your mother to go to your house.
If I were you, 我都唔放心 the maid alone to watch the two young children. They are too young to tell you anything.