Yes. I think it is better to give her back the passport since they have the right to call police for your keeping of her passport. I saw a memo from their labour union which teaches them to do so.
Some loan company didn't need passport, contract, just need a I.D. card copy. My first one resigned & the loan more than 10 thousand, we got phone call everyday & night, even letters..........anyway trouble