The agent keeps the passport to protect their interests. The maid usually needs to pay agency fees before getting a new job. Unless payments are all cleared, then the employers could on their maids' behalf to ask for the safekeeping of the passports at home.
I intend to do so as my local maid just finished the last payment this month.