don't worry, sometimes I have the same feeling too. I told myself, he is still young and doesn't know who is who and how to express himself, the situation will be better when he grows up more.
Besides, I have a lot of satisfaction when I could bring him to sleep peacefully at nite, and after breastfeeding. Sometimes, I just don't want to put him into his bed and would like to carry him longer.