I went for leisure trip with my husband earlier this year when my kids were 4 and 1 year old. I do not think it a problem as long as you arrange the baby sitting well. That time, my youngest kid was too young to be unhappy. Even for my 4-year old girl, she did not have any negative feeling either.