If your baby sleep very late at night, normally is because he sleep to many at day time. Try to wake him up around 7-8pm until around 11-12pm. You can turn on the TV and you can talk with your family members as usual (don't keep silent), and remember not to turn off the light in the room, in order to make him cannot sleep well in this period
Also, try to feed him to drink more milk at about 11-12pm, may be 1oz more, so that he can sleep much better and longer time at night.
You can try for a few days, he will get used to this habbit and sleep better at night.
Oh, my baby also cannot sleep well if we have go out or somebody come to our home on that day. May be when someone come, your home become noisy and the baby was not used to it. And he feels afraid at night, so he cannot sleep well.