SO, are you punishing your kids for the sin that is your husband's? he did not give you enough love, so you don't love your kids? sorry, but really wanna know, what kinda mother you are?? try to think if you were you son, how would you feel??
beside, one day, soon or later, you so-called boyfriends must find out you have 2 sons, and what he would think when knowing that you would banish your own sons for him, a man?! would not his assume that you would do the same thing for baby that you may have with him?!? would not he back up from you?
I am a single mother of a 6 years old, 27 myself. I don't see myself 'old', I still have a life, not easy thing to be a single mom, but this is something I have done to make myself what I am now. So, stick to it!
Don't demand people must understand you, but it is the road you have chosen, and so you must go on, with you responsibilities. Especially to the kids.
With you banishing your kids like this, what makes you better than you ex?? He might not be a good husband, but still yet to know he is a good dad?? and frankly speaking, that is not your say now, since you have shown not a very good mother yourself.
Let him at least take care of the kids, or you must consider more for your son rather than only yourself.