Why does my nephew act like this? please please help?

Because that's what he's use to. I'm surprised he didn't react to you smacking him. Probably in him automatically to stop with the first or he'll get beat. Cause at 20 he could of done something about it. Besides putting up with his abusive mom he could of also had to put up with several abusive men. He is troubled seems like he wants whats best for his son even sought you out after all these years for help. I do think he needs to be told that this behaviour well not be tolerated. Also some professional even just talking to some one in confidentiality like a therapist could do him some good.