Well, you just don’t call them any more and you don’t visit them. If they call or visit you then be friendly with him and listen to them and stuff. But then continue not calling them or visiting them. Eventually they will drift away from you without anyone getting hurt.