it could mean there are some unresolved issues there that you are not aware of. maybe unwittingly, you hurt them deeply and they would need to have that issue resolved before being comfortable around you in social or personal settings. it's not anything negative on you so much as just old baggage that probably needs to come out so it can go completely away on their part.