Sometimes is not something that you did, but something that happened to you that makes you hard to trust.
Like, being from Yomi Island. Sure, someone from Yomi would have had to do a lot of messed up stuff to get through it, but it's not just a single action that makes it bad for their rep. Kinda like being a child soldier from africa or something. It's not just one event, it's a whole upbringing.