In the least offensive way of speaking, people look kinda a bit alike 
so it would make sense that people might forget your face! if you do actions that make people annoyed the less personal they are they will probably just forget it!
the more "personal" an action is the longer they will remenber you, for example if you make a cop hostile because you broke some garbage they will remenber you for less time than if you ran them over with a car and then started throwing banana peels at them.
if a person SORTA remnebers you they will not be neutral but rather Suspecting, this means that upon being around them long enough or talking to them they will go "HEY WAIT A MINUTE" and return to their either annoyed or hostile state.
this rewards less direct brute force confrontation and rather more planned actions.