In Open World PvP, it seems that I will lose rep for defending myself when attacked until their health bar turns red, but player names do not go red until very significant damage has been inflicted, thus preventing players from defending themselves in time to survive encounters if they want to maintain their reputation.
I just had an encounter with three unguilded players (Green health bars to me) who attacked me. Their health bars did not go red until I was below 50% health, and then I was near-instantly boarded. There was no instance where I could return fire and sink these players (without losing rep) before I was effectively dead.
I suggest players become hostile at much lower damage thresholds so they we don't have to worry about reputation loss when defending ourselves.