This is an interesting one, thanks for posting~
I'm going to take this from a psychological perspective.
Your partner tells you that they believe you'd cheat on them if you could
On a side note, in my experience this is a relatively common cognitive distortion among couples. It certainly touches on the topic of trust and values.
If that happened to me, I'd feel like I was harmed but I'm not sure why.
This can go from a couple of angles. First, it seems to be that the initial belief suggests "Partner A" (the belief holder) distrusts "Partner B" (the other one lol). From that angle, it would make sense to me that if Partner B trusts Partner A, they would feel upset that this is not reciprocal. Second, it could be that Partner A is attempting to normalize cheating, which is against Partner B's values and thus creating negative feelings. There's likely more angles, but I'll stop there for now.
What might make a belief or actions based on a belief bad or harmful?
Heavily depends on the context, I would say. I find that generally speaking making critical assumptions/beliefs about the other in your romantic/intimate relationship as harm-inducing.
And if them believing in something like that is bad or harmful, why?
I'd go back to it undermining trust and potentially challenging an important/non-negotiable value.
If it seems true to them, then would they really even have a choice in believing it?
Yes. In my view, all beliefs can be changed, if one is willing and open-minded enough. I think holding too hard to a single belief can be detrimental to one's overall mental health.. though I suppose it really does depend on the belief.