Do I have a right to be angry in this situation? Do I have enough of a right to end it?

If you feel you can't trust him then you should end it. I'm not saying you are right about this friendship he has being wrong, but what I do know is that relationships can't work without trust. If you don't have trust in him, there's no point in continuing to be together.