Nope, the outer handle was the issue.
I figured out what the problem was. There was a piece bent, and it was also wearing on the handle itself, not to mention where the handle broke too.
The piece was bent like this...
So I bent it back to what it was supposed to be, like this...
And this is where the handle was worn. The part that's rubbed off is where the bent part was rubbing...
