Look, you don't get to be to his age without being paranoid. He takes his ID back, and heads out after the guy. He briefly considers knocking him out and running away, but that's not going to help anything. Not when he can't leave town, anyway.

So he just heads after him, trying to look casual. Nothing to see here, just two guys who know each other. No big deal.