As someone who has done work helping human trafficking victims, there's a possibility this was a trafficking match. Near a sketchyish area at a biz with a subpar rating (meaning it's likely to be less crowded and few potential witnesses lingering in area), and unwilling to go anywhere else.... my alarm bells are going off. Regardless, even if he's just a regular jerk, you made the right call.