Guardians learn to be punctual. If you're a professional hero and you get in the habit of being late, the consequences can be worse than annoying people or missing a show.

I don't think it bothers him too much if other people are a bit late, but if they cause him to be late, that can be a problem.