Hi brokenhearted,
yes he can make an issue with it...if he is mean enough he can shorten the visit time or indicate to access within his supervision.
it's good to keep to the schedule timing...given before hand and it's a courtesy to inform before hand you would be late to fetch or sending back late.
after all he gt his own life too and can't be always waiting for you to pick the girl up or drop the girl.
5mins late is actually reasonable as traffics have issues also..but highly depends on the person taking it.