Uber making you pay for drivers making unexpected stops

So yesterday I requested a ride like any other day and my driver decided he was going to make an unexpected stop (not requested by me) after telling him I will be late for work he brushed me off and then continues to take a longer route. I later get billed by uber (almost twice of what I was supposed to pay) went to the neccessary channels to get this rectified and the support team is going to send me an email saying there is no issue with my trip and I won't be refunded.

Like someone please explain to me since when can drivers just make unexpected stops and you as a client WILL PAY for it, no questions asked.