Gonubie Post Office witholding key deposit!

My husband and I went to Gonubie Post office this morning to cancel our postbox. The gentleman asked for our keys and we gave him back our 2 keys. I asked him if we need to fill in some kind of cancellation form, and he said no. He then said we're all done we can go. My husband asked if we can get our key deposit back, at which he rep****, No. We asked why and he mumbled something we couldn't make out, so we asked again why, and got the same response. He just kept saying "you're done, you're done, you can go" Not wanting to make a scene and delay the queue behind us, we left without getting our deposit back.