Asking you to leave if not buying a main meal

We sat down for nice hot beverages and wine to start off and we were not looking to dine. We went for dessert only since we have heard good comments about their desserts. Immediately, waiter says \I thought you were going to dine