Holiday rental not fit for purpose

We booked Froggy Towers for 5 weeks. On arrival we were confronted with extensive building works being carried out on the house next door bordering the pool and outdoor area. This consisted of several builders on the roof looking down onto the pool area relentless noise and brick dust pollution preventing doors and windows being opened. The agent visited at our request and declared the property unlettable. The owner agreed to a prorata refund if we found another property, however, we spent 3 days searching and moving and because of the short notice the cost was doubled. We could not stay in the house during the day. We asked for a full refund as advised by the tourist info but this was rejected. The owner never visited us and denied knowing about it despite his relatives having been there until we arrived. He has only contacted us by email once after we asked for a written reply. Apparently building works are like an act of God so he stated. He seemed more concerned with his loss of earnings than our ruined holiday having travelled to SA from the UK. We feel strongly that we are entitled to a refund as compensation for the extra expense and inconvenience and distress caused to us