How is this a conflict of interest?

My husband and I recently purchased our first home. We are currently going through the loan process (haven't closed yet). My father is our Realtor. Instead of taking the commission he would make he is going to give it to us to go towards our down payment. He has to fill out a Gift Letter (to make sure he isn't actually loaning us the money). Now the processor at our Mortgage Company just called and said it is a conflict of interest for him to give us the gift check. I'm curious. I understand why they are looking at it this way but I'm just wondering what a Realtor could do that would somehow manipulate the system in a situation like this. Obviously we are not doing anything underhanded, I'm just wondering how there could actually be a conflict of interest.