POA is a thief!

Good advice from Alva, and if you do take him to an elder law attorney who specializes in financial abuse, remind him to bring his checkbook. He should know that the consult may not be free, and if it isn't he needs to be willing and able to pay him on the day of the appointment. I'm saying this because if this turns out to be a man with dementia imagining things, then the attorney is out of his fee. My mom will sometimes call vendors/service people to do work on the house that isn't necessary. If they show up, I still have to pay them (and should pay them) for their trip charge.

A person confers POA on a person who is trusted to have control of his finances. If your father doesn't trust his children he should request an accounting each month of income to his assets and of distribution from them. He should see a lawyer to withdraw POA and go to his bank. If he is truly too frightened to take care of his finances, there is truly nothing that YOU can do. Either HE is in control of his money or he puts someone ELSE in control of his money. If he cannot take action certainly you as a bystander cannot. You can offer to accompany him to a lawyer. But if he is knowingly and without dementia letting criminals handle his affairs, then likely going into a nursing home is the very least of it. Offer to accompany him and then let it be. When he complains to you, as clearly he is doing, tell him that he is in charge of his affairs; no one can do this for him. If he has dementia, then this is reportable to APS so they can investigate the facts.

If it's true there is nothing you can do. If this man doesn't have cognitive issues he doesn't have to be controlled by the PoA since this authority doesn't kick in until the assignee has diagnosed impairment. You can encourage him to act to protect himself but if he's unwilling to let the offenders suffer the consequences, then maybe he can't have it both ways.

Notice I said "if it's true..." This is because you may not know the whole story. This man may have mental decline that you're not seeing. There are many posts on this forum from adult children who are acting legally to manage their parents' affairs yet the deluded parent thinks they are being robbed and actually call the police on them, caused distress and chaos.