So to be more detailed, now that I'm home.
The guy was yelling at me and flipping me off because in his words "you cut me off, how can you be so stupid, I have fishing lines in the water" (fishing poles, not commercial fishing gear) Clearly... The right of way was mine, and he was also required to yield to the dive flags.
I was very clear when I pointed to the TWO dive flags on my boat, both the Alpha Flag, and the Diver Down flag. Said, There are ACTUAL SCUBA DIVERS IN THE WATER, among other things.
The guy didn't leave until the passengers started shouting out his registration number to me, as I said, "OK *%@# $@#@ I'm calling the cops right now, let's see who is right"
Anyway... That's what happened. Luckily, nobody got hurt, and we all had a good laugh about what a idiot this guy was. I really hope that he tried to complain about me to the coast guard or something. LOL.