I'll even back off from an angry Sargeant Major. They're pretty cute when they're angry - but I also know they're protecting a nest so it's not fair to tease them.
I once made a terrible mistake with this. I was snorkeling with my non-diving wife, and I saw several sergeant majors defending their eggs below us. I wanted my wife to see how such a little fish would try to drive away a creature as big as I, so I went down and confronted it, intending to swim away as soon as it threatened me. But it didn't threaten me--it swam away in fear. I surfaced immediately, but the damage was done. As soon as the sergeant major went away from its nest, the other fish swarmed in. The sergeant major returned and tried to drive them away frantically, but there were too many for him, and soon there was not a touch of purple on the rock. Not a single egg was left to defend. I felt horrible, and I will never make that mistake again.