The major downside to a prescription mask is it means you have to deal with glasses on the boat. Glasses on boats are easy to break or lose (especially since they aren't nicely on your face at all times, you have to take them off and leave them)- and glasses cost a fortune. I'm a bit biased because I don't have prescription sunglasses, so if I wore my glasses (which I wear pretty much full time in regular life) on the boat, that would also suck.
I just put my glasses in my mask box when I'm not wearing them. Keeps them safe.