Just to follow up, this is one of the reasons I think all scuba divers should be comfortable, confident swimmers before they take scuba, and why allowing people to swim any stroke (including dog paddling, which isn't really a stroke) is inadequate preparation. If you know how to do freestyle properly, you're already comfortable with your face in the water while keeping water out of your nose and breathing out through it, and mask removal/replacement/maskless swim is a non-event. I think it was only after swimming for about 5 years (including lots of swimming underwater) that I got my first mask, so being without it was no more than what I was used to. It improved my vision but not my comfort (back then, with a rubber skirt it was considerably less comfortable than nothing).
I agree that divers should be proficient swimmers, but I don't think that's the whole story here. I passed OW just fine, and recently when taking another class had a devil of a time doing a no-mask swim or even mask removal. The problem was the cold water on my face while breathing in. I tried sticking my face in the cold water, no problem. I went to the pool and swam around with no mask and breathed through a snorkel, no problem. Sticking my face in cold water and breathing through the reg? Couldn't get the breath in. Just. Physically. Could. Not. Breathe.
Solution for me was to breathe out through my nose when I exhaled, and to spent ten minutes without my mask while diving (shock therapy!).
But I'm a proficient swimmer and comfortable underwater at this point, and still had trouble, so I have a hard time seeing swimming as the solution to everyone's problem with mask removal.
I agree that the OP should get comfortable enough to where he/she can just rip it off, though, because someday someone's stray fin will likely do just that.