Honestly, you only have to be an Ok-ish swimmer to pass the OW swim test. You have to swim 200 Meters without fins or snorkel OR 300 with fins and snorkel. (For PADI at least) I was able to pass it no problem, and I was a pretty sucky swimmer at the time. Although, I did enroll in an adults swimming class later, before getting Rescue. My reasoning being that I wasn't comfortable in the water without scuba gear, which was kinda ridiculous.
If you think it will make you more comfortable do it, otherwise you'll most likely be fine.