I don't mean to push any buttons here, but you are joking about this swimming ability stuff, right?
But no one should ever dive and surpass their physical limits.
Im not saying you have to be a good swimmer, and hadnt noticed anyone else saying that either. Its personal preference really, and for myself I enjoy diving in challenging conditions, so I stay in shape for these situations.
As for not surpassing your limits, the best planning and intentions can sometimes place you outside your comfort zone, especially in the open ocean.
Factoring in the need to assist or rescue someone, physical fitness can play a major role also.