Being comfortable in the water is important.I agree that diving is not swimming and speed is not required to meet the minimum requirements but there is a bigger picture to consider. Being comfortable in the water is important to self-confidence, which is important to enjoying diving and reducing the risk of panic.
A broken fin or mask strap is an inconvenience or annoyance to a good swimmer but can be downright scary to a new diver who is very dependent on their gear.