I don't mind diving in bad weather. My buddy and I have gone in when many other have cancelled. Most of the time, once you get below 15-20ft, you're fine.
However, there is one case where I cancelled a dive, and would never do it in that condition. That day, the charter trip for which I signed up was cancelled due to extremely high winds. By extreme, I mean the power in my neighborhood was out too. I drove down to Ogden Point with my gear to see how it looked. At Ogden Point, divers enter at the lowest blocks. That day, the waves and break continuously swept over the entire wall (at least 25 feet up, I believe), and the waves were probably a few meters high with plenty of debris (like logs) ripping around in the water.
Personal preference and whatever you're comfortable in, as long as you're safe.