As someone who's kept live coral in aquariums for years, I've got some experience with this.
A single light touch is unlikely to kill a coral. Sand and other detritus (think parrotfish poops) get dropped on them all the time. And numerous fish take small bits out of coral. But all of this activity can stress the animal. If the mucous layer is scraped off, or the tissue torn, a healthy coral will likely recover, but may get a bacterial infection.
However, if a coral isn't otherwise healthy (maybe it's stressed from heat or pollution), then it is less likely to heal. And if it is continually abused, at some point it can't heal fast enough and will likely sucumb to infection.
A big part of the problem with divers touching corals is that the same ones get touched over and over. The one a couple feet in front of the rare fish that the divemaster points out to the group. The one on the down-current side of that swimthrough. And then there are novice divers who aren't just going to lightly touch it, but bump hard enough to break pieces off.
A broken coral isn't necessarily doomed. It's a form of reproduction that happens frequently on the reef. If the piece that was broken off lands so that it is not entirely shaded, nor smothered in sand, it may well survive and eventually grow into a new colonly. Of course, many places where a coral fragment might fall, it won't get enough light or water movement, and will die.
So one person touching something probably won't kill it, but there is damage being done. Best not to touch anything, so that the reef will look at good tomorrow as it does today.
-Mark