I've posted on this issue in other threads here. As a marine biologist with 37 years of diving in SoCal kelp forests, there are species that are touchable and those that are not. Handling a warty sea cucumber would generally be fine as long as one did so gently. Killing a sea urchin with a knife just to feed fish is, IMHO, not OK.
I often touch marine life that I know that easily tolerate gentle handling. I do so to verify identification or to film them or their behavior for the purposes of educating divers and the general public. I don't see a significant problem with touching species you are familiar with and know the degree to which they can be handled safely.
However, most divers are not marine biologist (and many are very poor at species identification) so I always suggest people not touch unless they are very sure about what they are doing.
I try to limit my touching to my female dive buddies... if they'll let me!