The fact of the matter is that the dermal denticles which are found on sharks and rays are apatite covered collagen pieces which are similar to the enamel over our teeth, and with chondrichthyes (or cartilaginous fish) the denticles dont grow larger but increase in number as the fish grows bigger. From what I remember from previous research on sharks, there is very little mucous and most of it is found around the head region and is usually associated with the gel capsules along the ampullae of lorenzini which are used for electroreception. The primary defence mechanism for sharks are the rough dermal skin which is partnered with an efficient wound recovery system (the denticles are known to grow back slowly) with no known studies if the small amount of mucus plays any role.
However, this is not necessarily the case for rays which actually do have a more pronounced mucous layer with less "rough" denticles. There have been reports that riding mantas can cause infections though whether its case of people removing mucous or transferring bacteria across from hands/gloves is still unstudied.
So in summary, there is no evidence to show that touching whale sharks is detrimental to their health, but can be an issue for manta rays if the animals are handled excessively.
My personal opinion about "joy rides" is from a slightly different perspective. I don't approve of such behaviour primarily because I put myself in the position of the these animals which are hydrodynamic in their design and shape. As such, anything/anyone hanging on decreases this locomotive efficiency and increases drag requiring more energy to move, thus needing more sustenance. Another reason is that people may not recognise when these fish are feeding (e.g. whale sharks filter feeding on the surface) and its unknown if these joy rides might actually be interrupting their normal feeding behaviours or patterns (it takes a lot to fill up a whale shark).
