I've dived mostly from boats (speed boats, cruise boats, live-aboard boats), in the warm tropical waters of Malaysia. In some of the dives, we have faced some current, nothing TOO bad (if you give enough effort, you can swim against the current and stay mostly stationary). I've never felt insecure in it, but then again, I don't just dive with buddies, we always follow a DM/dive guide who has dived the sites thousands of times. From the few dives I have had, we dodge currents by descending closer to the bottom or hiding behind a rock/reef most of the time. However, in cases when the current carries us in the right direction, i.e. to where we agreed to meet the boatman, we just drift. In any case, we always have a "check out dive" where the DM assesses the skills of the whole dive group, if even 1 of us cannot control buoyancy or swim well enough, we would not be led to challenging sites (or we are separated into smaller groups, where the more advanced group goes to more challenging areas).
Point is, in my opinion, as long as you are diving from a good DC with good DMs, drift diving should be quite relaxing. If you ARE the DM and customers who you know can't do it insist on doing it, then yeah, I would say, that is not relaxing...
Now down currents on the other hand...