I agree with the labor. I don't like the idea of trying to do back to back, day after day dives because my wetsuit is still wet and uncomfortable and I'd have to drag it out to another beach. I have to shlep all that gear in and out of the car, wash it all, hang it all to dry, and wait 2 days and check everything. Two seconds in the water, or two hours in the water, the gear still has to be washed (Unless I really am going back in the next day). Packing all that gear and loading it into my car takes more space and trips than any other sport that I do.
However, I don't think this is an initial deterrent to new divers. I, for one, don't ever mention that part to friends when I'm trying to convince them to get certified! I'm relatively young compared to the majority of the dive community that I see (mid 20s) and my friends, once passed the fear of the water or sharks, cite cost as the major factor. Like many people have mentioned before, it's hard to spend 500 dollars (after initial non-rentable gear costs) just to find out whether you are going to like it or not.
The initial investment costs are high. For full gear setup and cert, you are talking thousands of dollars. And people are seldom happy with what they have with dive gear. There's always that new thing you want that does much more. It's not like most other sports, like skiing, boarding, surfing, climbing, etc. Gear in those sports are expensive too but not as much and if you are putting the amount of money you put into scuba gear for middle of the line stuff, you are probably buying top of the line for other sports. Not to mention, I can take my surfboard anywhere in the world and surf. I can take my skiis anywhere in the world (where there is snow) and ski. But I have to have warmer exposure protection for colder areas and less protective gear for warmer areas for scuba.
Scuba gear is also life support. If I don't take care of my surfboard or skiis. So I end up with some messed up equipment but my life isn't in immediate danger because of it.
Like other people have mentioned, there is planning involved as well. During snow season, I can up and hit the slopes on a whim. For many people, this isn't the case with diving. One of the minor issues I have as a diver my age is trying to split my time between diving and my social life. People my age tend to still move in large packs! I have to decide on whether I want to plan a vacation with friends that most likely won't involve diving, or go on a dive vacation with people I don't know or hardly know.
All in all, I think it's a few large barriers, such as cost, along with many smaller barriers, like the idea of being in deep water or fear of hopping in the ocean with 50lbs of gear strapped to you, that prevent more people from joining our sport.