The investment, particularly in money but also time, in a hobby that in many cases the rest of the family doesn't share, to do the kind of diving that many of us are after (e.g.: coral reef and maybe 'big animal' oceanic diving, particularly in warm seas).
For me, a solo trip to the Caribbean aiming for 20-something dives is around $3,500 - 4,000 in terms of total trip cost. That same amount of money would pay over half of a cruise vacation for my wife and I with our daughter.
There's also the backdrop of economic realities; I read articles on personal finance pitched that idea that if people with a Starbucks habit or similar would just stick to cheap coffee from home and invest the cost of those lattes, etc..., they could save and make yadda-yadda. I'm not a coffee drinker, but the principles the same with some other things.
If one or two exotic vacations per year is your 'thing,' that's a serious outlay of money. If your spouse doesn't dive, that may mean running off to the tropics and leaving the spouse back home sometimes. Kids under 10 are pretty much shut out of it.
This isn't all unique to scuba. Hunter/fisher types with off-road SUV, ATV, bass boat and gun collection, people who enjoy European vacations from America, they, too, pay serious money.
You California resident shore divers are cheating!!!