Doubles are not as streamlined in the water. They take more effort to move around when you're kicking. You present a bigger cross section in the water to create more drag, plus the extra weight means more inertia, so more force required to propel yourself. Also, I've been out on some charter boats that didn't allow doubles. One told me they just banned them because someone with doubles slipped and fell and broke another diver's foot with their tanks. When I'm on a boat in rough seas, I'd definitely rather be moving around the boat with just a single on my back.
I have singles and doubles. I'd always rather dive my single 100 or 120 with a 23# wing than dive my double 100s or 120s with my 45# wing. If the dive is suitable for diving with a single tank. The difference in balance and closeness of the center of gravity doesn't affect me in any way that I notice. The more drag in the water does.
As mentioned, doubles is also twice as expensive for annual VIP, plus hydros, if you're replacing one single tank with one doubles set. For me, it would be 2 x singles versus 1 x doubles set, so VIP and hydro is more or less the same.