I write about this periodically, but I haven't for a while, so . . .
The minimum amount of volume you have to move through the gas exchange part of your lungs is determined by your production of carbon dioxide. There is no way around that; you have to breathe what you have to breathe, to keep the CO2 level in your blood normal. Muscle activity causes increased CO2 production, so for the diver who wants to maximize his time underwater, becoming efficient and moving slowly is key, and I will talk about things that impact that a little later.
But it's important to note that I said that what we are talking about is the air that moves through the GAS EXCHANGE part of the lungs. Your lungs resemble upside-down trees -- as the air enters them, it moves through the trachea (trunk) into the major bronchi (primary branches) and then out through smaller bronchi until it reaches the air sacs, or alveoli, which are analogous to leaves. It is really only in the "leaves" that gas exchange takes place -- all the air that is sitting in the branches is not participating in the removal of CO2. As you exhale, you fill the branches with CO2-rich air, and when you inhale, you take that gas back into the alveoli. That volume, stuck in the branches, is called "dead space", and there is nothing you can do to diminish it (although you can increase it by adding a snorkel to the picture!).
So, let's say you are going to move 5l of air through your lungs in a minute. If you do it with 5 1l breaths, the dead space will be a small proportion of the volume of each breath, and most of the air will go to the air sacs. If you do it through 20 1/4l breaths, most of the gas is going to oscillate back and forth in the dead space, and very little CO2 elimination will take place. Anxiety tends to cause a rapid, shallow breathing pattern, which is not only very inefficient for CO2 reduction, but, because it allows CO2 to rise, causes additional anxiety! This is the root of the advice to use a slow, deep breathing pattern on Scuba, although too deep can cause buoyancy issues. If you did the "fin pivot" exercise in your OW class, you were looking for the breathing pattern that had you begin to exhale just as your body began to rise, and begin to inhale just as it began to fall, which is about the most efficient pattern you can use.
Now, assuming that you have conquered the anxious panting, the next thing to look at is minimizing exertion. The fastest place to address that is to SLOW DOWN. Most new divers swim constantly. In part, this is due to instability in the water -- if they stop, they pitch head or feet down, or roll on their sides. Swimming is like moving a bicycle, because it gives you stability. But much marine life survives by camouflage, so you see much less if you are zooming . . . and you have less time to see it in, too.
So, if people swim because they are unstable, how do you develop stability? It starts with weighting. Carrying the correct amount of weight is important, because being significantly overweighted causes buoyancy instability, and the diver who is constantly adjusting his BC, and swimming up or down to correct deviations from depth, is using more gas. But what is also important is having that weight in just the right places, because physics is inexorable. If you don't have the things that sink balanced over things that rise, and you do not do anything to counter the forces (eg. swim), you will rotate until those pieces have lined themselves up. For new cold water divers using integrated weight BCs, that will often mean rotating into a significantly feet-down position. Occasionally, especially if using aluminum tanks, it will result in feet up.
A diver who is not horizontal, and who is kicking, will of necessity be propelling himself to a different depth. If his feet are below him, he's swimming up; if they are above him, he's swimming down. Now, assuming he wants to stay where he is, there is only one solution . . . he has to adjust his buoyancy to counter the force he's creating with his fins. If he's swimming up, he has to remain negative in order not to rise. And of course, when he attempts to stop, he will SINK, which immediately results in more kicking. This vicious cycle uses gas very fast, and is the primary cause of increased gas use in novice divers who have made it past the anxious panting stage and who aren't incorrigible zoomers.
So, time spent determining the actual required weighting, and balancing your equipment so that you can hover in a horizontal position WITHOUT kicking, is key in reducing gas consumption. Pair those with relaxed breathing and a slow pace, and you'll see your bottom time increase enormously.
However, as a final note -- CO2 production is proportional to body mass. Bigger people need to move more volume through their lungs than smaller people, and people with more muscle mass will use more than people with a higher body fat proportion (because fat has a low metabolic quotient). So if you are significantly bigger than your buddy (as many men who dive with their wives are, for example) it's unlikely you will ever match consumption. If you want to match bottom time, you will need a bigger tank.