Obviously the diver's orientation, trim and size are irrelevant when the diver is passively drifting. As other have mentioned, position relative to the bottom is often very important. If you want to slow down, hug the bottom, figuratively or literally. If you want to "catch up' with the group, swim 10 feet off the bottom, get up in the water column where you will be less effected by the reduced velocity at the bottom.
One thing i can envision is that if diver A stops and holds onto the bottom (in a current) and when they release from the bottom, if they are facing into the current and are streamlined, they might give a kick or two into the current and their reduced drag might cause them to accelerate slower than diver B who leaves the bottom at a more aggressive angle of attack and then spins around quicker, since we normally don't drift backwards when drift diving. if there are a lot of starts and stops, I could see where it may appear that one diver is going to drift faster than another.