I can't tell you for sure what is your problem, without seeing the way you dive. But I will bet 10 to 1 all of the other folks here were wrong. and the reson for you problem is simple- You do not swim right.
From my experience, A lot of people do not use their legs correctly and thus tend to ascend when strting to swim. I will give here a thorogh description of the whole process, and than advice on it-
First, what happens is that you probobly descend first while you are in vertical position. Than when you reach your final depth, you atain a horizontal posiotion. Fact is, that while horizontal, you tend to float more than while vertical (It has to do with static fraction and higher center of gravity).
Now-About this specific point-Men tend to have heavy feet, which couses their feet to sink, and put them in vertical poistion. If this is true in your case, that every time you stop, you became vertical, it will couse you to retain more bouyancy when strting to swim again (and thus cousing you to ascend).
Second-Swimming right- As I said, I am willing to put money (well, when I'll get some, at least) on a bet that you do not use your legs properly. You probobly tend to bend your knees too much. Bending you knees while divig will couse you to ascend. why? becouse bending you knees, in addition to the vector pushing you forward, will also creat a vector pushing you upward (the movement of the fin down, pushes more water than the movement up). I see people do it on every course I teach, some of them will not notice it, and even refuse to acknowledge the that they bend their knees, even though I clearly see it happen. They simply don't feel it, and think they are working properly.
Three-This thing is creating a cycle- While you start ascending, you get annoyed with it ("Why the hell do I go up?! I simply push harder down!"), and try to kick yourself down using the fins agressivly. Using your fins agresivly will couse you to breath more- and breathing more will couse you to... ASCEND. Why? becouse when you breath more the avarage volume of your lungs is bigger and this couses you to have larger volume on the same wegiht i.e-makes you lighter.
So- you empty your BC manage to achieve neutral bouyancy- BUT the moment you stop swimming to look at somthing- Wham! you fall down. So you fill it again. And than when starting to swim empty it, and so on and so on.
Ok- now to the part that really interests you-How to solve it?
few things-
First- A rule-Be calm. Being calm will make you more neutraly bouyant.
two- Swim SLOWLY. Nowhere to rush down there- the reef won't go anywhere. Swimming fast will couse you to bend your knees more, and breath more. Two things that will couse you to ascend.
Three-Practice correct swimming. Go to a pool with a snorkel, have someone watching your legs and correcting you if nessacery. Think about every movement your leg makes, dont let it do it's own "will".
Four-The above should work for 90% of peope. Some people simply can't swim right-in this case, what you'll have to do is adjust to a new diving position- Swim with your legs slightly higher than you head. This will creat a vector down to negate the up vector. If your phisique makes it tough (heavy legs) the solution is simple-Either place your weight belt higher (personaly, I usually dive with the weightbelt almost as high as my armpits), or attach small weights to the shoulder straps of your BC.
Well- That's about it.
Please tell me how it's going.
Hope I helped.