Then you're doing it wrong. And if you were to get a massive headache while diving while everybody else is fine, then you'd know that you had not been breathing properly and had CO2 build-up. One uses breathing to minutely change buoyancy, not to control it. That's why you have a Buoyancy Control Device.
When you can float effortlessly without moving much other than your fins to keep yourself stabilized, when you hve the feeling of weightlessness, when your breathing doesn't appreciably change your depth.
If you're at neutral buoyancy, you can inhale deeply to start the ascension process because the air in your lungs will cause your body to be positively buoyant. Once you start to ascend, breath normally again and start deflating your BC until you get to new desired, shallower depth.
Yeah, I think this is what the OP meant by controlling buoyancy by breathing.
I know that's what I have to do, as no matter how close I get to neutral with my BCD, I'm so skinny that breathing in or out will significantly alter my buoyancy.
I consider myself close enough when my depth doesn't change a lot when I'm breathing normally (and I may counter the small changes with swimming), but either a deep breath in or deep breath out will cause me to change depth rather rapidly.