Oddly enough, I think it seems to happen most when I'm horizontal. When I'm descending or ascending and vertical, no trouble.
To give an example, I was swimming horizontally at about 70 ft.; when I gradually swam to a slightly lower depth (65 ft. or so), still horizontally, I noticed I had too...