I can honestly say that diving in no way worries me above 100 feet, but that is because I have fins on and I know I can make it to the top, and I practiced acsending while slowly breathing out (without my reg in my mouth) from 50 feet quite a few times, until I got totally comfortable with it, I started by practicing from 30 feet and worked my way up to 50 feet, I am going to wait until I have more practice before I practice it from 60 feet, but I am pretty sure I can do it from a 100 feet if I have to without a whole lot of over anxiety. When I go a little deeper that is when my caution alarm comes on. Around 70 feet I start to be more conscience of situation.
I think that anxiety (which is what I like to call, hidden fear) manifest when you are out of your comfort zone.
Not too long after I got certified I went with my buddy that certified with me, and another guy that had been certified for a couple years before us, and we went out to do a shore dive in fairly decent surf. 4-6 foot. There was a small craft advisory, just to give you an idea, but there were still a lot of small boats out, so there was definately swell, but not enough to keep the more seasoned boaters in dock.
I had no problem walking in with all my gear on, I just backed in, and I was completely comfortable, and totally relaxed about it, but that is because I surf, I am use to waves, they were little to me. My buddy on the other hand although he backed in fine enough, he was definately a little anxious about it, and the my other buddy that had been certified for a while couldn't make it out passed his waist.
So we finally decided to leave the one guy on shore and go do the dive just the two of us (since he would have been out of his comfort zone and ability) right when we dove in, my other buddy signaled me to surface, he couldn't do it, he hit his comfort level, he didn't want to go hang out under those waves. I spent the rest of the day thinking that they were chickens, but the truth is, they were playing it smart, it was out of thier ability. Too Anxious, Too Nervous, it is a recipe for for Panic, and Panic is the main ingrediant to accidents.
So I think that it is about acclimating to things, the fear of diving goes away pretty quick, I think with most people, but depending on how you adjust to some of the other aspects of diving might be different.
Personally, I am looking forward to doing plenty of ice diving this winter, and the thought of that would scare the crap out of most people, you can't just surface if something goes wrong, you have to find the whole you climbed in at.