I don't know if you've ever had the pleasure of sitting in an advanced mathematics or statistics class with a professor who just happens to be a pedantic PhD., but sometimes the instructor, because of his or her in depth knowledge of a subject, has no clue, whatsoever, as how to present the subject matter to the student in a comprehensible manner.
Different people's minds work in different ways. This is often referred to as "learning styles." Unfortunately, most people who know something about learning styles took a workshop in which a learning style theory is presented. This leads people to think that it is as simple as the presentation they saw. In fact, there are at least 100 different formal theories on learning styles, and many contradict the others. We really don't fully understand how it works, but different people do indeed learn differently.
After a lifetime in education, I believe math is where the difference comes through the most. I once heard a theorist say that only 15% of the population is geared to learn math the way it has been traditionally presented. The problem is that the population of math instructors usually comes from that 15%, and they then use the "it worked for me" approach to teaching.
Much of traditional math instructional texts feature page after page of stacks of numbers, an approach the National Council of Teachers of Mathematics (NCTM) has been striving to change for years. Once a student begins to get the "I'm a math failure" attitude, just seeing a stack of numbers is like Turning a switch off in the head--all learning stops before it begins.
It is not about being dumb--it is about learning in a way that makes sense to you.