I had great hopes for merino, but I can tolerate it for about 20 minutes before I want to rip my skin off. I do a little better with lambswood/angora, but even those garments are for brief, evening wear.
A friend who used to own sweater stores explained it to me. It's a combination of fiber length and stiffness. Wool, by its nature, has a short fiber length, so when you spin it into yarn, there are a lot of little ends sticking out. If the fiber is also stiff, as standard wool is, the yarn is effectively "prickly". Some people are more sensitive to this than others, and I am one of the sensitive sort. Artificial fibers like microfiber and the fiber of which Polarfleece is made can be as long as the manufacturer wants them to be, resulting in a much sleeker yarn. This is why silk used to be so prized -- it was the only natural material with incredibly long fibers.