I have used both and prefer neoprene, especially for the neck seal.
With a neoprene neck seal you can get a better and more comfrotable seal that will not leak if you twist youe neck to far one way or look straight up - you cannot say the same thing for latex.
The extremely limited insulation of a latex seal also causes a great deal of heat loss given the presence of the large arteries and veins just below the skin in the neck and the fact that the body never reduces flow of blood to the head (although the actions of some divers will make you question this from time to time.) In really cold water you need a neck warmer of some type with a latex seal, while with a neopren seal it is built in.
I have not had problems with neoprene wrist seals but people with prominent tendons sometimes do, especially when pinching their thumb and pinky together.
In addition to the skin out/fold over variety, neoprene seals can also be had in a skin in variety. They have half the bulk and do not generally leak with skinny wrists. But they require a lubricant like K-Y to get on easily and can burp air out the seal if you do not have the cuff of a glove adding a little pressure over them and you raise your arm straight up.
It is generally really hard to tear a neoprene seal, I did manage to tear a neoprene neck seal once but this was due to a defect in the seal (the seam was not taped on the inside). However, unlike a latex neck seal would have been, it was simple to repair in a few minutes with neoprene adhesive and was ready to dive (and properly taped) the next day.
In my opinion, the only clear advantages with latex are for persons with latex allergies and for persons who prefer to use dry gloves as most (effective) dry glove ring systems require a latex seal.