Are there potential toxicities to formaldehyde and toluene and so on? Of course there are. But the doses required for toxicity are orders of magnitude greater than anything you are going to be breathing, simply from being in proximity to finished neoprene items.
This is one of the favorite health store things to do -- tell you something contains a poison, but not tell you that the amount that is in the substance is so minute as to be harmless. Remember that you will be in contact with your neoprene for pretty short periods of time, too. It isn't as though you were lining your house with it. And your neoprene isn't even new, and new neoprene would be the most likely to be emitting any volatiles. All these substances disappear into the air pretty quickly.