There are only two real downsides to nitrox: cost and availability.
Cost may be a couple dollars, or maybe even $10 more than an air fill, but for most people that is just a small fraction of the cost of diving and doesn't affect the choice of whether or not to use nitrox.
Availability is the big issue with me. Some areas -- SE Florida, for example, nitrox is readily available. Other areas -- Maui, for example, it is more of a problem since most charter boats don't have it available.
It isn't just a coincidence, however, that SE Florida, with the square profile dives at 60 to 90' is where it is most useful; while many dives in Maui are multilevel wall dives where one can have a nice profile on air as you gradually ascend over many minutes.
Using nitrox does require that you be more aware of maximum depths and requires that you analyze gas and track O2 exposure. Most CNS oxtox incidents are caused by using a gas other than what the diver thought he was breathing --- either by inadvertentaly breathing a high %O2 deco mix while at depth, or by diving a mis-mixed/mis-labeled/mis-analyzed mix beyond the limits for what the mix really is. In other words, the real danger is from gross stupid mistakes, not from exceeding your planned depth by a few feet.
I definitely recommend that you get the nitrox certification. Even if you mostly continue to dive air, it is good to know about the dangers, limitations, and advantages of nitrox.