If you have a small tank and very little wetsuit, then you don't need to put much air in your BC and you can dive without one without too much trouble. However, it is not really safe, especially if you dive with a buddy. In an emergency, a BC will be invaluable. I would hate to have a bad outcome in an accident and know that I could have done more (in a rescue scenario) -if I were wearing a BC.
If you are solo, shallow, warm water (and probably carrying some ditchable lead) then it's not a big deal.
What many warm water people fail to realize is the magnitude of the huge shifts in buoyancy associated with a double layer 7 mm suit, and say maybe a 120 cu-ft or more capacity tank(s). If you are going to pull yourself down and up an anchor line, or crawl up and down a slope, it can be done, but it is not safe, it is not efficient and not something I would choose to do. The swings in buoyancy can be over 25 lbs! This is not something you can swim up or deal with using lung volume manipulation.
I agree that many people wear more lead than they should, however I worry much more about people who carry a lot of non-ditchable ballast and a wetsuit - rather than someone who simply chooses to be over-weighted and wears an extra 8 or 10 lbs in order to deal with a strong current or something (assuming they can ditch all or some of it in an emergency).
Sometimes, I dive without a BC