For over a decade now, I've done almost all of my diving without a BC and I've enjoyed my dives more as a result of the improved streamlining and light/compact size for traveling.
Are you diving off boats at resorts? If so, have they been accepting of you diving sans BCD?