Smoking causes lung tissue damage and scarring which would naturally reduce the effectiveness of air exchange which equates to wasted gas and the need to breathe more frequently so you've got a smoker with smaller usable lung area and less effective gas exchange which can lead to more frequent inefficient shallow breaths compared to a nonsmoking diver with larger and more effective lung capacity who can take deeper, slower, less frequent breaths.
It's been my experience that drift diving can require an intensive amount of physical exercise for a boatload of reasons- working to stay on a patch of reef by turning 180 degrees to the current, maintaining trim and buoyancy in the stiff current, maintaining a safety stop without a line to hold onto, and swimming back to the boat is sometimes necessary even in a "live boat" situation. Lots of times the current abates and you've got your typical dive which can involve a lot of finning to see interesting areas of reef. Plus the climb up the boat ladder and the work involved getting the fins off while you're holding on can be rather strenuous. At any rate, it's surely more of a workout than a walk around the block