I'd much rather swim less and sit more, checking out the little things or waiting to see if the big things will cruise by (maybe more than once).
Definitely try it! Especially if there's anything to see when you start.
The sharks of Providenciales came much closer, and came back, when I (and the rest of the group) moved the least. Stopping and waiting for everybody to drop into one of the swim throughs, and then waiting a little longer...and a little longer...rewarded me with a school of Atlantic spadefish who had to split into two groups to swim past me they came so close. I just sat their watching them approach, enjoying it, and selfishly enjoying that I was the only one seeing them since the rest of the group was focused on moving together. I spent the last 10 minutes or so before my safety stop on several dives hovering near the bottom near the boat watching the rays gliding and the schools of jacks swirling in fish cyclones (jacknado! lol).
My last dive on Dominica last month I spent the last 35ish minutes in 15ish feet of water within sight of the boat, at three different coral heads/sponges. Watching a juvenile frogfish watching me watching him...watching Pederson cleaning shrimp, arrow crabs, and banded coral shrimp poke around...and best of all, finding two red snapping shrimp deep in a corkscrew anemone. Sitting there for a bit encouraged them to poke out a little bit, and then play with the bolt snap I slowly moved near them. The best part was one of them using his snapping claw to make a loud PING!!! several times on the snap.
Near home I spent 20 minutes watching nudibranchs, hermit crabs, other crabs, and sea stars on a single rock last spring. If it wasn't 50ish degrees and I was dry instead of wet, I might have stayed their longer. One of my favorite sites is a granite wall and boulders on one side of a cove with a sandy middle. I almost always stop for at least 5 minutes, especially on night dives, to kneel on the sand with the wall behind me and watch for whatever might cruise by (squid, single and schools; stripers; skates).