Virgin Gorda could be an option.
The island mostly consists of two peaks to hike plus there's trails to deserted beaches, a national monument, the famous Baths is a hike down etc. Diving is generally with DiveBVI, there's no chain resorts/restaurants but mostly a few beachfront condo type places most of which sit on their own private snorkeling beach. One advertises there's no need to worry about boats (although there's a lot of sailboats there) since the coral would shred any that approach.
I enjoyed the diving, it's very diverse. Some deeper wrecks, the world famous Rhone, some good swim-thru's etc. Not really my definition of "advanced" though.
You get there by flying into St. Thomas and taking the fast ferries over. If you're willing to devote a day to it, you can also visit a couple of the other BVI's several of which must also have hiking - esp. Tortola which is a bigger island with lots of hills.
Virgin Gorda only has about 4000 permanent residents and under a dozen resorts. Lots of expensive villa rentals also - sometimes you can get a deal on vrbo/airbnb etc. so they have some really excellent restaurants considering the relative uncrowdedness. Our one non-diver spent a couple days exploring - he said at some places he was the only one there.
Compared to some of the other Antilles I've been to it's pretty quiet and calm. Yet you can find enough to do, we found dancing one night, went boat bar-hopping one non-dive day etc. You can even rent/book a trip with a couple operators who will drop you and lunch on a small island that you'll have to yourselves - or a beach not reachable via road.
I can't help with resorts as we rented a villa. Leverick Bay Resort was nice but a little removed being on the far north end. They have a great beach BBQ on Friday - locals, tourists all go. Mango Bay looked decent according to our friend who drove in there once - it's a waterfront small villa/condo property. The nicest resort - Little Dix Bay is closed for remodeling. Also some like Bitter End really are at the end of the island - upscale clients fly in there on a helo due to the remoteness. Too quiet for us. We stayed in the Windy Hill area which was nice and quiet except the morning roosters exceeded the area residents - yet close to everything - we walked home from town once.
Virgin Gorda | The British Virgin Islands
Dive BVI - Your stop for BVI Adventures -It's what we do!