The three places I gave coordinates for are within swimming distance of shore in an emergency, there are lots of boats around, if your boat floats off, somebody will pass by to pick you up. Two of the sites are visible from the pass.
We dive from and anchor our small boat unattended regularly.
There are no mooring buoys so certainly the OP will need rode, chain and anchor.
Inside the bay and inlet, in addition to the jetty, some people dive under the bridge, stay out of the channel. There are also a couple of places inland from the bridge but I don't have coordinates, I just know where they are by sight.
I have paddled my kayak to the Bridge Rubble, launching from the area on the southeast of the bridge. It is not like we are talking about a huge distance, with a little caution and common sense there should be no problem.
The biggest thing, already mentioned, is that on an outgoing tide with wind onshore, waves can stand up surprisingly large over the shallow bar at the inlet. I usually try to avoid that circumstance. Knowing how to pilot an inlet, how to ride the back of a wave, when to chicken out, when not to go, what sea states are acceptable and what my be expected over the course of a day are why paying a professional captain can be a wise thing. Problem is here, the only places the local boats go nine times out of ten is the Jetty and the Bridge Rubble, over and over and over and over and they charge like 80 bucks for that.
On some fish just out of the pass;
Heading out, Crab Island behind us, take old people fishing, they like that, oh, wait, I am old:
The Jetty as it was circa 1970s:
On the Louise a friendly 'cuda hangs in the shade under our little Whaler:
N