Niamh,
Navigation has been an incredible challenge for me as well. It is something I try to work on whenever I dive.
I took a great class that offered what I thought were incredibly good points on how to improve navigation.
Different dives require different navigation techniques. For example, a boat dive requires you to return to the anchor line to make your ascent. Whereas most shore dives, you need to know, generally speaking, where the shore is and whenever you are done, you just head back in that direction. There are advanced shore dives where you may be entering in an area that is protected and must return back to that area, otherwise, you risk making your return in something like a rocky area that is being pounded by waves. Anyway, each type of dive might require different "exploration" techniques which also entail slightly different navigation techniques.
Since you asked about boat dives, let me tell try my best to answer for that specific type of dive. For boat dives, it is highly recommended to make your ascent at the anchor line. This enables you to surface in the protection of the boat (presumably other boats are avoiding your boat). Ascending away from your anchor line may make it prudent to deploy a surface marker to passing boats know that there is a diver making an ascent somewhere other than an anchor line. The anchor line also gives you a visual reference to make your ascent. This makes it easier to know when you are making a slow steady ascent and also helps you during your safety stop.
Keeping in mind that you want to make you ascent using the anchor line, you should plan you dive so that you finish at the anchor line. On boat dives to sites you have never been to, this can be pretty hard. So one technique I was taught in sites like this is to explore the area using the star technique. This means that you use the anchor line as your base. You take a heading of say, due north and kick away from the anchor for say 20 ft. You explore all the way out. And then, when you are done, you take your reciprocal and head south for 20 ft. and you are back at the anchor line. Now presumably, you still have plenty of gas for your dive. So now you might decide to head west for 20 ft. Then head back to the anchor line. Then south... and so on.
This technique ensures that you explore a pretty good sized area and still return to the anchor line to make a safe ascent and is probably the best way to go for newbies as well as divers in unfamiliar sights and with limited vis.
Another technique is to use topography and your compass in combination. This type of navigation requires you to think through the dive a little more carefully from the beginning. For example, if the boat takes you to a set of pinnacles, you landmarks are different than say when the boat takes you to a low reef or a wall. In these types of dives, having a look at an underwater sonar reading of the area prior to your dive makes it helpful to make a plan on how to navigate a dive with the end goal of making it back to the anchor line. As an example, with a pinnacle, you can typically circumnavigate the pinnacle and find your way back to the anchor line assuming that it is not a huge pinnacle and assuming that you took a depth reading of where the anchor is located. If you wanna jump from one pinnacle to another, obviously, you need to know ahead of time that there are other pinnacles there and know which pinnacles you would want to jump to as well as which direction those other pinnacles are. Take a reading on which direction you are jumping just to double check so that you can remember which direction the anchor line is.
Anyway, these are just small tidbits. I'm sure others may have other tips they can offer. Also, if you think you are interested in the other navigation techniques offered in the class I took, you should look into the class. It is called Recreational Diving - Level 2 and is offered by a company in California called Breakthrudiving. Their website is Breakthrudiving.com. We use these techniques in Monterey where diving conditions are incredibly challenging with limited vis, strong current and cold water. These techniques seem to work well here.
