This sounds similar to most dives and dive ops I experience here on Grand Cayman. I also dive mostly as a single. We always have the option of diving on our own with our own buddy, e.g., if we are diving with friends, but also are always given the option to follow the guide (which most do). The biggest difference from your experience is that I have never been in a group with any dive op where there were more than 8 divers per guide and almost always less. Often we'll get two DMs for the group.
About half the dive ops I go with will buddy us up before the dive and we're supposed to stay buddied up for the entire dive. The other half will just have us follow the leader as a group and when we get back to the boat will buddy us up based on amount of air left and we just cruise around the boat until low on air. Most of the instabuddies I've had assigned before the dive have been pretty clueless in my experience (they usually just take off on their own), so like you I just try to stay somewhat close to the DM and the group - and as the DMs usually turn around to check on us every minute or so, it's pretty safe. (I used to dive with a pony tank just to be safe, but gave that up a couple of years ago.) When I get buddied up under the boat with that style operator, I always indicate to my buddy for them to lead and that I will follow them. At least I know we'll be in close contact. And being under/close to the boat in relatively shallow water is pretty low risk, so I'm way less concerned about who I'm diving with.
I think you handled the depth gauge issue well. As long as you were on a guided dive you were not going to go too deep, and assuming you were not on nitrox, the depth is not that big a deal. As long as your computer was working and you could track NDL you were ok. I would, however, have had a nice conversation with my instabuddy during the SI if we were to do the second dive together, to make sure you were both on the same page. I do realize, however, again given my experience, that if they were that irresponsible to you the first dive, that conversation might not make a difference.
One thing I have found that works well sometimes is if I am diving with an "instabuddy operator" I will look for other singles on the boat and engage them in conversation to try to suss out if they would be a good buddy. If so, I'll ask if they want to buddy up and will then make sure we're on the same page as to how it will work. Interestingly, I had this situation on a dive last week and had one of the best buddies ever. We both immediately agreed to buddy up for the second dive.