I know at least some of the companies that came out with Velcro integrated pockets changed that design to something less vulnerable to accidental loss, but not all of them. They are not a good thing, IMO. On one trip in Cozumel years ago, I found two such weight packets on two different days of diving.
On the other hand, I think you can do things that will make that not such a big deal:
1. Make sure you are as properly weighted as you can be. You were near the end of the dive at 20 feet. I am guessing you she had a fairly thick wet suit, which does tend to get you overweighted. At the end of the dive, having lost most of the air from the tank, you should not have a lot of air in the BCD. This has another advantage in that Velcro pockets with too much weight in them are good candidates for falling out.
2. Next, distribute the weight for the purpose of achieving trim. If I were diving a BCD of that type, I would try to have about half my weight in trim pockets near the shoulders, either in the BCD if it has them or on the cam bands if it does not. Let's say that a diver wearing a heavy wet suit needs 16 pounds in salt water. If only half of it is in the two weight pockets, then losing one of those pockets would only cost that diver 4 pounds--an amount that should be controllable.
I dive with weights distributed quite a bit. I was diving not all that far from you last year when I saw a small weight pouch lying on the rocks in the kelp right below me. I thought, hey, that's the same brand of pouch I use on my cam bands. (It was the kind that can slip over and be held in place by the pressure of the cam band's tightness against the tank. I picked it up and noticed that a name was written on it--mine! I have no idea how it managed to come loose and drop off, but the amount of weight in it was so small that I had not even detected the difference.