It is all about buoyancy. No need to guess, just do a proper weight check. That means empty or near empty tank, empty BC, you should be able to exhaul and sink. If you can't get properly weighted with HP100 because you are too heavy, your setup is too heavy. Changing HP100 to AL80 is just one way to change your weight. There could be other method. On the other hand, if you still need to add weight to archieve proper weighting, then HP100 isnt too heavy. When allow, I think HP100 is a better tank than AL80.
I also respectfully disagree about using the same weight between different tank and just add air to BC. It is dangerous practive. In OP's example, HP100 is -2lb empty, AL80 is +4lb empty. That means you need 6lb more while diving AL80. Now, keeping the same wieght and switch to HP100 meanings you are 6lb over weighted with a empty tank. Add 8lb of air of HP100, you are 14lb negative at the beginning of the dive. Even not considering wetsuit compression, if BC fail, you will sink to the bottom like a rock.