Hi!
So, of course, you should be a member of DAN. That gives you a professional opinion just a phone call away!
As to the rest:
Anxiety is often considered a contraindicator for diving, because underwater panic is so dangerous! Statistically, though, anxiety disorders are quite common, and it's reasonable to assume that many divers do dive with these conditions (even if one assumes that many anxious people self-select away from diving). We live in a culture that stigmatizes this, and it often seems that this goes double in the diving community.
This is fir a good reason. Please consider that your own safety is on the line, but ALSO that diving with any condition like this can put your buddy at risk too. Panicked divers are dangerous. You have a responsibility to those you dive with to (a) inform them and (b) to ensure you have prepared properly to minimize your risk.
In your favour is the fact that you, through lived experience rather than theory, know what panic feels like and know that it passes. You hopefully know what the very beginning of panic feels like. It's all about catching it at that point.
I, too, have had anxiety issues in the past and I have dived without major incident. I have felt the start of anxiety a couple of times: when narced at depth for the first time, and another time when exhausted at 35m in strong current overbreathing my reg. Because I know what anxiety feels like when it starts to build, I caught it very early both times, signaled my buddy, stopped, calmed myself, and continued.
Some tips that I think are good.
Be super duper conscientious about practicing good buddy and team skills at all times.
Practice a sign for 'I'm feeling dodgy' and a protocol for this with your buddy. Having someone there to make eye contact with you and even hold your hand while you collect yourself might make the world of difference.
Use the boat ride out to get very present and calm.
If you can, take some time in the water to get very zen. You are about to enter a calm world. Slow your breathing and relax your body. Be zen before you head down.
If you feel anxiety coming on, slow your breathing and in particular be sure to exhale fully. You have enough air coming in. What you want to do is make sure you are getting rid of your CO 2.
Throughout the dive, keep checking yourself frequently and relax, relax, relax.
Build up slowly, as you mention you are doing. Dive in benign conditions for your first 20 dives at least. New divers are task loaded already and as such are more prone to panic. There is no hurry. Build up slow and enjoy each step.
Be committed to skills. Practice your drills each dive. Accept nothing less than personal excellence with skills, trim and buoyancy, and buddy skills.
Stop exhaling through your nose. If you do get anxious, this is a bad habit. You can expel air faster through the mouth to get rid of CO2. Also, if you get a head cold you are toast. You will notice divers often come up with a fair bit of gunk coming out of their noses due to pressure and the sinuses etc. breathing out through your nose will make for a messy experience in more ways than one. Please make unlearning this bad habit of yours a priority. Bad headaches and so on after a dive are often a sign that you aren't breathing properly and are retaining CO2. Retaining CO2 will often lead to panic as it is a signal that the body interprets as 'I'm not getting enough air'. See if fixing your breathing helps with the symptoms you are experiencing also.
Happy bubbles!