Cheap diving can be done quite easily.
During the worst of the recession I was starting to wonder, if I lost all my gear tomorrow how would I replace it to get to just blow bubbles.
The way I would do it would be to get a used cheap wetsuit, booties, gloves, hood, all the neoprene. I'd look on Craigslist or look for rediculous sales at dive shops or online.
Fins, same thing, and I wouldn't be picky.
Reg, I would look at the most basic setup I could possibly find used and rebuild it myself. For example, I have been given for free many old used regs that the owners though were obsolete or were replaced with somethging that was considered more updated and better. Two that come to mind were a Aqualung Conshelf VI and the other a Scubapro MK3/108, and come to think of it there were also two other Scubapro MK5/109's that I got for free, plus a slew of many others.
All these regs can be rebuilt to work like new if you know where to get parts, know the right people, and learn to work on the stuff, which is about as easy as changing a watch battery or a tire.
SB is a great place to get to know those people BTW.
For tanks I also have been given many old steel 72's because they were considered no good anymore for some reason. I dive with them and they work great, and they were free!
A mask I would get the best that fits the best, regardless. A good mask is very important, but I would still look or the best deal.
For BC, personally I would go without one, but for the sake of others I would probably get the cheapest used one I could find that works. If I found a wing cheap enough I would fabricate a plate out of a stop sign and go to the dive shop and buy enough webbing and a belt buckle to put together something that I could strap a tank on my back and call it a day.
For weights, I love the suggestion of pouring lead into tuna cans, that's excellent!
I figured out once that if I had to replace everything just to get a dive in I could do it for between a few hundred bucks to a grand tops, for everything.
And then I would shore dive.