Knowing when to call a dive is always unpleasant but rarely all that difficult: when it doubt, abort. Easier said than done, I agree, but there you have it.
Ask questions and don't try to impress the charter with your skill. Rather, impress upon him the fact that you have concerns, aren't certain about your abilities and/or the conditions and might need extra help. If you don't know the site, the charter and the other divers, be extra cautious. If you think you might be getting a glad-handed back-slapping from a charter that is just looking to take your money, ask him about his insurance - that will almost always get him to stop talking for a minute and look closely at you.
As to gaining experience, the progression doesn't necessarily have to be slow but it does need to be steady and coherent. As japan-diver indicated, advancing in skill requires more diving than you can pack into a few dives on a weeklong trip. If time or money or interest conspire to keep you from diving more than occasionally, so be it. There is nothing wrong with being a once-a-year resort diver; if that's what you are, no amount of wishful thinking is going to make you an advanced diver.
The first step towards moving out of guppy status is to get trained through Rescue. IMNSHO, until you're Rescue certified, any class you take should be a Rescue prerequisite. No nitrox, no deep air, no night diving, no wreck diving, no nothing that doesn't lead directly to Rescue. You have to be able to take care of yourself and your buddy in an emergency. Depending upon your local conditions, a drysuit class might be an exception to this rule.
Past Rescue, training can be had from a variety of sources other than classes, although classes are usually the best place. Sometimes an alternate source could be a charter DM. It's imperative that this option not be assumed to be available and that arrangements are made with both the operator and the DM long before you reach the dock to have whatever special services you need provided. Plan on hiring your own private DM (and paying accordingly for both his/her time and expenses) and call it an introduction to local diving program, if need be. Another good source of training is a dive club. You can often find instructors, dive masters or just highly qualified buddies to share experiences and knowledge with - both on the surface and beneath.
The first step to expanding your limits is to know what your current ones are. Aside from making you a safer diver, knowing your limits allows you to create a realistic plan to develop the skills and gain the experience to become the diver you want to be.