If you made the agreement you need to stick to it. HOWEVER I wouldn't omit any required stops or go up fast, although I might omit/abbreviate the safety stop if I were genuinely worried. If you and/or your buddy really shouldn't be alone, but it's already happened nonetheless, better to be alone at 15' not 70' => so start up as soon as you've had a fair look around you to make sure surfacing is really necessary for reuniting.
If your buddy is very green or just a jittery person, he might get unreasonably much stressed unreasonabaly quickly, then do something unfortunate. At the least he might get all bent out of shape now, then bore the hell out of you about in the future.
Next time you will probably make a somewhat different agreement. Or you might make the very same agreement if you buddy is an anxious type, or someone who just is best not left alone for whatever reasons. Whatever you agree, think it through BEFORE saying "OK", then stick to it as best you can.