I had an interesting experience a few years back. My husband and I got lost on the way to the dive site, and got into a furious argument about it. By the time we got there, everybody else was already getting in the water, because we were late. We geared up anyway, and tried to go diving. You should see the video my husband shot -- I look like a bad OW student. I called the dive about 15 minutes into it because things were just horrible. The only problem with the dive was that my head wasn't in it -- my head wasn't anywhere that had to do with diving.
If your head isn't in the game, you shouldn't be diving. You don't need any other reason. And you don't need to explain to anybody what the problem is. I think Ron is right that, if you lose out on a dive for some reason that's something you can address (getting more comfortable with night dives or current or something) that's fine. But I think you can "wimp out" just because you aren't feeling it. I'd MUCH rather have a buddy do that, than go diving when they aren't happy and focused on the dive.
Of course, if somebody develops a PATTERN of doing that repeatedly, I probably will stop asking that person to go diving with me . . .