4000 calories a day is obscene? I think that number sounds low, given an average adult male would need somwhere between 2500 and 3000 calories a day. Just to keep your body functioning you tend to need 1500-2000 calories and thats without any activity.
People who exercise need a lot more and I think its safe to say Phelps, an olympic athlete, exercise.
If you want to talk obscene ammounts of calories, look to tour de france where the cyclists consume over 6000 calories a day and they still lose weight over the course of the tour. They actually struggle with stuffing their face well enough..
EDIT: Using myself as an example of the "average adult male" Im about 75kg, 178cm (165 lbs/5'10") wich makes my BMI 23,6 which is high end of ideal weight.
My BMR (Base metabolic rate) would then be ~1750 calories burned just being alive.
Without any activities accounted for this Mount Hægefjell - Elevation 1021m by Even.A.Lie at Garmin Connect - Details workout, which is good, but by no means extreme puts me well over 3000 calories needed and thats before the days even half done..
I believe Phelps' number was closer to 11000 calories, not 4000. I know when I was 22 and working out every day I ate 12000 calories per day (yes, for real) and gained exactly 4 pounds in a month and a half of working out that much. After that I decided I didn't enjoy the gym enough to spend 3+ hours there every day and brought my caloric intake back to a more reasonable number. I was still exercising a lot, though, so I was still around 6,000 per day. I have a ridiculously high natural metabolism, though. In the end, Freewillie's statement is what makes the most sense. Eat a balanced diet, exercise a reasonable amount and you will most likely be able to eat what you want occasionally.