I was recently diving in Myrtle Beach and got this explanation. In the old days, sailing craft transported bananas up the coast. People would go down to the beach after a big blow, and if they saw bananas that meant a ship had sunk in the storm.
Nobody got their bananas tossed overboard...