Walking in a BP&W without a wetsuit for padding not comfortable at all. The straps will dig in and it will be to painful to walk any distance, especially after you've been in the water awhile. I always wear something just for padding to prevent chaffing.
Years ago, this question was asked on Rodale's, and I was the only dissenter. When the poster came back from vacation with his new BP&W, he was upset that everyone told him that he could wear a plate with no suit.
Even a 1/2mm would probably be comfortable, but I think you'll find that you have to wear something.