My experience diving around Cape Charles: Limited.
I have made dives on the Texeco Wreck off Cape Charles Harbor. Visibility is extremely poor and currents can be very dangerous. I would stongly advise NOT diving around the concrete ships. With many of the breaks and holes around the vessels it would be easy to get siphoned in/out. Then there's all the fishing line and hooks.
Diving off the entrance to the Chesapeake Bay usually brings crappy vis. It always choppy where the Bay meets the Atlantic, no matter the tide or wind conditions. On the Eastern shore side you have a large shoal area (off Fisherman's Island) that stirs the sand and silt. I doubt you'll find even good diving off the beach.
As for Flat Fish: With the warmer water the boats are catching them closer in. Some very large Flounder are taken by the CBBT High Rise section of the span. Boats, Current, poor vis...not recommended either. I know some divers that dove recently on the Tiger (6 miles off Sand Bridge) and bagged their limit of very good size Flounder. You can bet they're all over the other wrecks too. If you've got a boat you can easily hit the Tower Reef area. It's got lots of things to hook into and dive on.
Safe Diving,
Wil