You may not be a party boy, but you may end up right in the middle of it, due to logistics and timing.
The Full Party on Koh Phangan attracts a lot of people, which means the ferries will book up fast. The general route people do is either go to Koh Tao before or after Koh Phangan, usually after, to "chill" after the party. (Koh Tao has the reputation of being more "chill".) I've seen situations where people who didn't book ahead get stuck on Koh Tao for two days due to full ferries. (They were buying tickets on the day of, instead of ahead of time, either online or right after arrival.) Also, if there are ferry scheduling issues, the ferry company will prioritize getting people to Koh Phangan. (I've had to wait for two hours when a ferry broke and they ferried people directly to Koh Phangan from Koh Samui and then came back to do the Koh Phangan-Koh Tao run. If you're doing the Surat Thani or Chumphon route, Koh Tao is the first stop, so this scenario won't happen.)